An Obstacle and (Hopefully) a Solution in Digital Research

This post is part of an ongoing series about my research on conspiracy theories and the tools I use to pursue it. You can read Part I: What is a Conspiracy Theory and Part II: Why Are Conspiracy Theories So Compelling? on Commons Knowledge.

Part of my research project, in which I am attempting to give an empirically-informed account of what constitutes a conspiracy theory, involves reading through a text that compiles a few hundred different conspiracy theories and gives brief accounts of them. By reading through them and coding for the presence of various features, I hoped to get some information on what features were most typical of conspiracy theories. My own suspicion is that an important part of the appeal of conspiracy theories is that, in general, we tend to find appeals to coincidence unconvincing. For example, if a student is repeatedly absent from class on test days and gives as an excuse a series of illnesses, we are inclined to find this unconvincing. It seems very coincidental that their illnesses always occur on test days. Of course, it’s possible that it really is a coincidence, but we strongly discount the explanatory weight of such an appeal. If we cast about for another theory to explain their absences, we quickly happen across another one: The student didn’t prepare for the tests and so wanted to avoid coming to class. Again, it’s possible this theory is incorrect, but it is much more satisfying than the theory that the student just coincidentally gets sick on test days.

I suspect that conspiracy theories derive much of their appeal from the unsatisfactory character of appeals to coincidence. To pick just one example: The plane crash that killed Senator Paul Wellstone in 2002 has been the focal point of a number of conspiracy theories. The standard account is that the crash was due to pilot error. One way that suspicion has been raised about this account is by noting a number of seeming coincidences. One coincidence is that Wellstone was one of the most outspoken voices against the Bush administration at the time. It raised some conspiracy theorists’ eyebrows that such a prominent liberal voice “just happened” to die unexpectedly in a plane crash. Alternative theories propose that members of the Bush administration arranged for the assassination of Wellstone. Another purported coincidence involved accounts of electronic malfunction: cell phones and automatic garage door openers in the vicinity supposedly malfunctioned at roughly the same time the plane crashed. This is accounted for in some conspiracy theories by appealing to the use of electromagnetic frequency weapon which disabled the controls of the plane, while also causing malfunction in nearby electronic equipment. My hypothesis is that this style of explanation and theory development is typical of conspiracy theories in general.

Federal investigators sift through debris in this Oct. 27, 2002 file photo, from the twin engine plane that crashed two days earlier near Eveleth, Minn,. killing Sen. Paul Wellstone, his wife Sheila, daughter Marcia and several others. The National Transportation Safety Board is ready to vote on the likely cause of the 2002 accident. (AP Photo/Jim Mone, File)

In my study, I have noted whether each conspiracy theory in my chosen compilation points to an appeal to coincidence in the rival (usually “standard”) account and, if it does, whether it then appeals to a conspiracy in order to provide a “better” theory to replace the one that appeals to coincidence. Unfortunately, this strategy has hit an obstacle. While there is a strong correlation between pointing to an appeal to coincidence as a problem with a theory and substituting an appeal to conspiracy in its place, there were relatively few theories that appeared to do this, based on the text. Even in cases where I knew the criticism of appeals to coincidence frequently played a large role in the justification of particular conspiracy theories, I often found no evidence of this in the brief accounts of the theories given in the book. It could be, of course, that my hypothesis is just mistaken; given that it didn’t match up with other conclusions I’d drawn based on other sources, however, I am inclined to think the problem is the source text. Thinking it over, it was clear that the nature of the text was to present the accounts “objectively,” stating the content of the views, normally without any effort to convince the reader one way or the other. Occasionally, talk of coincidences finds its way into the entries. Even then, it is only rarely explicit: for example, there are zero appearances of the word ‘coincidence’ or the phrase ‘just happened’, only three appearances of ‘coincidental’, and all appearances of ‘happened to’ are, upon checking the context, not related to appeals to coincidence in explanation. No other typically “coincidental” language makes a significant appearance. My concern is that this reveals only that my chosen text doesn’t address whether the presented theories are explanatorily superior to its rivals or explore how they developed in the first place. Since those are the areas in which coincidence would play a larger role, I’ve concluded that my chosen text is misleading as a source of data about conspiracy theories (at least with regard to the role of coincidence; in other areas, such as whether the theory is an official or unofficial account, it is much more reliable).

In order to resolve this obstacle, I have settled on using primary sources that are more likely to involve attempts to persuade the reader that the contained theory is correct and superior to its rivals. This includes books and websites that present a particular conspiracy theory and online fora where proponents of various conspiracy theories argue and collaborate in the development of conspiracy theories. This is obviously vastly larger than the single anthology I initially intended to use. My focus currently is finding a way to carve out a manageable chunk of this gigantic data set, most likely from online message boards, like Reddit, and use text from these fora to find evidence for my hypothesis about appeals to coincidence. This will necessitate the use of at least two kinds of digital techniques: web scraping, in order to extract usable text from a large number of individual websites, and topic modeling, in order to find meaningful relationships within an otherwise unmanageably large corpus. In my next post, I will talk about my initial forays into these techniques.

Introducing the Scholarly Commons Project Forum

A logo for the Scholarly Commons Project Forum.

The Scholarly Commons Project Forum is an hour-long bi-weekly meeting space for scholars who are interested in Digital Humanities questions regarding data and text. These meetings are an opportunity for informal, open-ended conversations about research where we will discuss conceptual, methodological, and workflow issues for projects. Those projects may be at any stage of development, whether still formative or largely complete. The goal is to think together about how to develop robust Digital Humanities research, whether as beginners interested in trying out DH techniques or those with more experience, and to make that research more legible to others.

These conversations will be facilitated by Interns at the Scholarly Commons, and will be held Mondays in Main Library 220 from 2:00-3:00 pm, starting March 5, and every two weeks following. Please RSVP to

Celebrating Frederick Douglass with Crowdsourced Transcriptions

A flier advertising the Transcribe-a-thon, which includes a photo of Frederick Douglass

On February 14, 2018, the world celebrated Frederick Douglass’ 200th birthday. Douglass, the famed Black social reformer, abolitionist, writer and statesman, did not know the date of his birth, and chose the date of Februar


y 14, 1818 to celebrate his birthday. This year, to celebrate the 200th anniversary of his birth, Colored Conventions, the Smithsonian Transcription Center, and the National Museum of African American History & Culture partnered together to host a Transcribe-a-thon of the Freedmen’s Bureau Papers in Douglass’ honor.

The Freedmen’s Bureau Papers consist of 2 million digitized papers through a partnership between the Smithsonian Transcription Center and the National Museum of African American History and Culture. It is the largest crowdsourcing initiative ever hosted by the Smithsonian. The Freedmen’s Bureau helped solve the everyday problems of formerly enslaved individuals, from obtaining clothing and food to helping find lost family members. The Bureau operated from 1865-1872 and closed due to opposition from Congress and President Andrew Johnson.

The Transcribe-a-thon was held on February 14th from 12-3 PM EST. According to the Smithsonian Transcription Center, over 779 pages of the Freedmen’s Bureau Papers were transcribed during this time, 402 pages were reviewed and approved, and 600 new volunteers registered for the project. Over sixty institutions hosted Transcribe-a-thon locations, many of which bought birthday cakes in Douglass’ honor from African American-owned bakeries in their area. Meanwhile, Colored Conventions livestreamed participants during the event.If you’re interested in seeing more from Douglass Day 2018, check out the Smithsonian Transcription Center’s Twitter Moment.

The Douglass Day Transcribe-a-thon was a fantastic example of people coming together and doing fantastic digital humanities work together, and for a great cause. While crowdsourced transcription projects are not new, the enthusiasm for Douglass Day is certainly unique and infectious, and we’re so excited to see where this project goes in the future and to get involved ourselves!


Digital Timeline Tools

Everyone has a story to tell. For many of us doing work in the humanities and social sciences, presenting our research as a timeline can bring it new depth and a wider audience. Today, I’ll be talking about two unique digital storytelling options that you can use to add dimension to your research project.


An image of Timeglider's sample timeline on the Wright Brothers

Timeglider is an interactive timeline application. It allows you to move in and out time, letting you see time in large or small spans. It also allows events to overlap, so you can show the relationship of things in time. Timeglider also gives some great aesthetic options, including what they call their “special sauce” — the way they relate the size of an event to its importance. This option emphasizes certain events in the timeline to the user, and can make getting important ideas across simpler.

Started in 2002 as a flash-based app, Timeglader is one of the older timeline options on the web. After a major redesign in 2010, Timeglider is now written in HTML5 and JavaScript. Timeglider is free for students for a basic package, and plans for non-students can choose to pay either $5/month or $50/year.

Overall, Timeglider is an interesting timeline application with numerous options. Give it a try!


A screenshot from a myHistro project on the Byzantine Empire.

myHistro uses text, video and pictures on maps and timelines to tell stories. Some of the power of myHistro comes from the sheer amount of information you can provide in one presentation. Presentations can include introductory text, an interactive timeline, a Google Maps-powered annotated map, and a comment section, among other attributes. The social aspect, in particular, makes myHistro powerful. You can open your work up to a large audience, or simply ask students and scholars to make comments on your work for an assignment. Another interesting aspect of myHistro is the sheer amount of projects people have come up with for it. There is everything from histories of the French Revolution to the biography of Justin Bieber, with everything in between!

myHistro is free, and you can sign up using your email or social network information.

Meet Dan Tracy, Information Sciences and Digital Humanities Librarian

This latest installment of our series of interviews with Scholarly Commons experts and affiliates features Dan Tracy, Information Sciences and Digital Humanities Librarian.

What is your background and work experience?

I originally come from a humanities background and completed a PhD in literature specializing in 20th century American literature, followed by teaching as a lecturer for two years. I had worked a lot with librarians during that time with my research and teaching. When you’re a PhD student in English, you teach a lot of rhetoric, and I also taught some literature classes. As a rhetoric instructor I worked closely with the Undergraduate Library’s instruction services, which exposed me to the work librarians do with instruction.

Then I did a Master’s in Library and Information Science here, knowing that I was interested in being an academic librarian, probably something in the area of being a subject librarian in the humanities. And then I began this job about five years ago. So I’ve been here about five years now in this role. And just began doing Digital Humanities over the summer. I had previously done some liaison work related to digital humanities, especially related to digital publishing, and I had been doing some research related to user experience and digital publishing as related to DH publishing tools.

What led you to this field?

A number of things. One was having known quite a number of people who went into librarianship who really liked it and talked about their work. Another was my experience working with librarians in terms of their instruction capacity. I was interested in working in an academic environment and I was interested in academic librarianship and teaching. And also, especially as things evolved, after I went back for the degree in library and information science, I also found a lot of other things to be interested in as well, including things like digital humanities and data issues.

What is your research agenda?

My research looks at user experience in digital publishing. Primarily in the context of both ebook formats and newer experimental forms of publication such as web and multi-modal publishing with tools like Scalar, especially from the reader side, but also from the creator side of these platforms.

Do you have any favorite work-related duties?

As I mentioned before, instruction was an initial draw to librarianship. I like anytime I can teach and work with students, or faculty for that matter, and help them learn new things. That would probably be a top thing. And I think increasingly the chances I get to work with digital collections issues as well. I think there’s a lot of exciting work to do there in terms of delivering our digital collections to scholars to complete both traditional and new forms of research projects.

What are some of your favorite underutilized resources that you would recommend to researchers?

I think there’s a lot. I think researchers are already aware of digital primary sources in general, but I do think there’s a lot more for people to explore in terms of collections we’ve digitized and things we can do with those through our digital library, and through other digital library platforms, like DPLA (Digital Public Library of America).

I think that a lot of our digital image collections are especially underutilized. I think people are more aware that we have digitized text sources, but not aware of our digitized primary sources that are images that have value of research objects, including analyzed computational analysis. We also have more and more access to the text data behind our various vendor platforms, which is a resource various researchers on campus increasingly need but don’t always know is available.

If you could recommend one book to beginning researchers in your field, what would you recommend?

If you’re just getting started, I think a good place to look is at the Debates in the Digital Humanities books, which are collections of essays that touch on a variety of critical issues in digital humanities research and teaching. This is a good place to start if you want to get a taste of the ongoing debates and issues. There are open access copies of them available online, so they are easy to get to.

Dan Tracy can be reached at

Why Are Conspiracy Theories So Compelling?

In my last post, I described the first phase of my research, in which I am attempting to develop an empirically informed definition of ‘conspiracy theory’. In this post, I want to discuss the second focus of my research: why it is that conspiracy theories are so compelling for so many people.

Newspaper article with headline "Kennedy Slain by CIA, Mafia, Castro, LBJ, Teamsters, Freemasons"Although the specifics can be debated, it is clear that conspiracy theories are very popular. In a recent survey, 61% of participants claimed belief in some form of a conspiracy theory about the assassination of John F. Kennedy. This could possibly be attributed to increased publicity about the event due to its impending fiftieth anniversary and coverage of the release of some previously classified documents regarding it. But in an even more wide-ranging study four years ago, the number was 51%. At the very least, it looks plausible that more than half of Americans believe in this particular conspiracy theory, and there are plenty of other theories out there. For example, approximately 40% of respondents endorsed the conspiracy theory that the FDA is withholding a natural cancer cure.

Conspiracy theories are often treated dismissively as the ravings of deranged paranoiacs. Yet, we have good reason to believe that a majority of Americans believe in at least one conspiracy theory, and we can’t dismiss all of them in this way. Why, then, are conspiracy theories so compelling? There are a number of predictors for belief in conspiracy theories. The best is belief in other conspiracy theories: if someone believes one conspiracy theory, the likelihood that they believe another goes up. Other predictors are useful for predicting if a subject believes in a particular conspiracy theory, but not for the likelihood that they believe in conspiracy theories generally. Belief in conspiracy theories is common regardless of race, but white Americans are more likely than African-Americans to believe in Sandy Hook conspiracy theories (in which the government supposedly faked the Sandy Hook shooting in order to initiate more stringent gun control laws), while African-Americans are more likely than white Americans to believe that the CIA developed AIDS in order to kill African-American populations. Similarly, political liberals are more likely to endorse GMO conspiracy theories, while political conservatives are more likely to endorse climate change conspiracy theories. Evidence does suggest that people are less inclined to believe in conspiracy theories the more educated they are, but exactly why this is the case is still unclear. Higher education is correlated with a complex of many other facts and it remains to be seen whether the education itself is the cause of decreased belief.

My own suspicion is that an important part of the appeal of conspiracy theories is that we tend to find appeals to coincidence unconvincing. This is often perfectly reasonable. If a recently-elected politician installs close friends and family to all important posts, insisting that, by coincidence, their friends and family were the most qualified individuals for the posts, we will be rightly suspicious. It can be a problem, however, when this suspicion transfers over to extraordinarily complex events. For example, there is a long-standing conspiracy theory that Bill Clinton arranged for the assassination of dozens of people with whom he had varying levels of contact. An enormous part of the appeal stems from the seeming unlikelihood of so many deaths that can be linked to Clinton. Of course, a president comes into contact with a staggering number of people, and some small number of these are bound to die in a variety of ways. It is not surprising that a number of people who met Clinton died; it is merely coincidental, and what would really be surprising is if no one who he met died. When a case is sufficiently complex (such as the network of everyone a United States president meets), coincidence will often be the explanation for events.

An image titled "The Clinton Body Bags" that lists people Bill Clinton came in contact with who are now dead.There are other cases where “conspiratorial thinking,” in which we are inclined to suspect agency is the cause of an event rather than coincidence, seems appropriate. It seems appropriate that homicide detectives presume agency was involved rather than coincidence when investigating an unexpected death, and that they ask questions like “Who would benefit from this?” in determining what agency was at work. On the other hand, it seems inappropriate that a voter should presume agency rather than coincidence was involved in explaining why a former member of the president’s staff died in a plane crash, and should not ask questions like “Who would benefit from this?” in order to discover who might have arranged the disaster.

Conspiratorial thinking, utilized in the appropriate circumstances, is a powerful tool that allows us to discount appropriately explanations that are, in other circumstances, much more plausible. When applied in inappropriate circumstances, on the other hand, conspiratorial thinking can metastasize and overwhelm our rational thinking. For instance, someone nearly always benefits from any event, so that asking “Who would benefit from this?” will nearly always yield a suspect. Without compelling reason to suspect agency in the first place, it is important to refrain from asking the question. My hope is to run a series of psychological studies to see whether people who believe in conspiracy theories are also more suspicious of coincidence as an explanation in general.

In my next post, I’ll talk some about some difficulties I’ve had running the initial portion of this study, as well as talk a bit about the digital tools I’m using.

Announcing Topic Modeling – Theory & Practice Workshops

An example of text from a topic modeling project.We’re happy to announce that Scholarly Commons intern Matt Pitchford is teaching a series of two Savvy Researcher Workshops on Topic Modeling. You may be following Matt’s posts on Studying Rhetorical Responses to Terrorism on Twitter or Preparing Your Data for Topic Modeling on Commons Knowledge, and now is your chance to learn the basics from the master! The workshops  will be held on Wednesday, December 6th and Friday, December 8th. See below for more details!

Topic Modeling, Part 1: Theory

  • Wednesday, December 6th, 11am-12pm
  • 314 Main Library
  • Topic models are a computational method of identifying and grouping interrelated words in any set of texts. In this workshop we will focus on how topic models work, what kinds of academic questions topic models can help answer, what they allow researchers to see, and what they can obfuscate. This will be a conversation about topic models as a tool and method for digital humanities research. In part 2, we will actually construct some topic models using MALLET.
  • To sign up for the class, see the Savvy Researcher calendar

Topic Modeling, Part 2: Practice

  • Friday, December 8th, 11am-12pm
  • 314 Main Library
  • In this workshop, we will use MALLET, a java based package, to construct and analyze a topic model. Topic models are a computational method of identifying and grouping interrelated words in any set of text. This workshop will focus on how to correctly set up the code, understand the output of the model, and how to refine the code for best results. No experience necessary. You do not need to have attended Part I in order to attend this workshop.
  • To sign up for this class, see the Savvy Researcher calendar

Save the Date: Edward Ayers Talk


We are so excited to be hosting a talk by Edward Ayers this coming March! Save the date on your calendars:

March 29, 2018 | 220 Main Library | 4-6 pm

Edward Ayers has been named National Professor of the Year, received the National Humanities Medal from President Obama at the White House, won the Bancroft Prize and Beveridge Prize in American history, and was a finalist for the National Book Award and the Pulitzer Prize. He has collaborated on major digital history projects including the Valley of the Shadow, American Panorama, and Bunk, and is one of the co-hosts for BackStory, a popular podcast about American history. He is Tucker-Boatwright Professor of the Humanities and president emeritus at the University of Richmond as well as former Dean of Arts and Sciences at the University of Virginia. His most recent book is The Thin Light of Freedom: The Civil War and Emancipation in the Heart of America, published in 2017 by W. W. Norton.

His talk will be on “Twenty-Five Years in Digital History and Counting”.

Edward Ayers began a digital project just before the World Wide emerged and has been pursuing one project or several projects ever since. His current work focuses on the two poles of possibility in the medium: advanced projects in visualizing processes of history at the Digital Scholarship Lab at the University of Richmond and a public-facing project in Bunk, curating representations of the American past for a popular audience.

We hope you’ll be able to join us at his public talk in March!

Open Source Tools for Social Media Analysis

Photograph of a person holding an iPhone with various social media icons.

This post was guest authored by Kayla Abner.

Interested in social media analytics, but don’t want to shell out the bucks to get started? There are a few open source tools you can use to dabble in this field, and some even integrate data visualization. Recently, we at the Scholarly Commons tested a few of these tools, and as expected, each one has strengths and weaknesses. For our exploration, we exclusively analyzed Twitter data.


NodeXL’s graph for #halloween (2,000 tweets)

tl;dr: Light system footprint and provides some interesting data visualization options. Useful if you don’t have a pre-existing data set, but the one generated here is fairly small.

NodeXL is essentially a complex Excel template (it’s classified as a Microsoft Office customization), which means it doesn’t take up a lot of space on your hard drive. It does have advantages; it’s easy to use, only requiring a simple search to retrieve tweets for you to analyze. However, its capabilities for large-scale analysis are limited; the user is restricted to retrieving the most recent 2,000 tweets. For example, searching Twitter for #halloween imported 2,000 tweets, every single one from the date of this writing. It is worth mentioning that there is a fancy, paid version that will expand your limit to 18,000, the maximum allowed by Twitter’s API, or 7 to 8 days ago, whichever comes first. Even then, you cannot restrict your data retrieval by date. NodeXL is a tool that would mostly be most successful in pulling recent social media data. In addition, if you want to study something besides Twitter, you will have to pay to get any other type of dataset, i.e., Facebook, Youtube, Flickr.

Strengths: Good for a beginner, differentiates between Mentions/Retweets and original Tweets, provides a dataset, some light data visualization tools, offers Help hints on hover

Weaknesses: 2,000 Tweet limit, free version restricted to Twitter Search Network


TAGSExplorer’s data graph (2,902 tweets). It must mean something…

tl;dr: Add-on for Google Sheets, giving it a light system footprint as well. Higher restriction for number of tweets. TAGS has the added benefit of automated data retrieval, so you can track trends over time. Data visualization tool in beta, needs more development.

TAGS is another complex spreadsheet template, this time created for use with Google Sheets. TAGS does not have a paid version with more social media options; it can only be used for Twitter analysis. However, it does not have the same tweet retrieval limit as NodeXL. The only limit is 18,000 or seven days ago, which is dictated by Twitter’s Terms of Service, not the creators of this tool. My same search for #halloween with a limit set at 10,000 retrieved 9,902 tweets within the past seven days.

TAGS also offers a data visualization tool, TAGSExplorer, that is promising but still needs work to realize its potential. As it stands now in beta mode, even a dataset of 2,000 records puts so much strain on the program that it cannot keep up with the user. It can be used with smaller datasets, but still needs work. It does offer a few interesting additional analysis parameters that NodeXL lacked, such as ability to see Top Tweeters and Top Hashtags, which works better than the graph.

These graphs have meaning!

Strengths: More data fields, such as the user’s follower and friend count, location, and language (if available), better advanced search (Boolean capabilities, restrict by date or follower count), automated data retrieval

Weaknesses: data visualization tool needs work


Simple interface for Documenting the Now’s Hydrator

tl;dr: A tool used for “re-hydrating” tweet IDs into full tweets, to comply with Twitter’s Terms of Service. Not used for data analysis; useful for retrieving large datasets. Limited to datasets already available.

Documenting the Now, a group focused on collecting and preserving digital content, created the Hydrator tool to comply with Twitter’s Terms of Service. Download and distribution of full tweets to third parties is not allowed, but distribution of tweet IDs is allowed. The organization manages a Tweet Catalog with files that can be downloaded and run through the Hydrator to view the full Tweet. Researchers are also invited to submit their own dataset of Tweet IDs, but this requires use of other software to download them. This tool does not offer any data visualization, but is useful for studying and sharing large datasets (the file for the 115th US Congress contains 1,430,133 tweets!). Researchers are limited to what has already been collected, but multiple organizations provide publicly downloadable tweet ID datasets, such as Harvard’s Dataverse. Note that the rate of hydration is also limited by Twitter’s API, and the Hydrator tool manages that for you. Some of these datasets contain millions of tweet IDs, and will take days to be transformed into full tweets.

Strengths: Provides full tweets for analysis, straightforward interface

Weaknesses: No data analysis tools

Crimson Hexagon

If you’re looking for more robust analytics tools, Crimson Hexagon is a data analytics platform that specializes in social media. Not limited to Twitter, it can retrieve data from Facebook, Instagram, Youtube, and basically any other online source, like blogs or forums. The company has a partnership with Twitter and pays for greater access to their data, giving the researcher higher download limits and a longer time range than they would receive from either NodeXL or TAGS. One can access tweets starting from Twitter’s inception, but these features cost money! The University of Illinois at Urbana-Champaign is one such entity paying for this platform, so researchers affiliated with our university can request access. One of the Scholarly Commons interns, Matt Pitchford, uses this tool in his research on Twitter response to terrorism.

Whether you’re an experienced text analyst or just want to play around, these open source tools are worth considering for different uses, all without you spending a dime.

If you’d like to know more, researcher Rebekah K. Tromble recently gave a lecture at the Data Scientist Training for Librarians (DST4L) conference regarding how different (paid) platforms influence or bias analyses of social media data. As you start a real project analyzing social media, you’ll want to know how the data you have gathered may be limited to adjust your analysis accordingly.

Preparing Your Data for Topic Modeling

In keeping with my series of blog posts on my research project, this post is about how to prepare your data for input into a topic modeling package. I used Twitter data in my project, which is relatively sparse at only 140 characters per tweet, but the principles can be applied to any document or set of documents that you want to analyze.

Topic Models:

Topic models work by identifying and grouping words that co-occur into “topics.” As David Blei writes, Latent Dirichlet allocation (LDA) topic modeling makes two fundamental assumptions: “(1) There are a fixed number of patterns of word use, groups of terms that tend to occur together in documents. Call them topics. (2) Each document in the corpus exhibits the topics to varying degree. For example, suppose two of the topics are politics and film. LDA will represent a book like James E. Combs and Sara T. Combs’ Film Propaganda and American Politics: An Analysis and Filmography as partly about politics and partly about film.”

Topic models do not have any actual semantic knowledge of the words, and so do not “read” the sentence. Instead, topic models use math. The tokens/words that tend to co-occur are statistically likely to be related to one another. However, that also means that the model is susceptible to “noise,” or falsely identifying patterns of cooccurrence if non-important but highly-repeated terms are used. As with most computational methods, “garbage in, garbage out.”

In order to make sure that the topic model is identifying interesting or important patterns instead of noise, I had to accomplish the following pre-processing or “cleaning” steps.

  • First, I removed the punctuation marks, like “,.;:?!”. Without this step, commas started showing up in all of my results. Since they didn’t add to the meaning of the text, they were not necessary to analyze.
  • Second, I removed the stop-words, like “I,” “and,” and “the,” because those words are so common in any English sentence that they tend to be over-represented in the results. Many of my tweets were emotional responses, so many authors wrote in the first person. This tended to skew my results, although you should be careful about what stop words you remove. Simply removing stop-words without checking them first means that you can accidentally filter out important data.
  • Finally, I removed too common words that were uniquely present in my data. For example, many of my tweets were retweets and therefore contained the word “rt.” I also ended up removing mentions to other authors because highly retweeted texts tended to mean that I was getting Twitter user handles as significant words in my results.

Cleaning the Data:

My original data set was 10 Excel files of 10,000 tweets each. In order to clean and standardize all these data points, as well as combining my file into one single document, I used OpenRefine. OpenRefine is a powerful tool, and it makes it easy to work with all your data at once, even if it is a large number of entries. I uploaded all of my datasets, then performed some quick cleaning available under the “Common Transformations” option under the triangle dropdown at the head of each column: I changed everything to lowercase, unescaped HTML characters (to make sure that I didn’t get errors when trying to run it in Python), and removed extra white spaces between words.

OpenRefine also lets you use regular expressions, which is a kind of search tool for finding specific strings of characters inside other text. This allowed me to remove punctuation, hashtags, and author mentions by running a find and replace command.

  • Remove punctuation: grel:value.replace(/(\p{P}(?<!’)(?<!-))/, “”)
    • Any punctuation character is removed.
  • Remove users: grel:value.replace(/(@\S*)/, “”)
    • Any string that begins with an @ is removed. It ends at the space following the word.
  • Remove hashtags: grel:value.replace(/(#\S*)/,””)
    • Any string that begins with a # is removed. It ends at the space following the word.

Regular expressions, commonly abbreviated as “regex,” can take a little getting used to in order to understand how they work. Fortunately, OpenRefine itself has some solid documentation on the subject, and I also found this cheatsheet valuable as I was trying to get it work. If you want to create your own regex search strings, has a tool that lets you test your expression before you actually deploy it in OpenRefine.

After downloading the entire data set as a Comma Separated Value (.csv) file, I then used the Natural Language ToolKit (NLTK) for Python to remove stop-words. The code itself can be found here, but I first saved the content of the tweets as a single text file, and then I told NLTK to go over every line of the document and remove words that are in its common stop word dictionary. The output is then saved in another text file, which is ready to be fed into a topic modeling package, such as MALLET.

At the end of all these cleaning steps, my resulting data is essentially composed of unique nouns and verbs, so, for example, @Phoenix_Rises13’s tweet “rt @drlawyercop since sensible, national gun control is a steep climb, how about we just start with orlando? #guncontrolnow” becomes instead “since sensible national gun control steep climb start orlando.” This means that the topic modeling will be more focused on the particular words present in each tweet, rather than commonalities of the English language.

Now my data is cleaned from any additional noise, and it is ready to be input into a topic modeling program.

Interested in working with topic models? There are two Savvy Researcher topic modeling workshops, on December 6 and December 8, that focus on the theory and practice of using topic models to answer questions in the humanities. I hope to see you there!