PART III ][ Chapter 19

Teaching Quantitative Methods: What Makes It Hard (in Literary Studies)

Andrew Goldstone

When I set out to teach a graduate English course titled “Literary Data: Some Approaches” in the spring of 2015, I was on a mission.[1] I wanted my eleven PhD students to learn not simply how to talk about digital humanities but also how to analyze data as part of their literary scholarship, to be able to argue not only about “data” but also with data. I wanted to prove that English graduate students could do more than play with computers in their first DH course. At the same time, I wanted students to acquire the conceptual sophistication that would make their practical knowledge meaningful. Though my students made remarkable practical and conceptual progress, at the end of the semester my high-flown aims still seemed to lie beyond our immediate grasp. Having made the attempt, however, I learned some lessons about what is needed to take on this pedagogical mission: lessons not only in course planning but also about what the scholarly community has to do—and what institutions must be prepared to supply—if quantitative methods are to fulfill their promise for the study of literature and culture.

There is, of course, more to the digital humanities than doing data analysis for literary study. Aside from a disciplinary slant toward my home field of literature in English, my course’s emphasis on data analysis meant giving relatively less attention to other prominent DH strands; for example, new media studies or digital editing and curation. Nonetheless, questions of what constitutes literary data and how it can be analyzed are central to DH discussions. The digital humanities has garnered so much attention and debate across humanistic scholarship in no small part because of the claim that studying aggregates of texts with the help of quantification might radically change our conception of literary and cultural history. The potential of aggregation is the reason I have participated in the digital humanities. And if it really does open up major new possibilities for research, then teaching the theory and method of literary data analysis to future researchers is something to which at least some literary scholars ought to devote serious energy.

My argument, in brief, is that teaching this material is really, really hard, for reasons that are more than technical or technological. The available strategies for teaching literary data analysis under the “DH” rubric, including my own, have so far been inadequate to the task of training scholars in research methods. In what follows, I summarize how I approached this pedagogical challenge in my course. But rather than offer my course as a model, I instead draw the following prescriptive lessons from my experience:

1. 1. Cultivating technical facility with computer tools—including programming languages—should receive less attention than methodologies for analyzing quantitative or aggregative evidence. Despite the widespread DH interest in the former, it has little scholarly use without the latter.
2. 2. Studying method requires pedagogically suitable material, but good teaching datasets do not exist. It will require communal effort to create them on the basis of existing research.
3. 3. Following the “theory” model, digital humanities has typically been inserted into curricula as a single-semester course. Yet as a training in method, the analysis of aggregate data will undoubtedly require more time and a different rationale from that offered by what Gerald Graff calls “the field-coverage principle” in the curriculum.[2]

These issues are not only pedagogical. I learned with my students that present-day literary scholarship is fundamentally uncertain about how to make a convincing scholarly argument using quantitative evidence, and it has a habit of postponing or avoiding methodological debate in the name of open-ended exploration or disciplinary pluralism.[3] If a more sustained research program including quantitative methods is to develop, those dispositions may be reaching the end of their usefulness.

Teaching “Literary Data”

Since I aimed for my students to learn practical skill and conceptual sophistication, I decided to divide each week’s three-hour meeting in two: half the time for seminar discussion of theoretical readings and half the time for practicum.[4] The major argument of the theory part was that the question of literary data is not restricted either to digital humanities or to its most obvious predecessor, humanities computing, but should be set in a longer and broader multidisciplinary trajectory. For example, we read Lévi-Strauss’s structuralist theory of myth as a transformation of narratives into data; everyone was immediately interested in his system of index cards for tracking mythemes.[5] We looked at McKenzie on scientific bibliography in “Printers of the Mind”; we had a lively discussion of one of Robert Darnton’s earliest essays on the history of the book. Turning more explicitly to literary study, our discussion of J. F. Burrows’s Computation into Criticism was particularly productive, not so much because his version of computational text analysis struck any of us as compelling, but because it presented a limit-case my students and I came back to over and over again: Burrows, fully committed to criticism in the sense of the exercise of judgment, claims that the quantitative patterns of word use he finds reveal Jane Austen’s authorial “genius.”[6] But we were also able to see Burrows, and the stylometric tradition he builds on, as only one among many possible models for a literary data analysis.

Our attention then turned to more contemporary work in digital humanities, with short side excursions into contemporary sociology of culture. In our discussions of recent work, two themes recurred: first, my students’ excitement and enthusiasm for the novel possibilities opened up in recent work in studying large-scale aggregates of texts; and second, their dissatisfaction with the way this scholarship has analyzed and interpreted its data. More than once, my students wished they could study the data being discussed in our readings themselves. They were looking not just for provocative claims—which we found and argued about in abundance—but also for models of research that they themselves could build on. I return to this fundamental point later.

The other half of the course was the practicum. My students were to learn R well enough to be able to prepare real-world, messy data for analysis and to describe it numerically and visually in basic but interesting ways. But I also wanted them to think programmatically, or, as I put it under “learning goals” on the syllabus, “understand the fundamentals of computation.” I know only one general way to teach this sort of skill, and that is through cumulative practice in which skills are used, tested, and reinforced. From this followed the improbable scenario of me devising, and English PhD students completing, weekly problem sets, following, more or less, the progression of a typical basic introductory course in computation.[7] I adopted Jockers’s Text Analysis with R for Students of Literature as a starting point, but used only parts of the book and added many exercises of my own. Fairly soon, as the challenges built up, I found myself needing to take class time to go over homework questions; since I was also presenting new material in the practicum, this half of the class was frequently overloaded. The seminar discussion felt positively relaxed by comparison.

All this led up to the final challenge of the course: a data analysis report.[8] Halfway through the term, my students formed pairs (and one triple). Then, over the succeeding weeks, I worked with them as they obtained a dataset, analyzed it, and used their findings to produce argumentative papers with code and data appendices. Each student wrote his or her own report on the basis of the group’s work. The folder containing these reports occupies more than seven gigabytes on my hard drive, representing analyses of regionalism in American newspapers; of deaths and absences in the eighteenth-century slave trade; of authorship in science-fiction pulps; of the reception of David Mitchell’s Cloud Atlas; and of the influences of and on Freud in turn-of-the-century writing. As this list suggests, my students’ interests were quite varied, and I encouraged them to apply quantitative methods to whatever topic interested them.[9]

The rationale for the assignment was to confront the class with making sense of data in the wild, with all the messiness that entails. My students’ reports bear witness that they all learned that getting the data into analyzable form—“cleaning” it—is a large proportion of the work of analysis. Here, their efforts at mastering R really bore fruit, allowing them to wrestle big bodies of data into sense. (There were more than a few computer crashes on the way, and one student’s hard drive filled to capacity in the course of corpus processing.) Everyone found interesting phenomena to comment on in the data, and everyone put particular effort into visualizing these phenomena thoughtfully.[10]

But, after an exhaustingly difficult semester for both my students and me, what of the hoped-for payoff—the promise of moving beyond experimenting with the computer and toward making quantitatively evidenced arguments? On the theoretical plane, a very healthy caution about drawing conclusions from the data prevailed, even as the students strove to respond to my demand that they make literary-historical arguments. I was gratified by the thoughtfulness—and the quite awesome amount of labor—represented in the reports. At the same time, it was clear that my students did not have all the methodological tools they needed to draw the conclusions they wanted. Most of the projects, for example, involved spotting trends over time: trends in word usage, in network clustering, or in some other phenomenon. But how could they be sure a trend is real and not a random fluctuation? Many also sought to explain contrasts among subsets of their data. But how far could the data allow them to assess the validity of such explanations? Other students needed more possibilities for choosing data to collect in the first place, but how were they to know which choices had a chance of yielding meaningful conclusions?

These questions—the classic questions of quantitative methodology—remained open in the final reports. But this is no criticism of my students who, as a group, did all I could have asked. Rather, it is a criticism of my pedagogy, which did not equip them with all they needed. Though no doubt my personal shortcomings and all the hiccups of a first-time course contributed to this problem, I think solving it is not simply a matter of tweaking the syllabus or assignments. Instead, it requires a reorientation of the course toward the aim of teaching methodology, by which I mean the formal study of the questions I have just posed about how to interpret data. But to do that raises the fundamental pedagogical problems I mentioned at the start of this chapter.

Problem 1: Programming Is Not an End in Itself

An informal consensus seems to have emerged that if students in the humanities are going to make use of quantitative methods, they should probably first learn to program. Introductions to this dimension of the field are organized around programming languages: The Programming Historian is built around an introduction to Python; Matthew Jockers’s Text Analysis with R is at its heart a tutorial in the R language; Taylor Arnold and Lauren Tilton’s Humanities Data in R begins with chapters on the language; Folgert Karsdorp’s Python Programming for the Humanities is a course in the language with examples from stylometry and information retrieval.[11] “On the basis of programming,” writes Moretti in “Literature, Measured,” a recent retrospective on the work of his Literary Lab, “much more becomes possible” (2).

Steven Ramsay makes one of the best-known cases for focusing on programming in DH teaching. In a 2012 essay, he describes a course that “proceeds, as most courses in programming do, through the major constructs and concepts germane to any programming language,” but that nonetheless has a specific rationale for humanists: “the particular form of engagement that characterizes the act of building tools, models, frameworks and representations for the traditional objects of humanistic study.” Here is the kernel of a broad justification for the study of programming in humanities fields, which I found appealing when I first planned my course: learning to program promises to open the door to sophisticated thinking about the scholarly uses of data in our disciplines. At my most idealistic, I even imagined that programming, by making abstractions more concrete, would reintroduce my humanist students to the beauties of mathematics.

The conceptual reward, furthermore, is in principle coupled to a practical one: programming means not being limited to the possibilities offered by any given all-in-one application or “tool.” Much DH instruction has forgone programming in favor of guiding students in the use of particular specialized programs.[12] A student who learns to operate the text-analysis web application Voyant Tools, for example, can generate its (long) list of tabulations and visualizations of a text corpus, but no others: the list of visualizations and possible dimensions of variation are closed to nonprogrammers. Additionally, Voyant makes a significant set of choices about how to convert the text into units of analysis, with no possibility of alternatives beyond the choice of stop words.[13] And if the project goes the way of most DH projects and stops being updated or maintained, what are students who have depended on it to do? By contrast, a student who learns to program in an established programming language has the ability to generate any tabulation or visualization of texts that she can dream of; she can rely on the relative longevity of the platform; and, most importantly, she has the means to transform her texts and other data as required to make her subsequent analysis meaningful.

In principle. In practice—and I had to teach programming to understand this—the distinction between the tool user and the heroically flexible programmer is not so clear. Beginning programmers typically begin by following recipes, modifying them only little by little as they learn what can and cannot be done in the language. The first thrill of programming, making the computer do something, is a tool-user’s thrill. And beginning data analysts cannot know the range of possible methods they might use. They know what they have studied, and especially in the early stages, they have to spend time simply learning how to apply the techniques they have been shown, mastering whatever particular steps are needed to use the technique in the programming language they are learning. It is no accident that the tutorial form is so ubiquitous in DH instruction. Chapter 3 of Jockers’s Text Analysis, for example, is a tutorial in finding the most frequent words in Moby Dick: the whole of the chapter is devoted to explaining the particular R steps necessary to go from the digitized text to the table (and graph) of words and frequencies. This is followed by exercises in finding the top ten words in Sense and Sensibility. What students are meant to learn is just precisely how to extract a table of top ten words and their frequencies. When I added my own exercises to this chapter, I found myself again asking students to make lists of “top words” in other texts. And this same process of extracting “top words” reappeared in my students’ final projects.[14] The infinite generality of programming does not survive classroom necessity.

Yet the problem is not that DH textbooks—or my own course—do not instill the ability to write sophisticated, original programs. In fact, judging from my students’ program code in their final analysis projects, either because or in spite of my methods, they made good progress in acquiring this ability. Anyone who completes Jockers’s textbook, Karsdorp’s lessons, or Arnold and Tilton’s book will have made strides as a programmer and will know, in a hands-on way, the elementary principles of both algorithms and data structures. But programming competence is not equivalent to competence in analytical methods. It might allow students to prepare data for some future analysis and to produce visual, tabular, numerical, or even interactive summaries; Humanities Data in R gives a fuller survey of the possibilities of exploratory data analysis than the other texts.[15] Yet students who have focused on programming will have to rely on their intuition when it comes to interpreting exploratory results. Intuition gives only a weak basis for arguing about whether apparent trends, groupings, or principles of variation are supported by the data. Without any sense of these possibilities, which are the stock in trade of statistical methodology in the social sciences, a programming curriculum can bring students—but really, I should say, it has brought digital humanities as a field—only up to the threshold of method and not over it.

Problem 2: Any Old Data Will Not Do

In the words of Miriam Posner, “it’s just awful trying to find a humanities dataset.” Until I taught my course, I did not anticipate just how hard it would be to find suitable datasets for study. I made the internet-age mistake of thinking all sorts of interesting things must already be out there for the downloading; failing that, I imagined that I would be able to construct interesting data sets out of big grab-bag text corpora, such as Project Gutenberg, the text section of the Internet Archive, and the Text Creation Partnership releases from Eighteenth-Century Collections Online (ECCO) and Early English Books Online (EEBO). I was thinking like an English professor: when I teach literature, I can normally choose a dozen or so interesting books and wait for students to find their own analyses. Students never fail, in the course of reading and discussion, to discover both challenging questions and meaningful evidence to address those questions. But an arbitrary corpus of texts is a different kind of challenge, because many possible corpora are entirely unsuited to answering any interesting questions.

Consider again Jockers’s textbook. His first corpus is simply the body chapters of Moby Dick (taken from a Project Gutenberg plain text). Text Analysis with R invites students to study how the words “whale” and “Ahab” occur with varying frequency across the chapters of the novel and to assess the correlation between them; later, students can also study the words that occur in proximity to “dog.” As an introduction to text-processing functionality in R, this works fine. But these exercises lead to nothing resembling a scholarly argument: What does knowing the changing counts of these words allow us to say about the novel?[16] Later, Jockers shows how to use hierarchical clustering on frequent words to discriminate authorship in a small corpus of Irish American novels. This is an impressive technical feat, but authorship attribution is rarely an important question in literary studies, and the field still awaits a demonstration that stylometry speaks to central research questions.

The problem is not really Jockers’s particular choices. Arnold and Tilton carry out a technically impressive demonstration analysis of a collection of Conan Doyle’s Sherlock Holmes stories using natural-language processing. But the most elaborate conclusion they reach is a visualization of where main characters occur within each story. Again, the effort is out of all proportion to the result, which any graduate student in English would immediately recognize as at best a further fact to be explained of at most moderate interest. In fact, most of the interesting questions about plot are not answerable within a single-author corpus. My own variation on the theme was no better: I spent a whole class carrying students through an analysis that showed the differentiation of fiction from poetry in terms of common-word frequencies in the modernist little magazine The Egoist. By crude measures, the diction of poetry differs from that of fiction in this publication. So what?

All of these examples have two things in common. First, they work with convenience corpora: collections chosen for practice, rather than having been designed to answer research questions. Second, though these collections encapsulate a certain amount of textual variation, they lack interesting metadata. But answering research questions about the relations between text and metadata is precisely what students really need to practice. Students must work with bodies of data that hold meaningful patterns, divergences, and historical developments, and they must learn the techniques necessary to bring those patterns compellingly to light.

Thus, the best data for teaching are not to be found by taking all of the bulk releases from big digital archives. Rather, what students need—how unhumanistic this will sound—are data about which at least some answers have already been given, so that instead of being forced to fish for interesting phenomena in an empty ocean, students can follow a trajectory from exploration to valid argument. Such datasets might be designed for teaching by a scholarly collective that sifts freely available materials and chooses samples to represent an interesting range of variation.[17] But the richest source for such data is research itself. Limiting myself to works from the seminar part of my syllabus, examples might include the following: the collection of novel titles (with dates and parts of speech) in Moretti’s “Style, Inc.”; a list of persons mentioned in each of the letters of Thomas Jefferson, as used by Klein to study the collocation network of James Hemings in her “Image of Absence”; the bibliographic metadata on Irish American novels analyzed by Jockers in Macroanalysis; and the catalog of geolocated place names found in nineteenth-century American novels by Wilkens in “The Geographic Imagination.” I am imagining that researchers like these might consider making their datasets, already cleaned and enriched, available for student use, within whatever restrictions of copyright are necessary.[18]

One rationale for my course’s final assignment, I mentioned earlier, was the importance of learning to deal with data in the wild. But I now think this emphasis was misplaced. Though data wrangling is indeed a crucial research skill, it should not come before the question of how to analyze data that are in analyzable form: otherwise, why wrangle? I thought I could combine wrangling with more complex forms of analysis, but it is simply too much to fit into a single course, and treating wrangling as an end in itself means deferring past the end of the semester the kinds of analysis that serve scholarly arguments. Better to start from tamed data—just as is standard in teaching quantitative methods across all the social sciences—in order to focus on methods for answering substantive questions.

Problem 3: The Singular DH Course Is a Bad Fit for the Subject

Data wrangling and programming skill often (and, I am suggesting, necessarily) competed with analytical method for time on the syllabus. Still, a more fruitful relationship sometimes developed between the theoretical readings and the practicalities of analysis. Starting about halfway into the course, it became possible to discuss theory and methodology in other scholars’ work from the standpoint of practitioners. Once I introduced the grammar of graphics via Wickham’s ggplot2 R package, we were able to have a very lively discussion of the construction and implications of specific visualizations in work by Lev Manovich and Lauren Klein.[19] Or again, after a lesson on defining and using functions, we were able to be much more concrete in our debate about Tara McPherson’s use of the concept of “modularity” in her important essay on race in digital humanities. And after tedious weeks of counting single words in texts, the rationale for topic modeling was particularly vivid.

Yet each of these exhilarating moments depended on the previous weeks of practicum preparation, and as soon as each session ended, it was time to turn to a new topic, with its own challenges. And because of my decision to devote so much time to programming, data analysis itself remained mostly obscure. For example, in the course of the semester I presented three ways of exploring variability in data: correlations between word frequencies (from Jockers), latent semantic indexing (LSI), and latent Dirichlet allocation (LDA). I barely explained any of these; the best I could do with LSI and LDA was handwaving and heuristics, and as for correlations, I rushed through them somewhere at midterm. When my students tried looking at word correlations themselves, I realized I had failed to convey how to use them effectively; nor had I been clear about the pitfalls of trying to interpret correlations between frequencies of rare words. My students thus knew much more about the theoretical problems of quantification in general than about the methodological problems of putting numbers (and computers) to work effectively.

In squeezing these quite disparate elements into the syllabus—and consequently scanting each of them for time—I was trying to answer a curricular imperative to provide a useful DH overview and a course satisfying a “theory” distribution requirement. In one form or another, my dilemma ought to be familiar to almost anyone who has tried to teach a DH course: since there is typically just one such course in the curriculum, it is necessary to offer some general DH overview alongside more specific topics. But quantitative method is a bad fit for the “patterned isolation” of the literary curriculum: even more than a literary-historical period, it does not constitute a free-standing topic. And the frequent disjunctions between the theory-seminar half of my course and the practicum parts suggest that the “theory course” is not an ideal home for quantitative method either.

Better models for the pedagogical rationale of quantitative methods are found in the other intensively skills-based courses in graduate education: premodern languages and bibliography. Old English is taught in many graduate departments, and language learning is usually the primary focus of the first-semester course, not debates in Anglo-Saxon studies, which are introduced in a later semester. This pedagogical division does not mean Anglo-Saxonists are tragically untheoretical or have falsely separated the organically unified tasks of studying grammar and interpreting monster slaying.[20] It means that their subject matter is distinctive—and cumulative—in ways that make a difference to the organization of the curriculum. Teachers of quantitative method have the same case to make for the distinctiveness of their subject matter.

Leaving both the breadth of the DH overview and the conceptual debates of theory for other courses, the quantitative analysis course (or, better, sequence) could focus on demonstrating, explaining, and above all practicing the fundamentals of exploratory and inferential data analysis on carefully curated data of known scholarly interest. There is no way to learn the required sort of logical problem solving except by deliberate practice, but solving problems efficiently requires ways of working that students cannot be expected to know in advance. At the start of the term I thought I needed only to give my students reasonable tasks and let them figure them out together. But eventually I learned that the problem sets I had written to take two or three hours were costing my students a dozen hours a week. It seems that, lacking training in solving problems in this domain, my students needed a great deal of time for trial and error.

How to teach better problem-solving technique? At a minimum, students need to see problem solving demonstrated, then to work through a problem together with an instructor, and then to practice it some more on their own. The once-weekly three-hour seminar meeting was not an ideal format for this sequence. In a sense the class was meeting much longer than that, not only in my office hours but also when the students met without me to do the homework. A certain amount of independent collaboration among students is very desirable, but really a TA would have helped a great deal: my students needed help from someone who was skilled enough to spend time doing the homework with them and reinforcing its lessons, and I needed someone to share the job of answering email. Just as learning quantitative methods is more work than fits into a single course, so is teaching them.

Toward Method

I have argued that teaching quantitative methods is hard, but I am not suggesting that it needs to be made easy. On the contrary, the digital humanities should be wary of promises of ease in prepackaged tools, in well-meaning introductory tutorials and workshops that necessarily stop short of what a researcher would need to draw conclusions, and in rationalizations of inconclusive arguments as exploration, play, or productive failure. Having organized workshops, produced tools, and talked elsewhere about data exploration myself, I’d like to think the desire for ease is understandable. These approaches can be useful in making the unfamiliar more accessible. Nonetheless, I learned from my students to be skeptical that the easeful way could naturally lead to the production of new knowledge, because a gentle introduction simply cannot get very far in developing ways of interpreting quantitative results. Good programming skills actually exacerbate the problem when they make highly complex transformations of data—matrices of frequency correlations, probabilistic topic models, and so on—simpler than figuring out whether those transformations are appropriate to the data at hand.

Indeed, the first two problems of quantitative methods pedagogy I have described here become most evident in the context of the third: if introductory programming with arbitrary yet convenient data were all learning methodology required, the subject would indeed fit in a one-semester course.[21] Yet the prospect of humanities departments offering multiple semesters of coursework in quantitative methods seems fanciful even to me: Who would teach such courses, and how many students would really take them? Humanities students who wish a fuller quantitative training are more likely to find it in the methods courses offered by social science or statistics departments, provided humanities departments recognize those courses as a possibility for credit within a degree program. Nonetheless, courses taught by humanities faculty can lay a meaningful foundation in data analysis, as long as they spend enough time introducing the problems of quantitative methodology as problems rather than passing them by or, worse, imagining they have been solved because students are learning to code.

The curricular practicalities can be addressed only once the true challenges of quantitative method are more widely understood. As long as confusion between programming and quantitative data analysis reigns, students and researchers alike may not even recognize when their problems are methodological rather than merely technological, and they will often be forced to stop short of making convincing arguments. Though my students’ and my experience was chastening in some ways, their determined commitment to rigor gives me hope. Despite the total unfamiliarity of much of the course practicum, my students hung together and mastered some complex skills; furthermore, they embraced the possibility of unfamiliar ways of producing knowledge, and, in the final reports, they wrestled ingeniously with the question of how to use numbers as part of their answers to research questions. To go further, what we needed was, above all, more models of how to take those questions on—not just for seminar discussion as exempla of “digital humanities” but also as paths from evidence to analysis we could retrace and then continue ourselves. If research provides more such pathways, including the evidence they start from, the challenges of quantitative method could be more clearly distinguished from those of programming and other dimensions of computer-assisted humanities. Then, I believe, a more focused quantitative literary-studies curriculum would become not easy, but just hard enough.

Notes

Bibliography