PART IV ][ Chapter 28

Volumetric Cinema

Kevin L. Ferguson

When A Square, the didactic narrator of Edwin Abbott’s Flatland (1884), imagines in his mind’s eye the haunting vision of a Cube transformed into a hypercube, the description of his ravished bewonderment anticipates the real-world response to cinema, which would be invented within the next two decades. With the techno-magical apparatus of motion pictures, we too can come into contact with a “more perfect perfection” of the real world by allowing our ravished eyes to pass through “a new kind of Space, with a wake of its own” (89). Of course, the point of Flatland is that our two-dimensional narrator comes from a lesser world, and after seeing him fumble through our familiar three-dimensional reality, we are meant to view his thought experiment in the fourth dimension as a strange new experience that can also transform our own understanding.

I begin with this moment of revelation not only because Abbott anticipates the looming invention of motion pictures but also because he affords us an opportunity to reimagine our own visual habits of mind, which we have developed to make sense of an increasingly media-saturated culture. Indeed, it is on this last point that A Square’s account is most relevant to my project; his description of a tesseract redefines four-dimensional volume not as a series of objective qualities or mathematical measurements, but as a spectacular visual experience that leaves him ravished. This sensory and affective response foreshadows precisely the ways in which film audiences have experienced motion pictures, which can similarly be understood as a cube passing through the “new kind of Space” of the cinema screen, leaving “a wake of its own” on viewers’ eyes that creates a “more perfect perfection” than still images alone. Researchers since the early twentieth century have explained the numerous optical illusions that account for the psychological persistence of vision that makes motion pictures possible, but what if there were a contemporary way to further extend the “wake” that A Square imagined, rendering it concrete and observable? I argue that the temporal and aesthetic trail Abbott imagined can help locate a new DH technique, one that encourages a skeptical stance toward the conventional faith in transparent, unmediated vision.

Unfortunately, the majority of DH approaches to cinema and media studies remain focused on the more easily quantifiable aspects of moving image texts, such as counting shot lengths, timing editing patterns, or comparing production budgets. For this reason, contemporary work at the intersection of digital humanities and media studies can be relatively unsophisticated when it also fails to critique spectators’ visual habits because it retains a faith in normative viewing practices. While recognizing a value in conservative statistical approaches, I argue that digital technologies can be used not only to produce new knowledge about film texts but also to fundamentally “transform cultural assumptions about the spatial properties of moving images” (Anderson).[1] Rethinking our viewing practices in a digital age, however, requires an investment in experimental, theoretical methods that run counter to the rationalist uses of quantitative data often employed in DH work. Is there not a way to rediscover a new kind of space in cinema, a digital humanities that can look past the flat screen and explore moving image texts as dimensional objects? In doing so, can we outline an experimental stance more broadly for the digital humanities, which would both position cinema and media studies more firmly within the field and generate new strategies for other DH practices to visualize collections of images as three-dimensional objects?

Cinema’s Obsession with Volume

As a function of its flat, two-dimensional technology, cinema is and always has been obsessed with volume. This is readily apparent across the history of film, beginning with the myths of audiences fleeing at the sight of an approaching train in one of the first motion pictures, L’arrivée d’un train en gare de la Ciotat (dirs. Auguste Lumière and Louis Lumière, 1896). Narrative accounts had audiences confusing the flat projection screen with an actual window and running for fear of their lives. “Today, we cannot comprehend the terror that gripped the 1895 audience facing the Lumière brothers’ arriving train,” claims one source (qtd. Loiperdinger, 90). While this foundational myth of naive spectators has been thoroughly debunked by Martin Loiperdinger, the fact that it persisted for more than one hundred years demonstrates a complicit desire on the part of filmmakers and audiences alike to insist that film is a magical art, able to render spectators helplessly spellbound in front of the screen. That is, even when promoting technological advancements such as 3D or virtual reality, film and media production strategies emphasize the wonder and captivity of spectators in front of moving images.

While this argument about the nature of cinema spectatorship has resurfaced with the introduction of each new technology such as synchronized sound, color film stock, or Smell-O-Vision, its main thrust is particularly coded in terms of a spatial spectacle: cinema as an art is powerful enough to transcend the flat screen and make spectators believe that moving images are three-dimensional reality. This effect is touted in contemporary immersive media techniques, such as IMAX and 3D filmmaking for theatrical blockbusters (and the future promise of holographic cinema); virtual reality, augmented reality, and motion-sensing video game technology (e.g., Google Cardboard, Google Glass, Project Tango, Microsoft HoloLens, Xbox Kinect, Oculus Rift); and echo-mapping laser cameras, trillion frame-per-second imaging, and other computational photography methods. Indeed, contemporary immersive media techniques move beyond the promise of convincing audiences that they are watching a real three-dimensional spectacle to the promise of actually interacting with a real three-dimensional world, manipulating virtual objects or overlaying virtual space onto reality.

Popular film narratives further encode this fascination with multidimensional space, as evident in prominent scenes of fantastic, interactive screen use in science fiction films like Fahrenheit 451 (dir. François Truffaut, 1966), Blade Runner (dir. Ridley Scott, 1982), Minority Report (dir. Steven Spielberg, 2002), and The Avengers (dir. Joss Whedon, 2012). Films like these promise a future of hybrid human–machine interactivity that further closes the gap between a remote experience of media and the lived embodiment of it. Among these examples, the single most emblematic scene of futuristic vision is detective Rick Deckard’s use of the fictional Esper device in Blade Runner. Using a complex but lo-fi home televisual device, Deckard inserts a flat, two-dimensional photograph and is able to navigate the space it represents in order to gain a needed clue: zooming in, enhancing details, and even introducing parallax to navigate around obscured corners. Deckard’s voice-activated machine manipulation of the photograph imagines an omniscient postcinematic vision, a fantasy of simple, consumer-grade, in-home technology that can reconstitute three-dimensional volumes in space from a flat two-dimensional image.

We see a fascination with transforming small screens into large volumes not only in the narratives of contemporary cinema but also in cinema technology itself, which is increasingly promoted as something hefty and large even as screens become smaller and more ubiquitous. This has been true from the beginning of cinema, as when Kristin Thompson describes how early copyright practice resulted in the curious medium of the “paper print,” “the main form of copyright deposit material” from 1895 to 1912, where “every frame of the original film was printed as a photograph on a long roll of paper. This practice arose because films could not be copyrighted as such and had to be copyrighted as a series of photographs” (7–8). The surviving paper prints from the early history of cinema testify to the legal and philosophical sense that movies were something “bulky” to be preserved, echoing Tom Gunning’s argument that early cinema was itself an “attraction”: the experience was much more about a display of technology than about watching a film narrative. Fast-forward one hundred years and we have no less an interest in displaying cinema technology as something weighty and large, evident in the fetishized display of massive projectors in the lobby of IMAX movie theaters or David Bordwell’s reporting about the “platter farms” needed to create 250-pound 70mm copies of Quentin Tarantino’s The Hateful Eight (2015) for theatrical distribution. As a response to television’s midcentury threat to cinema audiences, film production companies experimented with wider aspect ratios, new color techniques, 3D technology, and other gags and gimmicks, creating cinema as a huge experience impossible to contain in the home. In the same way, theatrical response to the contemporary miniaturization and proliferation of media screens (tablets, cell phones, watches) also leads to an expanded interest not only in the three-dimensionality of movies but also their physical size. IMAX and 3D releases of blockbuster films, innovative encouragement of second-screen use, or “luxury” viewing experiences with reclining chairs and reserved seating all work against the sense of movies as small, flat, transportable experiences.

The science fictional examples of interactive screen use are fantastic because they can impossibly draw out three-dimensional meaning from the limited two-dimensional photograph. But as futuristic as the Esper device scene from Blade Runner would have seemed in 1982, it has become partial reality well before 2019, the year in which Blade Runner is set. In a 2005 experiment, researchers at Stanford University developed a photographic technique they called “dual photography,” which uses the principle of Helmholtz reciprocity to switch the light source and camera in a scene, allowing the user to “take” a photograph from the perspective of the light source (Sen et al.). By measuring how light was reflected off the objects in the scene, the researchers were able to create remarkable images, such as one in which the markings on a playing card that was turned away from the camera but toward the light source are clearly visible. Another group of scientists from MIT invented an extremely high-speed photographic technique based on a process they call “femto-photography,” which allows one to take photographs around corners. In this method, they explain, “the camera uses light that travels from the object to the camera indirectly, by reflecting off walls or other obstacles, to reconstruct a 3D shape” (Velten et al., 2). By bouncing a laser off of a wall at a hidden mannequin, researchers could record the rebounding light using a streak camera and then reconstitute the hidden image, even rendering it from different views in three-dimensional space. Finally, recent applications in quantum photography have entangled photons of light, with the result that “ghost images” of objects can be made from light that was never itself reflected off of an object, but that nonetheless capture that real object precisely (Vergano). A still more perfect perfection of photographic representation, ravishing indeed.

Similarly, industrial and technological practices addressing the nature of cinematic volume were also theorized in early film history. Responding to criticisms that portrayed cinema as but the “feeble mechanical reproduction of real life” (a viewpoint actively encouraged by early distribution practices), art and film theorist Rudolf Arnheim argued in 1933 that “film pictures are at once plane and solid” (12). Arnheim drew evidence for the contrary argument that cinema was an art form by looking at the technical problem of how filmmakers translated three-dimensional reality into a two-dimensional experience. So, for example, he demonstrates the many nuances of the “projection of solids upon a plane surface,” describing how early film technicians thoughtlessly placed a camera in front of an object, whereas thoughtful film artists considered how camera placement could artistically manipulate perspective and result in aesthetic effects. As a result, “the effect of film is neither absolutely two-dimensional nor absolutely three-dimensional, but something between” (12). Hugo Münsterberg made a similar point at the time, noting how this inter-dimensionality “brings our mind into a peculiar complex state” where “we have reality with all its true dimensions; and yet it keeps the fleeting, passing surface suggestion without true depth and fullness” (emphasis in the original, 71).

As rudimentary as his argument may seem, Arnheim’s metaphysical conception of cinema has been almost entirely ignored in later empiricist digital approaches to media studies that prioritize the flat plane as the object of study. By refusing to accept the binarism of two- versus three-dimensionality, Arnheim instead advocates for a more complex thinking through of the moving image. While this trajectory might seem outdated in light of the technical evolution of cinema, we would be mistaken to think that spectators today have moved beyond the basic visual habits Arnheim outlined nearly a century ago. While our cinema technology has gotten more sophisticated, our viewing habits have only very slowly adapted to new contexts, and we continue to treat the moving image as simply a self-evident flat surface to be read in only two dimensions. The need for a more robust conception of moving images is evident not only in contemporary advertisements that sell cinema and video games as new media experiences that can transform the humdrum existence of spectators but also in immersive fads like stereoscopy or widescreen exhibition formats that ironically prevent audiences from considering the spatial aspects of moving image texts by creating a situation of spectatorial rapture, offering to viewers the promise that they will forget that they are even watching a film. The desire to create a media experience coterminous with a spectator’s own lived experience, aiming for the “mechanical reproduction of real life,” is built on flatness.

Similarly, many DH approaches to media studies also promise a spectacular refashioning of traditional humanities work, but instead obscure the deeper consideration of the intra-dimensionality of moving image texts by focusing on the statistical measurement of flat film details. For example, film historian Barry Salt’s monumental 1983 work, Film Style and Technology, takes a rigorously comprehensive approach to the quantification of film style, but it remains a decidedly precomputational project even as it finds second life in the Cinemetrics project.[2] Likewise, Michael Kipp’s ANVIL video annotation tool and the Centre Pompidou’s Lignes de temps each offer a workspace system for total film annotation, including timelines, subtitles, and shot notes. As promising as these approaches may be for one mode of film studies, they remain tied to traditional methods of evidentiary and stylometric investigation, which preserves conventional viewing habits and a theory of cinema spectatorship as a narrative-driven explication of meaning.

We need something stranger, something more ravishing. Arnheim concludes his defense of cinema as art: “Even in the simplest photographic reproduction of a perfectly simple object, a feeling for its nature is required which is quite beyond any mechanical operation” (11). Since cinema transforms the material reality it captures, there is an inherently aesthetic dimension, a “feeling” that resists a purely descriptive account of nature. In Chapter 20 in this volume, Jason Mittell describes how videographic criticism similarly enables conceptual transformations of film and media texts that, in the most successful cases, straddle both explanatory and poetic registers to create transformative insight. My project also examines the aesthetic dimensions of mechanical operations. By using scientific image analysis software designed specifically for anti-aesthetic, technical, and medical purposes, I encourage a DH media studies whose critical focus is more experimental than explanatory. This is not data-bending video files, mash-ups of video essays, or even a kind of cut-up I Ching, but rather a “turning” of a media object on its side, presenting a new view not otherwise possible to attain.

A Surrealist Tradition

In advocating for an experimental form of cinematic criticism, I explicitly reject the “Scientific Realism” that Salt espouses in his project to turn cinema studies into a “real science” by the objective measurement of stylistic features such as shot length or scale (155). Instead, inspired by the avant-garde film criticism practiced by scholars such as Robert B. Ray and Tom Conley, I embrace the “subjective, semi-arbitrary, irrational, [or] associational” procedures that Salt mocks (1). For me, these are powerfully productive modes for engaging moving image texts beyond the narrow scope of a limited definition of science and toward an engagement with the broader “feeling” of the humanities. In fact, it is only through methods stemming from the avant-garde Impressionist-surrealist tradition that we can create new, otherwise impossible-to-see knowledge about cinema, rather than just explicating the already visible.

The surrealists were invested in challenging the intentionality of art, pursuing playful methods that used art in unexpected ways and directions. For examples of unintended reuses of art, we could point to the game of “irrational enlargement” that poses and answers unusual or trivial questions of films, Stephen Ramsay’s “screwmeneutical” wandering through the archive, the DVD supplementary features that reframe classic films, or the exemplary videographic criticism discussed by Mittell that reorients our engagement with a text. Indeed, new media technologies offer the promise to reorient viewer engagement to cinema in ways that would have been unanticipated but appreciated by the surrealists. While the increased paratextual framing of film releases today (DVD commentary, fan edits of trailers, podcast reviews) suggests a possibility of such engagement, more advanced digital video analysis tools such as the ones I describe later will allow us to embrace surrealism as a strategy even more fully, transforming film texts in radically new, but still internally consistent, ways. While film distribution strategies often recognize the promotional value of fan manipulation of media properties and thus encourage fan practices that would have fit in surrealist circles, these fan practices are extremely limited compared to the use of noncommercial digital tools to study moving image texts, which do not restrict manipulation to only a limited set of preestablished options determined by corporate producers of distributed media. Instead, the contemporary digital humanist can adopt an open-source, software-driven, method-based experimentation to research every parameter of the media text, similar to the surrealist fascination with automatic processes that involved manipulating the mechanical and technical methods of creating art.

Despite the potentially liberating surrealist uses for new media and the increased availability of digital analysis software, the rationalist, positivist, semiotic bent of film theory threatens to become even more exaggerated when approached with digital tools. I have argued that media studies should embrace surrealism as a strategy, but we also must extend that argument to digital humanities broadly. To date, the most familiar touchstone in this field is Lisa Samuels and Jerome McGann’s 1999 essay, “Deformance and Interpretation,” which argues that “‘meaning’ in imaginative work is a secondary phenomenon, a kind of meta-data [that] is important not as explanation but as residue” (48). While arguing for antithetical interpretive practices that initially set aside the illusory search for a concept-based meaning, Samuels and McGann stress that “deformance does not banish interpretation” (40). That is, instead of thinking that a work of art’s “intelligibility” stems from a critical interpretation, they argue that “all interpretation is a function of the poem’s systemic intelligibility”: our critical practice should set aside the notion of applying an interpretation to a text and instead work to discover a text’s system by performing actions on it that render it strange (emphasis in original, 40). Likewise, Johanna Drucker, among others, has criticized digital tools used for information visualization, data mining, and geospatial representation as reflective of “positivistic, strictly quantitative, mechanistic, reductive and literal” values that “preclude humanistic methods from their operations because of the very assumptions on which they are designed: that objects of knowledge can be understood as self-identical, self-evident, ahistorical, and autonomous” (86). But reinvigorating a digital surrealist tradition allows us to reconceive of digital tools not solely as quantitative and interpretative devices but also as processes and methods that can be bent to unintended purposes to create unexpected knowledge, expanding our object of study rather than simply adding to it.[3]

Volumetric Cinema

What does it mean to “extract a volume” from a film? On a technical level, this entails transforming one 3D representation into another, switching the temporal dimension of moving images with a third spatial one. On a conceptual level, this turns a film on its side, allowing for a spatial visualization of the time of a film. This reverses, in a sense, the process of traditional filmmaking, which transforms 4D reality into a discrete 3D time-space. By transforming the passing film frames back into discrete physical moments, we can concretize the spectator’s persistence of vision and conjure up Flatland’s “new kind of Space, with a wake of its own.” As with the surrealist interest in irrational knowledge, which breaks perceptual habits in search of a different kind of authenticity, this “volumetric” method allows us to approach moving image texts outside of their temporal tyranny. Volumizing cinema deforms our experience of the structure of moving images: we can manipulate a film we may have already seen numerous times, but from an otherwise impossible vantage point. By creating a 3D volumetric model of a film scene and then passing a slice through that volume in time, we can examine film texts from higher-dimensional perspectives (see Figure 28.1).

In contrast to the analog videotape distribution of motion pictures, today’s digital distribution technologies allow us to more easily return to film’s celluloid origins to do this volumizing. For example, every filmstrip has a discrete number of countable frames. While a videotape and a digital media file are both time based, the latter can be easily converted into a discrete series of frames in a matter of moments. To do so, I employ the scientific image analysis software ImageJ, in a process I liken to cutting the individual frames of a filmstrip and stacking them up one behind the other. However, working with virtual stacks of film frames in ImageJ allows for an additional dimension of analysis: users load a series of images into a “stack,” after which each pixel in each image becomes a “voxel,” a volumized pixel that exists in relation not only to other pixels on its own plane but also to all the pixels on that coordinate that came “before” and that will come “after.” By modeling a whole scene’s worth of voxels simultaneously, rather than examining pixels of a frame individually, we can gain a radically new perspective on cinema: we can draw a frame through any subset that we desire of the 3D voxel-space of the film. That is, rather than be limited to a “slice” of the film that corresponds to a single frame, we can create whatever 3D slice we want from the entire volume of the film. Working from a film’s two spatial dimensions and the dimension of time, we can recover the otherwise impossible-to-see volume of the film by modeling a series of frames in a single moment, rather than as something occurring over time.

Figures 28.1–28.3 demonstrate some of these impossible views. In these examples, we are looking closely at an individual scene rather than an entire film. In Figure 28.2, by angling the movie volume down, we can see two of its otherwise hidden sides and get a quick sense of some basic information about editing patterns; for instance, this scene is bookended by two longer shots. Whereas Figure 28.2 shows the outer frame edge of this scene, Figure 28.3 shows three orthogonal slices cut from the middle of the film volume, allowing us to better see patterns of camera or character motion or to get a sense of color or shape in the interiority of a scene. Last, with Figure 28.1 we can draw our own slice, bisecting the volume on a particular axis of interest. While the orthogonal slices reveal something akin to finish-line slit-scan photography—where an image is composed of the same thin strip taken at different times—by making a slightly oblique cut we can see both slightly ahead of and slightly behind a shot in time. That is, in this example, the Evil Queen’s back is slightly temporally “behind” her future hand movement shown on the left side of the screen. By moving the oblique cut through the volume in this way, we see a ripple of motion as the character’s body is stretched over time, visualizing her movement in both space and time.

We can also further manipulate the source frames, adjusting transparency or color to more clearly see a particular object or detail. One simple manipulation to help clarify the film volume in this vein is to use an edge-detection algorithm, such as the Sobel filter in ImageJ that works by finding areas of contrast in an image, identifying which pixels from each slice are the “edgiest” and then recoloring the image so that the identified edge regions are shown in starker contrast. The result is a mostly monochromatic outline of shapes and figures; stacking a series of these edge-detected frames together emphasizes a kind of wire-frame motion. In Figure 28.4, we see the beginning of the Evil Queen’s transformation as a staggered series of edges as she raises her glass and tilts her head back to drink. Rotating around the actual 3D space further reveals surprising details, such as the smoothness of a subtle camera movement versus the jerkiness of the Evil Queen’s limbs, or the elegant symmetry of the cup’s rise and fall. In this way, selectively manipulating the edge-detected volume allows for new motion studies, which enable the visualizing of a series of movements through time.

These transformations point to a mostly ignored trajectory of viewing a film not as a succession of images, but as a simultaneity of images: a view across the landscape of the film that allows modes of reception different from the strict time-bound mode of projection. For example, Edison Kinetoscopic Record of a Sneeze, known as Fred Ott’s Sneeze (dir. William K. Dickson, 1894), the oldest surviving copyrighted film, is familiar to film scholars as a grid of frames (see Figure 28.5). Convention tells us how to reconstitute the “order” of these frames, beginning in the upper left and working across and down to the lower right. In the same way, a folder of .jpg frames can be scanned as a film sequence. But this ignores the more valuable possibilities suggested: that “volumizing” the cinema is a strategy that can help break our visual habits, particularly the one that starts us in the upper left corner and asks us to “figure out what comes next.” Instead, we can use a different approach: to project the film differently than was intended, moving through the voxel space in otherwise impossible ways. Such an approach to the visual image offers new venues for critical and aesthetic meaning-making, and while the results can appear surreal, they were nonetheless always already contained in the image.

When Arnheim described in 1933 the cinematic man holding a newspaper in front of his face, which “seems almost to have been cut out of his face, so sharp are the edges,” his account would have been particularly resonant with audiences recently exposed in the previous decades to Cubism, among other visual styles that drew renewed attention to the picture plane, compressing apparent screen depth by flattening layers together (12–13). In contrast, the contemporary project I describe resonates with the “softwarization” of media, defined by Lev Manovich as “the systematic translation of numerous techniques for media creation and editing from physical, mechanical, and electronic technologies into software tools” (164). Softwarization means that methodologies for creating or manipulating media texts are not bound to a particular platform or technology, and a world of increased softwarization means an increasing familiarity with (and display of) new modes of media manipulation. So, the fact that contemporary films can show space in new ways is an effect of a filmmaker’s ability to rely on audiences’ familiarity, and perhaps even experience, with digital manipulation of moving image texts. That is, the effectiveness of even the most advanced computer-generated special effect is limited by a spectator’s understanding of the aesthetic context of moving images. So, while Blade Runner imagined a novel use of photographic space that only later became real, softwarization makes contemporary cinema’s strange visual strategies more legible to more spectators. Recent films about quantum space like Interstellar (dir. Christopher Nolan, 2014) and Ant-Man (dir. Peyton Reed, 2015) are filled with visual spectacles identical to the conceptual volumizing I describe, extending a figure in space and then rotating that extended figure around new axes. More than just fascination with representing the unrepresentable, these sequences show an evolution of narrative cinema’s interest in volumizing space, using special effects and narrative devices to train spectators in new modes of vision. Today, an aesthetic of softwarization accelerates spectators’ familiarity with nontraditional uses of the flat plane surface, seen in the resurgence of 3D filmmaking represented by Avatar (dir. James Cameron, 2009) and the intensive development of virtual and augmented reality by Google. From this perspective, a film like Avatar not only updates the techno-fantasy of Blade Runner for twenty-first-century audiences but also advances an argument that technologies of filmmaking are consonant with techniques of spectatorship.

What does it mean to insist on a volumetric cinema? Work at the intersection between digital humanities and media studies must willingly explore more experimental methods, arresting our temporal experience of films as narrative things to be consumed. “At once plane and solid,” we can align the trajectory of film theory with contemporary computational techniques for recovering the volume of cinema, using open-source software and edge-detection algorithms to create virtual models of the time-space of film scenes in ways that resonate with Arnheim and an otherwise abandoned critical surrealist avant-garde project, achieving a “more perfect perfection.” These experiments can change what we know about a film, not only by requiring practitioners to engage with the epistemic effects of volumization and the manipulation of medical imaging software but also by creating a “knowledge effect” around, particularly, the extension of physical volume through time. Camera movements, a character’s walk across a set, or a choreographed fight or dance can all be made visible as flowing physical shapes. With a little refinement, these physical shapes can also be 3D printed, making a real, tangible representation of cinematic time and movement. Digital humanities and media studies would do well to resurrect a fascination with volumes and the play of transformation between reality and the two-dimensional screen.

Notes

Bibliography