Movie Tagger Alpha
Experiments in Movie Timeline Tagging
What interesting issues and questions about new possibilities for film analysis emerge when rich time-based tagging is combined with more traditional film study? How might distinct approaches to film study be combined to multiply the strengths of each? How can interests typically associated with close reading of a film be complemented by tagging throughout a film, or even across multiple films?
we combined expert tags
from the USC Film School
with scene and actor information
from RCDb's Time Data
to produce a variety of visualizations
to foster analysis of films on a macro scale
In early 2010, Zane Vella, RCDb's founder, reached out to USC's School of Cinematic Arts to see what they were doing with movie metadata and to consider the possibility for a joint project. A collaboration formed, the project came to be called Movie Tagger Alpha (MTA), and commenced in the fall of 2010 with the goal of examining how tags on movie timelines (Time Data) can be used in cinema studies to probe the preceding questions.
Michael Naimark's vision for Movie Tagger is "a method and system for parsing and richly tagging every movie ever made". Movie Tagger Alpha was an early experiment exploring the potential of that vision.
Tags on the Timeline
Time Data is RCDb's platform for tagging movies and television shows second-by-second (additional descriptions of Time Data at WatchWith and here). The basic level of Time Data contains general purpose tags that are common to all films, such as scenes, characters/actors, and product information. This can be extended to include location, sound track, and more. The span of each tag is marked with an in-point and an out-point. RCDb coined the term MetaBubbles for the presentation of timed movie metadata overlaid on the content. The following examples show the MetaBubbles view of some of the movies we worked with during Movie Tagger Alpha. (click below to view)
For the Movie Tagger Alpha project, USC students tagged a dozen films with metadata specific to the interests of film scholars at The School of Cinematic Arts. These tags were more specialized than those in Time Data, and they ranged from unstructured free-text to slightly structured name/value pairs. The following tag clouds hint at some of the types of tags the USC students produced, with the size of text in the clouds reflecting how commonly various tags appeared. (click for details)
The MTA Process
For Movie Tagger Alpha, we were interested in a hybrid approach to tagging that combined the general purpose Time Data tags with special purpose SCA tags, and which included both purely objective tags (e.g. when a given actor appears) and somewhat more subjective tags (e.g. intensity of crowd movement, and whether the use of a magic wand constituted an act of touch). We also wanted to apply iterative tagging, where collaborative review of a first pass at tags for a given film could inform our analysis and lead to refinements in following rounds of tagging.
Our methodology involved a few fairly distinct stages. We began by consulting film scholars to consider potential approaches to metadata annotation. Joshua McVeigh-Shulz, the SCA graduate student who spearheaded the MTA project, interviewed about a dozen faculty members and mapped their insights and interests to possible metadata analysis and visualization. Next, three highly astute and dedicated undergraduates painstakingly tagged seven films, beginning with free-text tags.
The following scene stacks depict Time Data for the films that were tagged by SCA students (click to view):
We collectively reviewed the resulting tags and narrowed the set of films to a smaller set that we could effectively analyze in the compact timeframe of the project. We also adopted a tagging ontology (a specification of concepts) for each film in order to focus the tagging and to support visualization of the metadata. We then revised the existing tags to conform to each film's tagging ontology. The final set included four films which pertained to the interests of two SCA faculty members, Steve Anderson and Henry Jenkins.
interests:- Steve Anderson - the depiction of race, gender, crowds, and police violence in Strange Days and West Side Story
- Henry Jenkins - adult-child interactions in The 5,000 Fingers of Dr. T and Harry Potter and the Sorcerer's Stone
The tags for these films were edited and supplemented to accommodate refined interests and corresponding tagging ontologies. Finally, we produced a variety of visualizations and analyzed the results.
Visualizations
1 - Race, Gender, Crowds, and Violence
Steve Anderson's research has investigated the representation of virtual reality in Hollywood films, including the film Strange Days. As we tagged Strange Days, patterns in the depiction of race, gender, crowds, and violence rose to our attention. In response, Jason Lipshin adapted his tagging to annotate bodies in motion, distinguishing between dancing and mobbing types of motion.
We developed a prototype iPad app with a zoomable dial-like interface that presents tags on a circular timeline. Specific types of tags can be highlighted to see their distribution throughout the film. Patterns begin to appear at a glance. For example, depictions of racist violence appear often in the first and last quarters of the film, but almost not at all in the middle half, while scenes of violence towards women were more evenly distributed. The iPad app is synchronized to playback of the movie on a computer. As the movie plays, the dial slowly rotates in tandem, highlighting active tags in the current content. The circular timeline can also be manually scrubbed to quickly jump to scenes containing tags of interest. (click below for screenshots)
Taking what we learned from tagging Strange Days, we continued exploring the themes of race, violence, and crowds in West Side Story. Jason Lipshin expanded his tagging of crowd movement to include level of intensity. For the visualization, we turned to a more conventional linear timeline:
The detail above shows the first 30 minutes of the film. There are two instances of high-intensity motion (purple spikes at the top of the graph), at roughly 12 minutes ❶ and 14 minutes ❷, both involving interracial conflict. When the police appear at 15 minutes ❸ (amber strip in the bottom timeline), their presence mitigates tensions, and movement-intensity drops. The frequently calming effect of the presence of police in West Side Story contrasts with the often provoking presence of police in Strange Days.
Similar to the iPad app used with Strange Days, the West Side Story Movement Type graph can synchronize with the movie, and with its scene stack. This allows one to quickly jump to the content pertaining to points of interest on the graph, affording fluid switching between close reads and broader perspectives of the film.
2 - Adult-Child Interactions
Henry Jenkins is a media scholar who has written about contrasts between permissive child rearing and authoritarian alternatives in The 5,000 Fingers of Dr. T, a 1953 musical fantasy film written by Dr. Seuss. Jenkins identified how the film presents a contrast in parenting philosophies. This contrast is manifested in Dr. Terwiliker and Mr. Zabladowski, though neither is a parent of Bart, the child character. Dr. T is a domineering music instructor, and Mr. Z is a gentle, understanding father-figure. Drawing on Jenkins's research, we tagged instances of authoritative and permissive language towards children in the film, including which characters are involved in each action. We visualized this with a circular network diagram that shows aggregations of interactions between characters. Red connections indicate authoritative language and green signifies permissive. The opacity of each connection ramps such that the link is more opaque near the recipient, namely Bart for all adult-child language interactions we tagged in The 5,000 Finger of Dr. T. The connection for each individually tagged instance is also partially transparent so that the sum of all connections between two characters reflects the volume of interactions between the two.
In the following diagram, we see that language interactions were tagged between three adults and one child. It is clear that Mr. Zabladowski always speaks permissively to Bart, while his mother speaks both authoritatively and permissively, and Dr. Terwilliker speaks mostly authoritatively to Bart. It also appears that Bart's mother speaks infrequently to him, compared with Mr. Z and Mr. T.
Next, we looked at instances of touch in The 5,000 Fingers of Dr. T.
We were mostly interested in characteristics of adult-child interactions evidenced by touch, but we also tagged the touching of objects, which take most of the space in this graph. The most common touching events were tagged as affectionate, or aggressive,magic, in roughly equal frequency. An example of aggressive touch would be an adult firmly holding a child's wrist while scolding him, while patting a child on the shoulder for encouragement would be affectionate touch. This graph shows that both Mr. Zabladowski and Bart's mother touched Bart in a combination of affectionate and aggressive manners, while Dr. T exhibited only aggressive touch towards Bart.
As with our tagging of Strange Days and West Side Story, we were interested in applying the tagging scheme we used for The 5,000 Fingers of Dr. T to a second film. We chose Harry Potter and the Sorcerer's Stone, another film with ample interactions between adults and children.
The language and touching graphs for Harry Potter and the Sorcerer's Stone are infused with more subtlety than the respective graphs for The 5,000 Fingers of Dr. T. First, there are many more adult and child characters in Harry Potter, which yields a more complex graph. More interesting, there are nuanced interactions that challenged the parameters of our tagging scheme. In the language graph below, it is clear that Hagrid stands out as an adult who frequently speaks permissively to children. In contrast, although Professor Dumbledore and Professor McGonigal are friends and supporters of the children, they are stern with the children at times, exhibiting a mix of both authoritative and permissive language.
At first glance, it is surprising to see that Voldemort, the archenemy of Harry Potter, was tagged as speaking permissively to Harry. It turns out this was a case of Voldemort bargaining with Harry. Similarly, one wouldn't have expected Snape to have been tagged as speaking permissively to Harry. It turned out to be an instance of sarcasm (which couldn't be neatly labeled as authoritative or permissive in our scheme). Cases like these highlight the need for iterative tagging, where subtleties arise that may call for revisions to the tagging scheme. They are also occasions that beg for a combination of close and broad analysis.
The graph for touching in Harry Potter and the Sorcerer's Stone also contains instances where our metadata ontology did not adequately support the nuances of the actions. For example, when Harry encounters Voldemort, it is Voldemort's face on the back of Quirrell's head. As Harry tries to remove Voldemort's/Quirrell's grip, he does not realize that his power over Vodemort will burn Quirrell's hand. This was perhaps the most relevant instance of touch in the film, but it went untagged because it didn't map well to our categories of affectionate, neutral, and aggressive. We also ran into issues of whether remotely "touching" another character by waving a wand constituted an instance of touch. This illustrates again how we need to alternate between close and distant views while tagging, and how iterative tagging can allow ontologies and tagging schemes to evolve.
Comparing the graphs for The 5,000 Fingers of Dr. T with those for Harry Potter and the Sorcerer's Stone, we see distinct differences. The character count in Harry Potter is much higher than in 5,000 Fingers. And several characters in Harry Potter exhibit nuanced language (e.g. Snape's sarcasm, Voldemort's bargaining). These differences in complexity may result from several potential factors, including: the fact that the films were produced in different eras, that they may have aimed for different viewing age groups, and differences in Dr. Seuss's and J. K. Rawlings's writing styles. Taking a distant read of each film in the form of these graphs, then stepping back further still to compare graphs across films, supported by close reads of special instances, we arrive at cohesive perspectives that span micro and macro views.
In Sum, Alpha = Beginning Steps
The Movie Tagger Alpha experiment was fruitful in raising several notable issues worth considering in movie/TV/video tagging efforts. Among them:
- balancing curated tagging and unmediated tagging
- visualizations and user interfaces to support creation, review, and presentation of tags
- how to allow and embrace ambiguity
- options for applying natural language processing to unstructured free-text tags
- feasibility of employing semantic web methods and technologies to manage temporal metadata
- weighing folksonomy-oriented and ontology-based approaches to organizing tags
- how to manage data sourcing to support expected uses, but be adaptable to unanticipated use
One example: the anomalies cited above regarding nuances in tagging language-interactions in Harry Potter and the Sorcerer's Stone (e.g. Voldemort's bargaining and Snape's sarcasm both getting tagged as permissive language), emphasize the need for an adaptive tagging structure, to possibly lean towards structure that develops with folksonomy-style tagging (think del.icio.us and Flickr) more than towards strongly curated tagging structure.
More interesting, Movie Tagger Alpha began to hint at compelling prospects for bridging traditional close reading of films with emerging ways of distant reading over a whole film, or even across a collection of films. When tagging and review tools fluidly traverse between concrete close reading of material and more abstract distant reading, we can find broader perspectives of material.
It would be fascinating to have a forever evolving form of Movie Tagger that continually grows to encompass richer, deeper, and broader forms of tagging, perhaps a "Movie Tagger - perpetual beta".
Context & Tangents
The following assortment of notes and links aims to place Movie Tagger Alpha within the context of related work, and suggest ideas for further tagging efforts.
Joshua McVeigh-Shulz has a more insightful description of the Movie Tagger Alpha project:
Movie Tagger Alpha: Critical Tagging in Emerging Methods of Media Scholarship
For another glimpse at the ambition of Movie Tagger, see this introduction by Michael Naimark, Steve Anderson, and others:
Movie Tagger - a method and system for parsing and richly tagging every movie ever made.
Eddie Elliott's cine.viz page offers a catalog of data visualization work revolving around cinema.
Movie Tagger Alpha was an offshoot of RCDb's Time Data endeavor.
Steve Anderson runs Critical Commons, "an online tool for viewing, tagging, sharing, annotating and curating media".
Critical Commons uses tags/keywords for each whole clip to aid searching.
Henry Jenkins writes about The 5,000 Fingers of Dr. T:
No Matter How Small: Revisiting Seuss's 5000 Fingers of Dr. T
Hop on Pop: The Politics and Pleasures of Popular Culture
Cultural Analytics, coined by Lev Manovich poses the question: How and why study big cultural data?.
Martin Szomszor and others investigated using a movies folksonomy in a movie recommendation system:
Folksonomies, the Semantic Web, and Movie Recommendation
Clay Shirky debates categorization, folksonomy, and ontology: Ontology is Overrated: Categories, Links, and Tags
The New York Times ran an article about Franco Moretti's thoughts on distant reading, entitled What is distant reading?.
Jerremy Buttler's Shot Logger contains a database of shot lengths for over 600 films and TV episodes.
Yuri Tsivian also looks at shot lengths and offers a database of shot timing in his Cinemetrics project.