The Star Wars social networks – who is the central character?

How I did the analysis

As this is part of the F# Advent calendar, I used F# for most of the analysis. I combined it together with D3.js for the social network visualizations, and with R for the network centrality analysis. You can find all the source code on my GitHub. Because the whole code turned out to be relatively long, here I look only at some of the more interesting parts.

Parsing the screenplays

I started by downloading all the scripts for the 6 Star Wars movies. They are freely available from The Internet Movie Script Database (IMSDb), for example here’s the script for Episode IV: The New Hope. The screenplays are only in the form of drafts that sometimes differ from the actual films – the differences are however not very big.

The first step was to parse the screenplays. To complicate things, each of them used a slightly different format. The screenplays themselves were in HTML, either within the <td class="srctext"></td> tags, or within the <pre></pre> tags. To extract the contents of each script, I used the Html Parser from F# Data library which allows accessing individual tags in a HTML document using statements like

open FSharp.Data
let url = "http://www.imsdb.com/scripts/Star-Wars-A-New-Hope.html"
HtmlDocument.Load(url).Descendants("pre")

The full code for this part is available in the parseScripts.fs file.

The next step was to extract relevant information from the scripts. In general, a script looks like this:

INT. GANTRY - OUTSIDE CONTROL ROOM - REACTOR SHAFT

Luke moves along the railing and up to the control room.

[...]
                LUKE
        He told me enough!  It was you 
        who killed him.

                VADER
        No.  I am your father.

    Shocked, Luke looks at Vader in utter disbelief.

                LUKE
        No.  No.  That's not true!  
        That's impossible!

Each scene starts with its setting: INT. (interior) or EXT. (exterior) and the location of the scene. Then there can be some free text describing what is happening and how does the scene look. In the dialogues, the names of characters are written in capital letters (sometimes also in bold), followed by what they are saying.

The main signposts in the screenplay are the INT. and EXT. statements which serve as scene separators. These were written consistently in bold in all the 6 scripts and I used them to split them into individual scenes:

// split the script by scene
// each scene starts with either INT. or EXT. 
let recsplitByScene (script : string[]) scenes =
    let scenePattern = "<b>[0-9]*(INT.|EXT.)"
    let idx = 
        onmouseover="showTip(event, 'fs4', 10)" onmouseout="hideTip(event, 'fs4', 10)">script 
        |> Seq.tryFindIndex (fun line -> Regex.Match(line, scenePattern).Success)
    match idx with
    | Some i ->
        let remainingScenes = script.[i+1 ..]
        let currentScene = script.[0..i-1]
        splitByScene remainingScenes (currentScene :: scenes)
    | None -> script :: scenes

This is a recursive function that takes the full screenplay and looks for the specific pattern, which is EXT. or INT. in bold, optionally preceded by a scene number. The function goes through the string and when it encounters the scene break, it splits the string into the current scene and the remaining text. Then it runs recursively until all the scenes are extracted, using the scenes variable as an accumulator.

Getting list of characters

The previous function gave me a list of scenes for each of the movies. Extracting the characters out of them turned out to be more difficult. Some of the scenes followed the format that I showed in the example above, some of the scenes only used character names followed by a colon and their dialogue, all within the same line in the text. The only common property between the different formats was that the names were always written in capital letters.

I resorted to regular expressions, one for each screenplay format:

// Extract names of characters that speak in scenes. 
// A) Extract names of characters in the format "[name]:"
let getFormat1Names text =
    let matches = Regex.Matches(text, "[/A-Z0-9-]+*:")
    let names = 
        seq { for m in matches -> m.Value }
        |> Seq.map (fun name -> name.Trim([|' '; ':'|]))
        |> Array.ofSeq
    names

// B) Extract names of characters in the format "<b> [name] </b>"
let getFormat2Names text =
    let m = Regex.Match(text, "<b>[]*[/A-Z0-9-]+[]*</b>")
    if m.Success then
        let name = m.Value.Replace("<b>","").Replace("</b>","").Trim()
        [| name |]
    else [||]

Each of the regular expressions matches not only capital letters, but also numbers, hypens, spaces and slashes. All because the characters in Star Wars use names like “R2-D2” or even “FODE/BEED”.

To actually extract the list of characters that appear across the films, I also had to take into account the fact that many characters use multiple names. Senator Palpatine turns into Darth Sidious and then into The Emperor, Amidala disguises herself as Padme (or the other way around?).

Because of this I manually put together a simple file aliases.csv where I specified which names I consider to be the same. I used these as a mapping onto a unique name for each character (except for Anakin and Darth Vader).

let aliasFile = __SOURCE_DIRECTORY__ + "/data/aliases.csv"
// Use csv type provider to access the csv file with aliases
type Aliases = CsvProvider<aliasFile>

/// Dictionary for translating character names between aliases
let aliasDict = 
    Aliases.Load(aliasFile).Rows 
    |> Seq.map (fun row -> row.Alias, row.Name)
    |> dict

/// Map character names onto unique set of names
let mapName name = if aliasDict.ContainsKey(name) then aliasDict.[name] else name

/// Extract character names from the given scene
let getCharacterNames (scene: string []) =
    let names1 = scene |> Seq.collect getFormat1Names 
    let names2 = scene |> Seq.collect getFormat2Names 
    Seq.append names1 names2
    |> Seq.map mapName
    |> Seq.distinct

Now I could finally extract names of characters from the individual scenes! The following function extracts all the names of characters from all the screenplay urls.

let allNames =
  scriptUrls
  |> List.map (fun (episode, url) ->
    let script = getScript url
    let scriptParts = script.Elements()

    let mainScript = 
        scriptParts
        |> Seq.map (fun element -> element.ToString())
        |> Seq.toArray

    // Now every element of the list is a single scene
    let scenes = splitByScene mainScript [] 

    // Extract names appearing in each scene
    scenes |> List.map getCharacterNames |> Array.concat )
  |> Array.concat
  |> Seq.countBy id
  |> Seq.filter (snd >> (<) 1)  // filter out characters that speak in only one scene

The only remaining problem was that many of the names were not actual names. The list was full of people called “PILOT” or “OFFICER” or “CAPTAIN”. After this I manually went through the results and filtered out the names that were actual names. This resulted in the characters.csv file.

Interactions between characters

To construct the social networks, I wanted to extract all the occasions when two characters talk to each other. This happens if two characters speak within the same scene (I decided to ignore situations when people talk with each other over a transmitter, and therefore across scenes). Extracting characters that are part of the same dialogue was now similified because I could just look at the list of characters I put together in the previous step.

let characters = 
    File.ReadAllLines(__SOURCE_DIRECTORY__ + "/data/characters.csv") 
    |> Array.append (Seq.append aliasDict.Keys aliasDict.Values |> Array.ofSeq)
    |> set

Here I created a set of all character names and their aliases to use for lookup and filtering. Then I used it to search through the characters appearing in each scene:

let scenes = splitByScene mainScript [] |> List.rev

let namesInScenes = 
    scenes 
    |> List.map getCharacterNames
    |> List.map (fun names -> names |> Array.filter (fun n -> characters.Contains n))

Then I used the filtered list of characters to define the social network:

// Create weighted network
let nodes = 
    namesInScenes 
    |> Seq.collect id
    |> Seq.countBy id        
    // optional threshold on minimum number of mentions
    |> Seq.filter (fun (name, count) -> count >= countThreshold)

let nodeLookup = nodes |> Seq.map fst |> set

let links = 
    namesInScenes 
    |> List.collect (fun names -> 
        [ for i in 0..names.Length - 1 do 
            for j in i+1..names.Length - 1 do
                let n1 = names.[i]
                let n2 = names.[j]
                if nodeLookup.Contains(n1) && nodeLookup.Contains(n2) then
                    // order nodes alphabetically
                    yield min n1 n2, max n1 n2 ])
    |> Seq.countBy id

This gave me a list of nodes with the number of times they spoke across the whole script, which I used to specify the size of nodes in the visualizations. Then I created a link for each time two characters spoke within the same scene, and counted them to get the strength of each relationship. Together, the nodes and links defined the full social network.

Finally, I exported the data into the JSON format. You can find all the networks, both the global one and the individual episodes’ networks, on my Github. 
The full code for this step is in the getInteractions.fsx script.

Character mentions in the script

I also decided to look at where each of the characters appears throughout the six movie cycle, resulting in the timelines chart. For this I looked at all the times the name of the character was mentioned in the scripts, not limited to the number of times the character actually spoke. The code for this was largerly similar to extracting the interactions in the previous section, only here I was looking for all the mentions and not only for the dialogues. To get the actual timelines, I also kept track of the scene numbers. The following code snippet returns a list of scene numbers and characters that are mentioned in them.

let scenes = 
    splitByScene mainScript [] |> List.rev
let totalScenes = scenes.Length

scenes
|> List.mapi (fun sceneIdx scene -> 
    // some names contain typos with lower-case characters
    let lscene = scene |> Array.map (fun s -> s.ToLower()) 

    characters
    |> Array.map (fun name -> 
        lscene 
        |> Array.map (fun contents -> if containsName contents name then Some name else None )
        |> Array.choose id)
    |> Array.concat
    |> Array.map (fun name -> mapName (name.ToUpper()))
    |> Seq.distinct 
    |> Seq.map (fun name -> sceneIdx, name)
    |> List.ofSeq)
|> List.collect id,
totalScenes

To get the full timeline, I used the scene numbers to map each episode into a [episode index−1,episode index]interval, giving me a relative scale of where in the episode the character appeared. The scene times in intervals [0,1] are from Episode I, in [1,2] correspond to Episode II etc.

// extract timelines
[0 .. 5]
|> List.map (fun episodeIdx -> getSceneAppearances episodeIdx)
|> List.mapi (fun episodeIdx (sceneAppearances, total) ->
    sceneAppearances 
    |> List.map (fun (scene, name) -> 
        float episodeIdx + float scene / float total, name))

I saved this data into an ill-formated pseudo-csv file, where each row contains a character name and the relative times the character appeared across the 6 films, separated by commas.

The full code is in the getMentions.fsx file.

Adding non-speaking characters

When I compared the interactions and the characters that were mentioned in the scripts, I noticed that something was different: there was no R2-D2 and no Chewbacca. Not only the poor Wookiee didn’t receive any medal at the end of Episode IV (like the human heroes did), but he was missing from all the dialogues in the screenplays! Actually, both Chewbacca and R2-D2 appear throughout the scripts as non-speaking characters. They are mentioned in lines like

Artoo finds something that makes him whistle wildly.

or

A sudden frown crosses Chewbacca’s face and he begins yelling gibberish at the tiny robot.

It wouldn’t be a proper Star Wars social network if we ignored these two important characters. To correct for the racial discrimination against Wookiees and astromech droids, I decided to insert them into the social network defined by dialogues.

I looked again at all the times these characters got mentioned in the screenplay. The mentions also define a social network where two characters are linked together if they are mentioned in the same scene. Because the characters are included in the scene descriptions and they also talk about each other more than they actually appear on screen, the network defined by all mentions is much messier than the network defined by direct interactions. Because of this, I couldn’t use the Chewbacca’s and R2-D2’s connections directly in the social network. I could however use it as an approximation.

I extracted the node sizes and links for the two missing characters from the network defined by mentions. To transform them into connections from the social network, I decided to scale all their values by comparing them with similar characters. For this, I chose C-3PO as a proxy for R2-D2, and Han as a proxy for Chewie, assuming their interactions were largely similar. Then I applied the following formula to compute the link strengths in the dialogue social network (this example uses Chewbacca and Han):

$$// <![CDATA[ \text{link weight}_{\text{dialogues}}(\text{Chewie}, x) = w_{\text{Han}} \times \text{link weight}_{\text{mentions}}(\text{Chewie}, x) // ]]&gt;$$

I used this highly scientific equation to recompute all the link strenghts and also the node sizes. The weight factor was around 0.5 both for Han and C-3PO, which means the characters were on average mentioned two times more than they actually spoke in the films. After this, I discarded all the links that had their weight smaller than one. The code for this step is relatively long but not very interesting.

Visualizations

After manually adding back Chewbacca and R2-D2, I finally had the full set of social networks both for the individual movies and for the full franchise. I used the Force to visualize the networks… well, I actually used the force-directed network layout from the D3.js library. This visualization method uses a physical simulation of charged particles to generate a network layout. The most important part of the D3 JavaScript file is the following part:

d3.json("starwars-episode-1-interactions-allCharacters.json", function(error, graph) {
  /* More code here */
  var link = svg.selectAll(".link")
      .data(graph.links)
    .enter().append("line")
      .attr("class", "link")
      .style("stroke-width", function(d) { return Math.sqrt(d.value); });

  var node = svg.selectAll(".node")
      .data(graph.nodes)
    .enter().append("circle")
      .attr("class", "node")
      .attr("r", 5)
      .style("fill", function (d) { return d.colour; })
      .attr("r", function (d) { return 2*Math.sqrt(d.value) + 2; })
      .call(force.drag);
  /* More code here */
});

In the previous steps I saved all the networks in JSON. Here I loaded them and defined the nodes and links using the data from the JSON. For each node, I also added a colour for each node, and a value specifying the importance (given by the number of times the character spoke in the script). This defines the radius r attribute for the node object, scaling each node realative to its importance in the network. Similarly for the links, I also stored the strength of each link in the JSON file, and here I used it to define the stroke-width of each link.

Centrality analysis

As the final step, the analysis of centrality of individual characters was probably the simplest one. I used the RProvider together with the igraph R package to analyse the networks from F#. First I loaded the network using the JSON type provider from FSharp.Data:

open RProvider.igraph

let [<Literal>] linkFile = __SOURCE_DIRECTORY__ + "/networks/starwars-episode-1-interactions.json"
type Network = JsonProvider<linkFile>

let file = __SOURCE_DIRECTORY__ + "/networks/starwars-full-interactions-allCharacters.json"
let nodes = Network.Load(file).Nodes |> Seq.map (fun node -> node.Name) 
let links = Network.Load(file).Links

The links variable contains all the links in the network, where the end nodes are specified
by their indices. To simplify the interpretation of results, I mapped all the indices to the corresponding
character names:

let nodeLookup = nodes |> Seq.mapi (fun i name -> i, name) |> dict
let edges = 
    links
    |> Array.collect (fun link ->
        let n1 = nodeLookup.[link.Source]
        let n2 = nodeLookup.[link.Target]
        [| n1 ; n2 |] )

Then I created an undirected graph object using the igraph library:

let graph =
    namedParams["edges", box edges; "dir", box "undirected"]
    |> R.graph

Calculating the betweenness and degree centrality was then as simple as

let centrality = R.betweenness(graph)
let degreeCentrality = R.degree(graph)

The full code that I used to produce the tables comparing the individual episodes is
available here.

Summary

As with most data science tasks, the most difficult step here was getting the data into a good shape. The Star Wars screenplays all had slightly different formats so I spent most of the time figuring out the common properties of the HTML documents to create a common function for parsing them and for identifying character names. After that, it was just a little fight for Wookiee and droid equality when I decided to combine data sources to infer their social connections. I made the extracted networks available in JSON on my GitHub so you can also play with them.

I hope you endjoyed the visualizations, and may the force be with you!

Links

• Source code: github.com/evelinag/StarWars-social-network
• Extracted networks in JSON format: github.com/evelinag/StarWars-social-network/tree/master/networks
• Screenplays: imsdb.com

Original.

Related:

• OpenText Data Digest, Dec 18: The Data Awakens
• Deep Learning in a Nutshell – what it is, how it works, why care?
• Poincare Conjecture, Perelman way, and Topology of social networks