# The Star Wars social networks – who is the central character?

Data Scientist looks at the 6 Star Wars movies to extract the social networks, within each film and across the whole Star Wars universe. Network structure reveals some surprising differences between the movies, and finds who is actually the central character.

### How I did the analysis

As this is part of the F# Advent calendar,Â I used F# for most of the analysis. I combined it together with D3.js forÂ the social network visualizations, and with R for the network centrality analysis.Â You can find all the source code on my GitHub. BecauseÂ the whole code turned out to be relatively long, here I look only at some of the more interestingÂ parts.

### Parsing the screenplays

I started by downloading all the scripts for the 6 Star Wars movies. They are freely available fromÂ The Internet Movie Script Database (IMSDb), for example here’s the script forÂ Episode IV: The New Hope. The screenplays are onlyÂ in the form of drafts that sometimes differ from the actual films – the differences areÂ however not very big.

The first step was to parse the screenplays.Â To complicate things, each ofÂ them used a slightly different format. The screenplays themselves were in HTML, either within theÂ <td class="srctext"></td> tags, or within the <pre></pre> tags.Â To extract the contents of each script,Â I used the Html Parser from F# Data libraryÂ which allows accessing individual tags in a HTML document using statements like

open FSharp.Data
let url = "http://www.imsdb.com/scripts/Star-Wars-A-New-Hope.html"


The full code for this part is available in the parseScripts.fs file.

The next step was to extract relevant information from the scripts.Â In general, a script looks like this:

INT. GANTRY - OUTSIDE CONTROL ROOM - REACTOR SHAFT

Luke moves along the railing and up to the control room.

[...]
LUKE
He told me enough!  It was you
who killed him.

Shocked, Luke looks at Vader in utter disbelief.

LUKE
No.  No.  That's not true!
That's impossible!


Each scene starts with its setting: INT. (interior) or EXT. (exterior) and the location of the scene.Â Then there can be some free text describing what is happening and how does theÂ scene look. In the dialogues, the names of characters are writtenÂ in capital letters (sometimes also in bold), followed by what they are saying.

The main signposts in the screenplay are the INT. and EXT. statements whichÂ serve as scene separators. These were writtenÂ consistently in bold in all the 6 scripts and I used them to split them intoÂ individual scenes:

// split the script by scene
// each scene starts with either INT. or EXT.
let recsplitByScene (script : string[]) scenes =
let scenePattern = "<b>[0-9]*(INT.|EXT.)"
let idx =
onmouseover="showTip(event, 'fs4', 10)" onmouseout="hideTip(event, 'fs4', 10)">script
|> Seq.tryFindIndex (fun line -> Regex.Match(line, scenePattern).Success)
match idx with
| Some i ->
let remainingScenes = script.[i+1 ..]
let currentScene = script.[0..i-1]
splitByScene remainingScenes (currentScene :: scenes)
| None -> script :: scenes


This is a recursive function that takes the full screenplay and looks for theÂ specific pattern, which is EXT. or INT. in bold, optionally preceded by a scene number.Â The function goes through the string and when itÂ encounters the scene break, it splits the string into the current scene and the remaining text. Then itÂ runs recursively until all the scenes are extracted, using the scenes variable as an accumulator.

### Getting list of characters

The previous function gave me a list of scenes for each of the movies. Extracting the charactersÂ out of them turned out to be more difficult.Â Some of the scenes followed the format that I showed in the example above, some of theÂ scenes only used character names followed by a colon and their dialogue,Â all within the same line in the text. The only common property between the differentÂ formats was that the names were always written in capital letters.

I resorted to regular expressions, one for each screenplay format:

// Extract names of characters that speak in scenes.
// A) Extract names of characters in the format "[name]:"
let getFormat1Names text =
let matches = Regex.Matches(text, "[/A-Z0-9-]+*:")
let names =
seq { for m in matches -> m.Value }
|> Seq.map (fun name -> name.Trim([|' '; ':'|]))
|> Array.ofSeq
names

// B) Extract names of characters in the format "<b> [name] </b>"
let getFormat2Names text =
let m = Regex.Match(text, "<b>[]*[/A-Z0-9-]+[]*</b>")
if m.Success then
let name = m.Value.Replace("<b>","").Replace("</b>","").Trim()
[| name |]
else [||]

Each of the regular expressions matches not only capital letters, but also numbers, hypens, spaces and slashes. All because the characters in Star Wars use names like “R2-D2” or even “FODE/BEED”.

To actually extract the list of characters that appear across the films, I also had to takeÂ into account the fact that many characters use multiple names. Senator Palpatine turns intoÂ Darth Sidious and then into The Emperor, Amidala disguises herself as Padme (or the other way around?).

Because of this I manually put together a simple file aliases.csvÂ where I specified which names I consider to be the same. I used these as a mapping onto a uniqueÂ name for each character (except for Anakin and Darth Vader).

let aliasFile = __SOURCE_DIRECTORY__ + "/data/aliases.csv"
// Use csv type provider to access the csv file with aliases
type Aliases = CsvProvider<aliasFile>

/// Dictionary for translating character names between aliases
let aliasDict =
|> Seq.map (fun row -> row.Alias, row.Name)
|> dict

/// Map character names onto unique set of names
let mapName name = if aliasDict.ContainsKey(name) then aliasDict.[name] else name

/// Extract character names from the given scene
let getCharacterNames (scene: string []) =
let names1 = scene |> Seq.collect getFormat1Names
let names2 = scene |> Seq.collect getFormat2Names
Seq.append names1 names2
|> Seq.map mapName
|> Seq.distinct

Now I could finally extract names of characters from the individual scenes! The followingÂ function extracts all the names of characters from all the screenplay urls.

let allNames =
scriptUrls
|> List.map (fun (episode, url) ->
let script = getScript url
let scriptParts = script.Elements()

let mainScript =
scriptParts
|> Seq.map (fun element -> element.ToString())
|> Seq.toArray

// Now every element of the list is a single scene
let scenes = splitByScene mainScript []

// Extract names appearing in each scene
scenes |> List.map getCharacterNames |> Array.concat )
|> Array.concat
|> Seq.countBy id
|> Seq.filter (snd >> (<) 1)  // filter out characters that speak in only one scene

The only remaining problem was that many of the names were not actual names. The list was fullÂ of people called “PILOT” or “OFFICER” or “CAPTAIN”. After this I manually went through theÂ results and filtered out the names that were actual names. This resulted in the characters.csv file.

### Interactions between characters

To construct the social networks, I wanted to extract all the occasions when two characters talk to each other.Â This happens if two characters speak within the same scene (I decided to ignore situations when peopleÂ talk with each other over a transmitter, and therefore across scenes). Extracting characters that areÂ part of the same dialogue was now similified because I could just look at the list of characters I put togetherÂ in the previous step.

let characters =
|> Array.append (Seq.append aliasDict.Keys aliasDict.Values |> Array.ofSeq)
|> set

Here I created a set of all character names and their aliases to use for lookup and filtering.Â Then I used it to search through the characters appearing in each scene:

let scenes = splitByScene mainScript [] |> List.rev

let namesInScenes =
scenes
|> List.map getCharacterNames
|> List.map (fun names -> names |> Array.filter (fun n -> characters.Contains n))

Then I used the filtered list of characters to define the social network:

// Create weighted network
let nodes =
namesInScenes
|> Seq.collect id
|> Seq.countBy id
// optional threshold on minimum number of mentions
|> Seq.filter (fun (name, count) -> count >= countThreshold)

let nodeLookup = nodes |> Seq.map fst |> set

namesInScenes
|> List.collect (fun names ->
[ for i in 0..names.Length - 1 do
for j in i+1..names.Length - 1 do
let n1 = names.[i]
let n2 = names.[j]
if nodeLookup.Contains(n1) && nodeLookup.Contains(n2) then
// order nodes alphabetically
yield min n1 n2, max n1 n2 ])
|> Seq.countBy id

This gave me a list of nodes with the number of times they spoke across the whole script, which I used toÂ specify the size of nodes in the visualizations. Then I created a link for each time twoÂ characters spoke within the same scene, and counted them to get the strength of each relationship.Â Together, the nodes and links defined the full social network.

Finally, I exported the data into the JSON format. You can find all the networks, both the globalÂ one and the individual episodes’ networks, on my Github.Â
The full code for this step is in the getInteractions.fsxÂ script.

### Character mentions in the script

I also decided to look at where each of the characters appears throughout the six movie cycle, resultingÂ in the timelines chart. For this I looked at all the times the name of the character wasÂ mentioned in the scripts, not limited to the number of times the character actually spoke.Â The code for this was largerly similar to extracting the interactions in the previous section,Â only here I was looking for all the mentions and not only for the dialogues.Â To get the actual timelines, I also kept track of the scene numbers. The following code snippetÂ returns a list of scene numbers and characters that are mentioned in them.

let scenes =
splitByScene mainScript [] |> List.rev
let totalScenes = scenes.Length

scenes
|> List.mapi (fun sceneIdx scene ->
// some names contain typos with lower-case characters
let lscene = scene |> Array.map (fun s -> s.ToLower())

characters
|> Array.map (fun name ->
lscene
|> Array.map (fun contents -> if containsName contents name then Some name else None )
|> Array.choose id)
|> Array.concat
|> Array.map (fun name -> mapName (name.ToUpper()))
|> Seq.distinct
|> Seq.map (fun name -> sceneIdx, name)
|> List.ofSeq)
|> List.collect id,
totalScenes

To get the full timeline, I used the scene numbers to map each episode into aÂ [episode indexâˆ’1,episode index]interval, giving me a relative scale of where in the episode the character appeared. The scene times in intervals [0,1] are from Episode I, in [1,2] correspond to Episode II etc.

// extract timelines
[0 .. 5]
|> List.map (fun episodeIdx -> getSceneAppearances episodeIdx)
|> List.mapi (fun episodeIdx (sceneAppearances, total) ->
sceneAppearances
|> List.map (fun (scene, name) ->
float episodeIdx + float scene / float total, name))

I saved this data into an ill-formated pseudo-csv file,Â where each row contains a character name and the relative times the character appeared across the 6 films,Â separated by commas.

The full code is in theÂ getMentions.fsx file.

When I compared the interactions and the characters that were mentioned in the scripts,Â I noticed that something was different: there was no R2-D2 and no Chewbacca. Not onlyÂ the poor Wookiee didn’t receive any medal at the end of Episode IV (like the human heroes did),Â but he was missing from all the dialogues in the screenplays! Actually, both Chewbacca andÂ R2-D2 appear throughout the scripts as non-speaking characters. They are mentioned in lines like

Artoo finds something that makes him whistle wildly.

or

A sudden frown crosses Chewbacca’s face and he begins yelling gibberish at the tiny robot.

It wouldn’t be a proper Star Wars social network if we ignored these two important characters. ToÂ correct for the racial discrimination against Wookiees and astromech droids, I decided to insert themÂ into the social network defined by dialogues.

I looked again at all the times theseÂ characters got mentioned in the screenplay. The mentions also define a social network where twoÂ characters are linked together if they are mentioned in the same scene. Because the charactersÂ are included in the scene descriptions and they also talk about each other more than they actuallyÂ appear on screen, the network defined by all mentions is much messier than the network definedÂ by direct interactions. Because of this, I couldn’t use the Chewbacca’s and R2-D2’s connectionsÂ directly in the social network. I could however use it as an approximation.

I extracted the node sizes and links for the two missing characters from the networkÂ defined by mentions. To transform them into connections from the social network,Â I decided to scale all their values byÂ comparing them with similar characters. For this, I chose C-3PO as a proxy for R2-D2, and HanÂ as a proxy for Chewie, assuming their interactions were largely similar.Â Then I applied the following formula to compute the link strengths in theÂ dialogue social network (this example uses Chewbacca and Han):

I used this highly scientific equation to recompute all the link strenghts and also the nodeÂ sizes. The weight factor was around 0.5 both for Han and C-3PO, which means the charactersÂ were on average mentioned two times more than they actually spoke in the films. After this, I discardedÂ all the links that had their weight smaller than one.Â The code for this stepÂ is relatively long but not very interesting.

### Visualizations

After manually adding back Chewbacca and R2-D2, I finally had the full set of social networks both for theÂ individual movies and for the full franchise. I used the Force to visualize the networks… well, IÂ actually used the force-directed network layoutÂ from the D3.js library.Â This visualization methodÂ uses a physical simulation of charged particles to generate a network layout. The most important partÂ of the D3 JavaScript file is the following part:

d3.json("starwars-episode-1-interactions-allCharacters.json", function(error, graph) {
/* More code here */
.enter().append("line")
.style("stroke-width", function(d) { return Math.sqrt(d.value); });

var node = svg.selectAll(".node")
.data(graph.nodes)
.enter().append("circle")
.attr("class", "node")
.attr("r", 5)
.style("fill", function (d) { return d.colour; })
.attr("r", function (d) { return 2*Math.sqrt(d.value) + 2; })
.call(force.drag);
/* More code here */
});

In the previous steps I saved all the networks in JSON. Here I loaded them and defined the nodes andÂ links using the data from the JSON. For each node, I also added a colour for each node, and a value specifyingÂ the importance (given by the number of times the character spoke in the script). This definesÂ the radius r attribute for the node object, scaling each node realative to its importance in the network.Â Similarly for the links, I also stored the strength of each link in the JSON file, and here I used it toÂ define the stroke-width of each link.

### Centrality analysis

As the final step, the analysis of centrality of individual characters was probably the simplest one.Â I used the RProvider together with theÂ igraph R package to analyse the networks from F#. First I loaded the networkÂ using the JSON type provider from FSharp.Data:

open RProvider.igraph

let [<Literal>] linkFile = __SOURCE_DIRECTORY__ + "/networks/starwars-episode-1-interactions.json"

let file = __SOURCE_DIRECTORY__ + "/networks/starwars-full-interactions-allCharacters.json"
let nodes = Network.Load(file).Nodes |> Seq.map (fun node -> node.Name)
let links = Network.Load(file).Links

The links variable contains all the links in the network, where the end nodes are specified
by their indices. To simplify the interpretation of results, I mapped all the indices to the corresponding
character names:

let nodeLookup = nodes |> Seq.mapi (fun i name -> i, name) |> dict
let edges =
[| n1 ; n2 |] )

Then I created an undirected graph object using the igraph library:

let graph =
namedParams["edges", box edges; "dir", box "undirected"]
|> R.graph

Calculating the betweenness and degree centrality was then as simple as

let centrality = R.betweenness(graph)
let degreeCentrality = R.degree(graph)

The full code that I used to produce the tables comparing the individual episodes is
available here.

### Summary

As with most data science tasks, the most difficult step here was getting the data intoÂ a good shape. The Star Wars screenplays all had slightly differentÂ formats so I spent most of the time figuring out theÂ common properties of the HTML documents to create a common function for parsing themÂ and for identifying character names.Â After that, it was just a little fight for Wookiee and droid equalityÂ when I decided to combine data sources to infer their social connections.Â I made the extracted networks available in JSON on my GitHubÂ so you can also play with them.

I hope you endjoyed the visualizations, and may the force be with you!