Analyzing Analysts to Build Better Analysis Software

Our study how analysts used Mode led to major updates designed to fit how data analysts and business analysts actually use data - there's no one-size-fits-all tool and analysis doesn't end with the analyst.

By Benn Stancil (Mode Analytics).

Seven months ago, we launched a public beta of Mode, a tool for analysts to create and share work. Importantly, the beta product gave us something that research and intuition couldn’t: usage data on how analysts work. After studying the way analysts use Mode, we've recently released major updates designed around what we've found to be important to data analysts and data scientists. In the spirit of open-sourcing some of our analysis code, we want to share some of our findings.

There's No One-Size-Fits-All Tool

We are building a tool to integrate perfectly with an analyst's existing workflow. Along the way, one of the toughest decisions we had to make—and one where our own opinions were strongest—was how Mode's SQL editor should be oriented. Some people wanted the SQL query stacked on top of the data, some wanted a side-by-side view, and others (like me) waffled back and forth. So we decided to see if our data held the answer. What kind of editor would most users want?

After looking, we discovered the answer: it depends. But surprisingly, not only does it depend on the person, it depends on how far along an analyst is in their work.

When analysts are first writing a query, the queries tend to be short and the results usually include a lot of columns. Short queries and wide tables are better suited for a horizontal editor. But as analysts edit their reports and get closer to the final version, queries get longer and datasets get narrower. Both of these attributes are better suited for a vertical editor.

Mode: Query Length by Query Number

Mode Columns Returned by Query Number

This data guided the design of a flexible editor. But it exposed a more important truth—no one design works for everything. Analysis passes through different stages, and what's best suited for one stage might not be the best for another.

Analysis Doesn't End With The Analyst

Mode is used by both analysts (people who write queries) and business users (those who primarily view results). When studying the behavior of these two groups, we discovered that they need very different things from an analytics tool.

Analysts often start on Mode's home page, which contains a directory of reports and data. This suggests they're looking to understand what work their team has done, and want to stay on top of current projects. To make this experience better in Mode, we added an activity feed that shows when someone shares a report, updates a report, or creates a list.

Mode Sessions First Event Business users, by contrast, tend to enter the Mode via links to specific reports shared by analysts. This highlights how important it is for analysts to be able to easily distribute their work throughout their companies. The work of analyst doesn't end with producing a great chart or finding something insightful—it ends when that conclusion is also understood by the right person.

Bio: Benn Stancil is the Chief Analyst of Mode Analytics, a company building collaborative, SQL-based analysis tools for analysts working with both public and private data. Prior to Mode, Benn was a senior analyst at Microsoft and Yammer, where he helped lead product analytics. Benn also worked as an economic analyst at the Carnegie Endowment for International Peace in Washington, DC.