Top Data Scientist Daniel Tunkelang on Data Science Project Scope… and Reducing It

Respected Data Scientist Daniel Tunkelang shares some insight into problems lying at the crossroads of software engineering and data science, and prescribes one major solution: reduce scope!

By Daniel Tunkelang, Data Science Consultant.

Some code

When I left the cocoon of academia nearly two decades ago to become a software engineer, I quickly learned two lessons.

The first was a version of the project management triangle: you can commit to a project’s scope, time, or quality - but not to all three. As I was told, pick any two.

The second was that my main job as a software engineer was not to design algorithms or write code: it’s to reduce the uncertainty of project estimates.

These two lessons have served me well throughout my career as a software engineer and data scientist. But lately I feel there’s a simpler principle that encapsulates them: reduce scope.

I’m not just saying that scope creep is the root of all project management evil (though it is). I’m arguing that we can’t make meaningful estimates for large projects - no matter how much our managers and planning processes dictate that we do so. Our only remedy is to make the projects smaller - by reducing scope.

The project management triangle suggests that we can manage large projects if we’re willing to sacrifice our estimates of time or quality. But this suggestion is at best theoretical. Can you imagine committing to a project scope without a time line? Or worse, without committing to its quality?

Meanwhile, agile software development has taught us the importance of early and continuous delivery. In effect, agile principles tell us to reduce scope, driving our first deliverable to a true minimum viable product (MVP) and subsequent deliverables to minimum units of incremental improvement.

You may be thinking that this advice is common sense that is already reflected in the software engineering canon. Indeed it is. Yet I’m surprised by how often I find myself advising people to reduce scope.

So please consider this post a public service announcement to engineers and engineering leaders everywhere. When in doubt, reduce scope. The rest will work itself out.

ps. If you liked this post, you may also like “Dream Big. Execute Incrementally”.

Bio: Daniel Tunkelang is a data science and engineering executive who has built and led some of the strongest teams in the software industry.

Original. Reposted with permission.