Business Intelligence vs. Big Data

I was recently fortunate to speak at one of the database technology conferences held in Israel. Big parts of this conference revolved around 'Big Data' and I was asked to give the Business Intelligence perspective on this fascinating subject.

As part of my presentation, I attempted to show the impact of columnar database technology on the basic premise of business intelligence - the ability to have business users perform ad-hoc analytics and reporting tasks over as much data as possible

In order to do that, I represented a business user building a report over a very large operational database containing 13 tables, the largest of which hold 100 million and 40 million rows. While databases of this size were once rare - now, any company who has a properly tracked website quickly accumulates even more data than that.

To demonstrate, I was using a front-end analytics tool (SiSense Prism) to create reports that query the database directly - a feat not advisable with a relational database. So instead of querying the source database, the data was replicated (but unmodified) into a columnar database that was designed specifically for ad-hoc analytics - ElastiCube.

See video here.

... One thing you should understand from this video is how simple drag-and-drop operations of a business user (or multiple users) within his or her desktop tool turn into complex database operations that would choke any relational database (joining, grouping, aggregating), yet are handled by a columnar database without any difficulty.

Which brings me to my final, and perhaps most important point

Whichever business intelligence front end tool you pick, dealing with issues like this (and their subsequent side effects) is in fact 90% of the life-time cost of a BI solution, and often why the solution stops being used all together.

Read more.