KDnuggets Home » News » 2016 » Dec » Software » Navigating the World of Big Data Analytics ( 16:n44 )

Navigating the World of Big Data Analytics


Fulcrum Agile Analytics Lab- helps our partners test new technologies, new methodologies and new data sets quickly in an environment that can scale up and down and that meets all of their security and compliance requirements. Read to learn more and schedule a consultation.



Richard Vermillion, CEO Fulcrum Analytics.

New analytics software and methods are developing at such a rapid pace that it’s difficult for an enterprise to keep up. Even the most talented and innovative data science and analytics teams are faced with two key challenges. How do you stay on top of the ever-changing analytics landscape? And, once the right tools are identified, how do you get the analytics and technical resources to test them in a timely manner?

agile-big-data

Analytics teams we’ve worked with in banking and insurance have encountered some combination of the following problems:

  • They have a defined analytical question, but don’t know which of several potential approaches is best suited to answer it – and maybe aren’t even aware of all the latest relevant tools.
  • They have an idea for a technical solution to try, but don’t have enough hands-on experience with the latest software/versions/packages available to get up and running efficiently.
  • They have the capability to test a new solution, but for one reason or another (cumbersome procurement processes, compliance issues around open source software, budget, bureaucracy) cannot get easy access to the software they need.
  • They cannot easily get the right data and tools in the same place. Security concerns or current compliance regulations prohibit them from moving internal data to the cloud or from bringing external data sources inside, or they simply don’t have the resources and experience to navigate, integrate and clean the external data they want to use.

A truly agile analytics group would be able to bring external and internal data together in one place where they could quickly test out a variety of the latest tools available with experts to guide them along. Today’s financial service enterprise can’t achieve this analytical agility on its own – it’s simply too much to ask given the resource constraints, internal structure, and regulatory environment.

This is exactly why we created our Agile Analytics Lab - to help our partners test new technologies, new methodologies and new data sets quickly in an environment that can scale up and down and that meets all of their security and compliance requirements.

One recent example is our work with a large financial services company that was challenged with massive amounts of data stored across several different legacy databases, all managed by different teams. These datasets included purchase history, customer information and product information. There was no way for anyone in the company to view data across these systems and because of the size of the data sets, a unified view was not feasible with a single machine or a standard database.

Leveraging our Agile Analytics Lab, we created reusable data processing pipelines, loaded the data into Hadoop and readied it for analysis. Once the central repository was established, we then cubed the data using Hive query language with Spark as the compute engine. This enabled fast querying of metrics of interest across all combinations of relevant variables. The output table was in HBase, a key value store, and was queried with Impala, which was made for rapid response and returns the results quickly. Finally, a Tableau dashboard was created to view measures and dimensions so the business end users can utilize the results.

Aggregating this information in one location gave business users a holistic view of their data that was not previously available. Best of all, we were able to accomplish this without requiring time from their internal IT team. All Fulcrum requested was a file transfer onto our secure servers and our team assumed the burden of data management and processing. Through an iterative process, we tested multiple solutions in parallel and were quickly able to determine the best technologies for this use case. The client was up and running quickly and is now able to unlock new insights and new value from their data. This set them up for greater exploitation of their data assets. We’re helping them rapidly develop and tune machine learning models, while providing training and support to their analytics team on cutting edge technologies like Tensorflow, Keras, and Theano for deep learning. The lab proved to be the perfect instrument to bring agile analytics to this financial services company.

At Fulcrum we manage labs for some of the largest banks and insurance companies in the world.  We help them achieve analytical agility by providing them virtual infrastructure to spin up exactly the servers and software they need.  We guide our partners from the selection process to implementing packages to support in building models with the latest tools.  We curate a variety of external data sources so that they are easy to integrate with their existing data – this involves identifying, cleaning, and providing metadata so they are ready to go.  It’s the job of our data science team to stay on top of the latest developments so that our enterprise partners can focus on their job – obtaining the best analytical solutions for their business.

Click here to learn more about our Agile Analytics Lab and schedule a consultation