Big Data and Hadoop- Drivers of Future

Big Data & Hadoop are the buzz words now days. Every service provider is positioning itself as a champion in Big Data & Hadoop space. Organizations are finding it very difficult to understand & make it a workable solution because of the “infancy stage” and “complexity” involved.

Let’s start with answering a fundamental question “does my organization really need Big Data solutions?”  To answer this, one needs to understand the organizations’ current data, analytical practices and critically evaluate future requirements. Are they finding the current data, analytical processes & systemsinefficient to deliver results in fast and actionable manner? Or will they become inefficient and insufficient in near future due to fastgrowing requirements and new sets of data challenges like web-log, social media, and audio/video data? If answer to any of the above questions is yes, then organizations need to start thinking about Big Data strategy and road map seriously.

The first baby step towards drawing Big Data strategy is to understand it from IT/Data and Analytical point of view. Organizations need to decide if this Big Data strategy is an efficiency booster or path to new capabilities/discoveries or both. At the foundation level, either of the goals will require investment in IT and skill sets. The Investment in IT may be controlled using Apache Hadoop and other open source platforms but training and skill development is surely going to be an ongoing journey.  To achieve latter part of the goal (new capabilities/discoveries) organizations need to develop the Big Data strategy not only from IT point of view but mainly from business & analytical point of view. It’s like a baby learning not only how to walk but also where to head…..

If we see the market land space of Big Data and Hadoop, there are numerous player and they are providing solutions to different aspect of Big Data deployment.  These players can be categorized into a few broad categories.

  1. There are players who focus on technology development like Hbase, Hive and Hadoop HDFS etc.
  2. The Second category is that of Service Companies providing IT solutions like setting up Hadoop platform etc and possibly helps Integration between legacy system and Hadoop.
  3. Third category is of companies is in the space of development of framework and application that run on top of Hadoop.
  4. The Forth category of companies is the BI outsourcing companies that provide value by running BI jobs with shorter turnaround time.
  5. Highest value comes from another category of companies that use Analytics/data mining to generate new insights from the huge amounts of data, hitherto not feasible on legacy RDBMS systems.

Affine Analytics provides solutions into two of the above categories. One is using BI atop Hadoop and second is performing predictive analytics on the data which were huge and unmanageable in the past and also discovering hidden patterns. We use Big Data to take predictive analytics accuracy to next level deploying Machine Learning Techniques and other advanced techniques, made feasible by Big Data.

At Affine we are developing Big Data capability in literally a big way.Besides having practically every analyst trained on Big Data platform, wealso have an in-house Hadoop Analytical Lab “Hal”(A Hindi word meaning solution)which increase the efficiency of our operations multifold and gives us the capability to mine unstructured and semi-structure data to generate new insights which helps our esteemed clients’ business take faster and better decisions.

Affine is currently working on the Telecom CDR data to generate insights which were not known earlier and to improve performance of existing strategies. It requires mining of huge amount of data using Big Data platforms like Hadoop & Hive. Affine is trying to use call data records or voice/data transaction data to better manage churn, come up with better strategies to increase ARPU, Increase usage of VAS etc.

Ashish Maheshwari

Director – Client Delivery, Affine Analytics

For feedback and comments I can be reached at


Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Google+ photo

You are commenting using your Google+ account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )


Connecting to %s