Cognizant Minds

On a not so lazy afternoon, I bumped in to the terminology “Big Data” once again. This time due to official reasons where we need to formulate the strategy for our big data venture. While I have been exposed to the concept from early last year, for last 2 months, everyone is talking about it vehemently. There are clear trends related to acquisitions and product offerings that give very positive sign about the buzz around the concept. While this blog is not to bore you people with the theory of big data, it’s definitely would make sense to define it in a more comprehensible and easier way. Subsequently I would try to point out the signs and the reasons why I think Big Data has a definitive future and also mention which sectors of the industry have more prospects to get benefited.

Big Data:
What: This is a theoretical renaming of the volume of data that is created and dealt within an enterprise on a period basis. You could imagine the amount of data that gets created, re-created, interpreted, summarized in side any organization or outside. Think about the information posted in twitter, Facebook on a daily basis. Think about the wikis and the blogs and the forum posts that happen every minute, it will give us a general idea about the growth of volume of data.
Storage: Now the next step would be to understand the complexity arising due to this growth. The biggest challenge with the data is the storage requirement. Considering the rate of growth of this data, its definitely a challenging process for the legacy way of looking at data storage in a database.
Structure: One unique thing about Big data is the structure. It might be multi structured, structured or unstructured. Data collected is sometimes very complexly related. There is no way to fit them in a schema that is driven by the relational nature of data storage. Sometimes the data itself is not binary. Considering these, there is a new schema or storage scheme that needs to be encompassed to help this cause.
Processing: An enterprise collecting this humongous amount of data definitely wants to extract value from it and turning it into an asset How do you remove the noise from this high velocity, highly variable data and discover key insights while they are still relevant. These needs are driving the enterprise to create a value creation analytic platform which can model this data to user readable inference by identifying trends
Use: The clues that lie in big data can be the key to an enterprise’s future success, offering insights to optimize supply chains, uncover consumer behavior patterns, identify traffic and energy patterns, and much more. Rather than be a burden, all of this newfound knowledge can allow businesses to make effective long-term and real-time business decisions.

How : The best way of handling big data and supporting the extreme processing involved is to deploy custom hardware and software solutions for processing different types of big data workloads, and then combine these solutions with the existing enterprise data warehouse to create an integrated  information supply chain to deliver the analytical results to business users

Trends that suggest that Big Data is Happening.

Flurry of Custom Tools/Connector: Remember the lunch of IPHONE 4? Before even the due date arrived, a host of vendors started selling the case, screen protector ET all (accessories). That definitely gave a clue to the market that iPhone 4 is coming. Similarly there are flurries of custom tools offered in the markets which address to different segment of improvisation. Though the core offering is Hadoop, there are flavors of it mushrooming everywhere. Tablue and R have already devised and released connector to Hadoop for analytic and dash boarding.

Market Place: Biggest reason why all technology exists is that there are buyers or business reasons. Consider the possibility arising due to collaboration of market data with the data collected by weather stations (minute wise) and predicting the buying pattern of the demography. There are new collaboration and new market place arising to harness this possibility.

Change in Data Philosophy: Relational databases are completely failure in accommodating unstructured data and mulch-structured data. They need so much of tune and redesign to bring these formats to the form that can be stored in relational way that, the meaning and usability of the data is lost. A lot of important information is now being captured in silo and needs to be stored before relationship to other agents can be established. Shift in paradigm of usability of data has helped the cause of big data.

Need of Visualization : Enterprise and business users have started questioning the basis on which the analytic information is provided to the fraternity. So the need for visualizing the basis of deduction of the inference has become critical. A platform to facilitate is definitely most welcome.

Me too : This is a unique things about the indication of a successful trend. Every product vendor has lined up it Big Data offers. Be it IBMs Infosphere based platform or EMC or Oracle. When there are competing products on a same concept, it does a world of good – to the concept (remember cloud computing),

Though these are not the only reason for which Big data as a concept will be a buzz word for some time, these are definitely critical assessments and reason for which it will be considered.