What is BIG Data?

Often in the business world, things come along that drive change. Sometimes, the change could be subtle. Sometimes the change is dramatic. In a lot of cases, there are those that are early adopters, as well as those that one can count on to wait until the change is deemed mature by the industry experts. It is important also to put forth a value proposition to adopt the change; to change for change’s sake generally doesn’t help the bottom line. In some cases, this change crosses industries and technology, but sometimes the change is among clusters of business that gives one a competitive advantage over another.

Consider some of the transformations that have happened over the past few decades:

• • • • • The computer
        • The internet
        • Mobile technology
        • Laws (SOX, etc.)
        • Hybrid vehicles
        • Global economics
        • Social networks

Companies that were able to see these changes coming were able to prepare. Companies that didn’t see it coming were forced to adapt. It is obvious to see that being prepared is arguably the best alternative, but in many cases and for many reasons, this may not be financially, socially, or logistically possible during the early adoption period. Eventually, over time, more and more companies adapt the change into the culture or technology as it becomes more and more pervasive and valuable. Those that take the lead and the initiative generally experience growing pains, but also gain advantages of working out the kinks of emerging technology or change. Those that wait may need to face consequences for the delay but prolong the disruption to critical process for as long as possible.

So what would we say are some changes that are on the way if not already here that will help shape the next decade of business activity? One would argue that “Big Data” is a great candidate for the next major change we are beginning to feel the effects of as its existence begins to transform the business landscape.

So what is “Big Data”? According to Gartner[1] Big Data is defined as:

Big data is the term adopted by the market to describe extreme information management and processing issues which exceed the capability of traditional information technology along one or multiple dimensions to support the use of the information assets. Throughout 2010 and into 2011, big data has focused primarily on the volume issues of extremely large datasets generated from technology practices such as social media, operational technology, Internet logging and streaming sources. A wide array of hardware and software solutions has emerged to address the partial issue of volume. At this point, big data, or extreme information processing and management, is essentially a practice that presents new business opportunities.

The definition leaves the reader with a lot questions. What is “extreme information management”? What are the “hardware and software solutions” that attempt to address the issues?  What are the “new business opportunities”? Let’s take a look some of these questions in more detail.

Extreme Information Management

When we look at the sheer volume or magnitude of some of the datasets that would make up a Big Data solution, it is clear the traditional processes of data management will be challenged. The normal ways of loading, storing, processing, evaluating, and analyzing this information has to change in order to reap the benefits inside the datasets. This means the normal way a technologist would write a query needs to change. The usual backup and recovery process has to be re-considered. In addition, how this “extreme” information is combined with the “traditional” information is the new conundrum businesses that are going down this path are facing. In a practical sense, a company that has a traditional data warehouse in place will be faced with the challenge of aligning existing dimensions with the new “dimensions” from the big data arena. If this challenge can be solved for an enterprise, then the competitive advantage discussed earlier can be realized because the power of leveraging changes. If a company can at least begin evaluating the contents of big data as it relates to their enterprise, they will still gain valuable insights and correlations that are eluding the competitors that will wait out the change until it is mature.

Hardware and Software Solutions

When a company is considering a big data solution, immediately the existing infrastructure stack will be impacted. There are a lot of vendors that are stepping up into the technology gap to help customers move forward with initiatives. In a lot of cases, storage and processing power will become immediate needs. Deep in the internals of big data solutions is the idea of multiple worker servers that take on distributed work to break down the processing into smaller chunks. This can be achieved with potentially low cost servers, but a lot of them depending on the scale desired. The open source community has been a leader in this arena of solutions that help address the big data problem with low cost entry into public offerings. For the technologist, this means learning new ways of addressing data access using environments like Hadoop (http://hadoop.apache.org/) or HPCC (http://hpccsystems.com/). The traditional ETL tool vendors like Pentaho (http://www.pentaho.com/) are also attempting to get in the game by extending their products to work in a big data environment. In the long run, we expect the vendors to all address “Big Data” in some form or fashion, suggesting the change is here to stay.  Therefore, it would be wise to do some research in the current vendor offering or begin to experiment with some of the solution options that are available, in order to begin to reap some of the benefits of a big data solution.

New Business Opportunities

When considering what new business opportunities will be presented as a result of engaging in a Big Data environment, the potential is alluring. When dealing with datasets in the terabyte and petabyte range, the scope and makeup of the results go from specifics to generalities in rapid order. It is currently feasible to look at things in a dimensional model like perspective. It is difficult and in some cases no longer possible to calculate results as you would normally do in traditional database environments because the sheer physics have changed. But there are opportunities for businesses to integrate this new landscape into their current Business Intelligence environments. Some companies are finding ways to use sources from public web sites to private applications to drive value and competitive advantage. The government is even seeing the opportunity and critical need to get involved[2]. With this much activity related to a single discipline, we would recommend actively seeking out the opportunities in your own business environment. More than likely, your competitors already are.

What are the challenges?

Some of the challenges will be related to identifying sources, because sometimes these are external to a business and require creative thought to see the big picture and the analytical opportunities. Additional challenges will be related to bringing a technical staff up to speed on new technology and new ways of thinking related to size and scale of Big Data environments. Furthermore, there will be changes to “normal” data acquisition and storage policies. In addition to all this, there still is the integration issue. This issue can be more complex for companies that already have a mature data warehouse environment, and wish to navigate from a specific dimensional model to a less specific Big Data analysis. The bridge between the two can be daunting, physically and philosophically.

What does this mean for you?

Although Big Data is still an emerging technology and concept, there is enough momentum in the marketplace that warrants serious consideration for customers and clients. There are still significant challenges in bringing it all together in a comprehensive Business Intelligence environment. The ability for systems to collect more and more data points going forward is inevitable, and systems that can evaluate the large data sets will become more and more ingrained in the normal technical environment. The traditional systems and big data systems will become more seamless over time. The more companies can leverage the information investment on all the data points as they come available or affects their business, the more effective they will become.

If you would like to learn more about Big Data and what impact it may have in your environment or are just curious about how to approach the challenges in this new arena, we would love to help work through it with you.

For more information on BIG DATA or to see how we can help your business with its BIG DATA needs, contact us.