Go to Top

Innovation Breeds Complexity

The business intelligence and analytics space is in the midst of a new era of innovation and opportunity. Earlier this year I wrote an article posted on Inside Analysis titled “Wave of Analytic Innovation.” At its core, I explored how sophisticated users, quickly evolving technology, lower costs and new data are driving change within our analytic landscape. At the same time, the environment we rely on to support our business intelligence and analytic strategies continues to grow physically and geographically. The enterprise data warehouse  delivers critical value; however, innovative companies are supplementing this technology with purpose-built platforms designed to handle workloads beyond the functionality of most data warehouse infrastructures.

Watch Shawn Rogers in The Briefing Room with Noetix.

Big Data environments, analytic platforms and cloud-based solutions are all growing in popularity across the data management landscape. Each delivers a specialized value proposition to the enterprise by addressing the drivers just mentioned and by applying feature sets to meet the following requirements as shown in Figure 1.


Figure 1: Specialized Requirements

  • Response – The need for these platforms to respond at new speeds and scale has opened the door for original ways to leverage data and provide insights to end users. This is especially true in the area of Big Data analytics where end-user response rates are a key component to the value these platforms can deliver. Sub-second data delivery is not necessary for all applications and data-driven scenarios. But it is clear that real-time use cases are growing in importance and becoming more critical. New technology platforms such as Big Data Frameworks are at the core of this evolution and are powering new solutions and improved response. Innovators such as Yahoo and Google helped to pioneer this area and created technology foundations to meet the growing needs of response time within the enterprise. Cassandra, CouchDB and Mongo are being adopted into traditional data management ecosystems to address these new demands. These solutions are highly technical and not generally designed for the laymen, but they are creating new opportunities within the enterprise to leverage data at faster speeds and wider scale that were once thought to be prohibitive from both a technology and economic viewpoint.
  • Workload Complexity – Addressing the requirements of complexity within analytic environments is getting more difficult, while running highly complex analytic models over massive data stores is becoming more commonplace. Early adopter companies realized that Big Data platforms could play a role within their ecosystem to execute extremely complex analytic workloads and were willing to invest early in these solutions to gain competitive advantage.  When coupling the workload complexity to the response these new platforms, it creates a powerful tool of differentiation. The ability to introduce new data types such as social information or machine or process data could be leveraged to add even greater levels of insight and value.
  • Economics – The economy of technology is the great equalizer and often can attribute to an early majority adoption of the technology. This has been especially true with Big Data. Many companies have identified needs to address response and workload complexity, but the return on investment has slowed adoption. Big Data platforms are leveraging commodity hardware and, often the software is free, so it breaks through the economic barrier to adoption. Hadoop’s open source software cost model is changing the playing field for established and high license fee alternatives. Companies that plan to adopt Big Data should be warned that the barrier to entry is significantly reduced, but that doesn’t mean its cheap. Special skill set requirements and lack of mainstream management tools create hidden costs that need to be taken into account before adopting this type of technology.
  • Structure – Big Data frameworks provide a level of flexibility not present in traditional data platforms. These systems can load and store data without requiring the time investment of designing and building complex data models. Analytics can be executed on these platforms without models, while running at speeds that eclipse standard relational databases. Many users are employing “late binding” models to the data as they move it forward in the analytic process, enabling a smaller set of data to be manipulated and leveraged. Data structure flexibility is key to the foundation of Big Data utilization and adoption.
  • Load – Data loads are growing and the sources are more diverse. Driven by greater complexity and demand, Big Data adoption is based on the need to provide flexibility.  The power of Big Data platforms to load a mixture of data creates an opportunity to address both analytic and operational scenarios. Without this data to fuel these workloads, it would be impossible to execute against the growing demands of enterprise applications and analytic environments.

All five of these requirements play key roles in the adoption of new platforms.  Big Data adoption, specifically, has been driven forward along these themes and is now experiencing adoption at the  “early majority” level.  As more and more companies introduce this technology into their data management ecosystems, they will be faced with opportunity to innovate as well as challenges and hurdles to success. 

Big Data Adoption Curve


Figure 2: Adoption Curve

To move beyond the “early adopter” stage, Big Data platforms will need to surround themselves with features that make management and execution easier and available to a wider group of users. Presently Big Data is complex but it delivers greater response, more complex workload capabilities and can prove to be cheaper than traditional platforms. Big Data entered the industry with the standard level of marketing buzz and misunderstanding, but it is now maturing and moving forward into the Hybrid Data Ecosystem that many innovative companies are making part of their analytic and operational roadmaps.

Shawn Rogers

About Shawn Rogers

Shawn Rogers, Enterprise Management Associates Vice President, has more than 19 years of hands-on IT experience, with a focus on Internet-enabled technology. Shawn’s passions are rooted in the online world, focusing on analytics, social media, business intelligence, enterprise 2.0, SaaS and open source. He applies his nearly two decades of experience to every client engagement.

Shawn Rogers

About Shawn Rogers

Shawn Rogers, Enterprise Management Associates Vice President, has more than 19 years of hands-on IT experience, with a focus on Internet-enabled technology. Shawn’s passions are rooted in the online world, focusing on analytics, social media, business intelligence, enterprise 2.0, SaaS and open source. He applies his nearly two decades of experience to every client engagement.

Leave a Reply

Your email address will not be published. Required fields are marked *

You may use these HTML tags and attributes: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <strike> <strong>