Go to Top

Do We Still Need a BI Architecture? You Bet We Do!

There are four main challenges facing business intelligence (BI) systems today. The first is the cost. BI has evolved and everybody has some form of BI in place now, as it is becoming a fairly substantial cost item. The overall cost of BI – the cost of technology, upkeep and implementation – is certainly one of the challenges that implementers are facing.

Watch The Briefing Room with author Claudia Imhoff and Brad Peters of Birst.

The second challenge, believe it or not, is the number of users. The number of business users now tapping into BI is increasing dramatically, especially as we begin to move into operational intelligence. We’re seeing more naïve users – not the traditional analysts or data scientists – so it is not only the number of users but an increase in support for these users from an implementation standpoint.

The third challenge is in the area of operational BI and the new sources of data available. We are seeing a tremendous increase in the volumes of data (big data) being analyzed and stored in data warehouses and experimental areas. This data is used for complex advanced, embedded and streaming analytics. There are now very interesting sets of data in BI, which is certainly different from the traditional, more strategic or tactical forms of BI. This doesn’t diminish the need for traditional BI; it just means we must expand our BI architectures to embrace these new areas.

These big challenges lead to the fourth, which is the performance and scalability of the environment. Obviously, if we are starting to bring in operational people, operational BI, streaming analytics, big data applications, etc., it means that the performance has to be a major focus of the BI implementers – sub-second response time for many operational intelligence queries while simultaneously supporting the more strategic or long running queries as well. It’s a mixed workload environment, and that can cause a performance issue. So our technology also has to scale up to handle it. A terabyte used to seem like a lot of data, but not anymore.

An Architected Approach is Still Mandatory

While we are certainly in a very disruptive period – with all the entire groundbreaking and innovative technology available today, we cannot “throw the baby out with the bath water.”  In the rush to bring in these exciting new technologies, we must remember that there were reasons for the success of the architected BI approaches used in the past. An architected BI approach is one in which there is a rational and logical architecture designed and used as our roadmap from project to project.  It is an approach that builds upon a sound technological foundation. Because of the claims of some vendors, many implementers may believe that they no longer need an architected, disciplined approach to BI. The unarchitected BI approach is still chaotic, project-focused and results in a much less organized, consistent, reliable or reusable BI environment. Many times, it is individuals or even small subgroups in a department deciding that they want to build their own little environment, producing independent, siloed pockets of analytics, as opposed to a more architected approach where you can integrate and clean data, then share it, while building upon the foundation already in place.

One important thing about the architected approach is that it is a logical or conceptual architecture and how people physically implement it is up to them. It is a logical approach, and as such, how it is physically implemented varies greatly from company to company. It is a roadmap that directs an efficient and productive approach to BI by promoting reusability of data, for example. It also promotes the sharing of analytics, data and components. It promotes efficiency in the data integration and data quality processes. It’s a very well-documented approach to BI that has been around for more than a decade but even it is changing with today’s innovations.

Colin White and I have studied the new technologies and had many a discussion about how these are impacting the architecture for today’s BI environments. We concluded that the principles from the original architected approach are still solid. Generally the environment starts out consisting of a data warehouse and dependent data marts (that may be physical or virtual in nature) being sourced from operational systems. This environment is excellent for low-latency and historical reporting, multidimensional comparisons and many advanced analytics. Most traditional database management systems (DBMSs) and BI technologies work well in this environment too.  However, it must be emphasized that the new sources of data, new forms of analytics and new technologies supporting these (Hadoop and MapReduce, analytic DBMSs, BI appliances, data compression, in-memory and in-database analytics, stream analytics – just to name a few!) have expanded the traditional architecture – not replaced it.

For example, we expand the data warehouse and data mart environment by adding an experimental or sandbox area where data scientists and other analysts can run heavy-duty analytics against big data – such as analytics against machine-generated, sensor and social media data as well as against every transactional record captured in the business.  These sandboxes allow us to run very dynamic queries against a relatively static, low-latency set of data with incredible performance.

We further expand our BI environments to support operational intelligence – that is, analytics against real-time (not low-latency) data; analytics against data as it is streaming into our companies. Stream analytics runs fairly static queries against very dynamic data (think of fraud detection analyses running against every claim, credit card or other transaction coming into our organizations).

The architected approach can still fulfill its promises while meeting the ever-increasing needs of the business challenges of today. If we stick to what works while expanding our architectures, costs are reduced, support for the new and evolving business users is enhanced, operational intelligence is a reality and, of course, performance and scalability meet all the service levels required.

Claudia Imhoff

About Claudia Imhoff

Claudia Imhoff is President of Intelligent Solutions, Inc. and Founder of Boulder BI Brain Trust. As a thought leader, visionary and practitioner in the rapidly growing field of business intelligence, Claudia is an internationally recognized expert on analytics, business intelligence and the infrastructure to support these initiatives. Dr. Imhoff has co-authored five highly regarded and popular books on these subjects and writes articles (totaling more than 100) for technical and business magazines. The Boulder BI Brain Trust is a consortium of independent analysts and consultants dedicated to furthering business intelligence.

Claudia Imhoff

About Claudia Imhoff

Claudia Imhoff is President of Intelligent Solutions, Inc. and Founder of Boulder BI Brain Trust. As a thought leader, visionary and practitioner in the rapidly growing field of business intelligence, Claudia is an internationally recognized expert on analytics, business intelligence and the infrastructure to support these initiatives. Dr. Imhoff has co-authored five highly regarded and popular books on these subjects and writes articles (totaling more than 100) for technical and business magazines. The Boulder BI Brain Trust is a consortium of independent analysts and consultants dedicated to furthering business intelligence.

10 Responses to "Do We Still Need a BI Architecture? You Bet We Do!"

  • Kirk Anderson
    April 15, 2013 - 4:40 pm Reply

    Well said, please keep me informed.

  • Barry Devlin
    April 15, 2013 - 4:57 pm Reply

    Excellent article, Claudia! A cogent argument for an architected, properly planned approach to BI taking the best of the old and the new… I couldn’t have put it better myself.

  • Claudia Imhoff
    Claudia Imhoff
    April 15, 2013 - 5:31 pm Reply

    Thanks for the comments. It is a turbulent time, no doubt, so maybe I can shed a bit of light on people’s BI pathways…

  • Jake Freivald
    April 15, 2013 - 5:55 pm Reply

    I like it. I used to work with a consultant who said, “You do your architecture, or it gets done to you.” That hasn’t changed with the advent of big data, real-time streaming, or large user populations. If anything, it has become more important to recognize.

  • Ganesh
    April 15, 2013 - 4:54 pm Reply

    The reality is that most operational reporting environments do not use the traditional data warehouse. Nor, is the traditional data warehouse intended for operational reporting/analytics. In-memory and other such technologies should ideally serve the purpose as a layer on top of source systems instead of using the data warehouse. It is a over-kill to try to create another layer of reporting structures on top of source systems.

  • Ramesh Babu
    April 15, 2013 - 5:32 pm Reply

    Couldn’t agree. An excellent article at the right time. Thanks

    • Ramesh Babu
      April 15, 2013 - 5:34 pm Reply

      Sorry for the typo – missed out the word ‘more.’

  • Karien Verhagen
    April 16, 2013 - 6:46 am Reply

    Reading your article I plee for the statement : A BI Architecture is more necessary than ever. If nothing else please embed your BI initiatives in a BI architecture. Would you agree and go as far as that ?

  • Dominic Sagar
    April 16, 2013 - 8:13 am Reply

    Very good article. I am always amazed as I work with clients how often their BI environments are just thrown together from a logical and physical point of view. It is hard to persuade them to be systematic and consistent.

  • Claudia Imhoff
    Claudia Imhoff
    April 16, 2013 - 9:43 am Reply

    I too am always amazed at the mishmash of BI components implemented by many organizations. And yes, Karien, it would be ideal if all BI components could be part of the EXPANDED BI architecture. This means that streaming analytics, content analytics, and experimental sandboxes are all part of a broader BI architecture – with appropriate data flows between the components.

    Claudia

Leave a Reply

Your email address will not be published. Required fields are marked *

You may use these HTML tags and attributes: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <strike> <strong>