There are four main challenges facing business intelligence (BI) systems today. The first is the cost. BI has evolved and everybody has some form of BI in place now, as it is becoming a fairly substantial cost item. The overall cost of BI – the cost of technology, upkeep and implementation – is certainly one of the challenges that implementers are facing.
Watch The Briefing Room with author Claudia Imhoff and Brad Peters of Birst.
The second challenge, believe it or not, is the number of users. The number of business users now tapping into BI is increasing dramatically, especially as we begin to move into operational intelligence. We’re seeing more naïve users – not the traditional analysts or data scientists – so it is not only the number of users but an increase in support for these users from an implementation standpoint.
The third challenge is in the area of operational BI and the new sources of data available. We are seeing a tremendous increase in the volumes of data (big data) being analyzed and stored in data warehouses and experimental areas. This data is used for complex advanced, embedded and streaming analytics. There are now very interesting sets of data in BI, which is certainly different from the traditional, more strategic or tactical forms of BI. This doesn’t diminish the need for traditional BI; it just means we must expand our BI architectures to embrace these new areas.
These big challenges lead to the fourth, which is the performance and scalability of the environment. Obviously, if we are starting to bring in operational people, operational BI, streaming analytics, big data applications, etc., it means that the performance has to be a major focus of the BI implementers – sub-second response time for many operational intelligence queries while simultaneously supporting the more strategic or long running queries as well. It’s a mixed workload environment, and that can cause a performance issue. So our technology also has to scale up to handle it. A terabyte used to seem like a lot of data, but not anymore.
An Architected Approach is Still Mandatory
While we are certainly in a very disruptive period – with all the entire groundbreaking and innovative technology available today, we cannot “throw the baby out with the bath water.” In the rush to bring in these exciting new technologies, we must remember that there were reasons for the success of the architected BI approaches used in the past. An architected BI approach is one in which there is a rational and logical architecture designed and used as our roadmap from project to project. It is an approach that builds upon a sound technological foundation. Because of the claims of some vendors, many implementers may believe that they no longer need an architected, disciplined approach to BI. The unarchitected BI approach is still chaotic, project-focused and results in a much less organized, consistent, reliable or reusable BI environment. Many times, it is individuals or even small subgroups in a department deciding that they want to build their own little environment, producing independent, siloed pockets of analytics, as opposed to a more architected approach where you can integrate and clean data, then share it, while building upon the foundation already in place.
One important thing about the architected approach is that it is a logical or conceptual architecture and how people physically implement it is up to them. It is a logical approach, and as such, how it is physically implemented varies greatly from company to company. It is a roadmap that directs an efficient and productive approach to BI by promoting reusability of data, for example. It also promotes the sharing of analytics, data and components. It promotes efficiency in the data integration and data quality processes. It’s a very well-documented approach to BI that has been around for more than a decade but even it is changing with today’s innovations.
Colin White and I have studied the new technologies and had many a discussion about how these are impacting the architecture for today’s BI environments. We concluded that the principles from the original architected approach are still solid. Generally the environment starts out consisting of a data warehouse and dependent data marts (that may be physical or virtual in nature) being sourced from operational systems. This environment is excellent for low-latency and historical reporting, multidimensional comparisons and many advanced analytics. Most traditional database management systems (DBMSs) and BI technologies work well in this environment too. However, it must be emphasized that the new sources of data, new forms of analytics and new technologies supporting these (Hadoop and MapReduce, analytic DBMSs, BI appliances, data compression, in-memory and in-database analytics, stream analytics – just to name a few!) have expanded the traditional architecture – not replaced it.
For example, we expand the data warehouse and data mart environment by adding an experimental or sandbox area where data scientists and other analysts can run heavy-duty analytics against big data – such as analytics against machine-generated, sensor and social media data as well as against every transactional record captured in the business. These sandboxes allow us to run very dynamic queries against a relatively static, low-latency set of data with incredible performance.
We further expand our BI environments to support operational intelligence – that is, analytics against real-time (not low-latency) data; analytics against data as it is streaming into our companies. Stream analytics runs fairly static queries against very dynamic data (think of fraud detection analyses running against every claim, credit card or other transaction coming into our organizations).
The architected approach can still fulfill its promises while meeting the ever-increasing needs of the business challenges of today. If we stick to what works while expanding our architectures, costs are reduced, support for the new and evolving business users is enhanced, operational intelligence is a reality and, of course, performance and scalability meet all the service levels required.