This article is part of The Bloor Group’s research program, Philosophy of Data, which has been underwritten by IRI, the CoSort Company.
Businesses don’t grow by resting on their laurels. The savviest enterprises understand that building a foundation on best practices and smart systems leads to wins across the board. As new solutions and processes are added to the infrastructure, however, it can be easy to lose track of who’s doing what – a pitfall that can blur the enterprise vision and ultimately harm the business.
Dan Linstedt, of Empowered Holdings, said that solving business problems involves bridging the gap between the technology and the business, and it all starts with data. “In my particular case and with my customer’s data,” he said, “it’s absolutely useless or meaningless in most of its current states until or unless the business users can correlate it, assign value to it and, therefore, turn it into information.”
Of course, auditing and regulatory compliance does not care about the business value of data; they are only interested in how and when raw data was manipulated. For that, IT is necessary to ensure governance, metadata management, data flow management – essentially a solidly-built architecture. But IT cannot create this ivory tower of data for itself. Although the term “self-service BI” might be a misnomer, IT still must provide some type of access to business users.
“The whole flow is really all about correlating, aggregating, gathering all of this data from across usually 20 to 50 source systems or more and then putting it all together on the other side, which is the information side and delivering it to the business on-demand in a managed fashion,” said Linstedt.
That said, the data warehouse can not be a free-for-all repository that’s available to anyone with a login. With so many components and considerations for both business and technical users, it’s not uncommon to see a data warehouse lose steam. Whether it’s because of ignored processes, policies or practices, the advantages of a complex system can be lost when people aren’t on the same page.
As a veteran data warehousing expert, Linstedt developed the Data Vault, a database modeling and methodology solution designed to underpin the moving parts of an information architecture. A typical implementation begins with conversations around pain points and breaking down those issues into manageable chunks. The next step is to “divide and conquer,” where historical data storage and data warehousing are separated from the information delivery lifecycle.
With Data Vault, there is no wrong data, any data can be right at a given time, and all data is relevant. “There is no single version of the truth,” Linstedt said. “My truth is different than your truth is different than his truth. And, oh by the way, the minute I learn something, my truth changes.”
It’s this philosophy that drives the Data Vault. With automated pattern-based data integration and data virtualization processes, along with the separation of data and information, the Data Vault can keep data warehousing focused on gathering the right data and turning it into information for the business user.
Today’s lure from Big Data and Hadoop have caused a slight ripple in the data warehousing world. Many organizations have eyed the open source framework as an end-all-be-all solution. But Hadoop is far from a data warehouse replacement.
When using Hadoop alone, many critical components of a well-managed system are lost: lineage, governance, metadata, data definitions, etc. Linstedt said that while there might be appropriate use cases for using Hadoop simply as a storage mechanism, it doesn’t really augment the business in any way. “In order to apply any sort of business information on this data, you have to begin to stratify it, you have to begin to profile it, you have to begin to manage it and understand it and apply structure to it, so that you can get results from it,” he said.
Tools like Datameer, Domo and Alteryx can deliver various aspects of what’s missing from Hadoop. But again, it’s all too easy for these tools to fall into siloed hands, working away on one person’s desktop rather than driving value across the enterprise. The key is finding and using a solution that focuses heavily on governance and standards so as not to lose sight of the enterprise vision of data.
“Having the right process at the business level, having the right tooling at the technology level, and, of course, training goes a long ways to making all of this work,” said Linstedt, “but you do need a methodology that can help you govern all of this from soup to nuts, from end to end.”