A couple years back, the first moving assembly line for mass production celebrated its 100th birthday. Churning out Model Ts at a record pace, it forever changed the face of the manufacturing industry. But let’s consider the evolution of the automobile.
The earliest Model Ts required 84 distinct assembly steps, and one worker was trained to manually carry out one step. This process took 12 hours. Driven by the need to increase productivity and efficiency, Henry Ford built machines to stamp out parts and tirelessly worked to make the process more effective. After years of incremental improvements, he unveiled his now-famous assembly line, reducing the build time from 12 hours to two and a half.
His innovation ushered in the era of the automobile, and today, thanks to the age of industrialization and the machinery that followed, some factories can produce 100 cars an hour. For a century, we’ve seen improvements upon the early models: safety, performance, fuel efficiency, design, etc.
Although manual transmissions are still around, automatic gear shifting greatly reduced the amount of effort required to drive. We have gauges and sensors that inform the driver of system problems and potential hazards. Object-detection camera systems have made the rear view mirror almost redundant. And now, between Tesla and Google, it looks like driverless technology is coming down the pike.
But no one other than a vintage car enthusiast drives a Model T anymore, for good reason. By today’s standards, they are difficult to drive, costly to maintain without special skills, too slow to safely drive on the highway and unreliable in general. They are short on speed and performance.
In fact, the Model T was retired long ago, not long after the ten millionth Model T was produced. Other manufacturers and Ford himself continued to innovate, and consumers demanded more features and options. Of course we didn’t stop making cars because the Model T became outdated; we simply started making better cars.
It’s the same with data warehousing.
Data warehousing fuels the entire industry of business intelligence and analytics. Designed to hold the enterprise golden record of data, it serves as the backbone for enterprise systems and applications. But building a data warehouse has traditionally not been easy.
A typical deployment can take six, nine, even 12 months to complete. Adding new data sources and users is a highly manual process, and there is often a disconnect between what the business wants and what IT can give. ETL processes are cumbersome, and with the exploding number and variety of data sources, from within and outside an organization, data warehouse maintenance becomes increasingly challenging.
There have been claims that the data warehouse is dying, crippled by the overwhelming burden of big data and challenged by the data lakes of Hadoop. Is the data warehouse doomed? Most certainly not. But its traditional waterfall development cycle is no longer acceptable. Business does not have months to wait before insights emerge.
The data warehouse must evolve with the inevitable shifts in the information landscape. Without a method of unifying the myriad components that now comprise a data warehouse system, it will get dragged down by data gravity and stuck in quicksand. It cannot afford to remain a Model T.
The Age of Automation
Data warehouse automation is a fairly nascent technology. Yes, several vendors have been doing it for a while, but organizations have been slow to catch on. Maybe it’s because they simply don’t comprehend the improvements it can deliver. Maybe they’re afraid to give up control of the manual processes. Maybe they think it sounds too good to be true. But the fact is, automation is the future.
One company, TimeXtender, takes a decidedly distinct approach to data warehouse automation. Built exclusively on Microsoft SQLServer, it exposes the metadata layer as the primary model, allowing business users to define and identify to IT exactly which data they want for their applications. Once those decisions have been made, IT can turn around and model the metadata layer. No ETL needs to be written, and only minimal coding is required.
The type of automation TimeXtender delivers solves a number of problems. First, it eliminates the need for documentation, a process that, when actually done correctly, eats up a ton of time. What’s worse, if Joe the DBA has not been documenting his code and leaves the company, it could take months to recover his knowledge. TimeXtender automatically generates and documents ETL and processes, thereby taking the pain out of documentation and code retrieval.
Second, it simplifies modeling. Whether data sources and applications are on premises, in the cloud or somewhere in between, modeling at the metadata layer eliminates the complexity from the integration. TimeXtender leverages what it calls intelligent adapters to connect to common CRM and ERP systems. These adapters are pre-loaded with knowledge of the source systems’ structure, which enables a simpler connection process.
Perhaps most importantly, it means the business user can understand the data. At the end of the day, data analytics drives the enterprise. If the data warehouse is outdated or too difficult to access, it becomes obsolete. When that happens, the business user will take matters into his own hands, bypassing the data warehouse altogether and likely creating reports based on inaccurate and certainly incomplete data sets. With TimeXtender’s automation capabilities, IT can better maintain and model the data warehouse, keeping it as the most reliable and trusted source for analytics.
Automation will not solve 100% of data warehousing issues. Ultimately, there will be a need for human intervention or manual optimization. With that in mind, TimeXtender does not offer a black box solution. Users can peer into every single code index and, if needed, make changes. However, automating all the heavy lifting turns manual processes into more of a light maintenance situation, meaning more time can be spent on higher value tasks.
Data has such a short lifecycle these days, and data sources change all the time. The data warehouse processes of yore – ETL, modeling, building connectors, etc. – are failing to keep up. By leveraging automation over all these processes, TimeXtender delivers the time to value that drives so many organizations today.
Business needs to answer questions fast, and automating the tedious parts of data warehousing will foster the type of agility to turn IT into a profit center. The ability to respond quickly to change, i.e., new data sources, new applications, new users, can mean the difference between an organizational win and a missed business opportunity. Automation can make the win happen.