The term agile gets bandied about quite a bit, almost to the point that its meaning is a bit lost. Of course, no software vendor will advertise its product as offering brittle, clunky and sluggish performance, but that doesn’t mean it’s necessarily agile.
Techopedia defines agile software development as “a lightweight software engineering framework that promotes iterative development throughout the life cycle of the project, close collaboration between the development team and business side, constant communication, and tightly knit teams.” But what are these agile technologies?
According to the recently published Modern Data Warehouse (MDW) Research Report, they include Hadoop, cloud-based or hosted solutions, SQL-alternative databases and data warehouse automation. While it’s true that legacy RDBMSs still reign supreme, the concept of a hybrid architecture – a mix of traditional investments and agile technologies – is gaining serious traction.
As was discovered in the MDW survey, many organizations (18%) are interested in or are already using agile technologies to augment their environment. That percentage will undoubtedly grow as environments become more complex, though the concept of data warehouse modernization isn’t about replacing the data warehouse with new technology. It’s about building a comprehensive solution.
How to Stay Agile
There has always been a power imbalance when it comes to data. For years, the only way business could get to enterprise data was to ask IT. But nowadays it’s all too easy for business users to go rogue and circumvent the heavily guarded and governed data, which can cause a number of unfortunate circumstances.
No single solution can modernize the data warehouse, but instead, leveraging a combination of these technologies to revitalize the aging infrastructure will become the de facto method.
Mark Budzinski, President of WhereScape USA, described the balance of power as an IT/business pendulum, where IT held control over data for so long, then business took matters into its own hands with external solutions. “I think we’re finding an equilibrium now when it comes to that balance of power – the methodology of agile and the technologies, whether they be NoSQL, Hadoop or data warehouse automation, that bridge the gap between those two,” he said.
It obviously will take a collection of solutions to realize a modern data warehouse, but Budzinski said that at the end of the day, “If you’re serious about agile, if you’re serious about collaboration, if you’re serious about iteration, I don’t care if it’s going to be in the cloud or on-prem, if you’re a big company or a small company, that theme is best realized through automation.”
Automation for the People
If something can be automated, it probably should be automated, and the building and maintenance of a data warehouse is no exception. No business has time for a 12 to 18 month development cycle any more, and the warehouse is no longer the static creature it used to be. Big data, streaming data, the Internet of Things – they have all strained data warehouse design and maintenance.
Building a data warehouse is a big deal, and without proper care and feeding, it’s all too easy to let it fall into chaos. Documenting metadata, connecting to new data sources, developing data models and making sure the warehouse is meeting business requirements: these are all critical components of a well-oiled machine. And they are all capable of being automated.
The MDW Report showed that 24% of very experienced organizations (those with seven or more years of having a data warehouse) are already using a data warehouse automation tool for their primary platform. While less experienced organizations report lower adoption rates, clearly the ones who have traversed the trials and pains of a data warehouse implementation understand how automation can create value.
One company that delivers automation is WhereScape. Its flagship product, WhereScape RED, offers an end-to-end data warehouse automation solution that dramatically decreases development and implementation time and provides an extensive maintenance framework.
WhereScape RED is a choose-your-own architecture product. It can work with and integrate one or a combination of data warehouses, data marts, data lakes or any other data stores. It supports every common database, and as an ETL engine, it can extract data from virtually any source.
Using business rules and logic, it can automate table creation and population. It can also automatically merge or update tables, detect changes and process data. All automated events are written to the metadata as well, eliminating the pesky and often ignored process of manual documentation.
As a scheduling and workflow engine, WhereScape RED automates jobs and streamlines decision management. It offers a highly intuitive environment for business users and IT, allowing both to iterate during the development phase and collaborate after deployment.
Some naysayers of automation are wary of giving up control, concerned that going on autopilot will result in reduced discernibility. WhereScape RED provides complete visibility into metadata, data origination, data lineage, hierarchies, business definitions and so on. If something requires human intervention, it will let you know.
Emerging solutions like automation don’t threaten traditional systems – they augment them. Agile technologies are catching on, and companies that expect to grow the business and compete in the information economy will learn, if they haven’t already, that old approaches can’t solve new problems. Data warehouse automation is one of many ways to modernize the data warehouse, and as Budzinski said, “Automation is really the way out of that sort of chaos.”