Inside Analysis

Redefining Hot and Cold Data

The Teradata Intelligent Memory™ release is significant for data warehouse (DW) and business intelligence (BI) architects for several reasons: mainly it redefines the old “data temperature” paradigm to meet today’s BI and analytics workloads while still following established Information Lifecycle Management (ILM) principles. Yet, while the Teradata Intelligent Memory™ approach removes the primary barriers that have inhibited the adoption of ILM strategies in data warehouses, it raises a few new questions for architects to tackle moving forward.

Tune in to hear John O’Brien in The Briefing Room with Alan Greenspan of Teradata.

Data temperature, with its well-known “hot-cold” designations, has long followed a paradigm in which current data is hotter and more valuable to the business: as the data ages, it slowly loses value and is migrated to lower-cost, slower-performing storage tiers. The ILM principle is that persisting data should strive to match its value with the corresponding cost of the storage media in order to maintain an optimized cost-benefit ratio. However, when designing and configuring multitiered storage architectures, BI architects have had to rely on business analysts to define the data management policies for when to age and migrate data. This became cumbersome as different records had different policies and some data was based on status: business analysts had to first define what was “hot” data and then ask the business when slightly less access should occur. Then there was “cold” data: how long it should be retained online and accessible before being moved to offline archives. Moreover, there were questions of when the data should be marked as read-only and removed from database backups or as to the level of data detail stored – some data warehouses attempted to manage detail data aging quicker than the same summarized data, rationalizing that detail data was less valuable as it got older.

Data warehouses that could accomplish the feat of defining and managing data aging policies then had to script, code, automate, schedule, verify and monitor that the entire process was working correctly in production. Most of the time, data architects would take a simplified approach of identifying data classes (or use fewer storage tiers) to make this process achievable. Even with all the gains to be achieved with optimized, multitiered storage architectures, concern still lingered as to whether these gains would be lost to the cost of human development and management of such a system.

The Teradata Intelligent Memory™ provides a more “intelligent” approach to this problem based on monitoring system IOPs to learn what data users are accessing the most and persisting that data in the highest performing storage medium. This updates the old paradigm of data temperature, arguing that the data being accessed the most is the most valuable right now (or, “hot”) and therefore should reside in comparable storage medium, such as in-memory. When a data set is no longer being accessed (or, “cold”), it is migrated through its appropriate storage tiers. There is no longer a requirement to attempt to define the data aging policy beforehand for the unknown usage patterns of the enterprise.

Another advantage that Teradata’s Intelligent Memory™ technology has over ILM implementations – besides its “intelligence” to monitor system IOPS and employ secret algorithms to decide when and where data should optimally reside – is that Teradata Intelligent Memory™ has the ability to learn this behavior for any configuration that any company has chosen.

Yet, like any new technology, Teradata Intelligent Memory™ will not be adopted without addressing its own set of questions one of the first of which is likely to center on how to architect storage tiers if Teradata’s solution will optimize configuration after the fact. Our opinion is that while infrastructure cost is likely to become the deciding factor, some analysis and monitoring will allow most architects to get “in the ballpark.” And, with price-performance in storage continuously improving, we should expect to leverage this and modify configurations over time, too.

Another interesting aspect with an intelligent learning system will be the addressing situation where the user already knows that a given data set will be hot from the time that it’s initially loaded, or that a data set will become hot at a specific date and time. To this concern Teradata has already discussed the ability to have data being loaded placed in hot status and then allowed to cool off on its own: early users will likely not have to suffer through average performance while Teradata Intelligent Memory™ learns the usage profiles and keeps the data in-memory.

Radiant Advisors feels that, while it continues to embrace established ILM principles, this new paradigm toward data temperature by Teradata Intelligent Memory™ is more appropriate for today’s analytic and BI workloads. Though some may consider this to be merely a transitional technology as we move toward an increasingly all in-memory future, we believe this approach will be the bridge for that change and will intelligently adapt to the storage media ratios every step of the way. We look forward to talking with early customers of Teradata’s Intelligent Memory™ implementations to collect and analyze these benchmarks. In the meantime, it’s encouraging to see Teradata Labs working with existing customers to understand their challenges and create new technology that challenges old paradigms in data warehousing.

This content originally published as Radiant Response: Teradata Sets a New Paradigm with Teradata Intelligent Memory™ on May 10, 2013.

John O'Brien

About John O'Brien

John O’Brien, CBIP is principal and CEO of Radiant Advisors, a strategic advisory and research firm that delivers innovative educational materials, publications, and industry news.

John O'Brien

About John O'Brien

John O’Brien, CBIP is principal and CEO of Radiant Advisors, a strategic advisory and research firm that delivers innovative educational materials, publications, and industry news.

Leave a Reply

Your email address will not be published. Required fields are marked *