There is an old story about Yogi Berra and a teammate driving from New York to spring training in Florida. Along the way, the teammate asked if they were lost, Yogi responded with the following retort: “Yes, but we are making great time…”
Oftentimes in the world of business, the quest for real-time processing and response leads to a situation similar to Yogi’s. Yes, you can achieve real-time results, but do you know if those results provide the information that your data consumers require? Furthermore, are the proper operational “positions” receiving the information required to be effective?
Operational systems, in the past, have been able to perform policy- or rule-driven real-time results. However, older processing models and lack of context data, such as customer or product information, can limit the value of the real-time results. Due to the limited scope of information available to that kind of operational silo, any real-time results would be constrained to a particular operational process such as manufacturing, quality control or an ERP platform.
Alternatively, analytical platforms, such as enterprise data warehouses or data marts, have the contextual information and often the ability to process in real time. Yet, those analytical platforms are isolated from both operational processes associated with business transactions, such as point of sale systems, and operational decisions regarding concepts such as manufacturing quality.
In both of these situations, businesses might be “making great time,” but their real-time results value is limited or lost. The fact is the requirement for faster processing is not going away. Organizations will continually push for faster response times; it is one of the core competitive advantages in a data-driven world.
Real Big Data Use Cases
In late 2013, Enterprise Management Associates released the results of a second annual end-user survey of big data decision makers. This worldwide survey of 259 respondents asked a series of non-technical implementation questions to determine the big data challenges and requirements of organizations.
The EMA study asked respondents about which use cases were most important to them. The top five responses for these big data use cases were:
- Speed of processing response
- Combining data by structure
- Pre-processing data
- Utilizing streaming data
- Staging structured data
The response, speed of processing, displayed the largest jump from the 2012 EMA big data study and was the overall top response for Big data cases in 2013. This shows that implementers of big data solutions are focusing on not just how to collect data, but also ways to use that information for competitive advantage in a quickly evolving, technical environment.
Operations and Analytics: Big Data Bridges Two Disparate Groups
Whether the sources are business transactions, operational log information or a combination of both, big data has its roots in machine-to-machine (m2m) derived data. Many sources of big data are operational platforms. The owners of these operational platforms have defined processes and procedures that can be improved with information from their systems. When you link the importance of the speed of response use case with these operational data sources, new types of processing or workloads emerge. Big data is not just for data warehouse replacement or analytical exploration. Instead, big data promises to provide as much impact to operational use cases as it does to analytical ones.
Wide Variety of Big Data Projects
The EMA study asked respondents to name the reasons for initiating big data projects. The top five reasons for individual projects were:
- Risk management
- Ad hoc operational queries
- Asset optimization
- Operational event and policy processing
- Campaign optimization
Three of the top five answers were focused on the type of real-time analytical workloads that are often directly integrated into operational processes.
For example, risk management focuses on the ability to check fraudulent activity in the sale of goods and services. When these goods and services are intangible, it becomes all the more important to include real-time fraud management checks into an operational process. Since intangible goods and services such as software, apps and certain formats of movies and music are not recoverable, the risk exposure at the point of sale is 100%.
In another area, asset optimization focuses on the ability to appropriately allocate resources for maximum value. In the past when corrections to allocations were not practical, one could use existing optimization models for transportation logistics or staffing allocations for labor forces. However, the ability to use new optimization allocations has increased with the expanded use of compartmentalization in logistics (e.g., the ability to have modular shipments) and flex scheduling with temporary labor pools and on-call staffing groups.
Operational analytics (process-driven analytics with real-time requirements) is another type of project mentioned by the survey respondents. Figure 1 shows the inclusion of these expanded workloads in the EMA Hybrid Data Ecosystem.
For more information on where EMA sees the world of big data going and how end users are meeting the challenges of big data with the project implementations, see the results of the 2013 EMA Big Data research study.