Inside Analysis

How Do You Build Data Pipelines that Are Agile, Automated, and Accurate?

Hosted by Eric Kavanagh (The Bloor Group).
Sponsored by Syncsort.

Today’s data-hungry analytics applications falter when they can’t get the right data at the right speed. Data pipelines are a critical element of success with analytics, yet building and operating data pipelines can be difficult. Today’s data management systems are complex, with data deployed across multiple cloud platforms and in hybrid architectures that combine multi-cloud with on-premises databases. Data pipeline difficulty is further challenged by a shortage of data engineers and increasing needs for real time data.

Analytics success and sustainability depends to a large degree on our ability to build data pipelines that are agile, automated, and accurate. Agility is essential to keep pace with requirements for new data pipelines and to respond to changing business needs, data sources, and technologies. Automation is needed because the shortage of data engineers makes hand- crafted data pipelines impractical. Accuracy is the foundation of trusted data and analysis. Without trust, analytics fails to deliver business value. Together the 3 A’s of data pipelines—agility, automation, and accuracy—help to overcome the complexities and challenges of data pipeline development and operation.

You Will Learn:
– How multi-cloud and hybrid deployments add complexity to data pipelines
– Why data streams and real time data are data pipeline challenges
– Why hiring more data engineers is not the solution
– Nine components of data pipelines and the challenges inherent in each
– How data pipeline success improves the frequency of analytics success
– Why agile, automated, and accurate are data pipeline imperatives
– The role of technology in meeting the data pipeline imperatives


Leave a Reply

Your email address will not be published. Required fields are marked *