Inside Analysis

Getting Out of the Weeds: Self-Service Big Data Integration

One of the common complaints from information technology teams and IT departments as a whole is that they are too busy doing their jobs to consider new work or new projects. How many times has a business stakeholder heard the following from a technologist, “I can’t consider that now. However, based on our current backlog, it will take around 12-18 months to scope, code and get into production…”?

Unfortunately, that type of response is no longer acceptable. The current speed of business does not run in 12-18 month increments, but 12-18 week segments, if not faster. IT departments need to be more responsive to the business and to opportunities to gain advantage over the competition.

Listen to The Briefing Room, entitled “Big Data Refinery: Distilling Value for User-Driven Analytics,” featuring John Myers and Chuck Yarbrough of Pentaho.

Many times this response to requests from business stakeholders comes from too much work and not enough people. Sometimes it comes from ill-considered requests from business stakeholders who have not yet fully fleshed out their requests. However, this situation usually occurs because technologists rarely pull their heads up from the day-to-day tactical tasks that take up much of their time to consider how to address long-term strategic initiatives that could reduce or completely eliminate those day-to-day tactical tasks.

Job Security

In the early days of extract-transform-load (ETL) efforts, individual contributors and sometimes larger teams invested in PERL or other scripting efforts to meet the data integration needs of the organization. They became expert in scripting and understanding how PERL could be used to transform data. After a time, many of these technologists and teams became more involved in maintaining and supporting PERL environments for ETL rather than considering the use of more automated tools. Their thought seemed to be: “If I wasn’t maintaining PERL scripts, what would I be doing?”

This mindset focuses on a “tactical” job outlook as opposed to a “strategic” outlook that focuses on finding a way to do business more effectively. While coding in PERL can be a fair option for organizations with lots of time, limited options for automated tools and few data sources to support, there are very few organizations who have these “luxuries,” particularly in terms of time.

Like Water Finding the Cracks

When you hear “no” enough times, you will find a different approach. Without a willing and responsive partner, business stakeholders have taken these responses and situations as an opportunity to do two things:

  • Embrace cloud-based subscription services
  • Start “shadow” IT initiatives

The first, engaging cloud providers, is not necessarily a bad thing. Organizations can move their expenses from capital (CAPEX) layouts for infrastructure and licensing to operational (OPEX) layouts which provide a more flexible cost structure. Organizations can also dramatically speed up the provisioning of environments to support their requirements. However, IT departments lose their involvement in and “control” over those initiatives and often have no knowledge of what is being performed in the cloud and how that is being managed and governed.

The second, creating shadow IT support groups, can be beneficial. The technical aspects of data integration are no longer a black art to be kept from business stakeholders. Many business stakeholders have “grown up” in the age of Google and DIY technologies, and they are now able to handle their own data transformation, logic and design. In fact, many do, hence the rise and quasi-dominance of Microsoft Excel ( …and soon Google Docs… ) as a transformation- and analytics-like platform. When properly managed, these shadow IT skills to understand data, test transformation and prepare the transformation logic for an operational implementation can be an IT department’s best friend.

When “shadow IT” teams are allowed to operate without proper supervision, organizations can find themselves with a mashup of “spaghetti” spreadsheets and ill-formatted and calculated data sets that show up well in PowerPoint presentations at the department level, but are often confusing and contradictory when presented to executives who need consistent information to make decisions.

Carpe Diem

In instances like the one above, IT departments and technologists need to understand the value in being a strategic partner to the business. Replying to requests with a 12-18 month time frame is a way to say no as opposed to a way to engage with the business to secure opportunities to expand market share and capitalize on revenue opportunities.

Instead of focusing on tactical workload, IT departments can look for those opportunities to raise their profile and provide strategic value-add services. In the example of big data integration, organizations cannot put the genie back in the bottle regarding shadow IT skills and the cloud, but they can leverage those opportunities. IT departments should look for opportunities to provide not just the tools for business stakeholders to manage their own transformations, but to provide “blessed” data sources that business stakeholders can use without concern.

Providing the transformation tools empowers business stakeholders to utilize their growing knowledge of the data available to an organization and capitalizes on their interest to move projects forward. Engaging these business stakeholders to handle the definition of transformations and support that engagement for data integration allows IT organizations say “yes” to business and move into a strategic position.

Providing “blessed” data sources with defined lineage and making that information available to the transformation tools gets IT teams out of the business of tactical data provisioning at the ETL level and instead puts them in the business of adding additional and new data sources to the business. This is particularly important in the era of big data where new data sources are being discovered and utilized at an alarming rate – a rate that with the rise of the Internet of Things will only continue.

As technologists and IT organizations pull out of the weeds of data integration tasks, they will find themselves with additional time because their tactical role as “THE” transformer of big data will be greatly reduced. They will also find themselves in an elevated role where business stakeholders will not hide their activities in the cloud and with “shadow” IT teams, but ask for strategic help in implementing important initiatives that can improve revenues and capture market share.

John Myers

About John Myers

John Myers is Research Director of Business Intelligence at Enterprise Management Associates. In this role, John delivers comprehensive coverage of the business intelligence and data warehouse industry with a focus on database management, data integration, data visualization and process management solutions. John has years of experience working in areas related to business analytics in professional services consulting and product development roles, as well as helping organizations solve their business analytics problems, whether they relate to operational platforms, such as customer care or billing, or applied analytical applications, such as revenue assurance or fraud management.

John Myers

About John Myers

John Myers is Research Director of Business Intelligence at Enterprise Management Associates. In this role, John delivers comprehensive coverage of the business intelligence and data warehouse industry with a focus on database management, data integration, data visualization and process management solutions. John has years of experience working in areas related to business analytics in professional services consulting and product development roles, as well as helping organizations solve their business analytics problems, whether they relate to operational platforms, such as customer care or billing, or applied analytical applications, such as revenue assurance or fraud management.

Leave a Reply

Your email address will not be published. Required fields are marked *