Imagine a business intelligence (BI) and analytics platform comparable in its way to Snowflake, the platform-as-a-service (PaaS) data warehousing specialist. A PaaS designed with the pluses and minuses of the cloud in mind: a cloud-native BI and analytics platform. Wouldn’t it be nice?
True, there’s no shortage of cloud-native BI and analytics offerings; however, few of these services specifically addresses the embedded market. Start-up provider Qrvey is one that does.
And wouldn’t an embedded cloud-native BI-analytics PaaS be especially nice?
“We sort of sit at this intersection of BI and analytics, so the ability to build … dashboards, reports, data visualizations, [and] work with data, but we’re doing so in an embedded way,” explains Qrvey CTO David Abramson. “So we work with companies who want to embed the technology, those capabilities, the software, directly into their existing business processes or even into their own applications.”
Almost from the beginning … there was embedded BI …
This invites the question: what do we mean by “embedded” BI and analytics?
We mean features and functions that are not exposed via dedicated BI and analytic front-end tools but which are, rather, “embedded” in operational applications and services – that is, the software that powers core business and IT processes. In the background, under the covers, BI and analytic functions are embedded in the workflows that both animate and knit together these processes.
Take, for example, the data quality routines that validate the names, addresses, and telephone numbers of new customers. Or the sales workflow that determines whether or not a supplier decides to offer incentives to a customer – and, if so, which ones. In the background, this workflow traverses a network of internal and third-party systems – for example, calling a third-party API to perform a credit check and then calling the BI platform to make a decision based on the results. In practice, the workflow might have to accommodate one or more exceptions, too: for example, what happens if the product is out of stock? The workflow could trigger analytic processing to identify a next-best option; to decide whether or not to offer the customer a discount; or to identify some other incentive. None of these is explicitly a “BI” or “analytic” scenario; each involves BI and analytics functionality, however.
One distinctive aspect of embedded BI is that customers are usually not aware that they’re interacting with BI and analytics. For the most part, the embedded software runs in the background and provides certain functions; so far as the line-of-business user is concerned, she is interacting with (e.g.) an HR, finance, sales, etc. application. The BI-analytic functions are a non-obvious part of the user experience.
Back in the day, embedded BI thrived. But the old-school embedded BI model was neither designed nor suitable for the cloud. For example, adopting an old-school embedded BI product meant adopting a vendor-specific software-development kit, which sometimes meant using a proprietary software framework. In addition, it meant adopting supported middleware technologies, such as specific web or application servers, and, not least, using a supported app-dev environment. Just as important, it meant maintaining these assets: managing version dependencies or deprecations, patching security issues.
In the cloud, by contrast, the conditions for success with embedded BI have never been better.
“In sort of the traditional [embedded BI] systems, too, it wasn’t just about finding the right software tools and components, but also figuring out how to scale them the right way, how to make sure that it’s going to handle the types of load, whether it be users or data, how to secure them the right way and manage all of the interconnected workings of everything that’s going on,” Abramson told analyst Eric Kavanagh during a recent Inside Analysis podcast. In the cloud, he countered, “all of that is sort of taken care of for you, in the sense [that] all of these different services auto-scale, [so it is] very easy to work with, very easy to grow and sort of manage, even in microservices and … serverless architectural models. All of the security is … comprehensive and [it is] easy to integrate and manage and all of that.”
Exploiting available cloud services enables Qrvey to focus on core BI and analytics
Qrvey hosts its PaaS service in the context of Amazon Web Services (AWS); in this way, Qrvey exploit’s AWS services to provide ETL, SQL query, ML processing, function-as-a-service (FaaS, or “serverless”) computing, and other integral components of its PaaS BI and analytics offering.
This makes sense. AWS is a modular stack. It is accretive in the sense that many AWS services lead to other AWS services. So, for example, use of Redshift, AWS’ massively parallel processing database service, leads to use of other AWS services, including S3, AWS’ object storage service, or Athena, AWS’ SQL-query-as-a-service offering. Similarly, AWS EMR (compute-as-a-service); AWS Glue (ETL-as-a-service); AWS QuickSight (BI-as-a-service); and AWS SageMaker (ML-as-a-service) each address a set of specific, but also complementary, use cases. So, not only does each AWS service provide a set of potentially useful functions, but each is modular in the sense that it complements one or more other AWS services. It is modular in the special sense that it is theoretically replaceable, too. The set of functions that each of Amazon’s AWS services encapsulates is generalizable in the sense that Microsoft, with its Azure platform, or Google, with its Google Cloud platform – or Alibaba, Oracle, IBM, and others – have usually developed and implemented similar or analogous services.
“It’s almost like building products with new parts that have come out… now you have these new parts that are being offered by the manufacturer if you will, namely AWS, and you can piece those together to create the functionality that you want,” observed Kavanagh, who hosts the Inside Analysis podcast.
Abramson agreed. “It really dawned on us that architecture was kind of the key, and by ‘architecture,’ we really … focused on this concept of cloud-native architecture, and by cloud native it means really just taking advantage of all of the breadth of functionality and capabilities that exists in the cloud, in our case in the AWS ecosystem,” he said. “We’re not looking to reinvent the wheel when it comes to … these very rich, powerful services. If we can find them already available in the cloud and incorporate them to give our users and our customers value, we’ll go ahead and take advantage of that.”
Cloud services are best-in-class services
This confers another advantage, too. As Abramson said, most “generic” cloud services are anything but generic: instead, Amazon, Google, and Microsoft develop them as best-in-class as-a-service products.
For the (large) teams that support and maintain them, enhancing these ETL-, BI-, and ML-, etc.-as-a-service products is a core competency. The upshot is that a “generic” ETL or SQL query product hosted in AWS, Azure, or GCP exposes one or more rich access interfaces, provides a comprehensive set of functions, and (for these and other reasons) should prove suitable for a wide range of potential use cases, analytic and otherwise. Exploiting available services to support generic use cases or to provide a rich set of generalizable functions is good for PaaS vendors and their customers. As a PaaS “David” that has neither the financial nor the human capital of Amazon and other “Goliaths,” Qrvey cannot afford to spread its finite resources too thinly. Its architects and developers most focus their available time, effort, and talents on improving the features and functions of the core Qrvey platform.
For the same reasons, Qrvey does not have the resources to service a large amount of technical debt; rather, it must take on as little technical debt as possible. In this respect, Qrvey’s modular architecture gives it a sustainable foundation for maintaining, enhancing, and scaling its services, as well as for introducing new core BI services, exploiting new generic AWS services, and otherwise accommodating different kinds of economic, political, or technological change. As Kavanagh noted, this architecture also makes it possible for Qrvey to expand its stack to other platforms, such as Azure or GCP.
“What’s cool about your architecture … is that everything is already in the cloud and you’re just basically snapping a little … analytical widget into the workflow and, bam, you’ve got your analytics,” he pointed out, referring to Qrvey’s use of AWS services to provide ETL, SQL query, and other functions.
Cloud-native design aims to produce resilient, change-friendly software architecture
“The traditional tools … are definitely not optimized for the embedded [cloud] use case,” Abramson argued. “Oftentimes … they’re running on the old sort of client-server-based model which doesn’t necessarily even work in a cloud scenario.” This is slightly misleading. A decade ago, BI and analytics vendors basically forklifted their on-premises systems, middleware, applications, suites, etc. to run in the cloud context. (The early versions of Tableau Cloud and Teradata Cloud are exemplars of this.)
At this late date, however, virtually all BI and analytics PaaS offerings are – to varying degrees –designed with the pluses, minuses, and vicissitudes of the cloud context in mind. Obvious cloud pluses include both elasticity – i.e., the ability to scale capacity up or down – and on-demand capacity; obvious cloud minuses include, first, the need to (re)design software to permit elastic scaling and, second, the need to anticipate and mitigate the impact of virtualization on performance and availability.
Less common, however, are microservices- or FaaS-based takes on cloud-native software design.
For example, customers can use Qrvey to build the equivalent of composite BI and analytic applications: i.e., workflows that knit together and orchestrate discrete services, some developed by Qrvey itself, some by Amazon. They can likewise embed specific Qrvey or AWS capabilities in the workflows that power their own business and IT processes – just by calling the relevant services.
This model is consistent with microservices-like design principles, even if Qrvey’s PaaS is not a formal implementation of microservices architecture. “To be honest a lot of what we’re doing today … is the migration towards more of a microservices and serverless … approach,” Abramson told Kavanagh, explaining that “all of these solutions can now be accessed only when they’re needed and you can basically plug and play all of the different components and capabilities that you might need.”
This is why Abramsom believes paradigms such as microservices or FaaS are the way forward for software architecture in the cloud. Qrvey focuses on what it does best – i.e., maintaining and enhancing its BI-analytics feature set and functions – while Amazon focuses on what it does best: building, maintaining, and enhancing generalizable AWS services. “That microservices approach is sort of exactly why migrating towards that model gives you that flexibility so that you can enhance certain features or certain functions without impacting other features or … functions,” Abramson concluded.
About Stephen Swoyer
Stephen Swoyer is a technology writer with more than 25 years of experience. His writing has focused on data engineering, data warehousing, and analytics for almost two decades. He also enjoys writing about software development and software architecture – or about technology architecture of any kind, for that matter. He remains fascinated by the people and process issues that combine to confound the best-of-all-possible-worlds expectations of product designers, marketing people, and even many technologists. Swoyer is a recovering philosopher, with an abiding focus on ethics, philosophy of science, and the history of ideas. He venerates Miles Davis’ Agharta as one of the twentieth century’s greatest masterworks, believes that the first Return to Forever album belongs on every turntable platter everywhere, and insists that Sweetheart of the Rodeo is the best damn record the Byrds ever cut.