Computing’s New Logic: The Distributed Data Cloud
A common pattern in today’s analytics ecosystem is to move data generated in different areas of the company to a central location. Data flows into the data lake and is locked into a data warehouse managed by IT staff. The original creator of the data is often an expert in the subject area of the business, effectively losing control or becoming a layer removed from the data, making the job meaningful. This separation reduces the value of the data over time as it is routed from the business consumer. Imagine a new model that will change this ecosystem head-on by breaking down barriers and applying common standards everywhere.
Imagine an analytics stack you can deploy to your enterprise domain. It stays there and is owned by team members in that business area, but is centrally operated and supported by IT. What if all the data products generated there were completely managed in this domain? What if other business teams could subscribe to these data products and gain API access? One organizational pattern (data mesh) that facilitates this decentralization of ownership of data products has received a lot of attention lately. However, which ecosystem architecture is suitable for enabling the data mesh and providing a technical backbone that can handle the new patterns of data growth?
As the amount of data grows, the idea of moving the data to a central processing location is more costly and time-consuming, especially if the data is generated outside a traditional data center or public cloud. Instead, organizations are becoming more and more fond of deploying analytics processing where the data is generated. Easily geographically identifying data for latency, compliance, or security reasons turns the calculation method into a more sustainable, efficient, and logical reality. This is the realm of distributed data clouds. With seamless control of data everywhere, businesses are benefiting from the amazing amount of data we’re aiming for.
Trends Shaping Business Needs & Driving the Cloud Towards Data
A distributed data cloud is an ecosystem pattern that captures data in the right place, at the right time, in a secure, controlled, and reliable way, rather than a single tool or platform. This includes an integrated collection of data management and analytics services across public, private, and network edges.
A distributed data cloud managed by a single control plane is required by a customized combination of physical and virtualized infrastructure based on data gravity, data sovereignty, data governance, and latency requirements. You can deploy your analytics application at any time. Some important trends allow organizations to use this model to maximize the value of their data. In this model, the infrastructure is designed to democratize data rather than lock it down.
Edge Computing Strains Internet Capacity
By 2025, 75% of enterprise-generated data will be created and processed outside of traditional centralized data centers or the cloud. This is up from less than 10% in 2019. The explosion of data and devices at the edge and the rollout of 5G and planning for 6G — 100 Gbps networks over the next 10 years — has hastened the realization that the internet backbone doesn’t have enough capacity to backhaul all of the data activity at the edge over to centralized data centers for analysis.
Distributed Cloud Answers Hybrid Drawbacks
The Gartner Top Strategic Technology Trends for 2021 report suggests that the distributed cloud—the necessary infrastructure as a service precursor of a distributed data cloud platform implementation explained in this article—is emerging to address location impacted latency. The deployment of cloud software and hardware stacks outside a public cloud provider’s data center to afford a mesh of interconnected cloud resources is what’s meant by distributed cloud. Its stacks allow businesses to run applications developed for the public cloud in a company’s own data center and other locations, like multi-access edge computing centers connected to 5G cell tower groups, or on the factory floor in support of IoT applications in manufacturing. But enterprises still benefit from the value proposition of the public cloud and guaranteed SLAs.
Both hybrid cloud and hybrid IT break the fundamental value propositions of the cloud. Hybrids can be very difficult to run efficiently, maximizing the scalability and resilience of the services offered by the public cloud. Hybrids do not provide the efficiency of cloud operations, governance, and updates that public clouds provide, and these systems are not keeping up with public cloud innovation. A distributed cloud means the same seamless cloud experience everywhere.
Mobile and Multi-Experience Hyper Personalized Business
Ultimately, organizations want to put interactive predictive analytics into the hands of real consumers. To this end, data warehouses serve the user community of millions of end-users, rather than the user community of thousands of users. The current ubiquity of mobile device adoption points to the idea of where the enterprise experience of multi-sensor, multi-device, and multi-touch points with data is heading. Computers are rapidly becoming a user environment. API-driven culture everywhere, seamless UX/UI, and democratized data access across the enterprise facilitates the transition to real-time, hyper-personalized interactions between people, places, and things.
Use Cases & More
As these trends drive the emergence of distributed data clouds, several use cases are emerging. First, there is a widespread need for simplified hybrid and multi-cloud operations that provide a consistent environment across public clouds, on-premises, and edges. A compelling reason for this, particularly in regulated industries such as banking, is to help reduce cloud concentration risk by distributing data and analytics across more than one cloud provider or data center. To achieve this using a distributed data cloud, an enterprise can provision containerized data management and analytics applications and run them anywhere that Kubernetes is deployed — in a public cloud, on-premises, or at the edge. Everything happens via the same management UX and DevOps processes and from the same web console and APIs.
Second, processing personal identifiable information (PII) in residence is a scenario where localized access and regulatory compliance transition to the best solution. If you run an instance optimized for distributed data clouds on a public cloud stack involved in the hospital, patient data may remain in the source.
The third application is for IoT analysis. The ability to perform secure analysis on the network edge means a real-time response such as the consumer near the consumer via the distributed data cloud and implies real-time response such as connected cars, intelligent cities, energy grids, and much more. In a multi-access edge environment for monitoring network quality in real-time, optimization analysis at AWS wavelengths can be fully feasible.
Delivering a distributed data cloud that makes it easy to manage and use data anywhere is not and will never be a single-vendor game. Rather, a consortium of companies that unite and co-operate around this idea brings a party of data and success to companies that are ready to embrace a more logical future.