UNLEASHING THE DATA MESH REVOLUTION: EMPOWERING BUSINESS WITH CUTTING-EDGE DATA PRODUCTS

The principles of data mesh have moved beyond being just theoretical concepts for data architects and forward-thinking executives. It’s time to start delivering on data mesh’s promise of exceptional data products. Data mesh principles can help us uncover the valuable insights that businesses need.

Feedback Fusion: The power of continuous iteration for product success

When building a product, it’s crucial to understand the utility of the product and how any changes to the product will impact its utility over time.

If we consider building a mobile phone or any other product, the cost of building a phone that is unusable will be significant. Therefore, conducting thorough research in the beginning to understand what the market wants is critical before beginning the product-development process.

Once we have built and distributed a phone, we need to continually consider feedback from different channels, including social media and online reviewers, to continuously iterate and improve the phone.

This feedback loop is also imperative for data, however in the past data developers have typically waited for feedback from data consumers and then reacted. This has introduced time delays and ultimately frustration for data consumers.

With product thinking this approach is turned on its head, data product developers are continuously monitoring both quantitative and qualitative feedback from consumers.

This feedback allows data product teams to proactively evolve the data product to ensure that as data consumers need new capabilities, they are being built into the data product, thus avoiding delays and frustration, and enabling better outcomes for the organization.

Data mesh dilemma: Embracing innovation amidst fear and uncertainty

Data mesh principles, which focus on the notion of first-class data products and other factors, have gained an unprecedented amount of interest in the past eighteen months. The conversation in the data mesh community has largely focused on the principles data mesh and what they mean for each organization. Most organizations have invested heavily in cloud but are still struggling to keep up to the pace that the business requires. “Why does it take me three to six months to get a new or modified dataset? Who’s responsible for the data governance? How can I trust that the dataset can be trusted?” and the list of questions goes on.

What we discovered during these conversations with clients is there is an overall acceptance that data mesh and its principles make good sense, but there is the fear factor on the pain an organization needs to go through to get to the promised land of a truly federated data estate of quality, secured, discoverable data products. So, most organizations have kicked the can down the road.

Start small, think big, and design for industrialization

Here are useful guidelines to help reduce this fear of failure.

1. To effectively build data products, it’s crucial to identify the problem you’re trying to solve and determine why a data product is the appropriate solution from the beginning of the process. Taking the time to clarify the reasons behind your approach will ultimately save you a great deal of time, money, and effort. This fundamental step is applicable to any product-development process, and it’s no different when building data products.

A simple data product canvas together with the business and domain experts need to be committed to this phase. Note: Forget about all technology during this phase.

2. Many organizations have not changed their approach to data management in the last 30 years. It is commonly believed that all data must be centralized into a data warehouse or data lake before it can be analyzed, which is both difficult and costly in terms of human resources and technology. Today decision makers wait for data to be made available before it can be used. This means waiting for data pipelines to be specified and built, however this is typically done in the absence of the complete knowledge of the value of the data to a particular use case. This unnecessarily elongated process is fragile and has a negative impact on an organization’s ability to compete using data.

Fortunately, solutions like Starburst/Trino offer intelligent connectors and a highly optimized federated MPP SQL engine that enables the creation of data products by analysts in the lines of business (domains) with no need for intimate knowledge of the source technology. Lines of business can quickly access data and determine its applicability to a use case without having to rely on central data teams.

If we consider this in the context of cloud-data migrations, solutions like Starburst/Trino enable these data products to be created, managed, and retired while the underlying data platforms are migrated. The system administrators only need to update the connector to ensure uninterrupted service for business users. With Starburst we want to give the data-product teams the option to decide on what works best for them to deliver the best data product that will satisfy the requirements as outlined by the data product canvas.

3. Finally, to ensure that the quality of data products is maintained over time as business needs change, a continuous monitoring and feedback loop is key. Data-product producers need to understand who, how, and for what purpose their data product is being used, so they can proactively manage the data product. This management requires technology capabilities to provide this insight as well as an agile approach to streamline the pipeline from ideation to production and constantly improve efficiency. We look at this as the building of a factory like model for data products.

Data mesh in action

At online fashion retailer Zalando, various lines of business independently utilize Amazon S3 for storing and managing datasets, eliminating the need for a central data team. A central data “enabling team” oversees data-governance standards and identifies reuse opportunities, while a dedicated platform team supplies compute services including a distributed SQL Engine (Starburst) for analytics. This clear division of responsibilities – lines of business managing data, the enabling team governing it, and the platform team providing technology – prevents bottlenecks and centralization, fostering agility in leveraging data to maintain a competitive edge.

A prominent French state organization has been devising its data-estate roadmap for 2025 over the past year. Its current extensive data platform comprises batch processing, streaming processing, AI, and use cases, with concerns about cloud readiness. With a complex data estate plagued by performance and monitoring issues, its goal is to streamline operations using a new data platform based on Starburst and Apache Iceberg. The primary objective is simplification and reduced complexity, achieved by focusing on business outcomes and scaling with data-mesh principles.

“Start small, think big and design for industrialization.”

Dawn of a new era

The rise of data mesh and its principles plus the technical offerings from Starburst marks the dawn of a new era for data products. As businesses embrace the principles of data mesh, it’s essential to address the fear factor associated with adopting this approach. By following the guidelines outlined in this article – focusing on identifying the problem to be solved, leveraging modern solutions like Starburst/Trino for data management, and implementing continuous monitoring and feedback loops – organizations can confidently embark on their journey towards a truly federated data estate. Success stories like Zalando and the large French state organization demonstrate the transformative power of data mesh in improving efficiency, agility, and competitiveness. As we move forward, it’s crucial for businesses to embrace the promise of data mesh, shifting from theoretical discussions to real-world implementation. Only then will they be able to harness the full potential of exceptional data products and uncover the valuable insights needed for sustained success in an increasingly data-powered world.

INNOVATION TAKEAWAYS

OVERCOMING ADOPTION HURDLES IN A FEDERATED DATA ESTATE

Data mesh principles enhance data-product creation, driving valuable insights and competitiveness, but adoption is slowed by perceived challenges in achieving a federated data estate.

THE THREE PILLARS OF EFFECTIVE DATA MESH IMPLEMENTATION

Implementing data mesh effectively involves problem identification, utilizing modern data-management solutions, and establishing continuous monitoring and feedback loops.

DATA MESH IN ACTION

Success stories like Zalando and a large French state organization showcase the benefits of data mesh, including improved efficiency, agility, and competitiveness.

BRIDGING THE GAP, PRACTICAL STEPS TO DATA MESH SUCCESS

Moving from theory to practice in data-mesh implementation allows organizations to better harness data-product power and succeed in a data-powered world.

Interesting read?

Capgemini’s Innovation publication, Data-powered Innovation Review | Wave 6 features 19 such fascinating articles, crafted by leading experts from Capgemini, and key technology partners like Google,  Starburst,  Microsoft, Snowflake and Databricks. Learn about generative AI, collaborative data ecosystems, and an exploration of how data an AI can enable the biodiversity of urban forests. Find all previous Waves here.