Draft:Data marketplace

Data marketplace is an online platform for sharing and consuming data in the form of data assets or data products. Part of the data management stack, it aims to bring together data producers and data consumers (including business users and AI) in a single space, with the objective of increasing access to understandable, high-quality data.

Included within its Data Marketplaces and Exchange (DME) category by Gartner, data marketplaces can provide data internally within an organization, externally with partners, or as open data.

Concept

Digitization has dramatically increased data volumes within organizations, with IDC predicting that by 2025 the world will contain 175 zettabytes of data.

This has created a need to both manage this data and provide access to it to enable business intelligence and data analysis. However, data is often scattered within multiple systems (such as data warehouses and data lakes), and is in formats that are only understandable by technical experts, such as data scientists. According to IDC, 81% of IT leaders cite data silos as a major barrier to digital transformation. This means that data is not freely available to business users or external audiences such as partners or citizens, limiting its value, and holding back AI deployments.

Data marketplaces solve this issue, providing seamless, self-service access to high-quality data in an understandable, secure and auditable manner. They break down data silos, reduce friction in data access, and enable a broader range of users, including non-technical profiles, to find, understand, and consume data autonomously. Data assets on the marketplace can be raw data, data visualizations or data products.

Data marketplaces combine data management functions such as data governance with the user-friendly experience offered by e-commerce marketplaces in order to increase the usage of data. These include features such as powerful search engines, feedback, ratings, subscriptions and product description sheets. According to Gartner, data marketplaces provide infrastructure, transactional capabilities, and services for both consumers and providers of data assets.

History and timeline

Data marketplaces have evolved since they first emerged in terms of both their scope and usage.

2000s

With the rise of the internet, data brokers began collecting, aggregating, distributing and selling personal, financial and marketing data to third parties online. Data marketplaces were deployed to monetize this data, making it discoverable and accessible to users, either through subscriptions or one-off purchases.

At the same time, regulations, such as the US Open Government Initiative of 2009 and others around the world mandated greater transparency and data sharing with the public. Data sharing portals were created by public and government bodies to make this information available through self-service to all users.

2010s

Due to the growth of big data and cloud platforms, cloud-based data exchange platforms emerged. These were offered by major infrastructure providers, and included Amazon Web Services (AWS) Data Exchange, Snowflake Data Marketplace, and the Google Cloud Platform. These platforms moved beyond simple data brokerage or open data by providing structured, catalogued data sharing between organizations.

2020s

Driven by a need to increase internal data sharing with both business users and AI, organizations are now looking to adopt internal data marketplaces. These aim to democratize data consumption by providing seamless access for all employees and AI to trusted data, including data products, through an intuitive, e-commerce style experience. According to Gartner analyst Richa Jha, "by providing a single, governed platform for discovering, sharing, and scaling data products, data marketplaces drive productivity, collaboration, and ROI across the enterprise."

Data marketplaces within the overall data architecture

Data marketplaces provide a consumption and collaboration layer for data. That means they complement and integrate with other parts of the overall data architecture, including:

Data warehouses and data lakes

Data marketplaces connect to data sources, such as data warehouses or data lakes, to provide intuitive access to the data stored within them, enabling data to be shared and distributed to non-technical audiences. Access can be direct, with data and data products stored within the data marketplace or virtualized.

Data catalog

A data catalog provides a technical inventory of an organization's data estate. It collects technical information on all available data assets within an organization, based on metadata descriptions. This ensures traceability, and supports compliance and governance requirements. Unlike a data marketplace, a data catalog does not provide access to data, and is designed to be used by data professionals, rather than the business. This means it lacks an intuitive, understandable interface and is consequently not easily accessible by business users.

Data mesh

Data mesh is an architecture and framework for data management, first defined by Zhamak Dehghani in 2019. It aims to decentralize data ownership to delegate responsibility, empowering teams and focusing on delivering data to users in the form of self-service data products. The data marketplace is a central pillar of data mesh, providing intuitive access to these data products, and creating a collaboration space for data owners and data consumers.

Data product

Data products are high-value, consumable data assets that package high-quality data and associated tools to enable seamless usage by business users at scale. First defined by McKinsey in 2022, they have an identified owner, a service level agreement (SLA), and a reusability logic.

Core components of a data marketplace

A data marketplace typically includes specific core components:

E-commerce style interface

An e-commerce style experience that engages non-technical users, minimizes the need for training and builds confidence and trust in data. Look and feel should be customizable to incorporate corporate design guidelines to ensure consistency with other organizational applications.

Built-in data catalog

As in a standalone data catalog, this indexes all available data, based on metadata that includes type, source, owner, freshness, and quality level.

Discovery and search engine

This enables users to search, filter, explore and discover available data intuitively. As in an e-commerce marketplace, it should be intelligent, and provide relevant results based on natural language queries.

Access control and security management

Data marketplaces will contain data that needs to be protected under regulations such as the General Data Protection Regulation (GDPR) in Europe, the California Consumer Privacy Act (CCPA) in the United States, and sector-specific frameworks in industries such as finance and healthcare. To ensure both security and compliance while maximizing data consumption, the data marketplace should include granular access management and a full audit trail.

Semantic layer and business glossary

Different parts of the business are likely to use different terms to describe data. This leads to inconsistencies and an inability to share data across systems and teams. The semantic layer and business glossary standardize a shared vocabulary and common definitions of business indicators and concepts, providing a single language for data across the business and for AI agents.

Data governance mechanisms

These enforce corporate data governance policies, ensuring data traceability through data lineage, quality certification, usage monitoring, and continuous improvement through user feedback loops.

Collaboration features

As on an e-commerce website, a data marketplace should provide collaboration features that bring together data users and data owners. This includes the ability to rate data products, share use cases, and provide feedback to data owners, creating a community around data and supporting a data-driven culture.

Types of data marketplace

While they share the same underlying technology, data marketplaces can be deployed in three broad ways:

Internal data marketplaces

These bring together data from across an organization and make it available via self-service to employees from across the business. They aim to widen access to data and consequently to improve decision-making and reporting, increase performance and maximize efficiency.

Ecosystem data marketplaces

These extend sharing beyond a single organization, enabling multiple partners (public institutions, industry players, research bodies) to share and consume data within a governed framework. Data can be provided by all parties or simply by one organization and consumed by others. Ecosystem data marketplaces are particularly relevant in sectors such as healthcare, energy, smart cities, and supply chain, where cross-organizational data collaboration creates collective value. The Catena-X automotive data ecosystem is an example of this type of data marketplace.

External commercial marketplaces

An evolution of the first data broker marketplaces, these monetize data and enable it to be bought and sold. Data is offered through subscriptions, licenses, or APIs. As well as data brokers, other organizations, such as cellphone network operators, now monetize their operational data through commercial data marketplaces.

Use cases for data marketplaces

Data marketplaces find applications across many sectors and organizational contexts, including:

• Large organizations: Allowing teams to discover and consume data produced by other departments without requiring IT support. This accelerates decision-making, reduces duplication, and improves data quality through shared ownership.
• Financial services: Enabling regulatory data (including compliance, risk, and reporting) to be consolidated and shared across the organization, as well as distributing market data and Know Your Customer (KYC) information across business teams.
• Healthcare: Securely sharing clinical, administrative, and research data between institutions, providing a more efficient, patient-centric experience while adhering to strict confidentiality requirements.
• Retail and distribution: Centralizing and sharing product, customer, and logistics data to deliver a single source of truth that supports analytics use cases, personalization, and the training of AI models.
• Public sector and smart cities: Supporting open data initiatives to increase transparency, and enabling the sharing of critical environmental, demographic, mobility and infrastructure data, between public bodies and private partners. This improves collaboration, decision-making, and innovation.
• AI and machine learning: Providing a structured source of certified, documented, and semantically enriched datasets. This enables AI agents and models to access reliable, contextualized data at scale, underpinning trustworthy and high performing AI systems.

Challenges and limitations of data marketplaces

The effectiveness of a data marketplace depends on four key factors:

• Data must be trustworthy, high-quality and easily understandable by non-technical users. It should be comprehensive, with sufficient data assets to attract and engage users and data owners.
• Data must be easily discoverable, accessible and consumable through an intuitive experience that does not require technical skills.
• Organizations need to build a data culture, where data is freely shared across the business and all users confidently use it in their daily working lives.
• Data must be well-governed, compliant and shared securely, with a full audit trail of usage to meet corporate policies and regulatory requirements.

Without this mix of technology, accessibility, cultural and process factors, data marketplaces will fail to engage users or data owners, and will consequently not deliver benefits or ROI.

References

This page will be placed in the following categories if it is moved to the article namespace.

Categories:

• Data management
• Information technology
• Data