How Databricks Marketplace Transforms Data Science Workflows

Challenges in Traditional Data Marketplaces

Author: Junaid Amjad

Data consumers in existing marketplaces grapple with the lack of contextual information and user-friendly data evaluation, often finding themselves mired in deciphering data models and reliant on support teams. This is due to limited overviews and sample queries, which culminates in a frustrating and inefficient experience. Moreover, the common ‘walled garden’ approach in these platforms restricts data exchange, fostering platform dependency and a lock-in effect.

Concurrently, data providers aiming to boost sales and reduce operational costs face significant hurdles in these marketplaces. They are constrained to basic dataset packaging and minimal enhancements, leading to complex onboarding processes, protracted sales cycles, and potential revenue losses. Additionally, the necessity to conform data to proprietary formats and replicate across different clouds further escalates costs and operational challenges, often resulting in the sidelining of potentially valuable datasets.

Solution: Databricks Marketplace

The Databricks Marketplace emerges as a solution, addressing the needs of both data consumers and providers. It offers a streamlined, context-rich platform for data discovery and evaluation, breaking free from the constraints of traditional marketplaces. For providers, it presents an efficient way to increase sales while managing operational costs effectively.

The next section goes through benefits of the Databricks Marketplace, showcasing how it is set to redefine the data exchange landscape and open new opportunities for growth in data science and machine learning communities.

Databricks Marketplace Enhances Data Science Workflows

The Databricks Marketplace significantly elevates data science workflows by introducing several key advantages, both for data consumers and providers.

For data consumers, the marketplace offers a comprehensive suite of data assets. This includes datasets, dashboards, notebooks, and ML models, facilitating an easier and more efficient evaluation of data. This access goes beyond simple datasets, enabling consumers to quickly gain insights. For instance, they can use starter notebooks for exploratory data analysis or leverage pre-built ML models for predictive analytics. The availability of Databricks-hosted dashboards allows a live exploration of data without incurring additional costs. This holistic approach considerably speeds up the evaluation, acquisition, and analysis cycle, thereby enhancing the overall value derived from the data.

Another significant feature of Databricks Marketplace is its open nature. Thanks to Delta Sharing, data consumers are not restricted to the Databricks platform, allowing for seamless access to data products and avoiding platform lock-in. This openness empowers consumers to use their preferred tools, maximizing the value they extract from the data.

From the perspective of data providers, the Databricks Marketplace opens new avenues for distributing and monetizing a diverse range of data products. Providers are not limited to selling just datasets; they can also offer notebooks, dashboards, and models. These additional resources help consumers fully understand and utilize the data. For example, a provider selling Environmental Social and Governance (ESG) data can bundle it with a notebook for NLP analysis, a dashboard visualizing polluting companies, and a model predicting changes in a company’s ESG ranking. Such comprehensive offerings were not feasible in traditional data marketplaces.

Moreover, the marketplace broadens the reach of data products beyond the Databricks Platform users, increasing the potential revenue for providers. Importantly, it eliminates the need for replication of data products across different clouds. Providers can share their products from a single source without the obligation to replicate, thus reducing additional costs and operational complexities.

Navigating the Databricks Marketplace: A Guide for New Data Providers

Before You Begin:

• Agree to the Marketplace provider policies.
• Ensure your Databricks account is on the Premium plan or above.
• Have a Databricks workspace enabled for Unity Catalog.

Apply to be a Databricks Marketplace Provider:

• If not part of the Databricks Partner Program, apply through the Databricks Data Partner Program.
• Visit the Databricks Data Partner Program page, click “Apply Now,” and fill out the application form, selecting ‘Marketplace’ as your interest.
• For existing partners, contact partnerops@databricks.com directly.

Assign the Marketplace Admin Role:

• Once approved, assign at least one user the Marketplace admin role through the account console for managing your provider profile and listings.

Create Your Marketplace Provider Profile:

• Log in to your Unity-Catalog-enabled workspace and access the Provider console in the Marketplace section.
• Create a profile by providing details like Provider Name, Description, Organization Website, Business Email, Support Email, Terms of Service Link, Privacy Policy, and upload a Logo Image.

Create Shares:

• Add data tables and views to your Unity Catalog metastore.
• Create a share and add these tables and views to it.
• Ensure you have the necessary permissions for creating and managing shares.

Create Listings:

• Log into your Databricks workspace and navigate to the Marketplace section.
• Access the Provider console and go to the Listings tab to click “Create listing.”
• Fill in the required fields such as Listing Name, Short Description, Provider Profile, Terms of Service, Public/Private Marketplace selection, Data Availability, Categories, Attributes, Detailed Description, and optionally add Sample Notebooks, Documentation, and Privacy Policy.
• Save your draft, preview it, and then publish your listing.

How Pre-Built Models and Tools on Databricks Marketplace Help Productivity?

The Databricks Marketplace, underpinned by Delta Sharing, revolutionizes productivity in various industries by providing secure data sharing and collaboration. This platform is pivotal for organizations aiming to extract maximum value from both internal and external data sources. By enabling efficient third-party data utilization, it aids in generating new insights and fostering innovation across diverse sectors such as healthcare, finance, retail, and manufacturing.

This Marketplace isn’t just a data exchange hub; it’s a collaborative ecosystem connecting data consumers and providers. With over 500 offerings from more than 80 providers, it streamlines the search and evaluation process for data and AI products. This extensive range accelerates project timelines and enhances decision-making, making it a critical tool for organizations looking to quickly adapt and innovate.

A standout feature of the Databricks Marketplace is the introduction of industry-specific Solution Accelerators. These accelerators provide pre-built code, sample data, and other resources tailored to specific industries. This dramatically reduces the development and deployment time for custom solutions. Industries like Insurance Services, Healthcare, and Manufacturing benefit immensely from these accelerators, as they offer ready-to-use tools for rapid deployment.

These pre-built models and tools significantly contribute to organizational efficiency. By offering ready-made solutions. The Marketplace allows teams to avoid the lengthy phases of development and testing, leading to quicker insights and innovation. This not only conserves resources but also enables organizations to focus on strategic, value-adding activities rather than the technical aspects of model development.