Data Sharing with Databricks Delta Sharing: A Complete Guide

Databricks Delta Sharing for Secure Data Exchange

Author: Inza Khan

30 May, 2024

Databricks Delta Sharing emerges as a transformative solution, offering a streamlined platform for sharing and accessing data across diverse computing environments. Let’s explore how Databricks Delta Sharing works and its key features that enable organizations to collaborate effectively.

Understanding Delta Sharing

Delta Sharing is an open protocol developed by the Databricks platform for sharing data seamlessly. It allows organizations to share data in its original format, avoiding the need for complex data transformations. It aims to simplify the process of data sharing by providing a standardized, secure, and scalable way to share data, regardless of the computing platforms being used.

How Does Databricks Delta Sharing Work?

Integration with the Databricks Platform

Databricks Delta Sharing seamlessly integrates with Unity Catalog, allowing organizations to centrally manage and audit shared data assets. This integration simplifies the sharing process and ensures compliance with security and regulatory requirements, making it easy for organizations to share data with suppliers and partners.

Privacy-Safe Data Collaboration

Databricks clean rooms offer a secure environment for collaborating with customers and partners on any cloud platform. These environments enable organizations to share data from their data lakes without replication, ensuring data privacy and security. Collaborators can run complex computations and workloads in various languages, including SQL, R, Scala, Java, and Python, accelerating insights generation.

Accessing Data Products via the Marketplace

The Databricks Marketplace provides a convenient platform for discovering and accessing various data products, such as datasets, machine learning models, dashboards, and notebooks. This open marketplace allows organizations to explore and evaluate data products from anywhere, facilitating collaboration and innovation.

Simplified Share Management

With Databricks Delta Sharing, managing shares is simple. Organizations can create and manage providers, recipients, and shares through a user-friendly interface, SQL commands, or REST APIs. The platform also supports CLI and Terraform, enabling organizations to automate and streamline their data sharing workflows.

Three Paths to Data Sharing

Databricks-to-Databricks Delta Sharing

The Databricks-to-Databricks Delta Sharing model is designed to streamline data sharing between Databricks workspaces, offering a simplified and efficient workflow. It begins with the recipient providing a unique sharing identifier, establishing a secure connection. Subsequently, the data provider creates a share within their Unity Catalog metastore, comprising a collection of tables, views, volumes, and notebooks. The recipient object is then created to grant access to the shared data, ensuring authenticated access. Upon granting access, users can seamlessly access the shared resources through tools like Catalog Explorer or Databricks CLI.

Databricks Open Sharing Protocol

The Delta Sharing Open Sharing Protocol is designed for data providers in Unity Catalog-enabled Databricks workspaces who want to share data beyond Databricks. Unlike the Databricks-to-Databricks protocol, this one allows sharing across different computing platforms. Providers start by creating recipients, generating tokens and credential files for secure access. They then create shares with data collections from their Unity Catalog metastore, granting access to recipients through an activation link. Once activated, recipients can securely access the shared data using their preferred tools or platforms.

Delta Sharing Reference Server

The Delta Sharing Reference Server provides developers with a foundational framework for implementing the Delta Sharing Protocol, enabling them to build and test their connector implementations effectively. While it isn’t a fully secure web server, it serves as a valuable starting point for testing and experimentation. Security should be prioritized when deploying the server, with recommendations to utilize secure proxies for public exposure. Additionally, users should consider managed services offered by vendors like Databricks as alternatives for their data sharing needs, depending on their specific requirements and preferences.

Advantages of Databricks Delta Sharing

• Cross-Platform Compatibility: Databricks Delta Sharing allows sharing data across different platforms without being tied to a specific vendor. It supports Delta Lake and Apache Parquet formats, ensuring compatibility and flexibility.
• Real-Time Data Accessibility: Organizations can share live data across platforms, clouds, or regions without duplicating it. This ensures stakeholders have access to the latest information, speeding up decision-making.
• Streamlined Governance: Databricks Delta Sharing provides centralized management, governance, auditing, and usage tracking of shared data. This simplifies control and compliance efforts across the organization.
• Marketplace for Data Products: With Databricks Delta Sharing, organizations can build, package, and distribute data products like datasets and machine learning models through a centralized marketplace.
• Secure Collaboration Environments: Databricks Delta Sharing offers secure environments for collaborative data analysis while ensuring data privacy. This allows organizations to work with partners and customers while protecting sensitive information.

Conclusion

Databricks Delta Sharing is a reliable solution for secure data exchange, enabling smooth collaboration and data access across different computing setups. It offers a platform integrated with the Unity Catalog, making data sharing easier while ensuring adherence to security and regulatory standards. Features like Databricks clean rooms and the Marketplace provide secure collaboration environments and convenient access to data products. With simplified share management and three distinct sharing paths, Delta Sharing empowers organizations to share data efficiently and securely.