Company Description

A global leader in commercial real estate services that offers a wide array of services including, but not limited to, property management, investment management, and valuation services. One of their innovative approaches involves utilizing technology to optimize and enhance workflows, particularly in analytics dashboards that use data from various sensors installed in apartment buildings. This not only involves ingesting and validating data but also refining raw data into actionable insights, thereby enabling comprehensive analytics.

Challenge

Problem

The main challenge encountered in the Databricks case was the existence of issues and bottlenecks in the Databricks dashboard workflows, which were leading to workflow failures and, consequently, negatively impacting the performance of the analytics dashboards.

Project Goals

  • Optimize and enhance workflows in the Databricks dashboard.
  • Refine raw data into actionable insights.
  • Address and resolve workflow issues and bottlenecks.

Challenge

Problem

The main challenge encountered in the Databricks case was the existence of issues and bottlenecks in the Databricks dashboard workflows, which were leading to workflow failures and, consequently, negatively impacting the performance of the analytics dashboards.

Project Goals

  • Optimize and enhance workflows in the Databricks dashboard.
  • Refine raw data into actionable insights.
  • Address and resolve workflow issues and bottlenecks.

Solution

Xorbix made strategic use of the Databricks UI tools, including the Spark UI and Clusters UI, to accurately pinpoint and resolve a memory exhaustion issue that was plaguing a Spark structured streaming job. To ensure continuous and seamless operation, they implemented watermarking. Furthermore, to facilitate proactive issue resolution, they established email notifications to alert the team immediately in case of any workflow failures. This multifaceted approach not only addressed the memory exhaustion problem but also enhanced the overall robustness and responsiveness of the system.

Innovations

Proactive alerts via email notifications streamlined issue resolution, and watermarking ensured continuous operation of the Spark structured streaming job.

High-Level Architecture

Components

  • Amazon Kinesis for data ingestion.
  • Databricks for data aggregation and validation.
  • Postgres database for batch processing and analytics.

Core Technologies

  • Amazon Kinesis
  • Databricks
  • Postgres database

Process

Team

The team included a certified Databricks developer, contracted specifically for their expertise in optimizing and enhancing workflows within the Databricks dashboard. The team specialized in data ingestion, streaming, aggregation, and validation as well as database management and analytics.

General Development

Utilizing an agile methodology, the development involved addressing workflow issues, improving performance, and refining raw data into actionable insights. Regular iterations and proactive adjustments were made to ensure optimal performance.

Testing

Testing involved validating the functionality and responsiveness of the implemented solutions, such as watermarking and email notifications, and assessing their impact on workflow performance and issue resolution.

Process

Team

The team included a certified Databricks developer, contracted specifically for their expertise in optimizing and enhancing workflows within the Databricks dashboard. The team specialized in data ingestion, streaming, aggregation, and validation as well as database management and analytics.

General Development

Utilizing an agile methodology, the development involved addressing workflow issues, improving performance, and refining raw data into actionable insights. Regular iterations and proactive adjustments were made to ensure optimal performance.

Testing

Testing involved validating the functionality and responsiveness of the implemented solutions, such as watermarking and email notifications, and assessing their impact on workflow performance and issue resolution.

Results

Results Overview

The optimization and enhancement of workflows, implementation of watermarking, and introduction of email notifications for workflow failures resulted in streamlined issue resolution and continuous operation of the Spark structured streaming job. 

Get In Touch With Us

Would you like to discuss how Xorbix Technologies, Inc. can help with your enterprise IT needs.


Blog

Case Study

Blog

Case Study

One Inc ClaimsPay Integration

One Inc’s ClaimsPay integration is our major Midwest headquartered Insurance provider client’s ambitious electronic payment integration project.

Blog

Case Study

Blog

Case Study

One Inc ClaimsPay Integration

One Inc’s ClaimsPay integration is our major Midwest headquartered Insurance provider client’s ambitious electronic payment integration project.