Contact Us

Company Description

A global leader in commercial real estate services that offers a wide array of services including, but not limited to, property management, investment management, and valuation services. One of their innovative approaches involves utilizing technology to optimize and enhance workflows, particularly in analytics dashboards that use data from various sensors installed in apartment buildings. This not only involves ingesting and validating data but also refining raw data into actionable insights, thereby enabling comprehensive analytics.

Challenge

Problem

The main challenge encountered in the Databricks case was the existence of issues and bottlenecks in the Databricks dashboard workflows, which were leading to workflow failures and, consequently, negatively impacting the performance of the analytics dashboards.

Project Goals

  • Optimize and enhance workflows in the Databricks dashboard.
  • Refine raw data into actionable insights.
  • Address and resolve workflow issues and bottlenecks.

Challenge

Problem

The main challenge encountered in the Databricks case was the existence of issues and bottlenecks in the Databricks dashboard workflows, which were leading to workflow failures and, consequently, negatively impacting the performance of the analytics dashboards.

Project Goals

  • Optimize and enhance workflows in the Databricks dashboard.
  • Refine raw data into actionable insights.
  • Address and resolve workflow issues and bottlenecks.

Solution

Xorbix made strategic use of the Databricks UI tools, including the Spark UI and Clusters UI, to accurately pinpoint and resolve a memory exhaustion issue that was plaguing a Spark structured streaming job. To ensure continuous and seamless operation, they implemented watermarking. Furthermore, to facilitate proactive issue resolution, they established email notifications to alert the team immediately in case of any workflow failures. This multifaceted approach not only addressed the memory exhaustion problem but also enhanced the overall robustness and responsiveness of the system.

Innovations

Proactive alerts via email notifications streamlined issue resolution, and watermarking ensured continuous operation of the Spark structured streaming job.

High-Level Architecture

Components

  • Amazon Kinesis for data ingestion.
  • Databricks for data aggregation and validation.
  • Postgres database for batch processing and analytics.

Core Technologies

  • Amazon Kinesis
  • Databricks
  • Postgres database

Process

Team

The team included a certified Databricks developer, contracted specifically for their expertise in optimizing and enhancing workflows within the Databricks dashboard. The team specialized in data ingestion, streaming, aggregation, and validation as well as database management and analytics.

General Development

Utilizing an agile methodology, the development involved addressing workflow issues, improving performance, and refining raw data into actionable insights. Regular iterations and proactive adjustments were made to ensure optimal performance.

Testing

Testing involved validating the functionality and responsiveness of the implemented solutions, such as watermarking and email notifications, and assessing their impact on workflow performance and issue resolution.

Process

Team

The team included a certified Databricks developer, contracted specifically for their expertise in optimizing and enhancing workflows within the Databricks dashboard. The team specialized in data ingestion, streaming, aggregation, and validation as well as database management and analytics.

General Development

Utilizing an agile methodology, the development involved addressing workflow issues, improving performance, and refining raw data into actionable insights. Regular iterations and proactive adjustments were made to ensure optimal performance.

Testing

Testing involved validating the functionality and responsiveness of the implemented solutions, such as watermarking and email notifications, and assessing their impact on workflow performance and issue resolution.

Results

Results Overview

The optimization and enhancement of workflows, implementation of watermarking, and introduction of email notifications for workflow failures resulted in streamlined issue resolution and continuous operation of the Spark structured streaming job. 

Internet of Things (IoT)
Blogs
Using IoT for Energy Optimization in Manufacturing
The manufacturing industry is a significant energy consumer, accounting for nearly 54% of global energy...
Internet of Things (IoT)
Blogs
The Role of IoT in Energy Management
The pressure to achieve energy efficiency while maintaining productivity and profitability is greater...
Angular 4 to 18
Case Studies
Revitalizing a Legacy Portal: The Path from Angular 4 to 18
This project revolved around modernizing a critical management...
TrueDepth Technology
Case Studies
Modernizing Orthotics with TrueDepth Technology
The client, a leading provider of foot scanning technology for...

Let’s Start a Conversation

Request a Personalized Demo of Xorbix’s Solutions and Services

Discover how our expertise can drive innovation and efficiency in your projects. Whether you’re looking to harness the power of AI, streamline software development, or transform your data into actionable insights, our tailored demos will showcase the potential of our solutions and services to meet your unique needs.

Take the First Step

Connect with our team today by filling out your project information.

Address

802 N. Pinyon Ct,
Hartland, WI 53029

Billing Inquiries


(866) 568-8615

Information and Sales


info@xorbix.com


Services

Industries

Solutions

Solutions

Contact Us

802 N. Pinyon Ct, Hartland, WI 53029

(866) 568 8615

Facebook-f X-twitter Youtube Linkedin Instagram

Contact Us

802 N. Pinyon Ct, Hartland, WI 53029 (866) 568 8615
Facebook-f Twitter Youtube Linkedin Instagram