Signol
A scalable Lakehouse to reduce fuel consumption and visualize environmental impact
Summary
Building a scalable, cloud-based Lakehouse architecture on Databricks to process large volumes of IoT data from thousands of airplanes and ships, enabling advanced analytics, AI-based modeling, and faster client onboarding.
Signol needed a robust platform to accommodate its evolving needs in data access, management, and governance. We developed and deployed a scalable Lakehouse environment with streamlined, modular, and reusable workflows, designed to support both Data Engineering and Data Science teams while significantly reducing client onboarding time.
Services
Industry
The Challenge
Vast data volumes, slow client onboarding, and limited historical data access for advanced analytics.
Signol receives large volumes of complex IoT data from thousands of airplanes and ships. Existing infrastructure couldn't keep pace with growth. Managing vast data volumes was increasingly difficult, onboarding new clients was slow, and historical data access for advanced analytics was limited. The platform needed to integrate complex workflows, provide historical data access, and support advanced analytics and modeling.
Problem 01
Scaling bottlenecks and slow onboarding
- ▪Complex IoT data streams from thousands of vessels and aircraft required scalable infrastructure
- ▪Each new client required significant manual effort to set up data pipelines
- ▪Existing infrastructure couldn’t keep pace with the company’s growth
- ▪No standardized process for repeatable, efficient deployments
Problem 02
Limited analytics and data accessibility
- ▪Historical data was difficult to access and utilize for advanced analytics
- ▪No foundation for machine learning models or AI-based forecasting
- ▪Complex data workflows were fragmented across the organization
- ▪Data quality and governance lacked a structured framework
The Solution
We built a cloud-based Lakehouse on Databricks with a medallion architecture and Infrastructure-as-Code approach to optimize data management and processing. The modular design ensures component reusability, significantly reducing development time for onboarding new clients.
01
Bronze, Silver, Gold data layers
A multi-layered medallion architecture supporting data quality and accessibility at every stage. This ACID-compliant environment is optimized for efficient analytics and reporting, with rapid deployment of new data models.
02
Modular and reusable workflows
Streamlined, modular workflows that can be adapted across different client projects. Infrastructure-as-Code ensures consistent, repeatable deployments that maximize efficiency and scalability.
03
Analytics and ML foundation
A foundation for sophisticated data-driven decision-making based on advanced analytics and machine learning models, enabling Signol to elevate and scale their product offering.
Onboarding
Terraform / IaC
Landing Zone
External Ingestion
Bronze
Silver
Gold
Analytics
SQL & APIs
Production
Dashboards & Reports
Impact
Democratized data access and improved system flexibility, transforming how Signol processes IoT data and delivers environmental impact.
Faster Client Onboarding
Modular, reusable workflows allow new clients to be onboarded significantly faster by leveraging standardized data pipelines and Infrastructure-as-Code deployments.
Accelerated Development
Streamlined development timelines through component reusability and automated workflows, enabling the team to deliver new features and analytics faster.
Auto-Scaling Infrastructure
The platform automatically scales with data volume and complexity, ensuring optimal resource utilization as IoT data from thousands of sources continues to grow.
Robust Data Quality
A comprehensive data quality framework provides the foundation for improved accuracy and easy utilization of historical data for analysis and model building.
Technology Stack
Databricks (cloud lakehouse), dbt (SQL transformation), GitHub (version control, CI/CD), MLflow (AI modeling), Great Expectations (data quality).
“DEUS transformed our data infrastructure into a scalable, efficient platform that has fundamentally changed how we process IoT data and serve our clients. The modular design means we can onboard new clients faster and deliver more sophisticated analytics to drive real environmental impact.”
Michael Fanning, CEO Signol
Methods
A data engineering and platform approach, from architecture design through modular implementation to production-ready analytics.
Lakehouse architecture
Medallion data layers
Data pipeline automation
Infrastructure-as-Code
Data quality frameworks
Cloud-native deployment
ML foundation
Advanced analytics
IoT data processing
Workflow automation
Modular design
Auto-scaling infrastructure