Databricks has also announced the general availability of Zerobus Ingest, a new serverless streaming ingestion offering in the Lakeflow Connect family of products, which enables businesses to stream high-velocity data directly into managed Delta tables in the Databricks Lakehouse Platform. Zerobus Ingest eliminates the need for message brokers and complex streaming systems and provides a single sink architecture that is optimized for real-time analytics.
Zerobus Ingest is built to facilitate data teams in scaling event-level data ingestion with a total aggregate throughput of 10+ GB/sec to a single table and sub-five-second latency, supporting thousands of concurrent clients writing directly to Unity Catalog-managed Delta tables. This serverless function automatically scales based on workload requirements, allowing for near real-time operational analytics on applications such as telemetry pipelines, IoT data, clickstream data, and cybersecurity event data without having to deal with message brokers.
By embracing a single-sink architecture optimized for lakehouse consumption, Zerobus Ingest eliminates the complexity and expense of traditional multi-component streaming stacks, such as Apache Kafka clusters, schema registries, and connector frameworks. This simplified architecture minimizes engineering effort and eliminates redundant storage of data, while maintaining governance and observability capabilities via Unity Catalog.
Also Read: Keeper Security Adds Native Google Cloud Support to KeeperPAM
The Zerobus Ingest service can be connected to using gRPC and REST APIs, as well as Python, Java, Rust, Go, and TypeScript SDKs to cater to different ingestion needs. The gRPC API provides high-performance and low-latency streaming capabilities, while the REST API provides flexibility for simple integrations and webhooks.
Zerobus Ingest is now generally available on the major cloud platforms and as part of the Lakeflow Connect offering within the Databricks Data Intelligence Platform, with future releases planned to support the broader cloud ecosystem. The GA release is accompanied by a promotional pricing period using the Lakeflow Jobs Serverless SKU.