Google Cloud Dataflow

A fully managed service for executing Apache Beam pipelines to process batch and streaming data at scale.

Dataflow handles the heavy lifting of data processing by automating resource provisioning and horizontal autoscaling. It runs on the open source Apache Beam SDK, allowing you to write a single pipeline for both historical batch loads and real-time streaming events. The service features built-in optimizations like liquid sharding to eliminate bottlenecks and Streaming Engine to offload resource-intensive tasks from worker VMs. With exactly-once processing guarantees and tight integration into BigQuery and Pub/Sub, it is the standard choice for high-throughput ETL (Extract, Transform, Load) and real-time analytics on Google Cloud.

https://cloud.google.com/dataflow