Apache Arrow

Apache Arrow is the language-agnostic standard for columnar in-memory data that eliminates serialization overhead between systems like Python, R, and Spark.

Arrow provides a high-performance cross-language development platform for shared memory. By defining a standardized columnar format, it enables zero-copy data exchange across different runtimes (including C++, Java, and Rust) and hardware architectures like CPUs and GPUs. This architecture powers critical data tools such as Pandas, DuckDB, and Google BigQuery Storage, delivering up to 100x speedups for analytical workloads by maximizing cache locality and leveraging SIMD instructions.

https://arrow.apache.org