Technology

Apache Spark

Apache Spark is the unified, open-source analytics engine for high-speed, large-scale data processing across clusters.

Spark is your go-to engine for serious big data workloads: it's a unified platform designed for speed and versatility. The core advantage is in-memory processing, which makes it up to 100x faster than traditional MapReduce for iterative algorithms (like machine learning) and interactive queries. It provides high-level APIs in key languages—Scala, Java, Python (PySpark), and R—and integrates a full stack of tools: Spark SQL for structured data, MLlib for machine learning, Structured Streaming for real-time analytics, and GraphX for graph processing. Deploy it on Hadoop YARN, Kubernetes, or standalone; it handles batch, streaming, and advanced analytics on data of any size.

https://spark.apache.org/

What builders pair with Apache Spark

Projects using both technologies. Select a pairing to see a project.

Built with Apache Spark

BERT 1 shared project BLOOM 1 shared project ECC 1 shared project fastXML 1 shared project GitHub 1 shared project GPT-3 1 shared project

11 more pairings

GPT-4 1 shared project In-Context Learning 1 shared project Llama-2 1 shared project Mulan 1 shared project PaLM 2 1 shared project prompt space 1 shared project RAkEL 1 shared project RoBERTa 1 shared project scikit-learn 1 shared project TensorFlow 1 shared project XML-CNN 1 shared project

Pairing: BERT

Photo from the event — Event photo

In-Context Learning for Extreme Multi-Label Classification

Chicago · July 23, 2024

view project | view BERT

Recent Talks & Demos

Showing 1-1 of 1

Members-Only

Sign in to see who built these projects

Sign in View FAQ

Extreme Multi-Label ICL