multimodal API

The Multimodal API unifies diverse data streams (text, image, audio, video) into a single model, enabling advanced reasoning and cross-modal content generation.

This API is your single-point interface for complex AI tasks: it processes multiple data modalities simultaneously. It moves beyond text-only, integrating inputs like images, video, and audio to enable deeper context and more robust outputs. For example, the Gemini API allows you to upload an image and a text prompt to extract text, convert it to JSON, and answer questions about the content. This capability is critical for use cases like smart search, real-time video summarization, and advanced customer support systems. It supports both standard REST and real-time WebSocket streaming (BidiGenerateContent) for low-latency, interactive applications.

https://ai.google.dev/gemini-api/