Token streaming

Token streaming delivers Large Language Model (LLM) output one token at a time: it eliminates user wait time, drastically boosting perceived application speed and user experience (UX).

Token streaming exposes the LLM's sequential generation process, sending text chunks to the client immediately instead of waiting for the full response. This technique is critical for modern AI applications: while a complex generation on a model like LLaMA 30B might take 10+ seconds, streaming ensures the first word appears in under a second, effectively solving the high-latency problem. Developers implement this by setting a stream parameter (e.g., OpenAI's `stream=true`) and often use Server-Sent Events (SSE) to push the token-by-token output to the frontend. The core value is UX: a snappy, type-writer effect replaces a long loading spinner, dramatically increasing engagement.

https://python.langchain.com/docs/expression_language/streaming/