LangKit

LangKit is the open-source text metrics toolkit for monitoring Large Language Models (LLMs), providing comprehensive signal extraction for security, quality, and performance analytics.

LangKit provides essential observability and safety guardrails for LLMs deployed in production: It is an open-source library developed by WhyLabs, designed to extract critical telemetry data from both prompts and responses. This data enables real-time detection of high-risk issues, specifically targeting security vulnerabilities like jailbreaks, prompt injection, and sensitive data leakage (PII). Furthermore, LangKit assesses output quality by tracking metrics for toxicity, sentiment, and factual consistency to flag potential hallucinations. The toolkit integrates seamlessly with popular platforms, including OpenAI and Hugging Face, ensuring reliable and transparent LLM application governance.

https://whylabs.ai/docs/langkit/