DiscoLM Mixtral

DiscoLM Mixtral is an experimental 8x7b Mixture-of-Experts (MoE) model, a DiscoResearch fine-tune of Mistral AI's Mixtral 8x7b.

This is the **DiscoLM Mixtral 8x7b alpha**, a high-performance, experimental MoE model built on Mistral AI's Mixtral 8x7b architecture. Developed by DiscoResearch's Björn Plüster, the model uses a sparse architecture: 46.7 billion total parameters, with only 12.9 billion active per token for efficiency. It was fine-tuned on specialized datasets, including Synthia, MethaMathQA, and Capybara, with compute resources provided by HessianAI. The project delivers a strong, open-weight language model for advanced text generation and NLP research.

https://huggingface.co/DiscoResearch/DiscoLM-mixtral-8x7b-v2