hugpadhugpad

Locutusque / Hyperion-3.0-Mixtral-3x7B

Text Generation19B804Updated Mar 15, 2024
Hugging Face

Downloads

80

Likes

4

Last updated

Mar 15, 2024

License

apache-2.0

This is an experimental first attempt at creating a Mixture of Experts (MoE) language model by combining several Mistral expert models. The model uses the hyperion-3.0-beta architecture as the base, with a bfloat16 output dtype. The gating mechanism is set to hidden and two ex…

Tasks & library

  • text generation
  • transformers

Tags

  • transformers
  • mixtral
  • text-generation
  • moe
  • en

Technical specifications

Parameters19B
Librarytransformers
Languagesen
Licenseapache-2.0