DS-Archive / Mixtral-8x7B-Instruct-v0.1-LimaRP-ZLoss
Text Generation47B10818Updated Jan 7, 2024
Downloads
108
Likes
18
Last updated
Jan 7, 2024
License
apache-2.0
Experimental model, using a limarp qlora trained at 10k ctx length (greater than size of the longest limarp sample when tokenized via mistral's tokenizer) on mistralai/Mixtral-8x7B-v0.1 using Charles Goddard's ZLoss and Megablocks-based fork of transformers, and then fused to …
Tasks & library
- text generation
- transformers
Tags
- transformers
- mixtral
- text-generation
- conversational
- en
Technical specifications
Parameters47B
Librarytransformers
Languagesen
Licenseapache-2.0