hugpadhugpad

abacaj / llama-161M-100B

Text Generation162M14723Updated Jul 7, 2024
Hugging Face

Downloads

147

Likes

23

Last updated

Jul 7, 2024

License

apache-2.0

Trained on 100B tokens. - 1e-3 LR - 0.1 wd - WSD scheduler with 10% decay - 80% code, 10% NL, 10% instruction data - Dataset decontaminated against popular benchmarks following bigcode - 8x3090s 110~ hours

Tasks & library

  • text generation
  • transformers

Tags

  • transformers
  • onnx
  • llama
  • text-generation

Technical specifications

Parameters162M
Librarytransformers
LanguagesNot available
Licenseapache-2.0