kyutai / Sequential_Helium_6B

This repository houses the Helium 6B models, specifically designed to compare sequential pretraining on temporally ordered data against standard shuffled pretraining. This research aims to understand how the order of data affects a model's ability to retain facts and minimize …