sdadas / polish-roberta-base-8k

This is a small model trained using knowledge distillation from polish-roberta-8k. For distillation, we used a combination of several loss functions: KL divergence for the MLM head, as well as MSE and cosine loss to align the last-layer representations of the teacher and the s…