AWS and Booking.com shrink Llama-3.1 70B to 1.1B TinyLlama, 10-20x faster at matched quality
Amazon Web Services researchers working with Booking.com combined knowledge distillation and the Medusa-2 framework to train a 1.1-billion-parameter TinyLlama that matched the quality of a 70-billion-parameter Llama-3.1 teacher. According to the authors, it ran 10 to 20 times as fast depending on the use case.