AWS and Booking.com shrink Llama-3.1 70B to 1.1B TinyLlama, 10-20x faster at matched quality

Amazon Web Services researchers and Booking.com combined knowledge distillation (KD) with the Medusa-2 framework to produce a 1.1-billion-parameter TinyLlama that matched the quality of a 70-billion-parameter Llama-3.1 teacher while running 10 to 20 times as fast, depending on the use case. The work was published on the Amazon Science blog by Daniel Zagyva and Laurens van der Maas.

According to the authors, the combined approach maintains the quality of the best open-source LLMs at less than one-third the cost while running up to 20 times as fast. They presented the techniques at last year's meeting of the Association for Computational Linguistics (ACL), where the work received the Industry Track best-paper award.

The two techniques

Knowledge distillation transfers knowledge from a large teacher model to a smaller student. The teacher is fine-tuned on labeled data, then generates predictions on unlabeled, domain-relevant text. The student trains on the combined human-labeled and machine-labeled data.

Medusa reduces inference latency by adding prediction modules (Medusa heads) to the base LLM, one per token decoded in parallel, so the model predicts multiple tokens at once instead of one at a time. The authors say Medusa typically delivers 2-4-fold speedups without compromising generation quality.

In the unified pipeline, Medusa heads are attached to the student during training, and the student learns from the combined corpus of teacher predictions and labeled data. The result is a single compact model with the teacher's knowledge and multitoken prediction.

Experiment design

The team used TinyLlama (1.1B parameters) as the student and Llama-3.1 (70B) as the teacher, running four progressive experiments across three Booking.com features:

• AI Trip Planner: a conversational assistant that must extract travel preferences from dialogue in under a second.
• Smart Filters: converts natural-language search queries into structured filters in real time.
• Accommodation Description Generation: produces property descriptions at the scale of millions of listings, where cost is the priority.

Results by stage

• Fine-tuning TinyLlama on task-specific data made it significantly more accurate than few-shot GPT-4o, but it still fell short of the teacher.
• Knowledge distillation closed the student-teacher gap by more than 90% and made inference roughly 6 times as fast, which the authors say is still not fast enough for real-time use.
• Adding Medusa heads sped inference up by a further 2 to 3.6 times.

The team tested two Medusa variants. Medusa-1 fine-tunes the base LLM first, then freezes its weights and trains the heads separately. Medusa-2 trains everything in a single stage. Both achieved comparable speedups with no meaningful quality loss, but the authors found Medusa-2's simpler pipeline more practical for production teams managing multiple models.

Production outcomes

Each use case got a dedicated model fine-tuned on domain data:

• AI Trip Planner: a 2.9% increase in user engagement versus GPT-3.5.
• Smart Filters: responses 15 times as fast for 99% of queries.
• Accommodation descriptions: 1.4% more "helpful" ratings from users, at less than one-third the cost of GPT-4.

The post does not disclose per-token pricing, context window sizes or benchmark scores for the models.

What this means

The results show that the two techniques stack: distillation supplied roughly 6 times the speed and most of the quality recovery, and Medusa heads added a further 2 to 3.6 times on top. For latency-bound features such as Smart Filters, the authors report a 15-times speedup for 99% of queries. For cost-bound workloads such as description generation, they report user ratings above those of a larger proprietary model at less than one-third the cost. The authors also favor Medusa-2 over Medusa-1 for production, because a single training stage is simpler to run across several dedicated models.