Meta

AI · Open-Source Models · Foundation Models

Launch

Meta releases Llama 4 Scout, open-weights model with 1M-token context

A learned routing gate skips deep layers for simple tokens, letting the model match closed rivals while running on two gaming-grade GPUs.

• Meta's FAIR team released Llama 4 Scout, an open-weights foundation model built on a Mixture-of-Depths (MoD) architecture with a native 1,000,000-token context window.
• Scout carries 105 billion total parameters but activates only 24 billion per token, letting it run in 4-bit mode on 24GB of VRAM — a single RTX 4090 or 5090.
• A learned top-k routing gate decides how many layers each token traverses, skipping deep self-attention and feed-forward blocks for simple tokens and reserving full depth for complex ones.
• The model reportedly hit 99.8% retrieval accuracy on a needle-in-a-haystack test spanning the full million-token window across 2,500 document depths, and resolves 48.6% of SWE-bench Verified tasks.
• Meta says the dynamic depth routing cuts generation latency 58% versus dense 70-billion-parameter models even as total capacity grows to 105 billion parameters.
• Checkpoints are available on Hugging Face with FP8 and INT4 AWQ quantization, and the model is supported in vLLM, Ollama and LM Studio.
• By making reasoning depth itself a routable, learned decision rather than a fixed property of the architecture, Meta is pushing sparsity past simply choosing which experts fire — a shift that could keep narrowing the cost gap between open-weights models and API-gated closed systems.