Mistral Large 4 VRAM Requirements

Open-weight releases come with a question the launch post rarely answers: what does it take to load the thing? For Mistral Large 4, the answer is about 1.06 TB of GPU memory for the official FP8 weights, 610 GB for the NVFP4 build that targets NVIDIA Blackwell, and 233 GB to 1.1 TB for the community GGUF quants, all before the KV cache.

That rules out any single GPU and any single 8x H200 node for the FP8 checkpoint. It does not rule out a 512 GB Mac Studio running a 2-bit quant. The numbers below are file sizes read from Hugging Face on October 30, 2026, the hardware Mistral and the vLLM team say they validated, and a KV cache estimate computed from the model's own config.json. One example runs through all of it: an internal coding agent that needs 128K tokens of context.

Why 49B Active Parameters Still Need 1 TB

Mistral Large 4, nicknamed Le Chonk, is a Mixture-of-Experts model with about 1 trillion total parameters and 49 billion active per token (52 billion counting embeddings and the output layer), according to its model card. Mistral released the weights on October 29, 2026, after a public preview earlier in the month. Each of its 72 layers except the first routes every token to 4 of 128 experts plus one shared expert.

The router picks experts per token, so any expert can be needed on the next step, and all 128 have to sit in memory. Active parameters set how much compute each token costs and how fast decoding runs. They do not shrink the memory bill. A 1T-parameter MoE needs memory like a 1T-parameter model and runs closer to the speed of a 49B one.

Every Checkpoint and What It Weighs

Mistral publishes two checkpoints, AMD publishes a third, and two community repositories publish GGUF files for llama.cpp. Sizes below are the sum of each repository's weight files as listed by the Hugging Face API, in decimal gigabytes. The last column is Unsloth's own measurement of how often a quant's most likely next token matches the full model's, averaged over seven test categories.

The AMD size comes from the vLLM recipe, not a file listing. GGUF users who want image input also need the 3.4 GB mmproj-F16.gguf vision projector. Note that the NVFP4 build is only about 5 GB larger than Unsloth's 4-bit GGUF, but it runs on vLLM and SGLang at data center speed, while the GGUF runs on llama.cpp.

What the KV Cache Adds

Mistral Large 4 uses multi-head latent attention (MLA), the design DeepSeek introduced, so it does not cache keys and values for each of its 64 heads. Each layer stores one compressed latent of 512 values plus 64 position values per token. Nine of the 72 layers reuse layer 35's cache instead of keeping their own, and a sparse attention indexer stores a 128-value key on 44 layers to pick the 2,048 most relevant earlier tokens.

vLLM, which the vLLM recipe says uses an FP8 latent cache by default, stores each MLA entry as 656 bytes: 512 FP8 values, four FP32 scales and 64 BF16 position values, per the layout documented in its FlashMLA sparse backend. Applying that to the config:

Python
Copy
import json, urllib.request

url = "https://huggingface.co/mistralai/Mistral-Large-4-1T-A52B/resolve/main/config.json"
t = json.load(urllib.request.urlopen(url))["text_config"]

own_kv = sum(1 for i, src in enumerate(t["kv_source_layers"]) if src == i)
own_idx = t["indexer_types"].count("full")
mla = t["kv_lora_rank"] + 16 + t["qk_rope_head_dim"] * 2   # 512 fp8 + 4 fp32 scales + 64 bf16
idx = t["index_head_dim"] + 4                              # 128 fp8 + 1 fp32 scale
per_token = own_kv * mla + own_idx * idx

print(f"{own_kv} of {t['num_hidden_layers']} layers keep their own KV, {own_idx} keep indexer keys")
print(f"{per_token:,} bytes per token")
for ctx in (32_768, 131_072, 262_144, 1_048_576):
    print(f"{ctx:>9,} tokens: {per_token * ctx / 1e9:5.1f} GB")

text
Copy
63 of 72 layers keep their own KV, 44 keep indexer keys
47,136 bytes per token
   32,768 tokens:   1.5 GB
  131,072 tokens:   6.2 GB
  262,144 tokens:  12.4 GB
1,048,576 tokens:  49.4 GB

So the coding agent's 128K-token context costs about 6.2 GB of cache, which is small next to a terabyte of weights. The estimate agrees with what vLLM measured. On 8 GB300 GPUs, the recipe reports room for 3.60 million cached tokens with NVFP4 weights and 2.38 million with FP8. The 1.22 million-token gap at 47,136 bytes each is 57.5 GB, within 3% of the 55.8 GB difference in per-GPU weights the recipe gives (about 76 GiB for NVFP4 versus 128 GiB for FP8). Counting a cache on every layer would miss by 24%.

There is one catch. With plain tensor parallelism, every GPU holds a full copy of the latent cache, because the latent is not split by attention head. vLLM's decode context parallelism (DCP8) shards it instead and raised the measured capacity from 3.60 million to 28.76 million tokens. The model card advertises a 1,048,576-token window, but the vLLM recipe validated up to 256K and caps --max-model-len at 262,144. llama.cpp keeps its cache in its own format, so treat the numbers above as vLLM numbers.

Which Data Center GPU Nodes Fit

The vLLM recipe lists what was actually validated: GB300 (one 4-GPU tray with NVFP4, or two trays with either checkpoint), 8x B300 with NVFP4 or FP8, 8x B200 with NVFP4, two 8x H200 nodes with FP8, and 8x AMD Instinct MI350X with the MXFP4 build. H100 is not on the list.

The H200 case shows why. FP8 needs about 128 GiB of weights per GPU across 8 GPUs, and an H200 shows 143,771 MiB in nvidia-smi, about 140 GiB (NVIDIA sells it as 141 GB). That leaves about 12 GiB per GPU for the KV cache, activations and CUDA graphs, and at the --gpu-memory-utilization 0.85 the recipe uses, vLLM budgets only about 119 GiB per GPU, less than the weights alone. So the recipe splits the model across 16 H200s (tensor parallel 16, or pipeline parallel 2 by tensor parallel 8). NVFP4 cuts the per-GPU weights to about 76 GiB, which is why a single 8x B200 node handles it, though Mistral's model card says the NVFP4 build targets Blackwell GPUs.

Software is the other constraint. The recipe states that no vLLM release or nightly wheel includes Mistral Large 4 support yet, so you serve it from the vllm/vllm-openai:mistral-large-4 Docker image, or SGLang's lmsysorg/sglang:mistral-large-4. This is the model card's command with the NVFP4 checkpoint and the agent's 128K context swapped in; we did not run it, since it needs one of the GPU nodes above:

Bash
Copy
docker run --runtime nvidia --gpus all \
  -v ~/.cache/huggingface:/root/.cache/huggingface \
  -p 8000:8000 --ipc=host \
  vllm/vllm-openai:mistral-large-4 \
  --model mistralai/Mistral-Large-4-1T-A52B-NVFP4 \
  --tensor-parallel-size 8 \
  --max-model-len 131072 \
  --tool-call-parser mistral --reasoning-parser mistral \
  --enable-auto-tool-choice --gpu-memory-utilization 0.85

Running It on a Mac Studio or a Workstation

Unsloth's guide gives total memory targets (RAM plus VRAM, or unified memory) per quant level: 256 to 270 GB for 1-bit, 310 to 390 GB for 2-bit, 435 to 500 GB for 3-bit, 630 GB for 4-bit and 1.2 TB for 8-bit. The rule behind the table is the useful part: total memory has to exceed the file size with room left for the operating system, the KV cache and runtime buffers.

The current Mac Studio tops out at an M5 Ultra with 256 GB or 512 GB of unified memory and 1.2 TB/s of bandwidth. A 512 GB machine fits UD-Q2_K_XL (378.3 GB, 95.6% top-1 agreement) with about 170 GB left over (Apple's 512 GB is 549.8 GB in the decimal units used here), and UD-IQ3_XXS (423.5 GB, 96.2%) more tightly. A 256 GB Mac Studio, or two 128 GB NVIDIA DGX Spark units, runs only the 233 GB UD-IQ1_S, and 90.6% top-1 agreement means roughly one token in ten differs from what the full model would pick. On a Linux workstation, llama.cpp splits the model between GPU memory and system RAM, so a pair of 48 GB cards plus 384 GB of RAM counts toward the same total. We have not measured tokens per second on any of these setups.

llama.cpp merged Mistral Large 4 support on October 29, 2026 in pull request #31376, and the first release containing it is build b11998. Unsloth's server command, which caps context at 32K to limit cache memory (not run here, as it downloads 378 GB):

Bash
Copy
./llama.cpp/build/bin/llama-server \
  -hf unsloth/Mistral-Large-4-1T-A52B-GGUF:UD-Q2_K_XL \
  --ctx-size 32768 --temp 1.0 --top-p 0.95 --top-k 0 --min-p 0.0 \
  --chat-template-kwargs '{"reasoning_effort":"high"}' \
  --host 127.0.0.1 --port 8080

For the 128K coding agent, raise --ctx-size to 131072 and watch memory. llama.cpp's cache format differs from vLLM's, but on a 512 GB Mac the 2-bit quant leaves more than 100 GB for it.

Or Pay per Token Instead

Mistral serves the same model as mistral-large-4 on its API at $0.70 per million input tokens, $0.07 per million cached input tokens and $2.10 per million output tokens, with a 1M context window, according to its model docs on October 30, 2026. For the coding agent, a turn that reuses a cached 120K-token prefix, adds 8K new tokens and writes 4K back costs $0.0084 + $0.0056 + $0.0084, about 2.2 cents, so a thousand of those turns comes to about $22.

The license matters if you plan to serve it to others. The Mistral Ecosystem Open-Weight License is a modified MIT license that allows commercial use, but anyone running a "Model as a Service" business, meaning third parties get API or endpoint access with real control over inputs, needs a seperate commercial license from Mistral. Internal use is exempt, and so are products that only embed the model inside specific features.

Which Setup to Pick

• Already running 8x B200 or B300: use the NVFP4 checkpoint in the vLLM image with tensor parallel 8. The 128K agent's 6.2 GB of cache per GPU is easy room.
• On H200s: plan for two nodes and the FP8 checkpoint, or use the API until NVFP4-class hardware frees up.
• A 512 GB Mac Studio: UD-Q2_K_XL on llama.cpp b11998 or later, and accept that you are running a 2-bit model.
• 256 GB of memory or less: UD-IQ1_S is possible, but at 90.6% top-1 agreement a smaller dense model at higher precision may serve you better.
• Building a product on it: read the Model as a Service clause before you design the architechture, then price the API against your own hardware.

Conclusion

Mistral Large 4 needs memory for all 1 trillion of its parameters even though only 49 billion work on any one token, and its compressed cache barely moves the total: the 128K coding agent adds about 6.2 GB per GPU. At data center speed, that means one Blackwell system running the NVFP4 checkpoint (a 4-GPU GB300 tray, or an 8x B200 or B300 node), or 8x AMD MI350X with the MXFP4 build. H200 owners need two nodes for FP8, and a 512 GB Mac Studio runs the 2-bit quant at a speed we have not measured.

For the coding agent, our pick is the API unless you already run that hardware. A thousand agent turns cost about $22, cheap enough to test the model on your real tasks while vLLM support is still Docker-only and before you know how the 2-bit build handles your code. If anyone outside your company will call your endpoint, read the Model as a Service clause first, since that use needs a separate license from Mistral.

Related DevToolLab Tools

• GGUF Viewer - open the first GGUF shard you download and confirm its quant types and tensor count from the header before pulling the rest of the 378 GB.
• Safetensors Viewer - check that the FP8 and NVFP4 shards carry the dtypes and block scales the model card describes.
• Prompt Caching Calculator - enter Mistral's $0.70, $0.07 and $2.10 rates in custom mode to see what cached prefixes save at your request volume.
• LLM VRAM Calculator - size the smaller dense models you might run instead; it uses the standard per-head KV formula, so use this post's numbers for Mistral Large 4.

Related Guides

• Qwen3.8 27B VRAM Requirements, Measured shows the same weights-plus-cache math on a model that fits one GPU.
• Local LLM Hardware: GPU vs Mac in 2026 compares unified memory with discrete VRAM for local inference.
• Best GPU Cloud Providers for AI Workloads in 2026 covers where to rent the B200 and H200 nodes this model needs.
• How to Run an LLM Locally on Mac or PC walks through llama.cpp and Ollama setup for smaller models.