Context Engineering in 2026: Developer Guide for LLM Applications

In June 2025, Andrej Karpathy wrote something that shifted how the developer community talks about working with LLMs: "Context engineering is the delicate art and science of filling the context window with just the right information for the next step." He was endorsing the term over "prompt engineering" - arguing that prompts are a small slice of the real problem.

A year on, that framing has stuck. If you are building any serious LLM application in 2026 - a coding agent, a customer support bot, a document Q&A system - context engineering is the discipline you need to understand before you write another line of model code.

What is Context Engineering

Context engineering is the practice of deliberately designing, selecting, and managing all the information that flows into an LLM's context window at inference time. That includes your system prompt, the user's message, retrieved documents, memory from previous sessions, outputs from tool calls, and any structured data you inject.

Prompt engineering asks: how do I word this question? Context engineering asks: what does the model need to know to answer correctly? In production systems, that second question is much harder. You are dealing with:

• Conversation history that grows across turns until it overflows the context window
• Retrieved documents from vector search that may be partially relevant or stale
• Tool outputs from function calls that arrive as verbose, deeply nested JSON
• Multi-agent handoffs where one agent's full reasoning trace gets passed to the next
• User memory that needs to persist across sessions without bloating current context

None of these are solved by rewording a system prompt. They require engineering.

The 4 Core Context Engineering Strategies

LangChain's context engineering documentation describes four strategies. They apply regardless of framework.

Write - Persist information outside the context window and retrieve it when needed. Save conversation summaries to a database, write intermediate reasoning to a file, store user preferences in a memory store. Avoid keeping everything in-window at once.

Select - Pull only the relevant subset into the window. Instead of embedding an entire knowledge base, retrieve the 3-5 most relevant chunks for the current query. Apply the same idea to memory: retrieve only memories that match the current task.

Compress - Reduce token count of context you do need. Summarize long histories, extract key facts from verbose tool outputs, filter retrieved chunks through a re-ranker before final inclusion.

Isolate - Run different subtasks in separate context windows. One agent handles web search, another handles code execution, a third synthesizes results. Each sees only the context relevant to its task.

LLM Token Budget Management

Treat your context window as a budget. Every token you add costs money and can hurt reasoning quality. The question is not "can I fit this in?" but "should I include this at all?"

A practical allocation for a 128k context window:

Trigger context compaction when the window reaches 70-80% capacity. At that threshold, summarize conversation history, drop lower-relevance chunks, and compress tool outputs. Wait too long and you hit the limit mid-reasoning, which produces degraded or truncated responses.

Python
Copy
import tiktoken

def get_token_count(messages: list[dict], model: str = "gpt-4o") -> int:
    enc = tiktoken.encoding_for_model(model)
    total = 0
    for msg in messages:
        total += 4  # per-message overhead
        for value in msg.values():
            total += len(enc.encode(str(value)))
    return total

def should_compact(messages: list[dict], max_tokens: int = 128_000, threshold: float = 0.75) -> bool:
    return (get_token_count(messages) / max_tokens) >= threshold

if should_compact(conversation_history):
    conversation_history = summarize_and_trim(conversation_history)

For Claude models, use the Anthropic SDK's built-in count_tokens method instead of tiktoken. For quick in-browser estimates, the DevToolLab AI Token Counter supports GPT-4o, Claude 4, and Gemini without an API key.

The Primacy and Recency Problem

LLMs do not pay equal attention to all parts of the context window. The "lost in the middle" problem is well documented: accuracy drops by over 30% when relevant information sits in the middle of a long context compared to the beginning or end.

The practical layout rule:

• Start of context - system prompt and non-negotiable instructions
• Middle - retrieved documents, historical memory, tool outputs
• End of context - the current user message and any facts the model must not miss

Never bury a hard constraint in a document pile. If the rule is "do not modify the database schema," that belongs in the first 200 tokens of the system prompt, not buried in message 15 of a long conversation history.

Compressing Tool Outputs

Tool calls are the most common source of uncontrolled context growth in agentic systems. A single database query can return thousands of tokens. The LLM rarely needs all of them.

Run a lightweight extraction step on every tool output before it re-enters the context:

Python
Copy
from anthropic import Anthropic

client = Anthropic()

def compress_tool_output(raw_output: str, task_description: str) -> str:
    response = client.messages.create(
        model="claude-haiku-4-5-20251001",  # cheap model for compression
        max_tokens=500,
        messages=[{
            "role": "user",
            "content": (
                f"Task: {task_description}\n\n"
                f"Raw tool output:\n{raw_output}\n\n"
                "Extract only the fields relevant to the task. "
                "Return compact JSON, no extra formatting."
            )
        }]
    )
    return response.content[0].text

raw_result = call_my_tool(args)
compressed = compress_tool_output(
    raw_output=json.dumps(raw_result, indent=2),
    task_description="Check whether the user's subscription is active"
)
conversation_history.append({"role": "tool", "content": compressed})

A typical verbose API response compresses 80-99% once you extract only relevant fields. Use the DevToolLab JSON Formatter to inspect raw payloads and identify which fields to keep before writing the compression prompt.

RAG Context Engineering

RAG is the most-used context engineering technique, but the tutorial defaults are not production-ready.

Chunk size: 500-1,000 tokens per chunk with 10-15% overlap is a common starting point. Too small and you split coherent information across chunks. Too large and one partially-relevant chunk consumes a large share of your budget.

Two-stage retrieval: Vector similarity search retrieves semantically similar documents, but semantic similarity does not always mean task relevance. A re-ranker scores each retrieved chunk against the current query and keeps only the top results - cutting included chunks by 50-70% while improving answer quality.

Python
Copy
from sentence_transformers import CrossEncoder

reranker = CrossEncoder("cross-encoder/ms-marco-MiniLM-L-6-v2")

def rerank_chunks(query: str, chunks: list[str], top_k: int = 5) -> list[str]:
    pairs = [(query, chunk) for chunk in chunks]
    scores = reranker.predict(pairs)
    ranked = sorted(zip(scores, chunks), key=lambda x: x[0], reverse=True)
    return [chunk for _, chunk in ranked[:top_k]]

# Retrieve broadly, re-rank tightly
initial_results = vector_store.similarity_search(query, k=20)
final_context = rerank_chunks(query, initial_results, top_k=5)

Cross-Session Memory

At the end of each session, generate a structured summary and store it. At the start of the next relevant session, retrieve only the memories that match the current task.

Python
Copy
def summarize_session(messages: list[dict]) -> dict:
    transcript = "\n".join(
        f"{m['role']}: {m['content']}"
        for m in messages if m["role"] != "system"
    )
    response = client.messages.create(
        model="claude-haiku-4-5-20251001",
        max_tokens=400,
        messages=[{
            "role": "user",
            "content": (
                f"Session transcript:\n{transcript}\n\n"
                "Extract in JSON: preferences, decisions, open_items, facts."
            )
        }]
    )
    return json.loads(response.content[0].text)

For production, Mem0 handles storage, retrieval, and deduplication automatically.

Common Mistakes

Injecting entire files - Pasting a 1,000-line codebase when the agent needs two functions. Retrieve the specific functions relevant to the current task.

Never compressing tool outputs - After 10 tool calls in an agentic loop, the context is full of verbose payloads the model no longer needs.

Ignoring primacy and recency - Critical constraints in the middle of a large document block get underweighted reliably.

One context window for everything - Long agentic tasks accumulate intermediate reasoning and partial results. Isolate subtasks into separate agents with clean context.

Treating context as free - Every token costs money and can degrade reasoning quality. The context window is infrastructure with a real budget.

Pre-Ship Checklist

Before shipping any LLM-powered feature:

• System prompt is under 300 words; critical constraints appear in the first 200 tokens
• Retrieved documents pass through a re-ranker; no more than 5 chunks per query
• Tool outputs are compressed before being added to context
• Context compaction triggers at 70-80% of window capacity
• User's current message appears at the end of the context
• Session memory is stored as structured summaries, not raw transcripts
• Token counts are logged at each step during development

Conclusion

Context engineering is not new - developers have been managing LLM inputs since GPT-3. What changed is the term gave the practice a systematic framework. The four strategies - write, select, compress, isolate - cover the majority of production context problems.

If you implement one thing first, make it context compaction at 70-80% window capacity. That single change delivers more reliability and lower costs than any other optimization in a typical LLM application. Work through the checklist before the next feature ships.

Related DevToolLab Tools

These tools are directly useful when building and debugging LLM context pipelines.

• AI Token Counter - Count tokens for GPT-4o, Claude 4, and Gemini before sending a request. Essential for checking whether your assembled context fits within budget before paying for an oversized API call.
• JSON Formatter - Inspect and format raw tool output payloads during development. Makes it immediately clear which fields are bloated and can be stripped from the compression prompt.
• Text Size Calculator - Calculate exact byte size and rough token estimate for any text payload. Useful for sizing RAG chunks and confirming targets match what lands in the context window.
• JSON Schema Validator - Validate structured output schemas you inject to constrain model responses. Catches definition errors before they cause parsing failures in downstream agents.
• JSON Flattener - Flatten deeply nested JSON from API tool results into a flat key-value structure. Flat JSON is more token-efficient and easier for LLMs to reason over.