Context Engineering: The Real Skill Behind Great LLM Apps

What Is Context Engineering, and Why Should You Care?

We've all been there: you spend hours perfecting a prompt, only to realize the real problem was the stuff surrounding it. That's the shift from prompt engineering to context engineering. Prompt engineering is about what you say to the model. Context engineering is about everything the model sees — system prompts, retrieved documents, conversation history, tool outputs, structured metadata, and the exact order of each piece within that finite context window.

Here's the thing: an LLM's output quality is fundamentally capped by its input quality. A beautifully crafted instruction paired with irrelevant retrieved documents will still produce garbage. But a so-so prompt paired with exactly the right context? That often produces excellent results. Once you internalize this, you start treating the context window as a first-class design surface — with the same rigor you'd apply to an API contract or a database schema.

The context window isn't a text box — it's an engineering artifact. Every token that goes in should earn its place through measurable impact on output quality.

In production, context is almost never static. It gets assembled on the fly from vector databases, knowledge graphs, user session state, tool call results, and cached summaries of prior conversations. The hard part is selecting, ordering, compressing, and formatting all these heterogeneous signals into a coherent input that maximizes your chances of getting a correct, grounded, useful response — all within a hard token limit.

Treat the Context Window as a Design Surface

Modern LLMs offer context windows ranging from 8K to over 1M tokens. It's tempting to treat all that space as a dumping ground — just concatenate every document that might be relevant and hope the model figures it out. In practice, this backfires in several well-documented ways.

• Attention dilution: As context grows, the model's attention spreads thinner. Critical information gets less weight, and the model starts missing things.
• The lost-in-the-middle effect: Research shows that LLMs attend most strongly to the beginning and end of the context. Stuff in the middle? Recall drops significantly.
• Latency and cost: Inference time and API costs scale with token count. Unnecessary context hits you twice — slower and more expensive.
• Conflicting signals: Redundant or contradictory documents confuse the model, leading to hedging, hallucination, or inconsistent outputs.

When you treat the context window as a design surface, you make deliberate choices about what goes in, what stays out, and where each element sits. The system prompt typically gets the top spot, followed by structured metadata, then retrieved context, and finally the user's query. Each section has a job, and the boundaries between them should be explicit — XML tags, markdown headers, or other structural markers the model can parse reliably.

Context Assembly Patterns

Context assembly — building the final input from raw materials — can follow several architectural patterns. Which one you pick depends on your latency requirements, how predictable user queries are, and how many different knowledge sources you're pulling from.

Static Context

This is the simplest pattern: your system prompt and supporting documents are fixed at deploy time. It works well for narrow, well-defined tasks — a customer support bot for a single product, or a code review assistant with a fixed style guide. Static context is fast, deterministic, and easy to version-control. The downside? It can't adapt to novel queries or evolving knowledge.

Dynamic Context

Dynamic assembly selects and formats context elements at inference time based on the user's query, session state, or other runtime signals. A typical setup routes the query through an intent classifier, then assembles a context template for that intent — pulling in the right tool schemas, few-shot examples, or domain-specific instructions. This pattern shines for multi-capability agents that need to handle diverse request types within a single deployment.

Retrieval-Based Context (RAG)

RAG is the most common dynamic pattern, and you've probably already used it. The user's query gets embedded into a vector space, nearest-neighbor documents are fetched from a vector store, reranked for relevance, and injected into the context. More advanced variants combine dense retrieval with sparse keyword search (hybrid search), use query decomposition for multi-hop questions, or let the model request additional context mid-generation through iterative retrieval.

Token Budget Management

Every context window has a hard token limit, and you need to manage that budget explicitly. One of the most common failure modes I see: retrieved documents eat up most of the available tokens, starving critical elements like system instructions or conversation history. The fix is straightforward — allocate a fixed or proportional share of your total token budget to each context component, then enforce those limits during assembly.

Here's a concrete example. Each component gets a maximum token share, and the assembly function truncates anything that exceeds its allocation.

from dataclasses import dataclass
from tiktoken import encoding_for_model

@dataclass
class TokenBudget:
    """Manages token allocation across context components."""
    model: str
    max_context: int
    # Reserve tokens for the model's response
    response_reserve: int = 4096

    def __post_init__(self):
        self.encoder = encoding_for_model(self.model)
        self.available = self.max_context - self.response_reserve

    def count(self, text: str) -> int:
        """Return the token count for a given text."""
        return len(self.encoder.encode(text))

    def allocate(self, components: dict[str, str],
                 priorities: dict[str, float]) -> dict[str, str]:
        """
        Allocate tokens to components based on priority weights.
        Higher-priority components are truncated last.
        Components: {"system": "...", "retrieved": "...", "history": "..."}
        Priorities: {"system": 1.0, "retrieved": 0.6, "history": 0.4}
        """
        # Calculate raw token counts
        counts = {k: self.count(v) for k, v in components.items()}
        total_needed = sum(counts.values())

        if total_needed <= self.available:
            return components  # Everything fits

        # Sort by priority — lowest priority truncated first
        sorted_keys = sorted(priorities, key=lambda k: priorities[k])
        remaining = self.available

        # First pass: calculate allocation per component
        allocations = {}
        for key in reversed(sorted_keys):
            # High-priority components claim what they need
            claim = min(counts.get(key, 0), remaining)
            allocations[key] = claim
            remaining -= claim

        # Truncate each component to its allocation
        result = {}
        for key, text in components.items():
            max_tokens = allocations.get(key, 0)
            tokens = self.encoder.encode(text)[:max_tokens]
            result[key] = self.encoder.decode(tokens)

        return result

In practice, you'll want to tune those priority weights empirically. System instructions almost always deserve the highest priority — they define the task and guardrails. Retrieved documents come next, then conversation history, which you can summarize or truncate more aggressively without killing quality.

Context Ordering and How Attention Actually Works

Where you place things in the context window matters more than most people realize. Multiple studies have confirmed the "lost-in-the-middle" phenomenon: when relevant information sits in the center of a long context, recall drops significantly compared to placing it at the beginning or end. This has real architectural implications.

1. Put system instructions and critical constraints at the very beginning. They benefit from the primacy effect and set the behavioral frame for everything that follows.
2. Put the user's query and any final instructions at the end. The recency effect means the model pays strong attention to the last tokens it sees before generating.
3. Place retrieved documents in the middle, but rank them by relevance — most relevant first. This partially offsets the lost-in-the-middle problem by front-loading what matters most.
4. Use structural delimiters (XML tags, markdown headers, separator tokens) to help the model tell context sections apart. This improves source attribution and reduces confusion.
5. When conversation history gets long, summarize older turns and keep only the most recent exchanges in full. You preserve recency while compressing the middle.

There's also context caching, now offered by several API providers. It lets you cache the prefix of your context — typically the system prompt and static instructions — and reuse it across requests. This cuts both latency and cost, but it also reinforces why you should place stable content at the beginning: that's what qualifies for caching.

Measuring Context Quality

You can't improve what you don't measure. Without quantitative evaluation, optimizing your context assembly is just guesswork. The metrics you need fall into two buckets: retrieval quality (did the right documents make it into the context?) and downstream task quality (did the model produce a correct, useful output given what it received?).

For retrieval quality, use the standard IR metrics — precision, recall, mean reciprocal rank (MRR), and normalized discounted cumulative gain (nDCG). Downstream quality is task-specific: accuracy for factual QA, ROUGE or BERTScore for summarization, pass rates for code generation, and human preference ratings for open-ended tasks.

Below is an evaluation harness that measures context quality in a RAG system. It compares retrieved documents against ground-truth relevant documents while also tracking end-to-end answer quality.

import numpy as np
from dataclasses import dataclass, field

@dataclass
class ContextEvalResult:
    """Stores evaluation metrics for a single query."""
    query: str
    precision: float = 0.0
    recall: float = 0.0
    mrr: float = 0.0
    context_utilization: float = 0.0
    answer_correct: bool = False

@dataclass
class ContextEvaluator:
    """Evaluates context quality for a retrieval-augmented system."""
    results: list[ContextEvalResult] = field(default_factory=list)

    def evaluate_retrieval(
        self,
        query: str,
        retrieved_ids: list[str],
        relevant_ids: set[str],
        total_context_tokens: int,
        used_context_tokens: int,
        answer_correct: bool,
    ) -> ContextEvalResult:
        """
        Compute retrieval and context quality metrics.
        retrieved_ids: ordered list of document IDs in the context
        relevant_ids: ground-truth set of relevant document IDs
        """
        # Precision: fraction of retrieved docs that are relevant
        relevant_retrieved = [d for d in retrieved_ids if d in relevant_ids]
        precision = len(relevant_retrieved) / len(retrieved_ids) if retrieved_ids else 0.0

        # Recall: fraction of relevant docs that were retrieved
        recall = len(relevant_retrieved) / len(relevant_ids) if relevant_ids else 0.0

        # Mean Reciprocal Rank: 1 / rank of first relevant result
        mrr = 0.0
        for rank, doc_id in enumerate(retrieved_ids, start=1):
            if doc_id in relevant_ids:
                mrr = 1.0 / rank
                break

        # Context utilization: how much of the budget carried useful info
        utilization = used_context_tokens / total_context_tokens if total_context_tokens else 0.0

        result = ContextEvalResult(
            query=query,
            precision=precision,
            recall=recall,
            mrr=mrr,
            context_utilization=utilization,
            answer_correct=answer_correct,
        )
        self.results.append(result)
        return result

    def summary(self) -> dict[str, float]:
        """Aggregate metrics across all evaluated queries."""
        n = len(self.results)
        if n == 0:
            return {}
        return {
            "mean_precision": np.mean([r.precision for r in self.results]),
            "mean_recall": np.mean([r.recall for r in self.results]),
            "mean_mrr": np.mean([r.mrr for r in self.results]),
            "mean_utilization": np.mean([r.context_utilization for r in self.results]),
            "accuracy": np.mean([r.answer_correct for r in self.results]),
        }

Pay special attention to the context utilization metric. It captures the ratio of tokens that actually contributed to the answer versus the total tokens consumed. If you have high recall but low utilization, you're wasting budget on irrelevant content — a clear signal that your retrieval or reranking stage needs work. Tracking this over time tells you whether your context assembly improvements are translating into real efficiency gains.

Key Takeaways

Context engineering is quickly becoming a core skill for teams building production LLM apps. As models get smarter, the bottleneck shifts from model intelligence to input quality — and input quality is an engineering problem with engineering solutions.

• Treat the context window as a design surface, not a dumping ground. Every token should justify its spot through measurable impact on output quality.
• Pick your context assembly pattern — static, dynamic, or retrieval-based — based on what your application actually needs. Consider hybrid approaches that combine stable instructions with adaptive retrieval.
• Implement explicit token budget management with priority-based allocation. Your system instructions and guardrails should never get sacrificed to make room for retrieved content.
• Respect how attention works: put critical information at the beginning and end, use structural delimiters, and summarize long conversation histories.
• Measure context quality rigorously with both retrieval metrics (precision, recall, MRR) and end-to-end task metrics. Track context utilization to spot wasted budget.
• Keep iterating. Context engineering isn't a one-time setup — it's an ongoing optimization loop driven by evaluation data and evolving model capabilities.

The field is still young and best practices are shifting fast. But the core insight is durable: what goes into the context window determines what comes out. Engineering that input with care, measurement, and rigor is one of the highest-leverage things you can do when building with LLMs.