The Shorter-Prompt Myth: Cutting Words Is the Last Thing to Try

    November 20, 2026
    10 min read
    By Rowan Thaldenby
    Share this article

    Most teams attack a long system prompt the way an editor attacks a bloated essay: shorten sentences, delete adjectives, merge rules. After a week the prompt is 30 percent shorter, the bill is 8 percent lower and a support bot that used to refuse refund promises now makes them. The wording was never where most of the tokens, or most of the savings, were hiding.

    A system prompt is paid for on every single request. At 9,000 tokens and 150,000 requests a day, that is 1.35 billion input tokens daily before a user types anything. The cheap wins are structural: knowing which sections cost what, making the stable part cacheable, and loading rarely used knowledge only when it is needed. Rewording comes last, behind an evaluation set, because it is the step most likely to change behavior.

    Measure the prompt section by section before cutting

    You cannot reduce what you have not counted, and word count is a poor proxy. Tokenizers split code, URLs, numbers, JSON and non-English text very differently from plain English prose, so a 400-word block of example JSON can cost more than a 900-word block of policy.

    Count real tokens. OpenAI publishes the tiktoken library for its encodings, and Anthropic offers a token counting endpoint that returns the input token count for a full request, including system prompt and tools. Split your prompt into its logical sections (role, policies, product facts, formatting rules, few-shot examples, tool definitions) and count each one separately. A spreadsheet with one row per section and a column for tokens is enough.

    Then add a second column: how often does each section actually matter? Pull 200 to 500 recent conversations and tag which sections were relevant to the answer. A typical outcome looks like this: policies 1,800 tokens, relevant in 90 percent of turns; product catalogue 3,200 tokens, relevant in 12 percent; few-shot examples 2,400 tokens; tool schemas 1,600 tokens. The catalogue is now an obvious candidate to move out, and nobody had to argue about adjectives.

    Also read the usage fields returned with each response. Input tokens, output tokens and, where caching applies, cached input tokens tell you what you are paying for in production rather than in a playground.

    Make the prefix cacheable before making it shorter

    Prompt caching changes the arithmetic of long prompts more than any edit. Anthropic's prompt caching bills cache reads at 10 percent of the base input price, with cache writes at a premium over the base price. OpenAI applies caching automatically to prompts of 1,024 tokens or more and discounts cached input tokens, with the discount depending on the model. Exact prices change, so check the current pricing page, but the shape holds: a cached prefix is far cheaper than an uncached one.

    Caching only works on an identical prefix. That single rule explains most cache misses. A timestamp, a user name, a session id or a randomly ordered list near the top of the prompt makes every request unique from that point on, and the cache never hits. Reorder the prompt so that everything static comes first (role, policies, tools, examples) and everything dynamic comes last: today's date, the user's plan, retrieved documents, the conversation.

    Watch serialization too. If your code builds the tools list from a dictionary whose order can vary between deploys or workers, the JSON differs byte for byte, and so does the prefix.

    Illustrative numbers: an 8,000 token prompt, 100,000 requests a day, a base input price of 3 dollars per million tokens. Uncached, that is 2,400 dollars a day for the prompt alone. With a 90 percent cache hit rate at a 90 percent discount, the prompt costs roughly 460 to 520 dollars a day, depending on the cache write premium. Cutting the uncached prompt to 5,000 tokens would have saved only 900 dollars.

    The order of operations that keeps behavior intact

    Once you know the token map and the cache is working, reduce in an order that puts the safest changes first and the riskiest ones behind a test.

    1. Build an evaluation set before touching text: 50 to 200 real conversations covering normal requests, edge cases and the refusals the prompt exists to enforce. Record the current outputs and score them, with a rubric or a model-graded check.
    2. Delete dead sections: instructions for features that no longer exist, duplicated rules, a style guide that also lives in a second place. Run the evaluation set.
    3. Move low-frequency knowledge out of the prompt into retrieval or a tool the model calls when needed. Run the evaluation set.
    4. Trim few-shot examples: keep the two or three that demonstrate distinct behaviors and drop those that repeat a pattern. Run the evaluation set.
    5. Compress formatting: minify example JSON, shorten tool descriptions to what the model needs to choose the tool, remove decorative Markdown. Run the evaluation set.
    6. Only now reword instructions, one section at a time, rerunning the evaluations after each section and reverting any change that lowers the score.
    7. Ship behind a flag to a share of traffic, compare cost per conversation and quality metrics for a week, then roll out.

    The loop is slow by design. A prompt edit that saves 300 tokens and quietly breaks one refusal costs far more than 300 tokens in support tickets.

    What usually inflates a long prompt, and the mistakes made trimming it

    Long prompts grow by accretion. Each incident adds a rule, each new feature adds a paragraph, nobody removes anything, and after a year the prompt carries three versions of the same instruction. The fat tends to sit in predictable places: example dialogues pasted in full, product data that changes weekly, legal text copied verbatim, and tool descriptions written like documentation for humans.

    Teams trimming such a prompt make a recurring set of mistakes:

    • Rewording first, measuring never, so the edits land on cheap sections while a 3,000 token catalogue stays untouched.
    • Putting the current date or the user's name in the first line, which breaks the cached prefix for every request.
    • Deleting "obvious" rules that were added after a specific incident, without checking the incident log.
    • Replacing explicit instructions with vague summaries ("be helpful and safe") that the model interprets more loosely.
    • Removing all few-shot examples at once, when one or two were carrying the output format.
    • Testing on five handpicked prompts in a playground instead of a fixed evaluation set.
    • Moving facts to retrieval without checking retrieval quality, so the model now answers from a missing or wrong chunk.
    • Forgetting tool schemas entirely, though they are injected on every call.

    There is a reasonable counterargument to all this caution. For a low-traffic internal tool with 500 requests a day, the whole prompt may cost a few dollars, and an evaluation harness is not worth building. In that case, trim by judgment and move on. The process above pays off when volume or risk is high.

    Move knowledge out and load it only when needed

    The biggest structural reduction comes from accepting that a system prompt does not have to carry everything the model might ever need. It has to carry what applies to almost every turn.

    Knowledge that applies to a minority of turns belongs somewhere else. Product catalogues, pricing tables, help center articles and long policy documents can sit in a retrieval index and be inserted at the end of the prompt only when the user's question touches them. The cached prefix stays stable and short, and the variable part is paid for only when used.

    A second pattern is routing. A cheap classifier, or a small model, decides which kind of request this is (billing, technical, sales) and the request is sent with a prompt variant that includes only the relevant policy block. Each variant has its own cache. Three 4,000 token variants are cheaper and often clearer than one 9,000 token prompt that tells the model to ignore two thirds of itself.

    Tools can carry knowledge as well. Instead of describing 40 plan limits inline, expose a get_plan_limits tool that returns the limits for the user's plan. The model learns from a one-line description that the tool exists and fetches details on demand.

    Long context has its own quality cost. The 2023 paper "Lost in the Middle" by Liu and colleagues found that models used information at the start and end of long inputs better than information buried in the middle. A shorter, focused prompt is often not only cheaper but more reliably followed.

    An illustrative case and the checklist that came out of it

    This is an illustrative composite, not a specific company. A B2B SaaS support assistant runs on a 9,400 token system prompt and handles about 60,000 conversations a month, averaging 6 turns each, so 360,000 requests. Measured by section: role and tone 600 tokens, policies 2,100, product catalogue and limits 3,400, eleven few-shot dialogues 2,200, tool schemas for 18 tools 1,100. The date and the customer's company name sat in the first two lines, so caching never hit.

    The team moved the date and company name to the end, which brought the cache hit rate from near zero to about 85 percent within a day. They moved the catalogue into a tool, cut few-shot examples from 11 to 4 and shortened tool descriptions by about a third.

    The cached static prefix fell to 4,100 tokens. Wording changes to policies saved another 250 tokens, and one of them was reverted because it lowered the refusal score on the evaluation set from 47 of 50 to 43 of 50.

    Their before-shipping checklist now reads:

    • Token count per section recorded and compared with the previous version.
    • No dynamic value (date, name, id, retrieved text) before the last static section.
    • Tool list serialized in a fixed order.
    • Evaluation set run, scores equal to or above baseline on every category.
    • Every deleted rule checked against the incident log.
    • Cache hit rate and cost per conversation tracked for 7 days after release.

    Questions teams ask about trimming system prompts

    Do whitespace, Markdown and indentation really cost tokens? Yes. Runs of spaces, deep indentation in pasted code and wide Markdown tables all tokenize, and pretty-printed JSON examples can take 20 to 40 percent more tokens than the same JSON minified. Keep structure that helps the model, such as headings or tags around sections, and strip decoration that only helps a human reader.

    Are tool and function definitions part of the input bill? They are. Tool schemas are serialized into the model's input on every call, and providers that support tool use add a few hundred tokens of their own instructions on top. A product with 25 tools, each with a long description and nested parameters, can spend more on tool definitions than on the written system prompt.

    Does writing the prompt in English save tokens if users speak another language? Often it does, because most tokenizers encode English more compactly than languages such as Ukrainian, Greek or Hindi. Many teams write instructions in English and add one line telling the model to answer in the user's language. Test it: tone and terminology in the target language can drift.

    Can prompt caching change what the model answers? No. Caching reuses the computation for an identical prefix, so the output distribution is the same as without the cache. What changes is cost and time to first token. If answers differ between cached and uncached runs, the prefixes were not actually identical.

    If traffic is steady and the prompt is above the caching threshold, fix the prefix order first and expect most of the savings from that alone. If traffic is bursty, with gaps longer than the cache lifetime, caching helps less, and moving knowledge out becomes the main lever. Reword instructions only with an evaluation set in place, and only after both.

    Related Articles