AdvancedLesson 26 of 305 min read

Context, cost and speed

Understand context windows and tokens, trim and summarise conversations, keep a facts list, and use cost and speed levers sensibly.

In this lesson you will

  • Explain the context window and tokens
  • Trim and summarise long chats
  • Keep a facts list
  • Use cost and speed levers

About 18 minutes

The AI has a desk of limited size

Waleed’s customer assistant works well for the first few messages. By the thirtieth message of a long chat it forgets the order number the customer gave at the start, and it takes longer to answer. He did not break anything. He ran into the context window: the amount of text a model can consider at one time, including your instructions, the whole conversation so far, any pasted documents and the answer being written. Managing it well improves accuracy, speed and cost.

Good to know

Limits, speeds and prices differ by tool and change often. Check your tool’s current documentation. The numbers in this lesson are illustrative.

What uses up the window

Part Notes
System or standing instructions Sent every time, so keep them tight
The conversation so far Grows with every message
Pasted documents and examples Often the biggest part
The reply Counts too; very long replies use space

Tokens in plain words

Models read text in small pieces called tokens: parts of words, whole short words, punctuation. In English a rough rule is that one word is a little more than one token, but real counts depend on the tool and the language. Costs and limits are usually measured in tokens, so it helps to think in them even approximately.

Trimming a conversation

When a conversation grows past the budget, something must go. The simplest approach drops the oldest messages first. This program estimates token counts (very roughly) and trims a pretend chat to a budget. The sizes are illustrative; the logic and output are real:

def rough_tokens(text):
    # very rough rule of thumb for English: about 1.3 tokens per word (real counts differ by tool)
    return round(len(text.split()) * 1.3)

system = "You are a helpful shop assistant. " * 10
history = [
    ("user", "Hi, my order is late. " * 8),
    ("assistant", "I am sorry to hear that. " * 12),
    ("user", "It was order 4471, placed on Monday. " * 6),
    ("assistant", "Thank you, I am checking now. " * 10),
    ("user", "Also, can I change the address? " * 8),
]
BUDGET = 260

def total(msgs):
    return rough_tokens(system) + sum(rough_tokens(t) for _, t in msgs)

print("Before trimming:", total(history), "rough tokens")
kept = list(history)
while total(kept) > BUDGET and len(kept) > 1:
    removed = kept.pop(0)           # drop the oldest message first
    print("  dropped oldest message from:", removed[0])
print("After trimming: ", total(kept), "rough tokens, kept", len(kept), "messages")
Before trimming: 419 rough tokens
  dropped oldest message from: user
  dropped oldest message from: assistant
  dropped oldest message from: user
After trimming:  218 rough tokens, kept 2 messages

The risk of dropping the oldest messages is that important facts vanish, such as the order number. Better approaches keep what matters:

Better ways to manage context

  • Summarise old turns. Replace twenty old messages with a short summary of facts and decisions, and keep the latest messages in full.
  • Keep a facts list. Maintain a small block of confirmed facts (name, order number, preferences) that is always included.
  • Retrieve, do not paste. For big knowledge sources, find the relevant sections first and include only those.
  • Start fresh when needed. When a chat gets confused, begin a new one with the instructions and a short state summary.
  • Trim your instructions. Remove sentences that do not change behaviour.
Summarise this conversation so far in under 100 words for use at the start of a new chat. Include: confirmed facts (names, numbers, dates), decisions made, open questions and the next step. Do not include anything that was not stated.

Cost and speed levers

Lever Effect
Shorter prompts and examples Less to process, cheaper, faster
Shorter replies (set a limit) Faster, cheaper
Less irrelevant context Better accuracy as well as speed
Fewer retries Fix the prompt, not the loop
Smaller or faster model for easy tasks Often enough for simple labelling (tool options vary)

Position effects

Models can pay more attention to the start and end of long inputs than to the middle. Put the most important instructions first and repeat the key rule at the end, and place the facts that matter where they are easy to find, with clear headings or markers.

A budget habit

  1. Decide how much context a task really needs.
  2. Count roughly: instructions, examples, documents, expected reply.
  3. Remove anything that does not change the answer.
  4. Test whether answers get worse when you cut. Often they do not.
  5. Record the final size in your template notes.

Practice

Take a long prompt you use. Cut it in half without losing the rules, then run both versions on three inputs and score them. If scores hold, keep the short one. Next, take a long chat and write a state summary; start a new chat with it and see whether the new chat continues smoothly.

Practice answer

Most prompts contain sentences that never change the output, such as repeated politeness or duplicate rules. Removing them rarely hurts and often helps. If a cut causes a regression, restore only that sentence.

Common mistake

Pasting entire documents ‘just in case’. Extra text can distract the model, slow replies and raise costs. Include only what the question needs.

Check what you learned

Five quick questions. Pick an answer and you will see straight away whether it is right and why.

What is the context window?

  1. A browser tab
  2. A spreadsheet
  3. A password
  4. The text a model can consider at one time

It covers instructions, history, documents and reply.

What is a token?

  1. A small piece of text the model reads
  2. A coin
  3. A login
  4. A picture

Costs and limits use tokens.

What is better than dropping old messages blindly?

  1. Delete everything
  2. Add more messages
  3. Summarise them and keep a facts list
  4. Use bigger fonts

Keep what matters.

Why trim your instructions?

  1. To hide rules
  2. To avoid tests
  3. To lose facts
  4. Every sentence should change behaviour

Short prompts are cheaper, faster and often clearer.

Where do models often pay less attention?

  1. The first word only
  2. The middle of very long inputs
  3. Titles
  4. Numbers

Put key rules first and repeat them last.

Summary

  • Instructions, history, documents and replies share one window.
  • Summarise and keep a facts list instead of dropping blindly.
  • Include only what changes the answer.
  • Check current limits and costs in your tool’s documentation.

Next lesson

Next: prompting beyond text: images, code and tools.

Questions

How many tokens is a page of text?

It varies by language and tool. Use your tool's counter or documentation for real numbers.

Does a longer prompt always cost more?

In many paid tools, yes, since input and output are measured. Check your tool's pricing page.

Why does the AI forget earlier details?

Long chats can exceed the window or the model can overlook details. Restate key facts or start a new chat with a summary.

Progress is saved in this browser.