AdvancedLesson 22 of 305 min read

Long Documents: Chunk, Map and Combine

Handle text too long for one go: cut it into chunks with overlap, run the same prompt on each, combine the results, and watch a real program lose a fact at a boundary and then find it again.

In this lesson you will

  • Cut a long text into overlapping chunks
  • Run the same prompt on every chunk
  • Combine the results from quotes, not memory
  • Check the seams for lost facts

About 24 minutes

A report too long for one go

Priya has a ninety-page annual report and needs a one-page summary for the trustees. Even if the whole report fits in the conversation, a very long input is harder to handle well, and in many setups it may not fit at all. So she splits it into parts, summarises each part, and then combines the summaries. This is the chunk-and-combine method, sometimes called map and reduce. It works well, and it has one classic way to fail quietly: a fact that is cut in half by the boundary between two chunks.

The method in four moves

  1. Chunk. Cut the text into parts that are comfortably small, at natural breaks such as headings where you can.
  2. Overlap. Let each chunk repeat the last few words of the one before it, so that anything near a boundary appears whole in at least one chunk.
  3. Map. Give each chunk the same prompt: for example, list every date, amount and decision, with the exact words.
  4. Reduce. Put the lists together, remove duplicates, and ask for the final summary from the combined list, not from memory.

Anthropic’s guidance for long material suggests putting the document first and the question after it, and asking Claude to pull out relevant quotes before answering. Both fit here: the map step is a quote-extraction step, and the reduce step works only from those quotes. Anything that never makes it into a map list cannot reach the summary, so the quality of the whole method depends on the boundaries.

Watch a fact fall through the gap

The program below takes a short made-up notice, cuts it into chunks of eight words, and uses a pattern to pick out the seven facts it contains: dates, times and amounts. It does this twice, once with no overlap and once with an overlap of three words. A pattern stands in for the map step, so the effect of the boundary is easy to see and count.

import re

text = ("The fair is on Saturday 17 June. Stalls open at 9 am. Each stall pays 12 pounds. "
        "Tea is served by the church group. The total budget is 400 pounds and it cannot be raised. "
        "Posters must be printed by Friday 9 June. Volunteers meet at 8 am. "
        "Parking is free at the school. The raffle closes at 3 pm.")
words = text.split()
FACT = re.compile(r"(?:Saturday|Friday) \d+ June|\d+ pounds|\d+ [ap]m")

def chunks(words, size, overlap):
    out, i = [], 0
    while i < len(words):
        out.append(words[i:i + size])
        if i + size >= len(words):
            break
        i += size - overlap
    return out

truth = set(FACT.findall(text))
print("facts in the whole text:", len(truth))
for size, overlap in [(8, 0), (8, 3)]:
    parts = chunks(words, size, overlap)
    found = set()
    for part in parts:
        found |= set(FACT.findall(" ".join(part)))
    sent = sum(len(p) for p in parts)
    print(f"size {size}, overlap {overlap}:")
    print(f"  chunks {len(parts)}, words sent {sent}")
    print(f"  found {len(found & truth)} of {len(truth)}")
    print("  missed:", sorted(truth - found) or "none")
facts in the whole text: 7
size 8, overlap 0:
  chunks 8, words sent 60
  found 6 of 7
  missed: ['12 pounds']
size 8, overlap 3:
  chunks 12, words sent 93
  found 7 of 7
  missed: none

Without overlap the program finds 6 of 7 facts and misses 12 pounds. The reason is plain once you see it: the number 12 ended one chunk and the word pounds began the next, so neither chunk contained the whole fact. With an overlap of three words every fact appears whole in some chunk, and the program finds 7 of 7. The price is more work: 60 words sent without overlap, and 93 with it. That is the trade. A little duplication costs a little more, and it buys back the facts that sit on the seams.

Checks that catch what chunking loses

  • Count the facts. If you can count the dates or amounts in the original, count them in the combined list.
  • Spot-check the seams. Pick two boundaries and read the text either side yourself.
  • Keep source labels. Ask each map step to say which chunk or page a fact came from, so you can trace it.
  • Ask for gaps. In the reduce step, say: if two chunks disagree, show both and do not choose.
Common mistake: too many rounds

Summarising summaries of summaries. Each round loses detail and can add small errors. Do one map round and one reduce round where you can, and go back to the quotes when something looks odd.

Use chunks only when you need them

Where the full document fits comfortably, giving it to Claude whole is often simpler, and then you can ask for quotes with page numbers. Chunking is for when it does not fit, or when you want a separate answer for each section.

Take a long text. Cut it in two places by hand. Pick a fact that sits near one cut and check it still appears whole in one chunk. If not, move the cut or add overlap.

Practice for this lesson

Check what you learned

Five quick questions. Pick an answer and you will see straight away whether it is right and why.

Why add overlap between chunks?

  1. To make chunks shorter
  2. So facts near a boundary appear whole in some chunk
  3. To avoid the map step
  4. To change the topic

Overlap repeats words at the seam so a cut fact is not lost.

In the program, what was missed with no overlap?

  1. The date of the fair
  2. Every time
  3. 12 pounds, cut across two chunks
  4. Nothing was missed

The number 12 ended one chunk and pounds began the next.

What is the map step?

  1. The same prompt run on each chunk
  2. Drawing a map
  3. Joining the lists
  4. Deleting the chunks

Map means giving every chunk the same task, such as listing facts with quotes.

What is the cost of overlap?

  1. Facts are lost
  2. The text changes
  3. More words are sent
  4. Nothing at all

The program sent more words with overlap than without.

Why avoid summaries of summaries?

  1. Each round loses detail and can add errors
  2. They are always longer
  3. They are not allowed
  4. They need no checking

Every extra round drifts further from the source.

Summary

  • Chunk, overlap, map, then reduce.
  • A fact cut in half by a boundary can be lost; overlap fixes this.
  • Combine from extracted quotes, not from memory.
  • Count facts and spot-check seams.

Next lesson

Lesson 23 builds evaluation sets to measure quality.

Questions

How big should a chunk be?

Small enough to handle well and big enough to keep context, such as a section or a few pages. Natural breaks such as headings are better than arbitrary cuts.

Do I always need chunking?

No. If the whole document fits comfortably, giving it whole is often simpler. Chunking helps when it does not fit, or when you want section by section answers.

Does the program use Claude?

No. A pattern stands in for the map step so you can see and count the boundary effect exactly. With Claude the map step would be a prompt.

Progress is saved in this browser.