On-Device LLM Long Docs: Chunking Past the Context Limit

This sentence appeared on the summary screen.

…이어서마침
표없이아주…

The chunk was cut in the middle of a syllable. I chose the wrong boundary when cutting a long note.

There was another problem at the same time. One 10-page document was split into 19 chunks. The assembled summary grew into 19 sections about the same topic.

Korean notes are not separated into paragraphs by blank lines

The first implementation looked for only two boundaries: two blank lines and sentence punctuation.

The punctuation list was ., !, ?, 。, ！, and ？, chosen for English and Japanese documents.

An actual Korean note looked like this.

회의 정리
다음주 화요일까지 시안 두 개
디자인 쪽에 미리 공유하기
예산은 아직 확정 아님

There are no blank lines. One newline separates items. There is no period either.

When it cannot find a boundary, the code simply cuts at the budget. That produced …이어서마침. A Korean syllable can be composed from several scalars, so cutting at a scalar boundary can also break a character.

I made five levels of boundary

// TextChunker.swift:107-109
/// 경계 우선순위: 문단(빈 줄) > 줄바꿈 > 문장 끝 > 낱말(공백) > 강제.
/// 각 등급에서 하한 위쪽의 **가장 뒤** 경계를 쓴다 — 예산을 최대한 채워야
/// 모델 호출 횟수와 지연이 줄기 때문이다.

I stop at the first boundary found from the top. If there is a paragraph boundary, I cut there; otherwise a line break, then a sentence end, then whitespace.

The last level is a forced cut.

// TextChunker.swift:144-148
if paragraph > 0 { return paragraph }
if line > 0 { return line }
if sentence > 0 { return sentence }
if word > 0 { return word }
return upper  // 공백조차 없는 연속 CJK — 이때만 스칼라 단위로 자른다.

Only a long, continuous block of Chinese or Korean with no spaces reaches this point. Otherwise it never cuts in the middle of a syllable.

I choose the latest boundary at each level

When finding a boundary, I search from the end rather than the beginning.

If the budget is 1,200 characters and paragraph boundaries occur at 300 and 1,100, I use the latter. Using the earlier one makes a 300-character chunk, which increases chunk count, which is model-call count.

// TextChunker.swift:125-142
var i = upper
while i > lower {
  let prev = scalars[i - 1]
  if prev == "\n" || prev == "\r" {
    if line < 0 { line = i }                       // 처음 만난 것이 가장 뒤입니다
    if paragraph < 0, hasBlankLineBreak(...) { paragraph = i }
  } else if sentence < 0, isSentenceEnd(...) {
    sentence = i
  } else if word < 0, isWhitespace(prev) {
    word = i
  }
  if paragraph >= 0 { break }   // 최고 등급을 찾았으면 더 볼 필요가 없습니다
  i -= 1
}

I fill the maximum position for each level in one backward scan. Because the scan comes from the end, the first point found at each level is the latest boundary.

I needed a lower bound to prevent tiny chunks

The reason for lower was a real bug.

// TextChunker.swift:67-70
/// 한 조각이 지나치게 짧아지는 걸 막는 하한 비율. 이게 없으면 문단 경계가
/// 맨 앞에 하나 있을 때 `"짧다."` 3글자가 독립 청크가 되어 모델 호출 한 번을
/// 통째로 낭비한다(구 코어의 실제 동작).
private static let minimumFillRatio = 2.0 / 5.0

Below 40% of the budget, I do not look for a boundary. Calling the model once for a three-character chunk is wasteful. One on-device call takes seconds.

I attach the final chunk to the previous one

The lower bound applies only when choosing a boundary. It does not apply to the final chunk, because the chunker returns the remaining text as-is.

A 31-character tail became an independent chunk. Two things followed: it consumed a model call, and because it contained only one sentence, the excerpt fallback was rejected and an empty section remained in the document.

// SummaryEngine.swift:424-433
static func mergingTail(_ chunks: [String], budget: Int) -> [String] {
  guard chunks.count >= 2, let last = chunks.last else { return chunks }
  let floor = budget * 2 / 5                       // 예산의 40% 미만이면
  guard last.unicodeScalars.count < floor else { return chunks }
  let joined = merged[merged.count - 1] + " " + last
  // 붙여서 예산을 크게 넘기면 그대로 둔다 — 컨텍스트를 넘기는 것이 더 나쁘다.
  guard joined.unicodeScalars.count <= budget * 13 / 10 else { return chunks }

The second guard is this function's safety guard. If joining would exceed 130% of the budget, I leave them separate. Exceeding the context is worse than one empty section.

The same 40% serves two different purposes. When choosing a boundary it means “do not cut here”; when examining a tail it means “this is not a chunk.”

Both reasons for the fixed 1,000 were stale

I hard-coded a chunk size of 1,000 scalars. I had two reasons for that value.

First, I assumed the model context limit was 4,096. Second, I assumed 1,000 scalars were about 2,700 to 2,800 tokens.

On 2026-08-13, I reran 131 documents on an iPhone 17 Pro and iOS 27. Both assumptions were wrong.

Value used as a basis	Measured value
Context limit 4,096	The window is 8,192 on iOS 26.4 and later
1,000 scalars ≈ 2,750 tokens	Korean is 0.627 tokens/scalar, so 1,000 scalars ≈ 627 tokens

4,096 was the fallback when the window could not be queried. SystemLanguageModel.contextSize returns 8,192.

The second error is larger. 2,750 and 627 differ by more than four times. The old number incorrectly applied prompt-template overhead as a multiplier of body length. I treated the fixed cost of instructions and schema as proportional to the body.

The two errors multiplied. The budget was roughly eight times too conservative.

The user saw the cost. A 10-page document split into 19 chunks and produced 19 sections about the same topic.

I now calculate from the window

// TextChunker.swift:39-44
public static func budget(forContextTokens contextTokens: Int) -> Int {
  let usable = Double(contextTokens) * windowShare - Double(reservedTokens)
  guard usable > 0 else { return minimumBudget }
  let windowCeiling = Int(usable / worstCaseTokensPerScalar)
  return max(minimumBudget, min(pageSizedBudget, windowCeiling))
}

Three values go in.

Value	What	Basis
windowShare 0.5	Use only half the window for source	Leave room for long instructions or expensive tokenization
reservedTokens	Response limit + 60	A margin over the measured 59 instruction tokens
worstCaseTokensPerScalar 0.64	Worst tokens per scalar	Korean prose 0.627, documents with tables 0.637

I use the worst value for a reason. An average calculation overflows the window on table-heavy documents, and then summarization fails.

I checked it by measurement. At budget 6,000, the worst prompt and response together were 4,119 tokens. It stopped around half of the 8,192 window.

Using everything the window allows makes the summary disappear

Once the formula looked right, I raised the budget to the maximum the window allowed. It produced 5,868.

I ran one 7-page PDF four times, changing only the budget.

Budget	Chunks	Bullets	Summary tokens
1,000 (first fixed value)	—	—	1,873
1,200	14	41	1,635
1,800	—	—	1,131
5,868 (window maximum)	2	6	346

Increasing the budget fivefold reduced the summary from 1,635 tokens to 346. Fourteen chunks and 41 bullets became two chunks and 6 bullets.

When one chunk contains three or four pages, the model folds those pages into one headline and three points. Since points has a maximum of three, more source content is discarded as chunks grow.

The comment says it plainly.

창은 넘지 말아야 할 선이지 채워야 할 목표가 아니다.

Chunk count is volume

Measuring three values on a real document made the relationship clear: 346 tokens at budget 5,868, 1,131 at 1,800, and 1,873 at 1,000.

Increasing the budget shortens the summary because chunk count falls and each chunk is capped at three points.

The user described the old result as “dense as it originally was.” That density became the reference.

// TextChunker.swift:46-52
/// 원문 한 쪽 남짓.
///
/// 값을 실물로 골랐다(2026-08-13, 7쪽·16쪽 PDF, 정리본 토큰 기준):
/// `5,868 → 346` · `1,800 → 1,131` · `1,000(옛값) → 1,873`. 조각 수가 곧 분량이다.
private static let pageSizedBudget = 1_200

I settled on 1,200. It is about one page of source. The summary's grain generally matches the document's pages and does not conflict with the screen saying “summarizing based on N pages.”

The window-calculated value remains only an upper bound over that value. On a device with a smaller window, it is reached first.

// TextChunker.swift:43
return max(minimumBudget, min(pageSizedBudget, windowCeiling))
//                        ^^^ 창이 허락해도 1,200을 넘지 않습니다

I put each chunk into the model once and assemble them in order

Once chunks exist, I call the model once per chunk. That is map.

Because each chunk is an independent call, the window always contains one chunk. It does not matter whether the document is 100 pages. What grows is time.

Input	Chunk count	Model calls
Short note	1	1
7-page PDF	14	14
30-page document	Capped	Up to the cap

There is a call limit. The note profile allows 64 calls per request and the document profile 256. Once the limit is exceeded, it stops with modelCallBudgetExceeded. Stopping and saying so is better than running forever.

A failed chunk keeps its place

When one chunk fails, I must not remove it from the result. A seven-page document would look like six pages.

So a failed chunk keeps its place. I try three times; if it still fails, I fill it with sentences extracted verbatim from the source; if that fails too, I leave an empty place marked as not summarized.

Chunk state	What enters the result
Model summarized it	Headline and points
Excerpt after three failures	Source sentences
Excerpt also failed	Empty place and marker

Saving these three states separately lets the screen say “This part could not be summarized.”

I summarize the chunks again to make a title

After chunk summaries finish, I need a title. I collect the chunk headlines and put them into the model once more. That is reduce.

Here I use a title-only schema rather than the chunk schema. Reusing the chunk schema makes the model create points too, creating values to discard where only a title is needed.

If there is only one chunk, I skip the title call. Its headline is the title.

Changing the terminology was also a fix

The unit shown to users in this pipeline was called a “page.” In reality it is a piece cut by character count.

When a 10-page PDF became 14 chunks, the screen said “summarizing 3 of 14 pages.” The document the user sees has 10 pages. The numbers do not match.

So I changed it to “chunk.” This is screen wording, not a code variable name.

At the same time I changed how the summary range is shown. I removed the page-count limit and describe the amount processed instead. If the cap is 30 pages and the document is 100 pages, saying “I read only 30 pages” tells the user the summary's range.