LLM Summary Quality: Scoring Copy Rate and Repetition in Code

I asked for a summary and the model returned the source unchanged. Our check passed it.

The cause was ordering. We compared with the source after cleaning the model's answer and truncating its length. The unchanged source hit the length limit and was cut; because it was cut, we judged it different from the source.

The code, not the model, passed the check.

I compare with the source before cleaning

I changed the order.

// SummaryJudgment.swift:63-70
// **다듬기 전에 원문과 견준다.** 예전에는 sanitize·truncate를 거친 뒤 비교했고,
// 원문을 그대로 되돌려준 응답이 잘려 나간 덕에 그 문을 통과했다.
let asWritten = ([rawHeadline] + rawPoints).joined(separator: " ")
if copiesSource(asWritten, source: source),
  !preservesShortSingleLine(headline: rawHeadline, points: rawPoints, source: source)
{
  return .reject(.copiedSource)
}

These are rawHeadline and rawPoints, exactly what the model supplied, not cleaned values.

Normalization can defeat a check. If I clean the input and then inspect it, I am looking only after the inconvenient facts have disappeared.

If what I need to check is a property of the original, I must inspect the original.

The copy threshold needed to be 0.6, not 0.9

I had to decide what percentage of the source appearing unchanged counts as copying.

I first thought 0.9: nearly all of it must be the same. In practice I had to lower it.

// SummaryJudgment.swift:44-49
/// 원문의 이 비율 이상이 요약에 그대로 실렸으면 복사로 본다.
///
/// 0.9가 아니라 0.6인 이유: 앞부분만 요약하고 뒤에 원문을 이어 붙인 응답이 실제로
/// 관찰됐고(사용자 지적), 그 경우 복사율이 절반 남짓이었다. 정상 요약은 원문의
/// 20~40% 길이이므로 0.6에 닿지 않는다.
private static let copyCeiling = 0.6

The model produced a careful summary at the front and appended the source unchanged at the back. Half was summary and half was copy.

With 0.9, this answer passes because the copy rate is only about half.

0.6 is safe because normal summaries are 20 to 40% of the source. They do not reach 0.6, leaving room between the two.

Answer property	Copy rate	Decision
Normal summary	20~40%	Pass
Summary first, source afterward	About 50%	Reject
Source unchanged	100%	Reject

When choosing a threshold, I cannot look only at the bad side. I also need to see where the good answers are to draw the line between them.

Without counting unique sentences, normal summaries are rejected

I made another mistake in how I calculated copying.

At first I counted every sentence in the source and checked how many appeared in the summary. The problem is a source with the same sentence repeated.

// SummaryJudgment.swift:118-127
// **고유 문장으로 센다.** 같은 문장이 여러 번 나오는 원문(쪽마다 반복되는 머리글,
// 시세 알림처럼 형식이 같은 줄)에서 중복을 각각 세면, 요약이 그 문장 하나를 담기만
// 해도 복사율이 100%로 부풀려진다 — 정상 요약이 복사로 거부됐다.
var unique: [String] = []
var seen = Set<String>()
for sentence in sentences(in: source) {
  let key = compact(sentence)
  guard key.count >= 12, seen.insert(key).inserted else { continue }
  unique.append(key)
}

Consider a header repeated on every page of a PDF. A 30-page document contains the same line 30 times. If a summary includes it once, counting duplicates says all 30 were matched.

Lines with the same format, such as market alerts, have the same issue.

There is also key.count >= 12. Sentences shorter than 12 characters are not counted. Accidental overlap such as “Yes” or “Confirmed” should not be treated as copying.

A source with one sentence is different

A ratio is meaningless with fewer than two unique sentences.

// SummaryJudgment.swift:128-132
guard unique.count >= 2 else {
  // 문장이 하나뿐인 짧은 원문은 길이로만 판정한다.
  return Double(compactSummary.count) >= Double(compactSource.count) * copyCeiling
    && compactSummary.contains(compactSource)
}

I require both a length ratio and containment. It is copying only when both hold.

I make an exception for a short single line

A one-line note such as “Buy milk” has no room to compress. It is normal for the summary to resemble the source.

The exception has six conditions, all of which must hold.

Condition	Value
Source length	Between 24 and 240 characters
Line breaks	None
Point count	1 to 4
Headline	Nonempty and different from source
Headline length	Shorter than source
Joined points	Exactly equal to source

The final row defines the exception. It passes only when the model preserves the source exactly as a point and creates a separate short title. An answer that leaves the source unchanged and also uses it as the title is caught here.

Multiple lines do not get the exception. Tables and OCR fragments are filtered at line boundaries. An entire long copy over 240 characters still fails.

I fold a headline that equals the first point

The model sometimes creates a headline and writes the same sentence again as the first point. This appeared in the path that combines multiple chunks. I recorded it as DEF-047 with severity P2. The one-chunk path already folds it in the core, so the issue was narrowed to the range-combining path.

// SummaryJudgment.swift:74-78
// 모델이 헤드라인을 첫 요점으로 그대로 되풀이하는 일이 잦다(실측 DEF-047).
// 화면에서는 제목과 첫 줄이 같은 문장으로 겹쳐 보인다 — 요점 쪽을 접는다.
var points = dedupe(
  rawPoints.map(tidy).filter { !$0.isEmpty }.flatMap(readablePoints)
).filter { compact($0) != compact(headline) }

I fold it rather than reject it. The remaining points are useful, so there is no reason to discard the whole answer.

The important detail is comparing with compact(). Removing spaces and punctuation makes “Meeting schedule confirmed” and “Meeting schedule confirmed.” equal.

I discard an incomplete final point

The last value is cut at the token budget. Guided generation closes it syntactically, but the content ends in the middle.

// SummaryJudgment.swift:80-83
if let last = points.last, !isCompletePoint(last) {
  points.removeLast()
}
guard !points.isEmpty else { return .reject(.incomplete) }

I check only the last one because the last value is the one that gets cut.

If no points remain after discarding it, I reject the answer. There was one point and it was truncated.

The key set must match exactly

// SummaryJudgment.swift:58
guard Set(object.keys) == ["headline", "points"] else { return .reject(.wrongKeys) }

It is ==, not contains. I check not only that required keys exist, but also that there are no extras.

If the model returns {"headline": ..., "points": ..., "confidence": 0.95}, I reject it. An unrequested field means it left the schema, and I cannot trust the rest of that answer.

I split rejection reasons into eleven kinds

If judgment is only pass or fail, I cannot improve the prompt. I need to know why it failed.

// SummaryJudgment.swift:9-32
enum Reason: String, Sendable, Equatable {
  case notJSON                 // JSON이 아니다(코드펜스, 산문, 잘린 응답)
  case wrongKeys               // 키가 다르다(여분 키, 누락 키)
  case empty                   // 헤드라인이나 요점이 비었다
  case incomplete              // 마지막 요점만 완결되지 않아 안전하게 버렸다
  case overlong                // 헤드라인이 화면 제목으로 읽기에 너무 길다
  case copiedSource            // 원문이 그대로 실렸다
  case repeated                // 같은 글자·구문의 되풀이
  case wrongLanguage           // 대상 언어와 다른 필드를 발견했다
  case languageIndeterminate   // 언어를 확정할 수 없어 승격하지 않았다
  case translationFailed       // 번역 또는 번역 후 검증이 실패했다
  case modelFailed             // 모델 호출 자체가 실패했다
}

Each reason calls for a different response.

Reason	What must be fixed
notJSON	Whether schema enforcement works
wrongKeys	Schema definition
copiedSource	Compression instruction
overlong	Headline length guidance
repeated	Temperature or budget
wrongLanguage	Language specification in instructions

These reasons accumulate per chunk, including successful chunks. I need the record to distinguish success on the first attempt from success on the third.

Repetition judgment checks two patterns

It checks a line made of the same character four or more times, and a 1-to-4-word phrase repeated at least three times in one line. 1111111111 is the first; “Approval complete. Approval complete. Approval complete.” is the second.

Rule	Trigger	Catches
Same character	A line consists only of at least 4 copies of one character	1111111111
Same phrase	A 1~4-word phrase appears at least 3 times in one line	승인 완료. 승인 완료. 승인 완료.

Both rules operate per line. They do not catch repetitions spanning lines. If the model puts the same sentence into three separate points, this check passes and duplicate folding handles it.

Word comparison lowercases and splits on non-alphanumeric characters. Punctuation between words therefore still yields the same phrase.

Sentence boundaries use only five characters

To count copying, I split the source into sentences using ., !, ?, \n, and 。.

The problem from chunking appears here too. A Korean note without periods uses line breaks as boundaries. A long note with neither line breaks nor periods becomes one sentence; then the unique-sentence count is one and it falls to length and containment instead of a ratio.

Failing to make a title is not failure

Judgment for a title spanning chunks is slightly different.

// SummaryJudgment.swift:94
if object.isEmpty { return .accept(nil) }

An empty object passes and returns nil.

A title is useful but optional. With one chunk, its headline becomes the title. There is no reason to discard the entire summary because the model could not make a title.

I separated judgment from language checking

The boundary is written in the comment at the top of the file.

// SummaryJudgment.swift:5-6
/// 출력 언어는 여기서 모델 재시도의 이유로 쓰지 않는다. `SummaryLanguageAudit`가 각
/// 필드를 판정하고, 불일치만 번역한 뒤 다시 증명한다.

When the language is wrong, I do not call the model again. I translate it.

The reason is cost. Calling the model again rebuilds the summary from the beginning. Translation moves an already-made sentence and is much cheaper.

I check again after translation. I write the result only after proving the translator used the target language.

Problem	Response
Format, copying, repetition	Call the model again
Language mismatch	Translate and check again

I assign different actions to the two kinds of failure.