iOS On-Device OCR: Removing the Server with Vision and PDFKit

While checking the consent basis for enabling server OCR by default, I read section 8 of the privacy policy.

"OCR(이미지·문서 내 텍스트 인식)은 기기 내에서 처리되며 원본 이미지를 서버로 전송하지 않음"

("OCR — recognizing text inside images and documents — is processed on the device, and the original image is not sent to a server.")

It directly contradicted the facts. The app had no on-device OCR path at that point. There were zero import Vision statements. It was sending attachment originals as multipart data to ocr.example.com.

It started on device, then moved to the server for performance

I was reading image text with Apple's Vision framework. It ran on the device and needed no network.

Quality was the problem. Recognition was poor for Korean documents, especially scans.

Moving to a server allowed a larger model. I put a gateway at ocr.example.com and called the DeepSeek OCR API.

During the move I consolidated the app's photo OCR paths. Five paths were each calling Vision.

Path	What
Compose	Attach a photo while writing a note
Home capture	Take a picture from Home
Add detail photo	Attach a photo to a saved note
Add file image	Import an image from Files
Live seed	Inject a sample for tests

I put all five behind OCRGateway.index. They share the login session, 25MB upload limit, and server usage limit with document OCR.

I then deleted the Vision implementation and tests and the Xcode project references. Keeping both paths would make it impossible to know which one runs.

The server was not better in every format

After the move I measured the server engine by format. I counted error as CER, the character error rate; closer to 0 is more accurate.

Input	CER	What happened
Multi-page PPTX	0.073	Normal
Multi-sheet XLSX	0.317	Dates and product names were cut; column boundaries collapsed
hwpx approval table with many empty cells	3.12	The model entered an infinite repetition loop

The last row is a complete failure. CER above 1 means there are more errors than source characters. When empty cells continued through a Korean approval table, the model emitted the same character forever.

The XLSX problem was not the model. The render stage did not fit column widths, so text was cut. After adding an auto-fit preprocessing step, CER fell from 0.172 to 0.0055 under the same normalization.

I tried another engine. The repetition loop disappeared and table CER improved, but it was 20 to 40 times slower: 15 to 40 seconds per page. Its license also imposed a revenue cap.

The premise that a larger model improves quality differed by format.

Deleting it broke the build twice

Deleting the Swift file was not enough.

The first failure was an Xcode project reference. The file was gone, but build, file, and group references remained in .pbxproj and broke the build. All three had to be removed together.

The second failure was a function signature. I had missed adding row-array and index parameters to OCRGateway.index, causing a compile error.

I verified with seven AttachmentClassifierTests and Go tests on the server. TestImageUploadBypassesNormalizer confirmed that PNG goes directly to the OCR upstream without the normalizer.

Trying to enable the default revealed a false policy

Server OCR shipped disabled. Users had to enable it in Settings.

Few people enabled it, so I tried to turn the default ON. Since the feature sends original images to a server, I first had to check the consent basis.

That is where I read section 8. The policy said processing happened on the device, while the code sent it to the server.

I searched for import Vision. There were zero results. The on-device path did not exist.

This is how the order diverged

Moment	Code	Policy
On-device OCR period	Process on device with Vision	Process on device
After moving to server	Send originals to the OCR gateway	Process on device

I moved the code without changing the policy. Once a policy is written, nobody reads it. There was no mechanism to change it when code changed.

I treated correcting the policy and enabling the default as one task. I deployed the correction to justsend.cloud/legal/privacy, then enabled the default.

Three silent failures also became visible

The same task put three previously invisible failures on screen:

• OCR was off, so text could not be read.
• The attachment was excluded from processing.
• AI summarization was unavailable.

All three silently did nothing. Users believed the app had read the content and moved on.

Not knowing that a feature is absent is worse than having no feature.

I returned to on-device processing

Between changing the policy to match code and changing code to match the policy, I chose the latter.

I removed the server OCR path and returned to extracting on the device. This time I combined three approaches rather than using only Vision.

Input	How it is read
Photos (JPEG, PNG, etc.)	Open with ImageIO and recognize text with Vision
PDF with a text layer	Extract text directly with PDFKit
Scanned PDF (no text layer)	Render pages as images, then Vision
Document files (docx, pptx, etc.)	JustSendDocKit parser
Plain text	Read as UTF-8

I split PDFs into two paths. With a text layer there is no reason to render. Extracting it is accurate and fast. Only empty pages fall through to Vision.

I put the five paths behind one AttachmentTextExtractor boundary. Home capture, detail attachments, the share-extension return path, and test samples use the same boundary.

I did not disable it; I deleted it

I did not leave server OCR behind a setting.

The plan says this.

서버 OCR을 단순 비활성화하지 않고 OCRClient와 token/consent/entitlement
배선을 clean cutover로 제거한다.

I deleted four things:

• OCRClient and gateway call code
• The serverOCREnabled runtime setting
• The consent screen and its text
• The StoreKit connection treating server OCR as a paid benefit

I cleaned the database too. Migration v52 removes the server-OCR consent column.

If I only disable it, that code remains alive. Nobody knows under what condition it might be enabled again. For the policy to say “processed on device,” there must be no code capable of sending it to a server.

I tested with the network blocked at the source

The problem was how to prove “it does not send anything to a server.” Reading code and saying there is no call was not enough.

I blocked the entire network in tests. With a global URLProtocol block in place, I ran OCR tests against real files.

If code tries to use the network, the test fails. Passing means there was no call.

I created failing tests first

I wrote six verification items and first secured a failing state for each.

What to verify	How
Preserve original bytes	Stored file equals input
Page order	Page order of a 3-page PDF
Zero network calls	Pass with URLProtocol blocked
Same result for same input	Run twice and compare
Search after reopening app	Find text after reopening DB
Share-extension retry while waiting	Failed item is processed again

I made fixtures from real files: JPEG and PNG containing Korean and English, and a 3-page scanned PDF with no text layer.

Synthetic data cannot verify OCR. Lighting and tilt in a real photo determine recognition rate.

The final result was 424 JustSendTests executed, 7 skipped, and 0 failures. The 7 skipped tests depend on the environment.

A fixture with a text layer makes verification invalid

There was a trap in the OCR integration test.

// app/JustSend/Tests/OCRDeviceIntegrationTests.swift:57-58
.isEmpty
}, "The PDF fixture must be image-only so PDFKit cannot satisfy the OCR test")

A PDF can contain a text layer. PDFKit simply extracts the text from such a PDF, so OCR does not run.

With that file, the test passes even if OCR is broken, because text came out.

The test therefore first checks that the fixture is image-only. I put the reason in the failure message. If someone changes the fixture later, this assertion catches it first.

On-device processing introduced limits

On the server, a large file could simply be sent. On the device, memory and time come from the user's device.

So I fixed upper bounds.

// app/JustSend/Sources/OCR/AttachmentTextExtractor.swift:23-29
maxImagePixels = 4_096      // 이미지 썸네일 한 변의 상한
maxPDFRenderPages = 64      // OCR 대상 쪽 상한
maxPDFRenderPixels = 2_048  // PDF 쪽을 렌더할 때 한 변의 상한

maxPDFRenderPages 64 is the limit visible to the user. A 100-page scanned PDF reads only through page 64.

The screen must say that. If the range is silent, the user believes the summary covers the entire document.

I judge by extension, not MIME

There is another rule for deciding how to read an attachment.

// JustSendMemoryCore/.../AttachmentReadPlan.swift:57-95
// 확장자를 lowercase해 PDF·이미지·평문·파싱 문서·미리보기·보관 전용 순으로
// 판정하며 MIME은 보지 않는다.

Not looking at MIME is a choice. MIME from a share sheet or Files is not reliable. The same file arrives with different types depending on its source.

The extension is what the user sees, so the decision does not conflict with the screen.

Line-based recognition broke tables

After returning on device, I had to handle recognition quality directly. Tables were the biggest problem.

// app/JustSend/Sources/Meaning/PageSources.swift:49-53
/// 줄 단위 인식(`VNRecognizeTextRequest`)은 표를 만나면 셀 하나를 한 줄로
/// 떨어뜨린다. 급여명세서가 `발급 / 일 / 2025 / . 11. 11 / 회사명`처럼 흩어져
/// 나왔고, 그 상태로는 어떤 요약도 정확할 수 없다 — 입력이 이미 망가져 있다.
/// iOS 26의 `RecognizeDocumentsRequest`는 표를 행·열로 돌려주므로 표를 표로

발급 and 일 were separated, as were 2025 and . 11. 11. One payslip line became five fragments.

A summary cannot be accurate in that state. The model is not the problem; the input is already broken.

iOS 26's RecognizeDocumentsRequest returns tables as rows and columns. Keeping a table as a table preserves neighboring-cell relationships.

Before trying to improve summary quality on the model side, I need to inspect the input. Here changing the recognition API mattered more than changing the prompt.

I fixed recognition to two languages

// app/JustSend/Sources/Meaning/PageSources.swift:69-73
var options = request.textRecognitionOptions
options.recognitionLanguages = [
  Locale.Language(identifier: "ko-KR"),
  Locale.Language(identifier: "en-US"),
]

I include only Korean and English. A language not in the list is not recognized.

Adding languages increases candidates, false recognition, and processing time. It is more accurate to include only languages that actually arrive in documents.

The line-based fallback uses the same values. If the two paths use different language lists, results differ depending on which path runs.

Each format follows a different path

An attachment may be a PDF or an hwp. How to read it differs by format.

// JustSendMemoryCore/Sources/JustSendMemoryCore/AttachmentReadPlan.swift:6-10
/// - PDF는 `PDFKit`으로 **쪽마다** 골라 그릴 수 있다. 다중 페이지 OCR의 유일한 경로다.
/// - `QLThumbnailGenerator`는 페이지 지정이 없어 **대표 이미지 한 장**만 준다.
///   doc·xls는 내용을 그려 주지만, hwp는 파일 아이콘만 준다.
/// - 파서가 있는 포맷은 글자가 이미 있으므로 그릴 이유가 없다.

There are 12 parser extensions: docx, xlsx, pptx, hwpx, odt, ods, odp, epub, pages, numbers, key, and rtf. The list appears in two places because the core does not import JustSendDocKit. Contract tests instead check that the lists do not diverge.

Format	Path	Limit
PDF	Render each page with PDFKit	The only path for multi-page OCR
12 parsed formats	JustSendDocKit	Text already exists, so no rendering
doc·xls·ppt·pps	QLThumbnailGenerator	Only one representative image
hwp	Nowhere	Store only

Korean documents split into two paths. hwpx is read by the parser because it is an XML container. The older hwp is not in any list and falls to storeOnly. The file is stored and the body is empty.

The comment explains why hwp is not in the preview list. In measurement, the system could not draw its contents and returned a file icon. OCR on that icon stores the filename as the body.

Without knowing this, I would debug “why does hwp OCR return empty?” The cause is before OCR.

The core does not import PDFKit

// JustSendMemoryCore/Sources/JustSendMemoryCore/PageOrchestrator.swift:5-7
/// PDF는 PDFKit이, 미지원 포맷(hwp·doc·xls·ppt)은 미리보기 뷰어가 구현한다.
/// `JustSendMemoryCore`는 PDFKit·Vision·WebKit을 모른다 — 그래야 오케스트레이션을
/// 파일 없이 테스트할 수 있고, 뷰어를 바꿔도 파이프라인이 그대로 남는다.

The core knows only protocols. The app implements actual rendering.

This gives two benefits: pipeline tests need no real PDF files, and the pipeline code stays when the renderer changes.

The second benefit mattered in this round trip. I changed server OCR without touching the orchestration layer.

The two moves carried different baggage

I moved the same feature twice. Different things came along.

	On-device → server	Server → on-device
Gained	Recognition quality	Zero network, policy and code agree
Lost	Offline operation, privacy	Quality of the large model
New	Upload cap, usage limit, consent screen	Page and pixel limits
Removed	Vision implementation, tests, Xcode references	OCRClient, settings, consent UI, StoreKit connection, DB column
Unexpected cost	Three .pbxproj reference types	Network-blocked test environment

The right column's “removed” has five items. There are more places to touch when removing a feature than when adding one.

I also need to look at when the policy/code mismatch was found. It was not while building the feature, but while checking the consent basis to change its default. Without that check, the mismatch would have remained.