On-Device Inference Heat: thermalState Gating and a Serial Queue

A heat report came from a real device. It was an iPhone Air.

I needed a profiler to find the cause, but I could not attach one to that device. I had to narrow it down from code and logs.

Two causes emerged. Several model calls could run at once, and a retry was repeating even though it could never succeed. The second was worse: every retry ran the entire model again.

Heat comes from generated tokens

First I needed to decide what was expensive. In on-device inference, the cost is in output rather than input.

I measured the same summary two ways, with five runs each on an M4 Max and an 8K window.

Receiving method	Time	Generated length
Structured output	1.89 seconds	115 characters
Markdown plain text	2.16~2.36 seconds	173~202 characters

The shorter one is faster. One generated token is one computation, and computation is heat.

So the first way to reduce heat is not throttling. It is receiving a shorter answer. If the schema forces the shape, the model does not add phrases like “Here is the summary.”

The executor comment says it directly.

생성 토큰이 곧 연산이고 발열이므로 산출 억제가 그대로 이득이다.

The second method is concurrency control, and the third is thermal-state gating. The order is: reduce output, prevent overlap, and stop when hot.

I put model calls in one line

Saving ten documents can start ten summaries. If each has 14 chunks, that is 140 calls.

The device cannot sustain that concurrently. So I put every model response into one queue.

// FoundationModelRuntime.swift:193-199
/// Serializes every live Foundation Models response. Unsafe device conditions
/// are waited on outside the shared queue, then checked again immediately
/// after queue acquisition before the model operation starts.
static func withAdmission<T>(
  for job: AdmissionJob,
  operation: @Sendable () async throws -> T
) async throws -> T {

Every place that calls the model passes through this function. There are three kinds now.

// FoundationModelRuntime.swift:164-168
enum AdmissionJob: String, Sendable {
  case profileSummaryChunk   // 조각 요약
  case profileSummaryTitle   // 제목 만들기
  case summaryMap            // 음성 전사 요약
}

The names are for logs, so I can separate how long each kind takes.

Concurrency of one increases total time. 140 calls run in order. Instead, the device does not become hot enough for the system to kill the app. A slow completion is better than a dead app.

I periodically replace the queue

The queue itself has one limit.

// FoundationModelRuntime.swift:7-44
private final class FoundationModelAdmissionQueue: @unchecked Sendable {
  static let resetThreshold = 256

  private func resetIfIdleLocked() {
    guard pending == 0, completed >= Self.resetThreshold else { return }
    queue = SequentialOrganizerQueue()   // 새 큐로 교체합니다
    completed = 0
  }
}

When more than 256 jobs have completed and no job is waiting, I create a new queue object.

The reason is visible inside the queue. For every job it stores enqueued, started, and ended events in an array, and puts job names and cancellation state in a dictionary. There is no code to reduce the event array. Summarizing 100 documents would leave 1,000 call events in place. Replacing the queue removes that history too.

pending == 0 is the safety guard. If I replace the queue while work is waiting, I cannot know where that work goes. I replace it only when nobody is waiting.

I do not start when it is hot

Even concurrency of one becomes hot if it runs for long enough. So I check device state before starting.

// FoundationModelRuntime.swift:230-237
private struct Conditions: Sendable {
  let thermalState: ProcessInfo.ThermalState
  let lowPowerMode: Bool

  var isUnsafe: Bool {
    thermalState == .serious || thermalState == .critical || lowPowerMode
  }
}

ProcessInfo.ThermalState is the device's thermal state reported by iOS. There are four levels.

State	Meaning	Our decision
nominal	Normal	Run
fair	Slightly warm	Run
serious	Hot; the system reduces performance	Wait
critical	Very hot	Wait

Stopping from serious is a choice. iOS is already reducing CPU clocks at that point. Starting heavy inference then makes it run slowly while producing more heat.

I also check Low Power Mode. Starting inference that takes minutes while a user has asked to save battery goes against that intention.

The device heats while waiting for the queue

The subtle point in this design is waiting for the queue.

In sequence: check the state, enter the queue if safe, and run when it is your turn.

The problem is the waiting time in the queue. If 100 jobs are ahead, the wait lasts minutes. Those jobs heat the device during that time. When my turn arrives, the state I checked is no longer current.

So I check twice.

// FoundationModelRuntime.swift:200-226
let deadline = ContinuousClock().now.advanced(by: Self.admissionConditionWaitTimeout)
while true {
  try await waitForSafeConditions(for: job, until: deadline)   // 큐 밖에서 대기

  do {
    return try await admissionQueue.run(named: job.rawValue) {
      let conditions = Self.currentConditions()
      guard !conditions.isUnsafe else {                        // 큐 안에서 재확인
        Self.logDeferred(job: job, conditions: conditions)
        throw DeferredAdmission()                              // 큐를 놓아 줍니다
      }
      return try await operation()
    }
  } catch is DeferredAdmission {
    // Queue ownership has been released. Re-check conditions outside the
    // queue before attempting this same request again.
    continue
  }
}

DeferredAdmission creates the loop. If the state is bad inside the queue, I throw, release the queue, and wait outside it again.

I must not wait while holding the queue. That blocks every job behind it. Waiting means giving up the place; running means taking it only when possible.

while true is that loop: wait outside until safe, enter and check, and leave again if unsafe.

The same condition is checked in three places, each for a different reason.

Where	What it does	If unsafe
Waiting outside queue	Waits on notification and 250ms polling	Keeps waiting, up to 30 seconds
Immediately after registering wait	Checks once immediately	Waits
After entering queue	Checks again before computation	Releases the queue by throwing

Checking the same thing three times looks redundant. The times differ. Time passes between waiting ending and acquiring the queue, and the device can heat again. The longer the queue wait, the larger that gap.

I wait for a notification, but give up after 30 seconds

There are two ways to wait for the device to cool: ask periodically, or ask to be notified when it changes.

I use the latter.

// FoundationModelRuntime.swift:87-98
observers = [
  center.addObserver(
    forName: ProcessInfo.thermalStateDidChangeNotification,   // 열 상태가 바뀌면
    object: nil, queue: nil
  ) { [weak self] _ in self?.signal() },
  center.addObserver(
    forName: .NSProcessInfoPowerStateDidChange,               // 전력 모드가 바뀌면
    object: nil, queue: nil
  ) { [weak self] _ in self?.signal() },
]

A notification is better than polling. Asking every 250 milliseconds uses battery by itself.

But I cannot trust notifications alone. Sometimes no notification arrives.

// FoundationModelRuntime.swift:171-174
// Conditions are rechecked at this monotonic deadline even when no system
// notification is delivered. The short interval bounds notification latency.
private static let admissionConditionWaitTimeout: Duration = .seconds(30)
private static let admissionConditionPollInterval: Duration = .milliseconds(250)

After 30 seconds I check again even without a notification. If a notification is late or lost, the upper bound prevents waiting forever.

I check once more when starting to wait

Immediately after registering the notification, I check the state again.

// FoundationModelRuntime.swift:62-69
let process = ProcessInfo.processInfo
let isUnsafe =
  process.thermalState == .serious
  || process.thermalState == .critical
  || process.isLowPowerModeEnabled
if !isUnsafe {
  finish()      // 이미 안전하면 알림을 기다리지 않습니다
}

The device may have cooled while the observer was being registered. In that case, no next notification arrives because it already changed to a good state.

Without this check, I wait 30 seconds for nothing. Checking once after registration removes that wait.

Retries were creating the heat

I was classifying model errors by string.

// FoundationProfileModelExecutor.swift:115-119 (주석)
// **타입으로 가른다.** 예전에는 `String(describing:)`을 영어 키워드로 훑었는데,
// Apple이 guardrail 거부에 주는 문구는 `"May contain sensitive content"`라서
// `guardrail`·`safety`·`refus` 어디에도 걸리지 않았다 — 안전 필터 거부가 조용히
// `.permanent`로 떨어져 "처리 중 문제가 생겼어요 · 다시 시도"로 보였고, 사용자는
// 될 리 없는 재시도를 반복하며 매번 모델을 통째로 다시 돌렸다(실기 발열).

The chain was this.

Step	What happened
1	The safety filter rejected the input
2	Apple's wording was "May contain sensitive content"
3	Our code searched for guardrail, safety, and refus; none matched
4	It was unclassified and shown as “Try again”
5	The user tapped it; the same input was rejected again
6	The model ran from the beginning every time

A rejection has the same result on retry. Yet the screen recommended “Try again.”

Type classification removed it

I classified by error type instead of string.

// FoundationProfileModelExecutor.swift:126-136
switch generation {
case .guardrailViolation: return .rejected("guardrailViolation")   // 다시 해도 안 됩니다
case .refusal: return .rejected("refusal")
case .rateLimited, .concurrentRequests: return .transient("rateLimited")  // 잠시 후 됩니다
case .assetsUnavailable: return .unavailable
case .exceededContextWindowSize: return .permanent("exceededContextWindowSize")
case .unsupportedGuide: return .permanent("unsupportedGuide")
case .unsupportedLanguageOrLocale: return .permanent("unsupportedLanguageOrLocale")
case .decodingFailure: return .permanent("decodingFailure")
@unknown default: return .permanent("unknownGenerationError")
}

I inspect the cases of LanguageModelSession.GenerationError directly. Apple can change the wording while the case remains.

The split between rejected and transient is the retry policy. rejected does not show a retry button; transient does.

String matching is tied to language and version. Apple can change one character and it silently breaks. Nothing tells me it broke.

Serialization does not reduce computation

withAdmission does two things: global serialization and waiting. It does not reduce the computation itself. There is no forced rest between successful inferences. Only one runs at a time, and that one continues without a break.

So if serious persists, the user sees a different problem. The loop of releasing the queue, waiting, and entering again continues up to 30 seconds, and the screen simply looks slow. Heat becomes delay.

When I measured on a real device, that interval did not occur.

Real-device measurement	Value
Consecutive actual model responses	8 (notes, transcription, documents)
Thermal state throughout	nominal
Thermal defers	0
Resumed after waiting	0
Battery	Stayed at 100%

When I rechecked the original heat report, admission was also innocent: thermal=nominal, lowPower=false, and .automatic and .userInitiated produced the same result. The cause of the heat is in the next section.

Eight runs are not a long validation. These numbers cannot say where a device used all day will end up.