Apple Foundation Models Framework: Taking On-Device LLMs to Production

# Apple Foundation Models Framework: Taking On-Device LLMs to Production

At WWDC25, Apple opened up direct Swift API access to the ~3-billion-parameter on-device language model that powers Apple Intelligence, via the `FoundationModels` framework. A year later at WWDC26, they abstracted that API behind the LanguageModel protocol and cracked the door open to third-party providers (Anthropic, Google). In this article I focus on the three things we ran into while using the framework in a real product: type-safe output through guided generation, access to real data through the Tool protocol, and the surprises that a 4096-token context window creates in production.

Don't confuse this framework with Core ML-based custom model pipelines — there you run a model you trained yourself, here you prompt Apple's ready-made, continuously updated system model. I covered separately, in my "Core ML on-device AI pipeline" article, when each of the two approaches is called for.

Architecture: SystemLanguageModel and the Session Lifecycle

At the center of the framework sits SystemLanguageModel — a reference to the system model on the device. Every interaction runs through a LanguageModelSession, and the session keeps the conversation history internally:

1import FoundationModels
2 
3let session = LanguageModelSession(
4    instructions: "You are an assistant that gives short, technical, and direct answers."
5)
6 
7let response = try await session.respond(to: "What's the main risk of SwiftData syncing with CloudKit?")
8print(response.content) // String

In production code, the first thing you need to do is not to call respond, but to check availability. The model can be "absent" for three reasons: the device isn't eligible, Apple Intelligence is turned off, or the model hasn't been downloaded yet.

1switch SystemLanguageModel.default.availability {
2case .available:
3    // proceed
4case .unavailable(.deviceNotEligible):
5    showFallbackUI(reason: .unsupportedDevice)
6case .unavailable(.appleIntelligenceNotEnabled):
7    showFallbackUI(reason: .settingsRequired)
8case .unavailable(.modelNotReady):
9    showFallbackUI(reason: .downloading)
10case .unavailable:
11    showFallbackUI(reason: .unknown)
12}

Pro Tip: Run this switch at the very top of the app, alongside your feature flag logic. The "model exists but isn't downloaded" state never shows up in the simulator; on a real device it's a common scenario the first time Apple Intelligence is turned on — make sure it's on your QA checklist.

Guided Generation: @Generable and @Guide

The framework's most practical feature is guided generation, which makes the model produce a Swift struct directly instead of free-form text. The @Generable macro generates a schema at compile time, and the model conforms to that schema through constrained decoding — meaning the classic LLM integration problems like JSON parse errors, missing fields, and hallucinated enum values disappear structurally.

1@Generable
2struct MeetingSummary {
3    @Guide(description: "One-sentence summary of the meeting")
4    let headline: String
5 
6    @Guide(description: "Action items, each short and in imperative form")
7    let actionItems: [String]
8 
9    @Guide(description: "Meeting priority level, 1 low 5 critical", .range(1...5))
10    let priority: Int
11}
12 
13let session = LanguageModelSession()
14let result = try await session.respond(
15    to: "Summarize these meeting notes: \(transcript)",
16    generating: MeetingSummary.self
17)
18 
19let summary = result.content // MeetingSummary, not String

@Guide constraints such as .range and the regex-based .pattern aren't just documentation — they're real constraints the model has to obey during decoding. In practice this is far more reliable than describing, in a prompt to the LLM, an API schema bound for your backend.

Pro Tip: Never make your @Generable structs your domain model directly. A mapping layer in between (MeetingSummary → Meeting) isolates schema changes and separates the fields the model produces from the fields headed to your backend — so when you want to extend the model schema, you don't break your domain model.

Streaming: Perceived Latency with Partial Generation

Fast as the 3B model may be, waiting for a long response to arrive all at once kills the UX. streamResponse yields a partial result of type T.PartiallyGenerated at every step — the compiler derives it automatically from your @Generable struct, so you get type-safe access to the fields even mid-stream:

1let stream = session.streamResponse(
2    to: "Suggest 5 creative fitness app names",
3    generating: AppNameSuggestions.self
4)
5 
6for try await partial in stream {
7    // partial.names is a [String]?, update the UI as it fills in
8    await MainActor.run {
9        viewModel.names = partial.names ?? []
10    }
11}

On the SwiftUI side, this gives you the ChatGPT-style "appears as it types" effect without building an extra state machine — the for try await loop can feed your @Observable view model as-is.

Tool Calling: Feeding the Model Real Data

A 3B-parameter model can't know anything past the moment its training data was frozen, nor anything in your own private database. The Tool protocol closes that gap: you tell the model which functions it can call, and it decides for itself which tool to call in which situation and with which arguments — the framework manages the parallel and serial tool call graph automatically.

1struct FindSavedArticlesTool: Tool {
2    let name = "findSavedArticles"
3    let description = "Searches among the user's saved articles."
4    let index: ArticleIndex
5 
6    @Generable
7    struct Arguments {
8        @Guide(description: "A short search phrase derived from the user's question")
9        var query: String
10 
11        @Guide(description: "Maximum number of articles to return", .range(1...5))
12        var limit: Int
13    }
14 
15    func call(arguments: Arguments) async throws -> ToolOutput {
16        let results = await index.search(arguments.query, limit: arguments.limit)
17        return ToolOutput(results.map(\.title).joined(separator: "\n"))
18    }
19}
20 
21let session = LanguageModelSession(tools: [FindSavedArticlesTool(index: articleIndex)])
22let response = try await session.respond(to: "Find the SwiftUI articles I saved last month")

The critical point here: the tool call happens on the device. Your call(arguments:) implementation may reach out to the network, but the model itself never sends raw data to Apple's servers — it only sees the arguments it produced and the result you returned.

When you define more than one tool, the framework calls them in parallel if needed and merges the results into a single final response — you don't write the orchestration logic by hand, you're only responsible for writing each tool's name and description clearly enough. In practice the most common mistake is two tools whose descriptions are too similar to each other: the model can't decide which one to call and triggers the wrong tool. Once you go past five tools, reviewing the descriptions and weeding out overlapping words pays off more than prompt engineering does.

Context Window: The 4096-Token Wall

Here we get to the thing that hurt us the most in a real product. SystemLanguageModel.default's context window is 4096 tokens — surprisingly small for a team used to GPT-4-class cloud models. When the window is exceeded, the framework doesn't silently truncate; it throws an .exceededContextWindowSize error, and that session becomes unusable from then on.

Apple Developer Technote TN3193 published a context management guide precisely to solve this problem; with iOS 26.4, a contextSize property and a tokenCount(for:) method were added to SystemLanguageModel — so you can now measure how many tokens a prompt consumes before sending it.

1let model = SystemLanguageModel.default
2let promptTokens = model.tokenCount(for: longTranscript)
3 
4if promptTokens > model.contextSize - reservedForResponse {
5    longTranscript = summarizeOrTruncate(longTranscript)
6}

Pro Tip: If you're building a multi-turn chat interface, never let the session grow without bounds. After N turns, summarize the older history with your own respond(to:generating:) call and inject it into a new session as a "context summary" — the user sees an uninterrupted flow, and you avoid the .exceededContextWindowSize crash.

Privacy: Where Is the Concrete Gain?

The biggest operational burden of cloud LLM integrations is KVKK/GDPR data processing agreements, data retention periods, and auditing "which data goes to which server." With Foundation Models, when the on-device model runs:

• The prompt, tool arguments, and tool results never leave the device.
• While Apple Intelligence is off, the framework already returns .unavailable — the data leakage risk is structurally absent at the framework level.
• For larger tasks that require the server side, Private Cloud Compute steps in; Apple's claim is that these requests are verifiably not logged either, and are used only to process the request — but that's a separate trust model, not the same level of guarantee as on-device.

For health, finance, or enterprise content applications with a "user data must never go to a third party" requirement, this makes the architectural decision easier: process the sensitive text with the on-device model first, and only an anonymized summary goes to the cloud service if needed.

WWDC26: The LanguageModel Protocol and Third-Party Providers

The biggest architectural change to arrive in the framework at WWDC26 was the opening up of the model abstraction layer. Thanks to the new LanguageModel protocol, both SystemLanguageModel and server-based models can feed the same LanguageModelSession; Anthropic and Google published their own Swift packages, opening up access to their frontier models through this API. The practical upshot: you keep the same @Generable/Tool codebase and can switch from the on-device model to a cloud model whenever you need to — the guided generation and tool calling contract doesn't change.

This is a real architectural win for the team: during feature development you can prototype quickly with the on-device model, then in user testing identify the scenarios that hit the 4096-token limit or the reasoning ceiling and route only those flows to the cloud provider. Because your Tool protocol and @Generable schema don't change, that switch isn't a separate integration project — it's a single parameter change on the session initialization line.

When to Use It, When Not To

Conclusion

The Foundation Models framework breaks the assumption that "every AI feature connects to a cloud API" — especially for short, structured, privacy-sensitive tasks, the on-device model really is good enough in production. But the 4096-token context window and the reasoning limits of 3B parameters make it a tool used in the right place, not "the solution to every LLM need." Our rule is simple: if the user's data must not leave the device and the task is short and structured, try Foundation Models first, and the moment you hit the context or reasoning ceiling, move over to the cloud side seamlessly with the LanguageModel protocol that WWDC26 opened up.