Building with Apple Intelligence

flagship · iphone app for riders

Record your rides, share them beautifully.

GPS track, speed, lean angle and weather, captured on the road and turned into a map poster. No accounts, no servers: everything stays on your iPhone.

Get Qroda ↗made for riders

I've been building Qroda, an iPhone app for recording motorcycle rides, in my spare time, and at some point I wanted to know what Apple gives developers for AI on its own devices. I wanted the view from Xcode rather than from the keynote: which frameworks exist, what each one is good at, what it costs, and where the limits are.

It turned out to be a good moment to look. At WWDC in June, Apple said it had rebuilt Apple Intelligence from the ground up, and iOS 27 shipped most of it on September 14: a new generation of models, a Foundation Models framework that takes images and can call Apple's server model on Private Cloud Compute, Siri AI in beta, and a new framework, Core AI, that opens up the engine behind Apple's own models. Custom adapters went away, and the API for generating images in code is deprecated. Most of what's written about Apple Intelligence for developers still describes iOS 26.

So I sat down with the current documentation, the WWDC sessions and the release notes, went through it one framework at a time, and wrote up what I'd have wanted to read before I started. Like my Google Cloud Networking 101 post, it tries to cover every major piece in one place: what it's for, when to reach for it, and how it fits with the rest. Where Qroda uses something, I say how it uses it.

The post goes roughly from least to most work: the models and the platform, the features you get almost for free, the Foundation Models framework, Siri and App Intents, the specialized frameworks, and running your own models. Xcode's own AI features get a short section at the end, since they changed as much as anything else this year. If you're new to the general ideas (prompting, structured output, tools, evals), my AI Engineering for Developers post covers them independently of any platform.

Everything here is as of early October 2026, on iOS 27.0 and Xcode 27.

What Apple Intelligence is now

Apple Intelligence is a set of models plus the system features and frameworks built on top of them. The models matter, because they decide what you can do on which device.

The models

At WWDC Apple introduced the third generation of its foundation models, which it calls AFM 3. There are five of them:

Each device runs exactly one of the two on-device models, and you don't get to choose. Core Advanced is a sparse model that streams its weights from flash, and it only runs on the newest devices with at least 12 GB of memory. Everything else gets Core. In practice your code has to work with the smaller one and benefit when the bigger one is there.

The other news in the technical report is that the models were built "in collaboration with Google and its Gemini models", following a deal the two companies announced in January. In interviews after the keynote, Apple's executives were careful to say that the models are Apple's own, trained by Apple and refined using outputs from Gemini, and that Siri isn't Gemini under the hood. For a developer, the practical effect is better models behind the same APIs.

Private Cloud Compute

Private Cloud Compute (PCC) is Apple's server side for requests that are too big for the device, and it comes with specific promises: requests are processed statelessly, nobody at Apple has privileged access to the running nodes, a request can't be targeted at a specific user, and Apple publishes the software images so researchers can check what's running. This year PCC grew beyond Apple's own data centers. AFM 3 Cloud Pro runs on NVIDIA GPUs in Google Cloud, using confidential computing on the GPUs and CPUs, and Apple says the same guarantees apply.

What matters more for developers is that your own app can now call PCC, which I come back to in the Foundation Models section.

Devices, languages and regions

Apple Intelligence runs on the iPhone 15 Pro and later (including every iPhone 16 and 17 model), the iPad mini with A17 Pro, iPads and Macs with M1 or later, and Vision Pro. Apple Watch Series 9 and later, Ultra 2 and later and SE 3 get it when they're paired with a supported iPhone. It supports 16 languages, among them English, most of the big European languages, Chinese, Japanese and Korean, and the device and Siri languages have to match.

Regions are uneven. Most Apple Intelligence features work in the EU, but Siri AI isn't available there on iPhone, iPad or Apple Watch, and in mainland China Apple Intelligence isn't available at all.

One thing changed quietly in iOS 27: Apple Intelligence is on by default and can no longer be turned off as a whole. Your code still has to handle devices that aren't eligible and models that are still downloading, but "the user switched it off" is no longer something you'll see on iOS 27.

What you get almost for free

Some Apple Intelligence features land in your app with no work at all, and the rest take a modifier or two. Start here, because the cost is close to zero.

Writing Tools

Writing Tools proofreads, rewrites and summarizes text, and it shows up in every standard text view automatically. The work is mostly deciding where you don't want it. In SwiftUI, .writingToolsBehavior(_:) takes .automatic, .complete, .limited (suggestions in a panel, without rewriting in place) or .disabled. Turn it off for code editors, search fields and anything where a rewrite makes no sense.

Qroda turns it on where it helps, in the name fields for rides, routes, spots, the rider and the bike, and ties it to the same Settings switch as the app's other Apple Intelligence feature. Each of those fields uses .writingToolsBehavior(aiEnabled ? .automatic : .disabled), so the switch really does turn everything off.

In UIKit you can also restrict the result types with allowedWritingToolsResultOptions (plain text, rich text, lists, tables) and react when a session starts and ends through the text view delegate, which is the place to pause syncing or autosave. If you have your own text engine, UIWritingToolsCoordinator lets you support Writing Tools without UITextView, and iOS 27 adds a grammar-checking presentation to it.

Genmoji

A Genmoji is an image that behaves like a character in text. In code it's an NSAdaptiveImageGlyph, which scales with the font around it. To accept them, set supportsAdaptiveImageGlyph on a UITextView; since iOS 26, a SwiftUI TextEditor bound to an AttributedString accepts them too. The trap is storage. A Genmoji lives in an attributed string, so if you save plain text it disappears. Save the attributed string (AttributedString is Codable, and RTFD works on the UIKit side), and use the glyph's contentDescription as the plain-text fallback for search, notifications and accessibility.

In iOS 27 Genmoji are generated in the cloud by the ADM 3 image model, and an app can let people create one directly by presenting the Image Playground sheet with the emoji style and an onAdaptiveImageGlyphCreation handler.

Image Playground

Image Playground generates images from text, photos and concepts you pull out of your own content. The way in is the system sheet: you pass one or more concepts and an optional source image to .imagePlaygroundSheet(...), the person refines the result, and you get back a file URL. Check supportsImagePlayground in the environment first, and copy the result out of its temporary location if you want to keep it.

import ImagePlayground
import SwiftUI
 
struct CoverButton: View {
    @Environment(\.supportsImagePlayground) private var supportsImagePlayground
    @State private var isPresented = false
    let story: String
 
    var body: some View {
        Button("Create a cover") { isPresented = true }
            .disabled(!supportsImagePlayground)
            .imagePlaygroundSheet(
                isPresented: $isPresented,
                concepts: [.extracted(from: story, title: "Cover")]
            ) { url in
                saveCover(from: url)
            }
    }
}

The .extracted(from:title:) concept is the useful one: instead of writing a prompt yourself, you hand the system a piece of your own content and it picks out what to draw.

iOS 27 changed a lot here. Generation moved to a new model on Private Cloud Compute, so images can now be photorealistic or any style (.any), with new options for output size and for editing an existing image versus generating a new one. There's a daily usage limit per person, which iCloud+ raises, and the system shows that UI for you. Generated images carry an invisible SynthID watermark. The ChatGPT style added in iOS 26 is still there as an external provider.

One API is on its way out. ImageCreator, which generated images in code without any UI, is deprecated in iOS 27, and Apple's guidance is to use the sheet or ImagePlaygroundViewController instead. If your design depended on generating illustrations silently in the background, it needs to change.

The features without an API

Plenty of Apple Intelligence has no developer API, and it's worth knowing about because it touches your app anyway. Notification and mail summaries rewrite what your app sends, so notifications with the important fact up front summarize better. Clean Up, Extend and Spatial Reframing in Photos, Image Wand in Notes, Smart Reply, Call Context, Workout Buddy and Safari's new features all exist only at the system level. So does the ChatGPT extension, which lets Siri and Writing Tools hand a request to ChatGPT when the user turns it on; it's still the only third-party model built into the system. Live Translation is a partial exception, because calling apps can opt into translated calls; more on that in the Translation section.

The Foundation Models framework

This is the centerpiece for developers. The Foundation Models framework, new in iOS 26, gives your app direct access to the on-device model from Swift: no server, no API key, no per-request cost, and it works offline. In iOS 27 it became much more than a wrapper around one small model.

What the on-device model is good for

The on-device model is small, so think of it as a capable text processor rather than a chatbot. It's good at summarizing, extracting fields, classifying, tagging, rewriting and short generation over content your app already has. It's bad at world knowledge, long reasoning and anything that needs a lot of context. Its context window is 4K to 8K tokens depending on the device, and that budget covers instructions, prompts, tool output and answers together. Read the actual size from SystemLanguageModel.default.contextSize rather than hard-coding it, and use tokenCount(for:) to measure what you're about to send.

That's the kind of job Qroda gives it. A feature called Ride in words writes one sentence about each ride, under its name on the summary screen, and a two-sentence intro for each monthly or season recap, in the rider's own language. A small model is plenty for that, and because it runs on the phone, the ride data never leaves it.

Sessions and guided generation

You talk to the model through a LanguageModelSession. A session keeps the conversation, so follow-ups have context. Instructions, which you write, tell the model what it is and how to behave, and the model gives them priority over the prompt. That makes the most important rule simple: never put user input in the instructions, only in the prompt.

The feature that makes the framework practical is guided generation. Instead of parsing free text, you mark a Swift type with @Generable, add @Guide descriptions and constraints, and the framework only lets the model produce a valid instance of it:

import FoundationModels
 
@Generable
struct RideSummary {
    @Guide(description: "A short, friendly title for the ride")
    var title: String
 
    @Guide(description: "Highlights of the ride", .count(3))
    var highlights: [String]
 
    var character: RideCharacter
}
 
@Generable
enum RideCharacter {
    case relaxed, sporty, epic
}
 
let session = LanguageModelSession(
    instructions: "You summarize motorcycle rides from trip data. Be brief."
)
 
let summary = try await session.respond(
    to: "212 km in 4h10. Passes: Giau, Falzarego, Valparola. Max lean 41°.",
    generating: RideSummary.self
).content

The model can't return a missing field or the wrong type. You can stream structured output too, and each snapshot is a partially filled version of your type, which is good for UIs that build up as the answer arrives. Properties are generated in the order you declare them, so put the ones the UI needs first at the top of the struct. When the shape is only known at runtime, like a form a user designs, DynamicGenerationSchema does the same job without a compile-time type.

A few details save time. Streaming snapshots contain the whole text so far, so you replace rather than append. prewarm(promptPrefix:) loads the model before the user hits send. A session handles one request at a time, and sending a second one while the first is running throws. For sampling, GenerationOptions takes a sampling mode and a temperature between 0 and 1; greedy sampling gives you the same output for the same input, which is what you want in tests.

Prompting a small model

The on-device model wants a different prompting style than the big cloud models. It does best with one narrow task per request. If a feature needs to extract facts, classify them and then write something, make that three calls with their own small prompts instead of one long one. Small steps are easier for the model and easier to test, and you can replace one later without touching the others.

Keep instructions short and plain, and when the output format matters, show one or two examples in the instructions instead of describing the format in prose. Let the type do as much work as possible: a precise @Guide description and a constraint like .range or .count steer the model better than another sentence of instructions. Give the model the content it needs rather than relying on what it knows, because its world knowledge is thin and can be wrong. Everything you send, including property names and guide descriptions, should be in a language the model supports.

Qroda's instructions show how plain they can be. The model gets the ride as a few short labeled lines, with every number already formatted by the app, and this:

let instructions = """
    You write one short sentence about a motorcycle ride, addressed to the \
    rider, in \(languageName). Use only the facts you are given. \
    Never use digits; write any amount in words. \
    Do not use abbreviations. \
    No exclamation marks, no emoji, no quotation marks, no hashtags. \
    Warm and plain. \
    Reply with the sentence only.
    """

Two choices in there matter. The numbers a rider sees always come from the app, in the rider's units, so instead of trusting the model to copy them correctly, the instructions tell it not to write digits at all. And the instructions stay in English while naming the target language, after a supportsLocale(_:) check, so the reply follows the rider's language without a prompt per language. Each ride also gets a fresh session, so nothing from one ride's prompt can leak into the next.

Guided generation is the right tool for structured output, but Qroda's output is one sentence, so it uses a validator instead. A reply that's empty, longer than 160 characters (240 for a recap), spread over several lines, contains a digit, or has more sentences than asked for is dropped, never repaired. If the model ignored "no digits", I don't trust it on the facts either. A cap on maximumResponseTokens makes a runaway answer fail the length check instead of running on.

Long inputs need a plan. Measure them with tokenCount(for:), and if a document doesn't fit, chunk it, summarize the chunks, and work from the summaries. Then test the prompt like code: a fixed set of inputs, greedy sampling so results are repeatable, and a way to compare outputs when you change something. Apple's new Evaluations framework, covered below, is built for exactly that.

Tool calling

The model doesn't know your user's data or today's weather. Tools let it ask your code. You implement the Tool protocol with a name, a description and @Generable arguments, pass the tools to the session, and the model decides when to call them:

struct FindRidesTool: Tool {
    let name = "findRides"
    let description = "Finds the user's recorded rides near a place."
 
    @Generable
    struct Arguments {
        @Guide(description: "A place, pass or road to search for")
        var place: String
    }
 
    func call(arguments: Arguments) async throws -> [String] {
        try await RideStore.shared.rides(near: arguments.place).map(\.title)
    }
}
 
let session = LanguageModelSession(
    tools: [FindRidesTool()],
    instructions: "Answer questions about the user's own rides."
)
 
let answer = try await session.respond(to: "When did I last ride the Stelvio?")

The model can call a tool several times, even in parallel, and everything a tool returns lands in the context window, so return short, relevant results rather than whole records. In iOS 27 you can also force or forbid tool calls per request with a tool-calling mode (.allowed, .disallowed or .required).

Apple now ships a few tools of its own. Vision provides OCRTool and BarcodeReaderTool, which the model can call to read text and codes in images you pass it. Core Spotlight provides SpotlightSearchTool, which gives the model retrieval over your app's own Spotlight index, the on-device version of RAG. It's a good fit for "answer questions about my notes" features. Apple's own session on it warns that the default configuration can overflow the smaller on-device context, so tune how much it retrieves.

Images in the prompt

The on-device model now takes images. You add an Attachment inside the prompt builder, from a CGImage, CIImage, pixel buffer, URL or UIImage:

let session = LanguageModelSession(tools: [BarcodeReaderTool()])
 
let response = try await session.respond {
    "What's on this receipt? List the items and the total."
    Attachment(receiptImage)
}

This covers a lot of what used to need a pipeline: describe a photo, compare two images, pull fields out of a screenshot. Combined with guided generation, it's a few lines from an image to a typed struct. For dense documents, pair it with the OCR tool rather than trusting the model to read small print.

Private Cloud Compute from your app

PrivateCloudComputeLanguageModel puts the server model, AFM 3 Cloud, behind the same session API, and switching to it is one line:

let model = PrivateCloudComputeLanguageModel()
let session = LanguageModelSession(model: model)
 
let response = try await session.respond(
    to: "Compare these three training plans and recommend one: ...",
    contextOptions: ContextOptions(reasoningLevel: .moderate)
)

You get a 32K context window and reasoning, with .light, .moderate or .deep levels (the on-device model doesn't reason). There's no API key and no bill. Instead, each person gets a daily quota tied to their iCloud account, which iCloud+ raises. The model exposes quotaUsage so you can show when someone is close to the limit, and a limitIncreaseSuggestion that presents Apple's upgrade options.

The catch is access. You need a managed entitlement, and right now only developers in the App Store Small Business Program with fewer than two million first-time downloads can get it. If you grow past that, you have six months to move off it. Apple says expanded access "for a fee" will come later. For indie developers that's a big deal: a capable server model with Apple's privacy guarantees, at no cost. For everyone else, it isn't an option yet.

Plan for it to fail sometimes. PCC requests need a network connection, availability can be .systemNotReady, and the quota can run out, so design the feature to degrade to the on-device model or to no AI at all. The server model is also available on watchOS 27, which makes it the way to use the framework on Apple Watch.

Bringing your own model

iOS 27 also turned the framework into a common interface. Anything that implements the new LanguageModel protocol works with LanguageModelSession, including guided generation and tools. Anthropic publishes a Swift package that brings Claude to it, Gemini is available through Firebase, and both Core AI and MLX can expose local models the same way. Dynamic profiles let a session change model, reasoning level and tool policy from one request to the next, and every response now reports token usage.

This is the part I find most useful as an architecture. Write the feature against Foundation Models with guided generation, use the on-device model when it's enough, and swap in PCC or a third-party model for the requests that need more, without a second code path.

On device, PCC or someone else's model

With one API in front of several models, picking one is mostly a product decision, so it helps to spell out the trade-offs.

The on-device model is free, works offline, answers fastest once it's warm, and keeps data on the device, which makes the privacy story easy to explain. Its limits are the small context, no reasoning, thin world knowledge, and the fact that it only exists on Apple Intelligence devices.

Private Cloud Compute keeps Apple's privacy guarantees and costs you nothing, and it adds a much larger context and reasoning. You pay for that with a network dependency, a daily quota per person, and an entitlement you have to qualify for.

A third-party model like Claude or Gemini is the most capable option and works for any developer, but you bring the API key, the bill and a server or proxy to keep the key off the device. Your users' data also leaves Apple's privacy guarantees and goes to another company, which belongs in your privacy policy and your App Store privacy details, and is worth asking people about before you send anything personal.

A model you run yourself through Core AI or MLX has no quota and doesn't depend on a network or another company. The cost shows up elsewhere: app size or a model download, the work of converting and tuning it, and battery and memory on the device.

My default is the on-device model for anything that fits in its context, PCC for the requests that don't if I qualify, and a third-party model only when neither is good enough and the feature is worth the extra privacy conversation. For Qroda the on-device model is enough, because a sentence about a ride needs neither a big context nor reasoning.

Availability, limits and errors

The model isn't always there, so check before you show an AI feature:

let model = SystemLanguageModel.default
 
switch model.availability {
case .available:
    print(model.variant.displayName, model.contextSize)
case .unavailable(.deviceNotEligible):
    // Hide the feature, or fall back to something without AI.
    break
case .unavailable(.modelNotReady):
    // Still downloading. Try again later.
    break
case .unavailable:
    break
}

variant tells you whether the device runs Core or Core Advanced, which is useful for deciding how much context to send. In the simulator, a scheme option lets you fake each availability state, so you can test the fallbacks.

Errors changed in iOS 27. The old GenerationError is deprecated in favor of LanguageModelError, whose cases are the ones you should handle in the UI: the context window is full, a guardrail tripped, the model refused, the request timed out, the language isn't supported, or you're being rate-limited. Rate limits mostly bite apps that generate in the background, and Apple's advice there is to use respond rather than streaming. When a long conversation fills the context, the fix is to start a new session from a trimmed transcript. Transcripts are Codable, so you can also save them and restore a conversation after the app restarts.

Guardrails are on by default. If your feature transforms content people wrote themselves (rewriting a journal entry, say), the permissiveContentTransformations setting reduces false positives.

In Qroda every failure is silence. The narrator returns either validated text or nothing, so an unavailable model, a refusal, a full context and a rejected reply all look the same to the UI: there's no sentence. The one file in the app that imports FoundationModels sits behind a small protocol, with an implementation that never writes for iOS 18 to 25 (the framework needs iOS 26) and a stub with canned sentences for previews and UI tests. The setting for the feature doesn't appear at all on phones that can't run it. Sentences are generated once, when the summary first appears, and stored with the ride; recap intros are cached with a fingerprint of their numbers, units and language, so they're only rewritten when one of those changes.

Tooling, and the end of adapters

The tooling got serious. In Xcode, the #Playground macro runs prompts inline next to your code and shows token counts. Instruments has a Foundation Models template for latency and token usage. The new Evaluations framework lets you measure the quality of an AI feature over a dataset. That's the piece I'd adopt first, because without it you can't tell whether a prompt change helped. Outside Xcode, macOS 27 ships an fm command-line tool, there's a Python SDK in beta, and Apple open-sourced a utilities package that includes a chat-completions adapter and runs on Linux.

Custom adapters are gone. In 2025 Apple let developers train them for the on-device model with a Python toolkit and a special entitlement, but they were deprecated in iOS 26.4 and removed in iOS 27, and the toolkit's last release doesn't support the new models. For custom models, Apple now points to Core AI. The built-in .contentTagging use case, a version of the model tuned for tagging and classification, is still available.

Siri, App Intents and Visual Intelligence

The second way Apple Intelligence reaches your app is by calling into it. That all goes through App Intents.

App Intents basics

An App Intent describes something your app can do, and once it exists, the system can run it from Siri, Shortcuts, Spotlight, the Action button, Control Center and widgets:

import AppIntents
 
struct SaveSpotIntent: AppIntent {
    static let title: LocalizedStringResource = "Save a spot"
    static let supportedModes: IntentModes = .background
 
    func perform() async throws -> some IntentResult & ProvidesDialog {
        guard try await RideRecorder.shared.saveCurrentSpot() else {
            return .result(dialog: "No ride is recording.")
        }
        return .result(dialog: "Spot saved.")
    }
}

Add an AppShortcutsProvider with spoken phrases (which must include your app's name) and the intent works with no setup by the user. App entities describe your data so intents can take it as parameters and the system can search it. If you wrote intents a while ago, note that supportedModes replaced openAppWhenRun, which is deprecated since iOS 26. Snippets, the small views an intent shows as its result, became interactive in iOS 26, so a snippet can have buttons and toggles that run other intents without opening your app. A snippet is its own intent that returns a view, it can reload itself after an action, and it's limited to 400 points of height, which keeps it to a card rather than a screen. Snippets don't run on tvOS, visionOS or watchOS. iOS 27 added long-running intents that go past the 30-second limit and show a Live Activity, plus cancellation, new parameter types for durations and people's names, and a simpler way to throw an error with your own message (AppIntentError(description:)).

Qroda's intents are small, and they still cover a lot. Start a ride, End my ride and Save a spot are App Shortcuts with phrases like "Start a ride in Qroda", so they work from Siri, Spotlight and the Shortcuts app with no setup. Start a ride is also a Control Center control, which the Action button can run. Save a spot is a LiveActivityIntent, so the button on the ride's Live Activity in the Dynamic Island runs it in the app's process without opening the app, and its dialog reports what actually happened (saved, no ride recording, no GPS fix yet) instead of always claiming success. Because Qroda still supports iOS 18, the intents that open the app declare both openAppWhenRun and, behind an availability check, supportedModes.

Siri AI and App Schemas

Siri AI, the personal, LLM-based Siri Apple first promised in 2024, shipped with iOS 27 as a beta. It understands personal context from Messages, Mail and Photos, knows what's on screen, can take actions across apps and answers from the web. As I write this, it's English only, there's a waitlist and a daily usage limit, and it isn't available on iPhone, iPad or Apple Watch in the EU, or in China.

For third-party apps, the way into Siri AI today is App Schemas. A schema describes a standard action in a domain Apple has defined, and you conform an intent or entity to it with a macro, like @AppIntent(schema: .messages.sendMessage). The domains that reach Siri are Audio, Calendar, Camera, Clock, Mail, Maps, Messages, Notes, Phone, Photos, Reminders, Video, and system and in-app search, and many of their schemas are new in iOS 27. Another set of domains (Books, Browser, Files, Journaling, Presentation, Reader, Spreadsheet, Whiteboard, Word processor) supports Shortcuts but not Siri. Conforming to a schema looks like this:

@AppIntent(schema: .messages.sendMessage)
struct SendMessageIntent {
    var content: AttributedString?
    var destination: MessageDestination
    var subject: AttributedString?
    var attachments: [IntentFile]
    var audioMessage: IntentFile?
    var locations: [GeoToolbox.PlaceDescriptor]
    var links: [URL]
    var scheduledDate: Date?
 
    func perform() async throws -> some ReturnsValue<[MessageEntity]> {
        let sent = try await Messenger.shared.send(content, to: destination)
        return .result(value: sent)
    }
}

The schema fixes the parameters and their meaning, which is what lets Siri use the intent reliably; you supply your own types for the destination and the message entity, and the code that does the work. Xcode helps: code completion suggests schemas by domain prefix, and it warns when you've implemented only part of a group of related schemas, like sending a message without drafting one.

A few other iOS 27 additions make schemas work better with Siri. When someone does something in your app by hand, you can donate the matching schema intent through IntentDonationManager, which teaches Siri what people actually do and improves its suggestions. Entities can conform to OwnershipProvidingEntity and say whether they're shared or public, so Siri asks for confirmation, with the right context, before it changes something other people can see. An entity that represents a person can convert to and from the system's IntentPerson through IntentValueRepresentation, which lets Siri carry a contact from one app into another. And entities can be marked as syncable, so a Siri conversation about them can continue on another device.

If your app fits one of the Siri domains, adopting the schemas is the most useful Apple Intelligence work you can do right now. If it doesn't, the effort still pays off through your entities, which is the next part. Qroda is in that second group: none of the Siri domains describes recording a ride, so its Siri support is the App Shortcuts phrases, and its data reaches the system as indexed entities.

On-screen awareness and the semantic index

Siri AI answers questions about what's on screen and about a person's own data, and both come from information your app provides. In iOS 26, with no work, Siri can already send a screenshot or your shareable content to ChatGPT. In iOS 27 you can be precise: annotate views with .appEntityIdentifier(_:), describe the primary item with an NSUserActivity, and annotate collections, so Siri knows which of your entities is on screen rather than guessing from pixels.

For personal context, entities that conform to IndexedEntity and are indexed with Core Spotlight end up in the semantic index Siri searches. Marking the properties that matter with @Property(indexingKey:) makes them searchable, and iOS 27 adds entity annotations on notifications, so a notification can point to the entity it's about. All of this is also what makes your app work well in Spotlight and Shortcuts, so it's worth doing regardless of Siri.

Every saved ride in Qroda is an IndexedEntity. Spotlight shows its name, or the place it happened, with a subtitle like "Bormio · Sep 24, 2026", and tapping a result opens the ride through an OpenIntent. The index lives on the device, which matters, because a ride's name and places are personal data.

Visual Intelligence

Visual Intelligence lets people point the camera at something, or take a screenshot, and search apps for it. Your app takes part by implementing an IntentValueQuery that receives a SemanticContentDescriptor (labels plus the pixels) and returns matching entities. Tapping a result opens it in your app through an OpenIntent, and a "more results" button can hand the search over to your app with the visual intelligence schema. In iOS 27 the same API works on iPad and Mac. If your app has a catalog of things people might photograph (products, plants, places, parts), this is a cheap way to show up.

iOS 27 also lets Visual Intelligence save what it recognizes straight into Calendar, Contacts and Health. If your app already reads from EventKit, Contacts or HealthKit, that data reaches you without any Visual Intelligence code at all.

Shortcuts and Spotlight

Since iOS 26, the Shortcuts app has a Use Model action that sends input to the on-device model, to PCC or to ChatGPT. When your app's entities flow into it, they're serialized to JSON from their properties, so well-described entities give better results. iOS 27 added newer models with web retrieval and an inspector that shows the model's transcript.

Two more iOS 27 changes in Shortcuts affect app developers. Describe a Shortcut lets people write what they want in plain language and builds the shortcut from the intents apps provide, so clear intent titles and parameter descriptions help decide whether your app's actions end up in it. And Shortcuts gained storage actions that sync through iCloud, plus automations that run when someone takes a screenshot, uses the keyboard or receives a notification. Stored entities need stable identifiers to survive that sync, which is worth checking if your entity IDs are generated on the fly.

For testing all of this, Xcode 27 has an AppIntentsTesting framework that runs intents and queries in a UI test bundle. Apple's suggested order is to test the intents first, then Shortcuts, then Spotlight, and Siri last.

The specialized frameworks

Before Apple Intelligence there was a long list of on-device ML frameworks, and they're still the right tool more often than an LLM. They're fast and free, they give the same answer every time, and most of them run on far more devices than Apple Intelligence does.

Vision

Vision analyzes images: text recognition, document structure, barcodes, faces, body and hand poses, saliency, image aesthetics and classification. Since iOS 18 it has a proper Swift API, where every request is a type with an async perform(on:). iOS 26 added RecognizeDocumentsRequest, which returns a document's structure (paragraphs, lists, tables and their cells, barcodes) rather than lines of text, plus a request that detects a smudged camera lens. Reading a table out of a photo is now a few lines:

import Vision
 
let request = RecognizeDocumentsRequest()
let observations = try await request.perform(on: imageData)
 
if let table = observations.first?.document.tables.first {
    for row in table.rows {
        print(row.map { $0.content.text.transcript }.joined(separator: " | "))
    }
}

Each cell's text also comes with detected data such as email addresses and phone numbers, so you don't have to parse them yourself.

iOS 27 added tap-to-segment: GenerateIterativeSegmentationRequest takes a point, a box or a scribble and returns a mask you can refine, with its model downloaded on demand. Vision also ships the OCR and barcode tools for the Foundation Models framework, and Vision now runs on watchOS too. For reading text and documents, Vision is still a better choice than asking a model to read the image.

Speech

SpeechAnalyzer, new in iOS 26, replaced the old speech recognition API for most uses. It's fast, handles long-form audio and runs entirely on device. The transcription models live in system storage, so they don't add to your app's size, and you manage them through AssetInventory. Use SpeechTranscriber on supported devices, fall back to DictationTranscriber elsewhere, and add SpeechDetector if you only want to process speech. The flow is always the same: create a transcriber for a locale, make sure its assets are installed, feed audio to a SpeechAnalyzer, and read the results as an async sequence of attributed strings that carry timing, so you can line text up with the audio. It isn't available on watchOS. iOS 27 made audio input easier, with providers that read straight from an AVAsset or a capture device and a converter for audio formats, which removes most of the plumbing older samples needed. SFSpeechRecognizer still works but is filed under legacy APIs. And if all you need is subtitles, AVKit in iOS 27 can generate English subtitles for video on device, with no code in your app.

Translation

The Translation framework translates text on device with downloadable language models. .translationPresentation shows the system translation UI, and TranslationSession translates in code, one string or a batch at a time. iOS 26.4 added a choice between high-fidelity and low-latency translation, and translation of attributed strings, so formatting survives. Translation isn't available on watchOS, tvOS or visionOS.

Live Translation, the system feature that translates calls as they happen, has a small API surface for calling apps: through CallKit or LiveCommunicationKit, an app can declare that it supports audio translation, and the system handles the rest.

Natural Language

Natural Language does language identification, tokenization, part-of-speech and named-entity tagging, sentiment and embeddings, including contextual embeddings from a small transformer. It hasn't changed since iOS 17 and it doesn't need to. For detecting a language, splitting text or computing similarity, it's faster and cheaper than any model call:

import NaturalLanguage
 
let recognizer = NLLanguageRecognizer()
recognizer.processString(text)
let language = recognizer.dominantLanguage

That's also a sensible first step before a Foundation Models call, since you can check that the text is in a language the model supports before you send it.

Speech synthesis, sound and music

AVSpeechSynthesizer reads text aloud with system voices, SSML and, with permission, a person's Personal Voice. Sound Analysis classifies audio with a built-in classifier that recognizes more than 300 sounds, in files or live. Both are unchanged and work everywhere. New in iOS 27 is Music Understanding, which analyzes a track on device for key, tempo, structure, instruments and loudness.

Running your own models

When nothing built in fits, you bring your own model. This changed the most in iOS 27, and the choice now comes down to the kind of model you have.

Core AI

Core AI is new in iOS 27 and available on every Apple platform. It's the inference framework that runs Apple's own on-device models, opened up to developers, and it's built for current architectures: transformers, diffusion models, segmentation. You convert a PyTorch model through torch.export with Apple's coreai-torch converter, compress it with the companion optimization package, and optionally compile it ahead of time for a platform. In your app, you load the model, get a function from it and run it, with support for stateful models like language models with a KV cache:

import CoreAI
 
// The first load specializes the model for this device and caches the result.
let model = try await AIModel(contentsOf: modelURL)
guard let classify = try model.loadFunction(named: "main") else { return }
 
var input = NDArray(shape: [1, 3, 224, 224], scalarType: .float32)
// Write the pixel data through input.mutableView(as: Float.self).
 
var outputs = try await classify.run(inputs: ["image": input])
let scores = outputs.remove("scores")?.ndArray

The first load can take a while for a big model, because Core AI specializes it for the device, so do it before the user needs the result. A Background Inference entitlement lets background tasks use the Neural Engine, and there's a debugger app and an Instruments template for profiling.

Apple also published a repository of ready-made export recipes and Swift packages, including a language model package that plugs into the Foundation Models framework. That's how a custom or open model ends up behind the same LanguageModelSession API as Apple's own. And if a model needs an operation no framework provides, Metal's tensor support lets you write it as a GPU kernel.

Core ML and Create ML

Core ML isn't deprecated. Apple now positions it for traditional machine learning (tree ensembles, regression, tabular feature engineering) and for existing pipelines, and points anything with a modern architecture to Core AI. It got no new APIs in iOS 27. Create ML still trains image, text, sound and tabular models on a Mac from labeled examples, without ML expertise, which is often all a custom classifier needs. Use its DataFrame-based APIs, since the old MLDataTable ones are deprecated.

MLX

MLX is Apple's open-source array framework for Apple silicon, with Python and Swift front-ends, unified memory and automatic differentiation. It's the tool for running and fine-tuning open models on a Mac: mlx-lm downloads, runs and LoRA-tunes most open LLMs, and mlx_lm.server exposes them through an OpenAI-compatible API. MLX supports the Neural Accelerators in M5 chips, and Apple reported up to four times faster time to first token than on M4. It can also spread inference and training across several Macs over Thunderbolt 5. Getting started takes two commands:

pip install mlx-lm
mlx_lm.generate --model mlx-community/Llama-3.2-3B-Instruct-4bit \
  --prompt "Explain a KV cache in two sentences."

Point mlx_lm.server at the same model and you have a local, OpenAI-compatible endpoint, which Xcode can use as a chat provider and which is handy for prototyping prompts before you move them into an app.

On the app side, mlx-swift-lm runs language and vision models inside an app, and its newest release bridges MLX models into the Foundation Models framework. My rule of thumb: MLX for experimenting, fine-tuning and Mac apps, and Core AI when a model ships inside an iPhone app.

Xcode's own AI

The tools for building all this changed just as much. Xcode 26 shipped with ChatGPT built in, sign-in for Claude, and support for any provider with a chat-completions API, including local models. Xcode 26.3 in February added agentic coding with Claude Agent and Codex, and exposed Xcode itself as an MCP server, so agents running outside Xcode can work with your project through xcrun mcpbridge. Connecting Claude Code is one command:

claude mcp add --transport stdio xcode -- xcrun mcpbridge

It's part of my own setup, which I described in How I Actually Use Claude Code. Xcode 26.6 added a Gemini agent and support for the Agent Client Protocol.

Xcode 27 builds agents into the editor: conversations next to your code, a /plan mode, subagents, and plugins that bundle skills, MCP tools and agents, with GitHub and Figma among the first. Any agent that speaks the Agent Client Protocol works, including local ones, and macOS 27 adds kernel-level control over which files an agent's subprocesses can touch. Xcode 27 runs only on Apple silicon.

What to use when

With everything laid out, most decisions come down to a few defaults:

• Text in standard views: Writing Tools, already there. Turn it off where it doesn't belong.
• Images from your app's content: the Image Playground sheet, not ImageCreator.
• Summaries, extraction, classification and tagging: the Foundation Models framework with guided generation, on device.
• Reading text, documents and codes: Vision, or Vision's tools inside a Foundation Models session.
• Bigger context or reasoning: Private Cloud Compute if you qualify, or a third-party model through the LanguageModel protocol.
• Search and Q&A over your app's data: index your entities in Core Spotlight and use the Spotlight search tool.
• Getting into Siri: App Schemas if your app fits a Siri domain, plus well-described, indexed entities and on-screen annotations either way.
• Transcription: SpeechAnalyzer.
• Translation: the Translation framework, and Live Translation for calls.
• Language ID, tokenizing and similarity: Natural Language.
• A custom modern model: Core AI. A custom classic model: Core ML or Create ML. Experimenting with open models on a Mac: MLX.

The end

Apple Intelligence for developers looks very different from a year ago. In iOS 26 it was a small on-device model and a handful of system features. In iOS 27 it's a layered platform: system features you get for free, a model framework that spans the device, Apple's servers and third-party providers, App Intents as the way into Siri, and Core AI underneath for your own models. The specialized frameworks are still there, and still the right answer for a lot of problems.

If you take nothing else away: design for the smaller on-device model and treat the larger ones as a bonus. Use guided generation instead of parsing text. Check availability and build the fallback first. Describe your app's data as App Entities, because Siri, Spotlight, Shortcuts and Visual Intelligence all read it. And remember that Private Cloud Compute is now an option for small developers, with a quota instead of a bill.

I hope you found this post useful. If you have any questions or feedback, please let me know!