iOS 18.2 beta 2 rolled out to developers and testers on the Developer Beta channel on Monday, as Apple prepares the next version of its smartphone operating system, which is expected to arrive with more Apple Intelligence features in early December . The latest beta release also includes support for a new application programming interface (API) that will allow developers to grant the system access to on-screen content, allowing Siri and Apple Intelligence to pass the information to third-party services for processing. Will be allowed to send.
Apple introduces API for Siri’s OnScreen Awareness feature
On the Apple developer website, the company has provided documentation (via MacRumors) for a new API titled ‘Making onscreen content available to Siri and Apple Intelligence’, designed to allow access to an app’s onscreen content. Which enables Siri and Apple Intelligence to understand. What content the user is accessing.
According to the company, if a developer adds support for the OnScreen Content API, their application will provide the screen’s content to Siri/Apple Intelligence if the user explicitly requests it. Information on the user’s screen may be shared with a third-party service (such as OpenAI’s ChatGPT).
Apple also provided an example of Siri accessing onscreen content. While browsing the web, a user can say or type “Hey Siri, what’s this document about?” Ask Siri to provide a summary of a document.
Developers can also add support for onscreen awareness in browsers, document readers, file management apps, mail, photos, presentations, spreadsheets, and word processing apps. Apple says this list is not exhaustive, so more apps should be able to take advantage of the API in the future.
It’s worth noting that iOS 18.2 won’t bring support for the new Siri, which is expected to offer significantly improved functionality. Support for in-app actions is expected to arrive on iOS 18.4, which will reportedly be released by Apple in April 2025, which is plenty of time for developers to integrate support for the API into their apps.
