How your phone reads its own screen

Chapter 1

The digital eyes

Chapter 2

The voice of the screen

Chapter 3

The brain upgrade

Ch 3 · The brain upgrade

sonia

sonia Context transfer

Sep 17

sonia

sonia App Intents and schemas

Sep 17

sonia

sonia Apple Intelligence arrives

Sep 17

sonia

sonia Siri On-Screen Awareness

I see your screen, Yeah, I see your screen, Mapping all the views to read what you mean!

Sep 17

sonia

sonia Multimodal thinking

Sep 17

sonia

sonia Siri can seamlessly move information from one active app to another.

Sep 17

sonia

Fun Fact

Siri's visual brain: Siri AI can identify specific pastries in videos just by looking at the active screen.

Sep 17

sonia

sonia Apps use structured blueprints to share their actions and content with Siri.

Sep 17

sonia

Glossary

reference resolution (noun)
The process of determining the specific entity to which a word or phrase refers in a given context. In April 2024, Apple researchers introduced ReALM to translate visual screen layouts into text-based formats.

Sep 17

sonia

Quiz

What system-wide index helps Siri search and organize your personal context on-device?

Sep 17

sonia

Gossip

Would you trust Siri to book a flight for you hands-free?

Sep 17

sonia

BETA

sonia Swipe the scanner over the screen to discover how Siri reads what you see.

Sep 17

sonia

Quiz

What framework do app developers use to let Siri take actions and find content inside their apps?

Sep 17

sonia

Quiz

If you ask Siri to identify a pastry from a video, which technology does it use?

Sep 17

sonia

Quiz

What does 'context transfer' mean for Siri AI on your phone?

Sep 17

sonia

Quiz

Why do third-party apps need to adopt App Intents schemas for Siri to understand them?

Sep 17

sonia

Quiz

What is the name of Apple's personal intelligence system that gives Siri advanced on-screen awareness?

Sep 17

sonia

Quiz

Which of these is a real-world example of Siri's on-screen awareness?

Sep 17

sonia

Quiz

What does it mean when Siri's underlying foundation models are 'multimodal'?

Sep 17