An iPhone Screen Recording Is Now How I Hand Work to My Agent

This morning I asked my agent to fix a tweet by recording my iPhone's screen and talking. I had the tweet open, said what was wrong with it and what I wanted, and stopped the recording. Less than two minutes later the recording was with my agent, along with a transcript of what I said and stills of what I was looking at.

The thing that does this is agent-dispatch, a small Mac app running on the Mac mini on my desk. Once a minute it checks Photos for new screen recordings from my phone, which iCloud delivers by itself. For each one it takes a still whenever the screen changes, reads the text in each still, and transcribes my voice, all on the Mac. Then Claude reads the transcript and answers one question: is Dan asking for something? If yes, the recording goes to my dot, the always-on agent in ChatGPT, through Clipshot, with a note telling it to watch, listen, and do what I asked.

What I learned

Most screen recordings are not requests, and silence catches most of those for free. The four phone recordings already in my library had no speech in them, so all four were skipped before a model was asked anything. Of the spoken ones I tested, it sent the one where I said "can you give me some examples as a comment reply here?" and skipped the one where I said I was only recording because the thing on screen was cool to watch.

Delivery was the easy part. Getting the agent to act was not. The first time, my dot read the recording and replied "Received," with a tidy summary of what I had asked for, and did none of it. Its standing instructions said to acknowledge a recording, and a note inside the recording did not outrank them. I had to tell the dot directly that a recording from my phone is a request. Only then did it open the stills and start on the work.

One thing to know before you try it: turn the microphone on. Long-press the record button in Control Center. A silent recording is skipped.