Agentic AI represents a significant leap in artificial intelligence, offering not just reactive bots—often providing potentially misleading advice—but also proactive systems that can act on our behalf. Envision a future where ChatGPT could scout hotel rates and finalize your travel arrangements, or Gemini could seamlessly order your groceries online.

This innovative technology is still in its early development phase; however, advancements in AI tools are happening swiftly. Early iterations of browsers capable of performing browsing tasks for users and bots designed to assist in apartment searches have already emerged. These intelligent agents will soon become accessible to the public.

Such developments prompt crucial reflections: Are you ready to grant an AI access to your personal calendar and payment details? Can you trust your chosen assistant to recommend the best dining options nearby? While the prospective convenience of these AI helpers is enticing, it’s essential to note that AI can make errors and should not be entirely relied upon for significant decisions just yet.

A recent highlight is PhoneAgent, an early-stage AI assistant designed for iPhones, conceived by developer Rounak Jain (as seen in coverage by 9to5Google). This tool isn’t available as a traditional app, but tech enthusiasts can give it a try with a little effort and a few OpenAI credits.

The PhoneAgent assistant, initially built during an OpenAI hackathon last year, is still in an experimental phase—Jain himself notes that the software may sometimes produce incorrect results. Yet, it offers a tantalizing insight into the potential directions Apple, Google, OpenAI, and others might pursue.

By connecting to OpenAI’s models, PhoneAgent utilizes resources akin to those in ChatGPT, allowing users to execute actions directly on their iPhones. This can be incredibly beneficial for quickly generating text in emails, messages, or documents simply by entering a prompt.

In a demonstration, Jain instructs the app to take a photograph, share it with his friend Ron, and even add a simple haiku to the message. In another scenario, the agent relays flight information to Ron and books an Uber for the airport, correctly identifying the terminal based on the flight number. It can also perform system-level operations, like opening the Control Center or activating the flashlight.

Commands can be submitted either via voice or text, and users receive notifications upon task completion, allowing for follow-up commands through these alerts (like turning the flashlight off, for instance).

During testing, the PhoneAgent exhibited occasional slowdowns and bugs, which is typical for such early-stage software. Overall, it executed commands successfully; for example, it managed to send a message to a friend named Beth expressing eagerness for an upcoming meeting, and it easily took a photo that opened in the editing interface with a single command.

Communicating through AI for a heartfelt message!
Credit: DailyHackly

App compatibility remains somewhat limited; for example, it failed to play the latest Self Esteem album on Spotify or attach a photo to a note. Robust third-party integrations are essential, as demonstrated by devices like the Rabbit R1, but the extent of access third-party developers will permit AI assistants remains unclear.

If Siri and Gemini evolve along these lines, they could genuinely transform daily activities: Picture commanding an AI to find directions to your next meeting or schedule a ride to the car service while you handle other tasks. Nevertheless, numerous challenges remain—Jony Ive and Sam Altman, take heed.

Setting up PhoneAgent involves several steps but is manageable—and costs only the OpenAI credits used to access its models (testing a range of commands typically costs around $1). Begin by downloading Xcode from the macOS App Store, and also install GitHub Desktop while creating a free GitHub account if you don’t have one already.

From GitHub Desktop, clone the PhoneAgent repository: Select Clone a Repository from the Internet on the main screen or from the File menu, select Clone Repository. In the URL tab, paste “https://github.com/rounak/PhoneAgent.git,” choose a destination folder on your Mac, and click Clone.

Installing PhoneAgent via Xcode on macOS.
Credit: DailyHackly

You now have the PhoneAgent code on your Mac. Navigate to the saved folder using Finder, double-click the PhoneAgent.xcodeproj file, and it should open in Xcode. To test it on your iPhone, connect the device to your Mac with a cable, unlock it, and confirm any prompts to trust the computer. Then go to Privacy & Security in iOS Settings, and enable Developer Mode.

After the iPhone reboots, open the Xcode dropdown menu (located next to PhoneAgent) and select your iPhone. Choose the PhoneAgent option from the sidebar, switch to the Signing & Capabilities tab, and use Add Account to enter your Apple ID. Following that, select your name from the Team dropdown list under Signing & Capabilities. Modify the Bundle identifier to something unique to avoid conflicts.

Next, click on PhoneAgentUITests.swift in the left pane, and locate the “func testLoop()” line within the code. Hovering near the code label will reveal a play button; click it to run PhoneAgent on your iPhone.

Selecting your iPhone as the target device for testing.
Credit: DailyHackly

The initial screen will prompt you to enter an OpenAI key. Create an OpenAI account through a web browser, then visit this page to add payment information and purchase credits (starting at a minimum of $5). Next, proceed to the API keys section and click Create new secret key to generate a key to use with PhoneAgent (ensure you save a copy of this key securely).

Once all set, you can start issuing commands by tapping the microphone icon or entering text into the input field at the bottom of the screen to instruct the agent on what to accomplish.