Recipe: Personal AI Phone Agent

Build a personal AI phone agent that can call your phone and hold real conversations — no Twilio account, no ngrok, no server to host. Just a few API calls or natural language in any MCP client.

Inspired by Garry Tan’s voice-to-brain recipe, which requires Twilio credentials, ngrok tunnels, OpenAI Realtime API keys, and a Node.js server. AgentPhone handles all of that for you.

Option 1: Natural language via MCP (fastest)

If you have the AgentPhone MCP server configured in Claude, Cursor, or Windsurf, just say:

“Create a voice agent called Brain that greets callers warmly, asks what’s on their mind, and has a thoughtful conversation. Buy it a phone number in the 415 area code, then call me at +1XXXXXXXXXX.”

That’s it. The AI client will:

  1. Call create_agent with a hosted voice and system prompt
  2. Call buy_number and attach it to the agent
  3. Call make_conversation_call to dial your phone

Your phone rings, and you’re talking to your AI agent.

Option 2: Python SDK (3 calls)

personal_agent.py
1from agentphone import AgentPhone
2import os
3
4client = AgentPhone(api_key=os.environ["AGENTPHONE_API_KEY"])
5
6# 1. Create a hosted voice agent
7agent = client.agents.create(
8 name="Brain",
9 voice_mode="hosted",
10 system_prompt=(
11 "You are Brain, a thoughtful personal AI assistant. "
12 "When someone calls, greet them warmly by saying 'Hey, it's Brain.' "
13 "Ask what's on their mind. Have a natural, helpful conversation. "
14 "If they want to brainstorm, help them think through ideas. "
15 "If they want to vent, listen empathetically. "
16 "Keep responses conversational and concise — you're on a phone call, not writing an essay."
17 ),
18 begin_message="Hey, it's Brain. What's on your mind?",
19 voice="11labs-Brian", # Run client.voices.list() to see all options
20)
21
22# 2. Buy a phone number
23number = client.numbers.buy(country="US", area_code="415", agent_id=agent.id)
24print(f"Your agent's number: {number.phone_number}")
25
26# 3. Call yourself
27call = client.calls.make_conversation(
28 agent_id=agent.id,
29 to_number="+14155551234", # Replace with your phone number
30 topic="You are Brain, the user's personal AI assistant. Have a natural conversation.",
31 initial_greeting="Hey! It's Brain. I was just thinking about you — what's going on?",
32)
33print(f"Calling... {call.id}")

Option 3: cURL (no SDK needed)

Requires jq for JSON parsing. Install with brew install jq (macOS) or apt install jq (Linux).

personal_agent.sh
$API_KEY="your_api_key"
$BASE="https://api.agentphone.to"
$
$# 1. Create agent
$AGENT_ID=$(curl -s -X POST "$BASE/v1/agents" \
> -H "Authorization: Bearer $API_KEY" \
> -H "Content-Type: application/json" \
> -d '{
> "name": "Brain",
> "voiceMode": "hosted",
> "systemPrompt": "You are Brain, a thoughtful personal AI assistant. Greet callers warmly, ask what is on their mind, and have a natural conversation.",
> "beginMessage": "Hey, it is Brain. What is on your mind?",
> "voice": "11labs-Brian"
> }' | jq -r '.id')
$
$echo "Agent: $AGENT_ID"
$
$# 2. Buy a number
$NUMBER_ID=$(curl -s -X POST "$BASE/v1/numbers" \
> -H "Authorization: Bearer $API_KEY" \
> -H "Content-Type: application/json" \
> -d "{\"country\": \"US\", \"agentId\": \"$AGENT_ID\"}" | jq -r '.id')
$
$echo "Number: $NUMBER_ID"
$
$# 3. Call yourself
$curl -s -X POST "$BASE/v1/calls/conversation" \
> -H "Authorization: Bearer $API_KEY" \
> -H "Content-Type: application/json" \
> -d "{
> \"agentId\": \"$AGENT_ID\",
> \"toNumber\": \"+14155551234\",
> \"topic\": \"Have a natural conversation as Brain, the personal AI assistant.\",
> \"initialGreeting\": \"Hey! It's Brain. What's going on?\"
> }"

Receiving inbound calls

Once your agent has a number, anyone can call it. For a hosted agent, inbound calls are handled automatically — the agent picks up and follows its system prompt.

To receive calls on your personal number and have them answered by your agent, forward your phone to the agent’s number (in your phone’s settings or carrier app).

Comparison: AgentPhone vs DIY

DIY (Twilio + OpenAI)AgentPhone
Setup steps10 (credentials, ngrok, server, TwiML, webhooks)3 API calls
Twilio accountRequiredNot needed
Server to hostRequired (Node.js + ngrok)Not needed
Time to first call30-60 minutesUnder 2 minutes
Voice AIWire up OpenAI Realtime yourselfBuilt-in hosted mode
Phone numberManual Twilio console purchaseOne API call
Inbound callsManual TwiML routingAutomatic
TranscriptsBuild your ownBuilt-in via API
Cost~$22/mo + server hostingUsage-based, no infra
MCP supportNone26 tools, works with any AI client

What’s next

  • Customize the voice: Use list_voices to browse options and update_agent to change
  • Add a webhook: Get notified of calls and messages via webhooks
  • View transcripts: Every call is transcribed — check call details in the dashboard or API
  • SMS too: Your agent’s number can also send and receive texts
  • Web calls: Let people talk to your agent from a browser — no phone needed