How is a production AI phone receptionist actually built?

A production AI phone receptionist is a real-time pipeline: a carrier delivers the call over SIP, audio streams as RTP media, speech recognition interprets it, a language model decides what to say and calls tools such as a booking calendar, and speech synthesis answers, all inside a tight per-turn latency budget, with a clean handoff to a person.

This guide walks that path layer by layer for technical and operational decision makers, including the federal and California rules that shape it. For the plain-language view, start with what an AI receptionist actually does day to day. Every standard and rule cited here links to its primary source.

What are the layers of an AI phone receptionist?

An AI phone receptionist is software that answers a business phone line, holds a spoken conversation, takes actions such as booking or messaging, and passes the call to a person when it should. It is built in layers, and each layer fails in its own way.

How does a phone call reach the AI in the first place?

A call to a business number starts on the public switched telephone network (PSTN), the interconnected system of carriers that routes ordinary phone calls. At some point the call is handed to an IP network, and from there to the receptionist software, usually over a SIP connection often called a trunk.

Numbers and porting basics

The number is the asset customers remember, so treat its ownership as an architecture decision. The Federal Communications Commission (FCC) consumer guide says that if you switch providers and stay in the same geographic area you can keep your number, and that porting can be done between wireline, IP, and wireless providers [10]. It states that once you request service from a new company, your old company cannot refuse to port the number even if you owe a balance or termination fee, that FCC rules require simple ports to be processed in one business day, and that you should not terminate existing service before initiating new service [10].

Design rule: register the number to the business, not the integrator, so it can move if the vendor changes.

SIP signaling: setting the call up

SIP (Session Initiation Protocol) is the signaling protocol that sets up, changes, and ends voice calls on IP networks. RFC 3261 describes it as "an application-layer control (signaling) protocol for creating, modifying, and terminating sessions" [1]. The basic sequence is short:

1. The carrier sends an INVITE to establish the session [1].
2. The receiving side may send provisional 1xx responses to show progress [1].
3. A 200 OK indicates the call has been answered, and the calling side confirms with ACK [1].
4. Either side ends the call with BYE [1].

In the SIP messages (typically the INVITE and its 200 OK), the two sides agree on codecs, addresses, and ports using the Session Description Protocol (SDP) offer/answer model defined in RFC 3264 [2]. A mismatch here is a classic cause of a call that connects with silence in one direction.

Media: where the audio actually travels

RTP (Real-time Transport Protocol) is the protocol that carries the call audio as a stream of timed packets. RFC 3550 says RTP provides end-to-end network transport functions for real-time data such as audio, and also says plainly that RTP "does not guarantee quality-of-service for real-time services" [3]. Its companion, RTCP, exists to provide feedback on the quality of the data distribution [3]. So production systems monitor jitter and packet loss instead of assuming a clean network.

Three details matter for AI specifically:

• Narrowband audio. The classic telephone codec, ITU-T G.711, recommends a nominal sampling rate of 8000 samples per second [8]. By basic sampling math (the Nyquist limit), that stream cannot carry frequencies above 4 kHz. Test speech recognition on real phone audio, never on studio recordings.
• Keypad tones. When a caller presses keys, the digits can travel inside RTP as events defined in RFC 4733 [5]. A receptionist that offers "press 1" must handle that path, not just speech.
• Encryption. SRTP, defined in RFC 3711, can add confidentiality, message authentication, and replay protection to RTP and RTCP [4]. Ask whether media is encrypted on every hop you control.

Caller ID authentication

Spoofed numbers feed the spam a receptionist will face. As of October 2026, the FCC's caller ID authentication page states that the FCC requires all voice service providers to implement the STIR/SHAKEN framework in their IP voice networks, and that this helps support providers' blocking decisions and call labeling [11]. Where your carrier passes verification results through, use them as one input to spam handling.

How does the AI hear the caller and talk back?

There are two broad designs, and neither is automatically better.

Cascaded pipeline. Speech recognition converts audio to text, a language model writes a text reply, and speech synthesis voices it. Each stage can be tested, swapped, and logged separately.

Speech-to-speech. A single model takes audio in and produces audio out. With fewer handoffs it can sound more natural and respond faster, but it is harder to inspect stage by stage.

Whichever design you choose, two rules hold. First, the model must not invent facts: NIST describes confabulation as "the production of confidently stated but erroneous or false content" [17], and on a phone line that means a wrong price or a slot that does not exist (see why your AI should never guess prices). Second, every fact the AI states about hours, prices, or availability should come from a tool or an approved knowledge source, not from the model's general training.

What is the latency budget for one conversational turn, and why does it matter?

A turn is one exchange: the caller speaks, the system decides the caller has finished, and the AI replies. The latency budget is the total delay you allow from the end of the caller's speech to the first sound of the reply, split across every component.

The telephone network has its own published reference. ITU-T Recommendation G.114 recommends not exceeding a one-way delay of 400 ms for general network planning, and says that if mouth-to-ear delays are kept below 150 ms, most applications "will experience essentially transparent interactivity" [9]. G.114 also notes that highly interactive tasks can be affected by much lower delays [9].

G.114 was written for transmission delay between people talking. An AI receptionist adds its own processing time on top of that in every turn, which is why the budget has to be engineered deliberately. The stages that consume it are network transit in, the endpointing wait (the pause the system takes to be sure the caller has finished), recognition, model reasoning up to the first words of the reply, any calendar or database lookup, synthesis up to the first audio chunk, and network transit back out.

Engineering guidance (not a standard or a measured benchmark): set a written target for the gap between the caller finishing and the first reply audio, measure it on every call, and track the slow tail (for example the 95th percentile), not only the average. Stream every stage so synthesis can start before the model finishes writing. When a tool call is unavoidably slow, use a short, honest holding phrase ("Let me check the calendar") instead of dead air.

How does an AI receptionist handle interruptions and know when the caller is done?

Three mechanisms make natural turn-taking work.

• Voice activity detection is the component that decides whether incoming audio contains speech or only noise.
• Endpointing is the decision that the caller has finished their turn. Wait too briefly and the AI cuts people off; wait too long and every turn feels slow. Callers reading digits or spelling a name pause more, so endpointing should loosen right after the AI asks for them.
• Barge-in is the caller's ability to interrupt while the AI is speaking. When speech is detected during playback, the system should stop audio quickly, discard the unspoken remainder, and record what the caller actually heard, so the model does not assume a cut-off sentence was delivered.

Echo matters: if the AI's own voice leaks into the inbound audio, it can interrupt itself. Test barge-in on speakerphones and car hands-free systems, not just headsets.

How does an AI phone receptionist book an appointment?

Booking is a tool call: the model outputs a structured request (for example "find open slots for a service estimate on Tuesday afternoon"), your code runs it against the real calendar, and the result returns to the model, which never writes to the calendar directly.

A sound booking flow for a home services company or a car dealership service lane looks like this:

1. Collect required fields. Service type, preferred day and time window, name, callback number, and location if technicians travel.
2. Check availability. Query the calendar for real open slots, with the business's time zone stated explicitly.
3. Offer two or three options. Long lists are hard to remember by ear.
4. Hold, then confirm. Place a short hold on the chosen slot so two callers cannot take it at once, then commit.
5. Make the commit idempotent. If the network drops and the request is retried, it must not create a second booking.
6. Send a written confirmation if the caller agrees to receive one.

For a restaurant the same pattern applies to party size and seating time. For a real estate office it applies to showing requests, with the agent confirming the final time.

How should the AI confirm details back to the caller?

Read back anything costly to get wrong: date and time, the phone number digit by digit, name spelling, and address. Then ask a direct yes or no question. Store the confirmed values, not the first transcript, and flag which fields the caller explicitly confirmed.

How does an AI receptionist hand a call to a person?

Decide in advance which moments belong to a person: complaints, pricing exceptions, financing, contract terms, anything the AI is unsure about, and any caller who asks for a human.

• Warm transfer. In SIP terms this is an attended transfer: the transferring side puts the caller on hold, calls the target to alert them, then completes the transfer. RFC 5589 describes this pattern using REFER with a Replaces header field [7]. Operationally, the AI briefs the person before the caller is connected.
• Blind transfer. RFC 5589 also documents the unattended or "blind" transfer, where the transferor ends its own leg of the call (sends BYE) as soon as the REFER transaction completes [7]. It is simpler but gives the person no briefing.
• Voicemail. If no one answers, the caller should land somewhere useful, not in a loop.
• Callback. The AI records the reason, the best number, and a time window, then creates a task for a person.

In every path, pass a context packet: caller number, name if confirmed, reason for the call, what has already been answered, and a link to the transcript. The success test is simple: the caller should never have to repeat themselves.

What should be logged, and how long should transcripts be kept?

• Per call: timestamps, call and SIP identifiers, outcome (booked, transferred, message, abandoned), transfer target, and per-turn latency by stage.
• Per turn: what the system recognized, what it said, and every tool call with inputs and results.
• Recordings and transcripts: only where consent and disclosure are handled (see the rules below).

Good retention practice: set a defined period for recordings, transcripts, and call records; delete on schedule; restrict who can play recordings; and mask card digits or other sensitive numbers callers say. Make sure the business, not the vendor, controls exports and deletion.

How do you monitor and test a voice AI system before and after launch?

Before launch (the same discipline applies when you evaluate an AI tool before you pay):

• Build a test set of real scenarios with private details removed: bookings, reschedules, cancellations, wrong numbers, a caller who changes their mind, and requests the business does not handle.
• Run them through the real phone path, not a text simulator, so codec, jitter, and endpointing are included.
• Cover varied accents, speaking speeds, background noise, and speakerphones.
• Force the failure cases: calendar unavailable, transfer target busy, call dropped mid-booking.

After launch:

• Track per-stage latency, transfer rate, abandonment, booking completion, and how often the AI says it does not know.
• Review a random sample of calls each week, checking facts that cost money first: prices, hours, availability, and anything resembling a promise.
• Alert on sudden changes, such as a spike in one-way audio or failed tool calls.

What are the common failure modes, and how do you design for them?

What rules apply to AI phone receptionists?

Three rules shape the design, and each one has a simple practice that covers it.

1. AI voices count as "artificial" voices. In February 2024 the FCC ruled that AI-generated voices are "artificial" voices under the Telephone Consumer Protection Act (TCPA) [12][13]. Outbound calls that use them follow the same consent rules as robocalls: prior express consent from the person called, and prior express written consent for telemarketing [12]. If your AI places callbacks, reminders, or follow-ups, collect and record consent first.

2. In California, recording needs everyone's consent. Recording a confidential call requires the consent of all parties (Penal Code 632), and calls involving cell or cordless phones have their own all-party consent rule (Penal Code 632.7) [15][16]. The simple practice: tell callers at the start that the call is answered by AI and may be recorded.

3. Identify the business at the start. FCC rules require artificial or prerecorded voice messages to state the identity of the business at the beginning and give a telephone number for it [14]. Doing this on every call is the simple standard.

The FCC ruling addresses calls placed using artificial voices; answering calls a customer makes is a different situation, and the disclosure practice above covers both.

In practice, on every call:

• Open with the business name and a plain statement that the caller is speaking with an AI assistant.
• If the call is recorded or transcribed, say so at the start, and offer a person where practical.
• Log that the disclosure played, with a timestamp.
• Use the same disclosure wording on every number and location.

Frequently asked questions

What is an AI phone receptionist? An AI phone receptionist is software that answers a business phone line, holds a spoken conversation, and takes actions such as booking appointments or taking messages. It connects to the phone network through a carrier, usually over SIP. It hands the call to a person when the situation calls for one.

Is a speech-to-speech model better than a cascaded speech pipeline? Neither is better in every case. A cascaded pipeline is easier to audit and to swap component by component, while a speech-to-speech model can sound more natural with fewer handoffs. Choose based on how much you need to inspect and control exact wording, such as prices and policy.

How fast does an AI receptionist need to respond? ITU-T G.114 says most applications feel "essentially transparent" when mouth-to-ear delay stays below 150 ms, and an AI receptionist adds its own processing time on top of the network. Set a written target for the gap between the caller finishing and the first reply audio, and measure it on every call. Track the slowest calls, not just the average.

Can an AI receptionist transfer a call to a real person? Yes. It can do a warm transfer that briefs the person first, a blind transfer, a voicemail drop, or a scheduled callback. In every case it should pass a summary so the caller does not have to repeat themselves.

Do I need consent to record AI receptionist calls in California? Yes. In California, recording a confidential call requires the consent of all parties (Penal Code 632), and calls involving cell or cordless phones have their own all-party rule (Penal Code 632.7). The simple practice is to tell callers at the start of every call that they are speaking with an AI and that the call may be recorded.

Does the FCC's AI voice ruling apply to inbound calls? FCC 24-17 treats AI-generated voices as artificial voices under the TCPA, so outbound callbacks and reminders placed with an AI voice need the called person's consent. Answering calls a customer makes is a different situation, and telling callers at the start that they are speaking with an AI and may be recorded covers both.

Sherwood designs and builds AI phone systems with this architecture in mind.

This article is general information, not legal advice.