How do you evaluate an AI tool before you pay for it?

To evaluate an AI tool before you pay for it, test it on real situations from your own business, including the awkward ones, and get clear written answers on five things: where your data goes and whether it trains the vendor's models, who owns your content and accounts, how accurate it is and how you can check, when a person takes over, and exactly what you will pay as usage grows. A scripted demo cannot tell you these things. Your own test and the vendor's written answers can. This guide ends with a scorecard you can use in your next vendor call.

Start with the job, not the tool

Write down the one job you want done, such as "answer after-hours calls and book showings" or "draft our weekly listing emails." Then write what good looks like: the volume, what must always be captured (a name, a callback number, the vehicle a customer is asking about), and what must never be said. That short page becomes your test plan. Without it, every demo looks impressive because the vendor chose what to show you.

Questions to ask the vendor

Ask these in writing, and get the answers in writing.

• What does the AI do, and what does a person do behind the scenes?
• What does it not do well? A confident vendor will tell you the limits. If the answer is "nothing," keep asking.
• Where is my data stored, and who can see it? This includes the vendor's own staff and any outside companies they rely on.
• Is my data used to train or improve your models, or anyone else's? Ask whether the answer differs by plan, and whether you can turn it off.
• How long do you keep my data, and can I delete it?
• Can I export my content, settings, and history if I leave? In what format?
• What happens to my phone number, accounts, or domain if I cancel?
• How do you price usage, and what happens when I go over a limit?
• How do you notify customers when your terms or data practices change?

Regulators take vendor promises seriously. The Federal Trade Commission has written that AI companies must honor privacy and confidentiality commitments wherever they make them, including in promotional materials and terms of service, and that what a company leaves out can matter as much as what it promises [1]. The FTC's advertising guidance also sets the baseline for any product claim: advertising must be truthful and non-deceptive, and advertisers must have evidence to back up their claims [2]. That applies to AI products. In February 2025, the FTC announced a final order barring a company that marketed an "AI lawyer" service from advertising that it performs like a real lawyer unless it has sufficient evidence [3]. Written commitments are the ones you can hold a vendor to.

Test it yourself with your real work

A demo shows a good day. Your own test, in a trial or pilot, shows your days.

Use real scenarios. Take ten to twenty situations from the last month, with private details removed. Run each one and compare the result with what a good employee would have done.

Include the edge cases. Try an upset caller, a question that mixes two topics, and someone who changes their mind halfway through. For a car dealership, try a customer asking about a trade-in, financing, and a service appointment in one breath. For a real estate office, try a buyer asking about a property you do not list, or asking for an opinion the agent should give personally.

Ask it something it cannot know. This is one of the most revealing tests. Ask about a policy you never gave it, a price that is not in its materials, or tomorrow's availability without connecting a calendar. A well-built tool says it does not know and offers a next step, such as taking a message or passing the question to a person. A poorly built tool makes something up and says it confidently.

Test it more than once. Run key scenarios two or three times and check that the answers stay consistent.

Check the handoff. Ask for a person. Note how long it takes and whether the customer has to repeat themselves.

Data and privacy: where does your information go?

The National Institute of Standards and Technology (NIST) notes in its Generative AI Profile that training these systems requires large volumes of data, sometimes including personal data, and that most model developers do not disclose the specific data sources their models were trained on [4]. The FTC has also pointed out that customers may reveal sensitive or confidential information, including internal documents and their own users' data, when they use AI models, and that companies offering those models have a business incentive to keep collecting data [1].

Get clear answers on four points:

1. Location. Where is the data stored, and which outside services process it?
2. Training. Is your data used to train or improve models? Is that the default, and can you opt out?
3. Retention. How long is it kept, including call recordings and chat transcripts? Can you delete it on request?
4. Access. Who at the vendor can see it, and under what circumstances?

Match your scrutiny to what the tool will touch: a caption writer needs less than a tool that reads every customer email.

Ownership: what is yours when you leave?

Imagine you cancel in a year. What do you still have?

• Your content and history. Scripts, knowledge bases, templates, call logs, transcripts, and lead records. Confirm in the terms that you own them, and try an export in a common format during the trial.
• Your phone number. If an AI receptionist gives you a new number and your customers learn it, that number becomes part of your business. The Federal Communications Commission's consumer guide explains that if you stay in the same geographic area you can generally keep your number when you switch providers, across wireline, IP, and wireless providers, and that once you request service from a new company your old company cannot refuse to port the number even if you owe a balance or termination fee [5]. The same guide notes porting is not always possible, for example if you move to a new area [5]. Ask the vendor in writing whether the number is ported in your name and whether they will release it when you leave.
• Your accounts. Social media, email domain, and listing accounts should stay registered to you, not the vendor. Anything customers use to reach you should be owned by your business.

Accuracy and hallucinations: how to spot-check

AI tools that write or speak can produce answers that sound right and are wrong. NIST calls this "confabulation," which it describes as confidently stated but false content, known colloquially as hallucinations, and explains that it is a natural result of how generative models are designed: they generate outputs that approximate patterns in their training data, for example by predicting the next word [4]. That means accuracy is something you keep checking, not something you confirm once.

A simple spot-check routine works for most businesses:

• During the trial, mark every test output correct, partly correct, or wrong.
• After launch, review a random sample of real conversations or drafts each week.
• Check the costly facts first: prices, hours, availability, terms, and anything that sounds like a promise.
• Log mistakes and send them to the vendor. Their response tells you a lot.

NIST also describes "automation bias," the tendency to defer too much to automated systems, and notes it can make other risks such as confabulation worse [4]. In plain terms: the more polished the tool sounds, the more deliberate your checking should be.

Human handoff: where a person should still decide

AI can take routine work off your plate, but decide in advance which moments belong to a person, for example:

• Pricing exceptions, discounts, and anything that commits you to a number
• Complaints, refunds, and disputes
• Contract terms, financing, and anything with legal weight
• Opinions a licensed professional is expected to give personally
• Anything the tool flags as uncertain

The NIST AI Risk Management Framework, a voluntary framework released in January 2023, lists characteristics of trustworthy AI that include being valid and reliable, safe, secure and resilient, accountable and transparent, explainable and interpretable, privacy-enhanced, and fair with harmful bias managed [6][7]. The practical takeaway: know who is accountable for each kind of decision, and make sure that person can see what the AI did.

If the tool makes outbound phone calls, be aware of one specific rule. In February 2024 the FCC ruled that calls made with AI-generated voices count as "artificial" under the Telephone Consumer Protection Act, which means they are held to the same standards as other artificial or prerecorded voice calls, including the consent rules for telemarketing robocalls [8]. Ask the vendor how their product handles consent, and confirm the details with your own counsel.

Cost structure: where the surprises hide

Ask the vendor to price a realistic month using your own volumes.

• Per-minute or per-message pricing: how is a minute counted, and do hold time or failed calls count?
• Per-seat pricing: what counts as a seat?
• Caps and overages: at the limit, does the tool stop or keep going and bill more? Do you get a warning?
• Add-ons and setup fees: numbers, integrations, premium voices, onboarding.
• Price changes: how much notice do you get?

Then price it at your current volume, at double, and in your busiest month.

Contracts and cancellation

Before you sign, look for:

• Term and auto-renewal: monthly or annual, and how much notice to cancel?
• How to cancel: online, by email, or only by phone?
• Data after cancellation: how long can you still export it?
• Changes to terms: can the vendor change terms or data practices without your agreement?
• Service commitments: what uptime and support response times are promised?

Start with the shortest commitment offered. A pilot that works is easy to extend; an annual contract that does not fit is hard to undo.

A simple AI tool evaluation scorecard

Score each line from 0 to 2: 0 means no or unclear, 1 means partly, 2 means yes with written evidence. Use the notes column for what the vendor actually said.

A total of 24 or more out of 30, with no zeros on lines 3, 7, 9, or 10, suggests the tool is worth a longer pilot. Any zero on those four lines is worth resolving before you sign, whatever the total. These thresholds are a starting point, not a standard.

Frequently asked questions

How long should an AI tool trial last? Long enough to cover a normal week and at least one busy period, so you see the tool under real volume.

Is a free trial enough to evaluate an AI tool? Only if it lets you test your own scenarios, export your data, and see real usage counts.

What is the most important question to ask an AI vendor? Ask whether your data is used to train or improve their models, and get the answer in writing.

How can I tell if an AI tool is making things up? Ask it questions it has no information to answer and see whether it admits that. Then review a random sample of its real outputs each week, checking prices, hours, and any statement that sounds like a commitment.

Should AI make final decisions for my business? For routine, low-stakes tasks it can act on its own within rules you set. For pricing exceptions, complaints, contracts, and anything with legal or financial weight, a person should make the final call.

Sherwood builds AI tools with these same standards in mind, and you are welcome to ask us every question on this list.

This article is general guidance for evaluating software, not legal advice. For contract terms, privacy obligations, or calling rules, check with your own counsel.