How a Regulated Buyer Should Run a UCaaS Evaluation

For a regulated buyer, the sequence of an evaluation matters more than the criteria. A requirement discovered in stage four can cost weeks of rework. The same requirement discovered in stage one costs a paragraph.

I have watched this play out from the vendor's side of the table more times than I can count. An evaluation starts with demos, builds enthusiasm, produces a shortlist, and then meets compliance, which asks one question that eliminates two finalists. Everyone goes back to the beginning, and the buyer blames the vendors for wasting their time.

The fix is not a better criteria. It is a better order. Here are the six stages I would recommend to any regulated, multisite organization. The order is the method

Stage 1. Compliance and legal set the requirements

Before anyone contacts a vendor, the people who own regulatory risk write down what the platform must do.

This stage produces one artifact: a list of requirements phrased as things you can test or evidence. Not aspirations, not categories.

Weak requirement: "The platform must be HIPAA compliant."

Testable requirement: "The vendor must execute a business associate agreement naming the specific services we will use, and provide it for review before signature."

Weak requirement: "Recordings must be retained appropriately."

Testable requirement: "Retention must be configurable per record type, and a legal hold must be applicable to a subset without extending retention globally."

Weak requirement: "The vendor must be secure."

Testable requirement: "The vendor must provide a current independent security attestation, such as a SOC 2 Type II report, covering the services in scope."

Who is in the room: compliance, legal, information security, and the person who owns records. Not the vendor. Not yet the broader IT team.

Gate to stage 2: every requirement is written, testable, and marked mandatory or preferred. If everything is mandatory, the list is not done.

Stage 2. Operations define the workflows

Now the people who run the phones describe what has to happen day to day.

This stage produces a second artifact: the workflows the platform must support, described as sequences rather than features.

Examples worth writing out in full: a new patient calls the main number during clinic hours; a site loses connectivity at 9 a.m.; a staff member covers at a second location for a week; a records request arrives with a deadline; a new location opens.

Feature lists are how vendors compare themselves. Workflows are how you compare vendors. They are not the same exercise, and the second one is yours to write.

Gate to stage 3: at least five workflows written end to end, including one failure scenario and one administrative scenario.

Stage 3. Market screen, on paper

Only now do vendors enter, and they enter on paper.

Send the stage one and stage two artifacts to a longer list than your eventual shortlist. Ask each vendor to respond in writing against every requirement, marking it as met, met with conditions, or not met.

Two things make this stage work.

Ask for mechanism, not capability. "Describe how you configure retention by record type" produces a very different answer than "Do you support configurable retention?" Capability answers are always yes.

Score the conditions, not just the yeses. A vendor who answers "met, at our Professional tier, with an additional license" has given you a more honest answer than one who simply says yes. Reward that candor.

On whether to issue a formal RFP: for most midsize organizations with multiple sites, a written response against your requirements is enough, and a full RFP adds process without adding information. If procurement policy requires one, use your stage one and stage two artifacts as its content rather than starting over.

Gate to stage 4: a shortlist of three or fewer, with every gap and every condition understood.

Stage 4. Scripted demonstrations

The default demo is led by the vendor. It shows what the product does best, in an order the team has rehearsed many times, and every one of them looks good. I know, because I have run thousands of them.

Replace it. Send each finalist the same script, drawn from stage two, at least a week in advance.

A workable script has six items:

1. Route a new patient call through the intended path, with the caller reaching the right queue.
2. Add a new user and assign them to two sites.
3. Apply a standard configuration to a new location.
4. Show retention settings across every site on one screen.
5. Retrieve every communication with one named party across a date range, and export it.
6. Describe, with the console open, what the front desk experiences when the site connection fails.

Have the same people attend every session. Score against the script during the session, not afterward, because impressions decay into preferences within a day.

Gate to stage 5: every finalist has performed the same script, and each has been scored.

Stage 5. References, chosen by you

Vendors choose the references they hand you. That does not make those references useless, but it means standard questions produce standard answers.

Ask each finalist for two references with a specific brief:

• One organization of similar size, in a similar regulatory environment, lives for more than a year.
• One organization whose implementation did not go smoothly.

The second request is the one that matters. Every vendor has had difficult implementations, mine included, and how a vendor talks about them tells you more than any success story. A vendor who cannot produce one is either very new or not being straight with you.

Four questions for every reference:

1. What took longer than you expected, and why?
2. What did you discover after launch that you wish you had asked about?
3. When you needed support urgently, what actually happened?
4. If you were doing it again, what would you change about the implementation, not the product?

Gate to stage 6: both references contacted for every finalist, with the difficult one contacted first.

Stage 6. Contract and implementation terms

The last stage is where most of the durable value is won or lost. It is also the stage teams rush, because everyone wants to be done.

Four things to settle before signature:

Pricing, now and at renewal. Know the full monthly cost, including taxes, regulatory fees, and emergency calling charges, and know exactly what governs the rate at renewal: fixed, capped, or reversion to list. Read the clause yourself.

The support commitment. Written into the agreement, with a definition of what "response" means.

Data portability and exit. Export format, timeline, and whether metadata travels with the media. Negotiate this at signature, because at renewal you have little leverage and at exit you have none.

Implementation scope. What the vendor includes, what it bills for, and the sequence across sites. Most organizations discuss implementation after the contract, which is exactly backwards.

Gate to signature: all four in writing.

On timelines

I deliberately do not put durations on these stages, and it is worth saying why.

Published evaluation timelines are unreliable, because the variable is not the process. It is how quickly your compliance function can produce a requirements list and how quickly your finalists respond. One organization has a compliance officer who can write requirements in two sessions. Another spreads that role across three people who all have other jobs. They will never move at the same speed.

The useful discipline is not a calendar. It is the gate at the end of each stage. Do not open the next stage until the current gate is met, and the evaluation will take exactly as long as it needs to.

Frequently Asked Questions

How long does a UCaaS evaluation take?

It depends on how fast your compliance function can produce a testable requirements list and how quickly finalists respond in writing. Rather than working to a calendar, use a completion gate at the end of each stage and do not begin the next until it is met.

Who should be involved in a UCaaS evaluation?

Compliance and legal first, before any vendor contact. Operations second, to define workflows. IT during technical validation. Finance at contract stage. The common failure is involving compliance after the shortlist is built.

What is a scripted demo?

A demonstration where the buyer supplies the tasks in advance and every finalist performs the same tasks in the same order, with the same people attending. It replaces the demo led by the vendor, which is rehearsed and therefore not comparative.

Should we issue a formal RFP?

For most midsize organizations with multiple sites, a written response against a testable requirements list provides the same information with less process. If procurement policy requires an RFP, use your stage one and stage two artifacts as its content.

What should we ask for a vendor reference?

What took longer than expected, what they discovered after launch, what happened the last time they needed urgent support, and what they would change about the implementation rather than the product. Contact the reference whose implementation went badly first.

Why does sequence matter more than criteria?

Because a requirement discovered late invalidates work already done. A compliance requirement found in stage four eliminates finalists you spent weeks evaluating. Found in stage one, it costs a line in a document.

Quick Answers

• Six stages: compliance requirements, operational workflows, paper screen, scripted demos, references you choose, contract terms.
• Stage one belongs to compliance, before any vendor contact.
• Requirements must be testable. "HIPAA compliant" is not a requirement.
• Ask every finalist for a reference whose implementation went badly.
• Plan by gates, not dates. Each stage ends when its artifact is complete.