Help & Onboarding

Vocalyze calls your production or staging Voice AI agents over the real telephone network using Twilio. During each call the platform acts as the guest: it either reads the exact lines you wrote (Scripted) or lets an AI guest improvise within a goal you described (Goal-based). The agent's replies are transcribed, the audio is recorded, and a second pass evaluates the transcript against expected outcomes and any rubric criteria. The result is a pass/fail status, a weighted score, a written summary, and a list of specific issues.

1. 1

   Make sure you have a voice agent

   Go to Voice Agents and add at least one agent with a reachable phone number. The call cannot start without it.

2. 2

   Create the scenarios you want to test

   In Scenarios, click New scenario. Choose the conversation mode: • Scripted — best for deterministic regression checks. Write each guest line exactly as it should be spoken. The platform will read the lines one by one. • Goal-based — best for realistic, open-ended conversations. Describe the guest's goal and personality in the AI scenario prompt; the simulated guest will improvise replies based on what the agent says. Pick a language, add tags for filtering, and write Expected Outcomes in plain language (e.g. "Reservation should be created", "AI should ask for phone number"). These outcomes are checked during evaluation.

3. 3

   (Optional) Create a rubric for weighted scoring

   Go to Rubrics and click New rubric. Give it a name and add criteria: • llm — the model reads the transcript and scores the criterion 0-100. Use it for subjective quality checks like "Agent asked for all required details" or "Tone was polite and professional". • latency — automatic pass/fail against a threshold in seconds. Use it for SLA checks like "Agent answers within 3 seconds". • outcome — literal pass/fail based on whether a specific expected outcome was observed. Set weights to reflect importance. A criterion with weight 2 counts twice as much as a weight-1 criterion.

4. 4

   Build the suite

   In Test Suites, click New suite. Enter a name and description, then select the scenarios to include. The picker shows each scenario's mode and tags so you can filter quickly. Enable Run this suite every morning as an automated sanity check if you want it included in the daily schedule.

5. 5

   Run the suite

   Click Run suite. The platform creates a suite run and queues one execution per member scenario. It starts up to 3 calls in parallel; as each call finishes, the next queued scenario starts automatically. Open the suite run page to watch live progress: queued / running / passed / failed counts update in real time. Each row links to the execution detail page for that scenario.

6. 6

   Review results and iterate

   When the run completes, review the aggregate pass rate and average score. Failed rows show why they failed. Open an execution to see: • Overall score and the weighted formula used. • A criterion-by-criterion breakdown. • The transcript with timestamps and latency chips. • The call recording (use Skip to conversation if the audio starts with ringing silence). If a few scenarios failed, use Retry failed only to re-run just those scenarios without rebuilding the queue.

7. 7

   Schedule daily runs and watch trends

   With the daily toggle enabled, the suite runs automatically every morning via a scheduler hitting /api/public/cron/daily-tests. Return to the Dashboard to see pass-rate and score trends, latency distribution, agent scorecards, suite scorecards, and regressions.

Every execution is scored against a rubric if one is attached, otherwise against the expected-outcome checks. Each criterion contributes a 0-100 score multiplied by its weight. The overall score is the weighted average:

For example, if four criteria score 80, 70, 60, and 90 with equal weights of 1, the overall score is (80×1 + 70×1 + 60×1 + 90×1) / 4 = 75. The execution detail page shows the exact formula used for that run under the score badge, plus a table that lists each criterion, its score, whether it passed, and any note. A run typically passes when the overall score is 70 or higher and no high-weight criterion failed outright.

Latency measures the perceived response time: the moment the guest finishes speaking to the moment the AI starts its real answer. The platform estimates when the guest line finished playing and uses Twilio's partial result callback to detect when the agent's actual reply began.

If the agent says filler speech like "One moment" before answering, that filler period is captured separately and the latency chip shows how long the real answer took after it. The Dashboard's p95 latency metric and the per-suite scorecards use these values to flag slow responses.

Agent

The Voice AI being tested — configured with a name, restaurant, phone number, and environment.

Scenario

A single test case describing what the guest will say and what the agent is expected to do.

Suite

A reusable group of scenarios that can be run together with one click.

Suite run

A single execution of every scenario in a suite, limited to 3 concurrent calls at a time.

Execution

One phone call and its resulting transcript, recording, and evaluation.

Rubric

A named set of weighted criteria used to score an execution.

Criterion

One rule inside a rubric (llm, latency, or outcome type).

Regression

A scenario that recently failed or dropped 2+ points compared with its previous run.