Skip to content
Operations November 7, 2026 10 min read

How to A/B Test Outbound Call Campaigns

How to A/B test outbound call campaigns: what to test, one metric per test, randomising the list, sample size before you start and reading results honestly.

D
November 7, 2026

Quick answer

To A/B test call scripts, change one thing, pick one metric, split the list at random into two equal groups and run both at the same times with the same agents. Work out the sample size before you start, do not stop early, then compare the two proportions with a z test. Keep every variant compliant.

Many outbound teams change a script, watch results for a week and decide it "feels better". The trouble is that call results swing for reasons unrelated to the script: a different list, a different day, a stronger team on shift. Without a proper split, you cannot tell a real improvement from noise.

A/B testing a call campaign is not complicated, but it has rules, and calling has traps that email tests do not. This guide covers what you can test, how to choose a single metric, how to randomise, how to size the test before dialling, how to read the result in plain words and what compliance expects while you experiment.

What you can test on a call campaign

Each of these is a candidate for a test, and each affects a different part of the funnel.

What you changeMetric it should moveNotes
Opener (first 10 seconds)Conversation continues, then conversion per contactSee the script templates
Offer or price framingConversion per contactOnly with approved, accurate offers
Voicemail messageCallbacks or later contactSee voicemail drop practices
Calling windowContact rateStay inside permitted hours
Dialing mode or pacingAgent talk time, contacts per hourWatch abandon limits; see the power dialer

For ideas that raise the chance of an answer in the first place, see how to improve outbound contact rates, and for the wider habits around testing and coaching, the outbound calling best practices playbook.

One metric per test: contact rate or conversion per contact, not both

Choose the metric before the test starts and write it down. Two metrics matter most, and they answer different questions.

  • Contact rate: contacts divided by dials. It tells you whether people answer. It is affected by timing, caller ID, list quality and answering-machine handling, and hardly at all by what an agent says after the greeting. See the glossary entry on contact rate.
  • Conversion per contact: sales, appointments or promises to pay divided by live conversations. It tells you whether the conversation worked. It is affected by the script, the offer and the agent, and not by whether the phone was answered.

Mixing them causes confusion. A new opener cannot change the contact rate, because the person has not heard it yet. A new calling window cannot sensibly be judged by conversion alone. If you measure per dial instead, you blend both effects and cannot tell which one changed. Definitions of related measures are in the KPI formulas guide.

The danger of looking at five metrics after the fact is real: with enough numbers, one will favour either version by chance. Name a primary metric. Treat the rest as context.

Randomise the list, not the agents' choice

Randomisation is what makes the comparison fair. If agents choose which script to use, they will use the new one on the leads they like. If you split by alphabet, region or upload date, you are comparing different kinds of people.

  1. Take the full list for the test and remove contacts that fail consent or do-not-call checks first, so both groups start clean.
  2. Add a column of random numbers (a spreadsheet RAND function is enough), sort by it and cut the list in half. Alternatively, assign contacts alternately after sorting by the random number.
  3. Check that the halves look alike on the things that matter: number of contacts, time zones, age of lead, source. If one half has most of the best leads, re-randomise.
  4. Upload each half as its own list, attached to its own campaign: Campaign A with script A, Campaign B with script B.

Keep records of which contact went to which group. You will need them to explain any odd result.

Work out the sample size before you start

This is the step call tests often skip, and it decides whether the test can ever answer your question. A small test can only detect a big difference; a real but modest improvement will look like noise.

The NIST/SEMATECH e-Handbook of Statistical Methods gives the method for sizing a test on a proportion by normal approximation, using the significance level, the power, the baseline rate and the difference you want to detect, with a continuity correction added (NIST/SEMATECH e-Handbook, section 7.2.4.2, checked November 7, 2026). The handbook's worked example is a single-process one, detecting a rise from 10 percent to 20 percent defective. Comparing two scripts is a two-group version of the same approach, and the figures below are our own computation using the standard two-group form of that normal approximation.

Illustrative example. Your baseline conversion is 10 percent of live conversations, and you want to be able to detect a rise to 14 percent. Use a 5 percent significance level (two-sided) and 80 percent power.

  • Without correction: about 1,035 live conversations per script.
  • With a continuity correction: about 1,085 per script.
  • For a smaller rise, 10 to 12 percent, the need rises to roughly 3,840 to 3,940 per script.
  • For a larger rise, 10 to 20 percent, about 200 to 220 per script.

Note the unit: these are conversations, not dials. If only 20 percent of dials reach a person, 1,085 conversations means about 5,400 dials per script. If that is more list than you have, test a bigger change, run the test longer, or accept that the test can only detect large effects. Writing this down first prevents the common mistake of treating a weak test as a negative result.

Running the test: same windows, same agents, no peeking

  • Run both arms at the same time. Not script A this week and script B next week. Time-based differences, such as pay day, weather or a news event, would pollute the result.
  • Use the same calling windows for both campaigns, so each sees the same hours.
  • Use the same agents if you can. If agents are dedicated to one campaign each, agent skill becomes part of the result. Where your set-up allows it, let every agent work both campaigns, or swap the agents halfway.
  • Do not change anything mid-test. If you must edit a script, restart.
  • No peeking at winners. Look for operational problems if you need to, but do not stop because version B is ahead on day two. Early leads can shrink or reverse. Stop at the planned number.

Reading the result: the two-proportion test in plain words

When the planned sample is in, you have two proportions. The question is whether the gap between them is bigger than chance would typically produce. The NIST handbook sets this out as a z test: pool the two groups to get an overall rate, work out how much random variation to expect from groups of your sizes, and divide the observed gap by that expected variation. Reject "the two rates are equal" when the absolute z value exceeds the critical value for your significance level (NIST/SEMATECH e-Handbook, section 7.3.3 on testing proportions, checked November 7, 2026). For a two-sided 5 percent test that critical value is about 1.96, from the standard normal table (NIST/SEMATECH e-Handbook, section 1.3.6.7.1, checked November 7, 2026).

Worked example, illustrative. Script A: 60 conversions from 500 conversations (12.0 percent). Script B: 85 from 500 (17.0 percent). The pooled rate is 145 out of 1,000, or 14.5 percent. The z statistic is about 2.25, which is above 1.96, so the gap is unlikely to be chance at the 5 percent level.

Second example. Script A: 70 from 500 (14.0 percent). Script B: 92 from 500 (18.4 percent). The pooled rate is 16.2 percent and z is about 1.89, below 1.96. B looks better, but the test does not reach the bar. That is not proof that the scripts are equal; it means you have not shown a difference. Either run more contacts or accept that the effect, if any, is small.

The handbook uses the normal approximation when the samples are reasonably large; for small samples it points to Fisher's exact test, which works on the two-by-two table of outcomes (NIST/SEMATECH e-Handbook, section 7.3.3). A spreadsheet or statistics package will calculate both. Also report the size of the difference, not only whether it passed: a statistically significant gain of half a point may not repay a script change.

Confounders that are specific to calling

ConfounderHow it spoils a testGuard
Agent skillThe strong agents end up on one scriptSame agents on both, or swap halfway
Time of dayOne arm called in better hoursRun in parallel, same windows
List age and sourceFresher contacts answer and buy moreRandomise and check balance
Answering-machine detectionDifferent handling of machines changes who counts as a contactIdentical AMD settings in both campaigns
Abandon caps and pacingDifferent pacing changes wait time and caller moodSame dialing mode and limits
Retry rulesMore attempts on one arm raises its contact rateSame attempts and intervals

If you are testing a dialing mode or pacing change deliberately, then of course those settings are the variable, and everything else should be held fixed. The playbook explains the trade-offs between modes.

Compliance guardrails during a test

  • Every variant is an approved script. A test is not a licence to improvise claims. Both versions need the same review.
  • Required disclosures stay in. Do not test a version that leaves out identification, recording notices or other mandatory statements.
  • Consent, DNC and calling hours apply equally. Scrub the whole list before randomising, and apply the same permitted-hours rules to both arms. The calling hours, DNC and consent guide covers the settings.
  • Honest offers only. Testing two prices or promises means both must be real and available.
  • Keep records. Store the script versions, dates and groups, so you can show what was said to whom.

Running a test with two campaigns and the reports in DialerBee

The product pages do not describe a built-in split-test feature, so the method here uses what they do describe: separate campaigns and separate lists, and reporting. Create two campaigns with identical settings and dialing mode, upload one randomised half of the list to each and load script A and script B. Then compare them in the reports: the Campaign Summary report covers how a campaign performed from dials to connections to dispositions, with the Agent Performance and Lead Conversion reports beside it, and each exports as CSV, Excel or PDF. The analytics page adds campaign comparison, agent metrics and scheduled CSV exports, which suit a test that runs for days.

Use the dispositions you already track as the conversion event, so both arms are measured the same way. Export both campaigns' final numbers at the planned sample size, apply the test above and note the result with the date, the scripts and the sample.

Frequently Asked Questions

How do you A/B test a cold call script?

Split one list at random into two equal groups, give each group a different script, run both under the same calling windows with the same pool of agents, and compare one pre-chosen metric once the planned number of contacts is reached. Decide the metric, the sample size and the stopping point before the first dial.

How many calls do I need for a script test?

It depends on your baseline rate and the smallest improvement you care about. As an illustration, detecting a rise from a 10 percent to a 14 percent conversion per contact with a 5 percent significance level and 80 percent power needs roughly 1,035 contacts per script, or about 1,085 with a continuity correction. Smaller differences need far more.

Which metric should I use?

Choose one. If you are testing something that affects whether people pick up, such as the calling window or the caller ID, use contact rate. If you are testing what is said once someone answers, such as the opener or the offer, use conversion per contact. Do not pick the winner afterwards from whichever number looks best.

Can I look at the results while the test is running?

You can monitor for problems, but do not stop the test the first time one script is ahead. Repeated early looks make a chance difference appear real. Fix the sample size in advance and read the result when it is reached.

Does DialerBee have a built-in split test feature?

The product pages do not describe one. You run a test with two campaigns and two lists, split at random before upload, and compare them using the campaign, list and agent reports and CSV exports.

Is it compliant to test different scripts?

Testing does not change your obligations. Every variant must be an approved script, every contact must still pass consent, do-not-call and calling-hour checks, and required disclosures must appear in both. This is general information, not legal advice.

Sources

  1. NIST/SEMATECH e-Handbook of Statistical Methods, section 7.3.3 (z test for two proportions; Fisher's exact test for small samples). itl.nist.gov, section 7.3.3 (checked November 7, 2026)
  2. NIST/SEMATECH e-Handbook of Statistical Methods, section 7.2.4.2, sample sizes required for a proportion (normal approximation with continuity correction). itl.nist.gov, section 7.2.4.2 (checked November 7, 2026)
  3. NIST/SEMATECH e-Handbook of Statistical Methods, section 1.3.6.7.1, cumulative distribution function of the standard normal distribution (critical value 1.960). itl.nist.gov, section 1.3.6.7.1 (checked November 7, 2026)
  4. Sample size and z statistic figures in the examples were computed by DialerBee for this article and are illustrative only.

Related reading

Disclaimer: This article is general information, not legal advice or statistical consultancy. DialerBee does not provide legal advice or guarantee regulatory compliance. Calling-hour, consent and disclosure rules differ by country, and a test must follow them like any other campaign. Confirm requirements with qualified counsel.

Ready to see DialerBee in action?

Request a 15-minute live demo, or start a 14-day free trial — no slides, no commitment. Calling starts once your own SIP trunk or PBX is connected and approved.

14-day free trial · free AI credit · no card · 11 languages · BYOC · compliance-supporting controls