A/B testing
An A/B test sends two to ten versions of the same message to a slice of the audience, measures click-through per version, and then sends the winner to everyone who was deliberately held back. Assignment is deterministic, the holdback never overlaps the test group, and promotion can be manual or automatic.
All examples use https://app.openpush.ai, the OpenPush API base URL.
When to use this page
Use an A/B test when you have a real question about copy: does urgency beat curiosity, does naming the reward beat teasing it, does a shorter body get more taps. Use it on audiences large enough that a difference could be visible — a few hundred devices per arm at minimum.
Do not use it as a way to send different copy to different segments. That is just two messages.
Prerequisites
- The app's REST API key.
- An audience big enough to split. With the default 25% test share and two arms, a 10,000-device audience puts about 1,250 devices in each arm and 7,500 in the holdback.
- The receipt ladder wired up if you want click data — clicks arrive from the SDK via
/v1/ingest, not from the provider.
Creating a test
Add variants and, optionally, ab to an ordinary send.
Code
Arms are given ids server-side in order: A, B, C, and so on. You never supply an arm id.
The variants array
| Arm field | Type | Notes |
|---|---|---|
name | string | Truncated to 80 characters. Defaults to Variant A, Variant B, … |
title | string | Falls back to the message's title when omitted |
body | string | Falls back to the message's body when omitted |
languages | object | Per-language copy for this arm, same shape and same base-language fallback as a normal send |
Two to ten arms. Anything else is 400 variants must contain between 2 and 10 arms, and a
non-object entry is 400 every variant must be an object.
Liquid in every arm — including each arm's per-language title and body — is compiled and validated when you create the message, not when it sends. A syntax error in arm C's Portuguese body fails the create call with the field named.
Only
title,bodyandlanguagesactually differ per arm. An arm may carryimage_urlanddeep_linkand they will be stored on the message, but the payload delivered to a device uses the message-level image and deep link for every arm. If you need to test an image or a destination, test it as two separate messages.
The ab object
| Field | Type | Default | Notes |
|---|---|---|---|
test_pct | integer 1–100 | 25 | Share of the audience that gets a test arm. The rest is the holdback |
auto.enabled | boolean | false | Whether to promote a winner automatically |
auto.after_h | number | 24 | Hours to wait after message creation before auto-promoting |
auto.min_per_arm | integer | 100 | Minimum sends per arm required before auto-promotion will fire |
winner | string | — | Set by promotion; you do not normally supply it |
promoted_at | number | — | Set by promotion |
A test_pct outside 1–100 is 400 ab.test_pct must be between 1 and 100, and a non-integer value
is 400 ab.test_pct must be a whole percentage.
Setting test_pct: 100 means the whole audience is in the test and there is no holdback —
which is a legitimate way to split traffic evenly with nothing left to promote to. Attempting to
promote such a message returns 409 this experiment has no holdback to promote.
How devices are assigned
Assignment hashes the message id together with the subscription id and turns the result into a uniform number between 0 and 1. Devices above the test share are the holdback wave; the rest are spread evenly across the arms.
Three properties follow from that, and all three matter:
- It is deterministic. The same device on the same message always lands in the same place — on the first attempt, on a retry, on a quiet-hours release, and on a redrive after a process died mid-campaign. A device cannot receive arm A now and arm B on the retry.
- It is device-disjoint. The holdback is exactly the devices that were not in the test. Nobody gets the test copy and the winner copy for the same message.
- It re-rolls per message. Because the message id is part of the hash, a device that was in arm A last week is not systematically in arm A this week.
During the test send, holdback devices are simply not sent to — no delivery row is created for
them, and they consume nothing from their frequency cap. The message's reported Audience still
counts the full matched audience, so the funnel tells you the truth about how many people the
message is ultimately for.
Reading the results
The message report carries the experiment:
Code
Code
| Report field | Meaning |
|---|---|
testing | true while the message has variants and no winner has been promoted |
winner | The promoted arm id, or null |
significance | Confidence percentage that the two test arms genuinely differ, or null |
variants | The arm definitions as stored |
ab | The experiment configuration, including winner and promoted_at once promoted |
arm_stats | One row per arm per wave |
Each arm_stats row carries id, name, wave (test or winner), sent, accepted
(provider-accepted), clicked, failed, and ctr_value — clicks as a percentage of sends,
rounded to two decimals.
Significance
significance is a two-proportion comparison of click-through between the arms, expressed as a
confidence percentage. It is computed only when there are exactly two test arms and both have
sent at least one message; otherwise it is null.
Treat it as a guardrail, not a verdict. A reading of 96.4 means the difference between 5.56% and 8.66% is unlikely to be noise at these sample sizes. A reading of 61 means you are looking at noise, whatever the bar chart suggests. With three or more arms, no significance figure is produced at all and you are on your own.
Two honest caveats: clicks only exist if your app posts click receipts, and the comparison uses clicks over sends, not over delivered-and-displayed notifications.
Promoting a winner
Promotion sends the winning arm's copy to the holdback wave.
Code
The arm id is case-insensitive. The response is the message's delivery report after the winner wave has been queued, in the same shape as a send response.
| Error | Meaning |
|---|---|
404 unknown message | No such message on this app |
409 winner must name an existing A/B arm | The id is not one of this message's arms |
409 a winner has already been promoted | Promotion is one-shot |
409 this experiment has no holdback to promote | test_pct was 100 |
Promotion re-resolves the audience and then excludes every device that already has a delivery row for this message. That means the test group is never double-sent, and devices that joined the segment since the test went out will receive the winner. If that matters to your measurement, note the timing.
Quiet hours and frequency caps apply to the winner wave exactly as they do to any send.
Automatic promotion
Set ab.auto.enabled and the scheduler will promote for you:
Code
The rules are strict, and all must hold:
- At least
after_hhours have passed since the message was created — not since the last click arrived. - Every test arm has sent at least
min_per_armmessages. - No winner has been promoted yet.
When they hold, the arm with the highest click-through rate wins, resolved deterministically if two
arms tie. Note what is not in that list: auto-promotion does not consult significance. If
two arms are within noise of each other after six hours, it will still promote one. Set
min_per_arm high enough that the CTR comparison means something, or promote by hand.
If the conditions are never met — the audience was too small to reach min_per_arm — the message
simply stays in testing and the holdback never receives anything. Check on it.
Limits
- 2 to 10 arms.
test_pctis 1–100, whole numbers only.- Only title, body and per-language copy vary per arm. Image, deep link, custom data, action buttons, TTL, priority and collapse key are message-level and identical across arms.
- Promotion is one-shot. There is no un-promote and no second winner.
- Significance is two-arm only, and is not consulted by auto-promotion.
- The measured metric is always click-through. You cannot optimise for anything else, because nothing else is measured — there are no conversions, goals or custom outcomes on a message.
- There are no holdout groups. The holdback wave here is part of the experiment and receives the winner; it is not a permanently-suppressed control group for measuring the lift of pushing at all.
- Retrying a create-message call creates a second, independent experiment with a different message id and therefore a different assignment. There is no idempotency on message creation.
FAQ
Can I test send time instead of copy? Not as an A/B arm. Arms differ only in copy. Send two messages at different times to different halves of an audience if you need that.
What is the difference between the holdback and a holdout group? The holdback is the part of the audience saved for the winner — it is going to receive a message. A holdout group in the classical sense is a control cohort that is never messaged so you can measure the lift of messaging at all. OpenPush does not have the latter.
Do holdback devices count against their frequency cap during the test? No. They receive nothing during the test wave, so nothing is consumed. The winner wave is capped normally.
What happens if I promote while the test wave is still sending? Devices that already have a delivery row are excluded from the winner wave, so the test group is safe. But your arm statistics are still moving, so you are deciding on partial data.
Can I use best-hour delivery with an A/B test?
Yes. The audience is resolved and arms are assigned first; per-device timing is applied afterward.
Be aware that spreading a test over 24 hours means your arm statistics fill in slowly, and
auto.after_h counts from message creation regardless.
Why is arm_stats empty right after I create the test?
Rows appear per arm per wave once deliveries exist. If it stays empty, check that the audience was
not entirely capped or held.
Related
- Sending messages — the rest of the message body
- Personalization — Liquid inside arm copy
- Segments — sizing the audience before you split it
- Best-hour delivery
- Messages API reference