Email A/B testing: what changed after the first hour

At 60 minutes, 4,712 of the 5,743 eligible pairs had a lead: one version had a higher recorded click measure than the other. At 1,440 minutes, or 24 hours after each version’s send, 3,510 still favored the same version. Another 991 favored the other version, while 211 were tied. That is 74.5% retained, 21.0% reversed and 4.5% tied among the pairs that started with a lead. ^[2]

These are comparisons of a particular recorded measure. We divide each version’s unique-click counter by its reported sent emails and multiply by 100. The resulting unit is recorded clicks per 100 sent. A lead means a higher value of that measure, however small the difference. It does not mean a statistically significant result.

The majority is worth keeping in view. This archive does not show that first-hour reports were generally overturned. It shows that persistence, reversal and a later tie all occurred. A report that records only the early ordering loses the distinction between those later paths.

A first-hour report can accurately describe the information recorded then even when the later ordering differs. Calling a reversal a “false winner” would go further: it would treat the later ordering as truth and assume the early report claimed more than a current lead. Here, both observations are retained. The comparison shows a change in recorded ordering, not an error rate for a decision system.

The source is also narrower than the general phrase “email A/B testing” suggests. Of the eligible pairs, 5,647 were labeled sender name and 96 were labeled topic. Sender-name tests therefore account for 98.3% of this analysis. The findings should not be presented as a subject-line benchmark or an estimate for every email platform.

The chart below uses all 5,743 pairs as its denominator, including the 1,031 that began tied. This makes every observed path visible. The 991 reversals are 17.3% of all pairs, but 21.0% of the 4,712 with an early lead. Both numbers are correct; they answer different questions.

A tied first-hour report has no leading version to retain or reverse. In this archive, 588 of the 1,031 early ties later had a lead, while 443 remained tied. That is 57.0% of early ties. It should not be added to the reversal percentage, which starts from a different group.

Many early ties also contained little recorded activity. In 655 of the 1,031 tied pairs, both first-hour unique-click counters were zero. Other ties could contain positive counters with exactly equal ratios. We classify equality from the underlying counts and sent totals, not from rounded percentages on a chart. ^[3]

For a reporting team, this separates “there is no lead in the recorded measure” from “the two versions perform equally.” The former is an observation at a stated age. The latter would require evidence this descriptive comparison does not provide. The later separation of some ties does not establish which version should have been selected earlier.

The denominator can be checked with a short calculation using the actual archive counts. Divide 588 by 1,031 to describe what happened to early ties. Divide 991 by 4,712 to describe reversals among early leads. Divide either count by 5,743 only when the question concerns all eligible pairs. A report heading should make that starting group explicit.

The 991 reversed pairs had a median absolute gap of 0.062 recorded clicks per 100 sent at the first hour and 0.110 at 24 hours. These are differences between the two versions’ recorded measures, not additional people, attributed purchases or percentage changes in revenue. ^[3]

The sent denominator matters when reading that gap. A difference in raw click counters is not enough to compare versions with different sent totals. Each counter must be divided by its own reported sent total first. Nor should a gap of 0.110 in this measure be described as a 0.110% relative improvement: it is an absolute difference on the per-100-sent scale, with no sales value attached.

The size of the later gap varied. For reversed pairs, the middle half lay between approximately 0.025 and 0.336 recorded clicks per 100 sent. The median therefore describes a typical position in this group; it does not mean every reversal was small. An ordering can change across a small gap, and some recorded gaps were much larger.

The other paths provide useful context. Pairs that retained a lead had a median later gap of 0.291. Pairs that moved from a tie to a lead had a median later gap of 0.444. A later tie has an exact gap of zero by definition. The table keeps all five paths together rather than highlighting only the reversals.

The early and later columns are separate medians within each row. They do not trace one typical pair, and subtracting them does not give the median change across pairs. Read them as summaries of the gap at each checkpoint. To describe the direction of the comparison, use the path label alongside the size.

These groups are defined by what happened at both checkpoints. Their median gaps cannot be used as prospective rules for spotting a reversal in a new campaign. In particular, the smaller early median among reversed pairs is not a validated cutoff for deciding when a report is safe to act on.

Count context belongs alongside the gap. Among all 4,712 pairs with a first-hour lead, the median combined unique-click counter across the two versions was nine. In 488 of those pairs, the combined counter was exactly one. That describes the starting evidence across early leaders; it does not show that those particular pairs caused the reversal total or that low counts explain the changes.

We fixed additional comparisons before calculating outcomes. Starting at 30 minutes instead of 60 produced 4,507 pairs with an early lead. Of those, 24.3% reversed by 1,440 minutes. The first-hour result was 21.0% among 4,712 pairs with a lead. Their different denominators matter: some pairs tied at one checkpoint but led at the other. ^[4]

This does not demonstrate that waiting an extra 30 minutes prevents a particular share of wrong decisions. The two calculations describe observed orderings at different ages. They do not randomly assign a waiting policy or evaluate the consequences of a rollout decision.

We also restricted the first-hour comparison to the 686 pairs whose two versions had identical recorded send timestamps. Of the 560 with an early lead, 19.6% reversed later. This check reduces one timing difference in the recorded data. Matching timestamps do not prove simultaneous delivery or equivalent audience exposure.

Some client IDs contribute many more pairs than others. The main calculation gives each pair equal weight. A separate check first finds the reversal share within each of the 403 client IDs with at least one first-hour lead, then gives each such ID equal weight. The average is 19.2%. This describes the mean of their reversal shares, not the percentage of clients that experienced a reversal. ^[4]

Taken together, the fixed checks preserve the practical issue: an observed early lead can persist, reverse or become a tie. They do not produce a universal reversal probability. The archive has repeated records within client IDs, and those IDs are not a verified sample of independent businesses.

A useful report starts with the decision being made. Is the recipient choosing the next send, documenting a completed test, or reviewing past tests for a planning meeting? A first-hour observation may be relevant to each task, but a report prepared for one can leave out information needed for another.

Name the tested variable and the measure before naming the leader. Here, that means sender name or topic, followed by the unique-click counter per 100 reported sent. A team using delivered emails, verified human clicks or purchases should state that definition instead. Keeping the measure visible also makes later comparisons easier to interpret.

Next, put the checkpoint beside the lead or tie. Include the difference between the two measures and the underlying counter and sent totals for both versions. The person reading the report can then see a direction, a gap and the recorded activity behind it, rather than having to infer all three from a highlighted row.

An illustrative handoff sentence could be: “At [checkpoint], [version or tie] describes the current ordering in [defined measure]; the gap is [value], with [counters and sent totals] behind it.” The brackets are fields to complete from a real report, not campaign results from this study. A separate sentence should record the action chosen and the reason for it.

If a later checkpoint is available, retain the earlier observation and show whether the lead held, reversed or became a tie. This lets a retrospective review distinguish what was known at the time from what was recorded later. Include pairs that began tied as well, unless the report explicitly concerns only pairs that already had a lead.

Choosing a version still depends on the campaign’s objective, delivery constraints, measurement quality and the cost of either choice. The archive does not set that rule. It supplies a useful reason to avoid letting one word, “winner,” stand in for the observation and the decision together.

Earlier research already treats email performance as something that unfolds over time. Bonfrer and Drèze’s “Real-Time Evaluation of E-mail Campaign Performance,” published online in 2008 and in Marketing Science in 2009, describes a predictive method and a sequential decision framework, evaluated on 25 campaigns. The general idea that early and later results can differ is not new. ^[5]

This analysis answers a narrower descriptive question using a different archive: how often did recorded orderings persist, reverse or tie in the eligible pairs, and what gaps accompanied those paths? It keeps early ties visible and checks whether the account changes under fixed timing and weighting choices.

That distinction matters when reading the findings. A predictive model tries to estimate what will happen next under its assumptions. Our tables describe what the released records contain at specified ages. The observed percentages can inform how a team explains a report; they are not a model for selecting the next campaign’s version.

SendGuard900K contains 908,229 released campaign records. Its source description declares production metadata from 2022–2024. We screened 6,801 groups identified by the combination of client ID and test ID, then retained 5,743 groups with exactly two eligible observed test-part records and positive reported sent counts. Final-rollout records were excluded from the pair comparison; their presence was not required. ^{[1] [6]}

The retained pairs span 473 client IDs. Exactly two eligible observed records does not establish that every original experiment had only two configured variants, that its archive is complete, or that audiences were randomly assigned. We therefore describe observed pairs rather than a verified collection of complete randomized experiments.

For each version, the calculation uses its unique-click counter at 30, 60 or 1,440 minutes and the same archived reported-sent denominator. Each checkpoint is measured relative to that version’s send. A primary pair need not share a wall-clock send time. The same-timestamp check is a separate restriction, not an assumption applied to every pair.

All required counters in the eligible pairs were available, valid and nonnegative; none exceeded its reported sent count. No counter decreased across the three checkpoints. These checks support the recorded arithmetic, but cannot establish the meaning of a unique click, bot filtering, delivery, or whether later observations capture every eventual response. ^[6]

We determine leads and ties by exact comparisons of the underlying ratios before rounding. Gaps are absolute differences in recorded clicks per 100 sent. Medians and other quantiles use the nearest-rank rule: sort the values and select the stated rank, without interpolation. The figures round percentages to one decimal and median gaps to three; the figure download retains six decimal places where applicable.

The method fixed the eligible structure, checkpoints, five paths, count and gap summaries, timing checks and equal-client calculation before outcomes were read. All fixed aggregate tables remain in the research download, including zero counts and quality checks. Independent source reproduction matched the complete aggregate outputs. These are descriptive archive results, with no confidence intervals, significance tests or claim of representative industry sampling. ^[7]