Multivariate testing: what your creative test can tell you

We examined the headline and image combinations within individual Upworthy website tests. From 3,212 tests meeting the main scope rules, 188 varied both elements. Most of the remaining tests varied only one of them. We kept each test separate and counted each once, regardless of how many people might have seen it. ^[2]

Only 31 of the 188 contained every combination of their observed headlines and image IDs. The other 157 were incomplete grids. That sounds restrictive until you ask what comparisons remained inside those grids. Of those 157 designs, 75 had both a same-image headline comparison and a same-headline image comparison. Another 30 had only the first kind, and 45 had only the second. Seven had neither.

Taken together, 150 of the 157 incomplete designs, or 95.5%, preserved at least one way to compare an element while holding the other fixed. This does not prove that any comparison had enough traffic, sound assignment or a meaningful result. It does show why “not every combination was tested” is too broad a reason to dismiss a design.

The table places complete and incomplete designs together. Across all 188, 106 contained both comparison types, 30 only headline comparisons, 45 only image comparisons and seven neither. “Both” means at least one of each. It does not mean that every headline or image can be compared on equal terms.

The original archive paper already explains that editors often chose packages without testing every possible combination. Our contribution is this inventory of the comparisons those choices left available. We cannot reconstruct each editor’s objective or judge whether a design met it. ^[1]

Imagine a pottery studio promoting a weekend workshop. It has two headlines: A, “Make your first ceramic bowl,” and B, “Spend Saturday at the pottery wheel.” Image 1 shows a finished bowl; Image 2 shows the wheel. These are illustrative choices, not Upworthy examples or measured campaign results. Keep the offer and other displayed copy the same while considering three possible designs.

First, the complete grid contains A with Image 1, A with Image 2, B with Image 1 and B with Image 2. The team can compare the headlines while holding Image 1 fixed, then make the same headline comparison with Image 2. It can also compare the images under each headline. Four cells provide these opportunities; they do not establish how much evidence will be collected in each cell.

Second, the partial grid contains A with Image 1, B with Image 1 and B with Image 2. A with Image 2 is absent. The team can compare A and B with Image 1 held fixed. It can separately compare the images while holding headline B fixed. The missing cell does not erase those two comparisons.

But the partial grid leaves another question unanswered by a direct four-cell comparison: would the A-versus-B difference look the same with Image 2? There is no A-with-Image-2 result to use. Nor can the team identify the best of all four possible packages just by choosing among the three it observed. The untested package could behave differently.

Third, the diagonal design contains only A with Image 1 and B with Image 2. Neither element stays fixed between the two packages. A valid test could still support choosing between those two finished versions. It would not separately identify a headline contribution or an image contribution from that comparison alone.

There is no universal winner among these designs. If the studio must choose between two approved concepts for a specific launch, comparing the packages may fit its task. If it wants to reuse a headline across images, it needs evidence relevant to that broader choice. Additional combinations earn their place by answering a question the team actually has. ^[3]

The archive makes this distinction concrete. Of the 106 designs containing both comparison types, 74 did not contain even one complete two-headline, two-image rectangle. A design can therefore offer a headline comparison somewhere and an image comparison somewhere else without supplying the four cells needed for a direct comparison of their differences.

Consider the studio’s partial grid again. Comparing A and B with Image 1 asks one question. Comparing Image 1 and Image 2 under B asks another. Neither tells us whether the headline difference itself changes when the image changes. That third question is about an interaction: whether the relationship between one element and the response depends on the other element.

In the 188-design set, 32 contained at least one complete rectangle. That includes all 31 complete grids and one larger incomplete grid. Having such a rectangle supplies a place to form a direct interaction contrast. It does not show that an interaction exists, and it is not the only possible basis for studying interactions under other designs or modeling assumptions.

For a campaign team, the distinction belongs in the brief before it belongs in a statistical model. “Compare these two headlines with this image” is a narrower request than “find a headline that works across our images.” A result answering the first question should not quietly become the answer to the second.

The main inventory excludes tests whose creation-date span touches June 25, 2013 through January 10, 2014. Archive researchers identified possible assignment problems during that period. Those concerns matter for cause-and-effect research; they do not make the combination records unusable for a structural inventory. We therefore also repeated the comparison with that period included. ^[4]

The broader scope contains 239 tests varying both elements. Forty contain every combination, while 199 are incomplete. Among those incomplete designs, 180, or 90.5%, retain at least one comparison holding one element fixed, and 93, or 46.7%, retain both kinds. The corresponding main-scope figures are 95.5% and 47.8%. The exact shares move, but incomplete designs still commonly preserve a specific comparison.

We also checked a text-handling choice. The main analysis compares headline and excerpt strings exactly. Removing boundary whitespace and collapsing repeated whitespace produces 189 both-varying tests, with 31 complete grids, 108 containing both comparison types and 32 containing a complete rectangle. This check changes which designs qualify, so its counts are shown with their own denominator. ^[5]

Median coverage is 50% in all three scopes. Coverage is the number of observed headline-image combinations divided by the number possible using that test’s observed headlines and image IDs. The median is a description, not a quality score. A missing combination may have been irrelevant, unsuitable or outside the editor’s intended comparison; the archive does not settle why it was omitted.

For the pottery studio, start by deciding what will happen after the test. Will the team choose one finished workshop promotion, keep one image and revise its headline, or carry a headline into several future designs? Each decision calls for a different set of comparisons. A larger grid is not automatically a better use of the team’s effort.

If the decision concerns finished packages, list those packages without promising an explanation of each component. If the decision concerns a headline, identify the image that will stay fixed. If the team needs to examine whether a headline difference changes across two images, include the relevant four combinations or obtain a justified alternative design. Keep this separate from deciding the response measure, allocation and observation period.

Check whether the proposed combinations make sense before adding them. An offer-specific headline may belong with only one product image. Creating an unsuitable pairing simply to fill a grid would change the task. The team can instead narrow the question or develop versions that can sensibly be paired. A complete grid is useful when its comparisons serve the decision, not because a filled table looks more thorough.

Use the following fields as a short planning brief. The bracketed text is for your own campaign, not a finding from this study.

Decision after the test: [choose a finished package; compare a named element in a specified setting; examine a specified combination of elements].

Elements and versions: [exact headline texts; image files or stable IDs; any other changing content].

Included packages: [the headline-image pairs that will actually be shown].

Comparisons needed: [which pairs hold an element fixed; which four cells support the interaction question, if relevant].

Evidence plan: [response measure; audience and assignment; exposure conditions; planned observation and review rule].

Conclusion boundary: [what the design could support; which untested combinations or settings remain outside the decision].

For example, a studio using the three-cell partial design can write: “We will compare the headlines with Image 1 and compare the images with headline B. We are not testing headline A with Image 2.” That sentence keeps two useful questions and the missing comparison visible without declaring the design inadequate.

A creator preparing work for a client can use the same distinction when discussing a revised thumbnail and title. The question is which version or element the client wants to choose. Our counts come from a publisher’s website, however, and do not establish how a current social platform assigns or measures creative variants.

The source is the publicly released exploratory file of the Upworthy Research Archive, described by J. Nathan Matias, Kevin Munger, Marianne Aubin Le Quere and Charles Ebersole. It contains 22,666 package rows grouped into 4,873 tests from the historical 2013–2015 archive. We used the headline, image identifier, excerpt, test identifier, creation timestamp and archive row index. Clicks, impressions and editorial selections were not used. ^{[1] [6]}

A structural feasibility check preceded the final design-audit plan and informed its question. The audit is therefore an exploratory reanalysis, not a preregistered confirmatory study. The specified rules retain tests with usable structural fields, apply the main date restriction, and require identical excerpt text across their packages. The excerpt restriction is conservative because the paper and archive documentation disagree about whether that field was displayed. ^[6]

The selection table shows successive exclusions, followed by a breakdown of the eligible tests. Five tests lack a headline or image ID; 1,050 then fail the date-scope rule and 606 have varying excerpts. The 3,212 retained tests include 1,624 varying only headlines, 1,078 varying only image IDs, 188 varying both and 322 with a single observed combination. The last category can still contain multiple source rows.

Within a test, repeated identical headline-image pairs count once as design cells. Five of the 188 both-varying tests contain repeated cells. A complete grid contains every pairing of the headline strings and image IDs observed in that test. Different strings need not express different ideas, and different image IDs need not represent visibly different pictures; the image files are unavailable.

The results count designs equally, not people, impressions or businesses. They do not establish present-day marketing practice, performance or statistical power. The downloads provide the aggregate results and reproduction method, including both scope checks, so readers can inspect the distinction between an observed combination and a conclusion drawn from it. ^[7]