Ecommerce merchandising: test your product-position assumptions

A product-card comparison should help you separate the item being offered from the way it was presented. If one product usually appears first and another usually appears last, their totals alone do not answer whether shoppers prefer one offer.

The simple approach we test gives every step down the list the same adjustment to the model’s underlying preference score. The useful question is whether it predicts choices well enough, compared with allowing different positions to have different adjustments.

We tested those two approaches using a released book-choice experiment. The result is relevant to businesses and creators comparing products on a collection page or digital storefront. It supplies a way to examine a measurement assumption, rather than a rule for where to put your best-selling item.

This matters when an offer appears in more than one place. A creator might feature the same download in a collection and a launch page. A retailer might change the order of a category list. The item, price, audience and presentation can differ together. A position adjustment is an assumption about part of that comparison, and it deserves its own check.

Giovanni Compiani, Ilya Morozov and Stephan Seiler asked Prolific recruits to make hypothetical choices from ten e-books, with randomized order and prices. Participants had to select a book; there was no option to buy nothing. We use the first task within each retained submission, not the later task that removed the chosen book.^[1]

The authors already report responses to rank and price. Their wider research studies how text and images help estimate product substitution. Our question is narrower: does replacing a single position term improve prediction for participants whose choices were kept out of model fitting?

Following the original eligibility rules leaves 9,265 submissions with 9,264 distinct Prolific participant IDs. One ID appears twice. We keep both submissions together and give that participant the same total weight as every other participant. A submission is not automatically a separate person, and a first task is not necessarily that person’s first exposure to the experiment.^[2][3]

The retained source dates run from June 11 to June 26, 2024. The manuscript describes June 14–27, so we disclose the difference and apply no date exclusion. Country and age eligibility were not established in the reviewed materials; these are not estimates for U.S. shoppers or another national population.^[1][2]

The flexible position model predicted held-out choices better on the prespecified scoring measure. Both models account for which book was offered and its displayed price. They differ only in how they represent position.

The baseline uses one coefficient for each step down the list. The flexible model uses separate adjustments for positions two through ten, relative to the first. That gives it more freedom, but also more parameters to estimate. Better fit to its own training data would not, by itself, settle the comparison.

We therefore fitted both models using 6,176 participants and evaluated them on the other 3,088 participants. Every submission from a participant stayed on the same side of that split. The test group contains 3,089 submissions because it includes the repeated participant.

To compare predictions, we used a negative log score. A model receives a smaller penalty when it assigns more probability to the choice a person actually makes. It receives a larger penalty when it assigns very little probability to that choice. Lower average negative log scores are better.

The held-out average was 2.2260 for the baseline and 2.2154 for the flexible model, in natural-log units, also called nats. These are prediction scores, not percentages of correct answers. Training and test scores are shown together so the comparison does not rely on the easier task of describing data the model has already seen.

The paired improvement was 0.0106 nats per participant, with an approximate 95% interval from 0.0044 to 0.0168. Here, a positive improvement favors the flexible model. The result is small on this scoring scale, but the interval lies above zero.^[4]

This interval treats the two fitted models as fixed and assumes independent participant groups. It does not include uncertainty from fitting the models on another training sample. The finding supports a predictive improvement in this selected historical task; it does not turn the score difference into a sales increase.

The complete position comparison shows why the overall score should come with detail. For each position, we compare its observed choice share with the average probability assigned by each model. These are equal-participant averages in the held-out group, not click-through rates.

At position one, the observed share was 17.16%. The baseline predicted 14.64%, while the flexible model predicted 17.73%. At position ten, the observed share was 9.18%, compared with predictions of 6.37% and 8.74%. The flexible model was closer at both ends of this particular list.

The flexible model was not closer everywhere. At position four, the observed share was 10.91%. The baseline predicted 11.21%, while the flexible model predicted 9.26%. The extra flexibility improved the overall held-out score while leaving a larger discrepancy at that position.

The next table shows observed minus predicted shares. A positive gap means the model predicted too little choice at that position; a negative gap means it predicted too much. Keeping all ten positions visible prevents a good-looking endpoint from standing in for the whole comparison.

These position rows are descriptive, with no separate significance claims. The experiment did not measure eye movements or attention, and this analysis does not identify why a position differs. Nor does a curved pattern of observed shares, on its own, refute the baseline: its position term is linear in the model’s underlying preference score, not in choice percentages.

Simply subtracting the first and last observed shares would answer a different question. Those shares summarize choices across products and prices. Both fitted models account for book identity and price before assigning probabilities across the ten options. Neither the endpoint difference nor the model comparison tells you what would happen if you moved a particular product on a live store.

Before comparing offers, write a short measurement brief. The following template is ungated and illustrative. It applies the study’s lesson about checking assumptions; it does not promise that the same model will suit your storefront.

Decision: “We need to decide [which product comparison or presentation change] for [page or campaign] during [period].” Name the decision before choosing a favorable metric.

Outcome: “We will measure [choice, click, add-to-cart or purchase].” Select the outcome your records actually support. Keep the offered alternatives, their order and displayed prices alongside it. A forced-choice result should remain separate from a purchase result.

Unit: “Our unit is [participant or another clearly defined unit]. Repeat visits or submissions will be handled by [rule].” Keep the same person’s observations together when evaluating prediction for new people. Do not treat an anonymous session identifier as proof of a unique person.

Comparison: “Both models will use the same products, price information and eligible observations. They will differ in [the position assumption].” Specify the simple alternative as well as the more flexible one.

Evaluation: “We will reserve [a defined group] before fitting and compare [a named scoring measure]. We will show all positions.” Decide how to report an unresolved or worse result before seeing it.

Use: “This result will inform [measurement decision]. Evidence about [commercial outcome not measured here] is still needed before [commercial decision].” This last sentence keeps a prediction exercise from quietly becoming a revenue forecast.

Consider a fictional creator selling a writing workbook, a workshop and a bundle. The bundle appears at the top of a launch page and receives the most clicks. The creator is considering rewriting the other product descriptions.

The measurement brief changes the immediate question. The observed outcome is a click, not a sale. The available records need to show which offers were displayed, in what order and at what prices. If the bundle always occupies the first slot, those records cannot cleanly separate that bundle from that position.

The creator’s completed brief could read: “We want to know whether our way of accounting for position predicts clicks from new visitors. We will compare the same offers and price information, keep repeat observations together, and report the full position comparison. We will make no claim about purchases from click records.” This is a fictional example of a measurement question, not a proposed result.

A sensible next step is to plan a comparison with enough variation to distinguish product and position, while retaining the same outcome definition. The creator could then compare a simple position adjustment with a more flexible one on a reserved group. This would be a new study of that page; the book experiment supplies no expected click rate for it.

If the flexible approach predicts the reserved group better, the creator has a reason to revisit how the report accounts for placement. That still does not show that moving the bundle will increase revenue. A commercial decision would require evidence on the actual presentation change and the outcome that matters to the business.

If the comparison is unresolved, the creator should say so. Additional flexibility is not automatically a better business tool. The practical gain is a clearer sequence: understand what the records can separate, test the measurement assumption, then decide whether another experiment is needed.

We used the authors’ edited response release and original selection rules. The first 553 pilot or test records are excluded by their fixed file position. The remaining frame contains 10,775 submissions. The comprehension filters leave 9,344; the completion filter leaves 9,265.^[2]

Consent, finished-status and the minimum 60-second duration rule remove no additional submissions at their respective steps. Completion and duration are post-exposure filters, so the retained group is a selected sample.

Prolific participant IDs define the groups. A deterministic hash split assigns two-thirds of the groups to training and the rest to test, without using their choices. Within each group, every retained first-task submission shares that participant’s contribution equally. The same weighting applies to fitting, scoring and the position tables.^[3]

Both models are unpenalized multinomial logits with nine book indicators and one linear price term. The baseline adds one linear position term, for 11 parameters. The flexible model adds nine position indicators, for 19. Position one and the numerically smallest product ID provide the reference categories. All eligible choice sets contain the same ten products, with selected ID, price and position checked for agreement.

We fit the models on the training group, retain the fitted coefficients, and then score the test choices. The interval uses the standard deviation of the paired participant score differences, 3,088 participant units and 3,087 degrees of freedom. It is an approximate Student-t interval except under its exact normal-score assumptions. All ten position comparisons remain descriptive.^[4]

The baseline’s primary optimizer returned a numerical warning. It still met every accuracy condition specified before fitting, including a second-start comparison. We retained that original fit; the detailed diagnostics are included with the reproduction materials.^[5]

The downloads provide the displayed results, original analysis code and aggregate numerical output. They exclude source response rows, participant IDs and book artwork. Obtain the source data separately from the authors’ release. Influence IQ is responsible for this reanalysis and its interpretation.