How routing learns

Routing picks the variant of each slot that a visit sees, and learns from your conversions which variants work, and for whom. This page explains enough to predict what it will do and to decide well.

A contextual bandit

Routing is a contextual multi-armed bandit, one per page and objective:

The arms are a slot’s variants.
The reward is whether the visit converted.
The context is what RoutingCat knows about the visit.

A Bayesian model estimates how likely each variant is to convert, and how sure it is. The model is hierarchical: a slot’s variants share what they teach about each other, so an early run of luck is pulled back toward the rest.

Routing then uses Thompson sampling. Each variant gets visits in line with the probability, under the model, that it is the slot’s best for that kind of visitor. So exploring and exploiting are one rule: an uncertain variant keeps getting tried, and a variant that is likely worse fades.

A small random slice and a control keep measurement honest.

What it learns from

Each page’s objective, on that page. A page learns from its own visits and its own objective. What the pricing page learns doesn’t carry over to the home page, and outcomes tracked beside the objective never steer it.
Visits whose outcome is known. Routing learns once a day on Hobby, and on Pro as conversions mature, at most hourly, from visits whose window has closed. With the default 24-hour window, a visit starts to shape routing a day or two after it begins on Hobby, about a day after on Pro. An early conversion waits like the rest, so a variant that converts fast gets no head start for it.
From the first day. On a new page, every variant starts on an equal footing, and routing learns from the first matured visits.

Previews and automated traffic, such as crawlers, are left out. Changing a page’s objective in code starts its learning over.

It favors the likely best, and keeps exploring

Under Thompson sampling, each variant gets a share of visits in line with how likely it is to be the best. A variant that looks better gets more; an uncertain one keeps being shown until the evidence settles; one that looks worse fades, though floors keep every variant in play. On top of that:

a small random slice of routed visits, 5% by default, is split evenly, whatever routing believes;
the control, 10% of visitors by default, always sees the defaults and is never routed.

So shares change as evidence arrives: once a day on Hobby, up to hourly on Pro. A variant stays in routing until you pause or retire it.

It learns for whom

Visitors don’t all respond alike. When some kinds of visitor respond differently from others, routing can learn it and show them different variants: for instance, visitors on phones might do better with a shorter headline. This is the bandit’s context. The model holds such differences to a higher bar than overall ones, so each must earn its place with evidence. Until it does, everyone is routed alike, so a few lucky conversions don’t split your traffic.

With personalization off, a slot routes everyone alike, whatever the evidence.

Several slots at once

Each slot’s variant is picked separately, and all of a page’s slots learn together, from the same visits and the same objective. So several ideas on one page are tried at once, without splitting traffic between tests. The model adds up the slots’ effects, so it doesn’t learn that a headline works only beside a particular image. Pieces that must match belong in one slot: see Pages, variants and versions.

New variants start cautiously

A new variant starts from what the page’s other variants taught routing, never from a guess that it is better. It gets a limited share at first, which opens up as its own evidence builds, so a promising new variant can trail for a while. Its label reads prior until routing has learned from its own visits, and can add capped while its share is held back.

What an edit carries over

An edit under the same key builds on what the variant already showed, and isn’t held back like a new variant: fixing a strong variant doesn’t send it back to a cautious start.
A new key starts cautiously, however close its content is to an existing variant’s.
One declared parent gives a new variant a head start from part of what its parent showed, never all of it, since a child can still do worse.

This is why a fix belongs under the same key and a new idea under a new one: see Same key or new key.

What learning survives

A promote changes the default and what the control sees, never what routing has learned.
Turning routing off, for the whole site, serves the defaults everywhere; turning it on again resumes from what was learned.
A change of objective is what starts a page’s learning over.

Settings that change routing

Two settings change routing for one slot, in code or with npx routingcat set:

A fixed share takes part of the slot’s routed visits out of routing: that part is split evenly, or by weights you declare, and each visitor in it keeps the same variant across visits, even as variants are added, paused or removed. At 1, the slot is a classic A/B test. It is the right choice for prices: under a conversion objective, routing would drift toward the cheapest price.
Personalization off routes every visitor of the slot the same way. Routing still favors the likely best and keeps exploring; it only stops learning for whom.

Routing settings gives each setting with its range and default.