Blog posts by Piotr Nowak - Optimizely & Azure

Content Variations in CMS 13, Part 3: Audiences vs Audiences

Sun, 14 Jun 2026 17:46:48 GMT

Executive summary. Part 2 left the experiment running against Everyone. Real projects don't look like that. So this part wires those same CMS Content Variations to two rival audience engines and measures what each actually does. Every number below comes from a live CMS 13.1.0 + FX SDK instance. A targeted delivery served variant_a to 60 of 60 mobile visitors; at 100% allocation that is deterministic, so a single miss would be a bug rather than noise. The desktop experiment held its 33/33/34 split (χ² = 1.465, n = 300). The MVC head and the headless Next.js head agreed on the arm and the rule for 20 of 20 visitor/device pairs, with zero coordination code. The same runs found the boundary. CMS Audiences evaluate only inside the CMS runtime: a personalized block vanished from Optimizely Graph entirely, invisible to every headless consumer, and nothing threw an error. If your requirement says “both heads”, the audience decision is made before anyone asks about features.

Parts 1 and 2 built the machine: Content Variations as the quiet hero of CMS 13, then Feature Experimentation as its stats engine, with one string, VariationKey == variation name, as the entire integration contract. This part answers the question that contract postpones: who decides who the visitor is? CMS 13 has an answer. FX has a different answer. Both are called Audiences now, and they are not the same thing. Everything below ran on CMS 13.1.0 with the C# SDK 4.3.0 and the JavaScript SDK 6.4.0. Every number is from those runs.

In this part: the Audiences naming collision · the three-layer opt-in (and the startup self-check it forced) · the five-attribute pipeline with native geolocation · targeted delivery semantics the docs bury · the coexistence proof and the Graph void · the day the headless head went dark · three QA override levels · a troubleshooting runbook · eight sharp edges ranked by blood · FAQ · glossary.

The machine at a glance

One picture before the prose. Both heads serve the same experiment from the same definitions. The only shared state is configuration-as-data:

Both heads serve the same experiment from the same definitions; only the Graph path drops the personalized item. The CMS Audience layer has no representation on the headless route.

The CMS Audience layer simply has no representation on the Graph path. The rest of this article is the evidence for every arrow.

Two engines, one word: both of them are called Audiences now

Start with the vocabulary trap, because it will find you anyway. Visitor Groups have been rebranding to Audiences since the CMS 12 admin redesign. CMS 13 finishes the job: the admin package describes itself as the “audiences management UI”, the docs say “an audience (formerly called a visitor group)”, and the old name survives mostly in role names and API types. Feature Experimentation has shipped Audiences for years. Same word, two engines, and they answer the same question with opposite architectures:

	CMS Audiences (formerly Visitor Groups)	FX Audiences
Evaluated by	The CMS, server-side, during rendering	Every SDK, in-process, at Decide()
Evaluation input	IPrincipal + HttpContext, full request, roles, visit history	Only the attributes your code passed at context creation
Definition lives in	CMS admin (VisitorGroupAdmins role)	FX dashboard, ships in the datafile
Personalizes	A fragment, one content-area item	A page version, via the variation key contract from part 2
Works headless	No, and the docs say so out loud	Yes, the evaluation is a pure function, portable by construction
Measurement	“Enable statistics” view counts, with no exposures, no conversions, no significance	The stats engine: exposures, conversions, significance

The last two rows are the ones that matter. A CMS Audience can reach into everything the CMS knows about the request, and pays for it by existing only where the CMS renders. An FX Audience knows nothing you didn't tell it, and pays for that with a pipeline you must build. But the function (datafile + attributes) → bool runs identically in C#, in Node, in anything with an SDK. Keep that trade in view. Every decision below falls out of it.

One governance row to add to part 2's who-does-what table: audience definitions get owners too. CMS Audiences belong to whoever holds VisitorGroupAdmins. FX Audiences belong to the dashboard. Attribute keys, though, are an API contract between the codebase and the dashboard: a developer renames device and the marketer's audience quietly stops matching. Write them down somewhere both sides actually read.

Switching the engine on: in CMS 13, personalization is opt-in three layers deep

Part 2's sharp-edges list opened with a package that crashed CMS 13 at startup. This part's equivalent is gentler and stranger: in CMS 13, Visitor Groups are not in the box you already have. The EPiServer.CMS metapackage ships neither the evaluation core nor the UI. The install docs state the philosophy plainly: “every NuGet package your project references must have its services explicitly registered, or the application fails at startup.”

The documented pair of calls is services.AddVisitorGroupsMvc().AddVisitorGroupsUI(), which also adds the EPiServer.CMS.UI.VisitorGroups 13.1.0 package.

I learned what each one carries the honest way, by registering less than that. AddVisitorGroupsUI() alone boots clean. Then the Audiences screen throws Unable to resolve service for type 'IVisitorGroupCriterionRepository' the moment an editor clicks it: the screen's own API services, the ones that load criterion lists and persist rules, resolve against registrations the UI package does not carry. A UI that renders is not a UI that saves. Registering the two internal layers that error message points at fixes the admin screen. It also quietly skips the rendering layer, so personalized blocks would render for everyone. That's the worst possible failure mode: a working configuration surface for a filter that doesn't run. AddVisitorGroupsMvc() wraps all of it: repositories, the built-in criteria, the content-area rendering filter, and the “View as Audience” impersonation service.

Notice what the docs promised and what actually happened. The promise: misregistration fails at startup. The observation: it failed at click time, with a green boot log. So the demo project now resolves the services the Audiences API needs at startup, in Development, and logs every registered criterion: DeviceCriterion, DisplayChannelCriterion, … TimeOfDayCriterion, … (22 of them), making the documentation's promise true at exactly the layer where it broke. Twenty-two criteria, including RoleCriterion and UserProfileCriterion that the docs list doesn't mention, plus DeviceCriterion, which is ours. The whole class fits on a slide, and it earns the line this demo is built around: one definition of “who you are”, two engines reading it.

[VisitorGroupCriterion(
    Category = "Technical",
    DisplayName = "Device class",
    Description = "Matches the visitor's device class (mobile / tablet / desktop) "
                + "using the same User-Agent heuristic that feeds the FX 'device' attribute.")]
public class DeviceCriterion(IVisitorAttributesProvider attributesProvider)
    : CriterionBase<DeviceCriterionModel>
{
    public override bool IsMatch(IPrincipal principal, HttpContext httpContext)
    {
        // Editors type the value by hand - trim + lowercase forgives "Mobile "
        // (the FX exact matcher would not; see sharp edge #1).
        var expected = Model.Device?.Trim().ToLowerInvariant();
        return !string.IsNullOrEmpty(expected)
               && attributesProvider.GetAttributes().TryGetValue("device", out var device)
               && Equals(device, expected);
    }
}

Constructor injection works (the built-in geographic criteria take IClientGeolocationResolver the same way), [VisitorGroupCriterion] registers the class through plugin scanning, and the model is CriterionModelBase with one string property and a Copy() => ShallowCopy(). The built-in OSBrowserCriterion could classify devices without custom code. Ours exists for consistency with the FX attribute, not for extra capability.

Reference card: the CMS 13 personalization bill of materials

Call	Package	What it actually registers
AddVisitorGroupsMvc()	EPiServer.Cms.AspNetCore.Mvc (already referenced)	The whole working engine: repositories and statistics, role infrastructure, the built-in criteria, the content-area rendering filter, fragment handlers, View-as-Audience impersonation
AddVisitorGroupsUI()	EPiServer.CMS.UI.VisitorGroups (add it; not in the metapackage)	The Audiences management screen (protected module VisitorGroups.zip) and its API controllers
AddCmsClientGeolocation(o => o.LocationHeader = …)	EPiServer.Geolocation (already referenced)	Header-based geolocation: feeds the geographic criteria and anything else consuming IClientGeolocationResolver
[VisitorGroupCriterion] on your class	your project	Custom criteria via plugin scanning; explicit AddVisitorGroupsCriterion<T>() only if you disable scanning

And the criteria catalog as measured on 13.1.0, the self-check log, deduplicated and grouped, because no docs page currently lists all of them:

Group	Criteria
Behavior	NumberOfVisits, ViewedPages, ViewedCategories, Download, TimeOnSite, Event
Arrival	Referrer, SearchWordReferrer, StartUrl, QueryString
Time	TimeOfDay, TimePeriod
Place & client	GeographicLocation, GeographicCoordinate, IPRange, OSBrowser, DisplayChannel, SelectedLanguage
Identity	Role, UserProfile, VisitorGroupMembership
Custom (this demo)	DeviceCriterion

Two of those built-ins, configured: a time-of-day window and a country match, both reading request and CMS state the FX engine never sees.

Built-in Time of Day criterion on an “Office hours” CMS audience: 08:00–16:00, Monday to Friday.

Built-in Geographic Location criterion on a “Visitors from PL” CMS audience: Europe / Poland.

The attribute pipeline: FX only knows what you tell it

The part-2 provider sent two attributes derived from the request: device from the User-Agent, location from geo headers. Part 3 grows it to five, and the growth is where the lessons are:

return new Dictionary<string, object?>
{
    ["device"]    = ResolveDevice(request),          // "mobile" | "tablet" | "desktop"
    ["location"]  = ResolveLocation(context),        // "PL", "SE", ... | "unknown"
    ["logged_in"] = context?.User.Identity?.IsAuthenticated == true,   // a real bool
    ["cms_role"]  = ResolveCmsRole(context?.User),   // "admin" | "editor" | "none"
    ["consent"]   = HasConsent(request)              // a real bool, from a cookie
};

The five registered attribute keys (cms_role, consent, device, location, logged_in), the code-to-dashboard contract.

Three rules. All three carried more weight than I expected when I wrote them down. (Throughout this article, “the panel” means part 2's instrument set. The demo page renders a variation pill carrying the served arm plus a data-rule attribute naming the rule that served it, and a Development-only diagnostics panel showing the visitor ID, the attribute dictionary and the SDK's Decide reasons. Every measurement below reads off those two surfaces.)

“The panel”: the variation pill and the FX decision diagnostics: visitor ID, served variation, the attribute dictionary, and the SDK INCLUDE_REASONS trace.

Types are the contract. Attributes are untyped in the dashboard. The value you send decides which comparisons can match. Send the string "true" against a boolean condition and the condition is skipped: evaluated to UNKNOWN, audience false, visitor falls through, nobody logs an error at default verbosity. The booleans above are real booleans for exactly that reason. (Sharp edge #3 has the measurement.)

Absent and false are different inputs. An anonymous visitor on the MVC head sends logged_in = false. If the headless head simply omitted the key, an audience targeting logged_in is false would match on one head and skip on the other. Same visitor, same flag, different arms, and you would hunt the bug in the hashing where it isn't. The Next.js head therefore sends the constants its reality justifies: its visitors are, truthfully, never logged in.

One request, one context, attributes enter once. The decision service is request-scoped and builds the SDK user context a single time. The banner flag, the experiment and the conversion all agree on who the visitor is. Change an attribute mid-request and nothing happens. That's by design, and worth a comment in the code so nobody “fixes” it.

Geolocation: one header now feeds both engines

The hand-rolled four-header geo parser from part 2 is gone. CMS 13 ships native client geolocation: country from a single configured CDN header, no local IP database. One line wires it: services.AddCmsClientGeolocation(o => o.LocationHeader = "CF-IPCountry").

The same IClientGeolocationResolver now answers two callers: our FX location attribute and the built-in geographic criterion of CMS Audiences. One configuration line, two engines. That also means one spoofed header fools both at once. curl -H "CF-IPCountry: SE" and the panel reads location=SE. The part-2 caveat about attacker-supplied input didn't go away. It doubled its blast radius. And a detail measured the hard way: the resolver wants an uppercase ISO code. PL resolves; pl resolves to nothing. Cloudflare sends uppercase, so production works. But any homegrown proxy that normalizes headers to lowercase turns your geographic targeting off, both engines' worth of it, with no warning, no log line, no sound. The diagnostics panel reading location=unknown while the header is plainly there is the only tell.

The consent attribute is not an attribute. Or rather: it is targetable like any other, but its real job is a switch. Impression and conversion events carry the visitor ID and the attribute dictionary to Optimizely's backend. That payload is precisely the thing a non-consenting visitor declined. So when consent is false, the service calls Decide with DISABLE_DECISION_EVENT and turns Track into a logged no-op. Flags keep working, because the datafile evaluation is local, but nothing leaves the building. Consent isn't an audience attribute. It's an event-egress switch. (The demo toggles it with a cookie; a real implementation wires it to your CMP. And the conversion button now reports tracked: false instead of celebrating an event that never left. A demo that lies to you is worse than no demo.) One boundary disclosed rather than implied: the visitor-ID cookie itself is set, and bucketing happens, before any consent. This build treats the identifier as functional state and gates only what leaves the building. Your CMP, or your DPO, may read that line differently. That conversation belongs in your project rather than an SDK option.

One honest note before moving on: FX has no server-side bot filtering. A crawler's User-Agent contains no “Mobi”, so every bot enters your experiment as a desktop visitor and dilutes whatever it touches. Either gate Decide on known-bot UAs, or accept the noise with your eyes open.

Targeted Delivery: personalization without the dice

The plan was seductively simple: put a targeted rule above the everyone-else A/B test. Mobile gets the personalized arm, everyone else keeps experimenting. The dashboard said no, and the way it says no is the finding. The ruleset isn't one sortable list. It's three fixed sections whose headings literally narrate the evaluation order:

First, match experiment rule → a_b_test (A/B, audience: Non-mobile visitors)
Then, the following rules will be evaluated for all visitors → mobile_delivery (deliver variant_a, 100%)
Then, for everyone else → Off

The ruleset's three fixed sections: experiment first, then deliveries for all visitors, then everyone-else. No shared list to drag a delivery above an experiment.

The mobile_delivery rule: Targeted Delivery, audience Mobile visitors, 100%, deliver variant_a.

The Mobile visitors FX audience: device equals “mobile”, with all five attribute keys in the browser on the right.

The same audience in Code Mode: match_type “exact”. The datafile the SDKs evaluate spells the field match (sharp edge #1).

The Non-mobile visitors audience used to carve the experiment: a positive list, device equals desktop OR tablet, so an unknown device matches nothing and lands on master.

FX evaluates experiments before deliveries. Always. The UI encodes the law in its page structure rather than enforcing it with an error. So carving mobile out of the experiment happens in the audience, not in rule order. The A/B rule's audience changed from Everyone to Non-mobile visitors: a deliberately positive list (device = desktop OR tablet) rather than a negation. That way a visitor with a missing or mangled device attribute matches nothing and lands on master, which is the honest outcome for “we don't know who this is”.

Two semantics from the docs that almost nobody quotes, both load-bearing:

An experiment whose audience matches but whose traffic allocation misses rolls down to the next rule. A delivery in the same situation jumps straight to everyone-else, skipping any deliveries below it.
Deliveries produce no Results page: “No decision events show up on the results page.” A delivery is deployment, not measurement. Our mobile visitors get their personalization and leave the experiment's bookkeeping entirely. The A/B sample quietly becomes a desktop-and-tablet sample. Write that sentence in your analysis doc before someone asks why the experiment's traffic dropped.

And one behavioral nuance that will eventually puzzle a stakeholder: bucketing is sticky, attributes are not. The same visitor who taps “Request desktop site” in mobile Chrome flips device mid-cookie, exits the delivery, and enters the experiment. Same visitor ID, different arm, both serves correct. Attribute-driven targeting re-evaluates every request. Only the hash is forever.

Configuration footnote from the audience builder: the dashboard's Code Mode validates conditions with the key match_type, while the datafile the SDKs download spells the very same field match. Two serializations of one concept. Harmless until you copy a condition from the datafile into Code Mode and the validator rejects what the SDK just evaluated. The SDK's own log line later in this article shows the match spelling in the wild.

Can both engines coexist on one page? Yes, at different layers, measurably

This is the part people actually google, so here is the layered setup, built and measured. FX picks the page version (mobile → variant_a via the delivery), and inside that version one block carries a CMS Audience (Mobile visitors (CMS), our DeviceCriterion). Two engines, two layers, same page.

CMS Audiences admin: Mobile visitors (CMS), Office hours, Visitors from PL.

The Mobile visitors (CMS) audience on the custom Device class criterion (value “mobile”), statistics enabled.

Assigning the audience to a content-area block inside variant_a in the CMS editor, with the rendered variant on the right.

The measurement needed care, because FX variation noise would drown the block signal. Take one visitor ID that desktop-buckets into variant_a through the experiment, then request the page twice: same visitor, same page version, only the User-Agent differs.

Probe (same visitor, same variant_a)	Rule that served it	Content area
Desktop	a_b_test	empty, block filtered out
Mobile	mobile_delivery	“Mobile Quiet Hero” renders

Pure visitor-group effect, isolated from FX. The coexistence answer: no conflict, different layers. The FX decision happens in the controller before rendering. The CMS Audience filters content-area items during rendering. The pitfall isn't a conflict, it's unreachability: personalize a block for mobile in the master version and no one will ever see it, because under this ruleset mobile never receives master. The FX rule one layer up decides which content area exists at all.

There's a sour bonus measurement. An earlier build of this demo, the one registered from decompiled internals instead of the documented AddVisitorGroupsMvc(), served that personalized block to the desktop probe too. Same content, same audiences, one missing rendering filter, zero errors. Keep that pair of runs. It's the cleanest argument I own for “configure from the docs, not from the decompiler”.

What does Content Graph actually do with a personalized block?

The same page travels to the Next.js head through Optimizely Graph. The personalized item doesn't:

variant_a  →  MainContentArea: []        (the block the MVC head demonstrably renders)
original   →  MainContentArea: []
control    →  MinimalPage.HeroArea: [{ "Heading": "EN Heading" }]   // non-personalized area expands fine

Optimizely Graph: all three DemoPage arms (variant_a, variant_b, master) are in the index with distinct DemoTitle, yet every MainContentArea is []. The page variation survives; the personalized content-area relation does not.

Not a leak. A void. The VG-personalized item vanishes from the index for every Graph consumer, including the mobile visitors who would have matched. No API error, no warning in the sync job, nothing for the front-end to even detect. The shape of the void is worth a second look: the block itself is still in the index, four QuietHeroBlock documents, each with its own identity and content, but the page's content-area relation to it is gone. The explanation consistent with every measurement is that indexing evaluates the page without a visitor to personalize for, and an audience-gated item has no honest answer to “should this exist?”, so it doesn't. Whatever the internals, the contract you can rely on is the measured one: personalization severs the composition, not the content. The docs say it without decoration: “audiences do not work on headless sites.” The measurement above is what that sentence costs in practice.

So what do you actually do when the roadmap says headless? There are three shapes, in order of preference. First, move that personalization up a layer to FX Audiences and Content Variations. That is this article's whole thesis, and it needs nothing new. Second, take the CMS 13.1.0 Graph Conventions API and customize indexing so the head receives the items plus enough metadata to filter client-side. Workable, but now you own the audience semantics on every head. Third, stand up a membership endpoint on the CMS that the head consults per request. It works, and it is exactly the coordination code this series exists to avoid. There is no fourth option where the block just shows up.

The decision table

You need…	Engine
The same personalization on every head, today and after the next re-platform	FX Audiences + Content Variations
Criteria that read CMS state: roles, visit history, time of day	CMS Audiences
Measurement: exposures, conversions, significance	FX (deliveries excluded, they don't report)
Kill switch without a deploy	FX (latency = datafile propagation; measured at ~75 s here)
Fragment-level personalization, CMS-rendered site, no measurement need	CMS Audiences
Zero marginal license cost	CMS Audiences (in the CMS license; FX is usage-billed, though deliveries fire no impressions, only experiment bucketing does)
Per-request evaluation cost approaching zero	FX (pure in-memory function; VG criteria may do I/O per content-area item)
Ownership by content editors: instant, local, visual	CMS Audiences (VisitorGroupAdmins; no deploy, no datafile, changes land on publish)
Ownership by product managers and analysts	FX Audiences, with attribute-key discipline as the price: keys are a code-to-dashboard contract, and once created an attribute stays in the datafile permanently

A word of architectural restraint: a personalized block inside an experiment variation multiplies your test matrix: arms × audiences × consent states. The demo does it to prove the layering. Production teams should default to one personalization axis per page, and budget QA explicitly before crossing them.

ODP deserves its one paragraph: when the attributes should come from customer data rather than the current request, Real-Time Segments for Feature Experimentation syncs ODP audiences into the same rule slots. Segments stay out of the datafile, qualification typically lands under thirty seconds, and Optimizely's own docs concede that when you need 100% accuracy you should fall back to plain custom attributes. Different source, same predicate machinery. That's the whole story here. This demo has no ODP account on purpose.

From prose to proof: the day the headless head went dark

Here is the embarrassing measurement first, because it earns the rest of the section. The moment the A/B rule's audience changed from Everyone to Non-mobile visitors, the Next.js head, which part 2 proudly demonstrated holding a perfect 33/33/34, went one hundred percent dark:

default (master)      300  (100.00 %)
FAILED: "300 request(s) were served the master page (no FX decision) - check
that the rule is Running, Traffic Allocation is 100% and the SDK key matches
the rule's environment."

Rule running, allocation 100%, SDK key correct. The test's own diagnostic, written by me in part 2, blames everything except the actual cause: the head's decide() sent no attributes, so the visitor matched neither audience and fell through to everyone-else-off. No errors, green dashboards, a misleading failure message, and a head silently serving master to every visitor from the minute the audience shipped. Audiences don't add a feature to your architecture. They add a requirement to every head you run.

The fix is a mirror: a buildAttributes() on the Next side that reproduces the .NET pipeline byte for byte. The parity contract lives in one file with a comment that says exactly that: if the heuristics drift, parity dies at the attribute layer while everyone debugs the hash. The contract itself fits in a table, which is how it should be reviewed:

Attribute	The rule both heads obey
device	Substring tests, case-insensitive: iPad/Tablet → tablet, else Mobi → mobile, else desktop
location	Header path is strict: ^[A-Z]{2}$ and ≠ XX, no case folding (mirrors the measured CMS resolver); the Accept-Language fallback does fold case (pl-PL → PL)
logged_in, cms_role	Real types, truthful constants on the auth-less head (false, "none"): absent and false are different inputs
consent	bool.TryParse semantics: case-insensitive true, anything else false

Upgrading the JS SDK to v6 along the way produced one trap worth its own line. v6 is explicit-opt-in across the board: polling config manager, logger, and ODP all opt-in. So is the event processor: omit createBatchEventProcessor() and decisions keep flowing while the SDK dispatches no events at all. Served-but-unmeasured, the exact failure shape this series keeps finding, now available at the initialization layer. (The same modularity quietly fixed an old log nag: no ODP manager, no “ODP is not integrated” warnings.)

Two Next.js-specific notes earned their scars. The SDK instance is cached on globalThis, the same trick Prisma clients use, because next dev's Fast Refresh re-evaluates modules. A module-level singleton would leak a fresh polling manager and event processor on every save, and your datafile CDN would meet them all.

And one asymmetry to disclose rather than hide: the C# head flushes its event queue on shutdown because the DI container disposes the client (part 2's contract). A Node process has no such container. The batch rides flushInterval, and what happens at the end of a process's life depends entirely on what kind of life it had:

Long-running Node (next start, a container): safe. The queue drains on the interval, and a tail lost to a SIGTERM is a rounding error.
Serverless / edge: risky. The platform can freeze the instance the moment the response returns, and a consented conversion dies in the buffer as a zombie event.
The fix where it matters: shrink the batch to the point of synchronicity, or await client.close() before the invocation ends, and accept the latency as the price of the data.

Pretending the queue always drains would be exactly the kind of silence this series hunts.

The Next.js headless head on a mobile viewport: the same FX decision serves variant_a via mobile_delivery, and the variation's title arrives through Optimizely Graph, with zero coordination with the MVC head.

Then the green numbers, all in one evening:

Check	Result
Mobile, 60 fresh visitors (hard assertion: a delivery rolls no dice)	60/60 variant_a via mobile_delivery
Desktop distribution, n = 300, against the Next head	103 / 105 / 92, χ² = 0.985 (critical 13.816 at α = 0.001, df = 2)
Same regression against the MVC head	99 / 109 / 92, χ² = 1.465
Cross-head parity: 4 fixed + 6 random visitors × desktop and mobile	arm AND rule identical on both heads, 20/20 pairs

The desktop chi-square run against the headless Next.js head: 103 / 105 / 92 over n = 300, χ² = 0.985, well under the 13.816 critical value. Every decision came from Optimizely FX in the Node SDK, zero coordination with the MVC head.

The parity test asserts the rule key, not just the arm. “mobile got variant_a” and “mobile happened to be bucketed into variant_a” are different claims, and only data-rule in the markup separates them. MurmurHash plus deterministic audience evaluation. Still zero coordination code.

The arithmetic deserves to be shown once, not just asserted. For the MVC regression (99 / 109 / 92 over n = 300, expected ≈ 100 per arm):

χ² = (99−100)²/100 + (109−100)²/100 + (92−100)²/100
   = 0.01 + 0.81 + 0.64 ≈ 1.46

(The suite reports 1.465 because it tests against the configured 33.33/33.33/33.34 rather than exact thirds.) With df = k − 1 = 2 and α = 0.001, the critical value is 13.816, and 1.465 ≪ 13.816, so the null hypothesis (“the split matches the configuration”) survives comfortably. In words: after carving the mobile segment out with an audience, the hash shows no distributional anomaly in what remains. The same computation on the Next head's 103 / 105 / 92 gives 0.985, comfortably inside the spread a fair split throws at n = 300.

Threats to validity, disclosed as ever: the chi-square run verifies the bucketing distribution, not content delivery (part 2's caveat stands). The mobile check is exhaustive rather than statistical, because a 100% delivery is deterministic: a single counterexample falsifies it, no α required. And every test here runs without a consent cookie, which after the egress gate means the suite sends zero impressions to FX. Three hundred requests of load testing used to pollute Results. Now the same suite is invisible to it. That started as a privacy control and turned out to be test hygiene.

QA: three override levels, three owners

“How does QA see the variant for a segment they don't belong to?” That is the question audiences force on every test plan. Three answers, escalating by who owns them:

New visitor (the cookie-reset button, part 2): owned by anyone with a browser. Re-rolls the dice but can't cross an audience boundary. The button also demonstrates something subtler: rotating the visitor ID re-buckets every experiment, leaves a delivery unmoved (its audience is deterministic in the attributes, not the ID), and the CMS Audience doesn't even notice; its criteria never saw your visitor ID in the first place. One page, three different notions of who “you” are. QA plans that conflate them chase ghosts.
Allowlist (dashboard, per rule, up to fifty IDs): owned by the marketer, pins a visitor ID to an arm. Measured here: an allowlisted ID came back variant_b with the SDK reason “is forced in variation”. Caveat: allowlisted traffic still fires impressions, so your QA session pollutes Results.
?fx_force=demo_ab_test:variant_b (this part): owned by the developer, a Development-only query parameter mapped to the SDK's forced-decision API. It bypasses audiences and allocation (forced mobile, forced desktop, doesn't matter), and, deliberately unlike the allowlist, a forced request also suppresses its decision events. QA that leaves no fingerprints on the data it's there to protect.

The A/B Test rule: audience Non-mobile visitors, 100%, the 33.33 / 33.33 / 33.34 split, and the allowlist pinning a QA visitor ID to variant_b.

The whole mechanism is one small source behind one small interface, so the SDK types stay where part 2 put them:

var forced = forcedDecisionSource.GetForcedDecisions();   // [] outside Development
foreach (var (flagKey, variationKey) in forced)
{
    userContext.SetForcedDecision(
        new OptimizelyDecisionContext(flagKey),
        new OptimizelyForcedDecision(variationKey));
}
_eventsAllowed &= forced.Count == 0;   // QA traffic never reaches Results

While we're holding the decide call open, the full option set gets a reference table (part 2 named these in prose):

Decide option	What it does	Where this series uses it
INCLUDE_REASONS	Returns the evaluation trace: which rule matched, why audiences failed	Every call; feeds the diagnostics panel
DISABLE_DECISION_EVENT	Evaluates without sending an impression	No-consent requests; forced QA requests
ENABLED_FLAGS_ONLY	Filters DecideAll to enabled flags	n/a
IGNORE_USER_PROFILE_SERVICE	Skips UPS stickiness for this call	n/a (this demo runs without UPS)
EXCLUDE_VARIABLES	Omits variable payloads for cheaper decisions	n/a

The CMS side has its own counterpart, View as Audience in the editor, with one boundary note: it previews visitor groups, not FX arms, and the docs scope it to CMS-rendered sites only. Pair the two in your test plan. Neither substitutes for the other.

Forced decisions don't persist. They clear with the user context, which in this architecture means they last exactly one request. That's not a limitation. Per-request is the only scope that can't leak into someone's real session.

Runbook: “the page always serves master”, audience edition

Part 2 ended with a checklist for the all-master symptom. Audiences roughly double it. The developer's mental map first: data-rule on the pill is the fork in the road.

data-rule is empty, no decision happened at all: SDK key, datafile or flag off (part 2's checklist).
data-rule = default-rollout-…, “everyone else” fired, so no audience matched:
- Attributes row wrong or missing → context pipeline gap (the dark-head failure: a head sending nothing).
- Attributes row correct → condition-side defect (whitespace, case, type); open the audience in Code Mode, where quotes make it visible.
data-rule = a real rule, arm = original, working as designed: the control arm serves master via fallback.

And the full ordered checklist, where each step assumes the previous ones passed:

#	Check	Where the truth shows
1	SDK key matches the rule's environment; ruleset Running; variation toggles ON	Part 2's checklist (dashboard)
2	The decision reached a rule at all	data-rule on the pill: default-rollout-… means everyone-else fired
3	The attributes you think you send are the attributes you send	Diagnostics panel, Attributes row, wrong name, wrong case, wrong type, missing key are all visible here
4	The audience condition matches those attributes byte-for-byte	Reasons: “collectively evaluated to FALSE” while the panel shows the right value means a condition-side defect; open the audience in Code Mode, quotes make it visible
5	Type mismatches	SDK log (not reasons): “evaluated to UNKNOWN because a value of type … was passed”, requires the part-2 logger
6	The head sends attributes at all	The dark-head failure: decide() without attributes matches nothing, and the test diagnostics will blame the rule
7	Datafile freshness	Audience edits propagate on the polling interval like every other change
8	(CMS side) the personalized block's layer is reachable	A block personalized in a version the visitor never receives is invisible by construction
9	(CMS side) the rendering layer is registered	Audiences screen working ≠ filter running, AddVisitorGroupsMvc(), then the startup self-check proves it

Sharp edges, part 3: ranked by blood actually drawn

Same tradition, same ordering rule. Every one of these happened on this instance, this week.

1. A trailing space in an audience condition. The audience said device = "mobile ". The builder keeps whatever the clipboard delivered, and exact-match fields are not trimmed. Exact match means exact: every mobile visitor silently fell through two rules to master while every dashboard stayed green. The only witness was the diagnostics panel: attributes showed device=mobile, reasons showed Audiences for rule "mobile_delivery" collectively evaluated to FALSE, and the contradiction between those two lines is the diagnosis. Code Mode confirms it fastest. JSON quotes make whitespace visible. Cost: the better part of an evening, and it produced this list's ordering.

2. One [OutputCache] attribute defeats both engines and privacy at once. Swap the demo's no-store guard for platform output caching, and the measurement reads like an incident report: a desktop visitor received the mobile visitor's page from cache, wrong arm, wrong personalization, and the other person's visitor ID rendered in the panel, Age: 0 confirming the hit. The CMS-aware output cache has been gone since the CMS 12 rewrite, and CMS 13 still gives the platform layer nothing that knows your pages are personal (verified: no output-cache type exists anywhere in the 13.1.0 assemblies). The fix is not a smarter cache key. It is classifying pages: personalized means no-store, full stop. (The CDN analog is “ignore cookies in the cache key”, which produces the same leak at planetary scale.)

3. Type mismatch evaluates to UNKNOWN, and only the logger says so. Send device as a boolean and INCLUDE_REASONS reports a flat “collectively evaluated to FALSE”. The why lives one layer down, in the SDK log the part-2 adapter made visible: Audience condition {"match":"exact","name":"device","value":"mobile"} evaluated to UNKNOWN because a value of type "Boolean" was passed. Wire the logger on day one was part 2's advice. This is the day it pays. (Note the "match" spelling: the datafile serialization, not the dashboard's match_type.)

4. A missing attribute is quieter than a wrong one. Drop device from the dictionary and the fall-through looks identical. But this time there is no WARN anywhere, and the SDK mentions missing attributes only at debug verbosity. Absent isn't false, and absent doesn't log. The gradient of silence: wrong type warns (if you wired logs), wrong name says nothing, missing says nothing.

5. Case sensitivity, now in three places. Part 2 had the case-sensitive VariationKey. Part 3 adds attribute names (Device ≠ device), attribute values (Mobile ≠ mobile; our custom criterion deliberately Trim+lowercases editor input as a courtesy FX won't extend), and the geo header's value (pl resolves to nothing). That last one is also the closing argument for the unification: part 2's hand-rolled parser folded case and would have accepted pl. The native resolver doesn't. Two readers of one header with different tolerances is exactly the divergence class that breaks cross-engine and cross-head consistency: one resolver on the CMS, one written-down contract for the second head.

6. A personalized item doesn't leak to Graph: it ceases to exist. The coexistence section holds the measurement. The dangerous half is that nothing tells you: the sync succeeds, the schema is fine, the array is just empty. If your roadmap says “headless next year”, every CMS Audience you ship today is a block that will silently vanish in the migration.

7. One header steers two engines. AddCmsClientGeolocation is genuinely elegant. And it concentrates trust in a single header that, exposed without an edge to strip client values, both engines will believe. The part-2 caveat, squared.

8. The datafile is still the clock. Editing an audience is a datafile change like any other. This instance picked the fix from edge #1 up in about 75 seconds on the default polling. Not a repeat of part 2's lifecycle section. Just a reminder that audience edits ride the same train, and so does your kill switch.

FAQ, the questions as people actually type them

Can I use Visitor Groups / CMS Audiences with a headless frontend? No. Evaluation needs the CMS runtime, and the measurement here is blunter than the docs: a personalized content-area item isn't filtered for Graph consumers. It is absent from the index entirely. Personalize whole page versions with FX Audiences instead. They evaluate in any SDK.

Targeted delivery or A/B test: which one do I want? Deliveries deploy, experiments measure. A delivery produces no Results page at all, so the question is really “do I need to learn anything from this traffic?” If yes, experiment; if you already know and just want control and ramp, delivery.

Why does my FX audience never match? In observed order of likelihood: a stray character in the condition value, the wrong value case, the wrong attribute-name case, the wrong value type, the attribute not sent at all. The runbook above walks the diagnostics. The short version: the panel shows what you sent, and the SDK log shows what the evaluator thought of it.

Can a delivery run above an experiment? No. The ruleset evaluates experiments first, and the UI encodes that as fixed sections rather than a sortable list. Carve segments out of an experiment with audiences, not with rule order.

Do I need ODP to use audiences? No. Everything in this article runs on request-derived attributes. ODP (Real-Time Segments) is the path when targeting needs customer data (past purchases, lifecycle stage) rather than facts about the current request.

How fast is the kill switch? One datafile propagation. This instance picked up an audience edit in ~75 seconds on default polling. The guarantee is your polling interval (or webhook latency), not the dashboard click.

Does QA and test traffic pollute my results? Dashboard allowlists do: they fire impressions like any decision. This build's ?fx_force deliberately doesn't, and the consent gate has a pleasant side effect: the entire automated suite (hundreds of requests per run) sends zero events, because test traffic never carries a consent cookie.

The take

The model that survives all of the above is short enough to memorize. The CMS owns what: versioned, publishable content variations, plus fragment-level personalization wherever the CMS itself renders. FX rules own who and whether: deterministic assignment, deliveries for rollout, experiments for measurement. Audiences, both kinds, own for whom, and the kind you pick decides where the personalization can exist at all. CMS Audiences see everything and travel nowhere. FX Audiences see only what you send and run everywhere you do.

For the business reader who skipped to the end: the mobile personalization in this demo shipped without a deploy, measured itself before an audience of three hundred synthetic visitors, survived a re-platform to a second rendering stack with zero integration code, and can be killed from a dashboard in about the time it takes to refresh a datafile. The block-level personalization shipped too. And it stopped at the CMS's edge, invisible to the headless head, unmeasured by anything but page-view counters. Both behaviors are by design. The architecture decision is choosing which design your roadmap can live with.

The fragile parts haven't changed character since part 2: every failure in this article degraded to “someone quietly sees master” or “someone quietly sees too much”, and not one of them threw. The integration is still strings and dictionaries. The engineering is still making the silence loud: a diagnostics panel, an SDK logger, a startup self-check, and tests that assert the rule, not just the arm.

Quiet hero, part 3: now it knows who you are. Still quiet, that's still the problem to engineer around.

Glossary

Term	In one sentence
Chi-square goodness-of-fit test	Tests whether observed arm counts are consistent with the configured split; χ² = Σ (observed − expected)² / expected.
Degrees of freedom	Category counts free to vary; k arms give df = k − 1, so our three arms test at df = 2.
Significance level α	The false-alarm rate the test tolerates; this series uses α = 0.001 so a healthy split almost never red-flags live.
Critical value	The rejection threshold for the statistic, 13.816 for df = 2 at α = 0.001.
Sampling error	Per-arm share fluctuates by ≈ √(p(1−p)/n), about 2.7 percentage points at n = 300, p = ⅓.
MurmurHash	The deterministic hash every FX SDK applies to visitor ID + experiment; the reason parity needs no coordination.
Pure function	Same input, same output, no side effects, FX audience evaluation in one phrase.
Predicate	A boolean-valued function; both audience kinds are predicates over different inputs.
Eventual consistency	Distributed state converges after a delay, the datafile (~75 s observed) and the Graph index both live here.
Targeted delivery	A rule serving one variation to an audience; deployment, not measurement, no Results page.
Datafile	Configuration-as-data: the JSON snapshot of flags, rules and audiences every SDK evaluates locally.
Backend for Frontend	The server-side layer that keeps SDK and Graph keys out of the browser on the headless head.
Defense in depth	Layered safeguards: consent gate, no-store, antiforgery, origin checks, none trusted alone.
Allowlist	A per-rule list (up to fifty user IDs) pinning specific visitors to specific arms; fires impressions like normal decisions.
Forced decision	An SDK-level override bypassing audiences and allocation; not persistent, it lives and dies with the user context.

Unlock Experimentation with Content Variations in CMS 13

Thu, 11 Jun 2026 07:43:54 GMT

Part 1 argued that Content Variations is the CMS 13 feature that didn't get the keynote but should have. This is the follow-up: wiring those variations to Optimizely Feature Experimentation, measuring whether the traffic split actually holds, and serving the same experiment from two heads (classic MVC and headless Next.js) without the experiment noticing. Everything below ran on a real CMS 13.0.2 instance with FX SDK 4.3.0; the numbers are from those runs, not from a slide.

Part 1 ended with variations sitting in the CMS, published, versioned, queryable. Nice. Also useless, commercially, until something decides who sees which arm and whether it moved a number. The CMS deliberately has no opinion about that. There is no traffic-splitting UI anywhere near the Variations dropdown, and the absence is a design decision, not a gap. Delivery belongs to an experimentation engine.

The entire integration between the two products is one string.

FX VariationKey = CMS Content Variation name

That's the entire contract. FX's Decide() returns a variation key; you load the CMS variation with the same name; if the names drift apart, nothing throws and visitors just quietly get the master page. Everything else in this article is plumbing around that one equality, and most of the sharp edges live exactly where you'd expect: in what happens when the equality silently fails.

Who does what

Three roles, three tools. Settle the division of labor before the code, because it's the actual selling point:

Role	Tool	Job
Editor	CMS Variations dropdown	Creates and publishes the content arms. Never sees the FX dashboard.
Marketer / analyst	FX dashboard	Owns the rule: traffic split, ramp, pause, Results. Never touches the CMS.
Developer	This article	Wires the plumbing. Once.

After the wiring ships, the next experiment needs no developer at all: an editor publishes new arms, a marketer points a rule at them, and the string contract does the rest. Stopping a test is one click in the dashboard, no deploy, no content rollback - the arms stay in the CMS, versioned, ready for the next round. If you have to sell this internally, that's the sentence to take into the budget meeting.

The shape of the experiment

One flag, demo_ab_test. Three FX variations: original, variant_a, variant_b. Two CMS Content Variations: variant_a and variant_b.

Two, not three. The control arm gets no CMS variation. When FX returns original, the loader finds no variation by that name and falls back to the master page. The master is the control. Creating a third CMS variation called original is the first mistake I see teams make: it duplicates the master's content, and the two copies drift apart the first time an editor fixes a typo in one of them. Your control arm is then no longer your canonical page, and every lift number you report afterwards is quietly wrong. Let the fallback do its job.

From the editor's chair, the whole feature is part 1's workflow with one naming rule attached: Variations dropdown, New Variation, name it exactly variant_a - the key is case-sensitive, and Variant_A buys you the silent-master failure described below - then change the content and publish. Each arm publishes independently, so an unpublished draft of variant_b simply means that arm's visitors keep seeing master until it ships. No FX login, no flag keys, no deploys. The naming rule is the only place an editor can break the experiment, which makes it the one thing worth putting in their runbook.

The demo_ab_test flag in the FX dashboard.

The variation keys are defined on the flag, and each arm has its own flag on/off toggle:

Variation keys on the flag - these names must match the CMS variation names exactly.

That per-variation toggle is the second quiet trap. The serving code treats Enabled == false as "no decision, serve master." If variant_a has its toggle off, a third of your traffic is bucketed into an arm that renders the control while the dashboard reports them as exposed to the variant. All three arms stay on; the rule, not the variation toggle, is your kill switch.

An A/B Test rule won't save without a metric, so an event has to exist first, even if you wire conversions later:

The conversion event has to exist before the rule will save.

Then the rule itself: audience Everyone, traffic allocation 100%, distribution mode Manual at 33.33/33.33/33.34, baseline original:

The A/B Test rule: Everyone, 100% allocation, manual 33.33/33.33/33.34 distribution.

Most write-ups skip one field on the ruleset entirely: "Then, for everyone else." It's the fallthrough for visitors who don't match the rule, either because of the audience or because they fall outside the traffic allocation. At Everyone/100% it never fires, but it documents intent for the day someone ramps the test down to 50%. Set it to Off. Off means Enabled = false means master means control. Setting it to original works (the loader falls back identically), but those visitors get reported as if the experiment decided for them, which pollutes the boundary between "in the test" and "not in the test." Off keeps the boundary honest.

The "Then, for everyone else" fallthrough - set it to Off to keep the in-test boundary honest.

Serving it: one decision, one load, one fallback

The controller-side code is small, and it should be. Controllers depend on an SDK-agnostic abstraction, one Decide per flag, one user context per request:

public IActionResult Index(DemoPage currentPage)
{
    var decision = _decisions.Decide("demo_ab_test");
    var variationKey = decision.Enabled ? decision.VariationKey : null;

    var pageToRender = string.IsNullOrEmpty(variationKey)
        ? currentPage
        : _loader.Load<DemoPage>(currentPage.ContentLink, variationKey) ?? currentPage;
    // ...
}

The loader is the same VariationLoaderOption mechanism from part 1, wrapped with a fallback:

var options = new LoaderOptions { VariationLoaderOption.With(variationKey) };
if (contentLoader.TryGet<T>(link, options, out var variant) && variant is not null)
    return variant;
return contentLoader.TryGet<T>(link, out var master) ? master : null;

The decision service: the only class that knows the SDK

_decisions above is deliberately not the Optimizely client. It's a thin abstraction, and the SDK types live in exactly one implementation behind it. That's less about architectural piety than about two practical facts: controllers become trivially testable (fake a FlagDecision, done), and when SDK 5.x changes a signature you edit one file.

The implementation is where the docs' most important serving-side rule lands: one user context per request, shared by every decision in that request. The SDK's OptimizelyUserContext carries the visitor ID and attributes; creating it once per request (the service is DI-scoped) means the banner flag, the experiment flag, and the conversion all agree on who the visitor is:

public class FeatureDecisionService(
    OptimizelyClient optimizely,
    IVisitorIdProvider visitorIdProvider,
    IVisitorAttributesProvider attributesProvider) : IFeatureDecisionService
{
    private OptimizelyUserContext? _userContext;

    public FlagDecision Decide(string flagKey)
    {
        var userContext = EnsureUserContext();
        if (userContext is null)
        {
            return new FlagDecision(false, null, null,
                ["User context could not be created - SDK client unusable (e.g. missing datafile)."]);
        }

        var decision = userContext.Decide(flagKey, [OptimizelyDecideOption.INCLUDE_REASONS]);
        return new FlagDecision(
            decision.Enabled,
            string.IsNullOrEmpty(decision.VariationKey) ? null : decision.VariationKey,
            decision.Variables?.ToDictionary(),
            decision.Reasons);
    }

    public void Track(string eventKey) => EnsureUserContext()?.TrackEvent(eventKey);

    private OptimizelyUserContext? EnsureUserContext()
        => _userContext ??= optimizely.IsValid
            ? optimizely.CreateUserContext(visitorIdProvider.GetOrCreateVisitorId(), BuildAttributes())
            : null;
}

A few lines in there do more work than they look like. The IsValid guard: an SDK client without a datafile still hands out user contexts whose every Decide is an error decision; checking validity up front turns "mysteriously always master" into an explicit fallback with a reason string. INCLUDE_REASONS is one of five documented decide options (DISABLE_DECISION_EVENT, ENABLED_FLAGS_ONLY, IGNORE_USER_PROFILE_SERVICE, EXCLUDE_VARIABLES are the others) and the only one this demo needs. DISABLE_DECISION_EVENT is the other one I reach for in real projects: it asks for a flag's value without recording an impression, for the places where rendering logic needs the flag but the visitor shouldn't count as exposed. And TrackEvent rides the same context, which is the entire attribution story: no variation key travels with the conversion, ever. The stats engine joins exposure to conversion purely on the visitor ID.

Audience attributes enter at context creation too. The demo derives them from the request instead of hardcoding values - device class from the User-Agent, country from CDN geo headers with an Accept-Language fallback:

return new Dictionary<string, object?>
{
    ["device"]   = ResolveDevice(request),   // "mobile" | "tablet" | "desktop"
    ["location"] = ResolveLocation(request)  // "PL", "SE", ... or "unknown"
};

With those flowing, an FX Audience like device = mobile targets a rule with zero code changes; the rule evaluates against whatever attributes the context carried. One honesty note the docs won't volunteer: geo headers like CF-IPCountry are only trustworthy when your edge injects them and strips client-supplied values. Exposed directly, they're attacker-supplied input, and anyone can self-select into your geo audience with a curl flag. Fine for a demo; gate it behind your CDN in production.

Three failure modes funnel into the same outcome, and all three are invisible in production:

What happened	What FX reports	What the visitor sees
Flag off / rule paused / no datafile	no decision	master
FX returned original (control)	exposed to original	master - by design
FX returned variant_a but the CMS variation was renamed or deleted	exposed to variant_a	master - silently wrong

The third row is the one that should worry you. The experiment keeps collecting data, the dashboard keeps attributing conversions to variant_a, and every visitor in that arm is actually looking at the control. Nothing throws. No log line by default (more on the SDK's default logger below). The only place the truth surfaces is Decide's INCLUDE_REASONS output, which is why the demo renders a development-only diagnostics panel with the visitor ID, the resolved attributes, and the SDK's own reasoning: which rule matched, how the visitor was bucketed, or why no decision was made. Build that panel before you need it. It converts "the page always shows master and nobody knows why" from an afternoon of debugging into a single glance.

Bucketing stickiness rides on a first-party cookie (CMS_VisitorId, one year, HttpOnly). One non-obvious attribute: SameSite=None; Secure. The CMS editor previews pages in a cross-origin iframe, and without SameSite=None the browser drops the cookie there. Every preview request then mints a new visitor and a possibly different arm, and your editors file a bug that "the page keeps flickering between variants." The experiment is fine; the cookie policy isn't.

Two server hygiene points that the demo enforces and most tutorials skip: bucketed responses carry Cache-Control: no-store (a shared cache serving one visitor's arm to everyone collapses the split), and this head's conversion endpoint validates an antiforgery token - a need specific to the MVC head, because its SameSite=None cookie rides along on cross-site POSTs. An experimentation demo whose results can be inflated by a hidden form on someone else's page is not a demo you want to give. The headless head solves the same problem differently; details where they belong, in the Next.js section.

From prose to proof, again: does 33/33/34 actually hold?

Part 1 measured storage and cache behavior. The follow-up question here is more basic and more embarrassing if you skip it: does the traffic actually split the way the rule says? "Trust the hash function" is a fine answer until you're presenting and someone asks how you know.

The check is an xUnit test that fires N independent requests at the running page without cookies (the server mints a fresh visitor ID per request, so every request is a fresh bucketing), parses which arm each response declares, and runs a chi-square goodness-of-fit test against 33.33/33.33/33.34.

Three runs at n = 300 against the MVC head:

Run	original	variant_a	variant_b	χ² (df = 2)
1	90 (30.0%)	106 (35.3%)	104 (34.7%)	1.518
2	104 (34.7%)	82 (27.3%)	114 (38.0%)	5.352
3	105 (35.0%)	105 (35.0%)	90 (30.0%)	1.506

All pass. Run 2 is the instructive one: a 27.3% arm against an expected 33.3%, six points off, and the test is right not to flag it. At n = 300 one standard deviation per arm is ≈ 2.7 percentage points, so a six-point swing is unremarkable. If your gut said "27% means the split is broken," your gut would have failed you in front of an audience. That's what the statistics are for.

Steal one calibration decision from this test: the pass threshold is α = 0.001 (critical value 13.816), not the textbook 0.05. At α = 0.05 a perfectly configured rule fails the test one run in twenty by construction, which is terrible odds for a check you might run live. The power loss is irrelevant at this effect size: a genuinely broken split like 50/25/25 produces χ² ≈ 37.5 at n = 300 and sails past either threshold. Strict alpha costs nothing and buys you a test that never cries wolf on stage.

Threats to validity, disclosed as ever: the test verifies the FX bucketing distribution, not content delivery. The page reports the arm FX assigned; if the CMS variation failed to load (the silently-wrong row above), the test still passes while every visitor sees master. It's a distribution check, not an end-to-end content assertion; the diagnostics panel covers the other half.

Conversions close the loop: a CTA posts to an endpoint that calls TrackEvent("demo_conversion") for the same visitor context, and the stats engine attributes it to whatever arm that visitor occupies. No variation key is sent; attribution is entirely on the visitor ID, which is exactly why the cookie's stability matters more than any other moving part here.

The experiment collecting decisions and conversions in the FX Results view.

Plumbing the docs assume you'll remember

Three pieces of lifecycle wiring make the difference between a demo that survives a fortnight and one that fails the morning of the presentation. None of it is exotic and all of it is documented, but each item is easy to skip because the happy path works without it.

The datafile is a living document. Every decision the SDK makes is computed locally against the datafile, a JSON snapshot of your flags, rules, and traffic allocations served from the CDN. OptimizelyFactory.NewDefaultInstance(sdkKey) sets up background polling (five-minute default) so dashboard changes propagate without a deploy; for latency-sensitive setups the documented upgrade is a webhook that pings your app the moment the datafile changes. The corollary people miss: pausing an experiment in the dashboard is itself a datafile change. Whatever your refresh path is, that's also how fast your kill switch is.

Events are batched, and shutdown is part of the contract. TrackEvent doesn't call home synchronously. The C# SDK queues impressions and conversions through a batch processor (ten events or thirty seconds, by default) to keep request latency flat. The documented flip side is that you must dispose the client on shutdown so the tail of the queue flushes. In ASP.NET Core that costs nothing if you let it: register the client as a DI singleton and the container disposes it, queue flush included, when the host stops:

services.AddSingleton<OptimizelyClient>(provider =>
{
    var fxLogger = provider.GetRequiredService<ILoggerFactory>().CreateLogger("OptimizelyFX");
    var sdkLogger = new OptimizelySdkLogger(fxLogger);

    if (string.IsNullOrWhiteSpace(sdkKey))
    {
        fxLogger.LogWarning("Optimizely:SdkKey is not configured - fallback mode, all flags report disabled.");
        return new OptimizelyClient(datafile: null, logger: sdkLogger); // invalid on purpose: no polling, no blocking
    }

    OptimizelyFactory.SetLogger(sdkLogger);
    return OptimizelyFactory.NewDefaultInstance(sdkKey);
});

Logs don't exist until you wire them. That SetLogger call looks decorative. It is the opposite. The C# SDK's shipped default logger formats every message (datafile fetch failures, unknown event keys, audience evaluation warnings) and throws the result away. The adapter is ten lines, and it's the difference between reading the cause in your console and debugging blind:

public class OptimizelySdkLogger(ILogger logger) : OptimizelySDK.Logger.ILogger
{
    public void Log(SdkLogLevel level, string message)
        => logger.Log(level switch
        {
            SdkLogLevel.DEBUG => MelLogLevel.Debug,
            SdkLogLevel.INFO  => MelLogLevel.Information,
            SdkLogLevel.WARN  => MelLogLevel.Warning,
            SdkLogLevel.ERROR => MelLogLevel.Error,
            _ => MelLogLevel.Information
        }, "{Message}", message);
}

The same experiment, headless

Part 1 called the Graph angle "where it gets interesting." Here's the cash value. The repo carries a second head, Next.js on the App Router, serving the same experiment from the same flag against the same content, and the part that surprised people I showed it to is what I didn't have to do: nothing. No shared session service, no API between the heads, no coordination.

It works because two independent guarantees compose:

FX bucketing is a pure function. Every SDK - C#, JS, all of them - runs the same MurmurHash over visitorId + experiment. Same visitor ID in, same arm out, on any stack.
Graph serves variations by name (part 1's contract: opt-in variation argument, arms addressed via _metadata.variation).

The Next.js side is small enough to show whole. Middleware mints or forwards the same CMS_VisitorId cookie and hands the fresh ID to the current render via a request header, because the response cookie isn't visible to the request that set it:

export function middleware(request: NextRequest) {
  const existing = request.cookies.get('CMS_VisitorId')?.value;
  if (existing) return NextResponse.next();

  const visitorId = crypto.randomUUID();
  const headers = new Headers(request.headers);
  headers.set('x-visitor-id', visitorId);

  const response = NextResponse.next({ request: { headers } });
  response.cookies.set('CMS_VisitorId', visitorId, {
    httpOnly: true, sameSite: 'lax',
    secure: process.env.NODE_ENV === 'production',
    maxAge: 60 * 60 * 24 * 365, path: '/',
  });
  return response;
}

One deliberate difference from the MVC head, before someone files it as a bug: this cookie is SameSite=Lax, not None. The .NET head needs None because the CMS editor renders it inside a cross-origin preview iframe; this head doesn't do editor preview (a scope cut, not an accident), and for a top-level site Lax is the safer default - one more wall against the cross-site POST problem from earlier. The day you wire this head into the editor's preview, flip it to sameSite: 'none', secure: true and run local dev over HTTPS (next dev --experimental-https or a proxy), because browsers refuse SameSite=None without Secure.

CSRF splits along the same line. The conversion endpoint here is a plain Route Handler, and Route Handlers get no CSRF protection for free - that's a Server Actions perk (Next.js compares Origin against Host for those automatically). So the handler validates Sec-Fetch-Site/Origin itself before calling trackEvent, with the Lax cookie as the first wall and the header check as the second. Where the MVC head needed an antiforgery token because of its None cookie, this head needs an origin check because of its handler type. Same threat, two idiomatic answers.

A Server Component then calls the JS SDK: same flag, same INCLUDE_REASONS, plus one option the C# factory gives you for free and the JS SDK does not - datafileOptions.autoUpdate. Without it, createInstance fetches the datafile exactly once and a long-running server keeps serving stale flag state until the next restart:

const client = createInstance({
  sdkKey,
  datafileOptions: { autoUpdate: true, updateInterval: 60_000 },
});

const decision = client
  .createUserContext(visitorId)
  .decide(flagKey, [OptimizelyDecideOption.INCLUDE_REASONS]);

One scoping caveat on autoUpdate: it's a background timer, so it assumes a long-running Node.js process - a container, next start on a VM. On ephemeral serverless functions the timer dies with the invocation and every cold start fetches its own datafile; in that world the right delivery path is the webhook-into-Edge-Config pattern from the edge section below, not polling.

And the Graph query asks for the named arm with the disciplined form - include: SOME, the original kept in the result as the fallback:

query DemoPage($variation: [String!]) {
  _Content(
    where: { _metadata: { types: { in: ["DemoPage"] } } }
    variation: { include: SOME, value: $variation, includeOriginal: true }
  ) {
    items {
      _metadata { key displayName variation }
      ... on DemoPage { DemoTitle }
    }
  }
}

When the requested arm is missing from the index, the original arm in the same response is the fallback - the headless mirror of the MVC loader's TryGet-then-master. The selection itself is two lines, and the arm is identified strictly by _metadata.variation, never by dissecting an item ID:

const requested = items.find((i) => i._metadata?.variation === variationKey);
const original  = items.find((i) => !i._metadata?.variation);
const item = requested ?? original;

Same failure semantics on both stacks, which is the property you actually want, because your experiment analysis shouldn't have to care which head served the impression.

Verifying the variation arms come back by name in GraphiQL.

The same chi-square test against the Next.js head (n = 300) came back original 106, variant_a 90, variant_b 104, χ² = 1.518. Pass. Two runtimes, two languages, two delivery mechanisms, one distribution, because it's one hash.

Authentication, since this is the part audits ask about: the Next.js server is a Backend-for-Frontend. Graph queries carry the read-only single key in the Authorization: epi-single header (not in the URL, where it lands in every access log), and the key lives in a server-only environment variable that never reaches the client bundle. The HMAC app key and secret exist solely in the CMS's user-secrets for the sync job. The FX SDK key likewise never ships to the browser; decisions happen server-side. Single key for delivery, HMAC for management, neither in the frontend: that's the documented split, and it survives a security review.

Sharp edges, part 2: all of these drew blood

Same tradition as part 1: none of these are hypotheticals. Each cost me real time on a real instance; they're ordered by how much.

The CMS 12 Graph package kills CMS 13 at startup. Optimizely.ContentGraph.Cms 4.4.1 restores fine against CMS 13 (NuGet only warns, NU1608), then dies on boot with a TypeLoadException on PropertyContentArea. CMS 13 wants the version-matched Optimizely.Graph.Cms package (13.0.2 against CMS 13.0.2; the dependency pins are exact, so the package version tracks the CMS version). The registration is services.AddContentGraph() from Optimizely.Graph.DependencyInjection. Treat NU1608 on an Optimizely package as an error, not a warning.

The C# SDK's default logger is a no-op. Not "logs to a place you forgot to check": the shipped DefaultLogger.Log formats the message and discards it. A wrong SDK key, a blocked CDN, an unknown event key in TrackEvent: all invisible. Ten lines of adapter forwarding OptimizelySDK.Logger.ILogger into Microsoft.Extensions.Logging, registered via OptimizelyFactory.SetLogger() before creating the client, and the SDK starts telling you things. Do this on day one.

The JS SDK fetches the datafile once. The C# factory polls in the background by default; the JS createInstance does a one-time fetch unless you pass datafileOptions: { autoUpdate: true }. Without it, your Next.js server keeps serving yesterday's flag state until someone restarts it, and "pause the experiment" in the dashboard pauses nothing.

The empty-SDK-key fallback is a trap with the factory. Feed OptimizelyFactory.NewDefaultInstance a placeholder key and you get a real polling manager hammering the CDN for a datafile that doesn't exist (HTTP 403, every interval, forever), and the first Decide blocks the full default fifteen-second readiness timeout before giving up. First page view after deploy hangs fifteen seconds; everything after is fine; nobody can reproduce it. Construct an intentionally invalid client when the key is absent and Decide fails fast instead.

The sync job and the disabled scheduler. Dev environments routinely run SchedulerOptions.Enabled = false. Content publishes index into Graph incrementally via events, but the initial full sync is a scheduled job, which never fires on a disabled scheduler. Symptom: Graph authenticates, the schema is live, _Content returns total: 0, and every layer of your stack is working correctly. Run the synchronization job manually once from Admin → Scheduled Jobs.

Don't parse the item ID. The pre-release docs show variation items with IDs shaped like Guid_Status_Language_VariantKey. That's an example, not a contract. The supported address is _metadata.variation; the original arm is the item where it's null. String-splitting the ID is the kind of thing that works for eleven months and then doesn't.

What I deliberately left out

Scope is a feature. Here's what this build skips and why, so you can disagree on purpose rather than by accident.

User Profile Service. Out of the box, bucketing is sticky because the hash is deterministic: same visitor ID, same arm. But determinism has limits the docs are upfront about: change the traffic allocation mid-flight and some visitors re-bucket. UPS is the documented fix, a key-value store the SDK consults before hashing, pinning each visitor's first assignment forever. It's the right call for long-running production experiments; for a demo it's a database for a problem you don't have yet.

Allowlists and forced decisions. The rule editor lets you pin specific user IDs to specific arms (up to fifty), and the SDK has forced-decision APIs on top. Invaluable for QA ("make me always see variant_b"), and I'd wire it into any real project's test plan. Here, the New visitor button (delete the cookie, re-roll the dice) covers the demo's need with zero configuration.

Edge decisions. The architecture above decides on the server, per request. The documented next step for latency-obsessed setups is pushing the datafile to the edge (Vercel's reference implementation pipes it through a webhook into Edge Config) and bucketing in middleware, before the request touches your origin. Same SDK, same hash, same arms, just earlier. It composes cleanly with everything in this article precisely because the decision logic has no server-side state to migrate.

The take

The pitch for this pairing is genuinely short. The CMS owns the what: versioned, published, auditable content arms that editors manage with the tools they already know. FX owns the who and whether: deterministic assignment, exposure counting, a stats engine that survives scrutiny. The integration surface between two entire products is a string equality, and the same experiment served an MVC head and a headless Next.js head with zero coordination code, holding 33/33/34 within sampling noise on both.

For the business reader who skimmed to the end: this means an A/B test on real CMS content no longer requires a front-end rebuild, a personalization middleware, or a tag-manager hack that your security team hates. Editors make the arms; the dashboard runs the test; either rendering stack, today's or the one you migrate to next year, serves it unchanged. The experiment outlives your architecture decisions. That's the durable part.

The fragile part is everything in the sharp-edges list, and one sentence summarizes all of it: the integration fails silent, never loud. Wrong package, wrong key, renamed variation, stale datafile, unsynced index: every failure mode degrades to "everyone sees master" with green status everywhere. Wrap it in the diagnostics panel, the startup warnings, the distribution test. The contract is one string; the engineering is making sure you notice when the string stops matching.

Quiet hero, part 2: now with a stats engine. Still quiet. Still worth it.

The github repo: cms13-fx-content-variations.

Content Variations: Optimizely CMS 13's Quiet Hero

Mon, 08 Jun 2026 06:34:41 GMT

Every release has a headline act. CMS 13’s is the obvious trio. Visual Builder became the default editing surface. Optimizely Graph and Opti ID are now mandatory in every license. The whole platform moved to .NET 10. Those got the demo slots and the launch-day screenshots.

The feature I keep coming back to with clients isn’t any of them. It shipped quietly, got one modest line in the GA notes, and it’s going to change how editorial teams actually work day to day. Content Variations.

You only appreciate it after years of building the workarounds it kills. Let me make the case. Then, because these are field notes and not a brochure, I’ll show you where the sharp edges are and how we wrapped them.

What it actually is

Content Variations let an editor keep several published versions of the same content item, in the same language, from one source record. Not a copy in a sibling node. Not a language branch pressed into service as a fake A/B slot. The same item, with more than one published face.

The mechanics are better than the one-liner suggests. Variations are delta-based. A new variation starts with no property data and stores only the properties you actually change, not a full clone. Each one keeps its own version history and its own publishing lifecycle, and you can publish it independently of the original as long as the source itself is published. If the instance has a content-approval sequence configured, variations go through it too.

It shows up as a Variations dropdown in the editing toolbar, the “Select a variation of the page” control, with full autosave. You add one and name it. The name can’t include spaces and can’t start with a number. The docs use WinterCampaign as their example. Then you’re editing a divergent version in place. You can also seed a new variation from an existing one, which copies its content instead of starting from an empty delta. There’s one structural rule. You can’t create a variation off Root.

When you have a winner, you promote it back into the canonical line. That’s where the first sharp edge hides, so hold that thought.

Why this is the quiet hero

Remember how we used to do this.

Want an A/B test on a landing page? You duplicated the page, ran it through whatever experimentation tool you’d bolted on, and prayed the URLs and canonical tags behaved. Want different content for different audiences? You either bought a separate personalization product or built a content area full of visibility rules until every page was a logic puzzle. Want a seasonal variant that didn’t disturb the permanent URL structure? You forked the tree and took on the cleanup debt.

All of those share one disease. They treat a variation of one thing as several different things. The tree grew. Reporting fragmented. Six months later nobody could say which of the four near-identical pages was the real one, and everyone was too scared to delete any of them.

Content Variations collapse that into one canonical record with several published expressions, and a single answer to “what is this page.” The experimentation, personalization, and campaign work that used to need structural gymnastics now sits inside the content item, where it belongs. It isn’t flashy. It’s structural, and structural fixes pay off for years.

The Graph angle is where it gets interesting

CMS 13 is cloud-first and headless. Headless delivery runs through Optimizely Graph. Classic ASP.NET MVC/Razor rendering still works without it, but Graph is the spine. The real question isn’t whether editors can make variations. It’s how the front end gets the right one back. That’s Graph’s job, and the design choices here are worth reading closely, because they’re also where the risk lives.

Start with identity. Each variation is identified by a unique string held in a variation field on the content item, the variation key. That field is how you address a variation in a query. There’s deliberately no tidy GUID that means “this variation.” You select one by filtering on the key.

Then indexing. Every content variation, including unpublished drafts, is indexed to Graph with a unique identifier, so they’re discoverable for previews and experiments. The docs are careful about the result. A variation isn’t indexed inline with its source. Each one goes in merged with its source as a self-contained document, which is why a delivery query gets back fully resolved content instead of a bare delta. Treat the exact shape of the identifier as internal. It’s a composite of content GUID, version status, language, and variation key. The pre-release docs show it as Guid_Status_Language_VariantKey, but that’s an example, not a contract. Address variations through _metadata (_metadata { key displayName }). Don’t string-split an id.

One default decides everything here.

By default, GraphQL queries don’t return variations at all. A standard query gives you only the original, canonical content. To get a variation, or to preview one, you opt in explicitly.

The opt-in is a variation argument. include is an enum (ALL | SOME | NONE), value is an array of variation keys, and includeOriginal is a boolean.

# Default: variations are invisible. Canonical content only.
query {
  _Content {
    items { _metadata { key displayName } }
  }
}

# Opt in to everything. Now you own whatever comes back.
query {
  _Content(variation: { include: ALL, includeOriginal: true }) {
    items { _metadata { key displayName } }
  }
}

# The disciplined form. Name the arms you actually want.
query {
  _Content(variation: { include: SOME, value: ["WinterCampaign"], includeOriginal: true }) {
    items { _metadata { key displayName } }
  }
}

The C# SDK (Optimizely.Graph.Cms.Query) mirrors this with SetVariation(...), so the discipline is the same whether you write GraphQL or build the query in C#.

That default is a safety valve, and a good one. A developer who’s never heard of variations writes the obvious query and never accidentally serves experimental content to the public. The flip side is that the moment you opt in, you own everything that comes back. Don’t assume a server-side default for includeOriginal. Set it. The SDK makes you anyway. SetVariation takes includeOriginal as a boolean in its direct overloads, like SetVariation(includeOriginal: true, "WinterCampaign"), and through the options object in the builder overload. Be that explicit in raw GraphQL too.

The part the launch post skipped: where this bites

This is the OMVP half, the part I’d want a client to read before they get excited.

1. Drafts are in the index, and include: ALL is a loaded gun

Unpublished variation drafts are indexed. The default protects you. The opt-in is yours to get wrong. If a delivery query reaches for ALL without pinning which variation it wants, or someone pastes a preview query into a production resolver, you can ship half-finished, experimental, or off-brand content straight to anonymous visitors. include: ALL belongs in preview and selection logic, never as a casual “give me everything.” When you want one arm, use include: SOME and name the key.

Keep one distinction straight. Indexed is not the same as returned. Drafts sit in the index, but anonymous queries still resolve to published content, and pulling unpublished variation content needs a preview-authenticated context. The sharp public-leak risk is published variation content slipping out through an unscoped ALL, with draft leakage on top wherever a preview surface gets misused.

2. Promotion is two operations, and only one is dangerous

This is the bit I see told wrong most often, so let’s be exact.

“Copy changes to Original” pushes the variation’s modified (delta) properties back onto the original. If the original already has a published common draft, you get a new draft version of the original. It doesn’t overwrite the live published version, it doesn’t auto-publish, and version history stays intact. An editor still has to publish that draft. The UI even says “Changes copied to Original!” Nothing is destroyed, so this path is recoverable. It’s a merge into a new version, not a clobber.

Promoting a variation as the default version is the sharp one. It creates a new version of the source, merges the variation’s modified properties in, and can delete the variation afterwards. Because variations are deltas, only the properties the variation actually changed get merged. But deleting the variation takes that test arm’s separate history with it.

The risk is real and specific. It lives in the promote-as-default-and-delete path, not in every promote. One confident click on the wrong option can wipe a test arm’s history for good. That needs process, not just permissions, and ideally a diff in front of the editor before the merge runs.

3. Deltas are bigger than editors think

Delta storage sounds surgical. It isn’t, quite. The granularity is property-level. Change one value inside a complex property and the whole property gets copied into the delta. There’s no finer sub-field delta. It bites hardest on Visual Builder Experience content. The Experience composition is itself a complex property, so changing a single block value pulls the entire Outline and its Sections into the delta. “I only touched the hero” is rarely true at the storage layer. That changes how much actually diverges, which changes what a promote or a re-sync does.

I didn’t want this one to stay prose, so I measured it at the persistence layer. The harness, the byte dump, and the before/after table are in From prose to proof below. Short version. Touching one string inside one block of a Visual Builder Experience wrote ~3.7 KB of delta, the whole composition, for an 18-byte edit. That’s a ~207× blow-up. Even a plain ContentArea amplified ~37×.

4. Reference integrity has gaps

Two limitations sit right there in the docs’ own Initial Phase Limitations.

Variations currently vary only localizable properties. Non-localizable, culture-invariant properties can’t diverge. The “currently” is the docs’ word, so treat it as release-bound and re-check it on whatever version you ship.

Softlinks aren’t generated for published variations. Any reference that exists only inside a variation, a link, a content-area item, a media usage, is invisible to the platform’s reference tracking.

The consequence is the same either way. If your governance, link-checking, or “where is this used” reporting leans on reference data, it’s partly blind to variation-only references. Add experimental URLs to that and you have a real duplicate-content and broken-reference risk that won’t show up where you normally look.

5. Governance scales with the surface, and most teams forget

Approval workflow covers variations if you’ve set it up. But variations multiply the number of things that can be published. If your approval gate only sits on the main editorial flow, you’ve quietly opened a side door. Nothing in the box stops a page from collecting fourteen orphaned variations nobody remembers making. Variation sprawl is the new tree sprawl. It’s just harder to see, because it doesn’t show up as nodes.

6. Index state is an environment footgun

A feature flag controls variation indexing, and flipping it off takes the index with it. Clean kill switch, and an easy thing to get wrong across environments. Flag and index state can drift, and an indexing gap is silent. That silent failure is field observation, not a documented guarantee. A variation that should be live just isn’t in the index, your previews and experiments quietly vanish, and nothing throws. The pre-release content-variations docs call this out and the GA notes don’t repeat it, so confirm it on the build you ship. I wouldn’t hard-code a flag name in a runbook either. Treat it as the variations feature toggle and check it per environment. Silent failure is the worst kind.

7. Pulling variation sets at scale costs you on caching

This one is a general Graph property, and it’s documented. Broad items queries trash your cache. They force it to “account for potential changes across all content,” in the docs’ words, so it invalidates constantly. Narrow single-item lookups cache cleanly and hit far more often. The queries you write to pull variations across content are exactly the broad items kind. The docs don’t tie variations to caching directly, but the query shape you reach for is the one that caches worst. On a busy site that’s a real line item, not a footnote.

“Trash your cache” is a claim with no number attached, so I attached one. The environment, the harness, and the measured hit-rate table for item vs items and SOME vs ALL are in From prose to proof below. Short version. In a deterministic cache model of a 5,000-page set, a narrow item lookup held a 99.2% hit-rate while items(limit:100) with include: ALL collapsed to 7.7%. A single base publish invalidates one item key but hundreds of broad-listing keys.

None of this is a reason to avoid the feature. It’s a reason to wrap it.

From prose to proof: two measurements

Two of the edges above are the kind of claim a reader has to take on faith. The delta is “coarser than it looks.” Broad queries “trash the cache.” Faith is the wrong currency for an architecture decision. Both are measured below, with the environment described and a harness you can re-run. Both experiments ship in the add-on’s tools/research/ folder and were run for this article. dotnet run -c Release reproduces the tables byte-for-byte (deterministic, seeded), and the exact build/date stamp lives in the generated RESULTS.md. One honesty note up front. The delta figures are real serialization measurements of representative property graphs, not a dump from a live store (DeltaProbe.cs is provided for that). The cache figures come from a deterministic discrete-event model of Graph’s documented invalidation rule. The model measures the hit-rate that follows from that rule under stated fragmentation assumptions. The direction is robust, but absolute magnitudes, and any wall-clock latency (p50/p99), need the live gateway and the graph-cache.k6.js script, which isn’t run here. The full list of what these caveats mean is in the Threats to validity section below.

Measuring the delta: what “I only touched the hero” actually writes

The claim under test. Changing a single value inside a complex property copies the entire property into the variation delta, and for a Visual Builder Experience that means the whole Outline + Sections, not the field you touched.

Method. A CMS 13 variation version stores only the properties that diverge from its source. There’s no public “give me the raw delta bytes” call, but you can reconstruct it faithfully. Load the source version and the variation version, walk the property bag, and for each property serialize both values and compare. Properties whose serialized bytes differ are the delta. Their serialized size is what diverged. This byte-diff is the same set Graph merges into the variation document, so it’s a faithful proxy for the changed-property payload a variation carries. The exact on-disk bytes can differ, which is what the Threats to validity note covers.

// DeltaProbe.cs - reconstructs a variation's delta by byte-diffing it against its source version.
// Run once on a freshly-seeded variation (expect: empty delta), then again after touching ONE block.
public sealed record DeltaRow(string Property, string Type, bool InDelta, int Bytes);

public sealed class DeltaProbe
{
    private readonly IContentLoader _loader;
    public DeltaProbe(IContentLoader loader) => _loader = loader;

    public IReadOnlyList<DeltaRow> Diff(ContentReference original, ContentReference variation)
    {
        var src = _loader.Get<IContent>(original);
        var var_ = _loader.Get<IContent>(variation);
        var rows = new List<DeltaRow>();

        foreach (var vp in var_.Property)
        {
            var sp = src.Property[vp.Name];
            var sb = Serialize(sp?.Value);
            var vb = Serialize(vp.Value);
            var inDelta = !sb.AsSpan().SequenceEqual(vb);   // differs => part of the delta
            rows.Add(new DeltaRow(vp.Name, vp.GetType().Name, inDelta, inDelta ? vb.Length : 0));
        }
        return rows;
    }

    // Stable, value-only serialization so the comparison reflects content, not object identity.
    private static byte[] Serialize(object? value) => value is null
        ? Array.Empty<byte>()
        : Encoding.UTF8.GetBytes(JsonSerializer.Serialize(value, JsonOpts));

    private static readonly JsonSerializerOptions JsonOpts =
        new() { ReferenceHandler = ReferenceHandler.IgnoreCycles, WriteIndented = false };
}

Environment. Figures come from the deterministic harness in tools/research. It builds two representative property graphs. One is a standard page with a ContentArea of local blocks. The other is a Visual Builder Experience of five sections (hero, a 3-up feature row, a story split, a product band, a signup). It serializes each, applies exactly one nested edit (the hero headline "Winter Sale" → "Winter Sale - Up to 40% off", an 18-byte change), and re-serializes. The DeltaProbe.cs above is the same byte-diff run against live content via IContentLoader. The harness just removes the need for a seeded site to reproduce the effect.

Result, standard page (ContentArea).

Property	Type	In delta?	Serialized bytes
Heading	PropertyString	no	-
MainBody	PropertyXhtmlString	no	-
MainContentArea	PropertyContentArea	yes	664

The headline lives in a local block inside MainContentArea. Editing it marked the whole content-area property dirty, every block, not just the changed one. 664 bytes landed in the delta for an 18-byte edit, a ~37× amplification.

Result, Visual Builder Experience page.

Property	Type	In delta?	Serialized bytes
Name	(metadata)	no	-
Composition	Experience composition (Outline + Sections)	yes	3,727

There it is, at the persistence layer. The Experience composition is one complex property. Touching one string in one block rewrote the entire Outline and all five sections into the delta. That’s 3,727 bytes against the 18 bytes the edit actually represents, a ~207× amplification. And that’s a deliberately modest composition. A production Experience with imagery, more blocks, and richer settings runs to tens of KB, which pushes the ratio into the thousands. “I only touched the hero” is false where it counts. Seed twenty audience variations of this one page and you’ve written ~75 KB of near-duplicate composition, and every promote or sync moves all of it.

Why it matters beyond disk. That delta is exactly what a promote merges and what Sync from default has to reconcile. A coarse delta means a coarse merge surface. More properties in play, more chances for a stale-conflict, more to diff in front of the editor. The byte number isn’t vanity. It’s the size of the blast radius.

Measuring the cache: item vs items, and the cost of include: ALL

The claim under test. Broad items queries cache badly, narrow item lookups cache well, and opting into variations with include: ALL makes it worse. A single master publish invalidates broadly.

Environment. Two layers. The hit-rate numbers below come from a deterministic discrete-event model (tools/research, run on .NET 10) of Graph’s documented cache rule. A result is keyed by its query, and a write invalidates every key whose result could change. The model uses the article’s dataset shape, 5,000 pages and 500 varied × 3 = 1,500 variants, exercised over a fixed 100-id working set with 200,000 reads and one publish per 100 reads. It models the decisive variable explicitly. A narrow item lookup has one key per id, while a broad items listing fragments across filter/sort/page keys (150), with SOME (300) and ALL (600) fragmenting further. The graph-cache.k6.js script below is the live counterpart that measures p50/p99 latency against a real gateway. k6 wasn’t run for this article, so latency is left to you. The model measures the hit-rate that follows from the cache-key/invalidation rule under those stated fragmentation assumptions. The ordering is robust. The exact percentages move with the assumptions.

// graph-cache.k6.js — compares cache behaviour of item vs items, and SOME vs ALL.
import http from 'k6/http';
import { Trend, Rate } from 'k6/metrics';

const URL = `https://cg.optimizely.com/content/v2?auth=${__ENV.KEY}`;
const hit = {};      // Rate per scenario
const lat = {};      // Trend per scenario
for (const s of ['item', 'item_some', 'items', 'items_some', 'items_all']) {
  hit[s] = new Rate(`hit_${s}`);
  lat[s] = new Trend(`lat_${s}`, true);
}

const Q = {
  item:       (id) => `{ _Content(where:{_metadata:{key:{eq:"${id}"}}}, limit:1){ items{ _metadata{ key displayName } } } }`,
  item_some:  (id) => `{ _Content(where:{_metadata:{key:{eq:"${id}"}}}, limit:1, variation:{ include: SOME, value:["WinterCampaign"], includeOriginal:true }){ items{ _metadata{ key } } } }`,
  items:      ()   => `{ _Content(limit:100){ items{ _metadata{ key displayName } } } }`,
  items_some: ()   => `{ _Content(limit:100, variation:{ include: SOME, value:["WinterCampaign"], includeOriginal:true }){ items{ _metadata{ key } } } }`,
  items_all:  ()   => `{ _Content(limit:100, variation:{ include: ALL, includeOriginal:true }){ items{ _metadata{ key } } } }`,
};

export default function () {
  const id = IDS[Math.floor(Math.random() * IDS.length)]; // fixed 100-id working set
  for (const s of Object.keys(Q)) {
    const body = JSON.stringify({ query: s.startsWith('item') ? Q[s](id) : Q[s]() });
    const res = http.post(URL, body, { headers: { 'Content-Type': 'application/json' } });
    const cached = (res.headers['X-Cache'] || res.headers['Cf-Cache-Status'] || '').toLowerCase().includes('hit');
    hit[s].add(cached);
    lat[s].add(res.timings.duration);
  }
}

Result, hit-rate by query shape (model).

Query shape	Variation arg	Reads	Origin fetches	Hit-rate
item(key)	none (canonical)	200,000	1,673	99.2%
item(key)	SOME ["WinterCampaign"]	200,000	1,885	99.1%
items(limit:100)	none	200,000	129,679	35.2%
items(limit:100)	SOME [key]	200,000	165,132	17.4%
items(limit:100)	ALL + includeOriginal	200,000	184,516	7.7%

Hit-rate (modelled), drawn so the cliff is obvious:

item            ██████████████████████████████████████████████████ ≈99%
item + SOME     ██████████████████████████████████████████████████ ≈99%
items           ██████████████████                                 ≈35%
items + SOME    █████████                                          ≈17%
items + ALL     ████                                               <10%
                0%        25%        50%        75%        100%

Two readings. First, query shape dominates. A narrow item lookup is a clean, stable per-id key that effectively always hits, 1,673 origin fetches across 200,000 reads (99.2%). A broad items listing fragments across keys, each reused less and invalidated whenever any member changes, so it fell to 35.2%. Second, opting into variations compounds it. SOME halves the broad hit-rate again (17.4%). ALL folds all 1,500 variants into the candidate space and floors it at 7.7%, about 110× the origin traffic of the narrow lookup (184,516 vs 1,673 fetches).

Result, invalidation cost of one base publish (model). Warm every key, publish a single base content inside the working set, then count the keys each shape must re-fetch from origin.

Query shape	Warm keys	Keys invalidated by 1 publish
item	100	1
item + SOME	100	1
items	150	150
items + SOME	300	300
items + ALL	600	600

That’s the asymmetry that matters. A base publish punches a single hole in the item cache, the published id, nobody else. The same publish poisons every broad listing key that could contain it, because the gateway can’t prove it doesn’t, so it invalidates the whole family. The variation-inventory queries you’re tempted to write (“show me every variation across the site”) are precisely this broad items shape. That’s why the Auditor’s grid resolves variation keys through narrow per-item lookups and batches the deliverability probe one round-trip per content id, instead of one broad items(variation: ALL) sweep. The measurement drove the design.

Repeatability. The whole harness ships in the add-on’s tools/research/ folder: DeltaExperiment.cs, CacheExperiment.cs, the graph-cache.k6.js live script, and the generated RESULTS.md. dotnet run -c Release reproduces both tables byte-for-byte (the run is deterministic, seeded 1337). Point DeltaProbe.cs and the k6 script at a live CMS 13 + Graph environment for your own live numbers. Absolute values move with dataset size, composition size, and gateway region. The cliff between item and items + ALL, and the property-level delta amplification on Experience content, do not.

Threats to validity

Both experiments confirm the direction of the claims. Be precise about what they don’t prove.

Delta is a serialization model, not a live store dump. It demonstrates the mechanism (any nested change re-serializes the whole complex property) using representative graphs. The exact on-disk bytes CMS 13 writes can differ. Run DeltaProbe.cs against real content for authoritative figures.
Delta magnitude scales with composition size. The 207× here is from a deliberately modest five-section Experience (3.7 KB). A production Experience with imagery and richer settings amplifies far more. The ratio is not a constant.
The ContentArea result assumes local/inline blocks. If the area holds shared block references, editing the referenced block changes a different content item, not the page’s delta. The Experience composition embeds its blocks, so that case holds robustly.
Cache numbers are a model, not a live-gateway measurement. Hit-rates follow from assumed key-fragmentation counts (150/300/600) and an invalidation rule where every listing key covers the whole working set. That rule maximises the broad-query penalty. A gateway that scopes invalidation more narrowly would show a softer, same-direction cliff. Only graph-cache.k6.js against a real gateway settles the magnitudes.
No latency was measured. Every p50/p99 statement is deferred to the live k6 run. None is asserted here.

None of these soften the practical takeaway. Narrow lookups and bounded variation scoping win. Coarse deltas and include: ALL lose. But they mark exactly where “measured” ends and “modelled” begins.

Live spot-check. I did point a read-only delivery key at a real Graph gateway to ground-truth the mechanism, not the magnitudes. Confirmed live. The schema accepts the variation: { include: ALL | SOME | NONE, includeOriginal } argument. The gateway caches with Cache-Control: public, max-age=86400 and reports CF-Cache-Status: HIT/MISS. A cache-miss include: ALL query was the costliest at the origin (p50 ~506 ms vs ~406 ms for a narrow item miss). What that source couldn’t validate (it held 24 items and zero variations, and the key can’t publish) is the hit-rate cliff at scale and the invalidation asymmetry, which still rest on the model. Mechanism confirmed on a live gateway, magnitudes still modelled.

Where the add-on comes in: PiNo Labs Variations Auditor

Most of the risks above aren’t bugs. They’re sharp edges that come with a powerful primitive being honest about what it does. The platform gives you the capability and a sensible default. It doesn’t have opinions about how your team should use it. That gap is what a thin add-on fills.

In our Foundation solution we packaged the governance into a small CMS 13 add-on, PiNo.Labs.VariationsAuditor, that turns the variation estate from a blind spot into something you can see, measure, and act on. It installs as one self-registering protected shell module: an IConfigurableModule, a ProtectedModuleOptions self-registration, an /api/auditor/* REST surface, a shell menu entry, and a React UI embedded in the assembly and served straight from the DLL, with no wwwroot copy step. It authorizes against the canonical CMS roles Opti ID syncs onto the principal, so the gate behaves the same on DXP and locally. Every external dependency it leans on, Graph most of all, is resolved optionally and degrades gracefully when it’s missing.

A fair question before the code. Every platform seam the snippets touch (shell-module and menu registration, IConfigurableModule/ProtectedModuleOptions, the IContentEvents.PublishedContent hook, the versioning APIs) sits inside CMS 13’s breaking-change surface. They’re shown as they compile against the CMS 13 / .NET 10 assemblies the add-on actually ships on, written for those breaking changes rather than refactored from CMS 12. Here’s what it does, mapped to the edges above.

It starts from the one inconvenient truth the platform hands you. There’s no in-process, site-wide API for “all content that has variations.” A CMS 13 content variation is just a content version carrying a non-empty IVersionable.Variation key, and a variation’s identity is a tuple of content, language, and variation key, not a GUID.

// A CMS 13 "content variation" is a version with a non-empty IVersionable.Variation key.
var variants = versionRepository
    .List(new VersionFilter { ContentLink = contentLink }, 0, max, out _)
    .Where(v => !string.IsNullOrEmpty(v.Variation));     // empty key == language baseline, not a variant

Discovery uses Graph to enumerate the site, then reads the authoritative variation keys from IContentVersionRepository. If Graph is down or hasn’t finished indexing, it falls back to an in-process content-tree walk. The grid works on day one, indexed or not.

A pre-promotion diff, so promotion is a decision and not a reflex (edges #2 and #3). Every destructive action runs a mandatory, conflict-aware dry-run first. The divergence engine computes a property-level diff and a structural content-area diff, so the editor sees exactly which properties will land back on the original. It also catches the dangerous case the platform won’t warn about. Promoting a variation whose source has moved on since.

// Stale-conflict guard: refuse to silently overwrite newer master changes on promote.
var stale = divergence.GetStaleProperties(target);
if (action == BulkActionType.Promote && stale.Count > 0 && !cmd.ForceOverwriteStale)
{
    return Blocked(
        "CRITICAL REVERSION RISK: promoting would overwrite newer changes on the original for: " +
        string.Join(", ", stale) + ". Re-run with ForceOverwriteStale to proceed.");
}

Promote is a delta merge, full stop. It overlays only the properties the variation actually overrode onto a writable clone of the published original, and reports which ones it touched. That keeps CMS 13’s delta model intact instead of flattening master-only properties. We pulled an earlier “replace everything” mode because it broke exactly that model. The same dry-run, lock, and status gate guards Unpublish and Delete. The inverse action, “Sync from default,” lets an editor pull the original’s current value back into a stale variation, which is the safe fix for edge #3.

A site-wide orphan and sprawl report, so cleanup actually happens (edge #5). The grid is the inventory nobody had. Every variation across the site, the audience it targets (the key resolved to a readable visitor group, because editors think in audiences, not keys), its divergence state, and a needs-attention filter. Multi-select plus a dry-run bulk bar turns unpublishing or deleting a pile of orphaned arms into a two-click, fully previewed operation. Sprawl you can retire.

A deliverability probe that turns silent index failures loud (edges #4 and #6). This is the canary for “my variation should be live but isn’t in Graph.” It checks each published variant against the index, batched and de-duplicated per content id so a grid page costs one round-trip instead of N, and flags anything known to CMS but missing from the index as an orphan. Softlinks are blind to variations, so this Graph-backed inventory is the “where are my variations, and are they being served” view the reference system can’t give you. It doesn’t lie the other way either. If the Graph client isn’t registered at all, the probe returns deliverable rather than painting every variant red.

// Optional Graph client, never required. No AddGraphContentClient()? Degrade, don't crash.
_graph = serviceProvider.GetService<IGraphContentClient>();
...
if (_graph == null) return GraphStatus.Deliverable;        // unknown != orphan; never mislabel
var indexed = await IsIndexedAsync(identity.ContentId, ct);
return indexed ? GraphStatus.Deliverable : GraphStatus.IndexGap;   // IndexGap == orphan, surfaced in the grid

Proactive drift notifications, so editors come back when something changes (edge #5 again). A StaleDriftDetector subscribes to IContentEvents.PublishedContent. When a default publishes, it recomputes its variants’ staleness and tells the owner of each newly stale variation. The default just moved, your audience is now on yesterday’s content for these three properties. It’s bounded, de-duplicated per variant and save, runs in its own DI scope, and is wrapped so it can never fail the publish that triggered it. Delivery fans out across pluggable INotificationChannels (webhook to Slack/Teams, SMTP email, log), all off until the host opts in.

public void Initialize(InitializationEngine context)
    => context.Locate.ContentEvents().PublishedContent += OnDefaultPublished; // drift stops being silent

And the one guardrail that belongs at the delivery layer, not in the editor (edge #1). The auditor governs the editorial and estate side, which is the side it can see. The include: ALL problem is a delivery concern, and the cheapest place to close it is a thin convention around your Graph client. A typed query wrapper, or a CI lint over your resolvers, that refuses an unscoped ALL on the production delivery key and makes callers pass an explicit variation key or a preview context. The platform default already protects the naive query. This protects you from the clever one. Run both and you’ve covered what’s in the estate and what reaches the visitor.

The point isn’t the code. It’s the posture. The platform ships the engine and a sensible default. You ship the seatbelts that fit your team’s risk tolerance. That’s what an add-on is for.

The full source of the add-on is available on GitHub at https://github.com/pino-labs/pino-labs-variations-auditor.

The take

A year from now, Content Variations won’t be what people remember about CMS 13. Visual Builder and the headless story will take that slot. But it’s the feature that quietly removes a whole category of structural debt teams have carried for years, and it deserved more than one launch-day line.

Just go in with your eyes open. Drafts live in the index. The opt-in is yours to misuse. Promotion has two doors, and one of them deletes history. Deltas are coarser than they look. Softlinks and index state have edges. Put a thin layer of governance and tooling around the feature: a pre-promotion diff, a site-wide orphan report, a deliverability canary, a delivery-side guard. Then let your editors do the thing they’ve always wanted, which is to experiment on a page without turning the content tree into an archaeological dig.

Quiet hero. Sharp edges. Worth it..

Running 64 Sites on Headless Optimizely CMS with GraphQL

Sat, 14 Mar 2026 17:57:22 GMT

64 websites. Live. Running on headless Optimizely with GraphQL.

We just wrapped a major rollout for our Rockwool Digital Experience Platform and the early numbers already tell a story.

This wasn't about moving pixels. It was about rethinking how users actually interact with the brand. Here's what we're seeing so far:

Product navigation is pulling its weight. A leaner UI means visitors hit conversion pages faster.
Documentation is finally findable. Doc downloads among active users have jumped noticeably.
Dealer pages are surging. A smoother user journey created a much clearer path for low-touch leads.

What it took behind the scenes

This was not a small lift. Making it work across 64 sites meant going deep into the stack:

Custom scaling: We extended Algolia queries and our Product Service to fully support sub-brands.
Independent pipelines: Autonomous release processes and brand-specific Next.js configs keep things flexible.
Smart routing: Azure Front Door handles the transition from legacy systems to the new Next.js environment.
The details: Complex language mapping, domain consolidation, Inriver content tagging, massive redirect prep all of it had to land.

The result? A platform that doesn't just look better - it actually performs. We've built a foundation where updates ship in days, not months.

In the next article, I'd like to walk through the underlying architecture. I hope.

64 Websites Live on Headless Optimizely with GraphQL

Fri, 13 Mar 2026 00:00:00 GMT

Major rollout for Digital Experience Platform complete. Product navigation working harder, documentation downloads up, dealer traffic growing.

Moving Beyond "Lift and Shift": Lessons from an Optimizely DXP Migration

Sat, 14 Feb 2026 00:00:00 GMT

Successfully migrated from legacy Windows to Optimizely DXP Cloud-native Linux. Cross-platform compatibility, WAF rules, and Infrastructure as Code.

Rockworld 2.0: A Strategic System Migration

Sat, 14 Jun 2025 00:00:00 GMT

Complex migration from legacy system to modern headless architecture using Optimizely Headless CMS, GraphQL, Next.js, and full Azure infrastructure.

Blog posts by Piotr Nowak - Optimizely & Azure

Content Variations in CMS 13, Part 3: Audiences vs Audiences

The machine at a glance

Two engines, one word: both of them are called Audiences now

Switching the engine on: in CMS 13, personalization is opt-in three layers deep

Reference card: the CMS 13 personalization bill of materials

The attribute pipeline: FX only knows what you tell it

Geolocation: one header now feeds both engines

Targeted Delivery: personalization without the dice

Can both engines coexist on one page? Yes, at different layers, measurably

What does Content Graph actually do with a personalized block?

The decision table

From prose to proof: the day the headless head went dark

QA: three override levels, three owners

Runbook: “the page always serves master”, audience edition

Sharp edges, part 3: ranked by blood actually drawn

FAQ, the questions as people actually type them

The take

Glossary

Further reading

Unlock Experimentation with Content Variations in CMS 13

Who does what

The shape of the experiment

Serving it: one decision, one load, one fallback

The decision service: the only class that knows the SDK

From prose to proof, again: does 33/33/34 actually hold?

Plumbing the docs assume you'll remember

The same experiment, headless

Sharp edges, part 2: all of these drew blood

What I deliberately left out

The take

Further reading

Content Variations: Optimizely CMS 13's Quiet Hero

What it actually is

Why this is the quiet hero

The Graph angle is where it gets interesting

The part the launch post skipped: where this bites

1. Drafts are in the index, and include: ALL is a loaded gun

2. Promotion is two operations, and only one is dangerous

3. Deltas are bigger than editors think

4. Reference integrity has gaps

5. Governance scales with the surface, and most teams forget

6. Index state is an environment footgun

7. Pulling variation sets at scale costs you on caching

From prose to proof: two measurements

Measuring the delta: what “I only touched the hero” actually writes

Measuring the cache: item vs items, and the cost of include: ALL

Threats to validity

Where the add-on comes in: PiNo Labs Variations Auditor

The take

Further reading

Running 64 Sites on Headless Optimizely CMS with GraphQL

64 Websites Live on Headless Optimizely with GraphQL

Moving Beyond "Lift and Shift": Lessons from an Optimizely DXP Migration

Rockworld 2.0: A Strategic System Migration