technology – Sitecore Tackled

A/B/n testing is a powerful tool used to understand which content engages your visitors the most. I’ll explain the concepts, share insights on how to get started, and what to consider when working with SitecoreAI’s A/B/n testing.

What is A/B/n testing and why is it important?

A/B/n testing is about comparing different versions of a webpage or a content element to see which version performs best. By formulating a clear hypothesis, for example, that a new color or text of a CTA button will increase page views by 10%, you can make data-driven decisions that improve user experience and conversion rates.

What is the difference between A/B testing and A/B/n testing?

The main difference between A/B testing and A/B/n testing is the number of versions, so-called variants, being tested.

A/B testing compares two variants, while A/B/n testing evaluates multiple variants simultaneously. An A/B test is a simple comparison between two options and tests a specific change to determine which performs better. An A/B/n test evaluates several design options, layouts, or headlines at the same time and provides a faster overview of which of many ideas works best.

What distinguishes SitecoreAI from traditional A/B testing tools?

SitecoreAI differs from traditional A/B testing tools in that it is tightly integrated with the entire Sitecore ecosystem and can use artificial intelligence to optimize tests. This includes:

Faster insights: SitecoreAI helps analyze results in real time and can identify winning variants faster than manual testing. With SitecoreAI, you can compare up to six variants.

Automation: You can configure rules to automatically direct all traffic to winning variants once results are statistically significant, reducing manual work.

Integration with Sitecore: Because the testing tool is built into the platform, you can easily connect A/B/n testing to other Sitecore features, such as forms and content management, without needing external tools.

Effective default goals: SitecoreAI comes with predefined goals (e.g., page views, form submissions, bounce rate) that make it easy for marketers to get started quickly without advanced statistical knowledge.

In short, SitecoreAI makes it possible to combine A/B/n testing with AI-driven insights in a way that is difficult to achieve with standalone testing tools. It saves time, delivers more accurate results, and simplifies optimization of the entire website.

How A/B/n testing works in SitecoreAI

First, you develop a hypothesis for your test. For example, reducing the number of fields in a form may increase form submissions by 10% compared to the current version.

Based on your hypothesis, you define the test goals. SitecoreAI includes predefined standard goals:

Increase page views
Increase form submissions
Reduce bounce rate
Reduce exit rate

You then create a control variant (Variant A) and a test variant (Variant B) and split traffic between them, either 50/50 or using another distribution you decide. As mentioned, you can create up to six variants.

You determine how long the test should run, but Sitecore recommends running it for at least one week to ensure statistically significant results.

SitecoreAI then provides insights into which variant performed best, or whether there is no difference (a null result). You can configure the system so that the winning variant automatically receives all traffic, or if results are inconclusive, that the test is extended or traffic is reverted to the original variant.

What you can test

Anything you can change on a page or in a form can be tested. Examples include:

Content: Images, CTA copy, headlines, and components
Sitecore forms: Number of fields, button placement
Design and layout

Tips for effective testing

Change as few variables as possible at a time. This makes results clearer and easier to interpret.
Prioritize areas of the page with the greatest impact, such as hero banners or purchase buttons.
Run tests regularly to account for changes in visitor behavior over time.
Set a confidence score (e.g., 95%) to determine when a winning variant should receive all traffic.
Note that A/B/n testing does not work simultaneously with personalization. Because personalization shows different content to different users, it can conflict with the purpose of the test. If you want to run an A/B/n test on a personalized page, the solution is to temporarily pause personalization while the test is running.

Overview

If you have multiple sites under a Site Collection and want to manage access per site or across the entire collection, it’s best to create custom roles. This gives you flexibility, for example, some editors can be Authors for the whole Site Collection, while others only have access to a specific Site.

If you only have one site or don’t need granular access, you can use the default Site Collection roles and Site roles in Sitecore. In that case, simply run the Setup Security script on your Site Collection and then on the Site.

Recommended Role Structure

Create custom roles before running Setup Security.
Preferably create 4 roles per Site Collection and 4 per Site. Use the Sitecore domain.

Site Collection Roles:
sitecore/MySiteCollection Admin
sitecore/MySiteCollection Author
sitecore/MySiteCollection Designer
sitecore/MySiteCollection Member

Site Roles:
sitecore/MySite Admin
sitecore/MySite Admin
sitecore/MySite Admin
sitecore/MySite Admin

Create your new custom roles for a new Site by creating 4 new roles.
Create new roles in the Sitecore domain. Follow this pattern:

For the headless Site Collection, you create 4 new roles for that Site Collection aswell.

Role Membership Setup

In Role Manager, select the <Site> Author role, then click Member Of. Make it a member of:

sitecore\<Site> Member (itself)
sitecore\<SiteCollection> Member

You can add additional Sitecore roles (e.g., Sitecore Client Securing) if needed.
Read more about Sitecore roles

Running the Setup Security Script

In Content Editor, right-click your Site Collection and run Scripts > Setup Security.

2. Choose sitecore as domain

3. In the dialog, link the 4 custom roles to the appropriate fields before clicking Assign.

4. Then repeat for the Site.

You can now assign users to your new Author role and they should be ready for editing your site. Don’t forget to setup Workflows to make the editor experience as expected in Page builder!

Notes

If you only have 1 site or don’t see a reason to split the roles up in any way, there are default Site Collection roles and Site roles predefined in Sitecore that you could use instead and just run the Setup Security script on your Site Collection and Site. But for granular access rights I recommend setting up site specific roles.