A/B Testing

Beta

FilterIQ includes built-in A/B testing so you can compare different filter configurations and measure which one performs better. You manage experiments from the Experiments page (/app/analytics/experiments).

What A/B Testing Does

A/B testing splits your store traffic between two filter configurations (Variant A and Variant B) and measures key metrics for each. This lets you make data-driven decisions about your filter layout, ordering, and settings instead of guessing.

Common things to test:

Different filter ordering (e.g., Price first vs Color first)
Filter types (checkboxes vs swatches for color filtering)
Number of visible filters (showing 5 vs 8 filters in the sidebar)
Layout changes (sidebar vs horizontal toolbar)
Whether showing product counts improves click-through

Creating an Experiment

Go to Analytics > Experiments (/app/analytics/experiments)
Click Create Experiment
The Create Experiment modal appears with the following fields:

Test Name

Give your experiment a descriptive name (e.g., "Color swatches vs checkboxes" or "Filter ordering Q1 2026").

Variant A

This is typically your current (control) configuration. By default, Variant A uses your currently published filter configuration.

Variant B

This is the configuration you want to test against. You can:

Modify specific filters (change type, reorder, add, or remove)
Apply a different preset
Change appearance settings

Traffic Split

By default, traffic is split 50/50 between Variant A and Variant B. Each shopper is consistently assigned to the same variant for the duration of the experiment.

Click Start Experiment to begin

Running Experiments

Once an experiment is running:

Shoppers are randomly assigned to Variant A or Variant B
Assignment is consistent per session -- a shopper sees the same variant throughout their visit
The experiment runs until you manually stop it or it reaches a configured end date
You can monitor results in real time on the Experiments page

Experiment Status

Status	Meaning
Draft	Experiment is configured but not yet running
Running	Experiment is live and collecting data
Completed	Experiment has ended (manually stopped or reached end date)
Paused	Experiment is temporarily paused

Viewing Results

Click on any experiment to open the Results view. The results panel shows:

Key Metrics Per Variant

Metric	Description
Sessions	Number of unique shopper sessions in this variant
Filter Interactions	Total filter clicks/selections
Click-Through Rate	Percentage of sessions that clicked a product after filtering
Add-to-Cart Rate	Percentage of sessions that added a product to cart
Conversion Rate	Percentage of sessions that completed a purchase
Revenue Per Session	Average revenue generated per session

Statistical Significance

FilterIQ calculates whether the difference between variants is statistically significant. You will see:

Confidence level -- Displayed as a percentage (aim for 95% or higher before making decisions)
Winner indicator -- When confidence is high enough, the better-performing variant is highlighted
Sample size recommendation -- If results are inconclusive, a note indicates how many more sessions are needed

Interpreting Results

Clear winner

If one variant has a significantly higher conversion rate or revenue per session at 95%+ confidence, you can confidently adopt that configuration.

No significant difference

If the confidence level stays below 95% after sufficient traffic, the two configurations likely perform similarly. You can choose either one or test a more dramatic change.

Mixed results

If Variant A has higher click-through but Variant B has higher conversion, consider what matters most for your business. Revenue per session is usually the most reliable metric.

Ending an Experiment

To end an experiment:

Click on the running experiment
Click Stop Experiment
Choose whether to:
- Apply winner -- Publish the winning variant's configuration as your new default
- Keep current -- Discard results and keep your existing configuration
- Save both -- Save both variants as presets for future reference

When to Use A/B Testing

After major filter changes -- Test before committing to a new layout
Seasonal optimizations -- Test different filter configurations for holiday vs regular traffic
After adding new filters -- Verify that the new filter improves rather than overwhelms the experience
Periodically -- Customer behavior changes over time; re-test your assumptions every quarter

Best Practices

Test one change at a time -- If you change multiple things between variants, you will not know which change caused the difference
Run tests long enough -- Wait for statistical significance before making decisions (usually 1-2 weeks for medium-traffic stores)
Do not peek and stop early -- Early results can be misleading. Let the experiment reach the recommended sample size.
Document your tests -- Use descriptive experiment names so you can reference past results

Need Help?

If you need help designing an experiment or interpreting results, contact support@filteriq.app.