For Magento merchants considering a checkout migration, one of the most useful questions to answer is simple: how does the new checkout perform on your own store?
The open-source elgentos/magento2-hyva-checkout-ab-test module provides a way to compare Hyvä Checkout with another configured checkout using live traffic. It is designed to work with a licensed Hyvä Checkout installation.
Hyvä’s Checkout documentation notes that measuring the new checkout against an existing Luma-based checkout can be useful and points to the Elgentos module for this purpose.
Version 2.0.3 of the module was released on 27 February 2026. By the time this article was published in August, it was therefore already an available option for merchants evaluating Hyvä Checkout rather than a newly introduced testing tool.
Its value is that it gives merchants a practical framework for testing checkout variants with their own traffic rather than relying only on results from other stores.
Why checkout A/B testing is different from regular A/B testing
Many website experiments involve relatively contained frontend changes, such as alternative content, layouts or calls to action.
Checkout testing is more complex because the variant sits inside a transaction flow involving payment methods, addresses, shipping, tax calculations, order creation and potentially other integrations.
A meaningful checkout test therefore needs more than a visual variant. Both checkout experiences need to be configured well enough that differences in the results reflect the checkout experience being tested rather than missing functionality or inconsistent integrations.
The Elgentos module handles the checkout selection itself. In production mode, customers can be randomly assigned to configured checkout variants according to a percentage split.
According to the module’s documentation, this can include:
-
Hyvä Checkout and the Luma fallback checkout
-
different Hyvä Checkout configurations, such as one-page and multi-step variants
The module also supports manual checkout selection through a URL parameter when the relevant configuration allows it, which can be useful during development and QA.
What the module actually enables
Configurable traffic splitting
Merchants can configure how traffic is distributed between available checkout variants.
That makes it possible to expose only part of production traffic to a new checkout rather than switching every customer at once.
The appropriate split depends on the purpose of the test, available traffic and the amount of data required. A smaller treatment group limits exposure but will normally take longer to generate enough observations for a useful comparison.
Testing the actual checkout configuration
The module switches between configured checkout namespaces rather than creating a simulated checkout experience.
That allows merchants to test the checkout configurations they are actually considering for production, including a comparison between Hyvä Checkout and Luma fallback or between different Hyvä Checkout layouts.
However, merchants still need to verify that payment methods, address handling, shipping logic and other checkout dependencies behave correctly in each variant before starting a live experiment.
During QA, it is also worth confirming how variant assignment behaves for returning customers and resumed quotes so that the test setup matches the intended experiment design.
Checkout-level conversion reporting
The module records the active checkout namespace against Magento quote data.
Elgentos provides a reporting example in the module documentation that groups quotes and resulting orders by checkout variant and calculates a conversion percentage.
This gives merchants a starting point for comparing quote-to-order conversion between checkout variants.
The module should not, however, be treated as a complete experimentation analytics platform. Metrics such as checkout duration, payment errors, average order value or detailed behavioural differences require appropriate analytics or reporting instrumentation beyond the module’s example conversion report.
Why this can be useful for Hyvä Checkout decisions
Hyvä’s documentation notes that comparing Hyvä Checkout with an existing Luma-based checkout can help merchants understand how the new checkout performs on their own store.
That is useful because checkout performance is site-specific.
A merchant’s outcome can depend on factors such as:
-
customer and device mix
-
payment methods
-
shipping options
-
address requirements
-
custom checkout fields
-
regional requirements
-
third-party integrations
-
differences between one-page and multi-step flows
An A/B test can therefore provide evidence from the merchant’s own implementation rather than assuming that another store’s results will transfer directly.
Hyvä Checkout pricing and the business case
At the time of publication, Hyvä Checkout was priced at €1,000 as a one-time licence fee for one Magento 2 installation.
The licence covers unlimited domains and store views belonging to the same business entity, along with non-production environments used for development and testing. It also includes the first year of support and updates, with paid support and update options available afterwards.
The licence fee is only one part of the business case.
Merchants should also consider implementation, integration, testing, ongoing support and any custom checkout work required. An A/B test can then help determine whether the measured commercial and operational results justify those costs.
It would be misleading to assume in advance that a particular conversion improvement will repay the investment within a fixed period. The purpose of the experiment is to establish whether an improvement exists for the individual store and whether it is large enough to matter commercially.
How to structure a useful checkout A/B test
The Elgentos module provides the traffic-allocation mechanism, but experimental design still matters.
On Tap recommends the following approach.
Define the primary metric before starting
Choose the main outcome the test is intended to evaluate.
For many merchants, this will be checkout completion or quote-to-order conversion, which aligns with the reporting example provided by Elgentos.
Other measures can still be useful as secondary diagnostics, such as:
-
payment failure rates
-
checkout duration
-
average order value
-
technical errors
-
abandonment at specific stages
If these metrics are not captured by the existing analytics setup, additional instrumentation will be required.
Decide the sample requirement before interpreting the result
There is no universal rule that every checkout experiment needs to run for two or three weeks.
Hyvä’s documentation notes that meaningful A/B testing requires a sufficiently large dataset.
The data requirement depends on factors including baseline conversion rate, traffic volume, the size of the difference the business wants to detect and the traffic allocation between variants.
On Tap recommends agreeing the measurement approach before the test begins rather than stopping the experiment as soon as one variant appears to be ahead.
Make the checkout variants operationally comparable
A checkout experiment is difficult to interpret if one variant is missing functionality used by the other.
Before sending live traffic through the test, confirm the behaviour of relevant:
-
payment methods
-
shipping methods
-
address validation
-
custom checkout fields
-
tax calculations
-
promotional logic
-
customer login and guest checkout
-
region-specific requirements
-
third-party checkout integrations
Where a feature cannot be made equivalent, document the difference and account for it when interpreting the result.
Segment results carefully
Overall conversion should normally remain the primary comparison, but useful secondary analysis can include device type, customer type, market or payment method where there is enough data.
Avoid drawing conclusions from very small segments.
A difference visible only on mobile, for example, can be worth investigating, but it needs enough observations to distinguish a genuine pattern from normal variation.
Test the implementation before testing the business outcome
Before beginning the experiment, On Tap recommends QA testing each checkout variant separately.
The goal is to avoid turning configuration mistakes, payment failures or incomplete integrations into apparent evidence that one checkout performs worse than the other.
What the module does not prove
Installing the module does not prove that Hyvä Checkout will improve conversion.
It also does not mean that a test will automatically produce a statistically reliable answer.
The module provides a mechanism for assigning traffic to different checkout configurations and recording the checkout namespace so that outcomes can be compared.
The quality of the conclusion still depends on:
-
correct implementation
-
sufficient traffic and sample size
-
appropriate measurement
-
comparable checkout functionality
-
disciplined interpretation of the result
That distinction matters because A/B testing is most useful when it reduces uncertainty rather than being used to justify a decision that has already been made.
What if you do not have enough traffic for an A/B test?
Not every Magento store has enough checkout traffic to produce a useful controlled experiment within a practical period.
Hyvä also supports scoping Hyvä Checkout to specific store views, allowing merchants to introduce it gradually while keeping another checkout on other store views.
This serves a different purpose from a controlled A/B test and should not be treated as statistically equivalent. However, it can provide another rollout option for merchants that need to limit exposure or do not have enough traffic for a meaningful split test.
The broader lesson
For Magento merchants evaluating Hyvä Checkout, the Elgentos module offers a practical way to compare checkout configurations using their own customer traffic.
It does not remove the implementation work involved in a checkout migration, and it does not guarantee that Hyvä Checkout will outperform an existing checkout.
What it can do is replace part of the migration decision with measurable evidence.
Instead of asking whether Hyvä Checkout performs better in general, merchants can test a more useful question: how does it perform for this store, with this customer base and this checkout configuration?
About On Tap
On Tap is a growth-focused eCommerce consultancy and Hyvä partner, specialising in Magento and Adobe Commerce implementations for mid-market and enterprise merchants.
From checkout A/B test design and Hyvä migration planning to conversion rate optimisation and peak-season readiness, On Tap helps merchants make evidence-based decisions about commercially sensitive parts of their eCommerce experience.
If you are considering a checkout change and want help designing or implementing an A/B test, get in touch with the On Tap team.
