A/B testing with Custom Dimensions
So, you’ve setup your custom dimensions and now you want to see whether picking the wizard or the knight affects player behavior?
Below we’ve outlined where to look, and what useful insights you can find.
1. See who’s winning in the Progression dashboard
How do the two dimensions compare when we look at which users finish more levels? Let’s look at the data in the Progression dashboard. At first glance, they seem to be very well balanced – the win rate is almost symmetrical.
But other metrics tell a different story. Although players win at a similar rate, scores seem to be higher for the blue test group – especially for lost levels. This might prevent users in the green group from reaching leaderboards and, in time, can potentially decrease their engagement. Let’s check if this hypothesis is true!
2. Check out how engagement is impacted
Looking at DAUs, the numbers are slightly higher for the blue group, but the difference is very small. Things start to get interesting when we look at weekly users – the two groups clearly differentiate. And the difference continues to grow at a monthly level. So we’ve clearly found something we can improve in the game.
In the Engagement dashboard, you will notice that not all metrics can be analyzed. This is the case for Retention and for New vs. Returning DAUs. The explanation is simple: a custom dimension is an attribute of an event, not of a user, so they cannot be used to compare all users and events, but only the ones they apply to. A session start is a user attribute which does not have custom dimensions appended, therefore custom dimensions cannot be applied to new users or returns / retention.
If you’re thinking this sounds a little confusing… we agree! So we’re already working on changing this data structure, and in a little while you should be able to filter any type of data by a custom dimension. If this functionality is important to you, let our friendly support team know. They can provide an updated timeline for the release, as well as an alternative means of configuration.
Rest assured, even though you can’t currently filter by custom dimensions, you do have the possibility to compare event types and see which fares better for new users or retention. The handy Explore tool can generate reports for any event type. Here is an example:
3. Understand the effect on your bottom line
Wins, sessions, they are relevant to the user’s experience, but at the end of the day do differences this small really impact your growth? You can learn the answer in the Monetization dashboard, where you can filter all metrics by custom dimensions, regardless of whether they are set for progression, business, or economy events.
You can count converting players to see how many there are in each group, and also get revenue averages and transactions for each one. In our case, the blue group which had higher engagement also includes more paying users, and it ranks higher for average revenue per user as well as total earnings.
Does this really matter?
If you look at only one metric, it can seem like the differences are tiny. But dig in further and you may discover that the difference between breaking even and making a profit is that small gap – between the players with 65% win rate and those with 67%.
Custom dimensions provide you the data you need to A/B test the impact of different items, characters, or stages of your game, so you can get in-depth insight on player behaviour, and how it impacts your business.