Introduction

Providing decision support on product offerings in an online marketplaces environment requires digging deep on the granularity of the key business drivers to evaluate success. Using a method called “A/B testing” allows an FP&A leader to easily build a framework on key criteria to better understand the various behavioural impacts that consumers apply in different scenarios and to analyse their performance.  In this note, a holistic decision framework is applied to an online marketplace using A/B testing to evaluate whether to develop a Buy-Now-Pay-later (BNPL) offering in house or to work with a vendor to supply it.

What is BNPL?

BNPL is a mechanism for consumers to purchase a good or service without fully paying for it upfront.  Another term for this plan is "instalment payments", which are extremely popular in countries like Brasil where the average consumer does not have the same level of disposable income as countries such as the United States or the United Kingdom.  Think of it as making a purchase with your credit card, and paying it off later, but there is no interest applied. One example of a BNPL offering is that you want to buy a $10 avocado slicer but pay only $5 now, and then commit to paying the other $5 at some later date, either in one shot 30 days from now, or paying $1 each for the next 5 months, etc… This is a great option for the consumer as it provides flexibility but there is the risk to the merchant in terms of understanding the creditworthiness of the consumer and potential for default.
 
At Wish, we wanted to evaluate the benefits of offering BNPL to our consumers. However, we wanted to determine if the BNPL offering should be managed in-house or if it should be managed by a vendor on our behalf. The criteria for BNPL loans that we want to provide for the consumer is to pay for 50% of the product immediately, and then pay the remaining 50% of the product in 30 days.

A/B Testing

The beauty of an online marketplace is that there are tools available to segment consumers to what they see on the site into multiple groups, which is what we call "A/B testing": where one group of consumers sees a form of an offering and another group sees a slightly different form, all the while ensuring that there is enough randomisation between the two groups to eliminate selection bias between the two sets of users. The test is then run over a pre-determined period such that the samples from the test become statistically significant and key metrics are compared to better understand business performance.

Analysing the product offering

From a holistic perspective in a B2C environment, it is important not only to understand the financial impacts of a business initiative but also the consumer's behavioural impacts from the initiative.  Specifically, in comparing the in-house (A) to the vendor-based (B) offering for BNPL, we want to answer the following questions:

• How much do consumers buy with the BNPL offering in (A) vs. (B) groups?

• i.e. what is the loan penetration (% of people purchasing with BNPL), average item value, basket size (number of items per transaction), purchase frequency (transactions purchased in a period), etc…

• And what is the performance of these offerings in different countries?

• i.e., Brazilian consumers have instalment payments (a form of BNPL) ingrained in their culture, but not the Swiss – so would they even want the offering?

• What is the cancellation and refunds performance like?

• i.e., do users cancel more or ask for more refunds in (A) vs. (B)?  Specifically, are users more committed in one offering vs. another?

• Since the vendor will charge a risk premium to manage default rates in the form of fees, will the premium in (B) offset the default rates assumed in (A)?  

• Are there multiple fees in (A) vs. (B)?  How do these costs impact the gross margin?

These questions not only help flush out the key business drivers that impact user behaviour but can also be translated into financial results.

Evaluating the results

A framework to answer these questions is to first develop a P&L for each of the A and B groups, and then to create subsegments for those that took out BNPL loans vs. those that did not in each of the A and B groups. Essentially, you’re creating four different P&Ls for each of these subgroups which clearly identify the drivers at the right level of granularity. This helps identify not only the financial performance but also the behavioural drivers which might impact the product design in each of the A and B groups.  For example, did the loan penetration rate identify more friction to sign up for a loan in either A or B contributing to volume differences in one group vs. another?  And were the subsegments of the non-loan purchasers consistent in each of the A and B groups?

This method of P&L evaluation down to EBITDA will help determine if the short-term P&L impacts are more beneficial in each of the A and B buckets.  To also take a long-term view, you would want to look at consumer retention over time in one bucket vs. another to determine if the repeat rate for the customers would increase in the A vs. B bucket, which could then be incorporated into the overall analysis.

And finally, there are operational considerations and costs to the program that may not be immediately obvious on the P&L: Is this a core competency to invest in-house where operational processes for collection need to be maintained and managed? What is the overhead to manage the regulatory landscape for providing such an offering in the countries that the company does business in?  These are non-trivial questions that need to be addressed as part of the overall BNPL solution as well.

Summarising the process

An "A/B test" framework is a great method to evaluate whether a product offering should be made available in an online marketplaces environment.  Key criteria for the FP&A professional should always be to segment to the most appropriate level of granularity in subgroups for the A and B buckets on both operational and financial metrics in the evaluation. However, the FP&A professional should also not forget the operational and risk considerations to derive at a recommendation.