If you visit an eCommerce site, odds are good that you’ll be hit with an email opt-in popup almost immediately. Most popups will offer you a discount between 10-25% off your first purchase…if you opt into email.
This has been going on for so long that many brands do it reflexively. “If we don’t do it, we’ll get beat out by the competition” is the thought.
But is that true? No. Does that mean you should stop running your welcome offer? Also no.
But first, I want to thank this week's sponsor: Inventory Planner by Sage. If you're still doing planning in a spreadsheet, you're leaving money on the table. Inventory Planner's software is informed by decades of retail expertise to help you maximize the ROI of your product assortment.
A first order discount of even 10% is a tough pill to swallow for fashion brands who are trying to be first order profitable. The average return rate for a fashion brand that sells footwear or tailored items is 15-30%. In that context, you don’t want to offer folks a 10% discount unless it's really doing something.
So that’s exactly what I’m going to share here: a step-by-step testing framework that will help you understand if your welcome offer is worthwhile.
You’ll need Shopify, Klaviyo and Lifetimely to run this process yourself. If you don’t use those tools the testing framework is still valid, but you’ll have to figure out how to build the test yourself.
Setting The Right KPIs
List size is a vanity metric. List size tells you nothing about the quality of your subscribers. A list of a million emails is worthless if none of those subscribers ever purchase from you.
For this reason, opt-in rate is the wrong “north star” KPI for your email popup. If folks opt in and never purchase, you won’t make money. You can use “hacky” tactics to get more folks to opt in to your popup. But if those folks don’t convert you’ve accomplished nothing.
You might disagree with me. Your email service provider will probably disagree with me, because they bill you based on the size of your list.
If you’re invested in the idea that list size is what matters most, you can stop reading here I guess. But if you’re with me, I’m going to show you how to run a popup test that tracks subscriber opt-in rate and LTV, because that’s what really matters.
Test Overview
Here is a high level summary of how we’re going to structure this test:
- A/B test two email popup designs on Klaviyo where the only variable is opt-in offer vs no opt-in offer.
- Set a variable on the subscriber profile based on the popup they used.
- Use the variable to create a Klaviyo segment, and then use Lifetimely to track the behavior of that segment over time.
When you run a standard A/B popup test on Klaviyo, the platform uses popup opt-in rate to determine the winner. But that doesn’t tell us if those subscribers converted. That’s why we’re bringing Lifetimely into the mix.
This tutorial doesn’t require any coding, but it does assume that you have enough familiarity with Klaviyo to follow their documentation. If this is too technical, share it with your email agency/freelancer.
You should run this test for a minimum of 30 days. At the end of 30 days, you’ll check to see if your results are statistically significant (see the last section for more details).
If you hit stat sig, you can pause the test and implement your winning variant. If you don’t hit stat sig, keep the test running for another 30 days and check again. I wouldn’t keep it running for more than 120 days.
***
If you want your business to grow, you need to stop asking $5 questions and start asking $500k questions.
I'll be honest–the email popup question is closer to $5 than $500k. I'm covering because folks keep asking about it.
But inventory planning and forecasting is more than a $500k question. If you run out of best sellers, your ad performance tanks. If you overbuy the wrong styles, you can tie up cash for years.
Stop wasting time hacking together inventory forecasts in Excel. Inventory Planner by Sage was built by fashion professionals. It has the features you need to minimize fashion risk and maximize return on inventory.
Plus, it includes personalized guidance informed by 8 powerful strategies that have been battle tested by some of the world's biggest retailers.
Click here to book a demo and learn more about how Inventory Planner can improve the ROI on your product assortment.
***
Klaviyo Setup
These instructions assume that you’re currently running a Welcome offer. If you’re not, the TEST version of your assets will include the offer and your CONTROL version will reflect your current non-offer.
Step 1: Create a custom profile property called optin_method. This is where you’ll store the data about which form a subscriber used to opt in.
Step 2: Create a copy of your current “business as usual” opt-in popup. Change the name to include the word “control”. Add a hidden field to the popup that sets optin_method to “popup-offer-test-CONTROL”.
Step 3: Duplicate the form you created in step 2. In the form name, replace the word “control” with “test”. In the hidden field, change “popup-offer-test-CONTROL” to “popup-offer-test-TEST”.
Step 4: Change the copy in the TEST version of the popup to omit the offer entirely. This is the ONLY change you should make to either version of the popup.
Step 5: Duplicate your Welcome series. Your CONTROL popup will trigger your BAU welcome series. Your TEST popup will trigger the copy you just created. Edit the contents of the TEST welcome series to omit the offer.
Don’t make any other changes to the design or timing of the Welcome series. Just remove any mention of the Welcome offer. If emails in the series are 100% dedicated to the offer, rework them to showcase your hero product or category.
Step 6: Create a segment for anyone who is opted in AND has an optin_method that matches popup-offer-test-CONTROL. Create a second segment for anyone who is opted in AND optin_method that matches popup-offer-test-TEST.
These segments will have zero members until the A/B test starts running.
Step 7: Double check that your popups are triggering the appropriate welcome flows. Then set up an A/B test with both versions of the popup.
Don’t check the box to “run until the test reaches stat sig”, because that will be based on opt-in rate, not subscriber conversion rate.
Lifetimely Setup
You’ll want to wait until the A/B test has been running for a few days before you set this up. Before you run through the steps below, confirm that your Klaviyo segments are populating.
If they’re not, you need to troubleshoot your technical setup before moving forward with the test OR confirm that folks are opting in to your popups.
Step 0: Download Lifetimely if you haven’t already. You’ll have to pay based on your monthly order volume.
Hint: if this expense feels too steep for your brand, you need to focus on growth and unit economics, not testing your Welcome offer.
Step 1: Navigate to your cohort report. Change the view to a monthly view of lifetime net sales data. You want total net sales, not average net sales per customer. This is the monthly cohort behavior for everyone in your business.
Step 2: Put a filter on the report so that you’re viewing the CONTROL segment you created in Klaviyo. Download the .csv data.
Step 3: Put a filter on the report so that you’re viewing the TEST segment you created in Klaviyo. Download the .csv data.
Compare the cohort data between the two reports. Did TEST or CONTROL have more conversions? Did TEST or CONTROL generate more total net sales?
Step 4: Go back to your A/B test in Klaviyo and make note of how many subscribers opted in via each popup per month. Add a column to each .csv file with the number of total subscribers for each month.
Analyzing The Results
Wait 30 days before you download the Lifetimely data and perform any analysis.
If you start the test on October 1st and pull your first round of data on October 30th, all the subscribers and customers will be part of the October cohort. If you start mid-month and pull data in the next month, your results will span two monthly cohorts.
The longer you run the test, the more monthly cohorts you’ll have to analyze.
The main question you’re trying to answer is this: does the Welcome code drive a lift in sales that outweighs the hit in margin you’re taking on the promo?
There are three ways this can happen:
- The Welcome code drives more subscribers to convert.
- The subscribers who convert off the Welcome code spend more than those who don’t get an opt-in incentive.
- Both of the above.
Here are some example numbers to help make this more clear:
Control Group: 15% Off Welcome Offer
- Month 1 subscribers: 5,000
- Month 1 subscribers converted: 200
- Month 1 % subs converted: 4%
- Month 1 converted net sales: $30,000
- Month 1 net sales/subscriber: $6
- Month 1 net sales/converter: $150
Test Group: No Offer
- Month 1 subscribers: 4,500
- Month 1 subscribers converted: 225
- Month 1 % subs converted: 5%
- Month 1 converted net sales: $30,375
- Month 1 net sales/subscriber: $6.75
- Month 1 net sales/converter: $135
What this data means: The Welcome Offer popup drove 500 more email signups than the No Offer popup. If you were running this test in Klaviyo, the Welcome Offer would have been labeled the winner.
BUT the No Offer subscribers converted at a higher rate. Although the No Offer buyers had a lower AOV than the Welcome Offer buyers, the net sales per subscriber was slightly higher.
The Welcome Offer audience and the No Offer audience essentially generated the same amount of net sales. The Welcome Offer audience has a higher AOV, but it doesn’t offset the margin hit you’re taking from offering the 15% discount.
When I run these results through a statistical significance calculator, they’re stat sig. If this was my brand I would turn off the discount, and work on making my Welcome Series convert at a higher rate without it.
But if I had a lot of traffic and felt strongly about offering a discount, I would try:
→ Keep the test running and analyze the 90 day behavior of the first monthly cohort.
→ Review my Welcome series and see if there were tactical improvements that could increase the conversion rate. Rerun the test with the optimized Welcome series firing for both test and control groups.
→ Run the test again with a 10% discount and (if the brand was comfortable with it) a 25% discount.
→ Try alternative offers (ex. $25 off your first purchase, free shipping).
Here’s the thing: it takes a lot of time to run these tests the right way. If you’re a smaller brand and the Welcome offer is eating into your margin, I would just abandon it or figure out a compelling but less costly alternative.
Most of your email revenue is going to come from repeat purchases. So your checkout flow is your number one source of list growth. All roads lead back to new customer acquisition, so focus your time and attention on that.
It’s hard to grow email revenue from prospects without scaling up your site traffic (acquisition again!) or leaning into direct response marketing tactics that aren’t a good fit for many fashion brands.
Final takeaway: don’t feel pressure to use a tactic because “everyone else is doing it”.