Understanding popup measurement beyond sign-up rates

Popups have a central role in today’s ecommerce experience. Online merchants use them to welcome users, offer discounts on first orders, and encourage sign-ups for email or SMS distribution lists. Popups have become the most ubiquitous tool in ecommerce. When used strategically, they can increase sign-up rates to identify more users for personalization and increase conversion rates among new customers leading to higher average order value.

The most common evaluation of popup success is sign-up rate, how many people gave you their email address in return for a discount or other incentive. If you are in the business of generating sign-ups instead of revenue generation, feel free to stop reading now. This short-term perspective ignores the significant impact popups can have on customer behavior as well as overall revenue and 6-12 month lifetime value. In this blog post, we’ll showcase three different ways to evaluate popups, explore the pros and cons of each, and state the case for why tracking not one, not two, but all three metrics are worth tracking.

Three Key Metrics to Evaluate Popups

Signup Rate

Let’s be perfectly clear: signup rate, the percentage of users who interact with a popup and provide their email or phone number, is a simple and intuitive way to gauge whether your popup is effective at growing your audience. Signup rates are useful for brands just starting out or those focused on building a larger email or SMS list to sell to at a later date. There’s a catch: signup rates only show one side of the story. They don’t account for how much revenue those new signups generate or how much offering a discount might cut into your profit margins. If your goal is to purely grow your list, this metric is sufficient. But if you’re thinking about a holistic view of your customer journey, including profitability and customer lifetime value, it’s not enough.

Revenue from Signed-Up Users

The next logical way to evaluate popups is by looking at revenue from users who signed up. This approach digs deeper, tracking how much customers actually spent after subscribing. It’s a great way to see how effective your popup is at driving purchases and whether the discount and creativity are paying off in the short term.

While this metric is helpful for understanding direct revenue impact, it still doesn’t paint the full picture. By focusing only on users who successfully interact with the popup, you risk overestimating its effectiveness and overlooking potential cannibalization where high-intent customers who would have purchased anyway use the discount unnecessarily.

Revenue from All Users (Including Non-Signups)

The third and most comprehensive metric is revenue from all users who see the popup, whether they interact with it or not. This takes into account the overall impact of the popup on your site’s revenue, offering the clearest picture of its incremental value. Since it includes everyone, it eliminates the risk of cannibalization and ensures that any revenue lift is not just a reshuffling of high-intent buyers.

This is the best metric for understanding the true incrementality of your popups and how they contribute to your brand’s overall revenue. By evaluating the impact across all users, whether they engage with the popup or not, you get a clear and comprehensive view of how your popup directly impacts your bottom line. It’s ideal for brands aiming to measure the broader effects of their popups while ensuring that any uplift directly translates to incremental revenue and profitability.

Real world example using Monocle

Here’s an example data set based on a Monocle customer running a popup on their website. We’ve removed the branding, but their strategy focuses on growing revenue from their welcome popup. Keep in mind: This is data from an a/b test where 50% of popups offered 20% off and the other 50% offered $20 off. Evaluating the success of the popup helps us measure incremental value that can be driven when comparing the offers.

On the surface, offering $20 off compared to 20% off converts significantly better. We could stop the analysis there and move forward with offering $20 off and likely see an increase in email and phone submissions. But when we dig deeper, a different story is revealed.

Let’s start with the middle column. It shows how much revenue was generated from customers who submitted the popup. The numbers confirm what you’d expect based on the initial conversion rate, more revenue driven from the $20 off offer.

The right-most column shows revenue generated from customers who did not submit the popup. It turns out there is significantly more revenue driven from the 20% off group, a 4x difference for those who did not submit the popup. This is where we see the importance of evaluating popups from all three perspectives.

Finally, the left-most column aggregates total revenue from both submitted and not submitted customers. It turns out that there was actually more total revenue driven from the 20% off offer.

This test suggests there is some sort of cannibalization of revenue happening when customers submit the popup. A subset of customers were likely already going to purchase before submitting the popup, but when they saw they could get $20 off, they just submitted the form and took advantage of the free discount.

Without this key insight, this brand probably would have moved forward offering $20 off for all submissions. In the long run, this is a losing strategy, one that will generate less revenue.

Concluding Thoughts

Popups should be a cornerstone of a well-rounded ecommerce strategy, far more than just tools for capturing emails or phone numbers. As with many ecommerce metrics, truly understanding impact is about evaluating success from vantage points. That’s why tracking all three metrics - signup rates, revenue from signed-up users, and total revenue from all users - is critical. Each metric measures a different aspect of your popup’s performance and provides valuable insights that only paint a full picture when reviewed together.

If your brand is ready to unlock the full potential of your popups and gain a deeper understanding of their effectiveness, get in touch with Monocle. We can optimize for both signup conversion rate and all revenue metrics in order to determine which customers should see which offers.