Scribd: User Trust Initiative
A three phase initiative to reduce early trial cancellations for new members of Scribd
Overview
-
Time to launch
(For each phase)
• 1 week design
• 1 week development -
Team structure
• 1 Designer- Me!
• 1 Product Manager
• 1 Technical Project Manager
• 3 Developers
• 1 QA Engineers
• 1 Data Analyst -
Testing methods
• UserTesting
• Hotjar
• In product n-way test
Challenge
A large portion (~95%) of Scribd’s new trial users enter Scribd through a Google search for a specific document/PDF. These users are then pushed into a trial sign-up funnel in order to gain access to or to download their document. Many of these users immediately cancel their free trial after downloading.
How might we delay these early trial cancellations in order to give the user more time to explore the product?
Initial Explorations
I worked with my product manager and data analyst to build a funnel of key points in the day zero experience. We identified places where every user would land during the first day of their trial and that would have high visibility to the user. We decided on the following locations:
Welcome Screen
Logged-In Homepage
Document Page
Account Settings
Phase 1: Welcome Screen
Control Experience
This was the first screen a user would see post-credit-card, and the final sign up screen before arriving back on their document. The purpose is to quickly mention some of our most popular content types and then let them into the product
Test experience
For the test experience I swapped out the “and that’s just the beginning” line of copy for the reminder messaging. Having the reminder text located above the final CTA before entering the product elevated visibility. Furthermore, by styling it to look like a selected checkbox, It further commanded a users attention. This proved to be a happy surprise to most users– in interviews every user was drawn to the messaging above the CTA and felt rewarded for looking at it and seeing it was “good news”
Phase 1 Results
Test period: 5/1/2020 - 5/29/2020
Total assignments in test: 312K (evenly split across two variants)
For all users, we observed statistically significant lifts on retention metrics:
% Viewed Cancellation Flow within 14 days
-6.1%
% Cancelled Within 14 Days
-5.3%
Bill-through Rate
+3.0%
Phase 2: Reminder Banner
Banner on home
Putting the banner on home does a couple things for our goal. The first is that it puts a lot if high-value content in front of the user to showcase the product offering. The second thing this does is lower the urgency to cancel by showing the user just how much time they have remaining.
Feedback on this composite was mostly positive, but there were several internal notes that when the small banner included illustrations it felt too busy.
One other note here is “# of days remaining” was flagged by dev for complexity and we pulled that out of scope for MVP
Banner on document page
I also tested the banner on the one page where a user wanted to be- back on their document page. However, this proved problematic because once users land on the document page they generally immediately click download and then head for the door. So this felt like it was a little bit too far down-funnel to have meaningful impact for our users.
Phase 2: Final Composites
Reminder banner (day zero)
Since the day counter way pulled out of scope, I introduced one additional banner at this point that would change messaging after day zero. The goal being that it would provide a slightly more intelligent or living product vibe when the messaging adjusted to reflect time passing to the user.
Reminder banner (day 1-7)
In the above composite you can see the messaging has changed from “Welcome…” to “You’re currently…”
Note: This banner only shows for the user until day 7, as the reminder is sent out that day
Phase 2 Results
Test period: 5/29/20 - 6/5/20 and 6/24/20 - 6/30/20
(The test was paused to accommodate for a separate homepage test)
Total assignments in test: 70K, evenly split between two variants
For all users, we observed statistically significant reductions in cancellation metrics
% Viewed Cancellation Flow within 1 day
-2.8%
% Cancelled Within 1 day
-3.3%
These changes were primarily driven by our premium content users, who also saw statistically significant reductions in cancellation metrics
% Viewed Cancellation Flow within 1 day
-6.5%
% Cancelled Within 1 day
-8.2%
Phase 3: Account Settings
Banner in account settings
The account settings page is the last chance to get the reminder messaging through to a user before they enter the cancellation flow. I started out by repurposing an existing banner space in account settings (one surfacing premium content) to elevate the reminder language.
In user tests, however, we prompted users to land on account settings, review it, and proceed to cancel. Most users didn’t recall the reminder language here in any meaningful way
Interstitial page
The next exploration I tested was injecting a new page in between “end my membership” and the actual cancellation flow
This tested poorly in that the user had already committed to entering the cancellation flow by having left the account settings page
Confirmation modal
This modal served a few strategic purposes. The first thing this accomplishes for the user was meeting them where they’re at. I wanted to acknowledge we know they want to cancel and that we are going to do that, but then I wanted again to remind them how much time they have left. Now, Scribd does not instantly cancel a users trial membership when they cancel, but I felt I could make it feel slightly like instant cancellation to hopefully get the to stick around and check out some more content
Delaying a user completing a desired action (cancellation) does feel a little strange. However, convincing them to stick around during a free trial period feels like helping the user slow down and consider the product more fully to get the most out of their time
Phase 3: Final Composite
Final modal design
The success of the previous two tests allowed us bring in-scope the development time for both time remaining as well as a reminder email date. It felt like a great escalation where before we had been mentioning simply “we’ll remind you” to now saying “we’ll remind you before this specific date” which felt like further trust building.
Phase 3 Results
Test period: 8/27/2020 - 9/5/2020
Total assignments in test: 94K, evenly split across two variants
We noticed statistically significant drops in cancellation metrics:
% Viewed Cancellation Flow Within 7 days
-3.4%
% Cancelled Within 7 days
-9.6%
All rolled up
This sequence of projects was massively impactful and led to far more focus on user trust company-wide. It proved that we could be transparent and trust in our value to users, without being afraid of reminding users about upcoming billing cycles.
Internally these projects also forged a much closer bond between Product Management and Design.
Change to Bill-through Rate: