While studying at the College of Innovation and Design, I got a chance to learn many new concepts as well as the unique opportunity to conduct an extensive user study for our main product. A concept we covered and that piqued my interest was "evidencing."
Our app allows users to watch videos on the TV via various methods, but being a virtual service, it does not have the opportunity to provide evidence of service to the user in traditional ways. For our first evidencing experiment we decided to let the users know how many videos they have played with a simple non-intrusive message.
For the AB test we changed the welcome message for users that have played more than 5 videos. It isn’t possible to know if a user truly rated the app or how they rated it but we can ask the users if they are happy with the app. With this in mind, the AB test is only activated after more than 5 videos so anyone rating before that wouldn’t count.
Something we discovered was that a user will say they are happy with the app is 4.3% higher when they are shown a count of the number of videos they have watched. 4.3% may not seem like a high number, but when expanded to all users of the app that kind of percentage could have a significant impact on ratings for such a simple change to the app!
An unexpected result has been an increase in purchases. Users shown the message are
9.2% more likely to purchase the app. During one of our clases we were allowed to perform a user study. Many suggestions
came out of that user study and we have been able to implement some of those. Here
are AB tests of a couple of those.
One big suggestion was to have a tutorial video. We’ve been testing showing a tutorial
video to users and so far it is leading to 13% higher sales.
The other suggestion we had was to show a close button near ads so users will be aware that they can remove ads by paying. We also did an AB test for this. The results were pretty great, 29% higher sales for users shown this close button followed by an explanation when they press it.
