Analyze Marketing Experiment Results | Measurement Fundamentals | App Marketing | Udacity
- Articles, Blog

Analyze Marketing Experiment Results | Measurement Fundamentals | App Marketing | Udacity


After you start seeing changes
in the metrics, keep testing. See if you can further perfect
your offering, landing page or messaging or
whatever it is that you’re testing. I don’t want you to get discouraged
of your hypothesis was wrong. That’s actually great. You learned something new. You now know that’s not what you need
to do to improve your business and you can narrow down other options and
keep working through your list of ideas. So let’s hear from Talley, who was proven wrong with
a hypothesis she once made.>>So it’s interesting,
we ran some experiments lately and I’m going to say something
really controversial and all the design people
are going to be outraged, but what I found out is that the language
and the text that you use on your landing page have tremendously
more effect than the imagery. So it’s interesting, right? We were trying, what you really
want to get right is the message, is what you’re saying to
people that the product is and every single product can be represented
probably in 30 different ways. So figure those out and try to find the
one sentence that resonates with your user, that they read the sentence and
they say, yes, this is it, I want to sign up for that. And we experimented a lot
with this language and we improved our conversion rate on
the home page by hundreds and hundreds of percentages and then we did the same
with the imagery and we thought, oh, now we’ll do the imagery and it will be
another 100% increase, but not so much. Once we found the right
language people just knew, oh, they read it, they knew they wanted it,
and they signed up.>>When you’re setting up an A/B test,
one of the things you want to have in mind is what’s the effect
you want to say. Usually we will saw with
the metric we want to move. So for instance, if we have a new
product like voice messaging and we want more people to put use it,
we can assign a test group and a control group where we advertise
more heavily to the test group, and see how the product is used
differently by the testing control. The first step to diagnosing any
problem that you see in your data is cutting it by different dimensions. So a dimension can be
pretty much anything. It can be country, if you have
an app that’s out internationally, it can be by platform if you have out
on Android and iOS and by anything else that you know about your users that you
can use to try and isolate the issue. If for example, your app is aBarkment
and you suddenly see that the number of postings that are going up onto
your site have dropped really heavily, you could then cut that by the different
interfaces that people can post from. So you can look at the number of
posts which are coming from iOS, from Android, and
from your desktop site. If you see that one of those
is the one that’s dropping, you can then look in the code and
see if you have a problem with your. An important thing to look out for when
doing an A/B test is to make sure that you have enough people going into
either side of your experiment, that you can be confident that the
result that you’re seeing is real, and not just a result of random chance. If you Google for
A/B test significance calculator, you can actually get free on line tools
where you can put in the number of people going through your experiment and
the number of people successfully completing whatever objective it is you
have in mind and it can tell you what’s called a confidence interval and in
the industry normally, we look at a p95. And that means for any result that
you see, you can be 95% sure that the experiment is really giving the
result and it’s not just random chance. And also when you found out which one
of your experience is the better one, you can actually roll out that
new user experience to everyone. You can do it without having to go
through a new App Store submission, either through the Play Store or
the App Store.

About Ralph Robinson

Read All Posts By Ralph Robinson

Leave a Reply

Your email address will not be published. Required fields are marked *