Designing Online Marketplaces
- Articles, Blog

Designing Online Marketplaces

So we have a session this afternoon. I don’t have my piece of paper
actually on what the exact title is. On networking disciplines. And, first of all, I’d like to thank
my colleague Katie Ross in economics, who actually, organized this session. And, from my view point as an economist, this is remarkably a multidisciplinary,
session. And, that word’s used a lot. The chancellor was using it this morning. And particularly for people in economics,
they’re very, nervous about that. All right?
And, you know even publishing in PNAS, i mean, what does that count for,
you know? That’s not really
an economics journal right. [LAUGH] And, at the same time some
economists are reaching out, you know. Berkart Skipper’s been involved
in neuroscience, this morning’s, taught by Brenda McCowan, one of her
co authors of one of the papers is Oscar Jorder who is a macro, monetary
economist in the Economics Department. But the main reason I’m bringing this up
is that we have three speakers today. Two of them really, multidisciplinary in,
cross disciplinary. So one, Susan Athey is an economist
who has worked a lot with, people in other disciplines. And the other, Raisa D’souza, who I’ll
introduce later, is also really working, across the disciplines, and, you know, and this is one of the hopes, is, and actually
one of the interesting issues, you know. We’re looking at big data,
we’re looking at networks. People outside the social sciences
really developed those skills. To what extent will we,
just bring them in house and do it all on their own and
to what extent will we reach out and, and actually work with people in other
disciplines who are using these methods. We’ll see how it goes. Okay, so first of all i would like
to introduce, our first speaker, Susan Athey, who’s the Economics
of Technology Professor at the Stanford Graduate School
of Business. She has a PhD in economics many research
interests including internet, options, econometrics, and machine learning. She has reached out, she for a while
was Chief Economist at Microsoft and for quite a while has been
a consultant at Microsoft, and that’s really, influenced her research and interaction with others, and also,
influenced, the people she’s working with. She’s won many awards including member
of the National Academy of Sciences and American Academy of Arts and
Sciences, but in addition, economics is really,
really tough on giving out awards. You know, i don’t really understand why,
i don’t see how it helps the discipline. But there is one award given by
the American Economic Association. The John Bastes, John Bates Clark award is
given not every year but every other year to that American economist under the age
of 40, who is adjudged to have made the most significant contribution to
economic thought and knowledge and Susan Athey has won that award, so
we’re very delighted to have her speaking. Thank you [APPLAUSE]
>>Thanks so much and thanks for putting together such a wonderful conference, i was,
talking with, Matt Jackson earlier and said you know, one thing, one rule
i think I’ve learned from today is, you never wanna go on stage after
someone who has 750,000 YouTube hits. [LAUGH] So, you know,
we’ll all just have to do our best. That was really fabulous. I really enjoyed your presentation. I’m, so, I’m gonna talk to today
about a different type of network, a market network where
the interactions of the individuals are not as much social interactions but
actually competitive interactions. And, so this, in some sense,
this observation that you can think about markets as networks is an old one and
there are, you know, a variety of economic studies over
the years that have taken that, that tack, but i would say that the sort
of micro economics of the, especially that part that’s been sort of
steadying some of the modern platform markets, like eBay and so on,
haven’t really, taken that approach and so today I’m gonna be talking about bringing
together some tools from economics for studying competitive behavior with,
with networks. So i wanna, i wanna start out actually
by talking about, A/B tests and experimentation in general, and that’s
gonna segue into the, the network idea. So one of the things that,
if, for those of us who have been sort of collaborating with, with tech
firms and, and doing experiments we’ve, we’ve observed that, A/B testing,
that is randomized controlled trials,so that if you’re a tech firm you call
a randomized controlled tile an A/B test, has become incredibly important for
business and business practice. So, something,
somebody like Google will run 10,000 or more randomized controlled
trials every year, nothing changes on a website like
Amazon or Google without an A/B test. Okay?
So this is actually a, a really important phenomenon for
our society. It also is affecting, governments and development organizations who
are trying to also, test the effects of all sorts of policy interventions,
and, and really, technology and, and digitization has made it possible
to do that on a much larger scale. We also see that in,in newspapers and everybody else who’s
providing our information. So, one of the, the observations, though,
about how these, these tests are run and, and what they do is that they actually,
really miss out on a lot. So, a typical A/B test at Google would, would test sort of two different
versions of a search algorithm. And the way they would do
the randomization, is they’ll take, say, one percent of users or half a percent
of users as they come on to Google and they’ll allocate them
into a treatment group. They’re actually going to be running
hundreds of experiments at the same time. It’s a little more complicated than that,
but so, you’re much more likely to be in an
experiment than you are a control group. But you’re gonna get
allocated on a user basis. This is something that you
can learn very very quickly. Should i use a different color blue? Should i change my fonts? Should i re-rank out my
results in a different way? And i can very very quickly
learn about that in a day or so. I’m gonna have a result, I’ll have
a significant a significant result, and i can decide to, to roll out and
release these new features. It’s actually really important for
innovation, because say, at a company life Facebook, pretty much anybody can,
can change the code of Facebook. An intern can change the code that
produces what you see on Facebook. And so how do you, how do you allow that
kind of decentralized rapid innovation? Where you have a very objective
way to evaluate that impact. And that is to put it through an A/B test. And if it looks good,
then you ship that, you ship that code, you release it to the wild immediately. So it’s really kind of central to the way
innovation happens in these large firms. But if you’re going to be testing users,
it’s very cheap, very easy, you’re not really going to
have a good idea of how those tests affect the businesses that
are operating on your platform. So if you’re eBay,
you have two sides of the market, you have users and you have,
and you have sellers. If you’re a search engine,
you have users and you have advertisers. If you’re Amazon you’ll have,
you know, the users and you’ll have the third party sellers and
so on. Facebook would have users and advertisers. So it’s very easy to test what users like. But trying to understand,
advertiser responses or seller responses is a much
more difficult problem.>>And it’s these agents
that i wanna focus on today. So, interestingly even though it’s,
it’s, seems, should be like sorta obvious to everyone
that randomizing at the user level- Is not a good way to learn about
advertiser seller response. That is the standard practice
in every major tech firm. So what you would typically
do is you might say, okay, I’m going to do something that’s
going to re-rank my sellers. I’m going to make lower priced
sellers appear higher on eBay, something like that. I’m going to take particular types of
advertisers and rank them higher in, in, beside your search results,
and I’m gonna experiment on users. So what’s gonna happen
is each advertiser or each seller is only gonna see about 1% of
their traffic affected by the experiment, which is like background noise to them. So the, from the seller perspective,
this experiment really didn’t happen. So what we’re gonna learn
is how the users respond. So we can see if i put you know,
low price sellers higher on eBay, i can see how much more the users click,
how much more the users buy. So i can learn about the user effects. But I’ve no idea what happens to the,
to the, to the sellers, but it’s very common just to look at
the metrics and say, oh gee i got this many more sales, without actually
taking into account that the sellers might later change their prices or changed
their behavior or you can quit eBay. Okay so, with advertisers like what
might happen, is the first of all, the advertisers are going
to notice the effect. Second of all, the advertisers will influence each other
even if they don’t change their behavior. So if i take one advertiser and put it up
high on the page they’re gonna get lots of clicks and they’re gonna, they’re ha,
therefore have to spend a lot of money and if they have a budget they
might run out of money. If they run out of money they’re off
the page and that’s gonna lead another advertiser to go to the top position and
then they might run out of money, too. So there’s interaction effects, even if the advertisers don’t
change anything at all. But then of course in an economic world, they have incentives to change what they
do if you change the way the market works. And so indeed they’ll change that as well. And so
all of those things are generally ignored. So why are they ignored? It’s not that people are stupid. And it’s actually that
it’s just very hard and very expensive to do
something other than this. The user experiments are really cheap. So let me kind of give a general framework
for thinking about these kinds of problems and that is to think about
these marketplaces as bipartite graphs. So this is gonna, I’m gonna use the word
seller for, for sellers or advertisers, just, just have one term. In but all in e-commerce marketplaces
as well as online advertising sellers are going to compete with each
other to attract user attention and ultimately purchases. And so users or consumers on these
platforms are going to enter queries or select products from a menu, and the platform is going to use
various algorithms for ranking them. So the economic market design problem for
the platform and that’s who the economists often think from
the platform’s perspective they’re going to try to figure out what are the ways to
rank sellers that get the most consumers to my marketplace that keep the sellers
happy in my marketplace and so on. And there, the tools they have or
the little dials they turn, would be things like fees for listings,
fees for clicks, or, percentages of sales. So just to see how this might work,
you know, suppose I’m shopping for my kids on eBay. And, first of all they say, oh Mommy i
want a “How to Train Your Dragon” costume, so i can look on eBay for
“How to Train Your Dragon” costumes. Then my, my little one might see me
searching for my middle one and say, oh no, no mommy you were searching for
costumes, i want a “Frozen” costume. So f might also search for
Frozen costumes. And then we might start thinking
about toys and search for “How to Train Your Dragon” toys. So all three of these search queries are
queries that could be entered by a user. And so eBay is gonna show a set of
sellers in response to each one. But, of course, there’s gonna be overlap. People who sell How to Train Your Dragon
costumes also sell How to Train Your Dragon toys. But then there’s people who
specialize in costumes and people who specialize in toys. So this is going to lead to, a bipartite graph where I’ve
got two types of, of guys. I’ve got sellers and they’re competing because they’re
both showing up on the same query. And so, the sellers, some sellers might
show up on both query A and query C. Some sellers show up on query A and
query B Some might show up on B and C, some might show up on all of them, and this is going to lead to
interactions among all of them. So one thing that’s really cool about
this is that actually i can measure this. I can measure this directly and actually as the the marketplace designer,
i can control this, i can manipulate it. So, unlike, kind of, social networks where
we think of them as being sort of more organic, and, and we can manipulate them
somewhat i can put a monkey in the cage. But on here I’ve,
I’ve sort of got my monkeys but i can actually manipulate you know,
how close friends they are, if you like. Okay. And that, that’s a, a really
interesting and powerful methodology. But it’s one, this is one that’s really
been explored from this perspective in a only very little way across
all of the disciplines that are, that are engaging with
these tech companies. Okay. So let me tell you,
quickly sort of, this is, and this is a real world problem
i should say that, you know, I’ve, this is been a focus of a number
of years of consulting for me. So I’m sort of telling you stories about,
partly based on my experience with that. So there’s sort of three types of
solutions to dealing with this problems. The first two are to continue
to do your user experiments but to try to do a better job with them. And I’ll, and I’ll touch on those,
I’ll touch on the first one briefly, talk a little bit more about the second
one, and then the third one is the one that, it really uses the network
science most deeply. So, the first is to just modify your
evaluation of short-term user experiments. So this is where an economist or
a social scientist can come in. So just, actually, by the way, if those of
interested like how does social sciences add value to, to the hundreds of
machine learning people in a tech firm, well this slide is a very
simple example of how one social scientist can have
a very large impact, very easily. So you conceptualize the problem and you attempt to address it
with alternative measures. So, for example, on,
in search, an advertiser’s bid is going to be responsive to how,
what the quality of their clicks are. That is, if I’m selling auto insurance and
you put me up on, and you put my ad up when somebody searches for dental floss,
then the person might click on my ad, but I’m not going to sell very much auto
insurance so that’s a low quality click. On the other hand, if they put
me up on somebody searching for car insurance,
that might lead to a high quality click. And so if you ignore advertiser behavior
then you are going to be very tempted if you’re especially if you need to, like, make more money this quarter like
Yahoo did for a number of quarters. You might be very tempted to put auto
insurance ads on dental floss queries. Because in the experiment it’s going to
look like it’s going to make you a lot of money because the auto insurance
guy’s paying 30 dollars a click. So you get a few more clicks, that’s very
nice, except for later on you think they might lower their bids if you keep
throwing them up against dental floss. Okay. So they, but that will never show up in a
user level experiment cause the advertiser won’t actually see the change and
respond to it. So one way to kind of avoid this
innovating in the wrong direction is to change what you measure. So if, of course, if i put up auto
insurance on dental floss, then somebody clicks on the ad, they’re very likely
just to click, to click right back to the dental floss query cuz they weren’t
really interested in auto insurance. So what i can do is that even though i
actually made money from that click. When i count the money i made from the
experiment I’m through that money away, I’m not gonna count it,
I’m gonna say that’s bad money. That is bad money in the sense
that that’s not sustainable money. And so when i evaluate the experiment
I’m only gonna count good clicks. And this is also something that you know,
I’d advise to news media organizations as well, when they start becoming data
driven, there’s a temptation to put, like, kind of junky headlines and so on. You’ve all seen more and
more of this, right, like you’ve got weird in all
the newspaper headlines these days. The weird thing you found out about this,
that’s because people click on that, but it doesn’t really make you like
the New York Times in the long term. So we’d throw away those, those,
those clicks and not count them even though the New York Times in
the short run makes money from them. A second example of this is that
advertisers spending is gonna respond to the return on investment and so
you can say, you can just tell the, the, the engineers you know you cannot
build algorithms that, that- For whatever reason raise price is for
advertisers. You’re just, you’re not allowed to do it. You, you’re, you’re,
you have to solve a constrained problem. You can try to improve
your ranking algorithms, but if it raises prices we’re
not gonna ship that product. That’s also of course a very
limiting thing to do, but it’s a way to to, to sort of prevent
yourself from harming the marketplace. Now as you can see these types
of solutions, they can help, they’re sort of a band-aid, but
they’re not really a fundamental solution. So the second thing, and
this is actually really the bread and butter of economics, but
almost no other science. So this is where we’re kind of unique and
weird I would say, is that economist’s actually believe
that we can build models and make predictions about counter factual
worlds that nobody has seen before. So you know, many people think
we’re quacks for doing this. In fact, people inside economics
think that we’re quacks for doing that as well, but
it is something that we try to do and we have at least a, a set of,
of well, a pretty well defined and well developed literature about how
to do that as well as possible. So the, the kind of simple way would
be to try to just have a simple model that relates short term
outcomes to long term responses. The second is to, to do something more
complicated we call a structural model, where we actually write down
a functional form equation for what somebody’s objectives are, and then
assume that they follow those objectives. And so if we change their environment, we,
if we know what their utility function is, we can predict what they’ll do to
maximize that utility or even solve for a new equilibrium. Okay? So just to give some
examples of each of these, here’ s an experiment that I ran on
Bing where I, I swapped search results. So I tried do see what would happen if you
took the thing in the first position and moved it down tot he third postilion, or
moved it down to the fifth position, or moved it down to the tenth, ten,
the, to the tenth position. And so what we found in this
randomized controlled experiment was that you lose about half
your clicks a little more than half your clicks if you move from
the top position to the third position. Okay?
And of course, this is a very big policy issue right now,
as well. So this is a, a general effect, but you might imagine that this
effect is really quite heterogeneous. So, the effect of re-ranking
stuff is gonna matter a lot. Like, if you know what you’re looking for,
if you type in Bank of America, and you put the Bank of America link
in the third position, you’re, you’re just gonna be annoyed and click on
Bank of America in the third position. So in fact, in our experiment, we threw
out those really, really clear queries, because that was just viewed as
unacceptable to the user, and, but even within queries that
are not quite that clear cut, there often is something that,
that could be more or less navigational. That can be more or
less clear where you wanna go. And there’s other types
of effects as well. So, in some work with [INAUDIBLE]
we’ve been working on modifying machine learning methods to try to, to
estimate heterogeneity in causal effects. So this actually requires changing
machine learning methodology because that methodology is designed to predict
outcomes, not to predict causal effects. And causal effects are actually,
there’s no ground truth. So I don’t know for any of you in the audience exactly
what your personal causal effect is. Therefore, traditional
machine learning methods, which work on trying to predict people’s
outcomes using observed outcomes as a ground truth don’t
work exactly right. So I won’t go through all the,
the details of that. There’s a the National Academy
had a recent symposium and I have a talk on that that’s
available there if you’re interested. I’m just gonna show you now, applying that
method to the search the search example we’re using a machine learning
method called regression trees, except for modified to predict causal
effects rather than just predict outcomes. And so what we get when we apply this
method is that we get a partition of the query space into subsets, and within
each subset we estimate a causal effect. And so we’re able to see, for
example, if the query is classified to be a good candidate for images and
also to be likely to be about celebrities, there was no effect of re ranking, because
if you were looking for a movie star, if you were looking for a move star, and you
saw lots of pictures of the movie star, you don’t really care about
the informational query. The results, but for
other types of queries, there was actually a very large effect,
even larger than average, for example things that were, that were rated
to be related to Wikipedia references, more informational queries
had a much larger effect. And so, we can then, the,
each row here is a subset of the sample. We can actually estimate the treatment
effects in each part of the sample, and we can then use that to under, to do a much
better job predicting what’s gonna happen. Cuz we can say, well, you know, here is some here is some
branch where there’s a really big effect. And then I can try to understand,
well are, did those types of advertisers, are they very sensitive to quality? Are they very responsive? Are they very active? There, there’s more likely to be a big
a big response by the advertisers in, in those parts of the,
of the of the space. So, this is obviously sort of, it doesn’t
require a lot of assumptions, but, it’s also very limited. I can say qualitatively whether
a particular change is likely to affect a particular type of advertiser and
whether that’s gonna have an effect, but I’m not gonna be able to be,
do a very good job of putting quantitative responses on exactly how much
of an effect it’s going to have. So, a second approach that I’ve used, is to actually build this
full structural model. So we’re gonna assume in my first
paper on this we assumed that the, the advertisers maximized profits. It turned out that that model did
a pretty lousy job predicting behavior. So we then build a much more richer model
where we allowed advertisers to have one of a, of a dozen objective functions
with some of them trying to get to the top position, some of
them had budget constraints, and so we actually used the fact that we were
constantly experimenting on every week. We’re changing something about how
the system works and we have lots and lots of stimuli. And if we see the same person
responding to ten different stimuli, we can classify which
type of person they are. And so, we, we, use this behavioral model,
and then we have to go back and try to calculate if I made some other
change that I’ve never made before, how do I think they would respond? And that’s where we get into one of
the big network computational problems. My, my paper actually looks at a subset
of the keyword Bids on the system, I’ve got tens of millions of bids
that I’m modeling in my sample. That’s a subset of all of the different
keyword bids in the dataset. So in principle I would be
computing an equilibrium betwe-, of a game between a couple
of hundred million players. That’s hard, you know,
my computers can’t do that very well. But, if I can use something about the
network structure, to, to say that well, really, you know, these they don’t all the
hundred million, or couple hundred million don’t care equally about the other couple
hundred million, we can find communities and segments and try to compute equilibria
there, and then try to reconnect them. So I’d say this area is,
is very much open for research how to compute large scale
equilibrium in sort of a reasonable way. So we are doing it in a very hacky way for
our paper cuz that’s not the main point of our paper, but it’s a, it’s a,
something that is very useful for these, these tech firms and
it’s gonna become more and more common for all of people studying networks to
have these very, very big networks. To try to worry about. Okay? So the last thing I’ll mention,
talk about is network experimentation. And so in network experimentation,
we’re trying to think about actually running experiments
on the network itself and really take advantage of, it’s run, not
user experiments but cellar experiments. Directly experiment on them. So to do that the, the sort of way
that’s developing in a, in a but rapidly growing literature, is to think about
grouping my sellers into communities. Now I should say,
this is mostly applied to like Facebook, not to my market networks. But for, for
the user networks where it’s been applied. We, we group the sellers into communities,
in, in, my seller context I’m going to do it by taking a bipartite graph
weighting links between advertisers and queries using weights in terms
of how much business they do. And the big problem I have is
that spillovers are large. There’s it’s not like there’s just
the guys who do toys on Ebay and just the guys over here who do laptops. There’s a lot of people who
cross over a lot of categories. On eBay and on Search. And then we do the experiment at the
cluster level, watch them for a long time. So just to kind of see
how this might work, I could run a community
detection algorithm. Something like, oops, is that,
no, my fonts aren’t messing up, I thought they were okay. Let me see, am I gonna lose. Well, I’ll see,
I’ve got a PDF on here somewhere. So what the Newman Et Al algorithms
are doing is they are trying to find communities where there is lots of
interaction within a community and not a lot across and these orange units are
ones that have a lot of spill overs, okay? So then I am going to randomize at
the community level, and try to you know, analyze that data. Now, I’ve got a bunch of problems. One is how do I deal with the spillovers? My observations aren’t
actually independent. So, how do I do my statistics? And also these kinds of designs make it
easy to study, like, well, what would happen if a whole community got treated or
if a whole population got treated? But they make it actually very hard
to study pure effects, because most people either had all their peers
treated or none of their peers treated. You don’t see a lot of in between. So one, one, but, one approach to this,
proposed recently by some computer scientists, was to basically
define being treated as having, a, say over 60% of your neighbors treated. And then use propensity score
weighting to adjust for non-random assignment to this condition. But there’s, this is really an open
research area of how to do this, and this is something that
I’ve been working on. So let me briefly tell you about
the way that we’ve been studying, testing hypotheses about
peer effects in networks. So to do that, I first just need to really
briefly introduce the approach we would use to studying networks if we were
experiments on networks if I just had direct effects. So imagine I’m, I’ve got an experiment. I’ve got treated guys in yellow and
the control guys and I’m trying to test hypotheses. So, the approach that has
been growing in popularity in doing this is something
called randomization inference. This is how we can do the statistics. And so, the way it works, is I can say I’m gonna have
potential outcomes for each unit. Say, unit A, I saw him being treated. So Yi of one is his outcome
if he was treated, but I don’t know what his outcome would have
been if he’d been the control group, and that’s my fundamental problem for
inference. Wi is his treatment assignment
is one because he was treated. So now I can say, well, what-, how can I figure out,
what the affect of the treatment is? Well, first of all, I can say what’s my
point estimate of the treatment affect, the average of the control guys
is I’ve got two and four, the, so the, the, the sum is six,
the average is three. In the, in the treatment group I’ve got,
three and five the average is eight. Oops, I, I flipped the sign there. I want to see what’s the P value of,
oh, no, no. These are, I, I’m sorry. I did the opposite ones. Three, these are the real treatments so
three, two, five, four are the real treatments and I want
to understand what the P value is so the treatment effect is negative two. So, the Randomization Inference
approach says, well under the null hypothesis
there is no effect of anything. I actually know what you’re
outcome would have been if you hadn’t gotten the treatment. Or I know what your outcome would have
been if you had been in the control group. So I fill in under the null that
your outcome would have been the same under either of those and that’s
what’s called a sharp null hypothesis. A sharp null hypothesis is one where you
know exactly what would have happened in the alternative treatment regime,
without under the null hypothesis. And so then what I can do is I can say,
well I can imagine under the null, it doesn’t matter what the treatment
assignments were, so I can think about all sorts, redrawing in lots of different
ways what the treatment vectors are, each of those, if it was equal
probability of treatment and control, would have equal probability, and in each
case, I can figure out what the test statistics would have been if those,
that had been the treatment assignment. So I know that these two guys were
treated and these were control. If I re-labelled them then I would
get a different test statistic. But if I of course, if there’s no effect
then that test statistic would be just another estimate of what this
test statistic is under the null. I’ve got a full distribution,
so I can say, you know, the test statistic is less than
negative two a third of the time, it’s greater than two a third of the time,
and that’s gonna give me an, that’s actually gonna give me
a statistical significance level that doesn’t depend on large samples,
it doesn’t depend on anything really. And so it’s quite general, so
this approach is gonna be really good for working on networks where we might have,
we don’t wanna make a lot of assumptions. But the problem is that
a lot of the assumption, the hypothesis you might want to
test in networks aren’t sharp. So if I want to say, I wanna imagine
tests where there’s peer effects, the problem is that if
a treatment has an effect on me, then I don’t know what my outcome would
have been if I hadn’t been treated. So these are what we call
non-sharp null hypotheses and so in our recent paper what we do is we
try to actually figure out how to do those types of things, So
I’m going to, I’m gonna, I’m running out of time I’m just going
to give you one example that shows you how we can turn a non-sharp hypotheses
into a sharp hypotheses and actually use this this Fisher randomization inference
approach for very complex hypothesis. Which allows us to then do correct tests of very interesting experiments and
networks. So here’s my little network and what I’m, what I’m gonna start by doing
is picking two focal units A and B, this guys treated, this guys
controlled, the red ones are focal. And I want to test the hypothesis that
friends of friends don’t have any impact, that only your friends have an impact. So in order to test that hypotheses,
I’m gonna first take my focal units and take all of their direct friends and I’m
going to call those buffer units, okay? The buffer units are gonna
be their friends and then the auxiliary units are gonna
be their friends of friends. And so under my no hypotheses that
friends of friends don’t matter, changes to the treatment
status of the auxiliary units should not affect the outcomes
of my focal units. And so what I’m gonna do is,
I’m gonna take, instead of thinking that I’ve got
all this data from a big network, I’m gonna pretend that I only
ran an experiment on those guys. And I didn’t touch their friends
when I ran my experiment, and I’m gonna pretend that I only randomized
their friends of friends, and then I’m gonna use that as the variation
that I analyze in my experiment. So I can do that, and I can look at
the probability of all the different friends of friends assignments,
only randomizing the auxiliary units I can come up with a test statistic, and
then I can come up with a distribution of what the test statistics
should be under the null. And then, have a valid hypothesis test for
whether friends of friends matter. And this turns out to be a very, very
general approach you can work for lots and lots of hypotheses, you need a very good
network to get good statistical power. So, we’re trying to push this forward,
in terms of testing much more interesting hypotheses on things like Facebook and
so on, and network experiments. Okay, so let me stop there. Thank you. [APPLAUSE]

About Ralph Robinson

Read All Posts By Ralph Robinson

Leave a Reply

Your email address will not be published. Required fields are marked *