If I know the number of pageviews I can predict the number of pageviews with 100% accuracy
Here's a post that I reblogged from Tom Ewing over on Tumblr, followed by the reply I gave. The article in question is "How Promotion Affects Pageviews On The New York Times Website" by Brian Abelson.*
via @bmichael on Twitter. Long, knotty, work on the extent to which promotion causes pageviews, which is obviously important to know when “pageviews” is your metric of success. There’s also a good bit at the beginning about the perils of a metric dominating your approach to your job - germane to the pageview issue but worth bearing in mind more generally.
It’s complex work, but here’s my gloss after one reading.
Say you’re an editor, and want to allocate resources, know what and who to commission, etc. Pageviews are a powerful metric for doing this, because pageviews pay the bills. Obviously promotion is going to influence pageviews to some degree, and the important question becomes: how much?
If the level of influence of promotion is relatively low, then you can reasonably suppose that the extra pageviews of that article are down to its topic, its writer, its innate brilliance, etc.
If the level of influence of promotion is relatively high, then you can reasonably suppose that writer, topic, etc. don’t make that much difference. They don’t make NO difference - partly because to some extent judgements about what works are already baked into the system, and articles about “dull” topics don’t make the front page of the New York Times. So a high level of influence wouldn’t show that, eg, you can make Lorem Ipsum text a hit with promotion. But it would show that - on the scale of pageviews of a major website - the power of promotion to drive traffic outstrips the power of anything else.
So what is the level of influence? The piece suggests it’s about 90%. 90% of the variance in pageviews can be explained by the level of promotion a piece receives.
That sounds pretty big. It probably is big. The piece suggests focusing on finding metrics beyond pageviews which explain success at the margin - that extra 10%. But you could draw other conclusions, You could draw the chilly conclusion that talent doesn’t make a difference to views in most cases, and the experiences of most of my journalist friends in the web era suggest that publishers have indeed been drawing that conclusion. You could also draw the possibly more hopeful conclusion that received editorial wisdom about the quite narrow bounds of topics and opinions that “work” online is bunk - what gets promoted succeeds, and topic matters much less.
And you should certainly draw the conclusion that what applies at the NYT might not apply anywhere else.
But it also might - it fits with other marketing/business knowledge about the relative impacts of distribution and visibility against inherent qualities. Everyone seeks some things out, but at an aggregate level people consume what’s easiest to get to. This applies to content too, so of course promotion matters immensely within a site. (The question of how SITES become more or less popular is a different one, and quite possibly content is more important there.)
It’s easy to lose sight of the importance of promotion and distribution because the sheer scale of web media means that “viral hits” happen all the time and seem immensely important and frequent, when as a proportion of all content viral events are extremely rare and don’t pay many bills. But their visibility gives people the sensation that we live in a content meritocracy where the inherent qualities of a piece might count for more than the promotion it receives And this work shows that isn’t true.
Tom, I think you should rethink your analysis here: nothing in the article tells us how much of the correlation between "promotion" and pageviews is caused by promotion leading to pageviews, how much by high pageviews leading to further promotion, and how much is due to other causes. Yet your endorsing the 90 percent figure means that you take the correlation to be entirely down to promotion, with nothing going the other way and no other inputs.**
That I put the word "promotion" in scare quotes doesn't mean I think it's false but rather that something like "length of time a piece is linked on the homepage" isn't quite part of the normal connotation of the word "promotion" and this difference needs to be highlighted. I also think that when Abelson uses the words "predict" and "explain" they need scare quotes even more, that they're potentially wrong. Again, all he's showing is that if he's got one set of numbers he can get within 10 percent of another one; he's not explaining the connection between the numbers. He obviously believes that promotion is the main driver here, but he certainly hasn't shown that it's the main driver or proven that it explains the pageviews (and he's not saying that it's entirely responsible for the 90 percent, though he only gives one sentence to the possibility of influence feeding back from pageviews to promotion).
My problem with the word "predicts" is the connotation of one thing leading to a later thing - whereas "how long a piece is linked on the homepage" is as after the fact as "number of pageviews," and again we can envision pageviews influencing time on the homepage as much as vice versa.
In short, I don't think the piece earns the word "affects" in its title.
I'm not a statistician or a social psychologist, but my guess is that in a nonexperimental setting like the New York Times you can't control for feedback loops etc. So I don't know what someone like Abelson could do to follow up on his hypotheses. It would help if he knew that as of yet that's all they are: hypotheses.
Actually, though, the main reason for my reblog is that I want to challenge your tacit assumption that "promotion" on the one hand and "inherent qualities of a piece" ("merit," "content," "readers' interest," etc) are the only two phenomena at play here and that we can divide the pie up between them (whether it's 90%-10% or 60%-40% or whatever). (Minor point: we should use the word "appeal" rather than "merit" or "quality," since that way we allow that the masses can be asses, if they so choose, and we don't have to wrestle with the problem word "inherent.") Among other things, what you've left out is cumulative advantage, i.e., that people are interested in what other people are paying attention to, that people gather where other people gather, and so on. This is neither promotion nor appeal - well, you could say it's part of appeal, but as such it can't be inserted (via promotion or content or anything else) into something that is not yet popular - and it feels like less of a "thing" than promotion or appeal do. But it's always a factor when something becomes well-known - all success is viral - and there's an ineradicable element of chance as to what gets a feedback loop that leads to success in the first place.
Quick explanation of "cumulative advantage" for those who don't know the term: if I see two articles that are equally well-promoted and on the surface seem equally interesting but I only have time to read one, and I see that Tom, Kat, and Dave have read Article A but not Article B, I'm extremely likely to choose A over B. This is for two reasons: (1) I'm interested in what Tom, Kat, and Dave are reading, and know that, among other things, reading the piece gives me the chance to talk to them about it, to socialize; and (2) I think it's possible that the three of them know something in A's favor, which is why they've read it. But actually it's quite possible that the reason Tom, Kat, and Dave read it is that they saw that Alan, Mark, and Erika had read it, who in turn saw that Sarah and Sabina and Isabel had read it, etc., and that none of them know anything in the piece's favor. And such small essentially random differences (10 people in our circle having read A and zero people having read B) will quickly spiral into major differences in a piece's visibility, even where there's no difference in either its appeal or how much it was originally promoted. So there's an ineradicable element of luck as to what gets popular, above and beyond whatever promotion a piece gets and whatever appeal it has.***
Another feature of cumulative advantage, though, is that as far as I can figure out there's no way to quantify how large a role it plays in something's success. It's always there, and it contributes to both appeal (a writer who writes one successful piece has a chance of getting paid to write more, and so in effect can get paid to work on her writing) and promotion (and she's more likely to get her next piece promoted, which increases its chances, which gets her more work, etc.). But still, while I can at least conceptualize what it means to say that "promotion" contributed 20 percent to something's success and appeal 30 percent (though since the two feed back on one another maybe my attempt to conceptualize is misguided), I have no idea if attributing a percentage number to "luck" could ever even make sense, over and above the impossibility of coming up with a percentage. But Tom, since you're in a field (market research) that tries to quantify wherever doing so might do some good, maybe you can tell me if I'm wrong on this, that maybe there are ways to quantify cumulative advantage, if such a task is doable. In any event, this is a long-winded attempt to get you to read my post on how "Bar Bar Bar" became a hit, where I try to be clear as possible about my confusion.
*Posted originally on Abelson's own blog.
**Of course, your statement "to some extent judgements about what works are already baked into the system" indicates that you know better, but your waving the figure 90 percent at us implies that you forget what you know and will sloppily place the baking in a remaining 10 percent rather than considering that it might cause some of the promotion in the first place.
***I've put Duncan Watts' in the tags because his book Six Degrees and his pieces in the New York Times and in Science are what introduced me to the concept "cumulative advantage," the latter piece convincingly proving the concept.
via @bmichael on Twitter. Long, knotty, work on the extent to which promotion causes pageviews, which is obviously important to know when “pageviews” is your metric of success. There’s also a good bit at the beginning about the perils of a metric dominating your approach to your job - germane to the pageview issue but worth bearing in mind more generally.
It’s complex work, but here’s my gloss after one reading.
Say you’re an editor, and want to allocate resources, know what and who to commission, etc. Pageviews are a powerful metric for doing this, because pageviews pay the bills. Obviously promotion is going to influence pageviews to some degree, and the important question becomes: how much?
If the level of influence of promotion is relatively low, then you can reasonably suppose that the extra pageviews of that article are down to its topic, its writer, its innate brilliance, etc.
If the level of influence of promotion is relatively high, then you can reasonably suppose that writer, topic, etc. don’t make that much difference. They don’t make NO difference - partly because to some extent judgements about what works are already baked into the system, and articles about “dull” topics don’t make the front page of the New York Times. So a high level of influence wouldn’t show that, eg, you can make Lorem Ipsum text a hit with promotion. But it would show that - on the scale of pageviews of a major website - the power of promotion to drive traffic outstrips the power of anything else.
So what is the level of influence? The piece suggests it’s about 90%. 90% of the variance in pageviews can be explained by the level of promotion a piece receives.
That sounds pretty big. It probably is big. The piece suggests focusing on finding metrics beyond pageviews which explain success at the margin - that extra 10%. But you could draw other conclusions, You could draw the chilly conclusion that talent doesn’t make a difference to views in most cases, and the experiences of most of my journalist friends in the web era suggest that publishers have indeed been drawing that conclusion. You could also draw the possibly more hopeful conclusion that received editorial wisdom about the quite narrow bounds of topics and opinions that “work” online is bunk - what gets promoted succeeds, and topic matters much less.
And you should certainly draw the conclusion that what applies at the NYT might not apply anywhere else.
But it also might - it fits with other marketing/business knowledge about the relative impacts of distribution and visibility against inherent qualities. Everyone seeks some things out, but at an aggregate level people consume what’s easiest to get to. This applies to content too, so of course promotion matters immensely within a site. (The question of how SITES become more or less popular is a different one, and quite possibly content is more important there.)
It’s easy to lose sight of the importance of promotion and distribution because the sheer scale of web media means that “viral hits” happen all the time and seem immensely important and frequent, when as a proportion of all content viral events are extremely rare and don’t pay many bills. But their visibility gives people the sensation that we live in a content meritocracy where the inherent qualities of a piece might count for more than the promotion it receives And this work shows that isn’t true.
Tom, I think you should rethink your analysis here: nothing in the article tells us how much of the correlation between "promotion" and pageviews is caused by promotion leading to pageviews, how much by high pageviews leading to further promotion, and how much is due to other causes. Yet your endorsing the 90 percent figure means that you take the correlation to be entirely down to promotion, with nothing going the other way and no other inputs.**
That I put the word "promotion" in scare quotes doesn't mean I think it's false but rather that something like "length of time a piece is linked on the homepage" isn't quite part of the normal connotation of the word "promotion" and this difference needs to be highlighted. I also think that when Abelson uses the words "predict" and "explain" they need scare quotes even more, that they're potentially wrong. Again, all he's showing is that if he's got one set of numbers he can get within 10 percent of another one; he's not explaining the connection between the numbers. He obviously believes that promotion is the main driver here, but he certainly hasn't shown that it's the main driver or proven that it explains the pageviews (and he's not saying that it's entirely responsible for the 90 percent, though he only gives one sentence to the possibility of influence feeding back from pageviews to promotion).
My problem with the word "predicts" is the connotation of one thing leading to a later thing - whereas "how long a piece is linked on the homepage" is as after the fact as "number of pageviews," and again we can envision pageviews influencing time on the homepage as much as vice versa.
In short, I don't think the piece earns the word "affects" in its title.
I'm not a statistician or a social psychologist, but my guess is that in a nonexperimental setting like the New York Times you can't control for feedback loops etc. So I don't know what someone like Abelson could do to follow up on his hypotheses. It would help if he knew that as of yet that's all they are: hypotheses.
Actually, though, the main reason for my reblog is that I want to challenge your tacit assumption that "promotion" on the one hand and "inherent qualities of a piece" ("merit," "content," "readers' interest," etc) are the only two phenomena at play here and that we can divide the pie up between them (whether it's 90%-10% or 60%-40% or whatever). (Minor point: we should use the word "appeal" rather than "merit" or "quality," since that way we allow that the masses can be asses, if they so choose, and we don't have to wrestle with the problem word "inherent.") Among other things, what you've left out is cumulative advantage, i.e., that people are interested in what other people are paying attention to, that people gather where other people gather, and so on. This is neither promotion nor appeal - well, you could say it's part of appeal, but as such it can't be inserted (via promotion or content or anything else) into something that is not yet popular - and it feels like less of a "thing" than promotion or appeal do. But it's always a factor when something becomes well-known - all success is viral - and there's an ineradicable element of chance as to what gets a feedback loop that leads to success in the first place.
Quick explanation of "cumulative advantage" for those who don't know the term: if I see two articles that are equally well-promoted and on the surface seem equally interesting but I only have time to read one, and I see that Tom, Kat, and Dave have read Article A but not Article B, I'm extremely likely to choose A over B. This is for two reasons: (1) I'm interested in what Tom, Kat, and Dave are reading, and know that, among other things, reading the piece gives me the chance to talk to them about it, to socialize; and (2) I think it's possible that the three of them know something in A's favor, which is why they've read it. But actually it's quite possible that the reason Tom, Kat, and Dave read it is that they saw that Alan, Mark, and Erika had read it, who in turn saw that Sarah and Sabina and Isabel had read it, etc., and that none of them know anything in the piece's favor. And such small essentially random differences (10 people in our circle having read A and zero people having read B) will quickly spiral into major differences in a piece's visibility, even where there's no difference in either its appeal or how much it was originally promoted. So there's an ineradicable element of luck as to what gets popular, above and beyond whatever promotion a piece gets and whatever appeal it has.***
Another feature of cumulative advantage, though, is that as far as I can figure out there's no way to quantify how large a role it plays in something's success. It's always there, and it contributes to both appeal (a writer who writes one successful piece has a chance of getting paid to write more, and so in effect can get paid to work on her writing) and promotion (and she's more likely to get her next piece promoted, which increases its chances, which gets her more work, etc.). But still, while I can at least conceptualize what it means to say that "promotion" contributed 20 percent to something's success and appeal 30 percent (though since the two feed back on one another maybe my attempt to conceptualize is misguided), I have no idea if attributing a percentage number to "luck" could ever even make sense, over and above the impossibility of coming up with a percentage. But Tom, since you're in a field (market research) that tries to quantify wherever doing so might do some good, maybe you can tell me if I'm wrong on this, that maybe there are ways to quantify cumulative advantage, if such a task is doable. In any event, this is a long-winded attempt to get you to read my post on how "Bar Bar Bar" became a hit, where I try to be clear as possible about my confusion.
*Posted originally on Abelson's own blog.
**Of course, your statement "to some extent judgements about what works are already baked into the system" indicates that you know better, but your waving the figure 90 percent at us implies that you forget what you know and will sloppily place the baking in a remaining 10 percent rather than considering that it might cause some of the promotion in the first place.
***I've put Duncan Watts' in the tags because his book Six Degrees and his pieces in the New York Times and in Science are what introduced me to the concept "cumulative advantage," the latter piece convincingly proving the concept.