Guide to statistics (intro) — for Yelp-listed business owners


The issue has been near-beaten-to-death in the United States. In fact, the FTC just concluded a long investigation involving hundreds of thousands of pieces of potential evidence that Yelp attempts to pressure businesses to pay for listings in exchange for suppressing negative reviews, but failed to find a smoking gun.

The same business owner complaints are now coming to light in Canada. There’s no conspiracy north of the border, either.

It’s easy enough to sympathize what business owners are going through, of course, because it feels real to them. No one likes negative reviews. I often feel like some of the negative reviews on TripAdvisor or Yelp are unfair. One lady crazily calls out a B&B proprietor for his “heavy drinking,” when hundreds of other guests of the lodgings and restaurant (including us) saw him working hard and having no issues with partying, day after day after day. “We are open, warm people and we gather around the fireplace in proximity to some guests, and I drink one beer because that’s genuinely how I live, and this is the thanks I get, a public shaming?” It’s no fun being a hardworking small business owner some days.

But more to the point, the narrative of Yelp harassment feels quite real, too, even though it isn’t.

The illusion of some sort of conspiracy is easy enough to explain away with basic statistics, and that breaks down roughly into two easy points.

  • First, the law of small numbers. I think many business owners would simmer down and accept the pattern of their reviews and ratings if they had large numbers of reviews. The distributions would revert to an average and the pattern would seem very steady. It’s a basic fact that in small sample sizes, results can be much more volatile. You’re more likely to be taken aback by the numbers when you don’t have enough numbers to work with yet. The business owners that have two glowing 5-star reviews and no others certainly aren’t complaining. Those who have two terrible reviews and two good reviews believe something is fishy, because the result is extreme. Yeah, but that result would even out given enough time. It’s the law of small numbers, pure and simple.
  • I’m not sure what principle of statistics this illustrates, but let’s call it “Yelp calls everyone.” Yes, Yelp does call on business owners to attempt to sell them upgraded listings. This isn’t selling the ability to suppress negative reviews. Some business owners have said “they imply it.” How do you know? If Yelp calls everyone, then eventually they’re going to call on someone when they’re going through a crisis of “oh no, I got a couple of really horrible reviews,” based on the law of small numbers (applying to them, possibly randomly).

Statistics are a bewildering thing sometimes. There are small numbers at play, and very large numbers, and lots of potential randomness. Out of that, we see coincidences, because we’re wired to create narratives to explain events. But perhaps it’s time to put down the tinfoil hat in this  case.

Like the FTC said: case closed.

You may also like