The world of technology is rife with horse races–so much so that many tech blogs read like the Daily Racing Form. There’s PC sales vs. Mac sales. iPhone shipments vs. Android shipments. Internet Explorer usage vs. Firefox usage vs. Chrome usage. Everybody loves to keep track of who’s ahead and interpret the significance of any changes. (Including me.) The primary way that tech publications do that is by reporting on market share numbers reported by research firms and other third-party entities.

More and more, though, I’ve come to the conclusion that the good articles about tech market-share data may be outnumbered by ones that are, at best, not very enlightening. Good data gets mangled and misinterpreted; bad data gets treated like good data. Herewith, a few of the reasons why it’s so important to approach these stories with a healthy dose of skepticism.

Different companies provide different numbers at different times. In May of 2010, it was big news when Android phones outsold iPhones for the first time. And in August of 2010, it was also big news when Android phones outsold iPhones for the first time. The Groundhog Day effect occurred because the two batches of news stories were based on two different research firms’ reports on phone sales. And since all third-party market share numbers are based on estimates of one sort or another, they’re never gospel.

The numbers require explaining that doesn’t always happen. All market-share figures involve disclaimers, loopholes, and footnotes. You need to read the whole article to understand what’s going on, and that’s assuming that the reporter has clearly and accurately represented the data. For instance, NPD’s research on computer retail sales are both useful and widely quoted–but they don’t include information for the country’s largest merchant, Wal-Mart, or direct-sales sites such as those operated by Dell and HP.

Sometimes the numbers are just plain wrong. In April of last year, IDC said that they were up by eight percent year over year–less than half the industry average. Bleak!  Except that rival research firm Gartner simultaneously estimated that Mac sales were up a robust 34 percent. When Apple released its financials later that month, it said that said that Mac sales were up 33 percent. Those that paid attention to Gartner had a better idea of what was going on than anyone who listened to IDC.

Any digit to the right of the decimal point is meaningless. So are some of the digits to the left. It’s pointless to read anything into tiny shifts in market share, like Firefox going from 8.71 percent market share to 8.07 percent. That was from a study back in 2005, and anyone who considered the supposed decline to be meaningful was presumably startled when Firefox’s share continued to boom in subsequent years. And all browser market share stats are based on online studies that come to strikingly different conclusions: Right now, for instance, NetApps says that Internet Explorer has 57.07 percent of the market, and W3Counter says it has 41.3 percent.

I’m not saying that market-share numbers are meaningless. But I try to keep a few questions in mind as I read about them (and write about them, for that matter). Such as…

1. Do two or more research organizations agree about how things stand? If they do, that’s a great sign. If they’re at odds, you know at least of them is wrong.

2. Is it a blip or a trend? if the market share of a PC brand or a phone platform or a browser dips in one given report in a single month, it’s impossible to judge the significance of what happened, and therefore silly to even pay attention. If a product’s share keeps dipping month after month, however, you know that something is going on.

3. Does the analysis ring true? When an article reporting on a market-share study clearly explains the data–where and when it was collected, for instance–I have more confidence in the reporter’s ability to interpret its meaning. And when a journalist seems to be using numbers to confirm his or her preconceived notions, I get cynical fast.

[NOTE: This story is republished from last week’s Technologizer’s T-Week newsletter–go here to sign up to receive it each Friday. You’ll get original stuff that won’t show up on the site until later, if at all.]

 
2 Comments


Read more: Apple, Operating Systems, Web browsers