Pinch Media records a lot of data as part of our reporting on the AppStore, but there’s a lot we deliberately don’t publish. For instance, we don’t report on reviews. Reviews are currently the easiest-to-manipulate aspect of the AppStore. A person doesn’t have to use the application to review it, and the number of reviews are low enough that an unscrupulous developers can ’stuff the ballot box’ – similar to asking all of your friends to vote for your items on Digg.
For applications that collect e-mail addresses, I/m sure we’ll see the sort of aggressive review manipulation that comes to all review-based systems in time. Here’s a few of the tricks we commonly had to deal with back when I worked for a comparison-shopping engine:
a) asking users for reviews in corporate promotional e-mails,
b) conducting surveys on user satisfaction, and then asking only the happiest users to provide reviews,
c) tying reviews to incentives – only providing the incentives when the review had been written.
In short, online reviews are an extremely weak thing to base your rankings on. Which is why I’m happy to report that Apple doesn’t seem to do this. We took AppStore applications’ review scores (out of 5) and ranks (where they appeared on the ranked ‘top 100′ lists) and checked to see if the variables were correlated – that is, if higher ranks meant higher reviews, or vice versa. The result: there’s no significant relationship between reviews and AppStore ranking. (For the statisticians out there, the correlation coefficient was around -0.10.)
For AppStore rankings, reviews simply don’t matter. Considering the number of ways they can be manipulated, this is probably a good thing.

