The Power of Reviews

johnny_ultimate

Just had a quick read of this:

http://penny-arcade.com/report/editorial-article/the-power-of-review-scores-why-critics-have-more-control-than-we-think1

May not be the most scientifically rigid study ever, but the findings are really food for thought. Knowledge of a game's critical consensus can favourably skew a player's perception towards the game, whereas the players who went in with a negative preconception had a much more negative response (in this case, Plants vs Zombies). The blind study of no review scores, I'm sure the researchers were happy to find, was smack bang in the middle.

I'm a firm supporter of (good) game criticism, although always a bit critical of the obsession with box-ticking reviews and numbered scores. Nevertheless, I am always fascinated about the effect metacritic scores and the like can have on gamers, and probably most of us are guilty to varying degree of being swayed by such aggregation sites. While many always bemoan or question the need for critics, studies like this show they still have a surprising amount of sway. Not just on purchasing decisions, but perhaps our perception of the games themselves. It's not new information - well known how obsessed publishers and developers can get over metacritic - but still welcome to see some simple research done on the subject.

Any thoughts?

Retr0gamer

I have to totally agree with that, especially about people going in with negative impressions of a game and not enjoying it. When I was younger I had a negative opinion of Western RPGs and hated them and probably made some stupid posts about them here as well. I went into KOTOR with negative opinions of the game and genre and decided I hated it. It wasn't until I played Mass Effect after I had matured that I realised how wrong I was.

I'm ashamed to say that I was guilty of this a lot when was younger but thankfully have matured an awful lot in recent times. It's just a shame that the community has matured with me.

I think it's mostly down to immaturity in the videogame community and how people are afraid to express their own opinion even if it isn't popular because they will be shot down by the masses.

As Henry Jacobsen said, "People would rather be wrong than be different."

There's lots of other examples:

The zelda CDi games. The first two zelda CDi games are regarded as terrible while the third game is supposedly good and plays like the old game. Most people even in professional publications have obviously no played the games or played them on an emulator. The problem is that getting a working CDi is a very difficult task that I can attest to and the best and only CDi emulator didn't support gamepads until a few weeks ago so no wonder the games controlled badly. Trustworthy reviews by people that have played the games on HG101 and by halfblindgamer comment on how the first two zelda CDi games are actually quite good while the third game, which apparently is like the old zelda games according to some publications, is in fact one of the worst games ever made.

Other games like Donkey Kong Country and Goldeneye are still regarded as classics, even by people playing them now for the first time and it seems to me that reviews and opinions of these games have skews peoples perspectives. Both games rated very highly mostly due to technical innovation. Playing them now it's obvious that DKC is a decent but not an amazing game while Goldeneye is a relic of early 3D gaming and in the light of innovations in the genre is really showing its age. Yet there's people that insist DKC is better than Super Mario World and Goldeneye is the best FPS game ever.

TPD

I'd be curious to see the results if there was a test group shown reviews which were 50/50 positive/negative. I'd imagine the individual results would be more polarized than otherwise - picking a side as such.

Grumpypants

I've always found when a game gets slammed i sometimes get more out of it as I went in with low expectations and it works the other way around Ive played some loved games and just not connected with them at all.

Retr0gamer

I find that as well. Some of the best games I played this generation have gotten mediocre reviews. Stuff like Deadly Premonition, Fragile Dreams and Nier were excellent and much better than reviews suggested.

It also works the opposite way with highly praised games. I found Arkham Asylum good but way overhyped. Enslaved I thought deserved the reviews it got, it was a very mediocre game and it's really not as great as some fans will have you believe. I also found that the critically acclaimed the Darkness to be one of the worst FPS games in a long time.

abelard

I don't think reviews have ever effected how I actually perceive a game I'm playing, that I know of anyway. I think it comes from being a 6/7 year old kid in the days of the SNES and renting games from a local XtraVision where my decisions were pretty much based on the display boxes! I played some games that I later found out were regarded as classics and some that were howlers, but was able to make up my own mind playing them.

However, I'll easily admit that reviews have effected what games I've bought - and I've become a lot more conservative not wanting to take risks on games that review poorly. When it comes to games and books, reviews are a huge factor in deciding what I will read/play, but when it comes to music and films I don't really care about reviews and will base purchases more on my own knowledge of favoured genres etc.

richymcdermott

Exactly! Thats why i dont get all the hate towards sonic 06 , sonic 06 is the holy grail of gaming while shining force series despite its raving reviews is a pile of ****.

Retr0gamer

richymcdermott wrote: »

Exactly! Thats why i dont get all the hate towards sonic 06 , sonic 06 is the holy grail of gaming while shining force series despite its raving reviews is a pile of ****.

Reported for trolling :P

richymcdermott

Retr0gamer wrote: »

Reported for trolling :P

not my fault i couldn't last more than ten minutes playing it :pac:

Mr.Saturn

The publisher/reviewer relationship has become such a mutually-abusive one, escalating over the last decade especially, that it's hard not entirely jaded by the process, as the whole Gamespot/Kane&Lynch jazz can attest to, as a recent example.

johnny_ultimate

As an aside, Indie Game: The Movie offers two very different perspective from developers on review scores. For the Team Meat guys, the universal acclaim was a godsend. You can see the joy and genuine surprise in their eyes as the positive feedback comes flooding in, followed by equally surprising sales. For them, it was an affirmation of their hard work and effort.

Jonathan Blow (Braid), though, has a different opinion. While he also received great reviews and great sales, for him they were very much a mixed blessing. He openly criticises - and TBH I'd agree with him completely on this - the 'surface level' critiques his game received, with many ignoring what he felt were the more personal and important aspects of his game. While lots of people were talking about and playing his game, they weren't engaging with it at all, and it ultimately sent him through a period of frustration and depression as he tried to counter and explain his ideas online when having to face stuff like this.

I think they're both valuable takes on the role criticism and discussion can have for developers. I know I always wish there were less ethically dubious review sources (as mentioned by Mr. Saturn above) and ones that delved deeper than mere gameplay / graphics / music / 'replay value' / 'longevity' box-ticking (the latter two being ludicrous criteria to judge something). Yet at the same time a series of good reviews can be an almost life and death determiner for studios, particularly smaller ones. There's very little doubt that the Super Meat Boy team would have seriously struggled to make ends meet if their game had have been poorly received.

Retr0gamer

johnny_ultimate wrote: »

Jonathan Blow (Braid), though, has a different opinion. While he also received great reviews and great sales, for him they were very much a mixed blessing. He openly criticises - and TBH I'd agree with him completely on this - the 'surface level' critiques his game received, with many ignoring what he felt were the more personal and important aspects of his game. While lots of people were talking about and playing his game, they weren't engaging with it at all, and it ultimately sent him through a period of frustration and depression as he tried to counter and explain his ideas online when having to face stuff like this.

What? That the game was about the Manhattan Project. I think the problem there was his delivery of that message which was terrible rather than a problem with the reviewers.

blaa85

I think gaming revews are ridiculous. 6/10 is considered a sh!te game, even 7/10. Dredd got a 3 star review from most mags/sites and I though it was a savage film. You couldn't have a rotten tomatoes gaming site (there might be one, I don't know), since nearly every game is rated over 6/10.

richymcdermott

blaa85 wrote: »

I think gaming revews are ridiculous. 6/10 is considered a sh!te game, even 7/10. Dredd got a 3 star review from most mags/sites and I though it was a savage film. You couldn't have a rotten tomatoes gaming site (there might be one, I don't know), since nearly every game is rated over 6/10.

6/10 is considered sh!te i agree but in reality 6 means its passable or decent
7 means good, 8 very good , 9 amazing, 10 masterpiece . I believe there is a misconception when gaming journalist hand out 10s more than movie critics or even music critics because we all no nothing is perfect.

I think games like resi6 even if its a ****e game should not be getting 3s 4s or 2s , that tells me the game is broken and you cant progress in it any further.

I also think games should be valued not by their passed titles in the series but should be reviewed compared to games that come out in the same year and should be valued by its presentation , gameplay, audio and terms on content

I see always the gaming journalist say they dont except bribes which somewhat can be true but when companies start giving you free stuff and ya get to hang with them you can lead to being towards being abit biased even when you are not trying to be.

I only listen to small people who are just starting out or not that well known like helix here and retr0 but also listen to everyone else here.

I much enjoy playing the games that have 50s or 60 /100 games on metacritic

snausages

richymcdermott wrote: »

I think games like resi6 even if its a ****e game should not be getting 3s 4s or 2s , that tells me the game is broken and you cant progress in it any further.

But whether or not a game "works" is something that shouldn't even need to be commended in the first place. It should be a given. Praising a game for being playable is like praising the chef for not giving you salmonella. If the game is "broken" then it should just get a big, fat zero.

johnny_ultimate

I think reviewers (and possibly even gamers themselves) are actually far too forgiving and generous. Was it the founder of metacritics who said that he had to weight game reviews differently as they were inherently more favourable than music, TV or film?

Again, it's the problem with numbers. A review should speak for itself, rather than the number slapped down at the end of it. Merely being playable is not enough. You can watch a Transformers film - it's in focus, looks good from frame to frame etc... But it's an appalling film in pretty much every other respect, and deserving of critical ire. Similarly, if a game is just dull, derivative or even offensive then it deserves critical vitriol. There does seem to be a hesitancy by many major games journalists to break with consensus or attack high-profile games. Partially down to a cozy relationship with publishers and developers: 'exclusives', trade events, previews, review copies etc... Dismiss these entirely, of course, and a publication loses some of its profile, but there are many instances when a publication's viewpoints have clearly been skewed by other factors.

danthefan

johnny_ultimate wrote: »

I think reviewers (and possibly even gamers themselves) are actually far too forgiving and generous. Was it the founder of metacritics who said that he had to weight game reviews differently as they were inherently more favourable than music, TV or film?

That's true. It's very common for a blockbuster film that makes hundreds of millions to get slated by critics. Resi 6 is the first time I can remember such a big game getting such poor reviews.

Reekwind

I don't agree with the central finding of that piece: that "Good reviews sell games. Bad reviews hurt games". Or at least I have issues with the way they reached this conclusion. That's because they ignore the key question: how review scores are formulated

We've touched on this already in this thread. The bizarre scoring system that prevails in most reviews means that it is very, very rare for a AAA title to score poorly. To the degree that those that do (such as DNF or RE6) are worthy of comment, usually in hushed 'I-can't-believe-they-gave-it-less-than-a-nine' tones. The reality is that a big AAA title with a big AAA budget and a big AAA publisher who has big AAA connections will almost invariably get a big 90+% score

That's life, that's the nature of the industry and it's why game review scores are less reliable than their film or literature or music counterparts. Now we could go into the reasons behind this but it should be clear that, for whatever reason, something is badly off here

So we really shouldn't be surprised that good scores equates to high sales. What the article didn't show was whether both the scores and sales were caused by a third factor, ie what influences a review in the first place. In this light the correlation between sales and reviews is just that: a correlation. The experiment with positive/negative perceptions was interesting but not hugely relevant to the question at hand

blaa85 wrote:

I think gaming revews are ridiculous. 6/10 is considered a sh!te game, even 7/10

I remember way back in the day when PC Zone shifted to an actually logical system. Very few games got the 'Classic' award (ie, 90+%) with 70-80% being considered a good solid 'Recommended' and a decent spread of games across the lower grades

I think they had to revise the system (or at least make it less harsh) a few years before the magazine closed because it was causing problems with publishers and the likes. Certainly you couldn't operate such a system in today's environment where Metacritic is king and often built into contracts