How to run an experiment?

cavedave

A pub in town serves four stouts. I want to test if people can differentiate them.
So I will serve a glass of Guinness, Beamish,Murphys and O'Haras to each person and ask them to write down what each one is. So my questions are
1. What is the best way to run this
2. How do you tell how successful people are. As in what is the expected correct rate if people choose randomly etc.

For 1 my guess is. Get glass of each put each on a beer mat with a number 1->4. For each number write down what stout it is and who it was served to. Experimentor B (the server) collects this tray and serves it to someone else. The server then takes down what stout the person thinks is which. We do this for all N people and then compare what people got with what they thought they got. In this way the server does not know what stout is which and cannot then unknowingly tip the drinker off.

So any suggestions improvements etc for this test?

2Scoops

Each subject should be familiar with the taste of all 4 stouts before doing the test, otherwise it's meaningless.

Randomize the order of drinks in each test.

Chi-square for seeing if there's a difference.

cavedave

2Scoops

Each subject should be familiar with the taste of all 4 stouts before doing the test, otherwise it's meaningless.

Thanks for the reply. The subjects (well my mates) will have tasted all the beers before.

Randomize the order of drinks in each test.

Chi-square for seeing if there's a difference.

Great that was the test I was looking for. Now I need to figure out the expected values. Expected with the hypothesis that people cannot tell the difference between different stouts.

2Scoops

cavedave wrote: »

Great that was the test I was looking for. Now I need to figure out the expected values. Expected with the hypothesis that people cannot tell the difference between different stouts.

The expected value for a correct guess, assuming they couldn't tell the difference, would be 25% for each drink, since they would know it has to be one of the four.

I think you'll have to do 4 separate chi-square tests; one [correct-25%/incorrect-75%] for each drink, because it can't be used to test related-measures designs. There's probably a better way to do it tbh, to compare all answers together but I can't think of it right now. The maths forum might be able to help...

The tests should be secondary to simple observed trends, anyway. A mathematical icing on the cake, if you will. In fact, without a decent sample of friends (at least 25, I'd say), any chi square test result will be basically worthless. :eek:

2Scoops

Here's another thought: maybe literally blindfold the subjects as well! Some of these pints look different colour-wise, especially in the head, and that would tip some experienced drinkers off straight away.

Make sure they're served in the same shape glasses too. And is that Guinness 'extra-cold' stuff actually colder than the others as well? Not sure how you would combat that problem, bar waiting for all pints to settle to the same temp...

cavedave

2Scoops Here's another thought: maybe literally blindfold the subjects as well! Some of these pints look different colour-wise, especially in the head, and that would tip some experienced drinkers off straight away.

Make sure they're served in the same shape glasses too. And is that Guinness 'extra-cold' stuff actually colder than the others as well? Not sure how you would combat that problem, bar waiting for all pints to settle to the same temp...

If pints look different or of a different temperature and this tips people off such is life. Maybe I should rephrase the hypothesis as "People cannot tell the difference between stouts" and leave having to use taste out of it. Yes everything will have to be in the same type glass alright.

The expected value for a correct guess, assuming they couldn't tell the difference, would be 25% for each drink, since they would know it has to be one of the four.

That sounds about right but I might check the maths forum in case I am missing something.

Thanks again for the help.

MathsManiac

If you treat each type separately (right/wrong 25%/75%) then it won't be legitimate to combine the results afterwards, as the results won't be independant of each other. You'd really need then to use a separate series of trials for each type.

I can help you with one aspect, if you don't want to treat each type separately:
Q: What is the probability of an individual getting X right in a single trial purely by guessing?

Guess all 4 right: 1/24
Guess 3 right: 0 (impossible to get three right and one wrong!)
Guess 2 right: 6/24 = 1/4
Guess 1 right: 8/24 = 1/3
Guess 0 right: 9/24 = 3/8.

This could give you a set of expected frequencies for a chi-squared test, but it's a bit out my field so I'm not sure whether the appropriate conditions to make the test legitimate are satisfied.

2Scoops

MathsManiac wrote: »

Guess all 4 right: 1/24
Guess 3 right: 0 (impossible to get three right and one wrong!)
Guess 2 right: 6/24 = 1/4
Guess 1 right: 8/24 = 1/3
Guess 0 right: 9/24 = 3/8.

This could give you a set of expected frequencies for a chi-squared test, but it's a bit out my field so I'm not sure whether the appropriate conditions to make the test legitimate are satisfied.

I'm pretty sure that would be a legitimate set up for a chi square test but because the frequency for getting all 4 correct is quite rare, you would need a fairly impractical sample size for the test results to be meaningful (125+ probably).

cavedave

Thanks for the advice. I am going to try asking this question in the Mathematics section in case anyone over there has the full read on hypothesis testing and such

r3nu4l

Just make sure that the volunteers only taste one beer each and have not had strong flavoured/salty/sugary food in the previous 3-4 hours.

Ask them to drink some water first but make sure they each drink a similar volume.

Irish Halo

Also would help if it was a double blind (i.e. even you don't know which beers you are serving them) and run on more than one occasion with obviously the possibility that they get the same beer again

cavedave

so 9 people tried all four beers. They got 12 correct. At random you would expect 9 to be correct. So I now need to do the sums to see how likely you are to get 12 instead of nine with random picking. If this probability is low enough we can say people can tell which stout is which (barely)

r3nu4l

cheers cavedave! Can you post up the protocol you used? Just to give us an idea of how rigorous it was

cavedave

Someone went up and ordered Stout A. Brought down Stout A for whoever had newly arrived. Then they went up and got B, C ,D in order. A,B,C,D were randomised between the three different goes at this procedure *as in there were three different sets of people who arrived at different times. Everyone used the same type of glass.

The testers could wait till all four were in front of them or just taste them as they arrived. They would write their guessed beer on a sheet. After tester was finished the sheets were handed over.

Then when everyone had done the tests the results were calculated and read out.

Just in case your a stout drinker the results were interesting
Person|Beamish|Guinness|Murphy’s|O’Hara’s|Score
A|Beamish|Guinness|Murphy’s|O’Hara’s|4
B|Guinness|Beamish|Murphy’s|O’Hara’s|2
C|O’Hara’s|Guinness|Beamish|Murphy’s|1
D|Guinness|O’Hara’s|Murphy’s|Beamish|1
E|Beamish|Murphy’s|O’Hara’s|Guinness|1
F|Guinness|Murphy’s|O’Hara’s|Beamish|0
G|Beamish|Guinness|O’Hara’s|Murphy’s|2
H|Guinness|Murphy’s|Beamish|O’Hara’s|1
I|Guinness|Beamish|O’Hara’s|Murphy’s|0

Beamish and Guinness seem to be indistinguishable.

cavedave

I wrote up the experiment here. Any other beer related experiments you can think of?

Nothingcompares

what was the volume of the sample given to each person? a sup? a sip? a pint?

Jimoslimos

cavedave wrote: »

A pub in town serves four stouts. I want to test if people can differentiate them.

Mmm tasty experiment!, although one comment to make - you seem to be testing identification rather than differentiation.

so for example instead of serving 4 samples of stout, serve two at random (i.e Beamish-Guinness, Guinness-Murphys, Murphys-Murphys, etc.,) then ask whether the taster can tell any difference. This would also allow you to test your hypothesis that Beamish and Guinness are indistinguishable.

cavedave

othingcompares what was the volume of the sample given to each person? a sup? a sip? a pint?

a glass (1/2 pint)

you seem to be testing identification rather than differentiation.

Yes that is what we were testing. No we did not run a triangle test. Indistinguishable might be the wrong word. I'd say you could tell they were different but maybe not know which one was which.

a5y

One complication to this test: How much is drank.

Ignoring for a moment that alcohol can impair judgement, if the subjects are only drinking a sip of each of the beers then it means the test only measures how good they are identifying them based off of a sip of each.

If I remember correctly (I think I read this in either Freakonomics or Blink) when Pepsi started conducting blind tastes the results came back massively in favour of Pepsi over Cocacola, resulting in much panicking and the creation of New Coke.

To cut to the chase, on the sip test , the sweeter the cola the better, but ultimately most customers didn't want to drink a whole bottle, as reflected by the face that there wasn't an en-mass conversion of Coke customers to Pepsi as a result of the taste tests.

And this may mean if your friends don't drink the beers in the unit size they usually drink them in they might not be able to tell the difference. Which goes back to the first problem: alcohol impairs judgement.

(Wish I could suggest some way of helping rather than just pointing out a possible problem! Sorry)

cavedave

One complication to this test: How much is drank.

You cannot taste anything after a pint of stout. It is a fair pint the taste profile head etc is likely to change as you go through a pint

To cut to the chase, on the sip test , the sweeter the cola the better, but ultimately most customers didn't want to drink a whole bottle

Interesting point still that relates to preference rather then identification.