Advertisement
If you have a new account but are having problems posting or verifying your account, please email us on hello@boards.ie for help. Thanks :)
Hello all! Please ensure that you are posting a new thread or question in the appropriate forum. The Feedback forum is overwhelmed with questions that are having to be moved elsewhere. If you need help to verify your account contact hello@boards.ie

Research Pilot Sample Size and Chi Square

  • 22-04-2013 8:04pm
    #1
    Black Swan
    Moderators, Category Moderators, Science, Health & Environment Moderators, Society & Culture Moderators Posts: 47,218 CMod ✭✭✭✭


    We will run one or more pilot tests before administering the main study. Isaac and Michael in Handbook in Research and Evaluation have suggested that samples with n's of 10 or more may be useful for preliminary null hypothesis testing based upon pilot study small sample data.

    When running a 2 x 2 chi-square for nominal level data to test goodness of fit and if there may be significance at the p<.05 level to reject the null, we found no significant difference (i.e., supporting the null). Given that there were less than 5 subjects per cell, I suggested to the team that we needed to increase our pilot sample size to ensure that there were a minimum of 5 expected frequencies per cell in order to avoid the chance of committing a type II error (beta); i.e., accepting the null when it is false. Is this too conservative?


Comments

  • #2
    Ostrom
    Registered Users Posts: 3,483 ✭✭✭Ostrom


    Black Swan wrote: »
    We will run one or more pilot tests before administering the main study. Isaac and Michael in Handbook in Research and Evaluation have suggested that samples with n's of 10 or more may be useful for preliminary null hypothesis testing based upon pilot study small sample data.

    When running a 2 x 2 chi-square for nominal level data to test goodness of fit and if there may be significance at the p<.05 level to reject the null, we found no significant difference (i.e., supporting the null). Given that there were less than 5 subjects per cell, I suggested to the team that we needed to increase our pilot sample size to ensure that there were a minimum of 5 expected frequencies per cell in order to avoid the chance of committing a type II error (beta); i.e., accepting the null when it is false. Is this too conservative?

    Hi Blackswan,

    Technically this is all sound, but you're placing an awful lot of stock in the results of the test. The problem with the chi^2, and indeed much inferential work in the social sciences, is that significance is a function not only of variance but of sample size; you could add a constant number of cases to each cell, and the test would inevitably achieve significance, but this tells you little of practical use.

    Why are you starting with this test in particular? The results will establish possible dependence between the variables, but really the only thing of merit you get from a test such as this is a measure of the probability of dependence due to sample error. Since this is a component of sample-population inference, this is much like putting the cart before the horse. If the tables are 2x2, odds ratios might be more useful for establishing a possible effect size.

    Piloting makes an awful lot of sense if you are trying to refine the research design or to check potential reliability issues in the questions - I would be a lot more worried about these, as these are things that cannot be corrected after collection and will render any inferences moot regardless of significance.


  • #3
    scottdawson
    Site Banned Posts: 4 ✭scottdawson


    Blackswan

    Have you ran a power analysis?

    Scott


  • #4
    Black Swan
    Moderators, Category Moderators, Science, Health & Environment Moderators, Society & Culture Moderators Posts: 47,218 CMod ✭✭✭✭Black Swan


    efla wrote: »
    Technically this is all sound, but you're placing an awful lot of stock in the results of the test.
    Chi-square was not the only statistical test we have been running on the data from our pilot studies. It was the only one that we questioned, having confidence in the others. It pertained only to those few either/or questions that produced only nominal level data.
    efla wrote: »
    ... you could add a constant number of cases to each cell, and the test would inevitably achieve significance, but this tells you little of practical use.
    Rather than artificially adjust the counts by adding "a constant number of cases to each cell," I would rather run another pilot study with a larger n.
    efla wrote: »
    Piloting makes an awful lot of sense if you are trying to refine the research design or to check potential reliability issues in the questions - I would be a lot more worried about these, as these are things that cannot be corrected after collection and will render any inferences moot regardless of significance.
    Agree. We would rather pilot like crazy and preliminarily test the methodology and hypotheses in advance, and make adjustments and revisions, than to stumble into the main study not having done so.
    scottdawson wrote: »

    Have you ran a power analysis?
    Our pilot sample sizes were so small that we felt that a power analysis would not be useful.


  • #5
    Ostrom
    Registered Users Posts: 3,483 ✭✭✭Ostrom


    Black Swan wrote: »
    Chi-square was not the only statistical test we have been running on the data from our pilot studies. It was the only one that we questioned, having confidence in the others. It pertained only to those few either/or questions that produced only nominal level data.

    Rather than artificially adjust the counts by adding "a constant number of cases to each cell," I would rather run another pilot study with a larger n.

    Agree. We would rather pilot like crazy and preliminarily test the methodology and hypotheses in advance, and make adjustments and revisions, than to stumble into the main study not having done so.


    Our pilot sample sizes were so small that we felt that a power analysis would not be useful.

    Sorry, I was a bit unclear; I wasn't suggesting you add a constant, just illustrating that adding more respondents with the same distribution across categories brings you from nothing to significance. That's why I mentioned odds ratios - it is more important to get a sense of the underlying effect size, rather than potential significance which (in my perhaps bigoted opinion) should always be a lesser supporting piece of information.

    I'm thinking along the lines of Ziliak and McClosky url]http://www.deirdremccloskey.com/articles/stats/preface_ziliak.php[/url

    You could also apply Yate's correction if you were stuck with the small cell counts.


Advertisement