Qualitative Research: What Size Sample?

Black Swan

Selecting the appropriate number of subjects for a qualitative research study has been a controversial topic. Unlike quantitative research where mathematically calculated sample sizes can be estimated from statistical probabilities, in qualitative research the guiding principle was establishing saturation, or the extent to which the sample subjects represent all the qualitative differences in the population being studied. In some cases, researchers may continue to add subjects to the sample until the last missing qualitative difference had at least one representative.

Mason (2010) examined 2,533 qualitative PhD studies to establish how many subjects may be needed in qualitative research. Mason found the average sample size to be 31, with the most common sample sizes that were either 20 or 30 to establish saturation.

Establishing saturation appears to be a vague, and sometimes arbitrary concept to me. Further, why not use a quantitative, probabilistic method of sample selection to increase the likelihood of statistical representativeness, before employing qualitative methods to examine the research problem?

Your thoughts?

Reference:
Mason, M. (2010). Sample size and saturation in PhD studies using qualitative interviews. Qualitative Social Research 11(3).

puddles and umbrellas

Black Swan wrote: »

Mason (2010) examined 2,533 qualitative PhD studies to establish how many subjects may be needed in qualitative research. Mason found the average sample size to be 31, with the most common sample sizes that were either 20 or 30 to establish saturation.

I'm a true quantitative researcher , but I have worked as a qual R.A. in the past. I actually spoke about this topic before with a researcher who did her (qual) PhD with 32 participants. I'm not quite sure if her work was based solely on interviews, or if she used other research methods such as focus-groups, etc. - but she told me that the sample size for her PhD always was a problem in interviews, whereby potential employers exclaimed that even for a qual study, her sample was far too small! :rolleyes:

Now, I would have been under the impression that her thesis took a damn lot of work and required a very high amount of detail, so I don't know why this N was a concern for others. Indeed, it is quite a subjective figure. She said she always had a script prepared to justify the sample size - in that for what she was looking at, the detail garnered from 32 people provided her with more than enough info.


However, I do like your idea about using a quantitative method to determine an exact #.
There are a lot of other things that need to be considered for qual research though - in terms of what exactly you are looking at, what is the context, what are your methods of gaining this info, repeated conditions etc. So how would you enforce a general quantitative method which could used for all qual studies would be an interesting endeavour indeed

Black Swan

All too frequently when reading peer-reviewed and published qualitative studies this issue of what constitutes an appropriate sample size occurs. For example, in an action research study conducted by Torrance and Pryor (2001) that pertained to the integration of formative assessment within the classroom curriculum as an ongoing process, they used an original sample size of n=11 teachers as subjects.

How did they arrive by this sample size? They circulated flyers to several K-12 schools throughout a county and only 11 teachers replied that they were interested in participating in the study. Size had been determined by self-selection, not though some valid and reliable method, be it qualitative or quantitative. Further, to what extent was there bias in this small convenience sample of self-selecting subjects?

What was worse, through study attrition overtime 6 teachers left, leaving only 5 teachers to complete the study ending with the adjusted sample size of n=5.

The researcher's attempts to justify the representativeness of the remaining 5 teachers that completed the study, much less the nonprobabilistic self-selecting nature of the original 11 study subjects, was highly questionable.

Could this study have benefited from a quantitative method of determining sample size before implementing their qualitative action research? One approach would have been to obtain a list of the total number of teachers in the county K-12 schools (N), set the confidence level (e.g., 95%), and then calculate the sample size (n) that would be statistically representative of the population (followed by a random selection of teachers from the population).

Your thoughts?

Reference:
Torrance, H. & Pryor, J. (2001). Developing formative assessment in the classroom: Using action research to explore and modify theory. British Educational Research Journal 27(5), 615-631.

Ostrom

The difference is that in qualitative studies, population inference is typically not a core objective. Often it can deal with issues and mechanisms which transcend specific contexts, and in such cases, conventional sample selection criteria are unnecessary. Grounded theory approaches, for example, usually approach sample size in reverse - by suggesting that further subjects be added until no additional information or further relevant subjects/themes present in the data. If the purpose of the above paper is simply to inform theory - building, then an approach such as this seems ideal; it allows for inductive saturation, and assumes that the process of integration operates more or less similarly across contexts (i.e. it addresses specifics of interaction which present independent of specific cases). My own preference would be for some form of stratification / quota setting to control for between-group differences and characteristics, but I feel a probabilistic approach in such settings undermines objectives which are often exploratory. Ideally, it would combine both - a representative survey with exploratory interviews, but individuals often become heavily invested in methodological approaches rather than the requirements of research questions, which is unfortunate.

Black Swan

efla wrote: »

If the purpose of the above paper is simply to inform theory - building, then an approach such as this seems ideal;

The purpose of the paper was to inform both theory and practice, with a heavy emphasis on practice. Consequently the small sample size after attrition (n=5), the loss of more than half the original subjects during the conduct of the study for various reasons (-6), potential bias associated with self-selection, lack of representativeness and the inability to generalise the results to the larger population, made the conclusions and recommendations that pertained to practices (not theory) at the end of the article questionable.

efla wrote: »

Ideally, it would combine both - a representative survey with exploratory interviews

This was one possible solution raised earlier, and may have some merit. Such multimethod approaches (i.e., triangulation; combining quantitative and qualitative methodologies) have been suggested by Brewer and Hunter (1989), Creswell (2003), and Taylor (2000).

I am currently on a project team where we are developing the research proposal and writing the grant for a study that uses multi-methods. Our research design begins with qualitative interviews of key informants, focus groups of participants that resemble subjects (i.e., saturation), and observations of organisational sites, all to inform the construction of the quantitative survey questionnaire. The preliminary quantitative survey (which also has a qualitative "comment" section) is then pilot tested one or more times using small samples, along with feedback to key informants and focus groups for additional revisions to the quantitative survey questionnaire.

Admittedly, we are concerned with the sample size of focus groups in our research design. It is not clear what constitutes the appropriate size for this qualitative method of data collection. Of course we cite sources from the methodological literature (Isaac and Michael, 1997; Stewart and Shamdasani, 1990), and are using from n=10 to 12, but personally I continue struggle with what constitutes sample size for qualitative research.

References:
Brewer, J. and Hunter, A. (1989). Multimethod Research. London: Sage.
Creswell, J.W. (2003). Research Design: Qualitative, Quantitative, and Mixed Methods Approaches, 2nd Ed. Thousand Oaks: Sage.
Isaac, S. and Michael, W. (1997). Handbook in Research and Evaluation, 3rd Ed. San Diego: EdITS.
Stewart, D.W. and Shamdasani, P.N. (1990). Focus Groups. London: Sage.
Taylor, G.R. (2000). Integrating Quantitative and Qualitative Methods in Research. Lanham: University Press of America.