Questionnaire question

parc

Is it possible to combine a Likert-scale (5 point) and a multiple choice question in the same questionnaire.

For instance:

This is subject is boring "disagree

---

agree"

and

I have studied this subject this before "Yes, No"



Thanks

Ostrom

parc wrote: »

Is it possible to combine a Likert-scale (5 point) and a multiple choice question in the same questionnaire.

For instance:

This is subject is boring "disagree

---

agree"

and

I have studied this subject this before "Yes, No"



Thanks

In the same questionnaire form? Absolutely - you can put as many questions as are relevant/necessary/unlikely to bore the respondent.

The only problem you might have is building a composite scale from it - i.e. if you are trying to measure something like 'engagement' with multiple questions, then it is generally easier to use the same measurement scale for each individual question. Think also about whether you could use something at a higher level of measurement - do you just want to filter the sample on whether they studied the subject or not, or would it be useful to know how recently they studied it, or if they have any similar experience?

I'm complicating it however, short answer - yes, it is fine.

parc

Thanks a million. Should I group these different response together? Like have

Likert scale (maybe 6 questions)

Multiple choice (say 4 questions)

Also one more question about sampling. I want to get the perceptions of a population. They population is basically the whole city. This is obviously problematic. Should I restrict this down to one particular part of the city? Maybe even a particular street? (Bearing in mind it's an undergrad research project)

If I do this, how do I make my sampling frame? Is the sample frame the street? Thanks

Black Swan

parc wrote: »

Should I group these different response together? Like have

Likert scale (maybe 6 questions)

Multiple choice (say 4 questions)

Yes, otherwise you may cause unnecessary confusion and error.

parc wrote: »

Also one more question about sampling. I want to get the perceptions of a population. They population is basically the whole city. This is obviously problematic. Should I restrict this down to one particular part of the city? Maybe even a particular street? (Bearing in mind it's an undergrad research project)

These are population parameters. If you wish to make estimates and draw conclusions about the city, you will need to sample the city. If you only sample a street, you can only make estimates about the street you sampled (i.e., the street is the population).

There are sample validation techniques where a street may represent a city, but they are complex and probably go beyond the scope of your project. Such techniques are sometimes challenged too by reviewers; e.g., ecological fallacy, or reasoning from one unit of analysis to another, etc.

parc

Ah yes ecological fallacy. I guess it would be better to play it safe and say something along the lines of:

"I am sampling this street because it is in an area that is affected by what I'm researching."

Would it would be ok to say in the conclusion that further study could be directed at wider areas or even the city, time and resources permitting?

ulyssesfcohen

Yes, it is possible - but it does complicate matters, in two respects. One is that using different scale types to assess the same variable adds error variance in the statistical analysis. Second, as a previous poster mentioned, it makes it a little trickier to create a composite scale. One option in response to this is to standardize (z-score) the items before combining them to ensure comparability.

efla wrote: »

In the same questionnaire form? Absolutely - you can put as many questions as are relevant/necessary/unlikely to bore the respondent.

The only problem you might have is building a composite scale from it - i.e. if you are trying to measure something like 'engagement' with multiple questions, then it is generally easier to use the same measurement scale for each individual question. Think also about whether you could use something at a higher level of measurement - do you just want to filter the sample on whether they studied the subject or not, or would it be useful to know how recently they studied it, or if they have any similar experience?

I'm complicating it however, short answer - yes, it is fine.

ulyssesfcohen

Black Swan wrote: »

Yes, otherwise you may cause unnecessary confusion and error.


These are population parameters. If you wish to make estimates and draw conclusions about the city, you will need to sample the city. If you only sample a street, you can only make estimates about the street you sampled (i.e., the street is the population).

There are sample validation techniques where a street may represent a city, but they are complex and probably go beyond the scope of your project. Such techniques are sometimes challenged too by reviewers; e.g., ecological fallacy, or reasoning from one unit of analysis to another, etc.

Following on from Swan's point, there are two approaches. Either you do a purely random sample within the entire city - where each member of population has an equal chance of being selected. Alternatively, you do a stratified sample - in which you segregate sample into homogenous bundles, and randomly sample within each bundle. But you simply cannot focus on a street - this is a conevience sample, and results from your survey will lack external validity.

parc

ulyssesfcohen wrote: »

Following on from Swan's point, there are two approaches. Either you do a purely random sample within the entire city - where each member of population has an equal chance of being selected. Alternatively, you do a stratified sample - in which you segregate sample into homogenous bundles, and randomly sample within each bundle. But you simply cannot focus on a street - this is a conevience sample, and results from your survey will lack external validity.

Cool, I'm aware of the Modifiable Areal Unit Problem (data changes when you change the size of the area you are measuring). That sounds like what you mentioned in your previous post.

So how would you recommend I segregate the city? Would it be ok to use wards/parishes or something like that and sample from within those?

I can't do a postal survey as I don't have the time or the funds (as an undergrad), so it would either have to be administered online or do it by foot on the street. The online method would be tricky as I can't really get everyones emails for one particular area, so I don't realy know how I'd go about sampling like this? Also, I'm not sure what kind of sampling method the street surveys fall into. Any ideas for both of these questions? Thanks for the help so far, it's been really useful

ulyssesfcohen

What is purpose of the survey? Is it commercial, academic... etc.

If we know more about your research objectives, we can advise better

Black Swan

ulyssesfcohen wrote: »

What is purpose of the survey? Is it commercial, academic... etc.

Academic:

parc wrote: »

(Bearing in mind it's an undergrad research project)

Yes, research problem, objectives, or questions, etc., would be useful as ulyssesfcohen asks:

ulyssesfcohen wrote: »

If we know more about your research objectives, we can advise better

Ostrom

parc wrote: »

Thanks a million. Should I group these different response together? Like have

Likert scale (maybe 6 questions)

Multiple choice (say 4 questions)

Also one more question about sampling. I want to get the perceptions of a population. They population is basically the whole city. This is obviously problematic. Should I restrict this down to one particular part of the city? Maybe even a particular street? (Bearing in mind it's an undergrad research project)

If I do this, how do I make my sampling frame? Is the sample frame the street? Thanks

Ideally you would like a truly representative sample - this is a base assumption of many common inference tests, and is only possible in instances where a comprehensive sampling frame is available. In practice, these rules are constantly broken, and a significant proportion of published research reports inferential procedures performed on non-probability sampled data. Think carefully about your population, and your research questions - does your analysis concern issues specific to urban residents? Are there key axes of comparison that might explain variation in your outcome (i..e. gender, class)? Are you addressing a general phenomenon independent of locality (i.e. 'the formation of political values')? All of these considerations will determine the suitability of your sampling procedure. You can account for your own limitations by addressing these issues in your write-up.

Since this is an undergraduate thesis, don't kill yourself. You do not have the resources to extract a truly random, representative sample, but you can 'engineer' a number of ad-hoc procedures that will ensure minimized bias. You might examine SAPS census data for example, and introduce a quota into your sampling technique which matches the population distributions of gender, age, race, and class (which would involve you ensuring a certain number of respondents are interviewed matching these characteristics). Technically this is a non-probability approach, but it minimizes bias substantially more than simply adding more respondents from within a limited area (i.e. a single street).

It is more important that you are able to discuss the implications of competing sampling procedures in your thesis - that you are aware of the distinctions between techniques, that you understand the procedures involved in achieving representation (and their statistical importance). You have more direct control over the reliability of your measures, your research design, and the logical development of your analysis - i.e. the extent to which your approach addresses existing limitations in knowledge, and to which you employ existing theory in order to explain observed relationships between key variables.

bertusik

To summarize much of what has been said, the key issue in any research design is that of validity - whether the inferences you make from a study are defensible. For survey research, this is usually twofold. First, are you measuring what you say you are measuring? In other words do your measures adequately tap the content domain of the topic, variable, or issue you are studying. And second, how much are your inferences generalizable? This depends on the representativeness of your sample.

My advice - take the path of least resistance; rely on established measurement scales, and do a random sample (or as close to that as possible).

efla wrote: »

Ideally you would like a truly representative sample - this is a base assumption of many common inference tests, and is only possible in instances where a comprehensive sampling frame is available. In practice, these rules are constantly broken, and a significant proportion of published research reports inferential procedures performed on non-probability sampled data. Think carefully about your population, and your research questions - does your analysis concern issues specific to urban residents? Are there key axes of comparison that might explain variation in your outcome (i..e. gender, class)? Are you addressing a general phenomenon independent of locality (i.e. 'the formation of political values')? All of these considerations will determine the suitability of your sampling procedure. You can account for your own limitations by addressing these issues in your write-up.

Since this is an undergraduate thesis, don't kill yourself. You do not have the resources to extract a truly random, representative sample, but you can 'engineer' a number of ad-hoc procedures that will ensure minimized bias. You might examine SAPS census data for example, and introduce a quota into your sampling technique which matches the population distributions of gender, age, race, and class (which would involve you ensuring a certain number of respondents are interviewed matching these characteristics). Technically this is a non-probability approach, but it minimizes bias substantially more than simply adding more respondents from within a limited area (i.e. a single street).

It is more important that you are able to discuss the implications of competing sampling procedures in your thesis - that you are aware of the distinctions between techniques, that you understand the procedures involved in achieving representation (and their statistical importance). You have more direct control over the reliability of your measures, your research design, and the logical development of your analysis - i.e. the extent to which your approach addresses existing limitations in knowledge, and to which you employ existing theory in order to explain observed relationships between key variables.

parc

ulyssesfcohen wrote: »

What is purpose of the survey? Is it commercial, academic... etc.

If we know more about your research objectives, we can advise better

It's an undergraduate dissertation, so academic.

I just want to get perceptions and attitudes from a city. However, the topic is almost only a problem in the poorer communities. So I'm mostly looking for people who live in the poorer areas (although I am also targeting the more affluent areas to get their perspective)

parc

efla wrote: »

Ideally you would like a truly representative sample - this is a base assumption of many common inference tests, and is only possible in instances where a comprehensive sampling frame is available. In practice, these rules are constantly broken, and a significant proportion of published research reports inferential procedures performed on non-probability sampled data. Think carefully about your population, and your research questions - does your analysis concern issues specific to urban residents? Are there key axes of comparison that might explain variation in your outcome (i..e. gender, class)? Are you addressing a general phenomenon independent of locality (i.e. 'the formation of political values')? All of these considerations will determine the suitability of your sampling procedure. You can account for your own limitations by addressing these issues in your write-up.

Since this is an undergraduate thesis, don't kill yourself. You do not have the resources to extract a truly random, representative sample, but you can 'engineer' a number of ad-hoc procedures that will ensure minimized bias. You might examine SAPS census data for example, and introduce a quota into your sampling technique which matches the population distributions of gender, age, race, and class (which would involve you ensuring a certain number of respondents are interviewed matching these characteristics). Technically this is a non-probability approach, but it minimizes bias substantially more than simply adding more respondents from within a limited area (i.e. a single street).

It is more important that you are able to discuss the implications of competing sampling procedures in your thesis - that you are aware of the distinctions between techniques, that you understand the procedures involved in achieving representation (and their statistical importance). You have more direct control over the reliability of your measures, your research design, and the logical development of your analysis - i.e. the extent to which your approach addresses existing limitations in knowledge, and to which you employ existing theory in order to explain observed relationships between key variables.

Thanks that's some really really good information, thanks!

To answer your questions:

"Think carefully about your population, and your research questions - does your analysis concern issues specific to urban residents? Are there key axes of comparison that might explain variation in your outcome (i..e. gender, class)? Are you addressing a general phenomenon independent of locality (i.e. 'the formation of political values')?"

Basically I'm examining local perceptions to a phenomena that nearly only affects those in poor communities.