1. What can you do with survey methods that you cannot do with other types of research?

The main goal of survey research is to produce an accurate picture of the population from which the sample is drawn. P 171-172

2. What are the main strengths and weaknesses of survey methods?

Survey research methods are well suited for the task of using a sample to learn about a large population. Surveys are expensive and time-consuming. P 171 Where experiments are used to determine causal relationships between variables, surveys produce correlational results. While they may indicate that a pair of variables are strongly related to one another, they ordinarily are not able to determine cause-and-effect relationships. P 172

3. Compare and contrast Likert scales and semantic differential scales in terms of how they are structured, the situations in which they are most appropriate and useful, and their strengths and weaknesses.

See pages 177, 178, and part of 179.

4. Compare and contrast Likert scales and short-answer essay questions in terms of how they are structured, the situations in which they are most appropriate and useful, and their strengths and weaknesses.

Likert scales (named after Rensis Likert who developed them) are used for measurement of attitudes and opinions. A Likert scale contains several items, each of which consists of a statement and a request for the respondent to indicate whether they strongly agree, agree, disagree, or strongly disagree with the statement. Often a "neutral" option is included between "agree" and "disagree." The text uses a scale designed to measure opinions about Netscape and Microsoft Internet Explorer for an example. Likert items are ordinally scaled. It is not assumed that the difference between "strongly agree" and "agree" is the same size as the difference between "agree" and "neutral." However, the fact that the item scores are added together to calculate the score for the scale means that the item scores are treated as if they were interval scaled. For analytic purposes, the ordinal level data from Likert items is treated as if it were interval data. Notice that the questions were stated so that a person who felt strongly that one program was better than the other would have to choose "strongly agree" for some items and "strongly disagree" for others.

When the list of possible answers is long, or when you do not know what the possible answers might be, the multiple-choice format is either impractical or impossible, so you will need to use an open-ended approach. You may want to ask some questions that will require longer answers. You are probably familiar with the "short-answer essay" questions often used on exams. A benefit of questions like these is that they allow your respondent to provide longer, more complex answers than are possible with other formats. The main disadvantage of this approach is that you must study and analyze the longer, more complex answers before you can do anything with them. You will probably want to read all the answers your respondents provide, taking note of the range of different kinds of answers and making lists of specific words or phrases that you see. Then you would develop a coding system that would allow you to reduce the large set of long answers into a much smaller set of categories or themes. Finally, you would go through the original answers and code each one according to the system you developed. What you have effectively done is transformed the open-ended question with long, complex answers into a multiple-choice question for which your respondents (indirectly) provided the list of possible answers.

5. Discuss the strengths and weaknesses of ranking questions.

These questions are more difficult to answer than rating questions because each of the options must be compared to all of the others before a ranking can be made. It is likely that the answers your respondents provide to these questions are less reliable than the answers would be if a rating format was used instead of a ranking format. P 176.

6. When are multiple-choice questions especially useful?

Many survey questions can be answered by providing a number or a word or short phrase. If the list of all the answers that your respondents might give for a question is short (no more than ten or twelve possibilities), you might set the question up as a multiple-choice item.

7. Design a five-item Likert scale that measures the extent to which people are afraid of mathematics.

You have to do this yourself.

8. Design an eight-item semantic differential questionnaire that measures students' opinion of their instructor's competence as a teacher.

You have to do this yourself.

9. Discuss the relative strengths and weaknesses of face-to-face interviews, telephone surveys, and self-administered questionnaires.

The distribution method that involves the smallest amount of personal contact with your respondents is to mail them. This is also the least expensive way. The lack of personal contact may mean that confidentiality is less of a problem than for other methods. This and the low cost may be the only advantages of using this method. The main disadvantage is that you are likely to get a lower response rate with mailed questionnaires than with most other methods of distribution. This is likely due to one of several factors. A mailed questionnaire is often seen as an unwanted intrusion on the recipient's time and effort. Since there is no personal contact with the researcher, it is easy for the recipient to discard the questionnaire. In many instances, the recipient may misplace the questionnaire and find it several days or weeks later, only to conclude that it is now too late to bother. There are a number of other problems that make this method even less attractive. One is that you don't know who is actually filling out the questionnaire. It could be the intended person, a roommate, a spouse, a secretary or someone else. Another problem is that your respondents may take the questionnaire less seriously than they would in a situation where they had some personal contact with you. They may be less likely to answer all questions, especially if the questions take more than a small amount of effort to answer.

Face-to-face interviews are the most expensive, in terms of both time and money (you must travel to respondents' location), and the most intrusive of all the methods. If your sample is going to be more than a few dozen people, you will probably want to hire some additional interviewers. Before you send them out, you'll have to train them and run through some rehearsals with them so you can be sure they understand the procedures you want them to use. A major strength of the face-to-face approach is the ability to deal with complex topics. Because you can see how the respondent reacts to the questions as you ask them, you will have a better idea of how well they understood the questions and how they felt about answering them. You will be more likely to find out that some questions are confusing or don't make sense and you'll have an opportunity to discuss them with your respondents to see how they might be improved. In some circumstances the face-to-face method can be problematic because the respondent's identity is known and due to the lack of anonymity.

Telephone surveys solve many of the problems of the face-to-face approach. Because you don't have to travel, it takes a lot less time to administer the questionnaire to people who are scattered around a city and its suburbs or around a continent. This also means that it is easier to get a representative sample. It may be easier to deal with sensitive issues over the telephone than in a face-to-face setting, since your respondents are more anonymous when you can't see their faces and when you're not sitting in their living rooms or kitchens. There are some drawbacks. With telephone surveys, it is difficult to keep the respondents on the line for long questionnaires. The interviews must be done when respondents are home. Because you are interrupting them at home, you may get more refusals to participate and more people who end the conversation before answering all of your questions. Compared to the face-to-face approach, telephone interviewers are less expensive and easier to train.

10. What factors would be the biggest threats to validity with telephone surveys?

With telephone surveys, it is difficult to keep the respondents on the line for long questionnaires. The interviews must be done when respondents are home. Because you are interrupting them at home, you may get more refusals to participate and more people who end the conversation before answering all of your questions.

11. What factors would be the biggest threats to validity with self-administered questionnaires?

The main disadvantage is that you are likely to get a lower response rate with mailed questionnaires than with most other methods of distribution. This is likely due to one of several factors. A mailed questionnaire is often seen as an unwanted intrusion on the recipient's time and effort. Since there is no personal contact with the researcher, it is easy for the recipient to discard the questionnaire. In many instances, the recipient may misplace the questionnaire and find it several days or weeks later, only to conclude that it is now too late to bother.
There are a number of other problems that make this method even less attractive. One is that you don't know who is actually filling out the questionnaire. It could be the intended person, a roommate, a spouse, a secretary or someone else. Another problem is that your respondents may take the questionnaire less seriously than they would in a situation where they had some personal contact with you. They may be less likely to answer all questions, especially if the questions take more than a small amount of effort to answer.

12. What factors would be the biggest threats to validity with face-to-face interviews?

In some circumstances the face-to-face method can be problematic because the respondent's identity is known and due to the lack of anonymity. Face-to-face interviews are the most intrusive of all the methods. Also, if your sample is going to be more than a few dozen people, you will probably want to hire some additional interviewers. How will you know what they actually did while they were out interviewing? This may be a problem especially if you have to rely on relatively inexperienced unskilled interviewers.

13. What is factor analysis, when would you use it, and what do you do with it?

It is a procedure used to analyse semantic differential scales. This procedure identifies the underlying dimensions or "factors" along which responses to the word-pairs vary. Each word-pair has a "loading" on each factor. These loadings are like correlations; they tell the extent to which the word-pairs are associated with the factors. A word-pair having a high loading on one factor will have low loadings on all the other factors. Word-pairs that don't have any high loadings are usually ignored. You have to examine the factors to determine which word-pairs have high loadings on them in order to determine what dimensions the factors are measuring.

14. What is the best way to determine whether your respondents' interpretations of the questions in your survey are in agreement with one another and with yours? What would you expect to see if they weren't?

You should examine all of the questions in your survey and determine whether some of them are left unanswered more often than others. If some questions are frequently not responded to, you should try to determine whether there are certain types of people in your sample who don't answer them. If you are able to identify a number of problematic survey questions, it is a good idea to try to talk to a few of the people who didn't answer them to see what the problem was. It is important for you to remind yourself that the questions in your survey that make perfect sense to you will probably be somewhat confusing, completely nonsensical, or even insulting to some of your respondents. The reason this happens is that the experiences, families, and employment situations of your respondents are probably very different from yours. You may be amazed at the way some of your "perfectly clear and obvious" questions are completely misunderstood and interpreted in ways that you would never imagine. A thorough pre-test of your questionnaire will do a lot to reduce the severity of these kinds of problems.

15. What is wrong with the following survey questions?

a. If you went to a record store and saw an expensive boxed CD set that you have wanted for a long time but haven't purchased because of the cost, would you steal it?

Yes ___ No ___

Respondents may be unwilling to answer sensitive questions - questions about things that they feel are personal or questions about things they are not comfortable talking about. They may feel embarrassed about the answers they would have to give if they were being truthful, so they may simply refuse to answer.

b. Where do you get most of your information about political events, economic issues, and investment opportunities?

Newspaper ___ Television ___ Radio ___

This is asking about three kinds of information and allowing only one answer. What if you used newspaper for economic issues, television for political events, and radio for investment opportunities? How would you answer the question? How would the researcher know which source you used for each type of information?

c. How old were you the first time you rode a bicycle? ____

You probably can't remember this with any degree of accuracy. Your respondents may be unable to answer some questions because they don't have the required information. They may not know the answer, so they just make one up.

d. Do you agree or disagree with the new defense policy that says the government shouldn't get involved with military or political issues that aren't affected by the national interest?

Agree ___ Disagree ___

This question is confusing with all of the negatives - disagree, shouldn't, aren't. What does the new defense policy say? Can you understand it? I'm not sure I can. Your respondents may be unable or unwilling to answer questions they don't understand or questions that are poorly constructed, obscure, vague, or that don't make sense.

e. How often do you pick your nose?

Never ___ Once a week ___ Every day ___

Personal, embarrassing, not something upstanding respectable people are supposed to do. Will people tell the truth? Respondents may be unwilling to answer sensitive questions - questions about things that they feel are personal or questions about things they are not comfortable talking about. They may feel embarrassed about the answers they would have to give if they were being truthful, so they may simply refuse to answer.

f. How often do you talk to your best friend?

Sometimes ___ Once a week ___ Every day ___ Whenever I can ___ Often ___

How often is sometimes? Is that more or less than once a week? How often is whenever I can? Is that more or less than the other possible answers? This is a mixture of nominal, ordinal, and ratio scaling. The answers are not clear and they are not mutually exclusive.