Evaluate sampling methods and sources of bias

Detailed Explanation: Evaluate sampling methods and sources of bias

To evaluate a sampling method, ask whether the sample gives every member of the target population a fair opportunity to be selected and whether the results can reasonably represent that population.

Use this process:

1. Identify the target population.
   Who do we want to describe?

2. Identify the actual sample.
   Who was included in the data?

3. Name the sampling method.
   Is it random, stratified, systematic, cluster, convenience, or voluntary-response?

4. Look for bias.
   Check for:

   • Undercoverage: some groups have little or no chance to be selected.
   • Nonresponse: selected people do not provide data.
   • Voluntary-response bias: people choose themselves to respond, often because they have strong opinions.
   • Response or wording bias: questions or interview conditions influence answers.

5. Decide what conclusion is justified.
   A biased sample may not represent the target population, even if it is large.

Worked example

A high school principal wants to estimate the percentage of all 1,200 students who support starting school later. The principal posts a survey link on the school website. The first 300 students who choose to complete it are included. Of these students, $(80\%)$ support a later start time.

Step 1: Identify the target population.

The target population is all $(1{,}200)$ students in the school.

Step 2: Identify the sample.

The sample is the 300 students who chose to complete the online survey.

Step 3: Identify the method.

This is a voluntary-response sample because students decide for themselves whether to respond. It may also have undercoverage if some students do not regularly visit the school website.

Step 4: Look for bias.

Students with strong opinions about the start time may be more likely to complete the survey. For example, students who strongly want a later start might respond more often than students who do not care. This creates voluntary-response bias.

Students who see the survey but do not respond create nonresponse, although the main problem here is that the original sample was self-selected.

Step 5: Evaluate the conclusion.

The result $(80\%)$ describes the 300 respondents, but it may not describe all $(1{,}200)$ students. It would be inappropriate to conclude that exactly $(80\%)$ of all students support a later start time.

A better design would be to select students randomly, or use a stratified random sample by grade level so that each grade is fairly represented.

Increasing the sample from 300 to 600 would reduce random sampling variability only if the sampling method were reasonably fair. A larger voluntary-response sample could still be biased; more data does not fix a biased design.