1.2 Data, Sampling, and Variation in Data and Sampling
SLO 1
Assess how data were collected and recognize how data collection affects what conclusions can be drawn from the data.
Here you learn to name the method behind a study — simple random, stratified, cluster, systematic, or convenience — and to sort what was recorded as qualitative, discrete, or continuous. Both are how you judge what a study's numbers can honestly claim.
SLO 6
Evaluate ethical issues in statistical practice.
The Critical Evaluation checklist — self-funded studies, self-selected samples, loaded questions, confounding, a bar chart missing its Other/Unknown category — gives you concrete grounds for calling a study misleading rather than merely wrong.
Learning Objectives
By the end of this section, you will be able to:
- sort data into qualitative, quantitative discrete, and quantitative continuous categories, and give an example of each;
- choose an appropriate display — pie chart, bar graph, or Pareto chart — for a set of qualitative data, and explain what a display hides when categories are omitted;
- describe how a simple random, stratified, cluster, systematic, or convenience sample is drawn, and identify which method a study used;
- explain how sampling bias, sample size, self-selection, funding, and confounding can make a study's conclusions untrustworthy;
- explain why two honest samples from the same population give different results, and why that is expected rather than an error.
Data may come from a population or from a sample. Whichever it comes from, the very first question a statistician asks is what kind of data is this? — because the answer decides what you are allowed to do with it. You cannot average a list of hair colors, and you cannot make a pie chart out of a list of measured weights. Getting the type right up front saves you from an analysis that looks fine and means nothing.
1.2.1 Types of Data
Qualitative data are the result of categorizing or describing attributes of a population. They are also often called categorical data, and they are generally described by words or letters rather than numbers.
You may collect data as numbers and report them categorically. Quiz scores are recorded all term as numbers; at the end of the term they are reported as A, B, C, D, or F. The same underlying reality can be quantitative when gathered and qualitative when published — so always ask which version you are holding.
Hair color, blood type, ethnic group, the car a person drives, and the street a person lives on are all qualitative. Hair color might be black, dark brown, light brown, blonde, gray, or red. Blood type might be AB+, O−, or B+. Researchers often prefer quantitative data because numbers lend themselves more easily to mathematical analysis — it simply does not make sense to find an average hair color or an average blood type.
Quantitative data are always numbers. They are the result of counting or measuring attributes of a population.
Counting gives you whole things — you can have 3 phone calls, never 3.4. Measuring gives you a reading on a scale, and the scale can always be read more finely. "How many?" is discrete; "how much?" is continuous.
Amount of money, pulse rate, weight, the number of people living in your town, and the number of students who take statistics are all quantitative. Quantitative data split further into two types, depending on whether you counted or measured.
All data that are the result of counting are called quantitative discrete data. These data take on only certain numerical values.
If you count the number of phone calls you receive for each day of the week, you might get values such as zero, one, two, or three — never 2.4.
Definition 1.2.3 — Quantitative discrete data: counting lands only on certain values, with gaps in between.
Data that are not only made up of counting numbers, but that may include fractions, decimals, or irrational numbers, are called quantitative continuous data.
Continuous data are often the results of measurements like lengths, weights, or times. A list of the lengths in minutes for all the phone calls you make in a week, with numbers like 2.4, 7.5, or 11.0, would be quantitative continuous data.
Definition 1.2.4 — Quantitative continuous data: measuring can land anywhere on the scale.
Almost all data fall into one of two big families: qualitative and quantitative.
Figure 1.2.1 — Data-types classification tree: qualitative versus quantitative, and quantitative split into discrete and continuous.
The data are the number of machines in a gym. You sample five gyms. One gym has 12 machines, one gym has 15 machines, one gym has ten machines, one gym has 22 machines, and the other gym has 20 machines. What type of data is this?
Solution
Step 1 — Ask what was recorded: For each gym we counted the machines.
Step 2 — Apply the rule: A gym has 12 machines or 13 machines; there is no such thing as 12.5 machines, so only whole numbers can occur.
Answer: This is quantitative discrete data.
The data are the number of books students carry in their backpacks. You sample five students from Avery Bennett's study group. Avery and one classmate each carry three books; a third student carries four books, a fourth carries two, and the fifth carries one. Avery checks their own backpack twice before reporting the count.
What type of data is this?
Solution
Step 1 — Ask what was recorded: For each student we wrote down how many books were in the backpack — Avery counted theirs, and so did everyone else. That is a count.
Step 2 — Apply the rule: Counting gives whole numbers only. A backpack holds 3 books or 4 books, never 3.4 books.
Answer: The numbers of books (three, four, two, and one) are quantitative discrete data.
The data are the areas of lawns in square feet. You sample five houses. The areas of the lawns are 144 sq. feet, 160 sq. feet, 190 sq. feet, 180 sq. feet, and 210 sq. feet. What type of data is this?
Solution
Step 1 — Ask what was recorded: Area is obtained by measuring the lawn, not by counting objects.
Step 2 — Check the values: The listed areas happen to be whole numbers, but a lawn could just as easily measure 160.5 or 160.47 square feet. Landing on whole numbers here is rounding, not a restriction.
Answer: This is quantitative continuous data.
The data are the weights of backpacks with books in them. You sample five students, including Kiran Raval. The weights (in pounds) of their backpacks are 6.2, 7, 6.8, 9.1, 4.3 — Kiran's is the 6.2, and they are surprised it is the lightest of the five.
What type of data is this?
Solution
Step 1 — Ask what was recorded: Each backpack was put on a scale, so each value is a measurement, not a count.
Step 2 — Check the values: Decimals appear (6.2, 6.8, 9.1). With a more precise scale we could have written 6.24 or 6.238 — the values can be refined without limit.
Step 3 — Notice the wrinkle: Two backpacks carrying three books each can still have different weights, because the books themselves differ. Kiran carries three books and so does a classmate, yet their totals are not the same. Weight is not determined by the count.
Answer: Weights are quantitative continuous data.
The following list of materials was purchased by Grant Halloway, a purchase manager in a company. He needs to describe the order to his accountant:
- Two types of nails (2 kg box nails, 3 kg roofing nails)
- One type of oil (4 L machine oil)
- Four types of screws (3 kg wood screws, 5 kg machine screws, 1 kg set screws, 2 kg socket screws)
Name data sets that are quantitative discrete, quantitative continuous, and qualitative.
Solution
Step 1 — Counts (quantitative discrete): The two types of nails, one type of oil, and four types of screws are counts of how many kinds were purchased, so they are quantitative discrete.
Step 2 — Measurements (quantitative continuous): The weights and volume — 2 kg, 3 kg, 4 L, 3 kg, 5 kg, 1 kg, 2 kg — come from a scale or a measuring container, so they are quantitative continuous.
Step 3 — Categories (qualitative): The names — box nails, roofing nails, machine oil, wood screws, machine screws, set screws, socket screws — are qualitative.
Answer: Number of types = discrete; kilograms and liters = continuous; the product names = qualitative.
Jun Takeda and their partner go to the supermarket and purchase three cans of soup (19 ounces tomato bisque, 14.1 ounces lentil, and 19 ounces Italian wedding), two packages of nuts (walnuts and peanuts), four different kinds of vegetable (broccoli, cauliflower, spinach, and carrots), and two desserts (16 ounces pistachio ice cream and 32 ounces chocolate chip cookies).
Name data sets that are quantitative discrete, quantitative continuous, and qualitative.
Solution
Step 1 — Find the counts (quantitative discrete): The three cans of soup, two packages of nuts, four kinds of vegetables, and two desserts are quantitative discrete data, because you count them.
Step 2 — Find the measurements (quantitative continuous): The weights of the soups (19 ounces, 14.1 ounces, 19 ounces) are quantitative continuous data, because they were measured as precisely as the store scale allows.
Step 3 — Find the categories (qualitative): Types of soups, nuts, vegetables, and desserts are qualitative data, because they name a category rather than a quantity.
Answer: Counts of items are discrete; ounce weights are continuous; the names of the foods are qualitative. Try to identify additional data sets in this example — the dessert weights (16 ounces, 32 ounces) are another continuous set.
Sarah Whitfield walks her block and records the colors of five houses. The colors of the houses are white, yellow, white, red, and white. What type of data is this?
Solution
Step 1 — Ask what was recorded: Each observation is a color name.
Step 2 — Apply the rule: Colors sort houses into categories; they are words, not measurements or counts.
Answer: This is qualitative (categorical) data. Notice you can count how many houses are white (three) — that count is a summary of the qualitative data, not the data itself.
The data are the colors of backpacks. Again, you sample five students. Malia Fonoti has a red backpack — she chose it because it was the only one left in the campus store — two students have black backpacks, one student has a green backpack, and one student has a gray backpack.
What type of data is this?
Solution
Step 1 — Ask what was recorded: For each student we wrote down a color name, not a number.
Step 2 — Apply the rule: Colors describe an attribute — they place each backpack in a category. There is no arithmetic to do on them; "the average of red and green" is meaningless.
Answer: The colors red, black, black, green, and gray are qualitative data.
Determine the correct data type (quantitative or qualitative) for the number of cars in a parking lot. Indicate whether quantitative data are continuous or discrete.
Solution
Step 1 — Read the phrasing: "The number of" is the classic signal of a count.
Step 2 — Check the possible values: A lot holds 40 cars or 41 cars, never 40.7.
Answer: The number of cars in a parking lot is quantitative discrete data.
Work collaboratively to determine the correct data type (quantitative or qualitative). Indicate whether quantitative data are continuous or discrete. Hint: Data that are discrete often start with the words "the number of."
a. the number of pairs of shoes you own
b. the type of car you drive
c. the distance it is from your home to the nearest grocery store
d. the number of classes you take per school year
e. the type of calculator you use
f. weights of dogs at an animal shelter
g. number of correct answers on a quiz
h. IQ scores (This may cause some discussion.)
Solution
Step 1 — Spot the counts: Items a, d, and g all begin with "the number of" (or are a count of correct answers). Counts are quantitative discrete.
Step 2 — Spot the measurements: Items c, f, and h come from measuring instruments — a distance, a scale, a standardized test scored on a continuous scale. These are quantitative continuous.
Step 3 — Spot the categories: Items b and e name a make of car and a make of calculator. Those are qualitative, or categorical.
Answer: Items a, d, and g are quantitative discrete; items c, f, and h are quantitative continuous; items b and e are qualitative, or categorical.
The registrar at State University keeps records of the number of credit hours students complete each semester. The data collected are summarized in the histogram in Figure 1.2.2. The class boundaries are 10 to less than 13, 13 to less than 16, 16 to less than 19, 19 to less than 22, and 22 to less than 25.
Figure 1.2.2 — Histogram of the number of credit hours completed per student, grouped into five intervals from 10 to 25.
What type of data does this graph show?
Solution
Step 1 — Identify what is on the horizontal axis: Credit hours completed — a number, so the data are quantitative, not qualitative.
Step 2 — Decide discrete or continuous: Credit hours are counted in whole (or half) units and the intervals group those counts. The values come from counting, so the underlying data are quantitative discrete.
Answer: The histogram shows quantitative discrete data — the number of credit hours each student completed. (The bars are grouped into intervals to make the picture readable; the grouping does not turn the counts into measurements.)
Professor Mateo Delgado collects information about the classification of his statistics students as first-year students, sophomores, juniors, or seniors. The data he collects are summarized in the pie chart in Figure 1.2.3. What type of data does this graph show?
Figure 1.2.3 — Pie chart showing the classification of statistics students as first-year students, sophomores, juniors, and seniors.
Solution
Step 1 — Identify what each wedge stands for: Each wedge is a class standing — first-year, sophomore, junior, senior. Those are category names.
Step 2 — Check what the numbers on the chart are: The percentages describe how many students fall in each category. They summarize the data; they are not the data themselves.
Answer: This pie chart shows the students in each year, which is qualitative (or categorical) data.
Priya Raman is studying every apprentice enrolled in a statewide electrician programme and records, for each one, the number of hours logged before certification. She needs to describe what she collected. Is that variable numerical (quantitative) or categorical (qualitative)? What is the parameter of interest?
Solution
Step 1 — Identify what is recorded on each individual: The characteristic recorded on each apprentice is the number of hours logged before certification. Hours are measured on a continuous scale, so this is a numerical (quantitative) variable.
Step 2 — Identify the parameter: A parameter describes the whole population, not the sample. Here the population is every apprentice in the programme, and the quantity of interest is their mean hours to certification.
Answer: The variable is numerical. The parameter of interest is the mean hours to certification across every apprentice in the programme, written \(\mu\).
1.2.2 Qualitative Data Discussion
In a pie chart, categories of data are represented by wedges in a circle, and the wedges are proportional in size to the percent of individuals in each category.
A pie chart promises that every slice is a piece of one indivisible pie — each individual belongs to exactly one wedge. Overlapping categories break that promise. Bars make no such claim; each bar is just a measurement against the same ruler.
Definition 1.2.5 — A pie chart: categories as wedges of one whole, each sized by its percent.
In a bar graph, the length of the bar for each category is proportional to the number or percent of individuals in that category. Bars may be vertical or horizontal.
Definition 1.2.6 — A bar graph: each category's bar length measured against one common ruler.
A Pareto chart consists of bars that are sorted into order by category size, from largest to smallest.
Definition 1.2.7 — A Pareto chart: the same bars, sorted from largest to smallest.
Look at Figure 1.2.4, Figure 1.2.5, and Figure 1.2.6 and decide for yourself which display — the pies or the bars — makes the comparison clearer. It is a good idea to look at a variety of graphs to see which is the most helpful for a given data set. We might make different choices of what we think is the "best" graph depending on the data and the context, and our choice also depends on what we are using the data for.
Figure 1.2.4 — Pie chart of full-time and part-time enrollment at De Anza College.
Figure 1.2.5 — Pie chart of full-time and part-time enrollment at Foothill College.
Figure 1.2.6 — Bar graph of student status showing full-time and part-time counts side by side for De Anza College and Foothill College.
Each display answers a different question. A single pie chart answers "how is this college split?" — the wedges must add to 100%, so the whole circle is one college. The bar graph in Figure 1.2.6 answers "how do the two colleges compare?" — the raw counts sit next to each other, so you can see at a glance that De Anza is the bigger school. Notice the trade-off: the bar graph shows sizes but hides the percentages, while the pie charts show percentages but hide the fact that De Anza enrolls over 8,000 more students than Foothill.
Below are tables comparing the number of part-time and full-time students at De Anza College and Foothill College enrolled for the most recent spring quarter. The tables display counts (frequencies) and percentages or proportions (relative frequencies). The percent columns make comparing the same categories at the two colleges easier. Displaying percentages along with the numbers is often helpful, but it is particularly important when comparing sets of data that do not have the same totals — like the total enrollments at these two colleges. Notice how much larger the percentage for part-time students at Foothill College is compared to De Anza College.
| Status | De Anza — Number | De Anza — Percent | Foothill — Number | Foothill — Percent |
|---|---|---|---|---|
| Full-time | 9,200 | 40.9% | 4,059 | 28.6% |
| Part-time | 13,296 | 59.1% | 10,124 | 71.4% |
| Total | 22,496 | 100% | 14,183 | 100% |
Tables are a good way of organizing and displaying data. But graphs can be even more helpful in understanding the data. There are no strict rules concerning which graphs to use. Two graphs that are used to display qualitative data are pie charts and bar graphs.
Using Table 1.2.1, explain why comparing the percent of part-time students at the two colleges tells a different story than comparing the number of part-time students. Which comparison would you use to argue that Foothill serves a mostly part-time student body?
Solution
Step 1 — Compare the counts: De Anza has 13,296 part-time students; Foothill has 10,124. By raw count, De Anza has more part-time students.
Step 2 — Compare the percentages: Part-time students make up 59.1% of De Anza but 71.4% of Foothill. By share, Foothill is far more part-time.
Step 3 — Explain the difference: The two colleges have different totals (22,496 vs. 14,183). Counts are pulled around by the size of the school; percentages remove that effect and describe the make-up of each student body.
Answer: To argue that Foothill serves a mostly part-time student body, use the percentages — 71.4% of Foothill students are part-time, the largest share of the four figures in the table.
1.2.3 Percentages That Add to More (or Less) Than 100%
Sometimes percentages add up to more than 100% — or to less than 100%. In Table 1.2.2 the percentages add to more than 100% because a single student can fall into more than one category: a full-time student can also be under 25 and also intend to transfer. When categories overlap like this, a bar graph is appropriate to compare the relative size of the categories, but a pie chart cannot be used. A pie chart also could not be used if the percentages added to less than 100%.
| Characteristic/Category | Percent |
|---|---|
| Full-time students | 40.9% |
| Students who intend to transfer to a 4-year educational institution | 48.6% |
| Students under age 25 | 61.0% |
| TOTAL | 150.5% |
Figure 1.2.7 — Bar graph of De Anza College student characteristics, showing 61.0% under age 25, 48.6% intending to transfer, and 40.9% full-time, against 100% for all students.
Katherine Bruce runs a campus survey and reports that 55% of students work at least part-time, 62% commute more than 20 minutes, and 30% are parents. She wants to put all three numbers on one pie chart. Would a pie chart be an honest display of these three percentages? Explain, and say what she should use instead.
Solution
Step 1 — Add the percentages: \(55\% + 62\% + 30\% = 147\%\), which is more than 100%.
Step 2 — Ask why the total is over 100%: The categories overlap. A student can work part-time, commute 25 minutes, and be a parent — one person counted in all three categories.
Step 3 — Match the display to the structure: A pie chart requires that each individual land in exactly one wedge, so it cannot represent overlapping categories without misleading the reader.
Answer: No — a pie chart would not be honest here. Use a bar graph, where each bar is measured independently against the same 0–100% scale and no claim is made that the categories partition the student body.
1.2.4 Omitting Categories and Missing Data
Table 1.2.3 displays the ethnicity of students but is missing the "Other/Unknown" category. That category holds people who did not feel they fit into any of the listed ethnicity categories, or who declined to respond. Notice that the frequencies do not add up to the total number of students — 22,044 out of 24,382. In this situation, create a bar graph and not a pie chart.
| Ethnicity | Frequency | Percent |
|---|---|---|
| Asian | 8,794 | 36.1% |
| Black | 1,412 | 5.8% |
| Filipino | 1,298 | 5.3% |
| Hispanic/Latino | 4,180 | 17.1% |
| Native American | 146 | 0.6% |
| Pacific Islander | 236 | 1.0% |
| White | 5,978 | 24.5% |
| TOTAL | 22,044 out of 24,382 | 90.4% out of 100% |
Figure 1.2.8 — Bar graph of student ethnicity at De Anza College with the Other/Unknown category omitted.
Figure 1.2.9 is the same graph as Figure 1.2.8, but with the "Other/Unknown" percent (9.6%) included. That category turns out to be large compared to some of the others — Native American is 0.6% and Pacific Islander is 1.0%, so "Other/Unknown" is bigger than both combined and then some. This is important to know when we think about what the data are telling us: nearly one student in ten is invisible in the first graph.
This particular bar graph can be difficult to read visually. The graph in Figure 1.2.10 is a Pareto chart — the bars are sorted from largest to smallest, which makes it easier to read and interpret.
Figure 1.2.9 — Bar graph of student ethnicity at De Anza College including the Other/Unknown category.
Figure 1.2.10 — Pareto chart of student ethnicity at De Anza College with bars sorted from largest to smallest.
Pie charts: no missing data. The pie charts in Figure 1.2.11 and Figure 1.2.12 have the "Other/Unknown" category included, since the percentages in a pie chart must add to 100%. The chart in Figure 1.2.12 is organized by the size of each wedge, which makes it a more visually informative graph than the unsorted, alphabetical graph in Figure 1.2.11.
Figure 1.2.11 — Pie chart of student ethnicity at De Anza College with wedges in alphabetical order.
Figure 1.2.12 — Pie chart of student ethnicity at De Anza College with wedges sorted by size.
Club president Camila Reyes reports the majors of her members as: Business 40%, Nursing 25%, Engineering 15%. The club has 200 members but only 160 are accounted for in those three categories. What is wrong with drawing a pie chart of the three reported percentages, and how would you fix the display?
Solution
Step 1 — Add what is reported: \(40\% + 25\% + 15\% = 80\%\), so 20% of the membership is missing from the list.
Step 2 — Check the counts: \(200 - 160 = 40\) members are unaccounted for, and \(\frac{40}{200} = 20\%\) — the same gap.
Step 3 — Say why the pie fails: A pie chart claims the wedges fill the whole circle. Drawing only the three reported categories would silently re-scale them so they appear to be the entire club, hiding 40 people.
Answer: Either add an "Other/Undeclared" wedge of 20% so the pie honestly totals 100%, or use a bar graph, which does not promise that the categories are exhaustive.
1.2.5 Sampling
In a simple random sample, any group of \(n\) individuals is equally likely to be chosen as any other group of \(n\) individuals. In other words, each sample of the same size has an equal chance of being selected.
Definition 1.2.8 — A simple random sample: every individual had an equal chance of being chosen.
For example, suppose Lisa wants to form a four-person study group (herself and three other people) from her pre-calculus class, which has 31 members not including Lisa. To choose a simple random sample of size three from the other members of her class, Lisa could put all 31 names in a hat, shake the hat, close her eyes, and pick out three names. A more technological way is for Lisa to first list the last names of the members of her class together with a two-digit number, as in Table 1.2.4.
| ID | Name | ID | Name | ID | Name |
|---|---|---|---|---|---|
| 00 | Anselmo | 11 | King | 21 | Roquero |
| 01 | Bautista | 12 | Legeny | 22 | Roth |
| 02 | Bayani | 13 | Lundquist | 23 | Rowell |
| 03 | Cheng | 14 | Macierz | 24 | Salangsang |
| 04 | Cuarismo | 15 | Motogawa | 25 | Slade |
| 05 | Cuningham | 16 | Okimoto | 26 | Stratcher |
| 06 | Fontecha | 17 | Patel | 27 | Tallai |
| 07 | Hong | 18 | Price | 28 | Tran |
| 08 | Hoobler | 19 | Quizon | 29 | Wai |
| 09 | Jiao | 20 | Reyes | 30 | Wood |
| 10 | Khan |
Lisa can use a table of random numbers (found in many statistics books and mathematical handbooks), a calculator, or a computer to generate random numbers. For this example, suppose Lisa chooses to generate random numbers from a calculator. The numbers generated are as follows:
0.94360; 0.99832; 0.14669; 0.51470; 0.40581; 0.73381; 0.04399
Lisa reads two-digit groups until she has chosen three class members — that is, she reads 0.94360 as the groups 94, 43, 36, 60. Each random number may only contribute one class member. If she needed to, Lisa could have generated more random numbers.
The random numbers 0.94360 and 0.99832 do not contain appropriate two-digit numbers, because every pair is larger than 30. However, the third random number, 0.14669, contains 14 (the fourth random number also contains 14, but it is a repeat), the fifth random number contains 05, and the seventh random number contains 04. The two-digit number 14 corresponds to Macierz, 05 corresponds to Cuningham, and 04 corresponds to Cuarismo. Besides herself, Lisa's group will consist of Macierz, Cuningham, and Cuarismo.
Using rāSHio
To draw Lisa's three IDs without a graphing calculator:
- Open rāSHio and choose File → Random Numbers…
- Set the range to
0through30— the two-digit IDs in the class roster — and ask for3numbers. - Check No repeats so no ID is drawn twice, then click Generate — all three IDs appear at once. Look each one up in the roster.
Figure 1.2.13 — Drawing Lisa's three sample IDs in rāSHio: File → Random Numbers… with Min 0, Max 30, How many 3, and No repeats checked.
Besides simple random sampling, there are other forms of sampling that involve a chance process for getting the sample. Other well-known random sampling methods are the stratified sample, the cluster sample, and the systematic sample.
To choose a stratified sample, divide the population into groups called strata and then take a proportionate number from each stratum.
For example, you could stratify (group) your college population by department and then choose a proportionate simple random sample from each stratum — that is, from each department — to get a stratified random sample. To do that, number each member of the first department, number each member of the second department, and do the same for the remaining departments. Then use simple random sampling to choose proportionate numbers from each department in turn. The members picked from the first department, the second department, and so on together make up the stratified sample.
Definition 1.2.9 — A stratified sample: the population is split into strata and a few are drawn from every one.
To choose a cluster sample, divide the population into clusters (groups) and then randomly select some of the clusters. All the members from the selected clusters are in the cluster sample.
Stratifying takes a little from every group, so every group is represented. Clustering takes everything from a few groups, so entire groups are left out. Both are random — they just randomize different things.
For example, if you randomly sample four departments from your college population, the four departments make up the cluster sample. Divide your college faculty by department — the departments are the clusters. Number each department, then choose four different numbers using simple random sampling. All members of the four departments with those numbers are the cluster sample.
Definition 1.2.10 — A cluster sample: whole groups are chosen at random and every member of them is taken.
To choose a systematic sample, randomly select a starting point and take every \(k\)th piece of data from a listing of the population.
For example, suppose you have to do a phone survey. Your phone book contains 20,000 residence listings and you must choose 400 names for the sample. Number the population 1–20,000, then use a simple random sample to pick a number that represents the first name in the sample. Then choose every fiftieth name thereafter until you have a total of 400 names — you might have to wrap around to the beginning of your phone list. Systematic sampling is frequently chosen because it is a simple method.
Definition 1.2.11 — A systematic sample: a random starting point, then every kth item down the list.
A type of sampling that is non-random is convenience sampling, which involves using results that are readily available.
For example, a computer software store conducts a marketing study by interviewing potential customers who happen to be in the store browsing through the available software. The results of convenience sampling may be very good in some cases and highly biased — that is, they may favor certain outcomes — in others.
Sampling data should be done very carefully. Collecting data carelessly can have devastating results. Surveys mailed to households and then returned may be very biased, because they may favor a certain group of people who bothered to reply. It is better for the person conducting the survey to select the sample respondents.
Definition 1.2.12 — A convenience sample: only the readily available members are taken, so it is not random.
Sampling with replacement is truly random sampling: once a member is picked, that member goes back into the population and thus may be chosen more than once.
Definition 1.2.13 — Sampling with replacement: the drawn member returns to the population, so it can be chosen again.
In sampling without replacement, a member of the population may be chosen only once.
In Confidence Intervals you will meet formulas that set sample size from the precision you want — not from the size of the population. A sample of 1,000 can serve a population of 100,000 or 1,000,000 equally well. It feels wrong, and it is true.
Definition 1.2.14 — Sampling without replacement: a member chosen once is out of the population.
For practical reasons, in most populations simple random sampling without replacement is what actually gets done — surveys are typically done without replacement. Most samples are taken from large populations and the sample tends to be small in comparison to the population. Because of that, sampling without replacement is approximately the same as sampling with replacement: the chance of picking the same individual more than once is very low either way. The distinction becomes a real mathematical issue only when the population is small.
A sampling error is an error caused by the actual process of sampling — for example, the sample may not be large enough.
Definition 1.2.15 — Sampling error: sample estimates scatter either side of the true value by chance.
A nonsampling error is an error caused by factors not related to the sampling process — for example, a defective counting device.
Definition 1.2.16 — Nonsampling error: every estimate is pushed the same way, so a larger sample cannot fix it.
In reality, a sample will never be exactly representative of the population, so there will always be some sampling error. As a rule, the larger the sample, the smaller the sampling error.
In statistics, a sampling bias is created when a sample is collected from a population and some members of the population are not as likely to be chosen as others. (Remember: each member of the population should have an equally likely chance of being chosen.)
Left to ourselves, we pick the convenient, the nearby, the willing — and those people are not a cross-section of anybody. Handing the choice to chance removes the researcher's thumb from the scale, which is exactly what lets us generalize from a few hundred people to a few million.
When a sampling bias happens, there can be incorrect conclusions drawn about the population that is being studied — and unlike sampling error, more data will not fix it. A biased method run at ten times the scale produces a bigger, more confident wrong answer.
Gathering information about an entire population often costs too much or is virtually impossible. Instead, we use a sample of the population. A sample should have the same characteristics as the population it is representing. Most statisticians use various methods of random sampling in an attempt to achieve this goal. In each form of random sampling, every member of a population initially has an equal chance of being selected for the sample. Each method has pros and cons.
Definition 1.2.17 — Sampling bias: one group has no path into the sample, so the sample cannot speak for it.
You are going to use a random number generator to generate different types of samples from the data. Dawson Redbird is working through this exercise alongside you, and he will compare his four samples with yours at the end. Table 1.2.5 displays six sets of quiz scores (each quiz counts 10 points) for an elementary statistics class.
| #1 | #2 | #3 | #4 | #5 | #6 |
|---|---|---|---|---|---|
| 5 | 7 | 10 | 9 | 8 | 3 |
| 10 | 5 | 9 | 8 | 7 | 6 |
| 9 | 10 | 8 | 6 | 7 | 9 |
| 9 | 10 | 10 | 9 | 8 | 9 |
| 7 | 8 | 9 | 5 | 7 | 4 |
| 9 | 9 | 9 | 10 | 8 | 7 |
| 7 | 7 | 10 | 9 | 8 | 8 |
| 8 | 8 | 9 | 10 | 8 | 8 |
| 9 | 7 | 8 | 7 | 7 | 8 |
| 8 | 8 | 10 | 9 | 8 | 7 |
Use a random number generator to pick the following samples.
Every draw below is already done for you — the numbers in each step are what a random draw returned. Read the quiz scores straight off Table 1.2.5 and write down each sample, then compare the four.
- Create a stratified sample by column. Three quiz scores from every column.
- Number the rows one through ten, top to bottom.
- The draw gave rows
2, 5, 9for column #1;1, 4, 8for #2;3, 3, 7for #3 (a repeat is allowed);2, 6, 10for #4;1, 5, 9for #5; and4, 7, 8for #6. - Record the quiz score at each of those row numbers in its own column.
- These 18 quiz scores are a stratified sample — every column is represented.
- Create a cluster sample by taking whole columns.
- The draw gave columns
2and5. - Record all ten quiz scores in each.
- These 20 quiz scores are a cluster sample — two whole groups, nobody inside them left out.
- Create a simple random sample of 15 quiz scores.
- Number the scores one through 60, reading down column #1, then #2, and so on.
- The draw gave
3, 7, 11, 14, 19, 22, 27, 31, 36, 40, 44, 49, 53, 57, 60. - Record the quiz score at each.
- These 15 quiz scores are a simple random sample — every score had the same chance of being on the list.
- Create a systematic sample of 12 quiz scores.
- Keep the numbering one through 60.
- The random start is
4. Record that score, then every tenth one after it — 14, 24, 34, and so on — wrapping back to the beginning when you run past 60. - These 12 quiz scores are a systematic sample — one random start, then a fixed step.
Solution
Your four samples will not match anyone else's — that is the point of the exercise. What should match is the structure of each sample and the count of scores in it.
Step 1 — Stratified check: You drew 3 scores from each of 6 columns, so your stratified sample holds \(3 \times 6 = 18\) scores. Every column is represented. Because repeats were recorded, you may have the same score more than once — that is sampling with replacement within each stratum.
Step 2 — Cluster check: You chose 2 whole columns of 10 scores each, so your cluster sample holds \(2 \times 10 = 20\) scores. Four columns are entirely absent — that is what clustering does.
Step 3 — Simple random check: You drew 15 scores from the 60 in the table, with no attention to which column they came from. Some columns may be over-represented purely by chance.
Step 4 — Systematic check: You picked one random start out of 60 and then stepped by 10, wrapping around, for 12 scores total. Because there are 6 columns of 10, a step of 10 lands you in a predictable pattern — a good reminder that a systematic sample can accidentally sync up with hidden structure in the list.
Answer: Stratified = 18 scores (3 per column); cluster = 20 scores (2 full columns); simple random = 15 scores; systematic = 12 scores. Compare your four sample averages with Dawson's — his will differ from yours, and neither of you is wrong.
Nathan Prescott runs a study to determine the average tuition that San Jose State undergraduate students pay per semester. Each student in the following samples is asked how much tuition they paid for the Fall semester, and he uses a different method each time. What is the type of sampling in each case?
a. A sample of 100 undergraduate San Jose State students is taken by organizing the students' names by classification (first-year, sophomore, junior, or senior), and then selecting 25 students from each.
b. A random number generator is used to select a student from the alphabetical listing of all undergraduate students in the Fall semester. Starting with that student, every 50th student is chosen until 75 students are included in the sample.
c. A completely random method is used to select 75 students. Each undergraduate student in the fall semester has the same probability of being chosen at any stage of the sampling process.
d. The first-year, sophomore, junior, and senior years are numbered one, two, three, and four, respectively. A random number generator is used to pick two of those years. All students in those two years are in the sample.
e. An administrative assistant is asked to stand in front of the library one Wednesday and to ask the first 100 undergraduate students he encounters what they paid for tuition the Fall semester. Those 100 students are the sample.
Solution
Step 1 — Part a: The population is split into class-standing groups and a fixed number is drawn from each group. Taking some from every group is the signature of stratifying. → stratified
Step 2 — Part b: A random start, then every 50th name down the list. A fixed step through an ordered list is the signature of a systematic sample. → systematic
Step 3 — Part c: Every student has the same probability at every stage, with no grouping at all. → simple random
Step 4 — Part d: The years are the groups, two whole years are chosen at random, and everyone in those years is in. Taking entire groups is the signature of clustering. → cluster
Step 5 — Part e: Whoever happens to walk past the library that Wednesday. No chance mechanism at all — just who was handy. → convenience
Answer: a. stratified; b. systematic; c. simple random; d. cluster; e. convenience
Determine the type of sampling used: Principal Diane Kessler polls 50 first-year students, 50 sophomores, 50 juniors, and 50 seniors regarding policy changes for after-school activities. She wants every class year represented.
Solution
Step 1 — Identify the groups: The four class standings — first-year, sophomore, junior, senior — divide the student body into non-overlapping groups.
Step 2 — Ask how many groups were used: All four. Nobody's class standing is excluded.
Step 3 — Match to a method: Taking a fixed number from every group is stratifying. (Had she randomly picked two of the four classes and polled everyone in them, it would have been a cluster sample instead.)
Answer: This is a stratified sample, with class standing as the strata.
If we were to examine two samples representing the same population, even if we used random sampling methods for both, they would not be exactly the same. Just as there is variation in data, there is variation in samples. As you become accustomed to sampling, that variability will begin to seem natural.
Determine the type of sampling used (simple random, stratified, systematic, cluster, or convenience).
a. Coach Jordan Achebe selects six players from a group of children aged eight to ten, seven players from a group of children aged 11 to 12, and three players from a group of children aged 13 to 14 to form the recreational soccer team they coach.
b. A pollster interviews all human resource personnel in five different high tech companies.
c. A high school educational researcher interviews 50 public high school teachers and 50 private high school teachers.
d. A medical researcher interviews every third cancer patient from a list of cancer patients at a local hospital.
e. A high school counselor uses a computer to generate 50 random numbers and then picks students whose names correspond to the numbers.
f. A student interviews classmates in their algebra class to determine how many pairs of jeans a student owns, on the average.
Solution
Step 1 — Part a: The age brackets are the groups, and a set number is taken from each. Some from every group → stratified
Step 2 — Part b: Five companies are the groups, and all HR staff in those companies are interviewed. Everyone from selected groups → cluster
Step 3 — Part c: Public and private are the two groups; 50 are taken from each. Some from every group → stratified
Step 4 — Part d: Every third patient down an ordered list — a fixed step → systematic
Step 5 — Part e: Random numbers matched to names, no grouping → simple random
Step 6 — Part f: Whoever happens to sit in the student's own algebra class — the most readily available people → convenience
Answer: a. stratified; b. cluster; c. stratified; d. systematic; e. simple random; f. convenience
1.2.6 Critical Evaluation
We need to evaluate the statistical studies we read about critically, and analyze them before accepting their results. Common problems to watch for include the following.
- Problems with samples. A sample must be representative of the population. A sample that is not representative is biased, and biased samples give results that are inaccurate and not valid.
- Self-selected samples. Responses only from people who choose to respond — call-in surveys, online polls — are often unreliable.
- Sample size issues. Samples that are too small may be unreliable. Larger samples are better where possible. In some situations small samples are unavoidable and can still support conclusions: crash testing cars, or medical testing for rare conditions.
- Undue influence. Collecting data or asking questions in a way that influences the response.
- Non-response or refusal of a subject to participate. The collected responses may no longer be representative of the population. People with strong positive or negative opinions often answer surveys while everyone else skips them, which can distort the results.
- Causality. A relationship between two variables does not mean that one causes the other. They may be related (correlated) because of their relationship through a different variable.
- Self-funded or self-interest studies. A study performed by a person or organization to support their own claim. Is the study impartial? Read it carefully. Do not automatically assume the study is bad — but do not automatically assume it is good either. Evaluate it on its merits and the work done.
- Misleading use of data. Improperly displayed graphs, incomplete data, or a lack of context.
- Confounding. When the effects of multiple factors on a response cannot be separated. Confounding makes it difficult or impossible to draw valid conclusions about the effect of each factor.
As a class, determine whether or not the following samples are representative. If they are not, discuss the reasons.
1. To find the average GPA of all students in a university, use all honor students at the university as the sample.
2. To find out the most popular cereal among young people under the age of ten, stand outside a large supermarket for three hours and speak to every twentieth child under age ten who enters the supermarket.
3. To find the average annual income of all adults in the United States, sample U.S. Representatives. Create a cluster sample by considering each state as a stratum (group). By using simple random sampling, select states to be part of the cluster. Then survey every U.S. Representative in the cluster.
4. To determine the proportion of people taking public transportation to work, survey 20 people in New York City. Conduct the survey by sitting in Central Park on a bench and interviewing every person who sits next to you.
5. To determine the average cost of a two-day stay in a hospital in Massachusetts, survey 100 hospitals across the state using simple random sampling.
Héctor Ramos, a spokesperson for a shampoo company, presents a study his own company funded: 4,000 volunteers recruited from the company mailing list, of whom 92% saw improvement. Name at least three items from the list above that his study trips over, and say whether the large sample size fixes any of them.
Solution
Step 1 — Who paid for it: This is a self-funded / self-interest study. Héctor's company has a stake in the answer, so his design and reporting choices deserve extra scrutiny.
Step 2 — Who is in the sample: Volunteers from the company's own mailing list are a self-selected sample and a convenience sample. People already interested enough to join a shampoo mailing list are not a cross-section of shampoo users.
Step 3 — What "improvement" means: "Saw improvement" is a self-reported judgment with no comparison group. Non-response and undue influence are both live risks, and with no control group any change could be confounded with season, other products, or simple expectation.
Step 4 — Does 4,000 help? No. Sample size fixes sampling error, not sampling bias. A biased selection method run on 4,000 people produces a more precise estimate of the wrong quantity.
Answer: At least three problems — self-funded study, self-selected sample, and confounding with no control group — and the large sample size fixes none of them, because it addresses only random error.
In a matched pairs design, what is still left for chance to decide — and what goes wrong if the researcher decides it instead?
Solution
Step 1 — Recall the setup: In a matched pairs design, two treatments are compared within the same pair — the same person before and after, or two people matched on age, sex, and health. Pairing removes the person-to-person differences that would otherwise swamp the treatment effect.
Step 2 — Find the leftover source of bias: Something still has to decide which member of each pair gets which treatment, or which treatment comes first in a before/after design. If the researcher decides, that choice can track something else — the healthier twin gets the new drug, the well-rested trial always goes first.
Step 3 — Apply randomization: Within each pair, flip a coin to assign the two treatments. Do it independently for every pair.
Answer: Randomize the assignment within each pair. This is how, and the why is that it breaks any link between the treatment and hidden characteristics of the pair members — including order effects such as practice or fatigue — so a difference in the paired outcomes can be attributed to the treatment rather than to how the assignment was made.
1.2.7 Variation in Data
Variation is present in any set of data. For example, 16-ounce cans of a beverage may contain more or less than 16 ounces of liquid. In one study, eight 16-ounce cans were measured and produced the following amounts (in ounces) of beverage:
15.8; 16.1; 15.2; 14.8; 15.8; 15.9; 16.0; 15.5
Measurements of the amount of beverage in a 16-ounce can may vary because different people make the measurements, or because the exact amount — 16 ounces of liquid — was not put into the cans. Manufacturers regularly run tests to determine whether the amount of beverage in a 16-ounce can falls within the desired range.
Be aware that as you take data, your data may vary somewhat from the data someone else is taking for the same purpose. This is completely natural. However, if two or more of you are taking the same data and get very different results, it is time for all of you to reevaluate your data-taking methods and your accuracy.
Quality inspector Lucía Herrera reviews the eight can measurements above. Explain in one or two sentences why she would not conclude that her filling machine is broken. What would make you suspect the machine really was broken?
Solution
Step 1 — Look at the spread: The values run from 14.8 to 16.1 ounces, scattering on both sides of the 16-ounce target. Four are within half an ounce of 16.
Step 2 — Interpret the scatter: Some variation is unavoidable in any filling process and in any measurement of it. Values landing above and below the target, by small amounts, is exactly the fingerprint of ordinary variation.
Step 3 — Say what would be different if it were broken: A broken machine would show a pattern, not scatter — every can consistently under 16 ounces (a systematic shortfall), or a spread so wide that some cans held 12 ounces and others 19.
Answer: Lucía should not conclude the machine is broken, because the readings scatter on both sides of 16 by small amounts, which is normal variation. Consistent underfilling, or a dramatically wider spread, would be the signal that something is genuinely wrong.
Try it in rāSHio
Open rāSHio, choose File → Delimited List… and paste the eight can amounts exactly as the book lists them (15.8; 16.1; 15.2; 14.8; 15.8; 15.9; 16.0; 15.5 — semicolons are fine), then choose Stats → Summary Statistics to put a number on the spread the quality inspector is judging by eye.
Figure 1.2.14 — Putting a number on the spread in rāSHio: Stats → Summary Statistics.
1.2.8 Variation in Samples
It was mentioned previously that two or more samples from the same population, taken randomly and having close to the same characteristics as the population, will likely still be different from each other. Suppose Doreen and Jung both decide to study the average amount of time students at their college sleep each night. Doreen and Jung each take samples of 500 students. Doreen uses systematic sampling and Jung uses cluster sampling. Doreen's sample will be different from Jung's sample. Even if Doreen and Jung used the same sampling method, in all likelihood their samples would be different. Neither would be wrong, however.
Think about what contributes to making Doreen's and Jung's samples different. If Doreen and Jung took larger samples — that is, if the number of data values were increased — their sample results (the average amount of time a student sleeps) might be closer to the actual population average. But still, their samples would in all likelihood be different from each other. This variability in samples cannot be stressed enough.
Doreen's 500 students average 6.8 hours of sleep; Jung's 500 students average 7.1 hours. A classmate, Sam Whitlock, says one of them must have made a mistake — they are certain somebody miscounted. Respond to Sam, and explain what would happen to the gap if each of them sampled 5,000 students instead.
Solution
Step 1 — Name what causes the gap: Doreen and Jung surveyed different students. Two different random samples from the same population almost never produce the same average — that is sampling variability, not error.
Step 2 — Check the size of the gap: The difference is 0.3 hours, about 18 minutes. On a quantity that varies by hours from student to student, that is a small gap.
Step 3 — Predict the effect of larger samples: Larger samples pin down the population average more tightly, so each sample mean would sit closer to the true value — and therefore the two would typically sit closer to each other. The gap would shrink, but it would not vanish.
Answer: Neither made a mistake; the difference is ordinary sampling variability. With 5,000 students each, the two averages would very likely be closer together — but still not identical, because two different samples are still two different samples.
1.2.9 Size of a Sample
The size of a sample — often called the number of observations — is important. The examples you have seen in this book so far have been small. Samples of only a few hundred observations, or even smaller, are sufficient for many purposes. In polling, samples of 1,200 to 1,500 observations are considered large enough and good enough if the survey is random and well done. You will learn why when you study confidence intervals.
Be aware that many large samples are biased. Call-in surveys, for example, are invariably biased, because people choose to respond or not.
Station manager Rosa Villalobos runs a local radio station with a fan base of 20,000 listeners. She wants to know if her audience would prefer more music or more talk shows. Asking all 20,000 listeners is an almost impossible task.
Rosa uses convenience sampling and surveys the first 200 people she meets at one of the station's music concert events. 24 people said they would prefer more talk shows, and 176 people said they would prefer more music.
Do you think that this sample is representative of (or is characteristic of) the entire 20,000-listener population?
Solution
Step 1 — Identify where the sample came from: The 200 people were at a music concert put on by the station. Attending a music concert is evidence that you like music.
Step 2 — Ask who was excluded: Listeners who prefer talk shows had little reason to be at that event, so they had almost no chance of being surveyed. The selection favored one answer before a single question was asked.
Step 3 — Look at the result in that light: 176 of 200 — 88% — preferring music is exactly what a music-concert crowd would say. The result tells us about concertgoers, not about the fan base.
Answer: No, this sample is not representative. It is a convenience sample drawn at a music event, so it is biased toward the "more music" answer. To learn what the 20,000 listeners want, Rosa needs a random sample of her whole fan base — for instance, a simple random sample from the station's listener list.
Divide into groups of two, three, or four. Your instructor will give each group one six-sided die. Try this experiment twice. Roll one fair die (six-sided) 20 times. Record the number of ones, twos, threes, fours, fives, and sixes you get in Table 1.2.6 and Table 1.2.7 ("frequency" is the number of times a particular face of the die occurs).
Did the two experiments have the same results? Probably not. If you did the experiment a third time, do you expect the results to be identical to the first or second experiment? Why or why not?
Which experiment had the correct results? They both did. The job of the statistician is to see through the variability and draw appropriate conclusions.
| Face on Die | Frequency |
|---|---|
| 1 | |
| 2 | |
| 3 | |
| 4 | |
| 5 | |
| 6 |
| Face on Die | Frequency |
|---|---|
| 1 | |
| 2 | |
| 3 | |
| 4 | |
| 5 | |
| 6 |
Try it in rāSHio
Once both experiments are rolled, type your 40 face values into a column in rāSHio and choose Graph → Frequency Table with Discrete values checked — it counts each face for you, so you can compare the two experiments’ frequencies without tallying by hand.
Figure 1.2.15 — Counting each die face in rāSHio: Graph → Frequency Table with Discrete values checked.
Quinn Marsden studies ABC College, which has 10,000 part-time students (the population). They are interested in the average amount of money a part-time student spends on books in the fall term. Asking all 10,000 students is an almost impossible task, so Quinn takes samples.
First, Quinn uses convenience sampling and surveys ten students from a first-term organic chemistry class. Many of these students are also taking first-term calculus. The amount of money they spend on books is:
$128; $87; $173; $116; $130; $204; $147; $189; $93; $153
The second sample is taken from a list of senior citizens who take P.E. classes, taking every fifth senior citizen on the list, for a total of ten senior citizens. They spend:
$50; $40; $36; $15; $50; $100; $40; $53; $22; $22
It is unlikely that any student is in both samples.
a. Do you think that either of these samples is representative of (or is characteristic of) the entire 10,000 part-time student population?
b. Since these samples are not representative of the entire population, is it wise to use the results to describe the entire population?
Now suppose Quinn takes a third sample. They choose ten different part-time students from the disciplines of chemistry, math, English, psychology, sociology, history, nursing, physical education, art, and early childhood development. (We assume these are the only disciplines in which part-time students at ABC College are enrolled, and that an equal number of part-time students are enrolled in each.) Each student is chosen using simple random sampling. Using a calculator, random numbers are generated and a student from a particular discipline is selected if they have a corresponding number. The students spend the following amounts:
$180; $50; $150; $85; $260; $75; $180; $200; $200; $150
c. Is the third sample biased?
Solution
Part a — No, neither is representative.
The first sample probably consists of science-oriented students. Besides the chemistry course, some of them are also taking first-term calculus, and books for these classes tend to be expensive. Most of these students are, more than likely, paying more than the average part-time student for their books.
The second sample is a group of senior citizens who are, more than likely, taking courses for health and interest. The amount of money they spend on books is probably much less than the average part-time student. Both samples are biased. Also, in both cases, not all students had a chance to be in either sample.
Part b — No, it is not wise.
For these samples, each member of the population did not have an equally likely chance of being chosen. A number computed from a biased sample describes only that slice of the population, not the whole.
Part c — The third sample is unbiased.
Every discipline is represented and each student within a discipline was chosen at random, so no group was systematically favored. That said, a larger sample would be recommended, to increase the likelihood that the sample is close to representative of the population. Note the asymmetry: for a biased sampling technique, even a large sample runs the risk of not being representative.
Answer: a. No — both are biased. b. No — an unequal chance of selection makes the results unusable for the whole population. c. Unbiased, though a larger sample would be better.
Students often ask if it is "good enough" to take a sample instead of surveying the entire population. If the survey is done well, the answer is yes.
1.2.10 Two Ways a Sample Goes Wrong After You Have Chosen It
Everything in the previous section is about choosing who to ask. But a sample is not the list of people you selected — it is the list of people who actually answered. Those two lists are never the same, and the gap between them is where a surprising amount of bad statistics lives.
Nonresponse bias occurs when the members of a sample who do not respond differ in a meaningful way from those who do, so the responses that come back do not represent the sample that was drawn.
Notice what makes this dangerous: the sampling itself may have been flawless. You can draw a perfect simple random sample of 4,000 students, email every one of them, and receive 300 replies — and those 300 are no longer a random sample of anything. They are a sample of students who reply to emails, which is a different population with different habits.
Voluntary response bias occurs when people select themselves into a sample by choosing to respond, so the sample over-represents whoever felt strongly enough to participate.
With sampling error, more data helps — the estimate settles down around the truth. With self-selection, more data settles the estimate down around the wrong value, and does it more convincingly. Ten thousand voluntary responses give you a very precise measurement of what strongly-motivated people think. That precision is real, and it is not an answer to your question.
Voluntary response is nonresponse bias with the dial turned to maximum: nobody was selected at all, so the entire sample is self-chosen. Online reviews are the everyday example. The people who write them are disproportionately those who had an unusually good or unusually bad experience; the large middle group who thought the meal was fine simply ate and left.
The practical defence is unglamorous: report your response rate. A study that says "we sampled 4,000 students" is telling you about its intentions. A study that says "we sampled 4,000 students and 312 responded, a rate of 7.8%" is telling you about its evidence, and inviting you to judge it.
Definitions 1.2.18 and 1.2.19 — A flawless draw can still return a non-random sample; self-selection is the same failure with nobody drawn at all.
For each situation, say whether the main problem is nonresponse bias, voluntary response bias, or neither — and say who is over- or under-represented in the result.
a. A department mails a printed survey to a random sample of 500 alumni. 90 are returned.
b. A streaming service displays a pop-up reading "Rate this show!" after each episode and reports an average rating of 4.6 out of 5.
c. A researcher draws a random sample of 200 households and visits each one in person, returning up to three times to reach anyone not home. She obtains 191 responses.
d. A radio host asks listeners to call in with their opinion on a proposed tax and reports that 78% of callers oppose it.
Solution
Part a — Nonresponse bias. The sampling was random, but only 18% replied. Returning a printed survey by mail takes time and a stamp's worth of motivation, so the 90 respondents skew toward alumni with a stronger attachment to the department and more time on their hands. Recent graduates and the very busy are under-represented.
Part b — Voluntary response bias. Nobody was sampled; viewers opted in. Those who disliked the show enough to stop watching never saw the pop-up at all, which quietly removes the harshest ratings from the pool. The 4.6 describes people who finished episodes, and a rating averaged over finishers cannot be read as a rating of the show.
Part c — Neither, or very little. A 95.5% response rate with deliberate callbacks is about as good as field work gets. Some nonresponse bias remains in principle, but with 191 of 200 there is not enough missing for it to move the result much. This is what the defence against nonresponse actually looks like: persistence, not a larger initial sample.
Part d — Voluntary response bias. Callers select themselves, and people who are angry about a tax will pick up the phone at far higher rates than people who are mildly in favour or indifferent. The 78% measures intensity of feeling among listeners motivated to call, not opinion among listeners, and certainly not opinion among residents.
Problem Set 1.2
Problem 1. "Number of times per week" is what type of data?
a) qualitative (categorical)
b) quantitative discrete
c) quantitative continuous
Solution
Step 1 — Read what is being recorded: "Number of times per week" is a count of visits, not a category and not a measurement.
Step 2 — Check the possible values: A resident visits the park 3 times or 4 times a week — never 3.6 times.
Answer: b. quantitative discrete
A study was done to determine the age, number of times per week, and the duration (amount of time) of residents using a local park in San Antonio, Texas. The first house in the neighborhood around the park was selected randomly, and then the resident of every eighth house in the neighborhood around the park was interviewed. Use this setting for the next four problems.
Problem 2. The sampling method was
a) simple random
b) systematic
c) stratified
d) cluster
Solution
Step 1 — Find the selection rule: The first house was chosen at random, and then every eighth house after that was interviewed.
Step 2 — Match it to a method: A random start followed by a fixed step through an ordered list is the signature of a systematic sample. It is not simple random (not every group of houses was equally likely), not stratified (no groups were formed), and not cluster (no whole group was taken).
Answer: b. systematic
Problem 3. "Duration (amount of time)" is what type of data?
a) qualitative (categorical)
b) quantitative discrete
c) quantitative continuous
Solution
Step 1 — Read what is being recorded: Duration is the amount of time a resident spends in the park, read off a clock.
Step 2 — Check the possible values: A visit can last 42 minutes, 42.5 minutes, or 42.37 minutes. Any level of precision is possible, so the values are not restricted to counting numbers.
Answer: c. quantitative continuous
Problem 4. The colors of the houses around the park are what kind of data?
a) qualitative (categorical)
b) quantitative discrete
c) quantitative continuous
Solution
Step 1 — Read what is being recorded: Each house gets a color name — white, blue, tan — not a number.
Step 2 — Apply the rule: Colors sort houses into categories. There is no arithmetic to perform on them.
Answer: a. qualitative (categorical)
Problem 5. The population is _______________.
Solution
Step 1 — Ask who the study wanted to learn about: The study is about the people who live around the park and use it — not just the ones who happened to be interviewed.
Step 2 — Separate population from sample: The residents of every eighth house are the sample. The whole group they were drawn from is the population.
Answer: The population is all the residents in the neighborhood around the park in San Antonio, Texas.
Problem 6. Table 1.2.8 contains the total number of deaths worldwide as a result of earthquakes over a 13-year period.
| Year | Total Number of Deaths |
|---|---|
| 1 | 231 |
| 2 | 21,357 |
| 3 | 11,685 |
| 4 | 33,819 |
| 5 | 228,802 |
| 6 | 88,003 |
| 7 | 6,605 |
| 8 | 712 |
| 9 | 88,011 |
| 10 | 1,790 |
| 11 | 320,120 |
| 12 | 21,953 |
| 13 | 768 |
| Total | 823,856 |
Use Table 1.2.8 to answer the following questions.
a) What is the proportion of deaths between Year 8 and Year 13?
b) What percent of deaths occurred before Year 2?
c) What is the percent of deaths that occurred in Year 4 or after Year 11?
d) What is the fraction of deaths that happened before Year 13?
e) What kind of data is the number of deaths?
f) Earthquakes are quantified according to the amount of energy they produce (examples are 2.1, 5.0, 6.7). What type of data is that?
g) What contributed to the large number of deaths in Year 11? In Year 5? Explain.
Solution
Step 1 — Part a. Proportion of deaths between Year 8 and Year 13. "Between Year 8 and Year 13" means the years strictly in between — Years 9, 10, 11, and 12:
$$88{,}011 + 1{,}790 + 320{,}120 + 21{,}953 = 431{,}874$$ $$\frac{431{,}874}{823{,}856} \approx 0.5242$$Step 2 — Part b. Percent of deaths before Year 2. Only Year 1 comes before Year 2:
$$\frac{231}{823{,}856} \approx 0.00028 = 0.03\%$$Step 3 — Part c. Percent in Year 4 or after Year 11. Year 4 gives 33,819; after Year 11 means Years 12 and 13, giving \(21{,}953 + 768 = 22{,}721\):
$$\frac{33{,}819 + 22{,}721}{823{,}856} = \frac{56{,}540}{823{,}856} \approx 0.0686 = 6.86\%$$Step 4 — Part d. Fraction of deaths before Year 13. Everything except Year 13:
$$\frac{823{,}856 - 768}{823{,}856} = \frac{823{,}088}{823{,}856}$$Step 5 — Part e. What kind of data is the number of deaths? Deaths are counted, and only whole numbers are possible, so this is quantitative discrete.
Step 6 — Part f. What type of data is earthquake magnitude? Values like 2.1, 5.0, and 6.7 come from a measuring instrument on a continuous scale, so this is quantitative continuous.
Step 7 — Part g. Why so many deaths in Year 11 and Year 5? In both years, undersea earthquakes produced massive tsunamis. The deaths were caused mostly by the resulting flooding rather than by ground shaking, which is why those two totals dwarf the others.
Answer: a. \(\approx 0.5242\); b. \(\approx 0.03\%\); c. \(\approx 6.86\%\); d. \(\frac{823{,}088}{823{,}856}\); e. quantitative discrete; f. quantitative continuous; g. in both years, undersea earthquakes produced massive tsunamis.
For the following four exercises, determine the type of sampling used (simple random, stratified, systematic, cluster, or convenience).
Problem 7. A group of test subjects is divided into twelve groups; then four of the groups are chosen at random.
Solution
Step 1 — Identify the groups: The test subjects are divided into twelve groups.
Step 2 — Ask what happens to a selected group: Four groups are chosen at random, and everyone in those four groups is in the sample. Taking entire groups is the signature of clustering — the other eight groups contribute nobody.
Answer: cluster
Problem 8. A market researcher polls every tenth person who walks into a store.
Solution
Step 1 — Find the selection rule: Every tenth person to walk into the store is polled.
Step 2 — Match it to a method: A fixed step through the stream of arrivals is a systematic sample.
Answer: systematic
Problem 9. The first 50 people who walk into a sporting event are polled on their television preferences.
Solution
Step 1 — Find the selection rule: The first 50 people through the door are polled — whoever they happen to be.
Step 2 — Match it to a method: There is no chance mechanism at all; the researcher simply took the people who were easiest to reach.
Answer: convenience
Problem 10. A computer generates 100 random numbers, and 100 people whose names correspond with the numbers on the list are chosen.
Solution
Step 1 — Find the selection rule: A computer generates 100 random numbers and the matching names are chosen.
Step 2 — Match it to a method: No grouping, no fixed step — every individual, and every group of 100 individuals, had an equal chance of being selected.
Answer: simple random
Studies are often done by pharmaceutical companies to determine the effectiveness of a treatment program. Suppose that a new drug is currently under study to address a respiratory virus. It is given to patients once the patient exhibits symptoms of the virus. Of interest is the average (mean) length of time in days from the time the patient starts the treatment until the symptoms are alleviated. Two researchers — Dr. Amir Haddad (Researcher A) and Dr. Priya Chandra (Researcher B) — each follow a different set of 40 patients with the respiratory virus from the start of treatment until the symptoms are alleviated. Dr. Haddad reports his totals first. The following data (in days) are collected. Use this setting for the next eight problems.
Researcher A: 3; 4; 11; 15; 16; 17; 22; 44; 37; 16; 14; 24; 25; 15; 26; 27; 33; 29; 35; 44; 13; 21; 22; 10; 12; 8; 40; 32; 26; 27; 31; 34; 29; 17; 8; 24; 18; 47; 33; 34
Researcher B: 3; 14; 11; 5; 16; 17; 28; 41; 31; 18; 14; 14; 26; 25; 21; 22; 31; 2; 35; 44; 23; 21; 21; 16; 12; 18; 41; 22; 16; 25; 33; 34; 29; 13; 18; 24; 23; 42; 33; 29
Problem 11. Complete the tables using the data provided.
| Survival Length (in days) | Frequency | Relative Frequency | Cumulative Relative Frequency |
|---|---|---|---|
| 0.5–6.5 | |||
| 6.5–12.5 | |||
| 12.5–18.5 | |||
| 18.5–24.5 | |||
| 24.5–30.5 | |||
| 30.5–36.5 | |||
| 36.5–42.5 | |||
| 42.5–48.5 |
| Survival Length (in days) | Frequency | Relative Frequency | Cumulative Relative Frequency |
|---|---|---|---|
| 0.5–6.5 | |||
| 6.5–12.5 | |||
| 12.5–18.5 | |||
| 18.5–24.5 | |||
| 24.5–30.5 | |||
| 30.5–36.5 | |||
| 36.5–45.5 |
Solution
Step 1 — Sort each researcher's 40 values into the bins. Work down the list and tally. Researcher A's smallest value is 3 and largest is 47; Researcher B's smallest is 2 and largest is 44.
Step 2 — Divide each frequency by 40 to get the relative frequency, then add the relative frequencies down the column to get the cumulative relative frequency.
| Survival Length (in days) | Frequency | Relative Frequency | Cumulative Relative Frequency |
|---|---|---|---|
| 0.5–6.5 | 2 | 0.050 | 0.050 |
| 6.5–12.5 | 5 | 0.125 | 0.175 |
| 12.5–18.5 | 9 | 0.225 | 0.400 |
| 18.5–24.5 | 5 | 0.125 | 0.525 |
| 24.5–30.5 | 7 | 0.175 | 0.700 |
| 30.5–36.5 | 7 | 0.175 | 0.875 |
| 36.5–42.5 | 2 | 0.050 | 0.925 |
| 42.5–48.5 | 3 | 0.075 | 1.000 |
| Survival Length (in days) | Frequency | Relative Frequency | Cumulative Relative Frequency |
|---|---|---|---|
| 0.5–6.5 | 3 | 0.075 | 0.075 |
| 6.5–12.5 | 2 | 0.050 | 0.125 |
| 12.5–18.5 | 11 | 0.275 | 0.400 |
| 18.5–24.5 | 8 | 0.200 | 0.600 |
| 24.5–30.5 | 6 | 0.150 | 0.750 |
| 30.5–36.5 | 6 | 0.150 | 0.900 |
| 36.5–45.5 | 4 | 0.100 | 1.000 |
Step 3 — Check your work: Each Frequency column must add to 40, and each Cumulative Relative Frequency column must end at 1.000. Both do.
Answer: Researcher A: 2, 5, 9, 5, 7, 7, 2, 3. Researcher B: 3, 2, 11, 8, 6, 6, 4. Relative and cumulative relative frequencies as shown above.
Problem 12. Determine what the key term data refers to in the above example for Researcher A.
Solution
**Step 1 — Recall what data means:** Data are the actual values recorded on the individuals in the study — not the summary of them, and not the question being asked.
Step 2 — Apply it here: Dr. Haddad recorded, for each of his 40 patients, the number of days from the start of treatment until the symptoms were alleviated.
Answer: The data are the values for the survival length in days — 3, 4, 11, 15, and so on — one value for each of the 40 patients.
Problem 13. List two reasons why the data may differ.
Solution
Step 1 — Different people: The two researchers followed two different sets of 40 patients. Two different samples from the same population practically never give the same numbers, purely by chance.
Step 2 — Different circumstances: The patients may differ in ways that affect recovery — age, general health, how quickly they started treatment after symptoms appeared, or which region or hospital they came from.
Answer: Two reasons, among others: (1) the samples contain different patients, so ordinary sampling variability alone produces different values; (2) the two groups may differ systematically — in age, health, or how soon treatment began — which shifts the recovery times.
Problem 14. Can you tell if one researcher is correct and the other one is incorrect? Why?
Solution
Step 1 — Ask what "correct" would mean: To call one researcher wrong, we would need to know the true average recovery time for the whole population and see that one of them missed it.
Step 2 — Check what we actually have: We have two samples of 40 and nothing else. We do not know how either sample was drawn, and we do not know the population value.
Answer: No, we do not have enough information to make such a claim. Two different results are exactly what two different samples are expected to produce.
Problem 15. Would you expect the data to be identical? Why or why not?
Solution
Step 1 — Recall sampling variability: Even two well-run random samples from the same population differ from each other. That is not an error; it is the normal behavior of sampling.
Step 2 — Consider the size: With only 40 patients each, there is plenty of room for the two sets of values to look quite different.
Answer: No. We would not expect the data to be identical, because the two researchers observed different patients. Identical data sets would actually be suspicious.
Problem 16. Suggest at least two methods the researchers might use to gather random data.
Solution
Step 1 — Method one, simple random sampling: Assign a number to every patient in the population and use a random number generator to select the ones to follow. This gives every patient the same chance of being chosen.
Step 2 — Method two, stratified sampling: Divide the patients into groups first — by age band, by hospital, or by how soon they began treatment — then take a simple random sample from within each group so every group is represented.
Answer: Take a simple random sample from each group. One way is by assigning a number to each patient and using a random number generator to randomly select patients; another is to stratify by hospital or age band first and then sample randomly within each stratum.
Problem 17. Suppose that the first researcher conducted his survey by randomly choosing one state in the nation and then randomly picking 40 patients from that state. What sampling method would that researcher have used?
Solution
Step 1 — Look at the first stage: One state is chosen at random from the whole nation. The state is a naturally occurring group, and only one of them enters the study.
Step 2 — Look at the second stage: 40 patients are then chosen at random from inside that one state.
Step 3 — Name it: Selecting whole groups at random and then sampling within them is cluster sampling — here, a two-stage version, since the researcher samples inside the chosen cluster rather than taking everybody in it.
Answer: Cluster sampling, with the state as the cluster. Note the weakness: every patient in the study comes from a single state, so anything unusual about that state's climate, healthcare, or population is baked into the results.
Problem 18. Suppose that the second researcher conducted his survey by choosing 40 patients he knew. What sampling method would that researcher have used? What concerns would you have about this data set, based upon the data collection method?
Solution
Step 1 — Find the selection rule: The researcher chose 40 patients he already knew.
Step 2 — Name the method: Selecting the people who are easiest to reach is convenience sampling, and it involves no chance mechanism.
Step 3 — State the concern: People a researcher already knows are likely to be similar to one another — same city, same hospital, similar background — and they were not selected in a way that gives every patient a chance to appear. Any conclusion about the wider population of patients would be unreliable.
Answer: This would be convenience sampling and is not random. The concern is bias: the 40 patients are a self-contained circle rather than a cross-section of patients, so the sample cannot support conclusions about all patients with the virus.
Two researchers are gathering data on hours of video games played by school-aged children and young adults. They each randomly sample different groups of 150 students from the same school. They collect the data in Table 1.2.11 and Table 1.2.12. Use this setting for the next five problems.
| Hours Played per Week | Frequency | Relative Frequency | Cumulative Relative Frequency |
|---|---|---|---|
| 0–2 | 26 | 0.17 | 0.17 |
| 2–4 | 30 | 0.20 | 0.37 |
| 4–6 | 49 | 0.33 | 0.70 |
| 6–8 | 25 | 0.17 | 0.87 |
| 8–10 | 12 | 0.08 | 0.95 |
| 10–12 | 8 | 0.05 | 1 |
| Hours Played per Week | Frequency | Relative Frequency | Cumulative Relative Frequency |
|---|---|---|---|
| 0–2 | 48 | 0.32 | 0.32 |
| 2–4 | 51 | 0.34 | 0.66 |
| 4–6 | 24 | 0.16 | 0.82 |
| 6–8 | 12 | 0.08 | 0.90 |
| 8–10 | 11 | 0.07 | 0.97 |
| 10–12 | 4 | 0.03 | 1 |
Problem 19. Give a reason why the data may differ.
Solution
Step 1 — Note what is the same: Both researchers sampled 150 students from the same school, so the population is identical.
Step 2 — Note what is different: The two samples contain different students. Gaming habits vary a lot from student to student, so two random draws of 150 will land on different mixes of heavy and light players.
Answer: The data may differ simply because of sampling variability — two different groups of 150 students were surveyed, and no two samples from the same population give identical results.
Problem 20. Would the sample size be large enough if the population is the students in the school?
Solution
Step 1 — Identify the population: The students in the one school.
Step 2 — Compare it to the sample size: 150 students drawn randomly from a single school is a substantial share of that school and, more importantly, was drawn by a random method from exactly the group of interest.
Answer: Yes, the sample size of 150 would be large enough to reflect a population of one school.
Problem 21. Would the sample size be large enough if the population is school-aged children and young adults in the United States?
Solution
Step 1 — Identify the population: All school-aged children and young adults in the United States — tens of millions of people, spread across every region, income level, and school type.
Step 2 — Check whether the sample can represent it: The 150 students all come from one school. No matter how carefully they were randomized within that school, no student from any other school had any chance of being selected.
Answer: No. The problem is not really the number 150 — it is that the sample is drawn from a single school, so it cannot represent the nationwide population no matter how large it gets.
Problem 22. Researcher A concludes that most students play video games between four and six hours each week. Researcher B concludes that most students play video games between two and four hours each week. Who is correct?
Solution
Step 1 — Check each researcher's own data: In Table 1.2.11 the largest relative frequency is 0.33 in the 4–6 hour bin, so Researcher A's conclusion follows from A's data. In Table 1.2.12 the largest is 0.34 in the 2–4 hour bin, so Researcher B's conclusion follows from B's data.
Step 2 — Ask which one describes the population: Both conclusions are supported by their own sample, and neither sample is obviously flawed. We have no basis for picking a winner.
Answer: Even though the specific data support each researcher's conclusion, the different results suggest that more data need to be collected before the researchers can reach a conclusion.
Problem 23. As part of a way to reward students for participating in the survey, the researchers gave each student a gift card to a video game store. Would this affect the data if students knew about the award before the study?
Solution
Step 1 — Identify the incentive: The reward is a gift card to a video game store — a prize that is more attractive to students who play a lot of video games.
Step 2 — Trace the effect on who participates: If students knew about the reward in advance, heavy gamers would be more eager to take part, and the sample would tilt toward high hours played.
Step 3 — Trace the effect on what they report: Students might also exaggerate their hours, believing that heavier play makes them a better fit for the study.
Answer: Yes. Announcing a video-game-store gift card beforehand is a form of undue influence: it changes who volunteers and may change what they report, biasing the results upward. Awarding the card only after participation, without announcing it, avoids this.
A pair of studies was performed to measure the effectiveness of a new software program designed to help stroke patients regain their problem-solving skills. Patients were asked to use the software program twice a day, once in the morning and once in the evening. The studies observed 200 stroke patients recovering over a period of several weeks. The first study collected the data in Table 1.2.13. The second study collected the data in Table 1.2.14. Use this setting for the next five problems.
| Group | Showed improvement | No improvement | Deterioration |
|---|---|---|---|
| Used program | 142 | 43 | 15 |
| Did not use program | 72 | 110 | 18 |
| Group | Showed improvement | No improvement | Deterioration |
|---|---|---|---|
| Used program | 105 | 74 | 19 |
| Did not use program | 89 | 99 | 4 |
Problem 24. Given what you know, which study is correct?
Solution
Step 1 — Compare the two study designs: Both observed 200 stroke patients, both split them into users and non-users, and both reported more improvement among users.
Step 2 — Ask what would settle it: To call one study "correct" we would need to know how patients were assigned to the two groups, whether the groups were comparable to begin with, and how "improvement" was measured. None of that is given.
Answer: There is not enough information given to judge if either one is correct or incorrect.
Problem 25. The first study was performed by the company that designed the software program. The second study was performed by the American Medical Association. Which study is more reliable?
Solution
Step 1 — Ask who benefits from each result: The first study was run by the company that designed and sells the software. A favorable result is worth money to them.
Step 2 — Check the second party: The American Medical Association has no financial stake in whether the software sells.
Answer: The second study — the one performed by the American Medical Association — is more reliable, because the first is a self-funded, self-interest study by the party that profits from a positive finding. This does not automatically make the company's study wrong; it means it should be read with extra care.
Problem 26. Both groups that performed the study concluded that the software works. Is this accurate?
Solution
Step 1 — Check the first study's numbers: 142 of 200 program users improved, against 72 of 200 non-users. A large gap.
Step 2 — Check the second study's numbers: 105 of 198 users improved, against 89 of 192 non-users. A much smaller gap, but still in the same direction.
Step 3 — Weigh the sources: The smaller gap comes from the independent study, so it is the more trustworthy estimate of the effect.
Answer: The software program seems to work, because the second study shows that more patients improve while using the software than not. Even though the difference is not as large as in the first study, the results from the second study are likely more reliable and still show improvement.
Problem 27. The company takes the two studies as proof that their software causes mental improvement in stroke patients. Is this a fair statement?
Solution
Step 1 — Separate association from causation: Both studies show that program users improved more often than non-users. That is an association between two variables.
Step 2 — Look for what else could explain it: Patients were not randomly assigned to use the program. Perhaps healthier or more motivated patients were the ones who used it twice a day, and their better outcomes were going to happen anyway.
Answer: No, it is not a fair statement. A relationship between using the software and improving does not establish that the software causes the improvement. Only a randomized controlled experiment — where patients are assigned to use the program or not by chance — could support a causal claim.
Problem 28. Patients who used the software were also a part of an exercise program whereas patients who did not use the software were not. Does this change the validity of the conclusions from Problem 1.2.26?
Solution
Step 1 — Identify the second difference between the groups: Program users were also in an exercise program; non-users were not. So the two groups differ in two ways, not one.
Step 2 — Name the problem: The effects of the software and the effects of the exercise cannot be separated. That is confounding.
Answer: Yes, because we cannot tell if the improvement was due to the software or the exercise; the data is confounded, and a reliable conclusion cannot be drawn. New studies should be performed in which the only difference between the groups is the software.
Problem 29. Is a sample size of 1,000 a reliable measure for a population of 5,000?
Solution
Step 1 — Compute the share sampled:
$$\frac{1{,}000}{5{,}000} = 0.20 = 20\%$$Step 2 — Judge the size: One person in five is a large sample by any standard, and 1,000 observations is well above the 1,200–1,500 that pollsters consider ample even for populations of millions.
Step 3 — Add the condition: Size only helps if the 1,000 were chosen randomly. A biased method at this scale would still be unreliable.
Answer: Yes, a sample of 1,000 is a reliable measure for a population of 5,000 — provided the 1,000 were selected by a random method rather than by convenience or self-selection.
Problem 30. Is a sample of 500 volunteers a reliable measure for a population of 2,500?
Solution
Step 1 — Check the size: 500 out of 2,500 is 20% of the population — numerically ample.
**Step 2 — Check how they were obtained:** They are volunteers. Volunteers choose themselves, so people with strong feelings about the topic are over-represented and everyone else is under-represented.
Step 3 — Recall the key asymmetry: Sample size fixes random error, not bias.
Answer: No. Even though the sample is large enough, the fact that the sample consists of volunteers makes it a self-selected sample, which is not reliable.
Problem 31. A question on a survey reads: "Do you prefer the delicious taste of Brand X or the taste of Brand Y?" Is this a fair question?
Solution
Step 1 — Read the wording carefully: "Do you prefer the delicious taste of Brand X or the taste of Brand Y?" One option carries a flattering adjective; the other does not.
Step 2 — Name the problem: Loading one option with positive language nudges respondents toward it. This is undue influence built into the instrument itself.
Answer: No, it is not a fair question. A neutral version — "Do you prefer the taste of Brand X or the taste of Brand Y?" — treats both options identically and does not steer the answer.
Problem 32. Is a sample size of two representative of a population of five?
Solution
Step 1 — Compute the share: Two out of five is 40% of the population — a large fraction.
Step 2 — Look at the absolute number: Two responses is still only two responses. With so few observations, one unusual individual moves the result enormously.
Step 3 — Note the cheap alternative: The population is only five people. Surveying all five costs almost nothing and removes the question entirely.
Answer: No. Even though the sample is a large portion of the population, two responses are not enough to justify any conclusions. Because the population is so small, it would be better to include everyone in the population to get the most accurate data.
Problem 33. Is it possible for two experiments to be well run with similar sample sizes to get different data?
Solution
Step 1 — Recall what a well-run experiment guarantees: Good design controls bias — it does not, and cannot, eliminate chance.
Step 2 — Apply it: Two well-run experiments still observe two different sets of individuals, and individuals vary. The results will differ.
Answer: Yes. Two well-run experiments with similar sample sizes can easily produce different data, because sampling variability is present in every study no matter how carefully it is conducted. That is precisely why results are replicated before they are believed.
For the following exercises, identify the type of data that would be used to describe a response (quantitative discrete, quantitative continuous, or qualitative), and give an example of the data.
Problem 34. number of tickets sold to a concert
Problem 35. percent of body fat
Problem 36. favorite baseball team
Problem 37. time in line to buy groceries
Problem 38. number of students enrolled at Evergreen Valley College
Problem 39. most-watched television show
Problem 40. brand of toothpaste
Problem 41. distance to the closest movie theatre
Problem 42. age of executives in Fortune 500 companies
Solutions 34–42
Problem 34
Step 1 — Ask count or measure: Tickets are counted one at a time.
Step 2 — Check the possible values: A concert sells 150 tickets, never 150.4.
Answer: quantitative discrete — for example, 150 tickets sold.
Problem 35
Step 1 — Ask count or measure: Body fat percentage comes from a measuring device, not from counting.
Step 2 — Check the possible values: A reading can be 18%, 18.4%, or 18.42% — any level of precision the instrument supports.
Answer: quantitative continuous — for example, 18.4% body fat.
Problem 36
Step 1 — Ask what is recorded: A team name — a word, not a number.
Step 2 — Apply the rule: Team names place people into categories, and there is no meaningful arithmetic on them.
Answer: qualitative — for example, the Oakland A's.
Problem 37
Step 1 — Ask count or measure: Time in line is read off a clock, so it is measured.
Step 2 — Check the possible values: A wait can be 4 minutes, 4.5 minutes, or 4.52 minutes.
Answer: quantitative continuous — for example, 4.5 minutes in line.
Problem 38
Step 1 — Ask count or measure: Enrollment is a head count.
Step 2 — Check the possible values: A college enrolls 11,234 students; a fraction of a student is not possible.
Answer: quantitative discrete — for example, 11,234 students.
Problem 39
Step 1 — Ask what is recorded: The name of a television show.
Step 2 — Apply the rule: Show titles are category labels, not quantities. (The ratings would be quantitative, but the show itself is not.)
Answer: qualitative — for example, the evening news.
Problem 40
Step 1 — Ask what is recorded: A brand name.
Step 2 — Apply the rule: Brands sort people into categories described by words.
Answer: qualitative — for example, Crest.
Problem 41
Step 1 — Ask count or measure: Distance is measured, not counted.
Step 2 — Check the possible values: The theatre could be 3 miles away, 3.2 miles, or 3.24 miles.
Answer: quantitative continuous — for example, 3.2 miles.
Problem 42
Step 1 — Ask count or measure: Age is measured on a continuous time scale, even though we usually round it to whole years.
Step 2 — Check the possible values: An executive can be 47 years old, 47.3 years old, or 47.32 years old. Rounding to "47" is a reporting convention, not a restriction on the underlying quantity.
Answer: quantitative continuous — for example, 47.3 years.
Key Terms
qualitative (categorical) data — data that categorize or describe attributes, generally recorded as words or letters.
quantitative data — numerical data resulting from counting or measuring an attribute.
quantitative discrete data — quantitative data that come from counting, taking only certain numerical values.
quantitative continuous data — quantitative data that come from measuring, which may include fractions, decimals, or irrational numbers.
frequency — the count of individuals in a category.
relative frequency — the proportion or percent of individuals in a category.
pie chart — a circular display in which each category is a wedge proportional to its percent of the whole.
bar graph — a display in which each category's bar length is proportional to its number or percent.
Pareto chart — a bar graph whose bars are sorted from largest to smallest.
simple random sample — a sample chosen so that every group of the same size is equally likely to be selected.
stratified sample — a sample formed by dividing the population into strata and taking a proportionate number from each.
cluster sample — a sample formed by dividing the population into clusters, randomly selecting some clusters, and taking every member of those clusters.
systematic sample — a sample formed by choosing a random starting point and then taking every \(k\)th item from a list.
convenience sample — a non-random sample made up of results that are readily available.
sampling with replacement — sampling in which a selected member is returned to the population and may be chosen again.
sampling without replacement — sampling in which a member of the population may be chosen only once.
sampling error — error caused by the process of sampling itself, such as a sample that is too small.
nonsampling error — error caused by factors unrelated to sampling, such as a faulty measuring device.
sampling bias — bias created when some members of the population are less likely to be chosen than others.
confounding — when the effects of two or more factors on a response cannot be separated.
nonresponse bias — bias arising when those who do not respond differ meaningfully from those who do, so the returned responses no longer represent the sample drawn.
voluntary response bias — bias arising when people select themselves into a sample by choosing to respond, over-representing those who felt strongly.