B.1 Sampling Experiment
Learning Objectives
By the end of this section, you will be able to:
- demonstrate the simple random, systematic, stratified, and cluster sampling techniques on a single real population;
- describe each procedure you used in enough detail that a classmate could repeat it and get a comparably valid sample;
- explain why the same population, sampled four different ways, yields four different-looking samples — and what each design protects against.
B.1.1 Stats Lab: Sampling Experiment
This section is a lab, not a reading. In §1.2 you met four random sampling designs on paper. Here you will run all four of them yourself, on the same population, and see with your own eyes that they do not produce the same sample — and that this is not a flaw in any of them.
Class Time:
Names:
You will be asked to pick several random samples of restaurants. In each case, describe your procedure briefly — including how you generated your random numbers — and then list the restaurants in the sample you obtained.
A list of fifteen restaurant names tells a reader almost nothing on its own. What makes a sample defensible is the rule that produced it: whether every restaurant could have been picked, whether you chose the starting point before you looked at the list, whether you wrapped around when you ran off the bottom. When this lab asks you to "describe your procedure," it is not asking you to show your work for partial credit — it is asking for the only part of your answer that another statistician can actually check.
Keep a record of what you actually did, not what you meant to do. If you drew a number twice and had to redraw, if you rounded a stratum size up rather than down, if you gave up on a random number generator and used a die instead — write it down. Reporting the departures honestly is part of doing statistics ethically; quietly smoothing them over is where a great many published results have gone wrong.
B.1.2 The Restaurant Population
The table below is your population: 84 restaurants across seven cities in California's Santa Clara Valley. It has been arranged so that two different groupings are visible at once. The restaurants are stratified by city into rows, and grouped horizontally by entree cost — the four cost bands act as a second set of strata, and each city's row acts as a natural cluster.
| City | Under $10 | $10 to under $15 | $15 to under $20 | Over $20 |
|---|---|---|---|---|
| San Jose | El Abuelo Taq, Pasta Mia, Emma's Express, Bamboo Hut | Emperor's Guard, Creekside Inn | Agenda, Gervais, Miro's | Blake's, Eulipia, Hayes Mansion, Germania |
| Palo Alto | Senor Taco, Olive Garden, Taxi's | Ming's, P.A. Joe's, Stickney's | Scott's Seafood, Poolside Grill, Fish Market | Sundance Mine, Maddalena's, Spago's |
| Los Gatos | Mary's Patio, Mount Everest, Sweet Pea's, Andele Taqueria | Lindsey's, Willow Street | Toll House | Charter House, La Maison Du Cafe |
| Mountain View | Maharaja, New Ma's, Thai-Rific, Garden Fresh | Amber Indian, La Fiesta, Fiesta del Mar, Dawit | Austin's, Shiva's, Mazeh | Le Petit Bistro |
| Cupertino | Hobees, Hung Fu, Samrat, Panda Express | Santa Barb. Grill, Mand. Gourmet, Bombay Oven, Kathmandu West | Fontana's, Blue Pheasant | Hamasushi, Helios |
| Sunnyvale | Chekijababi, Taj India, Full Throttle, Tia Juana, Lemon Grass | Pacific Fresh, Charley Brown's, Cafe Cameroon, Faz, Aruba's | Lion & Compass, The Palace, Beau Sejour | |
| Santa Clara | Rangoli, Armadillo Willy's, Thai Pepper, Pasand | Arthur's, Katie's Cafe, Pedro's, La Galleria | Birk's, Truya Sushi, Valley Plaza | Lakeside, Mariani's |
Before you sample anything, count. The city totals are San Jose 13, Palo Alto 12, Los Gatos 9, Mountain View 12, Cupertino 12, Sunnyvale 13, and Santa Clara 13. The cost-band totals are 28 restaurants under $10, 24 from $10 to under $15, 18 from $15 to under $20, and 14 over $20. Both sets add to the same 84, which is the arithmetic check that you have read the table correctly.
Nothing about these 84 restaurants forces you to group them by city rather than by price. The table shows both groupings at once precisely so you can see that "stratum" is a choice the researcher makes, not a property the population has. Stratify by city and you guarantee geographic coverage; stratify by price and you guarantee you hear from cheap and expensive places alike. You cannot do both at once with a single stratified sample — which is why the lab makes you do it twice and compare.
There are no Sunnyvale restaurants in the "Over $20" band. That empty cell is not a typo, and it matters: a stratified sample by city can still reach Sunnyvale, but a stratified sample by cost will never reach Sunnyvale through its most expensive stratum. Real populations have empty cells, and a sampling design that assumes they don't will quietly mislead you.
To sample, you first need a way to number the restaurants. Read the table left to right, top to bottom, and assign 1 through 84 — El Abuelo Taq is 1, Pasta Mia is 2, and so on through Mariani's at 84. Write your numbering down before you draw anything. If you number as you go, you will unconsciously number the restaurants you have already decided to include.
Using the reading order just described — left to right, then top to bottom — which restaurant carries label 14, and which city is it in? Then say how many labels the San Jose row uses in total.
Solution
Step 1 — walk San Jose's row. Reading left to right: 4 restaurants under $10 (labels 1–4), 2 in the $10–$15 band (5–6), 3 in the $15–$20 band (7–9), and 4 over $20 (10–13).
$$ 4 + 2 + 3 + 4 = 13 $$Step 2 — cross into the next row. San Jose has consumed labels 1 through 13, so label 14 is the first entry of the Palo Alto row's first cell: Senor Taco.
Answer: label 14 is Senor Taco, in Palo Alto; San Jose uses 13 labels, 1 through 13. Do this bookkeeping before you sample, not while you sample. A numbering built on the fly is a numbering you can nudge, and once the labels can move, the random number generator is no longer making the decision — you are.
B.1.3 A Simple Random Sample
A sample chosen so that every group of \(n\) members of the population is equally likely to be the one selected. Operationally: number the population \(1\) through \(N\), use a random number generator to produce \(n\) distinct labels, and take exactly the members those labels name — no substitutions, no skipping a restaurant because you have already picked two on the same street.
If you close your eyes and stab at the table fifteen times, you have not taken a simple random sample. Your finger favors the middle of the page, the longer cells, the names you recognize. A random number generator has no such preferences, which is exactly why we hand the decision to one. The generator is not there to make the work easier — it is there to remove you from the selection.
Pick a simple random sample of 15 restaurants.
The population is numbered 1 through 84, so you need 15 whole numbers drawn from that range. Duplicates are the one thing you have to handle: if the generator hands you 41 twice, discard the repeat and draw again, because this lab samples without replacement — a restaurant cannot appear in your sample twice.
Try it in rāSHio
You do not need a graphing calculator to draw these 15 labels. Open rāSHio and choose File → Random Numbers…, then set Min 1, Max 84, How many 15, and tick No repeats — that last box is what makes this a sample without replacement, so you never have to spot a duplicate by eye. The same dialog draws your systematic starting position (How many 1), each stratum’s picks (Max = that stratum’s size), and your two clusters (Min 1, Max 7, How many 2).
Figure B.1.1 — Drawing the lab’s random labels in rāSHio: File → Random Numbers…
You number the 84 restaurants and your random number generator returns these labels, in this order: 41, 7, 41, 63, 12, 84, 29. Which restaurants have you selected so far, and how many more labels do you still need?
Solution
Step 1 — spot the duplicate. The label 41 appears twice. Because this is sampling without replacement, the second 41 is discarded and does not count toward the sample.
Step 2 — count the distinct labels. 41, 7, 63, 12, 84, 29 — that is six distinct restaurants selected.
Step 3 — find what is left to draw.
$$ 15 - 6 = 9 $$Answer: six restaurants are selected (labels 7, 12, 29, 41, 63, 84), and you still need 9 more distinct labels. Notice what you must not do here: you may not decide to keep the duplicate "because 41 came up twice, so it must be important," and you may not replace it with 42 "because that's close." Both substitutions break the rule that every group of 15 was equally likely, and once that rule is broken there is no way to repair it after the fact.
- Describe your procedure.
- Complete the table with your sample.
| No. | Restaurant |
|---|---|
| 1. | |
| 2. | |
| 3. | |
| 4. | |
| 5. | |
| 6. | |
| 7. | |
| 8. | |
| 9. | |
| 10. | |
| 11. | |
| 12. | |
| 13. | |
| 14. | |
| 15. |
B.1.4 A Systematic Sample
A sample built by listing the population, randomly choosing one starting position, and then taking every \(k\)-th member from there, where
$$ k = \frac{\text{number of individuals in the population}}{\text{number of individuals needed in the sample}} $$When you run off the end of the list, wrap around to the top and keep counting.
Pick a systematic sample of 15 restaurants.
With \(N = 84\) and \(n = 15\), the step size is
$$ k = \frac{84}{15} = 5.6 \approx 5 $$so you will take every fifth restaurant. Draw one random number between 1 and 84 to be your starting position, then count forward by five, wrapping to the top of the list when you pass 84.
In the simple random sample you made fifteen independent random choices. Here you make exactly one — the starting position — and the remaining fourteen restaurants are then fully determined by arithmetic. That is what makes systematic sampling so easy to carry out and so hard to fudge; it is also what makes it vulnerable in a way simple random sampling is not. If the list has a repeating pattern whose period lines up with \(k\), your sample will hit the same kind of item over and over. Glance at the ordering before you commit to a step size, and say what you saw when you report your results.
Try it in rāSHio
Labels come out of the generator in the order they were drawn, which is the worst possible order for looking restaurants up in the table. Paste them into rāSHio and choose Transform → Sort to put them in ascending order — then you can walk the table once, top to bottom, instead of hunting for each label separately. Sorting changes nothing about which restaurants are in your sample; it only changes the order you read them off in.
Figure B.1.2 — Putting the drawn labels in order in rāSHio: Transform → Sort.
Your random starting position is 79, and your step size is \(k = 5\). List the first six positions your systematic sample marks.
Solution
Step 1 — start where the draw told you. Position 79 is the first mark. Every position after this is arithmetic, not a new random draw.
Step 2 — add 5 each time, wrapping at 84. 79, then 84. The next would be 89, which runs off the end of an 84-item list, so subtract the list length:
$$ 89 - 84 = 5 $$Step 3 — continue from the wrapped position. 5, 10, 15, 20.
79, 84, 5, 10, 15, 20
Answer: the first six marks are positions 79, 84, 5, 10, 15, 20. The wrap is not a fudge or a correction — it is part of the procedure, and it is what keeps a late starting position from producing a short sample. Because 5 and 84 share no common factor, stepping by 5 will walk through all 84 positions before it ever returns to 79, so you will never mark the same restaurant twice.
Your classmate Rocío Vargas renumbers the population city by city, and every city happens to hold exactly 12 restaurants listed cheapest-first. They then take a systematic sample with step size \(k = 12\), starting at position 3. Which restaurants get marked, and what has gone wrong?
Solution
Step 1 — mark the positions. Starting at 3 and stepping by 12: 3, 15, 27, 39, 51, 63, 75.
Step 2 — locate those positions inside the cities. With 12 restaurants per city, positions 1–12 are city 1, positions 13–24 are city 2, and so on. Position 3 is the third restaurant of city 1; position 15 is the third of city 2; position 27 is the third of city 3.
$$ 3,\ 15,\ 27,\ 39,\ 51,\ 63,\ 75 \;\longrightarrow\; \text{the 3rd restaurant of every city} $$Step 3 — name the problem. Because the list is ordered cheapest-first within each city, "the third restaurant" always means "one of the cheapest few." The sample reaches all seven cities but never leaves the bottom of the price range.
Answer: the sample marks the third restaurant of each city — every one of them cheap — so it will badly underestimate the average entree price. This is periodicity: the step size lined up with a repeating pattern in the list, so the "randomness" of the one starting draw got locked into a single position within each repeat. Rocío did nothing incorrectly, which is exactly why it is dangerous — their arithmetic is right and their sample is still biased. The remedy is to look at the ordering before choosing \(k\), and to say in your write-up what ordering you sampled from.
- Describe your procedure.
- Complete the table with your sample.
| No. | Restaurant |
|---|---|
| 1. | |
| 2. | |
| 3. | |
| 4. | |
| 5. | |
| 6. | |
| 7. | |
| 8. | |
| 9. | |
| 10. | |
| 11. | |
| 12. | |
| 13. | |
| 14. | |
| 15. |
B.1.5 A Stratified Sample by City
A sample that guarantees each subgroup of interest is represented: divide the population into groups called strata, then take a separate simple random sample from within each stratum, sized so that each stratum contributes its fair share.
A stratified sample is less free than a simple random sample: many groups of 20 restaurants can never be selected, because the design forbids any sample that takes six from San Jose and none from Los Gatos. You are trading away some randomness on purpose, in exchange for a guarantee that every city gets a voice. That trade is worth making when the strata genuinely differ from one another — and pointless when they don't.
Pick a stratified sample, by city, of 20 restaurants. Use 25% of the restaurants from each stratum, rounding to the nearest whole number.
Here the strata are the seven city rows. Take 25% of each city's count, round, and draw that many restaurants at random from inside that city — never across cities.
Hannah Whitfield is filling in her lab sheet and needs the city allocations. Compute the 25% allocation for each city, round each to the nearest whole number, and total them. Does she get the 20 restaurants the lab asked for?
Solution
Step 1 — take 25% of each city's count.
$$ \begin{aligned} \text{San Jose: } & 0.25 \times 13 = 3.25 \\ \text{Palo Alto: } & 0.25 \times 12 = 3 \\ \text{Los Gatos: } & 0.25 \times 9 = 2.25 \\ \text{Mountain View: } & 0.25 \times 12 = 3 \\ \text{Cupertino: } & 0.25 \times 12 = 3 \\ \text{Sunnyvale: } & 0.25 \times 13 = 3.25 \\ \text{Santa Clara: } & 0.25 \times 13 = 3.25 \end{aligned} $$Step 2 — round each to the nearest whole number. 3, 3, 2, 3, 3, 3, 3.
Step 3 — add them up.
$$ 3 + 3 + 2 + 3 + 3 + 3 + 3 = 20 $$Answer: yes — Hannah gets 20 restaurants, exactly as asked. But look at what rounding cost her. The unrounded allocations sum to \(0.25 \times 84 = 21\), so rounding lost a restaurant. Every city that landed on \(.25\) got rounded down, and Los Gatos — the smallest stratum — lost the largest share of its allocation. This is the ordinary, unavoidable friction of stratified sampling: the design promises proportional representation, and integer arithmetic then breaks that promise by a little. Report which way you rounded; don't let the reader assume you got exactly 25% of every city.
Los Gatos is allocated 2 restaurants out of its 9. Describe exactly how you would draw those 2, and then explain why "take the first two Los Gatos restaurants in the table" is not an acceptable substitute.
Solution
Step 1 — number within the stratum only. Label the 9 Los Gatos restaurants 1 through 9, reading the row left to right: Mary's Patio, Mount Everest, Sweet Pea's, Andele Taqueria, Lindsey's, Willow Street, Toll House, Charter House, La Maison Du Cafe.
Step 2 — draw 2 distinct labels from 1 to 9 with a random number generator, redrawing on a repeat.
Step 3 — take exactly those two. You never look outside the Los Gatos row while filling the Los Gatos allocation; that is what makes it a stratified sample rather than a simple random one.
Answer: number 1–9 within the row, draw 2 distinct labels at random, and take those restaurants. "The first two in the table" fails because the row is ordered by entree cost — Mary's Patio and Mount Everest are both in the under-$10 band, so that shortcut would guarantee your Los Gatos contribution is the cheapest end of the town every single time. Stratifying protected you from missing Los Gatos; it does nothing to protect you from bias inside Los Gatos. That protection has to come from the random draw within the stratum, and it is the step students most often skip.
- Describe your procedure.
- Complete the table with your sample.
| No. | Restaurant | No. | Restaurant |
|---|---|---|---|
| 1. | 11. | ||
| 2. | 12. | ||
| 3. | 13. | ||
| 4. | 14. | ||
| 5. | 15. | ||
| 6. | 16. | ||
| 7. | 17. | ||
| 8. | 18. | ||
| 9. | 19. | ||
| 10. | 20. |
B.1.6 A Stratified Sample by Entree Cost
Now stratify the very same 84 restaurants a second way. Pick a stratified sample, by entree cost, of 21 restaurants — again 25% of each stratum, rounded to the nearest whole number.
The strata are now the four cost bands: 28 restaurants under $10, 24 from $10 to under $15, 18 from $15 to under $20, and 14 over $20. Draw at random from within each column of the table.
Both samples take 25% of the same 84 restaurants, so both "should" be 21. The city version came out to 20 only because seven strata each shed a fraction in the rounding, while the cost version has four strata that happen to round back to 21. Nothing about the population changed between the two samples — only the grid you laid over it. When a sample size lands somewhere you did not expect, look first at how many strata you cut the population into.
Kai Xiong runs a neighbourhood food blog with his husband Devon, and he wants his sample to reach every price range fairly. Compute the 25% allocation for each of the four cost strata. Two of them land on a half. Show that his four allocations can total 21, and explain what he had to decide along the way.
Solution
Step 1 — take 25% of each cost stratum.
$$ \begin{aligned} \text{Under \$10: } & 0.25 \times 28 = 7 \\ \text{\$10 to under \$15: } & 0.25 \times 24 = 6 \\ \text{\$15 to under \$20: } & 0.25 \times 18 = 4.5 \\ \text{Over \$20: } & 0.25 \times 14 = 3.5 \end{aligned} $$Step 2 — notice the tie. Two strata land exactly on a half, and "round to the nearest whole number" does not tell you which way to go. That is a decision you have to make and disclose, not a fact the arithmetic hands you.
Step 3 — check the total either way. Round 4.5 up and 3.5 down: \(7 + 6 + 5 + 3 = 21\). Round 4.5 down and 3.5 up: \(7 + 6 + 4 + 4 = 21\). Round both up and you get 22; round both down and you get 20.
Answer: Kai's four allocations total 21 provided he rounds one half up and the other half down — and he must say which he chose. This is the whole point of the "describe your procedure" instruction. A reader who knows he rounded the $15–$20 band up can judge whether his sample over-represents mid-priced restaurants; a reader who is only handed the list of 21 names cannot.
- Describe your procedure.
- Complete the table with your sample.
| No. | Restaurant | No. | Restaurant |
|---|---|---|---|
| 1. | 12. | ||
| 2. | 13. | ||
| 3. | 14. | ||
| 4. | 15. | ||
| 5. | 16. | ||
| 6. | 17. | ||
| 7. | 18. | ||
| 8. | 19. | ||
| 9. | 20. | ||
| 10. | 21. | ||
| 11. |
B.1.7 A Cluster Sample
A sample built by dividing the population into groups called clusters, taking a simple random sample of the clusters themselves, and then including every individual in the chosen clusters.
Pick a cluster sample of restaurants from two cities. The number of restaurants will vary.
Here each city row is one cluster. Randomly select two of the seven cities, then take every restaurant in those two cities — you do not sample within a cluster at all.
Both start by cutting the population into groups. Stratified sampling then takes a few members from every group, so no group is missed. Cluster sampling takes every member of a few groups, so most groups are missed entirely. Stratifying protects against under-covering a subgroup; clustering trades that protection away to make the fieldwork cheap — you only have to travel to two cities instead of seven. If the clusters differ substantially from each other, that trade will cost you accuracy, and you should say so.
Yasmin Haddad draws her two clusters and gets Los Gatos and Cupertino. How many restaurants are in her sample, and what fraction of the population is that? Compare the result to what she would have gotten had the draw returned San Jose and Sunnyvale instead.
Solution
Step 1 — count every restaurant in the chosen clusters. Los Gatos holds 9 restaurants and Cupertino holds 12. Every one of them is in the sample; there is no sampling within a cluster.
$$ 9 + 12 = 21 $$Step 2 — express it as a fraction of the population.
$$ \frac{21}{84} = 0.25 = 25\% $$Step 3 — try the other draw. San Jose holds 13 and Sunnyvale holds 13, giving \(13 + 13 = 26\) restaurants, or \(\frac{26}{84} \approx 0.31 = 31\%\).
Answer: Yasmin's Los Gatos–Cupertino draw gives 21 restaurants (25%); the San Jose–Sunnyvale draw would have given 26 restaurants (31%). That is the defining awkwardness of cluster sampling: her sample size was not something she chose, it is something the draw handed her. Notice too that the Los Gatos–Cupertino sample contains no Sunnyvale restaurants at all, so it cannot tell her anything about Sunnyvale prices — and a reader who only sees "we sampled 25% of the population" would never guess that five of the seven cities are missing entirely.
- Describe your procedure.
- Complete the table with your sample.
| No. | Restaurant | No. | Restaurant |
|---|---|---|---|
| 1. | 14. | ||
| 2. | 15. | ||
| 3. | 16. | ||
| 4. | 17. | ||
| 5. | 18. | ||
| 6. | 19. | ||
| 7. | 20. | ||
| 8. | 21. | ||
| 9. | 22. | ||
| 10. | 23. | ||
| 11. | 24. | ||
| 12. | 25. | ||
| 13. |
Marcus Adeyemi has both samples in front of him: the stratified-by-city sample takes 20 of the 84 restaurants, and the Los Gatos–Cupertino cluster sample takes 21 — nearly the same size. How many of the seven cities does each one reach, and what should he conclude about comparing two samples on size alone?
Solution
Step 1 — count the cities each design reaches. The stratified-by-city design allocates at least 2 restaurants to every stratum, so it reaches all 7 cities. The cluster design takes every restaurant in exactly 2 cities and nothing from the other five.
Step 2 — put the two side by side.
$$ \text{stratified: } 20 \text{ restaurants from } 7 \text{ cities} \qquad \text{cluster: } 21 \text{ restaurants from } 2 \text{ cities} $$Step 3 — say what size does and does not tell you. The cluster sample is larger, and it is the one that cannot say anything about Mountain View, Palo Alto, San Jose, Santa Clara, or Sunnyvale.
Answer: stratified reaches 7 cities, cluster reaches 2 — and the cluster sample is the bigger of the two. Marcus should conclude that a sample size on its own is not evidence of coverage. Two samples of nearly identical size can answer completely different questions, and the difference lives entirely in the design that produced them. This is why "we surveyed 21 restaurants" is not a finding, and "we surveyed every restaurant in two randomly chosen cities" is.
B.1.8 Discussion Questions
Answer these with your group. Every one of them is a judgement call, so what matters is not the answer you land on but whether you can point at your own five samples to defend it.
- Lay your five samples side by side. Which cities appear in all five? Which appear in only one? What does that pattern tell you about the design that missed them?
- Suppose you wanted to estimate the average entree price across all 84 restaurants. Which of your five samples would you trust most for that purpose, and why?
- Your cluster sample almost certainly missed several cities entirely. Is that a flaw in cluster sampling, or is it the price of something cluster sampling buys you? What does it buy?
- In the stratified samples you had to round. Did your rounding systematically favour large strata or small ones? Would a reader of your results be able to tell?
- Which of the four procedures would be hardest to carry out honestly if you were tired, in a hurry, and nobody was checking your work? Does that make it a worse design, or just one that needs a better record?
Key Terms
Cluster sampling — a method for selecting a random sample by dividing the population into groups (clusters) and using simple random sampling to select a set of clusters; every individual in the chosen clusters is included in the sample.
Population — all individuals, objects, or measurements whose properties are being studied; here, all 84 restaurants in the table.
Random sampling — a method of selecting a sample that gives every member of the population an equal chance of being selected.
Representative sample — a subset of the population that has the same characteristics as the population.
Sample — a subset of the population studied.
Sampling bias — the situation in which not all members of the population are equally likely to be selected.
Sampling with replacement — once a member of the population is selected for inclusion in a sample, that member is returned to the population before the next selection.
Sampling without replacement — a member of the population may be chosen for inclusion in a sample only once; if chosen, the member is not returned to the population before the next selection.
Simple random sampling — a straightforward method for selecting a random sample; give each member of the population a number, then use a random number generator to select a set of labels that identify the members of your sample.
Stratified sampling — a method for selecting a random sample used to ensure that subgroups of the population are represented adequately; divide the population into groups (strata) and use simple random sampling to identify a proportionate number of individuals from each stratum.
Stratum — one of the groups a population is divided into for stratified sampling; here, either a city or an entree-cost band.
Systematic sampling — a method for selecting a random sample; list the members of the population, use simple random sampling to select a starting point, let \(k\) equal the population size divided by the sample size, and choose every \(k\)-th individual from the starting point, returning to the beginning of the list if necessary.