Randomness and Survey Sampling

00:01 Welcome to Lecture 9, in which we'll address Randomness and Survey Sampling.

00:05 So randomness, you hear about this a lot but what’s the big deal about random sampling? We often emphasize in statistics that the sample must be drawn randomly, but why? Well, random sampling has two key advantages.

00:18 First of all, nobody can guess the outcome before it happens and we want things to be fair, so we want outcomes to be equally likely.

00:27 By doing this, we ensure that separate outcomes do not affect each other, and in other words, outcomes of separate draws are independent.

00:36 Random sampling is actually fairly hard to do, but there’s several software packages that do it for you including Excel, SAS, R and several internet widgets.

00:46 Let’s look at sample surveys. There are three key ideas in survey sampling.

00:52 The first is to examine a part of the whole.

00:54 Next, we randomize, and the big key is it is the sample size - the sample size is paramount in survey sampling.

01:04 In this section, what we’re gonna talk about our types of sampling, bad samples and consequences, and we’ll also address how we can prevent bad sampling.

01:13 Examining part of a whole. Why do we sample? Well, the goal is to learn about an entire population of individuals, but the problem is it’s usually impractical or even impossible to examine all individuals.

01:27 The solution is to draw a sample from the population.

01:31 We do this every day.

01:34 For example, if you’re interested in knowing how a pot of soup that you’re making taste, you’re not gonna eat the whole pot, that'd be mean because you’ve been making a big pot soup, you’re probably going to feed more than one person, so you don’t wanna eat the whole thing.

01:47 You take a spoonful or two and then you trust that these spoonfuls are representative of how the pot taste.

01:53 Here the population would be the whole pot of soup and the sample is just the two spoonfuls.

02:00 In sample surveys, what we do is we give them to a small group of people in hopes that they will give an insight into the views of the entire population.

02:09 Examples of these are opinion polls, election surveys such as how do you plan to vote and exit polls once you leave the polls, how did you vote? The “survey”, in the soup example, is asking how “does the soup taste?” And the hope is that the taste of the two spoonfuls of soup that you took are representative of how the entire pot tastes.

02:32 Bias is a big problem in many surveys and this happens when the sample is not representative of the population.

02:40 How do we avoid these problems? Well, the main goal of survey sampling is to get a good sense of what the views are in the entire population.

02:49 In order to do this, we try to get a sample that represents the population.

02:54 Sampling methods often over or under-emphasize some characteristics of the population, and these types of samples are said to be biased.

03:03 For example, in 1936 of presidential election poll for the election between Alf Landon and Franklin Delano Roosevelt was conducted.

03:11 Literary Digest used a sample of 2.4 million ballots from 10 million that they mailed out, but some problems occurred with that.

03:20 The result of the poll was that Literary Digest predicted that Alf Landon would win the election with 52% of the popular vote to 43% for Franklin Delano Roosevelt, but Roosevelt won the election with 62% of the popular vote to 37% for Landon.

03:36 What happened? But what Literary Digest did was they used the phone book to choose addresses to send the polls to, but in 1936, right after the depression, phones were luxury so the poll sampled more wealthy people than poor voters and the result was that the sample was not representative of the population.

03:56 They totally undercovered the poor part of the population.

04:00 Modern polls get around this problem because they get representative samples by selecting the individuals in the sample at random.

04:07 When we do random sampling, what we’re doing, if we go back to the soup analogy, is we’re stirring the pot. How do we get a representative sample? Well, suppose we add salt to the pot and then we just taste the soup from the top without stirring the pot or without stirring the salt in first.

04:25 Well, if you just take the spoonfuls from the top, then that might lead you to believe that the soup is salty because the top part of the soup is where you put the salt, and in order to put the salt in and randomize the pot, you need to stir it up.

04:40 Stirring the pot in survey sampling is done by randomizing.

04:44 How do we get a representative sample? Well, randomizing protects us from the influence of many features of the population by ensuring that on average, the sample looks a lot like the population.

04:56 This is a lot easier than trying to match the sample to the population because we can’t account for every characteristic of the population and we can’t usually match the sample to the population for each of these characteristics.

05:09 In survey sampling, it’s all about the sample size and the question is often, “is our sample large enough?” How large of a random sample do we need for the sample to be reasonably representative of the population? A common thought is that we need a large percentage of the population.

05:25 But this isn’t really true, the population size doesn’t really matter.

05:30 All that matters is the number of individuals that are in our sample.

05:33 For example, if we take a sample of 100 students at a particular college, that represents the student body about as well as a sample of 100 voters represent the electorate in a particular state, even though the population size is likely to be vastly different.

05:49 One way to get information about the population is to take a census.

05:55 This is where we ask everyone.

05:57 Formally, a census is a sample that includes everyone in the population, so we just look at everybody. There are lot of problems with the census.

06:07 First of all, a census can be really impractical or even impossible to carry out.

06:12 Populations change, babies are born, people die and so there are people that are entering and leaving the population all the time, and it can be very complex to do.

06:22 Surprisingly maybe, a census may not give us accurate information about the population as a random sample would, and this goes back to the second point.

06:31 By sampling everybody, we’re not accounting for changes in the population but random samples remain representative of the population.

06:38 Let’s look at the population parameters and sample statistics.

06:42 In the example, we’re trying to figure out how the population feels about a particular issue or what percentage of population feels a particular way about a certain issue.

06:52 This percentage is known as the population parameter.

06:55 This is an unknown quantity that pertains to the population.

06:59 We use the percentage from the sample to estimate the population parameter.

07:03 This is an example of what we call a sample statistic.

07:07 So to help you remember this, just remember that the piece go together, parameters pertain to the population, statistics pertain to the sample.

07:14 A sample statistic, formally, is any summary or quantity calculated from the sample data.

07:21 Other parameters that we’ve dealt with before were the population mean and a population standard deviation in the section where we talk about the normal distribution.

07:29 What’s the bottomline? The bottomline is that we draw samples because we can’t work with the entire population, but we want the sample to be representative of the population.

07:39 How do we select a representative sample? Well, we're gonna discuss four ways to do it.

07:45 The simplest is the simple random sample. We'll discuss stratified sampling.

07:52 We'll discuss cluster and multistage sampling, and we’ll discuss systematic sampling.

07:57 All these methods provide a mechanism for selection of a representative sample from the population.

08:03 Let’s start with the simple random sample. What is it and how do we get it? Well, a simple random sample is a sampling scheme that has two key properties.

08:13 First, every possible sample of the same size has the same chance of being selected.

08:19 Secondly, every individual has the same chance of being selected.

08:23 In other words, every combination of individuals has the same chance of being selected.

08:29 There’s not one sample that’s more likely than another under simple random sampling.

08:34 In order to do it, we need to first define where the sample comes from, and we call this a sampling frame.

08:41 The sampling frame is a list of individuals from which the sample is drawn, and the easiest way to use this is to draw a simple random sample is to assign a number to each element in the sampling frame and then, randomly chooses a set of numbers to comprise our sample, where each number is equally likely.

08:57 Samples drawn at random in variable going to differ from one another, so you draw one sample, you draw another one, they’re going to be different.

09:05 This leads to different values of statistics for each sample that we draw.

09:09 And the sample-to-sample differences are known as sampling variability.

09:14 This variability is not a problem, it’s just the nature of the beast, and it doesn’t mean our sample is not representative of the population.

09:22 Let’s do an example of a simple random sample.

09:25 Suppose we have 80 students enrolled in an introductory statistics course, and we wanna sample 5 of them.

09:31 We’ll assign each one a number, 1 to 80, no repeats, and we’re gonna draw 5 of them at random using some sort of random number generator, placing numbers into a bowl and shaking the bowl up, and the numbers we get out are 24, 13, 2, 74 and 54.

09:47 The students with these numbers are the individuals in the simple random sample.

09:51 The next type of sampling we’re gonna look at is stratified sampling, and this is -- this boils down to sampling from different groups.

10:00 Sometimes we need a more complicated sampling scheme in order to draw a representative sample from a large population, and one way to handle this is is to slice the population into homogeneous groups called strata, and then we carry out simple random sampling in each stratum and then the samples are combined, and this is what we know as a stratified sample.

10:20 Let’s look at an example of a stratified sample.

10:24 At a large university, there’s a question about how people feel about their football team.

10:29 The campus is 60% men, 40% women. We wanna sample a hundred people.

10:34 We might wanna do a stratified sample here.

10:37 If we do a simple random sample, we could end up with 80 men and 20 women.

10:42 This is not representative of the population, men are over-represented, women are under-represented.

10:47 We’re gonna divide the population into strata by sex and then we’re gonna take simple random samples of 60 men and 40 women from each of those groups and then, combine the results of these two simple random samples.

11:00 These reduces the sampling variability and this is the most important benefit of stratified sampling.

11:07 We also have cluster and multistage sampling, and we’ll do this one by example.

11:15 Supposed wanna assess the reading level of a book based on the length of the sentences in that book.

11:20 How might we do it? Well, simple random sampling might be awkward if we number each sentence and then find, say, like the 242nd sentence and the 3965th sentence in the book.

11:32 We might consider picking a few pages at random and then count the lengths of the sentences on those pages, and that works if we believe that each page is representative of the entire book, in terms of reading level.

11:45 Splitting the population into representative clusters can be more practical.

11:49 We choose a few clusters and then perform a census in each of them.

11:53 This is called cluster sampling.

11:55 To formally describe a cluster sample, let’s talk about how we collect it.

12:01 Clusters are usually selected for reasons of practicality, efficiency, or cost.

12:06 If each cluster represents the population fairly, then cluster sampling will provide a representative sample.

12:13 We have to be very careful in how we choose the cluster, however, because we wanna avoid introducing bias into our sample and we’ll talk more about bias here in a few minutes.

12:23 This is not the same stratified sampling, stratified sampling is done by dividing the population into groups based on one or more characteristics and then doing simple random sampling in each group.

12:34 Strata are not representative of the population. Clusters are.

12:39 Sometimes we do multistage sampling and this is where we combine sampling methods together.

12:44 For example, in the reading level example, maybe we think that the reading level increases as the book goes on because the concepts get harder.

12:53 We would wanna avoid samples selected heavily from the early parts of the book or from the later parts of the book.

12:59 Supposed the book has 7 parts, we might randomly choose one chapter from each of the 7 parts, and then randomly select a few pages from each of those chapters.

13:09 We may then randomly select a few sentences from each of those pages.

13:13 Let’s look a little bit closer on how this is working.

13:17 The sampling scheme is as follows: we first stratified by the part of the book.

13:22 We randomly chose a chapter from each part.

13:25 In each chapter, we chose pages as clusters, and then for each clusters, we did a simple random sample of sentences in that cluster.

13:36 There were three stages of the sampling here, we have the stratified first, then the cluster sampling, and then the simple random sample to finish it off, and this is an example of a multistage sampling scheme.

13:49 One more method of sampling that we can use to obtain a representative sample of the population is known as systematic sampling, and the idea is, for instance, we wanna survey every 20th person from an alphabetical list of students.

14:03 What we would have to do is to randomly choose the starting point in order to get a representative sample.

14:09 This makes the order of the list unassociated in any way with the responses that we want and this type of scheme is known as a systematic sample.

14:17 Now, let’s look at the problems we might run into a surveys and the desirable properties of surveys.

14:24 We want valid surveys, which means we’re getting the information we hope to get.

14:28 Does valid survey provides the information we’re hoping to obtain, or in other words, that answers the questions that we’re trying to ask? Four questions we need to ask before we set out to survey.

14:39 What do I wanna know? Am I asking the right people? Am I asking the right questions? What would I do with the answers if I had them, do they address the things that I wanna know? Common pitfalls in surveys, you need to know what you wanna know, what do you hope to learn? You need to use the right sampling frame.

15:02 In order to get a representative sample, the sampling frame has to match the population.

15:08 You don’t wanna ask questions that you don’t need to know the answer to, and you wanna ask specific questions instead of general ones.

15:18 For instance, what was your grade in a class? Is a much better question to ask than, how did you do in that class? You wanna ask for quantitative results when possible.

15:28 You wanna be careful in how you phrase your questions.

15:31 Often, it’s a good idea to do a trial on the survey that you eventually plan to give to a larger group, so that you can help identify problems before you survey too many people and have to deal with all of that information.

15:44 This type of survey is known as a pilot study.

15:47 Let's look at what’s not to do? Common errors in survey sampling.

15:53 Don’t sample volunteers, under any circumstances, don’t use volunteers as your sample.

15:58 A volunteer response sample is a sample in which a large group of people are invited to respond and those who choose to respond are counted.

16:06 There are many problems with this, but the biggest ones are that, people who have really strong feelings about the questions at hand are the ones who are most likely to respond and this leads to bias in your survey.

16:18 Internet surveys are the biggest culprits of this, like the ones that go on a news website and on espn.com, for example, are volunteer response samples cuz they just give the question and if you visit the website, you can answer the question, this is a volunteer response sample.

16:34 It’s really hard to define a sampling frame for these, so in general it’s best not to use them.

16:39 Don’t sample because of convenience.

16:42 When we talk about convenient sample, what we mean is a sampling which we include only individuals who are convenient for us to sample, and these are usually biased as well.

16:53 For example, if we go to the grocery store and we’re wondering about the percentage of the population that likes vegetables, and we stand at the produce section of the grocery store and we just ask people who come by whether or not they like vegetables.

17:05 Well, you’re probably not gonna get a representative sample because the people who don’t like vegetables are probably not going to the produce section.

17:13 This is a convenience sample, because it’s convenient to sample the people who come through the place where you're standing.

17:19 This is probably going to be a biased sample because it leaves out the people who don’t buy produce at the grocery store or people who don’t come thorugh the produce section, at all.

17:28 We don’t wanna use a bad sampling frame.

17:31 Again, this is very important -- you must make sure that your sampling frame matches your population.

17:36 For example, simple random sample from an incomplete sampling frame introduces bias since there are parts of the population that might not be represented, such as like people in prison, homeless people or students that may be missed.

17:49 We need to be careful of undercoverage.

17:51 Undercoverage is just the under-representation of a particular group of the population in the sample.

17:57 For example, a survey about the satisfaction with a particular product ordered online that is only offered to the person who ordered it may under-represent some groups of the population.

18:08 Often products are ordered online that other people use besides the person who ordered it, sometimes given as gifts, so it’s gonna under-represent the population of people who are actually using that product, and this is an example of undercoverage.

18:23 Several types of bias that affect our surveys, one is non-response bias.

18:29 Not everyone who's asked to complete a survey will do so, you might run out of time, you might just not feel like you're doing it.

18:35 Those who do not respond might differ from those who do and possibly just on variables of interest.

18:41 We also have response bias, and this comes from anything in the survey that can influence the responses.

18:47 The biggest offenders for response bias are the wording of the questions.

18:52 For example, questions about prior illegal or embarrassing activities, those suffer response bias because people don’t typically wanna admit to that, especially to somebody who’s just out there, giving them a survey and never met this person before, so they don’t wanna answer the questions honestly.

19:08 How do we prevent bias? Well, there are ways to do it.

19:13 We look for biases in any survey that we encounter, and there’s no way to come back from a biased sampling method.

19:19 You can’t take a large enough sample size to overcome a bad sampling scheme.

19:23 We need to spend time and resources reducing biases.

19:27 This should be our biggest focus when we construct our survey.

19:31 We need to think about members of the population who could have been excluded from the study.

19:36 If possible, you wanna do a pilot study, and you always wanna report your sampling methods in detail, that way, if any questions come up about how the survey was conducted, they’re easily answered.

19:49 Right. What we’ve done, we’ve talked about different types of sampling, why we randomize, how to prevent bad samples, what makes a bad sample and problems in sampling that we can avoid if we do it correctly.

20:02 This is the end of Lecture 9, and we’ll see you back here for Lecture 10.