External Validity

00:01 Let's talk about validity.

00:02 There are in general two types of validity, two large categories.

00:06 The first is internal validity and that's when, in the context of my study itself, what I observe is true, only within the context of my study.

00:17 So the relationships between variables that I've observed and measured is real, in that case, my study is internally valid.

00:25 External validity however is when I take the results from my study and try to apply it to the real word, that's external validity.

00:32 Internal is within the study; external, outside the study.

00:36 To consider internal validity we have to ask ourselves could there be alternative causes or explanations for the relationship between a risk factor and an outcome that I have observed? We have to ask that question.

00:50 If there can be other possibilities that reduces the internal validity that I think my study might have.

00:57 Certain requirements exist for internal validity and these may seem familiar to you, because there're three of them are from the Bradford Hill criteria.

01:06 The first of temporality.

01:08 Does the exposure that I've observed come before the outcome that I've observed, that's important? The strength, the relative risk, the odds ratios, is it strong? If it's strong there's a heightened likelihood that my study is internally valid.

01:21 And is it biologically plausible? Does it make sense scientifically? If so, there's a higher likelihood that my study is internally valid.

01:29 Things that may challenge internal validity include confounding.

01:34 You may recall what a confounder is, a cofounder is a variable that either creates an artificial or spurious relationships between exposure and outcome, or it's a variable that masks a real relationship, you have to control for confounding in order to maximize our chances of good internal validity.

01:52 Also selection bias.

01:53 If I am choosing my subjects in such a way that I cannot be guaranteed that the relationship I'm observing is real, that's selection bias.

02:05 External validity again is when we are trying to generalize from our sample, our study, to the rest of the world, to the reference population.

02:13 So high external validity means I can generalize; poor external validity means I probably shouldn't generalize.

02:21 So some of the things that challenge external validity include observer effects.

02:26 What's an observer effect? That's when the person doing the observing, the investigator, is somehow changing the outcome.

02:34 On our lecture about RCTs and biases, we talked about the Hawthorne effect.

02:39 You may recall, the Hawthorne effect is when subjects behave differently simply because they're being observed or they're in a study, that's an observer effect.

02:49 That's a challenge to external validity because suddenly, the people in my sample aren't behaving like the rest of the world.

02:57 Also, we like to have as few exclusion criteria as possible. Can you see why? If I'm choosing the individuals in my study or the conditions of my study in a very exact specific way, I am heightening the chances that my study does not resemble the external world, that's not good, I'd like to have as few as possible.

03:18 Only the absolute required ones we're going to include.

03:22 The major threats to external validity are overly specific study characteristics.

03:27 Again, that has to do with our Parsimonious criteria, like we don't like our study to be so specific that it no longer resembles the real world.

03:36 And we talk about Hawthorne Effect, as well Rosenthal Effect.

03:39 You may recall that Rosenthal Effect is when the investigator can elicit a change in the study behavior.

03:46 Ecological validity is a little harder to describe.

03:49 Let's say you're running a sociological or criminal law experiment or study, and you're trying to determine if in a mock-jury environment you can have the same jury outcomes as in a real-life criminal jury? To do this you have people pretending to be jury members and they're reading out the testimony from a real-life criminal cases from the past, and you're trying to see if your mock-jury will have the same results as the real jury.

04:13 That's great except reading out the content or testimony from a real trial is not the same as sitting in the jury booth in a criminal court listening to testimony, so the environment is different, that's when ecological validity has been violated.

04:30 It doesn't mean the results won't be good, it just means it's one more thing to consider when assessing whether or not your study can extrapolated to the rest of the world.

04:39 Some final thoughts to think about. This is a difficult thing to grasp, so think about it deeply.

04:45 An invalid study can be reliable. Okay, that's pretty heavy, let's think about it again.

04:51 That means that a study that has no internal validity, meaning the relationships that I've observed between the variables in my study population might not be real, it can still be reliable, because I might still observe it many times. Why do I observe it many times? Because the conditions that cause them to be unreal, like selection bias, like really extreme exclusion criteria, are still applied every time I do the study, so it's not valid, but it's reproducible.

05:22 Okay, on the other hand an unreliable study cannot be valid.

05:26 If I cannot reproduce the same results reliably, there is no way that the relationships that I observed in my study are real, simply not possible.

05:36 However, a relationship that does not represent the real world, in other words, an invalid relationship can be seen multiples times in the study.

05:44 I.E. Reliable. In other words, a reliable test can be invalid.

05:49 You can have invalid test, one that in which the relationships between variables are not real, but see it many times.

05:56 For example, let's say we're trying to measure intelligence by measuring the heads of subjects which is completely ridiculous.

06:03 There is no relationship between head size and intelligence.

06:06 You would get the same results all the time, because head sizes are not changing, so our results are reliable, they're reproducible, but they're invalid because on its basis, the relationship between head size and intelligence doesn't exist.

06:21 If that's the case, and for liability being present, but validity not being present.