External Validity

by Raywat Deonandan, PhD

My Notes
  • Required.
Save Cancel
    Learning Material 2
    • PDF
      Slides 12 ValidityReliability Epidemiology.pdf
    • PDF
      Download Lecture Overview
    Report mistake

    00:00 External validity again is when we are trying to generalize from our sample, our study, to the rest of the world, to the reference population. So high external validity means I can generalize; poor external validity means I probably shouldn't generalize.

    00:18 So some of the things that challenge external validity include observer effects, what's an observer effect? That's when the person doing the observing, the investigator, is somehow changing the outcome. On our lecture about RCTs and biases, we talked about the Hawthorne effect. You may recall a Hawthorne effect is when subjects behave differently simply because they are being observed while they are in a study, that's an observer effect. That's a challenge to external validity because suddenly the people in my sample aren't behaving like the rest of the world. Also we'd like to have as few exclusion criteria as possible, can you see why? If I'm choosing the individuals in my study or the conditions of my study in a very exact specific way, I am heightening the chances that my study does not resemble the external world. That's not good. We'd like to have as few as possible, only the absolute required ones we are going to include. The major threats to external validity are overly specific study characteristics. Again that has to do with our parsimonious criteria, we don't like our study to be so specific that it no longer resembles the real world.

    01:31 And we talk about Hawthorne effect and as well, Rosenthal effect. You may recall that Rosenthal effect is when the investigator can elicit a change in their study behavior.

    01:43 Ecological validity is a little harder to describe. Let's say you're running a sociological or criminal law experiment or study and you trying to determine if in a mock jury environment you can have the same jury outcomes as in a real life criminal jury. To do this you have people pretending to be jury members and they are reading out the testimony from real life criminal cases from the past and you're trying to see if your mock jury will have the same results as the real jury. That's great, except reading out the content or testimony from a real trial is not the same as sitting in the jury booth in a criminal court listening to testimony, so the environment is different, that's when ecological validity has been violated.

    02:25 It doesn't mean the results won’t be good, it just means it's one more thing to consider when assessing whether or not your study can be extrapolated to the rest of the world.

    02:36 Some final thoughts to think about and this is a difficult thing to grasp, so think about it deeply. An invalid study can be reliable. Okay that's pretty heavy, let's think about it again. That means that a study that has no internal validity, meaning the relationships that I observe between the variables in my study population might not be real, it can still be reliable, because I might still observe it many times. Why might I observe it many times? Because the conditions that cause them to be unreal, like selection bias, like really extreme exclusion criteria, are still applied every time I do the study, so it's not valid, but it's reproducible. Okay, on the other hand, an unreliable study cannot be valid.

    03:23 if I cannot reproduce the same results reliably, there is no way that the relationships that I observe in my study are real, simply not possible. However, a relationship that does not represent the real world, in other words an invalid relationship, can be seen multiple times in a study, i.e. is reliable. In other words a reliable test can be invalid. You can have an invalid test, one that in which the relationships between variables are not real, but see it many times. For example, let's say we're trying to measure intelligence by measuring the heads of subjects, which is completely ridiculous. There is no relationship between head size and intelligence. You would get the same results all the time, because head sizes are not changing. So our results are reliable, they are reproducible, but they are invalid, because on its basis, the relationship between head size and intelligence doesn't exist, so that's a case of reliability being present but validity not being present.

    About the Lecture

    The lecture External Validity by Raywat Deonandan, PhD is from the course Validity and Reliability.

    Included Quiz Questions

    1. Yes
    2. No
    1. Hawthorne effect
    2. Randomization
    3. Inclusion criteria
    4. Population sampling
    5. Quality of measurement scales

    Author of lecture External Validity

     Raywat Deonandan, PhD

    Raywat Deonandan, PhD

    Customer reviews

    5,0 of 5 stars
    5 Stars
    4 Stars
    3 Stars
    2 Stars
    1  Star