View All Pages

Statistical Conclusion Validity Notes

Internal vs. External Validity

  • Internal validity relates to how credible the study results are, within the context actually examined. It addresses the question, how credible is the claim made by the results of this analysis?
  • External validity relates to how well the results of a study or analysis can be applied to other situations. It addresses the question, how widely, or to what other contexts, can we apply the results of this study?

Statistical Conclusion Validity

Internal vs. External Validity

  • Internal validity relates to how credible the study results are, within the context actually examined. It addresses the question, how credible is the claim made by the results of this analysis?
  • External validity relates to how well the results of a study or analysis can be applied to other situations. It addresses the question, how widely, or to what other contexts, can we apply the results of this study?

Statistical conclusion validity

Statistical conclusion validity is a special subset of internal validity that deals specifically with ways that statistical results may not reflect reality.
The following are threats to statistical conclusion validity:

  • Low statistical power: There are not enough observations in the study to detect an effect.
  • Violated assumptions of statistical tests: You ran a test that was not appropriate to the data.
  • Fishing and the error rate problem: You ran multiple models (generally seeking a specific result) but did not adjust the math for the changes in error rates this “fishing” produced.
  • Unreliability of measures: You measured the variables in a manner that allowed more variation than reflects reality.
  • Restriction of range: You measured the variables in a manner that allowed less variation than reflects reality.
  • Unreliability of treatment implementation: The treatment variable (cause; “x”) was not implemented in the same manner for all units, but this variability was not accounted for in the measurement.
  • Extraneous variance in the experimental setting: There are other variables that may have influenced the result but these were not accounted for in the model.
  • Heterogeneity of units: There are variations across units that are unaccounted for in the model.
  • Inaccurate effect size estimation: The statistic is inherently faulty or biased in some way.

 

More information about evaluation/research validity (including text about statistical conclusion validity) can be found here:

Download Validity.pdf

Here is the height data for discussing how some statistics are flawed/biased: biased statistics.xlsx Download biased statistics.xlsx