Replication and Reproducibility in Psychological, Medical and other Sciences
Main Article Content
Abstract
As there are no universal constants in psychological, medical and economic sciences, only constructive-phenomenon replications are meaningful. Yet, psychologists continue to perform direct replications, as evidenced by recent preregistered multilab attempts at exact replications of the ego depletion effect. Statistics are driving the replication movement into a ditch because of an overemphasis on the determination of statistical magnitude of effects while ignoring commonsense magnitude and other criteria for evaluating phenomena’s validity, reliability, and viability. The nature of the human mind and the variability of psychological phenomena pose difficult challenges for the scientific method and insurmountable obstacles for precise replications in psychological sciences. The situation is no better in medical and economic sciences. The interaction effect of person (genetics) and environment (lifestyle) calls for constructive replications to determine, for example, drugs’ efficacy as a function of group and individual differences. The vaccine-vaccination paradox is an interesting case because psychological and medical sciences meet at this intersection. In all fields, science advances by theory building and model expansion, not by replication tests of statistical hypotheses. Rigorous logical and theoretical analysis always precedes and guides good empirical tests. The nonexistence of an effect is not viable if it can withstand rigorous logical and theoretical analyses. Empirical studies are mainly evaluated for their theoretical relevance and importance, not their success or failure to exactly reproduce the original findings.
Article Details
How to Cite
ISO-AHOLA, Seppo E..
Replication and Reproducibility in Psychological, Medical and other Sciences.
Medical Research Archives, [S.l.], v. 10, n. 2, feb. 2022.
ISSN 2375-1924.
Available at: <https://esmed.org/MRA/mra/article/view/2702>. Date accessed: 25 apr. 2024.
doi: https://doi.org/10.18103/mra.v10i2.2702.
Section
Research Articles
The Medical Research Archives grants authors the right to publish and reproduce the unrevised contribution in whole or in part at any time and in any form for any scholarly non-commercial purpose with the condition that all publications of the contribution include a full citation to the journal as published by the Medical Research Archives.
References
1. Gleiser M. The island of knowledge. 2014. New York, NY: Basic Books.
2. Iso-Ahola S. Reproducibility in psychological science: When do psychological phenomena exist? Frontiers in Psychology. 2017; 8 (879): 1-16.
3. Anderson C, Bahnik S, Barnett-Cowan M, Bosco F, Chandler J, Chartier C, et al. Response to comment on “Estimating the reproducibility of psychological science”. Science. 2016; 351: 1037.
4. Earp B, Trafimow D. Replication, falsification, and the crisis of confidence in social psychology. Frontiers in Psychology. 2015; 6: 621.
5. Iso-Ahola S. Replication and the establishment of scientific truth. Frontiers in Psychology. 2020; 11 (2183): 1-15.
6. Rubin M. What type of Type I error? Contrasting the Neyman-Pearson and Fisherian approaches in the context of exact and direct replications. Synthese. 2019; 196: 1-26.
7. Stroebe W, Strack, F. The alleged crisis and the illusion of exact replication. Perspectives in Psychological Science. 2014; 9: 59-71.
8. Baumeister R, Bratslavsky E, Muraven M, Tice D. Ego depletion: Is the active self a limited resource? Journal of Personality and Social Psychology. 1998; 74: 1252-1265.
9. Vohs K, Schmeichel B, Lohmann S, Gronau Q, Finley A, Ainsworth S. et al. A multiple preregistered paradigmatic test of the ego-depletion effect. Psychological Science. Psychological Science. 2021; 32: 1566-1581.
10. Carter E, Kofler L, Forster D, McCullough M. A series of meta-analytic tests of the depletion effect: Self-control does not seem to rely on a limited resource. Journal of Experimental Psychology: General. 2015; 144: 796-815.
11. Hagger M, Chatzisarantis N, Zwienenberg M. A multilab preregistered replication of the ego depletion effect. Perspectives on Psychological Science. 2016; 11: 546-573.
12. Hagger M, Wood C, Stiff C, Chatzisarantis N. Ego depletion and the strength model of self-control: A meta-analysis. Psychological Bulletin. 2010; 136: 495-525.
13. Iso-Ahola S. Conscious versus nonconscious mind and leisure. Leisure Sciences. 2015; 37: 289-310.
14. Enkavi A, Eisenberg I, Bissett P, Mazza G, MacKinnon D, Marsch L, et al. Large-scale analysis of test-retest reliabilities of self-regulation measures. Proceedings of National Academy of Sciences USA. 2019; 116: 5472-5477.
15. Hussey I, Hughes S. Hidden invalidity among 15 commonly used measures in social personality psychology. Advanced in Methods and Practice in Psychological Science. 2020; 3: 166-184.
16. Prentice D, Miller D. When small effects are impressive. Psychological Bulletin. 1992; 112: 160-164.
17. Funder D, Ozer D. Evaluating effect size in psychological research: Sense and nonsense. Advances in Methods and Practices in Psychological Science. 2019; 2: 156-168.
18. Meehl P. Appraising and amending theories: The strategy of Lakatosian defense and two principles that warrant it. Psychological Inquiry. 1990; 1: 108-141.
19. Iso-Ahola S. Exercise: Why it is a challenge for both the nonconscious and conscious mind. Review of General Psychology. 2013; 17: 93-110.
20. Iso-Ahola S. Conscious-nonconscious processing explains why some people exercise but most don’t. Journal of Nature and Science. 2017; 3 (e384): 1-16.
21. Iso-Ahola S. Human mind: Both the cause and solution to the global pandemic of physical inactivity. International Journal of Public Health Research. 2018; 6: 107-113.
22. Kahneman D. Thinking, fast and slow. 2014. New York, NY: Farrar, Straus and Giroux.
23. Trafimow D. Hypothesis testing and theory evaluation at the boundaries. Psychological Review. 2003; 110: 526-535.
24. McFall R. Making psychology incorruptible. Applied and Preventive Psychology. 1996; 5: 9-15.
25. Popper K. The logic of scientific discovery. 1959. London: Hutchison.
26. Lykken D. Statistical significance in psychological research. Psychological Bulletin. 1968; 70: 151-159.
27. Ackerman P. Nonsense, common sense, and science of expert performance: Talent and individual differences. Intelligence. 2014; 45: 6-17.
28. Bargh J. The cognitive unconscious in everyday life. 2021. In A Reber, R Allen (Eds), The cognitive unconscious. London: Oxford University Press.
29. Cumming G. The new statistics: Why and how. Psychological Science. 2014; 25: 7-29.
30. Cohen J. Statistical power analysis for the behavioral sciences. 1988. Hillsdale, NJ: Erlbaum.
31. Wetzels R, Matzke D, Lee M, Rouder J, Iverson G, Wagenmakers E-J. Statistical evidence in experimental psychology: An empirical comparison using 855 t tests. Perspectives on Psychological Science. 2011; 6: 291-298.
32. Kass R, Raftery A. Bayes factors. Journal of American Statistical Association. 1995; 90: 377-395.
33. Dienes Z. Bayesian versus orthodox statistics: Which side are you on? Perspectives on Psychological Science. 2011; 6: 274-290.
34. Bargh J. The historical origins of priming as the preparation of behavioral responses: Unconscious carry-over and contextual influences of real-world importance. In D Molden (Ed), Understanding priming effects in social psychology (pp. 218-233). New York, NY: Guilford Press.
35. Baumeister R, Tice D, Bushman B. A review of multi-site replication projects in social psychology: Methodological ideal or collective self-destruct mechanism? 2022. Submitted for publication.
36. Churchland M, Afshar A, Shenoy K. A central source of movement variability. Neuron.2006; 52: 1085-1096.
37. Bargh J, Hassin R. Human unconscious processes in situ: The kind of awareness that really matters. 2021. In A Reber, R Allen (Eds), The cognitive unconscious. London: Oxford University Press.
38. Baumeister R. Conquer yourself, conquer the world. Scientific American. 2015; 312: 61-65.
39. Inzlicht M, Schmeichel B. What is ego depletion? Toward a mechanistic revision of the resource model of self-control. Perspectives on Psychological Science. 2012; 7: 450-463.
40. Crandall C, Sherman J. On the scientific superiority of conceptual replications for scientific progress. Journal of Experimental Social Psychology. 2015; 66: 93-99.
41. Dang J, Barker P, Baumert A, Bentvelzen M, Berkman E, Buchholz N, et al. A multilab replication of the ego depletion effect. Social Psychological and Personality Science. 2021; 12: 14-24.
42. Garrison K, Finley A, Schmeichel B. Ego depletion reduces attention control: Evidence from two high-powered preregistered experiments. Personality and Social Psychology Bulletin. 2019; 45: 728-739.
43. Bryan C, Walton G, Rogers T, Dweck C. Motivating voter turnout by invoking the self. Proceedings of National Academy of Sciences USA. 2011; 108: 12653-12656.
44. Bryan C, Yeager D, O’Brien J. Replicator degrees of freedom allow publication of misleading “failures to replicate”. Unpublished manuscript. 2019. University of Chicago.
45. Ramscar M, Shaoul C, Baayen R. Why many priming effects don’t (and won’t) replicate: A quantitative analysis. Unpublished manuscript. 2015. Tubingen University, Germany.
46. Chen X, Latham G, Piccolo R, Itzchakov G. An enumerative review and a meta-analysis of primed goal effects on organizational behavior. Applied Psychology. 2021; 70: 216-253.
47. Weingarten E, Chen Q, McAdams M, Yi J, Hepler J, Albarracin D. From primed concepts to action: A meta-analysis of the behavioral effects of incidentally presented words. Psychological Bulletin. 2016; 142: 472-497.
48. Bargh J. The hidden life of the consumer mind. Consumer Psychology Review. 2021; 5: 1-16.
49. Hofmann W, Vohs K, Baumeister R. What people desire, feel conflicted about, and try to resist in everyday life. Psychological Science. 2012; 23: 582-588.
50. Kool W, Botvinick M. A labor/leisure tradeoff in cognitive control. Journal of Experimental Psychology: General. 2014; 143: 131-141.
51. Amrhein V, Greenland S, McShane B. Retire statistical significance. Nature. 2019; 567: 305-307.
52. Greenland S, Senn S, Rothman K, Carlin J, Poole C, Goodman S. et al. Statistical tests, p values, confidence intervals, and power: A guide to misinterpretations. European Journal of Epidemiology. 2016; 31: 337-350.
53. Iso-Ahola S, Dotson C. Psychological momentum--not a statistical but psychological phenomenon. Review of General Psychology. 2015; 19: 112-116.
2. Iso-Ahola S. Reproducibility in psychological science: When do psychological phenomena exist? Frontiers in Psychology. 2017; 8 (879): 1-16.
3. Anderson C, Bahnik S, Barnett-Cowan M, Bosco F, Chandler J, Chartier C, et al. Response to comment on “Estimating the reproducibility of psychological science”. Science. 2016; 351: 1037.
4. Earp B, Trafimow D. Replication, falsification, and the crisis of confidence in social psychology. Frontiers in Psychology. 2015; 6: 621.
5. Iso-Ahola S. Replication and the establishment of scientific truth. Frontiers in Psychology. 2020; 11 (2183): 1-15.
6. Rubin M. What type of Type I error? Contrasting the Neyman-Pearson and Fisherian approaches in the context of exact and direct replications. Synthese. 2019; 196: 1-26.
7. Stroebe W, Strack, F. The alleged crisis and the illusion of exact replication. Perspectives in Psychological Science. 2014; 9: 59-71.
8. Baumeister R, Bratslavsky E, Muraven M, Tice D. Ego depletion: Is the active self a limited resource? Journal of Personality and Social Psychology. 1998; 74: 1252-1265.
9. Vohs K, Schmeichel B, Lohmann S, Gronau Q, Finley A, Ainsworth S. et al. A multiple preregistered paradigmatic test of the ego-depletion effect. Psychological Science. Psychological Science. 2021; 32: 1566-1581.
10. Carter E, Kofler L, Forster D, McCullough M. A series of meta-analytic tests of the depletion effect: Self-control does not seem to rely on a limited resource. Journal of Experimental Psychology: General. 2015; 144: 796-815.
11. Hagger M, Chatzisarantis N, Zwienenberg M. A multilab preregistered replication of the ego depletion effect. Perspectives on Psychological Science. 2016; 11: 546-573.
12. Hagger M, Wood C, Stiff C, Chatzisarantis N. Ego depletion and the strength model of self-control: A meta-analysis. Psychological Bulletin. 2010; 136: 495-525.
13. Iso-Ahola S. Conscious versus nonconscious mind and leisure. Leisure Sciences. 2015; 37: 289-310.
14. Enkavi A, Eisenberg I, Bissett P, Mazza G, MacKinnon D, Marsch L, et al. Large-scale analysis of test-retest reliabilities of self-regulation measures. Proceedings of National Academy of Sciences USA. 2019; 116: 5472-5477.
15. Hussey I, Hughes S. Hidden invalidity among 15 commonly used measures in social personality psychology. Advanced in Methods and Practice in Psychological Science. 2020; 3: 166-184.
16. Prentice D, Miller D. When small effects are impressive. Psychological Bulletin. 1992; 112: 160-164.
17. Funder D, Ozer D. Evaluating effect size in psychological research: Sense and nonsense. Advances in Methods and Practices in Psychological Science. 2019; 2: 156-168.
18. Meehl P. Appraising and amending theories: The strategy of Lakatosian defense and two principles that warrant it. Psychological Inquiry. 1990; 1: 108-141.
19. Iso-Ahola S. Exercise: Why it is a challenge for both the nonconscious and conscious mind. Review of General Psychology. 2013; 17: 93-110.
20. Iso-Ahola S. Conscious-nonconscious processing explains why some people exercise but most don’t. Journal of Nature and Science. 2017; 3 (e384): 1-16.
21. Iso-Ahola S. Human mind: Both the cause and solution to the global pandemic of physical inactivity. International Journal of Public Health Research. 2018; 6: 107-113.
22. Kahneman D. Thinking, fast and slow. 2014. New York, NY: Farrar, Straus and Giroux.
23. Trafimow D. Hypothesis testing and theory evaluation at the boundaries. Psychological Review. 2003; 110: 526-535.
24. McFall R. Making psychology incorruptible. Applied and Preventive Psychology. 1996; 5: 9-15.
25. Popper K. The logic of scientific discovery. 1959. London: Hutchison.
26. Lykken D. Statistical significance in psychological research. Psychological Bulletin. 1968; 70: 151-159.
27. Ackerman P. Nonsense, common sense, and science of expert performance: Talent and individual differences. Intelligence. 2014; 45: 6-17.
28. Bargh J. The cognitive unconscious in everyday life. 2021. In A Reber, R Allen (Eds), The cognitive unconscious. London: Oxford University Press.
29. Cumming G. The new statistics: Why and how. Psychological Science. 2014; 25: 7-29.
30. Cohen J. Statistical power analysis for the behavioral sciences. 1988. Hillsdale, NJ: Erlbaum.
31. Wetzels R, Matzke D, Lee M, Rouder J, Iverson G, Wagenmakers E-J. Statistical evidence in experimental psychology: An empirical comparison using 855 t tests. Perspectives on Psychological Science. 2011; 6: 291-298.
32. Kass R, Raftery A. Bayes factors. Journal of American Statistical Association. 1995; 90: 377-395.
33. Dienes Z. Bayesian versus orthodox statistics: Which side are you on? Perspectives on Psychological Science. 2011; 6: 274-290.
34. Bargh J. The historical origins of priming as the preparation of behavioral responses: Unconscious carry-over and contextual influences of real-world importance. In D Molden (Ed), Understanding priming effects in social psychology (pp. 218-233). New York, NY: Guilford Press.
35. Baumeister R, Tice D, Bushman B. A review of multi-site replication projects in social psychology: Methodological ideal or collective self-destruct mechanism? 2022. Submitted for publication.
36. Churchland M, Afshar A, Shenoy K. A central source of movement variability. Neuron.2006; 52: 1085-1096.
37. Bargh J, Hassin R. Human unconscious processes in situ: The kind of awareness that really matters. 2021. In A Reber, R Allen (Eds), The cognitive unconscious. London: Oxford University Press.
38. Baumeister R. Conquer yourself, conquer the world. Scientific American. 2015; 312: 61-65.
39. Inzlicht M, Schmeichel B. What is ego depletion? Toward a mechanistic revision of the resource model of self-control. Perspectives on Psychological Science. 2012; 7: 450-463.
40. Crandall C, Sherman J. On the scientific superiority of conceptual replications for scientific progress. Journal of Experimental Social Psychology. 2015; 66: 93-99.
41. Dang J, Barker P, Baumert A, Bentvelzen M, Berkman E, Buchholz N, et al. A multilab replication of the ego depletion effect. Social Psychological and Personality Science. 2021; 12: 14-24.
42. Garrison K, Finley A, Schmeichel B. Ego depletion reduces attention control: Evidence from two high-powered preregistered experiments. Personality and Social Psychology Bulletin. 2019; 45: 728-739.
43. Bryan C, Walton G, Rogers T, Dweck C. Motivating voter turnout by invoking the self. Proceedings of National Academy of Sciences USA. 2011; 108: 12653-12656.
44. Bryan C, Yeager D, O’Brien J. Replicator degrees of freedom allow publication of misleading “failures to replicate”. Unpublished manuscript. 2019. University of Chicago.
45. Ramscar M, Shaoul C, Baayen R. Why many priming effects don’t (and won’t) replicate: A quantitative analysis. Unpublished manuscript. 2015. Tubingen University, Germany.
46. Chen X, Latham G, Piccolo R, Itzchakov G. An enumerative review and a meta-analysis of primed goal effects on organizational behavior. Applied Psychology. 2021; 70: 216-253.
47. Weingarten E, Chen Q, McAdams M, Yi J, Hepler J, Albarracin D. From primed concepts to action: A meta-analysis of the behavioral effects of incidentally presented words. Psychological Bulletin. 2016; 142: 472-497.
48. Bargh J. The hidden life of the consumer mind. Consumer Psychology Review. 2021; 5: 1-16.
49. Hofmann W, Vohs K, Baumeister R. What people desire, feel conflicted about, and try to resist in everyday life. Psychological Science. 2012; 23: 582-588.
50. Kool W, Botvinick M. A labor/leisure tradeoff in cognitive control. Journal of Experimental Psychology: General. 2014; 143: 131-141.
51. Amrhein V, Greenland S, McShane B. Retire statistical significance. Nature. 2019; 567: 305-307.
52. Greenland S, Senn S, Rothman K, Carlin J, Poole C, Goodman S. et al. Statistical tests, p values, confidence intervals, and power: A guide to misinterpretations. European Journal of Epidemiology. 2016; 31: 337-350.
53. Iso-Ahola S, Dotson C. Psychological momentum--not a statistical but psychological phenomenon. Review of General Psychology. 2015; 19: 112-116.