Data-Driven Hypothesis Generation in Clinical Research: What We Learned from a Human Subject Study?
Article Sidebar
Published
Feb 28, 2024
Downloads
Download data is not yet available.
Submit your own article
Register as an author to reserve your spot in the next issue of the Medical Research Archives.
Author RegistrationJoin the Society
The European Society of Medicine is more than a professional association. We are a community. Our members work in countries across the globe, yet are united by a common goal: to promote health and health equity, around the world.
Join Europe’s leading medical society and discover the many advantages of membership, including free article publication.
MembershipMain Article Content
Abstract
Hypothesis generation is an early and critical step in any hypothesis-driven clinical research project. Because it is not yet a well-understood cognitive process, the need to improve the process goes unrecognized. Without an impactful hypothesis, the significance of any research project can be questionable, regardless of the rigor or diligence applied in other steps of the study, e.g., study design, data collection, and result analysis. In this perspective article, the authors provide a literature review on the following topics first: scientific thinking, reasoning, medical reasoning, literature-based discovery, and a field study to explore scientific thinking and discovery. Over the years, scientific thinking has shown excellent progress in cognitive science and its applied areas: education, medicine, and biomedical research. However, a review of the literature reveals the lack of original studies on hypothesis generation in clinical research. The authors then summarize their first human participant study exploring data-driven hypothesis generation by clinical researchers in a simulated setting. The results indicate that a secondary data analytical tool, VIADS—a visual interactive analytic tool for filtering, summarizing, and visualizing large health data sets coded with hierarchical terminologies, can shorten the time participants need, on average, to generate a hypothesis and also requires fewer cognitive events to generate each hypothesis. As a counterpoint, this exploration also indicates that the quality ratings of the hypotheses thus generated carry significantly lower ratings for feasibility when applying VIADS. Despite its small scale, the study confirmed the feasibility of conducting a human participant study directly to explore the hypothesis generation process in clinical research. This study provides supporting evidence to conduct a larger-scale study with a specifically designed tool to facilitate the hypothesis-generation process among inexperienced clinical researchers. A larger study could provide generalizable evidence, which in turn can potentially improve clinical research productivity and overall clinical research enterprise.
Keywords:
Clinical research, scientific hypothesis generation, visualization, data-driven hypothesis generation, medical informatics, translational research
Article Details
How to Cite
JING, Xia et al.
Data-Driven Hypothesis Generation in Clinical Research: What We Learned from a Human Subject Study?.
Medical Research Archives, [S.l.], v. 12, n. 2, feb. 2024.
ISSN 2375-1924.
Available at: <https://esmed.org/MRA/mra/article/view/5132>. Date accessed: 22 dec. 2024.
doi: https://doi.org/10.18103/mra.v12i2.5132.
Section
Research Articles
The Medical Research Archives grants authors the right to publish and reproduce the unrevised contribution in whole or in part at any time and in any form for any scholarly non-commercial purpose with the condition that all publications of the contribution include a full citation to the journal as published by the Medical Research Archives.
References
1. Pruzan P. Research Methodology: The Aims, Practices and Ethics of Science. Springer International Publishing Switzerland; 2016.
2. Farrugia P, Petrisor B, Farrokhyar F, Bhandari M. Research questions, hypotheses and objectives. J Can Chir. 2010;50
3. Hicks CM. Research methods for clinical therapists: Applied project design and analysis. 1999;
4. Supino P, Borer J. Principles of research methodology: A guide for clinical investigators. 2012;
5. Browner W, Newman T, Cummings S, et al. Designing Clinical Research. 5th ed. Wolters Kluwer; 2023.
6. Jing X, Zhou Y, Cimino J, et al. Development, validation, and usage of metrics to evaluate clinical research hypothesis quality. BMC Medical Research Methodology, under review. 2023;doi:https://www.medrxiv.org/content/10.1101/2023.01.17.23284666v2
7. Jing X, Draghi BN, Ernst MA, et al. How do clinical researchers generate data-driven scientific hypotheses? Cognitive events using think-aloud protocol. BMJ Health & Care Informatics Under Review MedRxiv Preprint. 2023;doi:https://medrxiv.org/cgi/content/short/2023.10.31.23297860v1
8. Jing X, Patel VL, Cimino JJ, et al. A Visual Analytic Tool (VIADS) to Assist the Hypothesis Generation Process in Clinical Research: Mixed Methods Usability Study. JMIR Human Factors. 2023;10:e44644. https://preprints.jmir.org/preprint/44644. doi:doi: 10.2196/44644
9. Jing X, Cimino JJ, Patel VL, et al. Data-driven hypothesis generation among inexperienced clinical researchers: A comparison of secondary data analyses with visualization (VIADS) and other tools. Journal of Clinical and Translational Science. 2023;8(1):e13. doi:https://doi.org/10.1017/cts.2023.708
10. Jing X, Patel VL, Cimino JJ, et al. The Roles of a Secondary Data Analytics Tool and Experience in Scientific Hypothesis Generation in Clinical Research: Protocol for a Mixed Methods Study. JMIR Res Protoc. 2022/7/18 2022; 11(7):e39414. doi:10.2196/39414
11. Jing X, Cimino JJ. Graphical methods for reducing, visualizing and analyzing large data sets using hierarchical terminologies. presented at: AMIA 2011; 2011; Washington DC. https://www.ncbi.nlm.nih.gov/pmc/articles/PMC3243153/
12. Jing X, Cimino JJ. A complementary graphical method for reducing and analyzing large data sets: Case studies demonstrating thresholds setting and selection. Methods Inf Med. 2014;53:173-185. doi:10.3414/ME13-01-0075
13. Levine M, Osei D, Cimino J, et al. Performance Comparison between Two Solutions for Filtering Data Sets with Hierarchical Structures. J Computer Engineering & Info Tech. 2016;S1doi:http://dx.doi.org/10.4172/2324-9307.S1-003
14. Emerson M, Brooks M, Masters D, et al. Improved visualization of hierarchical datasets with VIADS. presented at: AMIA Annual Symposium; Nov 3-7, 2018 2018; San Francisco.
15. Jing X, Emerson M, Gunderson D, et al. Architecture of a visual interactive analysis tool for filtering and summarizing large data sets coded with hierarchical terminologies (VIADS). presented at: AMIA Summits Transl Sci Proc 2018;
16. Jing X, Emerson M, Masters D, et al. A visual interactive analysis tool for filtering and summarizing large data sets coded with hierarchical terminologies (VIADS). BMC Med Inform Decis Mak. 2019;19(31) doi:https://doi.org/10.1186/s12911-019-0750-y
17. Biesecker L. Hypothesis-generating research and predictive medicine. Genome Res. 2013;23:1051-1053.
18. The Cambridge Handbook of Thinking and Reasoning. Cambridge University Press; 2005.
19. The Oxford handbook of thinking and reasoning. The Oxford handbook of thinking and reasoning. Oxford University Press; 2012:xix, 836-xix, 836.
20. Klahr D. Exploring Science: The Cognition and Development of Discovery Processes. The MIT Press; 2000.
21. Dumas D, Dunbar K. The Creative Stereotype Effect. PLoS ONE 2016;11(2):e0142567. doi:doi:10.1371/journal.pone.0142567
22. Dunbar K, Fugelsang J. Causal thinking in science: How scientists and students interpret the unexpected. In: Gorman M, Kincannon A, Gooding D, Tweney R, eds. New directions in scientific and technical thinking. Erlbaum; 2004:57-59.
23. Fugelsang J, Dunbar K. Brain-based mechanisms underlying causal reasoning. In: Kraft E, ed. Neural correlates of thinking. Springer; 2009:269-279.
24. Patel V, Groen G. Knowledge Based Solution Strategies in Medical Reasoning. Cognitive Sci. 1986;10:91-116. doi:10.1207/s15516709cog1001_4
25. Joseph G-M, Patel VL. Domain knowledge and hypothesis generation in diagnostic reasoning. Medical Decision Making. 1990;10:31-46.
26. Arocha J, Patel V, Patel Y. Hypothesis generation and the coordiantion of theory and evidence in novice diagnostic reasoning. Medical Decision Making. 1993;13:198-211.
27. Kitano H. Nobel Turing Challenge: creating the engine for scientific discovery. npj Systems Biology and Applications. 2021/06/18 2021;7(1):29. doi:10.1038/s41540-021-00189-3
28. Misra DP, Gasparyan AY, Zimba O, Yessirkepov M, Agarwal V, Kitas GD. Formulating Hypotheses for Different Study Designs. J Korean Med Sci. 2021;36(50):e338. doi:https://doi.org/10.3346/jkms.2021.36.e338
29. Gasparyan AY, Ayvazyan L, Mukanova U, Yessirkepov M, Kitas GD. Scientific Hypotheses: Writing, Promoting, and Predicting Implications. J Korean Med Sci. Nov 25 2019;34(45):e300. doi:10.3346/jkms.2019.34.e300
30. Henry S, McInnes BT. Literature Based Discovery: Models, methods, and trends. J Biomed Inform. Oct 2017;74:20-32. doi:10.1016/j.jbi.2017.08.011
31. Swanson DR. Fish oil, Raynaud's syndrome, and undiscovered public knowledge. Perspectives in biology and medicine. Autumn 1986;30(1):7-18. doi:10.1353/pbm.1986.0087
32. Swanson DR. Undiscovered Public Knowledge. The Library Quarterly: Information, Community, Policy. 1986;56(2):103-118.
33. Swanson DR, Smalheiser NR. Implicit Text Linkages between Medline Records: Using Arrowsmith as an Aid to Scientific Discovery. Library Trends1999. p. 48.
34. Liekens AM, De Knijf J, Daelemans W, Goethals B, De Rijk P, Del-Favero J. BioGraph: unsupervised biomedical knowledge discovery via automated hypothesis generation. Genome Biol. Jun 22 2011;12(6):R57. doi:10.1186/gb-2011-12-6-r57
35. Wittkop T, TerAvest E, Evani US, et al. STOP using just GO: a multi-ontology hypothesis generation tool for high throughput experimentation. BMC Bioinformatics. Feb 14 2013;14:53. doi:10.1186/1471-2105-14-53
36. Spangler S, Wilkins AD, Bachman BJ, et al. Automated hypothesis generation based on mining scientific literature. presented at: Proceedings of the 20th ACM SIGKDD international conference on Knowledge discovery and data mining; 2014; New York, New York, USA. https://doi-org.libproxy.clemson.edu/10.1145/2623330.2623667
37. Sybrandt J, Shtutman M, Safro I. Moliere: Automatic biomedical hypothesis generation system. ACM; 2017.
38. Klauer KC, Stegmaier R, Meiser T. Working Memory Involvement in Propositional and Spatial Reasoning. Article. Thinking & Reasoning. 1997;3(1):9-47. doi:10.1080/135467897394419
39. Baddeley A, Emslie H, Kolodny J, Duncan J. Random generation and the executive control of working memory. The Quarterly journal of experimental psychology A, Human experimental psychology. 1998;51(4):819-852. doi:10.1080/713755788
40. Kuhn D, Amsel E, O'Loughlin M. The Development of Scientific Thinking Skills. Developmental Psychology Series. Academic Press; 1988.
41. Vosniadou SE. International Handbook of Research on Conceptual Change (1st ed.). Routledge; 2008.
42. Klahr D, Dunbar K. Dual Space Search During Scientific Reasoning. Cognitive Science. 1988;12(1):1-48. doi:https://doi.org/10.1207/s15516709cog1201_1
43. Klahr D. Patterns, rules, and discoveries in life and in science. The journey from child to scientist: Integrating cognitive development and the education sciences. American Psychological Association; 2012:263-292.
44. Thomas R, Dougherty M, Sprenger A, Harbison J. Diagnostic hypothesis generation and human judgment. Psychological Review. 2008;115(1):155-185. doi:doi:10.1037/0033-295X.115.1.155
45. Sprenger AM, Dougherty MR, Atkins SM, et al. Implications of cognitive load for hypothesis generation and probability judgment. Front Psychol. 2011;2:129. doi:10.3389/fpsyg.2011.00129
46. Dasgupta I, Schulz E, Gershman SJ. Where do hypotheses come from? Cogn Psychol. Aug 2017;96:1-25. doi:10.1016/j.cogpsych.2017.05.001
47. Donnelly MB, Sisson JC, Woolliscroft JO. The reliability of a hypothesis generation and testing task. Med Educ. Nov 1990;24(6):507-11. doi:10.1111/j.1365-2923.1990.tb02666.x
48. Alison L, Doran B, Long ML, Power N, Humphrey A. The effects of subjective time pressure and individual differences on hypotheses generation and action prioritization in police investigations. Journal of experimental psychology Applied. Mar 2013;19(1):83-93. doi:10.1037/a0032148
49. Merrifield PR, Erickson EB. System Support for Hypothesis Generation. Psychological Reports. 1965;16(2):475-490. doi:10.2466/pr0.1965.16.2.475
50. Kerne A, Smith S, Koh E, Choi H, Graeber R. An Experimental Method for Measuring the Emergence of New Ideas in Information Discovery. Article. International Journal of Human-Computer Interaction. 2008;24(5):460-477. doi:10.1080/10447310802142243
51. PATEL VL, GROEN GJ, AROCHA JF. Medical expertise as a function of task difficulty. Memory & cognition. 1990;18(4):394-406.
52. Kaufman DR, Patel VL, Magder SA. The explanatory role of spontaneously generated analogies in reasoning about physiological concepts. International Journal of Science Education. 1996/04/01 1996;18(3):369-386. doi:10.1080/0950069960180309
53. Kushniruk A, Patel V, Marley A. Small worlds and medical expertise: implications for medical cognition and knowledge engineering. Int J Med Inform. 1998;49:255-271.
54. Patel V, Groen G, Patel Y. Cognitive aspects of clinical performance during patient workup: The role of medical expertise. Advances in Health Sciences Education. 1997;2:95-114.
55. Ericsson KA, Simon eA. Protocol Analysis: Verbal Reports as Data. The MIT Press; 1993.
56. Li AC, Kannry JL, Kushniruk A, et al. Integrating usability testing and think-aloud protocol analysis with "near-live" clinical simulations in evaluating clinical decision support. Int J Med Inform. Nov 2012;81(11):761-72. doi:10.1016/j.ijmedinf.2012.02.009
57. McKeown K, Jordan D, Feiner S, et al. A study of communication in the Cardiac Surgery Intensive Care Unit and its implications for automated briefing. Proc AMIA Symp. 2000:570-4.
58. Kushniruk AW, Kan MY, McKeown K, et al. Usability evaluation of an experimental text summarization system and three search engines: implications for the reengineering of health care interfaces. Proc AMIA Symp. 2002:420-4.
59. Kaufman DR, Patel VL, Hilliman C, et al. Usability in the real world: assessing medical information technologies in patients’ homes. J Biomed Inform. 2003;36:45-60.
60. Kushniruk AW, Patel VL. Cognitive and usability engineering methods for the evaluation of clinical information systems. J Biomed Inform. 2004;37:56-76.
61. McKeown K. PERSIVAL, a System for Personalized Search and Summarization over Multimedia Healthcare Information. In: Proceedings of the 1st ACM/IEEE-CS Joint Conference on Digital Libraries. ACM; 2001:331-340.
62. Iyengar MS, Chang O, Florez-Arango JF, Taria M, Patel VL. Development and usability of a mobile tool for identification of depression and suicide risk in Fiji. Technol Health Care. 2021;29(1):143-153. doi:10.3233/thc-202132
63. Patel V, Halpern M, Nagaraj V. Information processing by community health nurses using mobile health (mHealth) tools for early identification of suicide and depression risks in Fiji Islands. BMJ Health Care Inform. 2021;28:e100342doi:doi:10.1136/bmjhci-2021-100342
64. Patel VL, Arocha JF, Zhang J. Chapter 30: Thinking and Reasoning in Medicine. In: Holyoak KJ, Morrison RG, eds. The Cambridge Handbook of Thinking and Reasoning. Cambridge University Press; 2005:727-750:chap 30.
65. Weeber M, Klein H, de Jong-van den Berg LTW, Vos R. Using concepts in literature-based discovery: Simulating Swanson's Raynaud–fish oil and migraine–magnesium discoveries. Journal of the American Society for Information Science and Technology. 2001;52(7):548-557. doi:https://doi.org/10.1002/asi.1104
66. Kilicoglu H, Rosemblat G, Fiszman M, Shin D. Broad-coverage biomedical relation extraction with SemRep. BMC Bioinformatics. 2020/05/14 2020;21(1):188. doi:10.1186/s12859-020-3517-7
67. NIH N. PubMed. 2006
68. Xu G, Wang C, He X. Improving Clinical Named Entity Recognition with Global Neural Attention. In: Lecture Notes in Computer Science. Springer, Cham; 2018:
69. Wei CH, Allot A, Leaman R, Lu Z. PubTator central: automated concept annotation for biomedical full text articles. Nucleic Acids Res. Jul 2 2019;47(W1):W587-w593. doi:10.1093/nar/gkz389
70. Hristovski D, Friedman C, Rindflesch TC, Peterlin B. Exploiting semantic relations for literature-based discovery. AMIA Annu Symp Proc. 2006;2006:349-53.
71. Fader A, Soderland S, Etzioni O. Identifying relations for open information extraction. presented at: Proceedings of the Conference on Empirical Methods in Natural Language Processing; 2011; Edinburgh, United Kingdom.
72. Bouraoui Z, Jameel S, Schockaert S. Relation Induction in Word Embeddings Revisited. In: Proceedings of the 27th International Conference on Computational Linguistics. Association for Computational Linguistics; 2018:1627-1637.
73. Marneffe M-Cd, MacCartney B, Manning CD. Generating Typed Dependency Parses from Phrase Structure Parses. European Language Resources Association (ELRA); 2006:
74. Akujuobi U, Chen J, Elhoseiny M, Spranger M, Zhang X. Temporal Positive-unlabeled Learning for Biomedical Hypothesis Generation via Risk Estimation. 2020:
75. Pedersen T, Pakhomov SV, Patwardhan S, Chute CG. Measures of semantic similarity and relatedness in the biomedical domain. J Biomed Inform. Jun 2007;40(3):288-99. doi:10.1016/j.jbi.2006.06.004
76. Kulmanov M, Smaili FZ, Gao X, Hoehndorf R. Semantic similarity and machine learning with ontologies. Briefings in Bioinformatics. 2020;22(4)doi:10.1093/bib/bbaa199
77. G S, A G, H G-A, D P. Soft similarity and soft cosine measure: similarity of features in vector space model. Comput Sist. 2014;18(3):491-504.
78. Petrič I, Cestnik B, Lavrač N, Urbančič T. Bisociative Knowledge Discovery by Literature Outlier Detection. In: Berthold MR, ed. Bisociative Knowledge Discovery: An Introduction to Concept, Algorithms, Tools, and Applications. Springer Berlin Heidelberg; 2012:313-324.
79. Sluban B, Juršič M, Cestnik B, Lavrač N. Exploring the Power of Outliers for Cross-Domain Literature Mining. In: Berthold MR, ed. Bisociative Knowledge Discovery: An Introduction to Concept, Algorithms, Tools, and Applications. Springer Berlin Heidelberg; 2012:325-337.
80. Kötter T, Berthold MR. (Missing) Concept Discovery in Heterogeneous Information Networks. In: Berthold MR, ed. Bisociative Knowledge Discovery: An Introduction to Concept, Algorithms, Tools, and Applications. Springer Berlin Heidelberg; 2012:230-245.
81. Workman TE, Fiszman M, Rindflesch TC, Nahl D. Framing serendipitous information‐seeking behavior for facilitating literature‐based discovery: A proposed model. Journal of the Association for Information Science and Technology. 2014;65
82. Petric I, Urbancic T, Cestnik B, Macedoni-Luksic M. Literature mining method RaJoLink for uncovering relations between biomedical concepts. J Biomed Inform. Apr 2009;42(2):219-27. doi:10.1016/j.jbi.2008.08.004
83. Workman TE, Fiszman M, Cairelli MJ, Nahl D, Rindflesch TC. Spark, an application based on Serendipitous Knowledge Discovery. Journal of Biomedical Informatics. 2016;60:23-37. doi:https://doi.org/10.1016/j.jbi.2015.12.014
84. Cohen T, Whitfield GK, Schvaneveldt RW, Mukund K, Rindflesch T. EpiphaNet: An Interactive Tool to Support Biomedical Discoveries. J Biomed Discov Collab. Sep 21 2010;5:21-49.
85. Goodwin JC, Cohen T, Rindflesch T. Discovery by scent: Discovery browsing system based on the Information Foraging Theory. 2012: 232-239.
86. Baek SH, Lee D, Kim M, Lee JH, Song M. Enriching plausible new hypothesis generation in PubMed. PLOS ONE. 2017;12(7):e0180539. doi:10.1371/journal.pone.0180539
87. Sang S, Yang Z, Li Z, Lin H. Supervised Learning Based Hypothesis Generation from Biomedical Literature. Biomed Res Int. 2015;2015:698527. doi:10.1155/2015/698527
88. Akujuobi U, Spranger M, Palaniappan SK, Zhang X. T-PAIR: Temporal Node-Pair Embedding for Automatic Biomedical Hypothesis Generation. IEEE Transactions on Knowledge and Data Engineering. 2022;34(6):2988-3001. doi:10.1109/TKDE.2020.3017687
89. Sybrandt J, Shtutman M, Safro I. Large-Scale Validation of Hypothesis Generation Systems via Candidate Ranking. Proc IEEE Int Conf Big Data. Dec 2018;2018:1494-1503. doi:10.1109/bigdata.2018.8622637
90. Callahan A, Dumontier M, Shah NH. HyQue: evaluating hypotheses using Semantic Web technologies. Report. Journal of Biomedical Semantics. 2011/05/17/// 2011;2:NA.
91. Soldatova LN, Rzhetsky A. Representation of research hypotheses. J Biomed Semantics. May 17 2011;2 Suppl 2(Suppl 2):S9. doi:10.1186/2041-1480-2-s2-s9
92. Whelan K, Ray O, King RD. Representation, simulation, and hypothesis generation in graph and logical models of biological networks. Methods Mol Biol. 2011;759:465-82. doi:10.1007/978-1-61779-173-4_26
93. Spangler S. Accelerating discovery: Mining unstructured informaiton for hypothesis generation. 2016;
94. Wang R, Zelikman E, Poesia G, Pu Y, Haber N, Goodman ND. Hypothesis Search: Inductive Reasoning with Language Models. arXiv:230905660 [csLG]. 2023;doi:https://doi.org/10.48550/arXiv.2309.05660
95. Park YJ, Kaplan D, Ren Z, et al. Can ChatGPT be used to generate scientific hypotheses? arXiv:230412208 [csCL]. 2023;doi:https://doi.org/10.48550/arXiv.2304.12208
96. Chen Y, Fu Q, Yuan Y, et al. Hallucination Detection: Robustly Discerning Reliable Answers in Large Language Models. presented at: In Proceedings of the 32nd ACM International Conference on Information and Knowledge Management; 2023;
97. Thirunavukarasu AJ, Ting DSJ, Elangovan K, Gutierrez L, Tan TF, Ting DSW. Large language models in medicine. Nature Medicine. 2023/08/01 2023;29(8):1930-1940. doi:10.1038/s41591-023-02448-8
98. Zhang Y, Li Y, Cui L, et al. Siren's Song in the AI Ocean: A Survey on Hallucination in Large Language Models. arXiv preprint. 2023;arXiv:2309.01219doi:https://doi.org/10.48550/arXiv.2309.01219
99. Wang C, Liu S, Yang H, Guo J, Wu Y, Liu J. Ethical Considerations of Using ChatGPT in Health Care. J Med Internet Res. 2023/8/11 2023;25:e48009. doi:10.2196/48009
100. Mynatt C, Doherty M, Tweney R. Confirmation Bias in a Simulated Research Environment: An Experimental Study of Scientific Inference. QUART J EXP PSYCHOL. 1977;29:85-95.
101. Dunbar K. How scientists think: On-line creativity and conceptual change in science. Creative thought: An investigation of conceptual structures and processes. American Psychological Association; 1997:461-493.
102. Dunbar K. How scientists really reason: Scientific reasoning in real-world laboratories. The nature of insight. The MIT Press; 1995:365-395.
103. Dunbar K. The analogical paradox: Why analogy is so easy in naturalistic settings, yet so difficult in the psychology laboratory. In: Gentner D, Holyoak K, Kokinov B, eds. Analogy: Perspectives from cognitive science. MIT Press; 2001:323-334.
104. Patel VL, Kaufman D, (Eds) TC. Cognitive Informatics in Health and Biomedicine: Case Studies on Critical Care, Complexity and Errors Springer; 2014.
105. Statistics CNCfH. NAMCS datasets and documentation. 2017;
106. Jing X, Zhou YC, Cimino JJ, et al. Development and preliminary validation of metrics to evaluate data-driven clinical research hypotheses. 2022:
107. Jing X, Patel V, Cimino J, Shubrook J. Hypothesis generation in clinical research: challenges, opportunities, and role of AI. IOS; 2022:
108. Draghi B, Ernst M, Patel V, et al. Number of scientific hypotheses and time needed in a 2-hour study session among inexperienced clinical researchers—preliminary results. Mar 18-21, 2023:
109. Brooke J. SUS - A quick and dirty usability scale. Reading, UK.
110. Brooke J. SUS: a retrospective. J Usability Studies. 2013;8:29-40.
111. Jing X, Patel V, Cimino J, Georgiou A. Scientific hypothesis generation in clinical research: cognition, visualization, and evaluation. IOS; 2023:
2. Farrugia P, Petrisor B, Farrokhyar F, Bhandari M. Research questions, hypotheses and objectives. J Can Chir. 2010;50
3. Hicks CM. Research methods for clinical therapists: Applied project design and analysis. 1999;
4. Supino P, Borer J. Principles of research methodology: A guide for clinical investigators. 2012;
5. Browner W, Newman T, Cummings S, et al. Designing Clinical Research. 5th ed. Wolters Kluwer; 2023.
6. Jing X, Zhou Y, Cimino J, et al. Development, validation, and usage of metrics to evaluate clinical research hypothesis quality. BMC Medical Research Methodology, under review. 2023;doi:https://www.medrxiv.org/content/10.1101/2023.01.17.23284666v2
7. Jing X, Draghi BN, Ernst MA, et al. How do clinical researchers generate data-driven scientific hypotheses? Cognitive events using think-aloud protocol. BMJ Health & Care Informatics Under Review MedRxiv Preprint. 2023;doi:https://medrxiv.org/cgi/content/short/2023.10.31.23297860v1
8. Jing X, Patel VL, Cimino JJ, et al. A Visual Analytic Tool (VIADS) to Assist the Hypothesis Generation Process in Clinical Research: Mixed Methods Usability Study. JMIR Human Factors. 2023;10:e44644. https://preprints.jmir.org/preprint/44644. doi:doi: 10.2196/44644
9. Jing X, Cimino JJ, Patel VL, et al. Data-driven hypothesis generation among inexperienced clinical researchers: A comparison of secondary data analyses with visualization (VIADS) and other tools. Journal of Clinical and Translational Science. 2023;8(1):e13. doi:https://doi.org/10.1017/cts.2023.708
10. Jing X, Patel VL, Cimino JJ, et al. The Roles of a Secondary Data Analytics Tool and Experience in Scientific Hypothesis Generation in Clinical Research: Protocol for a Mixed Methods Study. JMIR Res Protoc. 2022/7/18 2022; 11(7):e39414. doi:10.2196/39414
11. Jing X, Cimino JJ. Graphical methods for reducing, visualizing and analyzing large data sets using hierarchical terminologies. presented at: AMIA 2011; 2011; Washington DC. https://www.ncbi.nlm.nih.gov/pmc/articles/PMC3243153/
12. Jing X, Cimino JJ. A complementary graphical method for reducing and analyzing large data sets: Case studies demonstrating thresholds setting and selection. Methods Inf Med. 2014;53:173-185. doi:10.3414/ME13-01-0075
13. Levine M, Osei D, Cimino J, et al. Performance Comparison between Two Solutions for Filtering Data Sets with Hierarchical Structures. J Computer Engineering & Info Tech. 2016;S1doi:http://dx.doi.org/10.4172/2324-9307.S1-003
14. Emerson M, Brooks M, Masters D, et al. Improved visualization of hierarchical datasets with VIADS. presented at: AMIA Annual Symposium; Nov 3-7, 2018 2018; San Francisco.
15. Jing X, Emerson M, Gunderson D, et al. Architecture of a visual interactive analysis tool for filtering and summarizing large data sets coded with hierarchical terminologies (VIADS). presented at: AMIA Summits Transl Sci Proc 2018;
16. Jing X, Emerson M, Masters D, et al. A visual interactive analysis tool for filtering and summarizing large data sets coded with hierarchical terminologies (VIADS). BMC Med Inform Decis Mak. 2019;19(31) doi:https://doi.org/10.1186/s12911-019-0750-y
17. Biesecker L. Hypothesis-generating research and predictive medicine. Genome Res. 2013;23:1051-1053.
18. The Cambridge Handbook of Thinking and Reasoning. Cambridge University Press; 2005.
19. The Oxford handbook of thinking and reasoning. The Oxford handbook of thinking and reasoning. Oxford University Press; 2012:xix, 836-xix, 836.
20. Klahr D. Exploring Science: The Cognition and Development of Discovery Processes. The MIT Press; 2000.
21. Dumas D, Dunbar K. The Creative Stereotype Effect. PLoS ONE 2016;11(2):e0142567. doi:doi:10.1371/journal.pone.0142567
22. Dunbar K, Fugelsang J. Causal thinking in science: How scientists and students interpret the unexpected. In: Gorman M, Kincannon A, Gooding D, Tweney R, eds. New directions in scientific and technical thinking. Erlbaum; 2004:57-59.
23. Fugelsang J, Dunbar K. Brain-based mechanisms underlying causal reasoning. In: Kraft E, ed. Neural correlates of thinking. Springer; 2009:269-279.
24. Patel V, Groen G. Knowledge Based Solution Strategies in Medical Reasoning. Cognitive Sci. 1986;10:91-116. doi:10.1207/s15516709cog1001_4
25. Joseph G-M, Patel VL. Domain knowledge and hypothesis generation in diagnostic reasoning. Medical Decision Making. 1990;10:31-46.
26. Arocha J, Patel V, Patel Y. Hypothesis generation and the coordiantion of theory and evidence in novice diagnostic reasoning. Medical Decision Making. 1993;13:198-211.
27. Kitano H. Nobel Turing Challenge: creating the engine for scientific discovery. npj Systems Biology and Applications. 2021/06/18 2021;7(1):29. doi:10.1038/s41540-021-00189-3
28. Misra DP, Gasparyan AY, Zimba O, Yessirkepov M, Agarwal V, Kitas GD. Formulating Hypotheses for Different Study Designs. J Korean Med Sci. 2021;36(50):e338. doi:https://doi.org/10.3346/jkms.2021.36.e338
29. Gasparyan AY, Ayvazyan L, Mukanova U, Yessirkepov M, Kitas GD. Scientific Hypotheses: Writing, Promoting, and Predicting Implications. J Korean Med Sci. Nov 25 2019;34(45):e300. doi:10.3346/jkms.2019.34.e300
30. Henry S, McInnes BT. Literature Based Discovery: Models, methods, and trends. J Biomed Inform. Oct 2017;74:20-32. doi:10.1016/j.jbi.2017.08.011
31. Swanson DR. Fish oil, Raynaud's syndrome, and undiscovered public knowledge. Perspectives in biology and medicine. Autumn 1986;30(1):7-18. doi:10.1353/pbm.1986.0087
32. Swanson DR. Undiscovered Public Knowledge. The Library Quarterly: Information, Community, Policy. 1986;56(2):103-118.
33. Swanson DR, Smalheiser NR. Implicit Text Linkages between Medline Records: Using Arrowsmith as an Aid to Scientific Discovery. Library Trends1999. p. 48.
34. Liekens AM, De Knijf J, Daelemans W, Goethals B, De Rijk P, Del-Favero J. BioGraph: unsupervised biomedical knowledge discovery via automated hypothesis generation. Genome Biol. Jun 22 2011;12(6):R57. doi:10.1186/gb-2011-12-6-r57
35. Wittkop T, TerAvest E, Evani US, et al. STOP using just GO: a multi-ontology hypothesis generation tool for high throughput experimentation. BMC Bioinformatics. Feb 14 2013;14:53. doi:10.1186/1471-2105-14-53
36. Spangler S, Wilkins AD, Bachman BJ, et al. Automated hypothesis generation based on mining scientific literature. presented at: Proceedings of the 20th ACM SIGKDD international conference on Knowledge discovery and data mining; 2014; New York, New York, USA. https://doi-org.libproxy.clemson.edu/10.1145/2623330.2623667
37. Sybrandt J, Shtutman M, Safro I. Moliere: Automatic biomedical hypothesis generation system. ACM; 2017.
38. Klauer KC, Stegmaier R, Meiser T. Working Memory Involvement in Propositional and Spatial Reasoning. Article. Thinking & Reasoning. 1997;3(1):9-47. doi:10.1080/135467897394419
39. Baddeley A, Emslie H, Kolodny J, Duncan J. Random generation and the executive control of working memory. The Quarterly journal of experimental psychology A, Human experimental psychology. 1998;51(4):819-852. doi:10.1080/713755788
40. Kuhn D, Amsel E, O'Loughlin M. The Development of Scientific Thinking Skills. Developmental Psychology Series. Academic Press; 1988.
41. Vosniadou SE. International Handbook of Research on Conceptual Change (1st ed.). Routledge; 2008.
42. Klahr D, Dunbar K. Dual Space Search During Scientific Reasoning. Cognitive Science. 1988;12(1):1-48. doi:https://doi.org/10.1207/s15516709cog1201_1
43. Klahr D. Patterns, rules, and discoveries in life and in science. The journey from child to scientist: Integrating cognitive development and the education sciences. American Psychological Association; 2012:263-292.
44. Thomas R, Dougherty M, Sprenger A, Harbison J. Diagnostic hypothesis generation and human judgment. Psychological Review. 2008;115(1):155-185. doi:doi:10.1037/0033-295X.115.1.155
45. Sprenger AM, Dougherty MR, Atkins SM, et al. Implications of cognitive load for hypothesis generation and probability judgment. Front Psychol. 2011;2:129. doi:10.3389/fpsyg.2011.00129
46. Dasgupta I, Schulz E, Gershman SJ. Where do hypotheses come from? Cogn Psychol. Aug 2017;96:1-25. doi:10.1016/j.cogpsych.2017.05.001
47. Donnelly MB, Sisson JC, Woolliscroft JO. The reliability of a hypothesis generation and testing task. Med Educ. Nov 1990;24(6):507-11. doi:10.1111/j.1365-2923.1990.tb02666.x
48. Alison L, Doran B, Long ML, Power N, Humphrey A. The effects of subjective time pressure and individual differences on hypotheses generation and action prioritization in police investigations. Journal of experimental psychology Applied. Mar 2013;19(1):83-93. doi:10.1037/a0032148
49. Merrifield PR, Erickson EB. System Support for Hypothesis Generation. Psychological Reports. 1965;16(2):475-490. doi:10.2466/pr0.1965.16.2.475
50. Kerne A, Smith S, Koh E, Choi H, Graeber R. An Experimental Method for Measuring the Emergence of New Ideas in Information Discovery. Article. International Journal of Human-Computer Interaction. 2008;24(5):460-477. doi:10.1080/10447310802142243
51. PATEL VL, GROEN GJ, AROCHA JF. Medical expertise as a function of task difficulty. Memory & cognition. 1990;18(4):394-406.
52. Kaufman DR, Patel VL, Magder SA. The explanatory role of spontaneously generated analogies in reasoning about physiological concepts. International Journal of Science Education. 1996/04/01 1996;18(3):369-386. doi:10.1080/0950069960180309
53. Kushniruk A, Patel V, Marley A. Small worlds and medical expertise: implications for medical cognition and knowledge engineering. Int J Med Inform. 1998;49:255-271.
54. Patel V, Groen G, Patel Y. Cognitive aspects of clinical performance during patient workup: The role of medical expertise. Advances in Health Sciences Education. 1997;2:95-114.
55. Ericsson KA, Simon eA. Protocol Analysis: Verbal Reports as Data. The MIT Press; 1993.
56. Li AC, Kannry JL, Kushniruk A, et al. Integrating usability testing and think-aloud protocol analysis with "near-live" clinical simulations in evaluating clinical decision support. Int J Med Inform. Nov 2012;81(11):761-72. doi:10.1016/j.ijmedinf.2012.02.009
57. McKeown K, Jordan D, Feiner S, et al. A study of communication in the Cardiac Surgery Intensive Care Unit and its implications for automated briefing. Proc AMIA Symp. 2000:570-4.
58. Kushniruk AW, Kan MY, McKeown K, et al. Usability evaluation of an experimental text summarization system and three search engines: implications for the reengineering of health care interfaces. Proc AMIA Symp. 2002:420-4.
59. Kaufman DR, Patel VL, Hilliman C, et al. Usability in the real world: assessing medical information technologies in patients’ homes. J Biomed Inform. 2003;36:45-60.
60. Kushniruk AW, Patel VL. Cognitive and usability engineering methods for the evaluation of clinical information systems. J Biomed Inform. 2004;37:56-76.
61. McKeown K. PERSIVAL, a System for Personalized Search and Summarization over Multimedia Healthcare Information. In: Proceedings of the 1st ACM/IEEE-CS Joint Conference on Digital Libraries. ACM; 2001:331-340.
62. Iyengar MS, Chang O, Florez-Arango JF, Taria M, Patel VL. Development and usability of a mobile tool for identification of depression and suicide risk in Fiji. Technol Health Care. 2021;29(1):143-153. doi:10.3233/thc-202132
63. Patel V, Halpern M, Nagaraj V. Information processing by community health nurses using mobile health (mHealth) tools for early identification of suicide and depression risks in Fiji Islands. BMJ Health Care Inform. 2021;28:e100342doi:doi:10.1136/bmjhci-2021-100342
64. Patel VL, Arocha JF, Zhang J. Chapter 30: Thinking and Reasoning in Medicine. In: Holyoak KJ, Morrison RG, eds. The Cambridge Handbook of Thinking and Reasoning. Cambridge University Press; 2005:727-750:chap 30.
65. Weeber M, Klein H, de Jong-van den Berg LTW, Vos R. Using concepts in literature-based discovery: Simulating Swanson's Raynaud–fish oil and migraine–magnesium discoveries. Journal of the American Society for Information Science and Technology. 2001;52(7):548-557. doi:https://doi.org/10.1002/asi.1104
66. Kilicoglu H, Rosemblat G, Fiszman M, Shin D. Broad-coverage biomedical relation extraction with SemRep. BMC Bioinformatics. 2020/05/14 2020;21(1):188. doi:10.1186/s12859-020-3517-7
67. NIH N. PubMed. 2006
68. Xu G, Wang C, He X. Improving Clinical Named Entity Recognition with Global Neural Attention. In: Lecture Notes in Computer Science. Springer, Cham; 2018:
69. Wei CH, Allot A, Leaman R, Lu Z. PubTator central: automated concept annotation for biomedical full text articles. Nucleic Acids Res. Jul 2 2019;47(W1):W587-w593. doi:10.1093/nar/gkz389
70. Hristovski D, Friedman C, Rindflesch TC, Peterlin B. Exploiting semantic relations for literature-based discovery. AMIA Annu Symp Proc. 2006;2006:349-53.
71. Fader A, Soderland S, Etzioni O. Identifying relations for open information extraction. presented at: Proceedings of the Conference on Empirical Methods in Natural Language Processing; 2011; Edinburgh, United Kingdom.
72. Bouraoui Z, Jameel S, Schockaert S. Relation Induction in Word Embeddings Revisited. In: Proceedings of the 27th International Conference on Computational Linguistics. Association for Computational Linguistics; 2018:1627-1637.
73. Marneffe M-Cd, MacCartney B, Manning CD. Generating Typed Dependency Parses from Phrase Structure Parses. European Language Resources Association (ELRA); 2006:
74. Akujuobi U, Chen J, Elhoseiny M, Spranger M, Zhang X. Temporal Positive-unlabeled Learning for Biomedical Hypothesis Generation via Risk Estimation. 2020:
75. Pedersen T, Pakhomov SV, Patwardhan S, Chute CG. Measures of semantic similarity and relatedness in the biomedical domain. J Biomed Inform. Jun 2007;40(3):288-99. doi:10.1016/j.jbi.2006.06.004
76. Kulmanov M, Smaili FZ, Gao X, Hoehndorf R. Semantic similarity and machine learning with ontologies. Briefings in Bioinformatics. 2020;22(4)doi:10.1093/bib/bbaa199
77. G S, A G, H G-A, D P. Soft similarity and soft cosine measure: similarity of features in vector space model. Comput Sist. 2014;18(3):491-504.
78. Petrič I, Cestnik B, Lavrač N, Urbančič T. Bisociative Knowledge Discovery by Literature Outlier Detection. In: Berthold MR, ed. Bisociative Knowledge Discovery: An Introduction to Concept, Algorithms, Tools, and Applications. Springer Berlin Heidelberg; 2012:313-324.
79. Sluban B, Juršič M, Cestnik B, Lavrač N. Exploring the Power of Outliers for Cross-Domain Literature Mining. In: Berthold MR, ed. Bisociative Knowledge Discovery: An Introduction to Concept, Algorithms, Tools, and Applications. Springer Berlin Heidelberg; 2012:325-337.
80. Kötter T, Berthold MR. (Missing) Concept Discovery in Heterogeneous Information Networks. In: Berthold MR, ed. Bisociative Knowledge Discovery: An Introduction to Concept, Algorithms, Tools, and Applications. Springer Berlin Heidelberg; 2012:230-245.
81. Workman TE, Fiszman M, Rindflesch TC, Nahl D. Framing serendipitous information‐seeking behavior for facilitating literature‐based discovery: A proposed model. Journal of the Association for Information Science and Technology. 2014;65
82. Petric I, Urbancic T, Cestnik B, Macedoni-Luksic M. Literature mining method RaJoLink for uncovering relations between biomedical concepts. J Biomed Inform. Apr 2009;42(2):219-27. doi:10.1016/j.jbi.2008.08.004
83. Workman TE, Fiszman M, Cairelli MJ, Nahl D, Rindflesch TC. Spark, an application based on Serendipitous Knowledge Discovery. Journal of Biomedical Informatics. 2016;60:23-37. doi:https://doi.org/10.1016/j.jbi.2015.12.014
84. Cohen T, Whitfield GK, Schvaneveldt RW, Mukund K, Rindflesch T. EpiphaNet: An Interactive Tool to Support Biomedical Discoveries. J Biomed Discov Collab. Sep 21 2010;5:21-49.
85. Goodwin JC, Cohen T, Rindflesch T. Discovery by scent: Discovery browsing system based on the Information Foraging Theory. 2012: 232-239.
86. Baek SH, Lee D, Kim M, Lee JH, Song M. Enriching plausible new hypothesis generation in PubMed. PLOS ONE. 2017;12(7):e0180539. doi:10.1371/journal.pone.0180539
87. Sang S, Yang Z, Li Z, Lin H. Supervised Learning Based Hypothesis Generation from Biomedical Literature. Biomed Res Int. 2015;2015:698527. doi:10.1155/2015/698527
88. Akujuobi U, Spranger M, Palaniappan SK, Zhang X. T-PAIR: Temporal Node-Pair Embedding for Automatic Biomedical Hypothesis Generation. IEEE Transactions on Knowledge and Data Engineering. 2022;34(6):2988-3001. doi:10.1109/TKDE.2020.3017687
89. Sybrandt J, Shtutman M, Safro I. Large-Scale Validation of Hypothesis Generation Systems via Candidate Ranking. Proc IEEE Int Conf Big Data. Dec 2018;2018:1494-1503. doi:10.1109/bigdata.2018.8622637
90. Callahan A, Dumontier M, Shah NH. HyQue: evaluating hypotheses using Semantic Web technologies. Report. Journal of Biomedical Semantics. 2011/05/17/// 2011;2:NA.
91. Soldatova LN, Rzhetsky A. Representation of research hypotheses. J Biomed Semantics. May 17 2011;2 Suppl 2(Suppl 2):S9. doi:10.1186/2041-1480-2-s2-s9
92. Whelan K, Ray O, King RD. Representation, simulation, and hypothesis generation in graph and logical models of biological networks. Methods Mol Biol. 2011;759:465-82. doi:10.1007/978-1-61779-173-4_26
93. Spangler S. Accelerating discovery: Mining unstructured informaiton for hypothesis generation. 2016;
94. Wang R, Zelikman E, Poesia G, Pu Y, Haber N, Goodman ND. Hypothesis Search: Inductive Reasoning with Language Models. arXiv:230905660 [csLG]. 2023;doi:https://doi.org/10.48550/arXiv.2309.05660
95. Park YJ, Kaplan D, Ren Z, et al. Can ChatGPT be used to generate scientific hypotheses? arXiv:230412208 [csCL]. 2023;doi:https://doi.org/10.48550/arXiv.2304.12208
96. Chen Y, Fu Q, Yuan Y, et al. Hallucination Detection: Robustly Discerning Reliable Answers in Large Language Models. presented at: In Proceedings of the 32nd ACM International Conference on Information and Knowledge Management; 2023;
97. Thirunavukarasu AJ, Ting DSJ, Elangovan K, Gutierrez L, Tan TF, Ting DSW. Large language models in medicine. Nature Medicine. 2023/08/01 2023;29(8):1930-1940. doi:10.1038/s41591-023-02448-8
98. Zhang Y, Li Y, Cui L, et al. Siren's Song in the AI Ocean: A Survey on Hallucination in Large Language Models. arXiv preprint. 2023;arXiv:2309.01219doi:https://doi.org/10.48550/arXiv.2309.01219
99. Wang C, Liu S, Yang H, Guo J, Wu Y, Liu J. Ethical Considerations of Using ChatGPT in Health Care. J Med Internet Res. 2023/8/11 2023;25:e48009. doi:10.2196/48009
100. Mynatt C, Doherty M, Tweney R. Confirmation Bias in a Simulated Research Environment: An Experimental Study of Scientific Inference. QUART J EXP PSYCHOL. 1977;29:85-95.
101. Dunbar K. How scientists think: On-line creativity and conceptual change in science. Creative thought: An investigation of conceptual structures and processes. American Psychological Association; 1997:461-493.
102. Dunbar K. How scientists really reason: Scientific reasoning in real-world laboratories. The nature of insight. The MIT Press; 1995:365-395.
103. Dunbar K. The analogical paradox: Why analogy is so easy in naturalistic settings, yet so difficult in the psychology laboratory. In: Gentner D, Holyoak K, Kokinov B, eds. Analogy: Perspectives from cognitive science. MIT Press; 2001:323-334.
104. Patel VL, Kaufman D, (Eds) TC. Cognitive Informatics in Health and Biomedicine: Case Studies on Critical Care, Complexity and Errors Springer; 2014.
105. Statistics CNCfH. NAMCS datasets and documentation. 2017;
106. Jing X, Zhou YC, Cimino JJ, et al. Development and preliminary validation of metrics to evaluate data-driven clinical research hypotheses. 2022:
107. Jing X, Patel V, Cimino J, Shubrook J. Hypothesis generation in clinical research: challenges, opportunities, and role of AI. IOS; 2022:
108. Draghi B, Ernst M, Patel V, et al. Number of scientific hypotheses and time needed in a 2-hour study session among inexperienced clinical researchers—preliminary results. Mar 18-21, 2023:
109. Brooke J. SUS - A quick and dirty usability scale. Reading, UK.
110. Brooke J. SUS: a retrospective. J Usability Studies. 2013;8:29-40.
111. Jing X, Patel V, Cimino J, Georgiou A. Scientific hypothesis generation in clinical research: cognition, visualization, and evaluation. IOS; 2023: