Nonetheless, this is often where biomarker research begins. Guidance for reporting diagnostic accuracy studies is available and, contrary to frequent practice, should receive systematic application in biomarker investigations.

At a minimum, the evaluation of a candidate biomarker should not be limited to sensitivity and specificity, but should be expanded to include:

• Positive and negative likelihood rates

• Positive and negative predictive rates

• False discovery rates

• Area under the ROC (receiver operating characteristic) curve

These reports must include confidence intervals.

Willful or inadvertent errors in diagnostic hypothesis testing frequently occur. Head et al. concluded that “p-hacking is widespread in science.” In part, this reflects a misunderstanding of the p-value. A growing body of opinion maintains that better alternatives exist. This is considered in the American Statistical Association’s statement on p-values.

A vigorous summary of the views of some statisticians is given in Wasserstein et al.’s “Moving to a world beyond p-values”. Additional opinions should also be considered. Extensive documentation addressing the discussion is available online at the American Statistical Association.

In the specific context of classification, the important assessment is probability of classification error, PERRORP_{ERROR}PERROR​.

A low between-group p-value does not ensure successful classification. Rapp et al. constructed a two-group classification example where p=2×10−11p = 2 \times 10^{-11}p=2×10−11, but PERROR=0.4078P_{ERROR} = 0.4078PERROR​=0.4078.

Recalling that the error rate of random classification in a two-group classification is PERROR=0.5P_{ERROR} = 0.5PERROR​=0.5, it is seen that little better than random performance was achieved. In the case where a single continuous variable is used in a two-group classification, and where the distributions of the discriminating variable in each group are normal (or near-normal), it is possible to predict PERRORP_{ERROR}PERROR​ with the number of participants in each group and the group means and standard deviations.

This makes possible a re-assessment of some previously published candidate biomarker papers. These post-facto calculations can be disheartening.

III. Model selection and validation

Biomarker classifier performance can often be improved by incorporating more variables into a multivariate classification, but more is not necessarily better.

Watanabe et al. published an EEG classification example where PERRORP_{ERROR}PERROR​ decreased as measures were eliminated from the classifier.

Mathematically informed model selection is essential. Model selection is a process in which candidate measures are either incorporated into or rejected from a classifier.

LASSO (Least Absolute Shrinkage and Selection Operator) is often used in biomedical research and has the advantage of preventing overfitting and presenting results in a form that is readily interpretable.

An extension of this method, elastic net model selection, can outperform LASSO on some problems (Zou and Hastie show that LASSO is a special case of elastic net LARS-EN). Random forest methods can also be used in model selection.

As in the case of classification algorithms, the recommendation is to perform biomarker model selection with more than one algorithm. Significant divergences in selection should be investigated.

Once a model has been constructed, cross validation is a commonly used procedure for model validation. This procedure is not robust against misapplication. Indeed, errors in cross validation are so common that the standard textbook for statistical learning (Hastie et al.) includes the section:

“The wrong and the right way to do cross validation.”

Misapplied, cross validation can produce sensitivity, specificity, and positive prediction values in excess of 0.95 with random numbers.

IV. Longitudinal monitoring, quantifying change

A biological measure becomes a candidate biomarker if its values depart from normal values in a consistent manner when obtained from a well-defined clinical population.

Ideally, a candidate monitoring biomarker would accurately track clinically perceived patient status. For example, the value of the biomarker would increase as the patient improved. Reality is not so simple.

In some instances, the value of the candidate biomarker is largely unchanged even though the patient is improving. Kemp et al. provides an example. Heart rate variability was found to be reduced in depressed patients but did not change when patients improved clinically in response to medication.

There are several reasons that can explain the failure of a candidate monitoring biomarker to track clinical status. The first, and possibly most common, is that the original identification as a biomarker was flawed. Sample sizes in clinical studies can result in false positive identification of biomarkers. Hajcak et al. have directed attention to another possibility. It is possible that the candidate biomarker is not correlated with the disease process itself but rather is correlated with a risk factor for the disease. When this occurs, population studies will identify a correlation between the presence of the biomarker and the disease, but the biomarker will not change in response to treatment.

It is commonly argued that even if a biomarker’s between-diagnostic-group specificity is low, it may still be useful when used longitudinally to monitor patient status. Body temperature is an example. Used alone, temperature has low diagnostic specificity, but used longitudinally it can be an important indication of improvement or deterioration. Temperature is an effective monitor because:

1. Thermometers give precise measurement of temperature.

2. Temperature is stable in a clinically stable individual.

Thus, a critical step in establishing the utility of a candidate monitoring biomarker is determination of its test-retest reliability. In the case of psychophysiological biomarkers, the literature on test-retest reliability is limited and discouraging. This can also be true of neuropsychological assessments.

For example, Cole et al. conducted a reliability study of four neuropsychological batteries used in pre- and post-concussion assessments. They concluded:

“However, small test-retest reliabilities in four NCATs (Neuropsychological Assessments Tests) in a military population are consistent with reliability reported in the literature (non-military populations) and are lower than desired for clinical decision making.”

Reliability studies of biomarkers are now an urgent requirement. Linear correlation should not be used to quantify reliability. Reliability for continuous data should be quantified by the intraclass correlation coefficient (ICC). It is important to note that there is more than one variant of the ICC.

• Shrout and Fleiss have identified six versions.

• McGraw and Wong give ten versions.

• Müller and Büttner and Koo and Li provide directions for selecting among the possible ICC versions.

In a report of ICC results, it is essential to identify the version used and to include confidence intervals.

When the intraclass correlation coefficient is available, change in a biomarker can, to a degree, be interpreted by calculating the corresponding minimum detectable difference (MDD) and the standard error of measurement (SEM). It must be understood that the MDD and SEM characterize properties of measurement distribution. They are not equivalent to the minimal clinically important difference, which must be calculated by very different procedures.

Reliability as quantified by the ICC is necessary but not sufficient for clinical utility. Interpretation of the ICC is further complicated when it is recognized that the ICC is specific to a population. Young healthy controls typically have the greatest reliability. Some clinical populations have low reliability — for example, head injury patients. A change in a biomarker that is unremarkable in an elderly patient could potentially be a matter of concern in a young adult.

V. Sample sizes in biomarker validation

Inadequately powered studies are a severe and continuing issue in neuroscience.

In “Power failure: why small sample size undermines the reliability of neuroscience,” Button et al. reported:

“Here we show that the average statistical power of studies in neuroscience is low. The consequence of this includes overestimates of effect size and low reproducibility.”

We have published an example from our own research group. The investigation was directed to determining if event-related potentials.

obtained from returning service personnel could identify those who would present delayed onset PTSD in six months or one year after return from combat. The initial analysis indicated Sensitivity=0.8 and Specificity=0.87. Prior to publication, a further analysis confirmed those values of sensitivity and specificity but found that the corresponding confidence intervals were [0,1]. The sensitivity and specificity values were a fortuitous consequence of a small sample size. This example emphasizes two points; small sample sizes can produce false positive indications of biomarker utility, and, second, the determination of confidence intervals is an essential element of every analysis.

Sample size requirements are specific to the objective of the investigation. Zou has provided guidance for intraclass correlation coefficient test-retest studies. It should be noted, and Professor Zou has acknowledged, that there are a number of typographic errors in the equations in this paper. Our calculations have independently confirmed the contents of the sample size tables in Zou. Notably, the sample sizes identified by this analysis are much larger than those typically reported in biomarker test-retest studies.

Prodrome studies are constructed by identifying a population of individuals who do not meet diagnostic threshold for a disorder but where there is a reason to believe that a significant fraction of that population will subsequently present the disorder (an enriched population). The sample size required for this study will be critically dependent on the fraction of the intake population that will present the disorder (the converter versus stable fraction). If the converter fraction is low, which is often the case in neuropsychiatric disorders, the sample size required can be large. Consider a conversion rate of 10%, which is higher than typically obtained, and an acceptable uncertainty in sensitivity of ±0.1. Hoeffding’s inequality requires 185 converters and therefore a study population of 1850. Hoeffding’s criterion fixes the precision of the interval independently of the value of sensitivity and therefore gives a conservative (greater) estimate of the required sample size than the criterion constructed by Clopper and Pearson which requires 104 converters and a total population of 1048. The requirement for large sample sizes in prodrome studies, which by definition are longitudinal studies and therefore expensive, may be a critical limiting factor in prodrome discovery.

VI. Paths forward

The challenge of mathematical analyses of data obtained in biomarker studies should not be underestimated. An instructive example has been published by Botvinik-Nezer, et al. In this study the same fMRI data set was sent to 70 independent teams for analysis. They report that “no two teams chose identical workflows to analyze the data. This flexibility resulted in sizable variations of the results of hypothesis tests, even for teams where statistical maps were highly correlated at intermediate stages of the analysis pipeline.” No suggestion is made of incompetence or misbehavior. Simply put, the challenges of large data sets be it imaging data, electrophysiological signals, genetic studies or their combination are formidable.

In the specific case of psychophysiological biomarkers, the Society for Psychophysiological research made several recommendations that readily generalize to other types of biomarkers. The recommendations included data sharing with data format harmonization, analysis software sharing and preregistration of analysis plans. The free availability of powerful analysis software carries the danger of misapplication. To a degree, this can be addressed by applying software to be used in an anticipated study to publicly available data as part of the software validation process. Public data sources are expanding daily and include ERP data, connectome data, MEG data, EEG data, and genomic data (the NIH GenBank).

It has been proposed that preregistration of a research analysis plan can reduce bias, increase transparency and facilitate reproducibility of research. Registration of an analysis plan is not limited to a description of computational procedures. It includes

description of the study design, motivating hypotheses, variables, and data collection methods as well as planned statistical analysis. This document can be submitted to a public online repository such as the Open Science Framework. Preregistration resources are available at the American Psychological Association website.

Systematic implementation of data sharing, software sharing and preregistration will not come easily or quickly to the biomedical research community. We nonetheless suggest that the path forward in biomarker discovery will not only require scientific advances but also cultural change.

VII. Conclusion

An historical opportunity to advance clinical practice with biomarkers is now available to us. Data acquisition capabilities have expanded exponentially. Computational resources have expanded at a similar rate. Additionally, the essential underlying mathematical methods needed for large scale data analysis have been constructed and validated. The clinical realization of this opportunity has, however, expanded at a slower pace. Many factors contribute to this. In this contribution we identified four areas that have the advantage of being immediately addressable.

1. Appropriate mathematical methods for classifying individuals into recognized clinical populations have been developed.

2. It is recognized that the mathematical methods of statistical learning are not robust against misapplication, but procedures to prevent misapplication have been identified and can be implemented.

3. The critical need for systematic test-retest reliability studies has been established, and appropriate procedures for quantifying reliability are available.

4. Mathematical results establishing the need for much larger sample sizes in biomarker studies have been established, and sample size guidelines specifically for biomarker studies have been published.

The magnitude of the challenges ahead should not be underestimated, but, conversely, the magnitude of the opportunity should not be underestimated.

Author Contributions:

P.E.R. conceptualization, literature review, first draft;
A.M.K.G. revision of the final draft. Both authors have read and agreed to the published version of this manuscript.

Conflicts of Interest:

The authors declare no conflict of interest.

Funding:

This research did not receive external funding.

Institutional Review Board Statement:

Not applicable. This contribution is a review of publicly accessible previously published literature.

Informed Consent Statement:

Not applicable.

Data Availability Statement:

Not applicable.

Disclaimer:

The opinions and assertions contained herein are those of the authors and do not necessarily reflect the official policy or position of the Uniformed Services University, the US Department of Defense, or the Henry M. Jackson Foundation for the Advancement of Military Medicine.