AUC Estimation

Biometrics59,614–623September2003Partial AUC Estimation and RegressionLori E.Dodd1,∗and Margaret S.Pepe2,∗∗1Biometric Research Branch,National Cancer Institute,6130Executive Blvd,MSC7434,Rockville,Maryland20892,U.S.A.2Department of Biostatistics,University of Washington,Box357232,Seattle,Washington98195,U.S.A.∗email:doddl@∗∗email:mspepe@Summary.Accurate diagnosis of disease is a critical part of health care.New diagnostic and screening tests must be evaluated based on their abilities to discriminate diseased from nondiseased states.The partial area under the receiver operating characteristic(ROC)curve is a measure of diagnostic test accuracy.We present an interpretation of the partial area under the curve(AUC),which gives rise to a nonparametric estimator.This estimator is more robust than existing estimators,which make parametric assumptions.We show that the robustness is gained with only a moderate loss in efficiency.We describe a regression modeling frameworkfor making inference about covariate effects on the partial AUC.Such models can refine knowledge about test accuracy.Model parameters can be estimated using binary regression methods.We use the regression framework to compare two prostate-specific antigen biomarkers and to evaluate the dependence of biomarker accuracy on the time prior to clinical diagnosis of prostate cancer.Key words:Diagnostic testing;Mann-Whitney U-statistic;Regression;Receiver operating characteristic curve.1.IntroductionScreening and diagnostic tests are familiar and ever-evolvingtools of modern medicine.Populations of healthy individu-als characterized as at-risk are commonly screened for dis-eases such as cancer and heart disease.Early detection byscreening is considered essential to alleviate disease burden,and considerable resources have been devoted to developingnew screening tests.New diagnostic tests that are less inva-sive,less expensive,and more accurate than existing proce-dures are sought for diagnosis of many conditions.Technolo-gies that measure gene and protein expression,as well as newimaging procedures,all hold promise in this regard.Prior towidespread application,however,rigorous evaluation of testaccuracy and of factors that effect it is compulsory.Inherent in the analysis of screening and diagnostic tests arecosts and benefits,both monetary and nonmonetary,associ-ated with true-positive and false-positive diagnoses.Considera continuous test result Y for which Y>c indicates a positivetest result,and let D and¯D denote diseased and nondiseasedstates,respectively.The true-positive rate at a threshold c,TPR(c),is defined as P(Y>c|D)≡S D(c).The correspondingfalse-positive rate,FPR(c),is P(Y>c|¯D)≡S¯D(c).Costsand benefits are associated with any given{FPR(c),TPR(c)}pair.The receiver operating characteristic(ROC)curve plots{FPR(c),TPR(c)}for all possible thresholds c,and providesa visual description of the trade-offs between TPRs and FPRsas one changes the threshold stringency(Figure1).We canwrite the ROC curve as a function of t=S¯D(c)as follows:ROC(t)=S D{S−1¯D (t)}.An uninformative test is representedby a straight line from the(0,0)vertex to(1,1),while acurve pulled closer towards the(0,1)vertex indicates a better-performing test.Frequently,the best threshold is not known when a test isunder evaluation,and it may vary depending on the settingin which the test is implemented.A summary measure thataggregates performance information across a range of pos-sible thresholds is desirable.The area under the ROC curve(AUC),defined as1ROC(t)dt,summarizes across all thresh-olds,and is the most commonly used measure of diagnosticaccuracy for quantitative tests.However,the AUC summa-rizes test performance over regions of the ROC space in whichone would never operate,i.e.,for{FPR(c),TPR(c)}values ofno practical interest.In population screening,large monetarycosts result from high false-positive rates;hence the region ofthe curve corresponding to low false-positive rates is of pri-mary interest.In diagnostic testing,it is critical to maintaina high TPR in order not to miss detecting subjects with dis-ease.In this case,interest is in the region of the ROC curvecorresponding only to acceptably high TPRs.In this article,we consider a summary index for the ROC curve restricted toa clinically relevant range of false-positive rates.The partialAUC isAUC(t0,t1)=t1t0ROC(t)dt,(1)where the interval(t0,t1)denotes the false-positive rates of in-terest.The analogue that restricts to a range of true-positiverates will also be discussed.Selecting the interval(t0,t1)is an614Partial AUC Estimation and Regression615 important practical issue.The choice depends on the particu-lar setting and should depend on the costs of a false-positivediagnosis,as well as the benefits of a true positive.Baker(2000)develops a“utility”function to specify a target par-tial ROC region.Obuchowski(1997)uses decision analysis toassociate patient outcome with desirable diagnostic accuracyvalues.Such methods could be adapted,for example,to de-termine a maximum allowable false-positive rate or the lowestdesirable true-positive rate.This is a complex area requiringinput from health services and economic specialists.Although the partial AUC has been proposed before(McClish,1989;Thompson and Zucchini,1989;Jiang,Metz,and Nishikawa,1996)and has gained popularity,particularlyin screening research(Baker and Pinsky,2001),little attentionhas been devoted to statistical inference about it.We providea probabilistic interpretation for the partial AUC that givesrise to a novel nonparametric estimator.Through simulationstudies,the estimator is compared with the existing estima-tors that are all based on parametric assumptions.We showthat the increased robustness of the nonparametric estima-tor is gained at the expense of a moderate loss in efficiency.Then we present a regression framework for evaluating co-variate effects on the partial AUC.The approach extends amethod developed recently for regression analysis of the fullAUC(Dodd and Pepe,2003).Since the partial AUC has moreappeal in many practical settings,this represents an impor-tant generalization.The work is motivated by a study of prostate-specific anti-gen(PSA),a serum biomarker.PSA is a screening tool forprostate cancer that has been the focus of much research.Im-portant questions to consider when evaluating this biomarkerare,which form of PSA is best(total or ratio of free to totalPSA),and by how long does this test advance the lead time,or time prior to clinical diagnosis of disease?In this article,we show that a parsimonious regression model of the partialAUC with PSA type,and lead time as covariates,assists withthis type of evaluation.2.Partial AUCThe partial AUC is simply the area under the ROC curvebetween t0and t1(Figure1).With an uninformative test,TPR(c)=FPR(c)for all c,and the partial AUC is the areaof a trapezoid equal to1/2(t1+t0)(t1−t0).For a perfect test,ROC(t)=1for all t∈(0,1),and the partial AUC is the areaof the rectangle with height1and base t1−t0.2.1Interpretations of the Partial AUCAssume Y D and Y¯D are continuous random variables withsurvivor functions S D and S¯D,respectively.Let F=1−S.The partial AUC is the joint probability that Y D>Y¯D andthat Y¯D falls within the range of clinically relevant quantiles.To see this,observe that:AUC(t0,t1)=t1t0ROC(t)dt=t1t0S DS−1¯D(t)dt=S−1¯D(t0)S−1¯D(t1)S D(y)dF¯D(y)FPRTPR0.00.20.40.60.8 1.01Figure1.Illustration of an ROC curve and its partial AUC(t0=0.1,t1=0.3).=S−1¯D(t0)S−1¯D(t1)S D(y)f¯D(y)dy=PY D>Y¯D,Y¯D∈S−1¯D(t1),S−1¯D(t0)(2)To simplify notation let q0=S−1¯D(t0)and q1=S−1¯D(t1).Thepartial AUC can also be written as a weighted conditionalexpectation AUC(t0,t1)=(t1−t0)P{Y D>Y¯D|Y¯D∈(q1,q0)}.A second interpretation arises from the concept of place-ment values(Pepe,2003).The term S D(y¯D)is the placementof a given nondiseased test result,Y¯D=y¯D,in the survivorfunction of results of diseased.For a good test,the nondis-eased observations fall in the lower tail of the distributionof diseased.Hence,larger placement values indicate a bettertest.Hanley and Hajian-Tilaki(1997)showed a connectionbetween average placement values and the full AUC.We ex-tend their result to the partial AUC here,as it provides aninteresting interpretation of the partial AUC,as well as a sim-pler estimator:P{Y D>Y¯D,Y¯D∈(q1,q0)|Y¯D=y¯D}=P{Y D>y¯D,y¯D∈(q1,q0)}=I{y¯D∈(q1,q0)}P(Y D>y¯D)=I{y¯D∈(q1,q0)}S D(y¯D)≡Pl D(y¯D)(3)We refer to Pl D(y¯D)as a“restricted placement value”(withrespect to the distribution of results from the diseased pop-ulation).Note that a full placement value is defined as in(3),except there is no restriction to Y¯D∈(q0,q1).The re-stricted placement is weighted so that only those values thatfall within the relevant quantiles are considered.Note thatE{Pl D(Y¯D)}=AUC(t0,t1).Thus,the partial AUC can bethought of as an average of restricted placement values.It616Biometrics,September2003 is straightforward to show that the partial AUC is also theexpected restricted placement value from conditioning on ob-servations from the diseased population.One may wish to rescale the partial AUC,especially inregression analysis(see Section5).We define the partial AUCodds asAUC(t0,t1)/{(t1−t0)−AUC(t0,t1)}=P{Y D>Y¯D|Y¯D∈(q1,q0)}/P{Y D<Y¯D|Y¯D∈(q1,q0)}.(4) This is the ratio of the probability of a correct ordering ofa randomly selected diseased and nondiseased test result tothe probability of an incorrect ordering,with both probabili-ties conditional on the test result of nondiseased arising fromthe region of interest.These odds have value of(t1+t0)/{2−(t1+t0)}when a test is uninformative,and of infinityfor a perfect test.2.2Restricting the True-Positive RangeThe partial AUC just described restricts the ROC region ofinterest to false-positive rates that take values in(t0,t1).Insome settings,one may wish to restrict to a range of true-positive rates(Jiang et al.,1996).By transforming the ROCcurve to a plot of{S D(c),1−S¯D(c)},interpretations of a par-tial AUC corresponding to true-positive rates in an intervalare easily obtained.The curve is no longer the classic ROCcurve.It is simply a270◦rotation of Figure1.We refer to thiscurve as the specificity-ROC curve(ROC spe),since specificityis plotted on the y-axis.For u=S D(c),this curve is describedby ROC spe(u)=1−S¯D{S−1D(u)}=F¯D{S−1D(u)}.The partialAUC for a range of true positives,denoted AUC TP(u0,u1),isdefined asu1u0ROC spe(u)du=S−1D(u1)S−1D(u0){1−S¯D(y)}dS D(y)=PY D>Y¯D,Y D∈S−1D(u1),S−1D(u0).(5)Note that1S D(y)dS¯D(y)=1{1−S¯D(y)}dS D(y).Therefore,the two representations give the same full AUC. However,the quantiles that define the partial AUC are{S−1¯D (t1),S−1¯D(t0)},and are derived from the nondiseaseddistribution for the classic ROC curve.For the case justpresented,the quantiles are{S−1D (u1),S−1D(u0)},and arisefrom the distribution of tests of diseased subjects.With this exception,the interpretations are the same.Thus,we focus on the classic ROC curve,recognizing that methods developed here easily extend to the case in which restriction of true-positive rates is required.3.Estimators3.1Parametric EstimatorsBefore proposing our nonparametric estimator,we briefly de-scribe existing estimators.McClish(1989)describes what we refer to as the normal-distributions partial AUC esti-mator(NDE),which assumes that the test results for dis-eased and nondiseased populations follow normal distribu-tions with different means and variances.Maximum likelihood estimates of these parameters provide the ROC curve esti-mate,and numerical integration of it gives the partial AUC es-timator:AUC(t0,t1)=t1t0Φ{ˆa+ˆbΦ−1(t)}dt.HereΦdenotes the standard normal cumulative distribution function(CDF),ˆa=(ˆµD−ˆµ¯D)/ˆσD andˆb=ˆσ¯D/ˆσD,whereµandσ2denote mean and variance,respectively.Another approach is to parameterize the ROC curve di-rectly.The most common form is the binormal ROC curve, ROC(t)=Φ{a+bΦ−1(t)},for which the partial AUC is given by the same formula above.However,because this approach does not parameterize the distributions of test results,but only the ROC curve that describes the relationship between their distributions,it stipulates far weaker assumptions.Two methods have been proposed for estimating parameters a and b of the binormal ROC curve(Metz,Herman,and Shen,1998; Pepe,2000).Both methods are distribution-free in that they are functions only of the ranks of the data.3.2Proposed Nonparametric EstimatorDenote V q0,q1ij=I{Y D i>Y¯D j,Y¯D j∈(q1,q0)}and observethat E(V q0,q1ij)=P{Y D>Y¯D,Y¯D∈(q1,q0)}=AUC(t0,t1), according to the interpretation of equation(1).This suggests the following method-of-moments estimator:AUC(t0,t1)=(n D n¯D)−1ijV q0,q1ij,(6)where n D and n¯D denote the number of observations from the population of diseased and nondiseased,respectively.In some circumstances,the quantiles(q0,q1)will be known. In others,they will not,in which case empirical quantile es-timates are substituted.If the empirical quantile value does not coincide precisely with the desired value,as may hap-pen with small sample sizes,values are linearly interpo-lated.Observe that,when t0=0and t1=1,AUC(t0,t1)= (n D n¯D)−1ijI(Y D i>Y¯D j).Hence,for the full AUC the es-timator reduces to the Mann-Whitney U-statistic,the classic nonparametric AUC estimator.Further,note that this results in the same value as the area calculated from the empirical ROC curve,using the trapezoidal rule when there are no ties in the data.The same estimator is derived by consideration of empir-ical restricted placement values.LetˆS denote an empirical survivor function and write the empirical placement value cor-responding to an observation from a disease-free subject,Y¯D, as:Pl D(Y¯Dj)=IY¯D j∈(q1,q0)ˆSDY¯D j=IY¯D j∈(q1,q0)(1/n D)n DiIY D i>Y¯D j.(7) The sample average is(1/n¯D)n¯DjPl D Y¯Dj=(n D n¯D)−1n¯Djn DiIY D i>Y¯D j,Y¯D j∈(q1,q0).Partial AUC Estimation and Regression617Thus,the nonparametric estimator is an average of restricted placement values within the disease reference distribution.Likewise,the estimator can be written as the average of the placement values within the nondiseased reference distribu-tion.This generalizes the corresponding results for the full AUC given in DeLong,DeLong,and Clarke-Pearson (1988)and Hanley and Hajian-Tilaki (1997).Calculations using placement values are considerably faster computationally,and are used in the simulations described next.Asymptotic distribution theory is nonstandard because thebinary indicators,V q 0,q 1ij,are cross-correlated.It can be shown that the projection (n D n ¯D )−1ijPl ¯D (Y D i )+Pl D (Y ¯D j )is asymptotically equivalent to (6).Since this projection is a sum of independent terms,standard theory provides consis-tency and asymptotic normality results.For complete details,refer to Dodd (2001).We recommend using the bootstrap to obtain variance estimates.4.Simulations4.1Small Sample Performance of the EstimatorThree different ROC models were simulated to evaluate per-formance of the proposed method of inference.The models simulated are a normal-distributions model,a proportional-hazards ROC model,and an extreme-value ROC model.The proportional-hazards ROC model assumes that the hazard function for the disease test result distribution is proportional to that of the nondisease distribution.If the ratio of hazards is r ,then ROC(t )=t r .The extreme-value ROC model in-volves two parameters (e ,f ),and is of the form ROC(t )=exp[−(1/e )exp {−f Φ−1(t )}](Cai and Pepe,2003).Extensive simulations of these models described in Dodd (2001)showed that the nonparametric method produced estimates with little bias and that confidence intervals using the bootstrap stan-dard error estimator and normal quantiles provided good cov-erage probability.We present results for a normal-distributions modelthat assumes Y D ∼N (1.5, 1.44)and Y ¯D ∼N (0,1)(see Figure 1).Partial AUCs are considered for (t 0,t 1)={(0,0.1),(0,0.2),(0.1,0.2),(0.1,0.3)}when quantiles are both known and estimated empirically.Sample sizes were gener-ated with both equal and unequal numbers in each group,such that (n D ,n ¯D )={(10,50),(10,100),(50,50),(50,100),(100,100),(100,200),(100,300)}.All resulted in estimates with little bias (Table 1).The largest amount of bias (5.4%)occurs for n D =10and n ¯D =50,for (t 0,t 1)=(0.1,0.2),when known quantiles are used.Bias becomes negligible as sample size increases and as more of the curve is integrated.In most cases,bootstrapped standard errors,computed with 200bootstrap samples,slightly overestimate the truth when the quantiles are estimated.In the case when quantiles are known,the bootstrapped standard error estimator is rea-sonably unbiased.Consequently,confidence interval coverage tends to be above the nominal level with estimated quantile values,but close to the nominal level with the known thresh-old.Surprisingly,when empirical quantiles are substituted,there is an increase in efficiency compared to the case when the true quantile is used.The true standard errors associatedwith AUC {t 0(ˆq ),t 1(ˆq )}are consistently smaller than those for AUC{t 0(q ),t 1(q )}.The reason for this is unclear to us,but it is a real phenomenon observed throughout our simulation stud-ies.Simulations suggest that the empirical quantile estimatoris negatively correlated with the partial AUC estimator,hence reducing the variance.However,explicit characterization of this correlation is difficult to obtain.4.2RobustnessTo investigate if the nonparametric estimator gains robust-ness over other estimators,we considered settings where the ROC curve deviated from the classic binormal form.We gen-erate an ROC model in which test results arise from mixturesof distributions.Assume that Y ¯D ∼N (0,1),but that Y D is a mixture of two normal distributions,f Z 1and f Z 2,with Z 1∼N (µD,1,σ2D,1)and Z 2∼N (µD,2,σ2D,2),such that the prob-ability density of Y D is p f Z 1(y )+(1−p )f Z 2(y ),for a mixing proportion p ∈(0,1).The resulting ROC curve is simply a mixture of ROC curves weighted by the appropri-ate mixing probabilities and can be expressed as ROC(t )=p Φ{a 1+b 1Φ−1(t )}+(1−p )Φ{a 2+b 2Φ−1(t )},where a i =(µD,i −µ¯D,i )/(σD,i )and b i =σ¯D,i /σD,i for i =1,2.We setµD 1=5,σ2D 1=1.2,µD 2=0,σ2D 2=1,and p =0.3.The true ROC curve is the solid line shown in Figure 2.We use t 0=0and three separate maximum false-positive rates,t 1=0.05,0.1,1.0.Very low false-positive rates,such as t 1=0.05,have been advocated in settings such as cancer screening (Baker and Pinsky,2001).Estimates of ROC curves from the NDE,the Pepe esti-mator (PPE)and the Metz estimator (MZE)are also given in Figure 2.Recall that the normal-distributions estima-tor assumes that test results are normally distributed in diseased and nondiseased ing sample meansand variances,the ROC curve is calculated as ROC(t )=Φ{ˆa +ˆb Φ−1(t )}with ˆa =(ˆµD −ˆµ¯D )/ˆσD and ˆb =ˆσ¯D /ˆσD .Clearly,all of these curves fail to accurately represent the true0.00.20.40.60.8 1.0FPRT P RFigure 2.Estimates of the ROC based on data from the mixture model.Shown are curves that use the average pa-rameter estimates.618Biometrics,September2003Table1Bias and coverage probability of the nonparametric method with known and estimated quantilesAUC{t0( q),t1( q)}AUC{t0(q),t1(q)}n D n¯D Bias S E(SE)×100CP Bias S E(SE)×100CPAUC(0,0.1)=0.0421050 3.28% 1.578(1.600)0.950.71% 2.315(2.344)0.88 10100 2.11% 1.443(1.492)0.950.11% 1.971(1.905)0.93 5050 4.30%0.109(0.101)0.96 1.90%0.200(0.192)0.90 50100 2.60%0.085(0.087)0.94<0.01%0.152(0.146)0.93 100100 1.50%0.075(0.072)0.95−0.20%0.138(0.139)0.94 1002000.20%0.06(0.061)0.940.60%0.102(0.103)0.95AUC(0,0.2)=0.1071050 1.20% 2.957(2.978)0.96 2.98% 4.198(4.388)0.92 101000.58% 2.761(2.848)0.940.14% 3.557(3.347)0.93 5050 1.50%0.185(0.172)0.97 1.60%0.347(0.338)0.92 50100 1.20%0.152(0.155)0.94 1.80%0.269(0.252)0.92 1001000.80%0.128(0.127)0.94−1.40%0.224(0.237)0.96 1002000.00%0.107(0.109)0.94−0.20%0.178(0.178)0.95AUC(0.1,0.2)=0.0651050−0.17% 1.929(1.593)0.99 5.40% 3.119(3.403)0.88 10100−0.43% 1.616(1.530)0.95−0.23% 2.479(2.569)0.92 5050−0.30%0.136(0.086) 1.00 1.40%0.284(0.280)0.92 501000.40%0.094(0.080)0.98 3.00%0.217(0.203)0.92 1001000.30%0.082(0.065)0.99−2.10%0.191(0.197)0.95 100200−0.20%0.061(0.055)0.96−0.80%0.147(0.144)0.94AUC(0.1,0.3)=0.1401050−0.27% 3.113(2.877)0.96 2.01% 4.871(4.915)0.92 10100−0.30% 2.820(2.774)0.94−0.11% 3.939(4.037)0.94 5050−0.30%0.194(0.151)0.99 1.40%0.416(0.414)0.95 501000.10%0.151(0.142)0.96 1.90%0.301(0.305)0.94 1001000.30%0.126(0.114)0.97−2.10%0.281(0.293)0.95 100200−0.10%0.103(0.099)0.950.40%0.218(0.216)0.95 Note:SE is bootstrapped standard error(×100)from200bootstrap samples,SE is true standard error(×100),and CP is coverage probability for95%confidence intervals using SE with a normal quantile.Results represent1000realizations of normal-distributions model.curve,which is not of the binormal form.They parameterize the ROC curve incorrectly,and hence produce biased par-tial AUC estimates(Table2).The largest bias was observed with the normal-distributions estimator.In contrast to this estimator and those of Pepe and Metz,the nonparametric es-timators produce estimates with very small bias.Observe that the amount of bias decreases for both the Metz and the Pepe estimators as more of the curve is integrated.This is consis-tent with results showing that the binormal model estimators produce AUC estimates that are robust to departures from this model(Hanley,1996).Additionally,note that the more parametric methods model and estimate the ROC over the entire(0,1)range and then integrate the relevant portion to determine the par-tial AUC.In contrast,the proposed method directly estimates the partial AUC over the false-positive rate range of inter-est.A more robust approach might be to model over(t0,t1), which can be accommodated by the Pepe approach.When the estimates are computed while restricting to(0,0.05)and (0,0.2),however,there is still bias with the Pepe method.The mean estimates restricting to the corresponding intervals areAUC(0,0.05)=0.013(−18%bias)andAUC(0,0.2)=0.044 (−40%bias).4.3EfficiencyWe compare the efficiency of the methods for estimating the partial AUC under the normal-distributions model.In this setting,the normal-distributions estimator is more efficient than the others,as it is the maximum likelihood estimator and makes use of the information that the data are nor-mally distributed.The results presented in Figure3indi-cate that the nonparametric estimator with known quantile, NPE(q),is the least efficient of the methods.In agreement with Table1,we observe that the nonparametric estimator is more efficient when the quantile is estimated than when the known value is used.The distribution-free binormal es-timators are more efficient than the nonparametric estima-tor,but have reduced efficiency relative to the approach that models the test results.Although very similar,the Pepe es-timator appears to be slightly more efficient than the Metz estimator.The efficiency of all methods approaches that of the normal-distributions estimator as t→1.Recall that when t=1,the nonparametric estimator is the Mann-Whitney U-statistic.The normal-distributions estimator is simply a transformation of the standardized differences in means,i.e., of the Z-statistic.Hence,the comparison of the variancePartial AUC Estimation and Regression619Maximum False Positive RateR e l a t i v e E f f i c i e n c y0.00.20.40.60.8 1.0Figure 3.Efficiency of estimators relative to the maximum likelihood normal-distributions estimator.Shown are results with sample sizes of 200per group and 2000replicates per scenario.of the nonparametric estimator with the variance of the normal-distributions estimator is akin to the comparison of the efficiency of the Mann-Whitney U-statistic to that of the Z-statistic.It is well known that the asymptotic relative ef-Table 2Bias in partial AUC estimates from ROC mixture model.Shown are results with 200samples per group and 1000simulations of the model.MethodEstimatePercent biasAUC(0,0.05)=0.0158Nonparametric estimator (ˆq )0.0161Nonparametric estimator (q )0.016<1Pepe estimator 0.003−81Metz estimator 0.01222Normal-distributions estimator 0.02346AUC(0,0.2)=0.074Nonparametric estimator (ˆq )0.0751Nonparametric estimator (ˆq )0.074<1Pepe estimator 0.043−41Metz estimator 0.0785Normal-distributions estimator 0.11049AUC(0,1)=0.649Nonparametric estimator 0.651<1Pepe estimator 0.634−2Metz estimator 0.6754Normal-distributions estimator 0.7099ficiency of the Mann-Whitney U-statistic to the t-statistic is 0.95(Lehmann,1997).Thus it is no surprise that,in this study,the relative efficiency is 0.94at t =1.4.4RecommendationsThe normal-distributions estimator,while most efficient,produces unacceptably biased estimates.Estimators that pa-rameterize the ROC curve are similarly not robust.The nonparametric estimator provides substantially more robust estimation.The added robustness is at the expense of mod-erate losses in efficiency.Hence,we recommend use of the nonparametric estimator.Indeed,for partial areas with t >0.1,it has similar efficiency to the Pepe and Metz methods.5.Regression AnalysisAccuracy may depend on information other than the test re-sult itself.For example,accuracy may depend on how close in time a subject is to clinical diagnosis.Patient characteristics,such as age or family history of disease,may also be relevant determinants of accuracy.This information may guide deci-sions about what populations are most likely to benefit from testing or for what populations test innovations are needed.We extend regression methodology we previously developed for the AUC to a regression methodology for the partial AUC summary index of test accuracy.For a more extensive discus-sion of modeling approaches for the AUC and details about fitting,refer to Dodd and Pepe (2003).Here,we elaborate only on aspects unique to the partial AUC.620Biometrics,September20035.1ModelsTo assess the effect of covariates on test accuracy,we pro-pose the following partial AUC regression models.Considera vector of covariates X.Define the covariate-specific partialAUC as AUC X(t0,t1)=P{Y D>Y¯D,Y¯D∈(q0,q1)|X}.Fora specified link function g,the general model is given byAUC X(t0,t1)=g(X Tβ)(8)Possible link functions include the logit or probit forms.How-ever,since AUC(t0,t1)has an upper bound of(t1−t0),a generalization of the logit that incorporates this constraint is appropriate:g−1(u,t0,t1)=log{u/(t1−t0−u)}.When thislink is used,an interpretation that corresponds to the partialAUC odds defined earlier follows.Consider the modellog[AUC X(t0,t1)/{(t1−t0)−AUC X(t0,t1)}]=β0+β1X.(9)It follows that eβ1is a ratio of partial AUC odds for X+1to X.When eβ1>1,the partial AUC odds are an increasingfunction of X,and accuracy increases with X.If eβ1<1,thepartial AUC odds are a decreasing function of X.Refer toDodd and Pepe(2003)for details about model specification.5.2Estimating FunctionConsider the indicators V(q0,q1)ij as defined in(6),butnow we condition on the covariates X.They have meanE{V(q0,q1)ij |X}=AUC X(t0,t1),which suggests that binary re-gression methods can be used to estimate partial AUC model parameters with the following estimating equation:V nD,n¯D (β)=n Din¯Dj(∂θX/∂β)ν−1(θX)V(q0,q1)ij−θX=0.(10)Here,θX=AUC X(t0,t1),ν−1(θX)is the variance of V(q0,q1),and(∂θX/∂β)is a(p×1)vector of the partial derivatives ofθX with respect to the model parametersβ.This resemblesthe classic estimating equation for binary regression.However,the binary indicators are cross-correlated in the sense that,fora given i,the set of binary variables{V ij,j=1,...,n¯D}are correlated because they are a function of Y D i.Consistency andasymptotic normality for parameter estimates from a similarestimating equation is developed in Dodd and Pepe(2003).The same theory applies here,at least when the quantiles(q0,q1)are known.See Dodd(2001)for a full exposition.5.3Comparing Two TestsThe proposed estimating equation gives rise to astandard approach when a model to compare twotests is of interest.Consider the following model:log[AUC X(t0,t1)/{t1−t0−AUC X(t0,t1)}]=β0+β1X, where X is an indicator of test type.The scorelike test of β1=0,based on the estimating equation in(10),reduces toAUC X=1(t0,t1)−AUC X=0(t0,t1),whereAUC(t0,t1)isdefined as in(6).This statistic is a member of the familyof statistics proposed by Wieand et al.(1989)that takesintegrated differences in weighted ROC curves to comparetwo tests.The Wieand et al.approach does not requirebootstrapping for inference,as does ours when the quantiles,q0and q1,are unknown.The current approach,on the otherhand,provides a more general regression framework.5.4ImplementationTo obtain parameter estimates,algorithms developed for bi-nary regression are used.First,however,the quantiles(q0, q1)must be specified.If known,they are simply substituted into(10).If quantiles are unknown and do not depend on co-variates,they can be estimated empirically from the nondis-ease data at hand.If the unknown quantiles depend on co-variates,a quantile regression model may be specified(see Heagerty and Pepe,1999).Then,the binary indicators based on pairs of diseased and nondiseased observations are com-puted.When covariates are categorical and there are suffi-cient observations of diseased and nondiseased within each category,all possible pairs of diseased and nondiseased within a given category are created.That is,the n kDand n k¯Dobserva-tions corresponding to the category or level of covariate k areselected and we calculate V(q0,q1)ijat each X for k=1,...,K. Note that the estimating function is modified to indicate the sum over k.When covariates are continuous,pairing of dis-eased and nondiseased observations(Y D i,Y¯D j)may be under-taken for all possible pairs,although we prefer to pair those observations only with covariate values near one another.Fi-nally,if covariates are unique to the diseased group,as with the“time prior to diagnosis”covariate in the PSA example that follows,the covariate does not restrict the pairing.Pair-ing is based only on covariates common to D and¯D.We refer to the article on AUC regression by Dodd and Pepe(2003)for a detailed discussion of pairing in the presence of covariates. The same considerations are relevant here.Logistic or probit regression estimation routines in stan-dard statistical packages can be used to solve the estimat-ing equations.Note that standard errors reported from these packages will not be valid,even with a robust sandwich vari-ance estimator,because of the crossed-correlation structure. We use the bootstrap to calculate the standard errors for pa-rameter estimates.Bootstrap samples are taken by sampling subjects as the primary unit.6.Prostate-Specific Antigen AnalysisThe data analyzed here are taken from a retrospective sam-pling of stored serum from theα-Tocopherol andβ-Carotene Study(ATBC),described by Heinonen et al.(1998).Al-though the primary goal of this study was to evaluate the effect of dietary supplementation ofα-tocopherol and β-carotene on lung cancer risk,the development of prostate cancer was also recorded.Additionally,serum samples were collected and stored both at baseline and three years later. For those240subjects who were diagnosed with prostate can-cer during the eight-year follow-up period,their serum sam-ples were retrospectively evaluated for prediagnostic levels of prostate-specific antigen.Age-matched serum samples for 237non-prostate-diagnosed subjects were selected for com-parative purposes.Two ways of quantifying PSA were con-sidered,the total PSA in serum(denoted by“total”)and the ratio of free to total PSA in serum(denoted by“ratio”). Etzioni et al.(1999)previously compared these two measures using a different dataset.In addition to comparing total to ratio PSA,we examine the effect on accuracy of time from serum sampling to clinical diagnosis.One would expect that PSA levels taken close to the time of clinical time of diagnosis would be more predictive.Let“test”denote an indicator that takes a value of one for total PSA and zero for PSA ratio.。