Small Sample Properties of Nonparametric Bootstrap t Confidence Intervals

Porter, Rao, Ku, Poirot, and DakinsISSN 1047-3289 J. Air & Waste Manage. Assoc. 47:1197-1203Copyright 1997 Air & Waste Management AssociationSmall Sample Properties of Nonparametric Bootstrap tConfidence IntervalsP. Steven Porter University of IdahoS. Trivikrama RaoState University of New York, Albany, New YorkJia-Yeong KuNew York State Department of Environmental Conservation Richard L. PoirotVermont Department of Environmental Conservation Maxine Dakins University of IdahoABSTRACTConfidence interval construction for central tendency is a prob-lem of practical consequence for those who must analyze air contaminant data. Determination of compliance with relevant ambient air quality criteria and assessment of associated health risks depend upon quantifying the uncertainty of estimated mean pollutant concentrations. The bootstrap is a resampling technique that has been steadily gaining popularity and ac-ceptance during the past several years. A potentially powerful application of the bootstrap is the construction of confidence intervals for any parameter of any underlying distribution.Properties of bootstrap confidence intervals were determined for samples generated from lognormal, gamma, and Weibull distributions. Bootstrap t intervals, while having smaller coverage errors than Student's t or other bootstrap meth-ods, under-cover for small samples from skewed distribu-tions. Therefore, we caution against using the bootstrap toconstruct confidence intervals for the mean without first con-sidering the effects of sample size and skew. When sample sizes are small, one might consider using the median as an estimate of central tendency. Confidence intervals for the median are easy to construct and do not under-cover. Data collected by the Northeast States for Coordinated Air Use Management (NESCAUM) are used to illustrate application of the methods discussed.INTRODUCTIONThe bootstrap is a resampling technique 1 that has been steadily gaining popularity and acceptance during the past several years. A potentially powerful application of the boot-strap is the construction of confidence intervals for any parameter of any underlying distribution.2 In this paper we address the use of the bootstrap technique to construct confidence intervals for the mean of an unknown distribu-tion using small sample sizes. Confidence interval construc-tion for central tendency is a problem of practical conse-quence for those who must analyze environmental moni-toring data. For example, determination of compliance with relevant ambient air quality criteria and assessment of as-sociated health risks depend upon quantifying the uncer-tainty of estimated mean pollutant concentrations.In this paper we focus on distribution-free techniques and small sample sizes. Confidence intervals for the mean of an unknown distribution can be based on Student’s T,provided the sample size is large. When the underlying distribution is not normal and sample sizes are small,Student's t will under-cover , meaning the probability thatIMPLICATIONSDetermination of compliance with relevant ambient air quality criteria and assessment of associated health risks depend upon quantifying the uncertainty of estimated mean pollut-ant concentrations. The nonparametric bootstrap t has be-come a popular method for constructing confidence inter-vals for the mean of an unknown distribution. However, us-ers should be aware that this method does not provide nomi-nal coverage probabilities when used with small samples from probability distributions typically used to characterize pollutant concentrations. It is suggested that using the sample median and associated confidence interval is a more reliable estimator of central tendency for this application.Porter, Rao, Ku, Poirot, and Dakinsthe interval contains the parameter of interest is less than nominal. A nominal 95% confidence interval based on small samples from a skewed distribution and Student's t statistic will contain the mean less than 95% of the time in repeated sampling. This paper provides some guidance about adequate sample sizes for use with Student's t.Bootstrap methods offer the promise of confidence inter-vals having coverage close to nominal. However, one must choose from among several bootstrap confidence interval methods that have appeared in the literature. Perhaps the most well known bootstrap confidence interval methods are the percentile method (BSP), the bootstrap t (BST), and the bias-corrected and accelerated bootstrap (BCA). Large sample theory has shown the BST to provide narrower intervals, with smaller coverage errors than the BSP or BCA methods.3 The BST is particularly applicable to location statistics, such as the sample mean, where there is an appropriate estimator of its standard error.4 When there is no obvious standard error formula for the parameter of interest, as is the case for the sample median, one may resort to the BCA or double bootstrapping.4 Large sample theory assists the choice of a method, but does not always provide a reliable indication of perfor-mance with small samples. In this paper, we use simula-tion studies and data collected by the Northeast States for Coordinated Air Use Management (NESCAUM) to illustrate the small sample properties of the BST. Although these net-works have a primary focus on visibility impairment, the resulting trace element data also provide an excellent op-portunity for assessing long-term exposures to a variety of potentially toxic trace metals and other substances.5 BST intervals, while having smaller coverage errors than Student's t or other bootstrap methods, under-cover for small samples from skewed distributions. Therefore, we caution against using the BST to construct confidence in-tervals for the mean without first considering the effects of sample size and skew. When sample sizes are small, one might consider using the sample median as an estimate of central tendency. Confidence intervals for the median are easy to construct and do not under-cover. In addition, me-dian confidence intervals can be constructed for highly censored samples. As sample sizes increase, the actual cov-erage of the BST approaches nominal more rapidly than Student's t, and, if the underlying distribution can be iden-tified, better methods, including a parametric bootstrap, can be applied.METHODSDatabase for ExamplesData collected by NESCAUM were used to illustrate the meth-ods. The NESCAUM Regional Particle Monitoring Network samples fine particulate matter (< 2.5 microns) at seven air monitoring stations in New York, New Jersey, Connecticut, Rhode Island, Maine, Massachusetts, New Hampshire, and Vermont. Twenty-four hour composite samples are collected on teflon filters every Wednesday, Saturday, and every sixth day that is not a Wednesday or Saturday, for a total of about 145 samples annually. Samples are analyzed by Crocker Nuclear Laboratory at the University of California at Davis for mass (gravimetric), light absorption (integrating plate), and multiple trace elements (proton elastic scattering analy-sis and proton-induced X-ray emission).6-8Zinc concentrations at Ringwood, NJ were chosen to illustrate the methods discussed in this paper. Annual mean and day-of-the-week concentrations are useful for assess-ing human health risk associated with long-term exposure to toxic air contaminants. The zinc concentrations found at these locations are well below the 24-hour-maximum standard of 0.15 ng/m3 for particulate zinc. However, ar-senic concentrations detected in the network, while often well above the standard, are difficult to characterize statis-tically or track because of censoring. In addition, large amounts of zinc and arsenic or other toxic substances may originate from the same source. Hence, there is interest in using zinc as a surrogate for particulate arsenic.Confidence IntervalsStudent's t. A 1-2α confidence interval for the mean us-ing Student's t is given by:CI =[XSn,XSn](1)where X, S, and n are the sample mean, sample standarddeviation, and sample size, respectively. Values for t1-α,n-1 and tα,n-1are ordinates at the 1-α and α points of Student's t distribution with n-1 degrees of freedom. BecauseStudent's t is symmetric, t1-α,n-1= - tα,n-1, and tables need provide only the positive ordinate.Bootstrap confidence intervals. The nonparametric bootstrap is a distribution-free technique in which the original sample of size n is resampled N times with replacement. The statis-tics of interest are calculated for each bootstrap resample and averaged. The sample mean and a bootstrap replica-tion of the sample mean are given by:=Xni=1ni∑(2)X1NXn*i*=−∑in1where X is the sample mean, xian element of the origi-nal sample , *X a bootstrap replica of the sample mean, and xi* an element of the bootstrap sample. Elements of the original sample may appear zero, one, or more timesPorter, Rao, Ku, Poirot, and Dakinsin the bootstrap resample.The bootstrap is not a useful estimation technique for the sample mean because it will provide the same result as the regular sample mean.1 However, the N bootstrap replicates have information about the dis-tribution of the sample statistic of interest and can be used to construct confidence intervals. For example,the bootstrap t statistic is formed from the empirical distribution of the following statistic:j j *j*T =X - X S(3)where j *and S j * are bootstrap realizations of the mean and standard deviation, respectively, and X is the sample mean.The BST confidence interval is found by inverting eq 3:4CI = [X T S, X T S]U L −⋅−⋅(4)where S is the sample standard deviation and T L and T U are the lower and upper critical points, respectively,of the T values from eq 3. Equation 4 is analogous to eq 1, with the upper and lower tails of the T distribu-tion corresponding to the lower and upper bounds of the interval, respectively. However, instead of consult-ing a standard table, a unique table is generated for each sample.5 The critical points of the T distribution are any two numbers that include 1-2α of the T val-ues. The BST converges to Student's t as the sample mean converges to normality.The BST has been shown in theory to be better for constructing confidence intervals for the sample mean (shorter with smaller coverage errors) than several other bootstrap methods (percentile, bias corrected, and accel-erator methods).4 Different BST intervals can be defined according to how critical values are selected. The equal-tailed 1-2α BST confidence interval is formed from the (α• N)th smallest and [(1-α) • N]th largest T values. The short-est interval is the pair of T values that has the shortest length and covers 1-2α of the T values. One might also want a symmetric interval, which is formed from those values covering 1-2α of the T values and providing equal width below and above the estimate.Correct parametric intervals for samples from a lognormal dis-tribution. If it can be assumed that the underlying distribu-tion is lognormal, correct confidence bounds can be con-structed using the method and tables found in Land.9 BST intervals were compared with Land’s correct intervals to give some idea of the penalty incurred by not knowing the underlying distribution.Median confidence intervals. Distribution-free confidence bounds for the median are sample order statistics. Order statistics are derived from samples that are arranged in size from smallest to largest. The central value of an odd-sized sample is the sample median. When the sample size is even, the sample median is usually taken as the average of the two central order statistics.A confidence interval for the median is a set of two order sta-tistics, one larger and one smaller than the sample median.The coverage of a given order statistic can be found in tables ofFigure 1. Coverage of nominal 90% two-sided BST interval for threeparent distributions: (a) gamma, (b) lognormal, (c) Weibull.Figure 2. Coverage of nominal 90% two-sided student’s t interval for three parent distributions: (a) gamma, (b) lognormal, (c) Weibull.Porter, Rao, Ku, Poirot, and Dakinsthe cumulative binomial or tables of confidence intervals for the median, both of which can be found in Breyer.10 For ex-ample, when the sample size is 20, the table indicates that the 6th and 15th order statistics (6th and 15th largest values in the sample) provide a 95.9% confidence interval. Confidence in-tervals constructed from order statistics do not under-cover.Simulation StudiesSynthetic independent samples were drawn from log-normal, gamma, and Weibull parent distributions hav-ing coefficients of variation of 0.5, 1, 2, 3, and 4. This set of distributions includes several with small depar-tures from normality (those with coefficients of varia-tion of 0.5) and exponential distributions (gamma and Weibull with coefficients of variation of 1.0). The set also includes several skewed distributions with differ-ing amounts of weight in the tails. Sample sizes were 10, 20, 50, and 100. Nominal coverage ranged from 90% to 99% for two-sided intervals (95% to 99.5% for upper one-sided intervals). Bootstrap confidence in-tervals were constructed from 2,000 bootstrap resamples per each original sample. Two thousand rep-etitions were carried out for each simulation.In all figures, sample coefficients of skew are aver-ages of sample coefficients of skew for given sample sizes and distributions. For small samples, the sample values of higher central moments are much smaller than their theoretical values.11 For a given coefficient of variation,the sample coefficient of skew increases with sample size,hence the different values of coefficient of skew for each point in the figures.RESULTS AND DISCUSSIONConfidence Interval CoverageThe coverage of the BST falls short of nominal for small samples from skewed distributions (Figure 1) but still has much better coverage than Student's t (Figure 2). For example, the coverage of a nominal 90% two-sided interval for a lognormal parent with sample coefficient of skew of 2.1 and sample size 20 is actually only about 81%. Coverage also varies by parent distri-bution. For a given sample coefficient of skew and sample size,gamma coverage is greater than W eibull, which in turn is greater than lognormal.Undercoverage occurs in skewed distributions because the upper bound tends to be too small. This is evident from coverage probabilities for 95% upper one-sided bounds (Fig-ure 3), which are the same as the upper bounds for the 90%two-sided intervals. As a typical example, actual coverage of the point mentioned above (coefficient of skewness of 2.1 and sample size of 20) is about 83.5%. This means that the upper bound failed to cover 17.5% of the time, while the lower bound failed to cover, on average, only 1.5% of the time (1 - 0.81 - 0.175).Width of Confidence IntervalsTo provide some idea of the dependence of confidence interval width on sample size and coefficient of skew,expressions for the width of two-sided confidence inter-vals using nominal coverage were derived from simula-tion studies (for each case, the actual mean = 1):gamma width z n R =⋅⋅⋅⋅=−3820964438122.,..αγ(5)upper bound mean znR =+⋅⋅⋅=−αγ4809142097.,.Figure 3. Coverage of nominal 95% upper one-sided BST intervalfor three parent distributions: (a) gamma, (b) lognormal, (c) Weibull.Figure 4. Width of shortest nominal 90% BST interval relative to equal-tailed BST interval for three parent distributions: (a) gamma,(b) lognormal, (c) Weibull.Porter, Rao, Ku, Poirot, and Dakinslog .,...normal width z n R =⋅⋅⋅=−1410971661522αγ(6)upper bound mean Z nR =+⋅⋅⋅=−75809721522.,..αγWeibull width z n R =⋅⋅⋅=−14509521522.,..αγ(7)upper bound mean znR =+⋅⋅⋅=−1710962672262.,...αγShortest BST intervals are, of course, shorter than equal-tailed intervals (Figure 4) but have larger coverage errors (Figure 5).Comparison of BST Widths with Correct Lognormal WidthsIt is useful to compare the widths of BST intervals con-structed with samples from lognormal distributions with correct widths computed using the method of Land.9However, a fair comparison requires that actual coverage be the same for the two methods. Because the actual cov-erage of the BST method is less than nominal, widths should be recomputed using a nominal coverage that pro-vides the desired actual coverage.Asymptotically, the coverage error of a nominal 1-2αtwo-sided, equal-tailed BST interval is given by:4coverage error =(8)n -1(κ-3γ2/2) • z 1-α • (2z 21-α + 1) • φz /6where κ and γ are the coefficients of kurtosis and skew-ness, respectively; and φz is the density of the standard nor-mal distribution at z, the 1-α point of the distribution. Thesample skewness and kurtosis are given, respectively, by:Figure 5. Coverage of shortest BST interval relative to equal-tailed BST interval, 95% nominal coverage, for (a) gamma parent, (b)lognormal parent, (c) Weibull parent.Figure 6.Comparison of asymptotic and small sample coverage errors.Figure 7. Nominal coverage providing actual 95% coverage for one-sided confidence bound.Figure 8. Upper 95% bounds of BST with nominal 95% coverage,Land’s correct method, and BST with actual 95% coverage.Porter, Rao, Ku, Poirot, and Dakins$ = $n (X -)-3-1i 3γσ⋅⋅Σ(9)$ = $n (X - X ) - 3-4-1i 4κσ⋅⋅Σwhere and 2$σ are the sample mean and variance, re-spectively.The sample coefficient of skew is a measure of symmetry,and the kurtosis a measure of the mass of the distribution thatis found in the tails. The normal distribution has a skew andkurtosis of 0. For large samples, eq 8 indicates the extent towhich departures from normality affect coverage error. Increas-ing skew also indicates increasing correlation between thesample mean and sample variance. Student's t is premised onindependence of the sample mean and variance, which be-comes true for any distribution as the sample size increases.However, for small samples, departures from normal skew havea greater effect on coverage error than departures from normalkurtosis;12 hence, the focus here on skewed distributions, ratherthan symmetric but heavy-tailed distributions.Equation 8, an asymptotic result, is optimistic in com-parison with simulation results using samples fromWeibull and lognormal distributions, and conservativewhen applied to samples from gamma distributions (Fig-ure 6). Therefore, an empirical expression for coverageerror was derived from the simulation results. For samplesfrom the lognormal distribution and the range of condi-tions used in the simulation, the coverage error of 1-2αconfidence intervals can be estimated from:lognormal coverage error (one-sided) =5.1 z (z) n $, R = 0.962.6-1.5 2.42⋅⋅⋅⋅αφγ(10)Equation 10 was used to compute thenominal coverage needed for an actual 95%one-sided confidence bound (Figure 7). Equa-tion 6 was then used to compute the value ofthat upper bound for lognormal distributions hav-ing a true mean of 1 (Figure 8). That the upper bound of the BST with nominal coverage of 95% is actually smaller than the correct upper bound is misleading,because we know that BST bounds under-cover (Fig-ure 3). The distance between the correct upper bound and that given by the BST with adjustednominal coverage, a fairer comparison, repre-sents the penalty for not knowing the trueunderlying distribution. What is striking aboutFigure 8 is the large increase in the upper boundthat results from apparently small changes in thenominal coverage. In fact, moving a few percent-age points in the tail of the distribution of the sample mean can be a large distance in terms ofthe underlying variable. For example, when the sample size is 20, the sample coefficient of skew is about 1.9 (parent coeffi-cient of variation 2), and the nominal coverage is increased to 99.4% to achieve 95% actual coverage, the upper bound in-creases from 1.7 to 3.0.Nominal coverage error can be adjusted automaticallyusing a method called an iterated bootstrap.13 An iterated boot-strap uses a nominal coverage equal to the desired coverageplus the coverage error given by an expression such as eq 8.However, eq 8 is not accurate for small samples (Figure 6),meaning that an iterated bootstrap, while possibly an improve-ment over the simple BST , may still undercover for samplesfrom lognormal and Weibull distributions. Eq 10 is not usefulfor computing iterated BST intervals either, because it is validonly for samples from lognormal distributions.Application of Methods to NESCAUM DataSummary statistics. Summary statistics were calculated forzinc concentrations at Ringwood, NJ, using data from 1990(Table 1). Coefficients of skew ranged from 1.52 to 2.61.Confidence bounds for means and medians . Confidencebounds were calculated for mean zinc concentrations us-ing the BST and Student's t (Table 2). Most striking per-haps is the disparity in the size of intervals for BST andStudent's t at sample sizes of 8 and 49, both of whichwere nominally 95%. Lower bounds were not very differ-ent, but upper bounds for the BST were much larger thanfor Student's t. The disparity can be explained by differ-ences in actual coverage. The actual coverage for samples of size 50 originating from a lognormal distribution witha sample coefficient of skew of 2.6 is only about 82% for Student's t and about 88% for the BST. The disparity in coverage decreases with increasing sample size (Figures 1-3).Table 1. Summary statistics for zinc concentrations (in ng/m 3) (Ringwood, NJ, 1990).Day of Week Sample Size Mean Median Std. Dev. Coef. Var. Coef. Skew Tue 8 13.2 10.0 8.4 0.64 1.85Sat 49 13.5 10.8 11.4 0.85 2.61Annual 142 14.5 11.4 11.5 0.79 1.52Table 2. Confidence bounds for means of zinc concentrations (ng/m 3) (Ringwood, NJ, 1990). Student’s t BST Day Sample LowerUpper Two-sided Lower Upper Two-sidedSize (5%)(95%)Width (5%)(95%)widthTue 8 7.6 18.8 11.2 8.53 34.9 26.4Sat 49 10.8 16.2 5.4 10.9 18.2 7.3Annual 14212.9 16.1 3.2 12.9 16.6 3.7Porter, Rao, Ku, Poirot, and DakinsMost of the undercoverage occurs at the upper bound. Student's t always produces intervals symmetric about the sample mean, while the BST produces asymmetric intervals from skewed samples, reflecting the shape of the distribution of the sample mean. Considering the simulation study, the BST should pro-vide intervals with close-to-nominal coverage when used with sample sizes and coefficients of skewness like those in the an-nual NESCAUM data.Confidence intervals for the median (Table 3) were calcu-lated from order statistics using tables that indicate the order statistics that will provide the nominal coverage10 indicated. Order statistic intervals are distribution-free and do not under-cover. In addition, when used with skewed samples, order sta-tistics tend to produce confidence intervals that are asymmet-ric with respect to the median.Widths of intervals for the median cannot easily be compared to those for means, because coverage depends on sample size. For large samples, however, the nominal coverage using order statistics approaches 95%, enabling comparisons. With respect to the example, confidence intervals for the median are sometimes smaller than in-tervals for the mean (Table 3).SUMMARY AND RECOMMENDATIONSThe most common method used by non-statisticians for computing confidence intervals for the mean from small samples is Student's t. The actual coverage of Student's t is much less than nominal for samples from skewed parent distributions, even for quite large sample sizes. The BST is a distribution-free method that, while providing better coverage than Student's t for samples from non-normal parent distributions, is still less than nominal.The information presented in this paper can be used to temper judgment about health or environmental risks based on approximate methods such as the BST. Certainly, the BST is preferable to Student's t, and being aware of a method’s limitations is preferable to ignorance of them. The practical implication is that to achieve confidence intervals with nominal coverage the sample size may have to be greatly increased.An alternative is to use the median as an estimate of central tendency. Confidence in-tervals for the median are easy to construct anddo not undercover. Another alternative is tocharacterize concentrations using extreme val-ues (for example, the 95th percentile concen-tration). Many articles have appeared describ-ing bootstrap methods for estimating percen-tiles. However, confidence intervals for percen-tiles are very wide and strongly dependent onsmall changes in the tails of distributions. Per-centiles may be useful supplements to estimatesof central tendency, but alone will not provide much information when derived from small samples.As a final word, we caution against using the BST to con-struct confidence intervals for the mean without considering the effects of sample size and skew. When sample sizes are large, actual coverage approaches nominal, and if the underly-ing distribution can be identified, a parametric bootstrap4 can be used and coverage corrections, as described above, can be applied. Some guidance as to appropriate sample sizes for good-ness-of-fit testing can be found in reference 14.ACKNOWLEDGMENTSThis work was supported by Northeast States for Coordi-nated Air Use Management and the U.S. Environmental Protection Agency.REFERENCES1.Efron, B. The Jackknife, the Bootstrap and Other Resampling Plans; Soci-ety for Industrial and Applied Mathematics: Philadelphia, PA, 1982.2.Rao, S.T.; Sistla, G.; Pagnotti, V. “Resampling and extreme value sta-tistics in air quality model performance evaluation,” Atmospheric En-vironment1985, 19, 1503.3.Hall, P. “Theoretical comparison of bootstrap confidence intervals,”Annals of Statistics1988, 16, 927.4.Efron, B.; Tibshirani, R.J. An Introduction to the Bootstrap; Chapmanand Hall: New York, 1993.5.Porter, P.S.; Rao, S.T.; Ku, J.; Poirot, R.L. Statistical Methods for the Analy-sis of Pollutant Concentrations Affected by Measurement Uncertainty, Project report submitted to Northeast States for Coordinated Air Use Management, Boston, MA, 1993.6.Flocchini, R.G.; Cahill, T.A.; Eldred, R.A.; Feeney, P.J. “Particulate sam-pling in the northeast: a description of the Northeast States for Coor-dinated Air Use Management (NESCAUM) network.” In Visibility and Fine Particles; C.V. Mathai, Editor; Air & Waste Management Associa-tion (A&WMA): Pittsburgh, PA, 1990 pp. 197-206.7.Poirot, R.L.; Flocchini, R.G.; Husar, R.B. “Winter fine particle compositionin the northeast: preliminary results from the NESCAUM network,” Paper 90-84.5, 83rd Annual A&WMA Meeting, Pittsburgh, PA, 1990.8.Poirot, R.L.; Galvin, P.J.; Gordon, N.; Quan, S.; Van Arsdale, A.;Flocchini, R.G. “Annual and seasonal fine particle composition in the northeast: second year results from the NESCAUM network.” Paper 91-49.1, 84th Annual A&WMA Meeting, Vancouver, B.C., 1991. nd, C.E. “Tables of confidence limits for linear functions of thenormal mean and variance.” In Mathematical Statistics, Volume III;Markham Publishing: Chicago, 1975; selected tables.10.Handbook of Tables for Probability and Statistics; Beyer, W.D., Ed; Chemi-cal and Rubber Company Press: Boca Raton, FL, 1985.11.Dalén, J. “Algebraic bounds on standardized sample moments,” Sta-tistics and Probability Letters 1987, 5, 329-331.12.Kendall, M.G.; Stuart, A. The Advanced Theory of Statistics;Griffin:London, Vol 2, 4th ed., 1979.13.Loh, W. “Calibrating confidence coefficients,” J. Am. Statistical Assoc.1987,82, 155-162.14.Shapiro, S.S.; Wilk, M.B.; Chen, H.J. “A comparative study of varioustests for normality,” J. Am. Statistical Assoc. 1968, 39, 1343.Table 3. Confidence bounds for medians of zinc concentrations (ng/m3) (Ringwood, NJ, 1990).Order 95% Bootstrap Relative Length Ratio -Statistics Percentile(Median Width /Median)/(Mean Width/Mean)Day Sample Lower Upper Coverage Lower UpperSize(%)(5%)(95%)Tue 8 6.3 31.9 99.2Sat 49 8.0 14.0 95.6 8.0 14.0 1.03Annual 142 9.24 14.0 95.0 9.2 14.0 1.65。