This Article Full Text (PDF) References Alert me when this article is cited Alert me if a correction is posted Services Email this article to a friend Similar articles in this journal Alert me to new issues of the journal Download to citation manager Google Scholar Articles by Jose, V. R. R. Articles by Winkler, R. L.

Evaluating Quantile Assessments

McDonough School of Business, Georgetown University, Washington, DC 20057
Fuqua School of Business, Duke University, Durham, North Carolina 27708
vrjose{at}gmail.com
rwinkler{at}duke.edu

Quantile assessments are commonly encountered in the elicitation of probability distributions in decision analysis, forecasting, and risk analysis. Scoring rules have been developed to provide ex ante incentives for careful and truthful assessments and ex post evaluation measures in the context of probability assessment. We show that these scoring rules designed for probability assessment provide inappropriate incentives if used for quantile assessment. We investigate the properties of a linear family of scoring rules that are intended specifically for quantile assessment (including the assessment of multiple quantiles) and can be related to a realistic decision-making problem. These rules provide proper incentives for quantile assessment and yield higher expected scores for distributions that are more informative in the sense of having less dispersion. We discuss the special case of interval forecasts and a generalization involving transformations, and we briefly mention other possible extensions.

Subject classifications: probability; assessment; evaluation; decision analysis; expert information; forecasting; probability forecasts.
History: Received May 2008; revision received July 2008; accepted August 2008.