This Article Full Text (PDF) e-companion Alert me when this article is cited Alert me if a correction is posted Services Email this article to a friend Similar articles in this journal Alert me to new issues of the journal Download to citation manager Google Scholar Articles by Delage, E. Articles by Mannor, S.

Percentile Optimization for Markov Decision Processes with Parameter Uncertainty

Department of Management Science, HEC Montréal, Montréal, Quebec H3T 2A7, Canada
Department of Electrical and Computer Engineering, McGill University, Montreal, Quebec H3A 2A7, Canada
erick.delage{at}hec.ca
shie.mannor{at}mcgill.ca

Markov decision processes are an effective tool in modeling decision making in uncertain dynamic environments. Because the parameters of these models typically are estimated from data or learned from experience, it is not surprising that the actual performance of a chosen strategy often differs significantly from the designer's initial expectations due to unavoidable modeling ambiguity. In this paper, we present a set of percentile criteria that are conceptually natural and representative of the trade-off between optimistic and pessimistic views of the question. We study the use of these criteria under different forms of uncertainty for both the rewards and the transitions. Some forms are shown to be efficiently solvable and others highly intractable. In each case, we outline solution concepts that take parametric uncertainty into account in the process of decision making.

Subject classifications: Markov decision processes; parameter uncertainty; finite state; stochastic model applications; stochastic programming; value at risk; chance-constrained optimization.
History: Received November 2006; revision received January 2008; accepted April 2008.