|
|
|
Optimal stopping time on discounted semi-Markov processes |
Fang CHEN1, Xianping GUO1, Zhong-Wei LIAO2( ) |
1. School of Mathematics, Sun Yat-Sen University, Guangzhou 510275, China
2. College of Education for the Future, Beijing Normal University, Beijing 100875, China |
|
|
|
|
Abstract This paper attempts to study the optimal stopping time for semi- Markov processes (SMPs) under the discount optimization criteria with unbounded cost rates. In our work, we introduce an explicit construction of the equivalent semi-Markov decision processes (SMDPs). The equivalence is embodied in the expected discounted cost functions of SMPs and SMDPs, that is, every stopping time of SMPs can induce a policy of SMDPs such that the value functions are equal, and vice versa. The existence of the optimal stopping time of SMPs is proved by this equivalence relation. Next, we give the optimality equation of the value function and develop an effective iterative algorithm for computing it. Moreover, we show that the optimal and ε-optimal stopping time can be characterized by the hitting time of the special sets. Finally, to illustrate the validity of our results, an example of a maintenance system is presented in the end.
|
| Keywords
Optimal stopping time
semi-Markov processes (SMPs)
value function
semi-Markov decision processes (SMDPs)
optimal policy
iterative lgorithm
|
|
Corresponding Author(s):
Zhong-Wei LIAO
|
|
Issue Date: 01 June 2021
|
|
| 1 |
N Bäuerle, A Popp. Risk-sensitive stopping problems for continuous-time Markov chains. Stochastics, 2018, 90: 411–431
https://doi.org/10.1080/17442508.2017.1357724
|
| 2 |
N Bäuerle, U Rieder. Markov Decision Processes with Applications to Finance. Heidelberg: Springer, 2011
https://doi.org/10.1007/978-3-642-18324-9
|
| 3 |
F A Boshuizen, J M Gouweleeuw. General optimal stopping theorems for semi-Markov processes. Adv in Appl Probab, 1993, 25: 825–846
https://doi.org/10.1017/S0001867800025775
|
| 4 |
B Cekyay, S Ozekici. Mean time to failure and availability of semi-Markov missions with maximal repair. European J Oper Res, 2010, 207: 1442–1454
https://doi.org/10.1016/j.ejor.2010.07.019
|
| 5 |
Y S Chow, H Robbins, D Siegmund. Great Expectations: The Theory of Optimal Stopping. Boston: Houghton Mifflin Company, 1991
|
| 6 |
B Dochviri. On optimal stopping of inhomogeneous standard Markov processes. Georgian Math J, 1995, 2: 335–346
https://doi.org/10.1007/BF02255984
|
| 7 |
O Hernández-Lerma, J B Lasserre. Discrete-time Markov Control Processes: Basic Optimality Criteria. New York: Springer-Verlag, 1996
https://doi.org/10.1007/978-1-4612-0729-0
|
| 8 |
Y H Huang, X P Guo. Discounted semi-Markov decision processes with nonnegative costs. Acta Math Sci Ser A Chin Ed, 2010, 53: 503–514 (in Chinese)
|
| 9 |
Y H Huang, X P Guo. Finite horizon semi-Markov decision processes with application to maintenance systems. European J Oper Res, 2011, 212: 131–140
https://doi.org/10.1016/j.ejor.2011.01.027
|
| 10 |
A Jaśkiewicz, A S Nowak. Optimality in Feller semi-Markov control processes. Oper Res Lett, 2006, 34: 713–718
https://doi.org/10.1016/j.orl.2005.11.005
|
| 11 |
M Y Kitaev. Semi-Markov and jump Markov controlled models: average cost criterion. Theory Probab Appl, 1986, 30: 272–288
https://doi.org/10.1137/1130036
|
| 12 |
M Y Kitaev, V Rykov. Controlled Queueing Systems. Boca Raton: CRC Press, 1995
|
| 13 |
N Limnios, G Oprisan. Semi-Markov Processes and Reliability. Boston: Birkhäuser, 2001
https://doi.org/10.1007/978-1-4612-0161-8
|
| 14 |
M L Nikolaev. On optimal multiple stopping of Markov sequences. Theory Probab Appl, 1999, 43: 298–306
https://doi.org/10.1137/S0040585X9797691X
|
| 15 |
M L Puterman. Markov Decision Processes: Discrete Stochastic Dynamic Programming. New York: John Wiley & Sons, Inc, 1994
https://doi.org/10.1002/9780470316887
|
| 16 |
G Peskir, A Shiryaev. Optimal Stopping and Free-Boundary Problems. Boston: Birkhäuser, 2006
|
| 17 |
S M Ross. Average cost semi-Markov decision processes. J Appl Probab, 1970, 7: 649–656
https://doi.org/10.1017/S0021900200110617
|
| 18 |
J L Snell. Applications of martingale system theorems. Trans Amer Math Soc, 1952, 73: 293–312
https://doi.org/10.1090/S0002-9947-1952-0050209-9
|
| 19 |
L Ye. Value function and optimal rule on the optimal stopping problem for continuoustime Markov processes. Chin J Math, 2017, 1: 1–10
https://doi.org/10.1155/2017/3596037
|
| 20 |
M V Zhitlukhin, A N Shiryaev. On the existence of solutions of unbounded optimal stopping problems. Proc Steklov Inst Math, 2014, 287: 299–307
https://doi.org/10.1134/S0081543814080185
|
|
Viewed |
|
|
|
Full text
|
|
|
|
|
Abstract
|
|
|
|
|
Cited |
|
|
|
|
| |
Shared |
|
|
|
|
| |
Discussed |
|
|
|
|
