Review on ranking and selection: A new perspective

doi:10.1007/s42524-021-0152-6

Front. Eng

2021, Vol. 8

Issue (3) : 321-343 https://doi.org/10.1007/s42524-021-0152-6

REVIEW ARTICLE

Review on ranking and selection: A new perspective

L. Jeff HONG¹, Weiwei FAN²(

), Jun LUO³(

)

¹. School of Management and School of Data Science, Fudan University, Shanghai 200433, China
². Advanced Institute of Business, Tongji University, Shanghai 200092, China
³. Antai College of Economics and Management, Shanghai Jiao Tong University, Shanghai 200030, China

Download: PDF(392 KB) HTML
Export: BibTeX | EndNote | Reference Manager | ProCite | RefWorks

Abstract

In this paper, we briefly review the development of ranking and selection (R&S) in the past 70 years, especially the theoretical achievements and practical applications in the past 20 years. Different from the frequentist and Bayesian classifications adopted by Kim and Nelson (2006b) and Chick (2006) in their review articles, we categorize existing R&S procedures into fixed-precision and fixed-budget procedures, as in Hunter and Nelson (2017). We show that these two categories of procedures essentially differ in the underlying methodological formulations, i.e., they are built on hypothesis testing and dynamic programming, respectively. In light of this variation, we review in detail some well-known procedures in the literature and show how they fit into these two formulations. In addition, we discuss the use of R&S procedures in solving various practical problems and propose what we think are the important research questions in the field.

Keywords ranking and selection hypothesis testing dynamic programming simulation

Corresponding Author(s): Weiwei FAN,Jun LUO

Just Accepted Date: 30 January 2021 Online First Date: 29 March 2021 Issue Date: 13 July 2021

Cite this article:

L. Jeff HONG,Weiwei FAN,Jun LUO. Review on ranking and selection: A new perspective[J]. Front. Eng, 2021, 8(3): 321-343.

URL:

https://academic.hep.com.cn/fem/EN/10.1007/s42524-021-0152-6
https://academic.hep.com.cn/fem/EN/Y2021/V8/I3/321

Goal of R&S	Means	HT formulations
PCS	$μ [k] > μ [k − 1]$	$H 0 j : μ j ≤ max i ≠ j ? μ i v .s . H 1 j : μ j > max i ≠ j ? μ i, ∀ j$
PCS-IZ	$μ [k] − δ > μ [k − 1]$	$H 0 j, δ : μ j + δ ≤ max i ≠ j ? μ i v .s . H 1 j, δ : μ j − δ > max i ≠ j ? μ i, ∀ j$
PGS	$μ [k] > μ [k − 1]$	$H 0 j, G : μ j + δ ≤ max i ≠ j ? μ i v .s . H 1 j, G : μ j + δ > max i ≠ j ? μ i, ∀ j$

Tab.1 R&S problems and their HT formulations

Procedure 1 Rinott’s procedure
Require: Number of alternatives $k$ , common first-stage sample size $n 0 ≥ 2$ , PCS $1 − α$ , IZ parameter $δ$ , and a constant $h R$
1: Generate $n 0$ samples for each alternative $i$ , and calculate the sample variance $S i 2 (n 0)$
2: for $i ← 1 : n$ do
3: Let (14) $N i ← max {n 0, ? h R 2 S i 2 (n 0) δ 2 ?},$
4: Generate $N i − n 0$ samples from alternative $i$ , and calculate the sample mean $X ¯ i (N i)$
5: end for
6: Select $arg ? max i = 1,?2, ...,? k ? X ¯ i (N i)$ as the best

Procedure 2 $K N$ procedure
Require: Number of alternatives $k$ , common first-stage sample size $n 0 ≥ 2$ , PCS $1 − α$ , IZ parameter , and a constant $h$
1: Set $η = 12 [(2 α k − 1) − 2 / (n 0 − 1) − 1]$
2: $I ← {1, 2, ..., k}, h 2 = 2 η (n 0 − 1), n ← n 0$
3: Generate $n 0$ samples to each alternative $j$ and calculate $X ¯ i (n 0)$ . For $i, j ∈ I$ , $S j i 2 = 1 n 0 − 1 ∑ l = 1 n 0 [X j l − X i l − (X ¯ j (n 0) − X ¯ i (n 0))] 2$
4: while $\| I \| > 1$ do
5: Set $W j i = max {0, δ 2 n (h 2 S j i 2 δ 2 − n)}$ and $I ← {j : j ∈ I and X ¯ j (n) − X ¯ i (n) ≥ − W j i (n), ∀ i ∈ I, i ≠ j}$
6: Take an additional observation from each alternative $j ∈ I$ , and set $n ← n + 1$
7: end while
8: Select the alternative in $I$ as the best

Fig.1 Triangular region for the

K N

procedures.

Procedure 3 FHN procedure
Require: Number of alternatives $k$ , common first-stage sample size $n 0 ≥ 2$ , and PCS $1 − α$
1: Set $c = − 2 log (2 α k − 1)$
2: $I ← {1, 2, ..., k}, n ← n 0$
3: Generate $n 0$ samples to each alternative $j$ , and calculate $X ¯ j (n 0)$ . For $i, j ∈ I$ , $S j i 2 (n 0) = 1 n 0 − 1 ∑ l = 1 n 0 [X j l − X i l − (X ¯ j (n 0) − X ¯ i (n 0))] 2$
4: while $\| I \| > 1$ do
5: Set $t j i (n) = n / S j i 2 (n)$ and $g j i (t j i (n)) = [c + log (t j i (n) + 1)] (t j i (n) + 1)$ , and let $I ← {j : j ∈ I and t j i (n) [X ¯ j (n) − X ¯ i (n)] ≥ − g j i (t j i (n)), ∀ i ∈ I, i ≠ j}$
6: Take an additional observation from each alternative $j ∈ I$ , and set $n ← n + 1$
7: end while
8: Select the alternative in $I$ as the best

Procedure 4 OCBA procedure
Require: Number of alternatives $k$ , common first-stage sample size $n 0 ≥ 5$ , total sampling budget $N$ , and sampling budget $τ$ per stage
1: Generate $n 0$ samples from each alternative $i$
2: Set $t ← 0, n i, t ← n 0, b t ← ∑ i = 1 k n i, t$
3: while $b t < N$ do
4: Update the sample mean $x ¯ i$ ^{Most of DP procedures are described as Bayesian procedures. Therefore, they are used to represent random variables with upper-case letters and their observations with lower-case letters. To keep in line with the existing literature, we use $x ¯$ in Section 4 to denote the observation of the sample mean.¹⁾ and the sample variance $σ^i 2$ ; $(k) ← arg ? max i ? x ¯ i$ and $d (i) (k) ← x ¯ (k) − x ¯ i$}
5: Set $b t + 1 ← b t + τ$ . Calculate the new budget allocation $n 1, t + 1, n 2, t + 1, ..., n k, t + 1$ satisfying $∑ i n i, t + 1 = b t + 1$ according to $n i, t + 1 n j, t + 1 = (σ i / d (i) (k) σ j / d (j) (k)) 2$ , for $i ≠ j ≠ (k)$ and $n (k), t + 1 = σ^(k) ∑ i ≠ (k) n i, t + 1 2 / σ^i 2$
6: Generate $max {0, n i, t + 1 − n i, t}$ samples from each alternative $i$ . Set $t ← t + 1$
7: end while
8: Select $arg ? max? x ¯ i$ as the best

Procedure 5 EVI procedure for linear loss

Require: Number of alternatives

k

, common first-stage sample size

n 0 ≥ 2

, total sampling budget

N

, and sampling budget

τ

per stage

1: Generate

n 0

samples from each alternative

i

2: Set

t ← 0, n i, t ← n 0, b t ← ∑ i = 1 k n i, t

3: while

b t < N

4: Update

x ¯ i

and

σ^i 2

. Set

x ¯ (1) ≤ x ¯ (2) ≤ ... ≤ x ¯ (k)

and

L = {1,? 2, ...,? k}

5: Let

λ (i)(j) − 1 ← σ^(i) 2 / n (i), t + σ^(j) 2 / n (j), t, d (i)(k) ← x ¯ (k) − x ¯ (i)

. Set

b t + 1 ← b t + τ

6: For each alternative

(i) ∈ L

, calculate

n (i), t + 1 = (τ + ∑ (j) ∈ L n (j), t) (σ^(i) 2 η (i)) 1 / 2 ∑ (j) ∈ L (σ^(j) 2 η (j)) 1 / 2

, where

η (i) =

λ (i)(k) 1 / 2 n i, t − 1+ λ (i)(k) d (i)(k) 2 n i, t − 2 ψ n i, t − 1 [λ (i)(k) 1 / 2 d (i)(k)]

for

(i) ≠ (k)

and

η (k) = ∑ (j) ≠ (k) η (j)

, and

ψ s (?)

denotes the probability density function of a standard student-t distribution with

s

degrees of freedom

7: while

min (i) ∈ L (n (i), t + 1 − n (i), t) < 0

8: if

n (i), t + 1 − n (i), t < 0

then

9: Set

L ← L \ (i)

and

n (i), t + 1 ← n (i), t

10: end if

11: For each alternative

(i) ∈ L

, update

λ (i)(k) − 1 = {σ (i) 2 / n (i), t + σ^(k) 2 / n (k), t, if (k) ∈ L σ (i) 2 / n (i), t, if (k) ∉ L

12: Go back to Step 6

13: end while

14: Generate

n i, t + 1 − n i, t

samples from each alternative

i ∈ L

. Set

t ← t + 1

15: end while

16: Select

arg? max? x ¯ i

as the best

Procedure 6 KG procedure
Require: Number of alternatives $k$ , total sampling budget $N$ , common and known variance $σ 2$ , prior predictive mean $μ i$ , and variance $σ i 2$ for each alternative
1: Set $t ← 0$ . Let $μ i t ← μ i, β i t ← 1 / σ i 2 and β = 1 / σ 2$
2: while $t < N$ do
3: Calculate the variance of the change in predictive mean by taking a sample from alternative $i$ , $σ ˜ i 2 = (β i t) – 1 − (β i t + β) – 1$
4: Calculate (22) $ζ i = − \| μ i t − max j ≠ i ? μ j t σ ˜ i \|,$
5: Choose $z t + 1 = arg? max i = 1,?2, ...,? k ? σ ˜ i (ζ i Φ (ζ i) + φ (ζ i))$ , where $Φ (?)$ and $φ (?)$ denote the cumulative distribution function and probability density function of the standard Gaussian distribution, respectively
6: Take a sample $y z t + 1 t + 1$ from alternative $z t + 1$ . Update $β z t + 1 t + 1 ← β z t + 1 t + β$ , $μ z t + 1 t + 1 ← (β z t + 1 t μ z t + 1 t + β y z t + 1 t + 1) / β z t + 1 t + 1$
7: Set $t ← t + 1$
8: end while
9: Select $arg? max ? μ i t$ as the best

Procedure 7 APS procedure
Require: Number of alternatives $k$ , PCS $1 − α$ , IZ parameter $δ > 0$ , the first-stage sample size $n 0 ≥ 2$ , and the number of processors $m + 1$ (i.e., one master and $m$ workers)
1: Let $a = − log [2 α / (k − 1)]$ and $I ← {1, 2, ..., k}$
2: Let $p$ denote the phantom alternative queued after each round-robin cycle, and let the stage $r$ denote the current sample size of $p$
3: For all $i ∈ I$ , record the triple $(N i r, ∑ ℓ = 1 N i r Y i ℓ, ∑ ℓ = 1 N i r Y i ℓ 2)$ , where $Y i ℓ$ is the $ℓ$ th completed observation from alternative $i$ and $N i r$ is the total sample size obtained from alternative $i$ at stage $r$ . Set $r ← 1$ , and set $N i r ← 0$ , $∑ ℓ = 1 N i r Y i ℓ ← 0$ , $∑ ℓ = 1 N i r Y i ℓ 2 ← 0$ for all $i ∈ I$
4: Use the round-robin order to start simulations on workers $1, 2, ..., m$
5: while $\| I \| > 1$ do
6: Wait for the next observation $Y h ?$
7: if $h ∈ I$ then
8: Update $∑ ℓ = 1 N h r Y h ℓ ← ∑ ℓ = 1 N h r Y h ℓ + Y h ?$ , $∑ ℓ = 1 N h r Y h ℓ 2 ← ∑ ℓ = 1 N h r Y h ℓ 2 + Y h ? 2$ and $N h r ← N h r + 1$
9: else if $h ∉ I$ and $h ≠ p$ then
10: Drop the observation
11: else if $h = p$ then
12: for all $i, j ∈ I$ and $i ≠ j$ do
13: if $N i r ≥ n 0$ and $N j r ≥ n 0$ then
14: Let $τ i j, r = [S i 2 (N i r) N i r + S j 2 (N j r) N j r] − 1$ , where $S i 2 (N i r)$ and $S j 2 (N j r)$ are the sample variance of alternatives $i$ and $j$ , respectively
15: else
16: Let $τ i j, r = 0$
17: end if
18: end for
19: Set $I ← {i : i ∈ I and τ i j, r [Y ¯ i (N i r) − Y ¯ j (N j r)] ≥ min {0, − a δ + δ 2 τ i j, r},$ $∀ j ∈ I, j ≠ i}$ , where $Y ¯ i (N i r)$ and $Y ¯ j (N j r)$ are the sample mean of alternatives $i$ and $j$ , respectively
20: Set $r ← r + 1$
21: end if
22: end while
23: Select the alternative in $I$ as the best

Procedure 8

K T +

procedure

Require: Number of alternatives

k

, PCS

1 − α

, IZ parameter

δ > 0

, the first-stage sample size

n 0 > 2

, the parameter

λ (0 < λ < δ)

, number of alternatives

g ≥ 2

within a “match”, and the number of processors

m + 1

(i.e., one master and workers)

1: Let

I r s

be the set of alternatives in contention at the beginning of round

r

in processor

s

for

s = 1, 2, ..., m

2: Equally allocate

k

alternatives to

m

processors so that each processor handles the selection of approximately

k / m

alternatives, e.g., for

i = 1, 2, ..., k

, let

I 1 (i ?mod? m) + 1 = I 1 (i ?mod? m) + 1 ∪ {i}

3: for all

s = 1, 2, ..., m

4: Set

r = 1

5: while

| I r s | > 1

6: Let

I r + 1 s = ∅

. Group alternatives in

I r s

with the size of

g

. In the case of leftover ones, let them form a group. After grouping, a total of

? | I r s | / g ?

groups are formed. Let

I r, q s

denote the set of the alternatives in group

q

for

q = 1, 2, ..., ? | I r s | / g ?

of processor

s

at round

r

7: Let

α r = α / 2 r

. For each group

q = 1, 2, ..., ? | I r s | / g ?

, set

C ? = I r, q s

and compute

I r + 1 s = I r + 1 s ∪ {K N (C, α r, δ, n 0)}

8: Set

r = r + 1

9: end while

10: Let

I s

denote the index of the alternative in

I r s

11: Take

n 0

observations from alternative

I s

. Calculate its sample variance

S I s 2

based on these

n 0

observations. Set

r = ? log g ? k m ? + 1

α r = α / 2 r

, and

h (α r, m, n 0)

, which is the Rinott’s constant determined by

α r

m

and

n 0

12: Set

N max, I s = max {n 0, ? (h (α r, m, n 0) S I s δ) 2 ?}

. Then, take additional

N max, I s − n 0

observations from alternative

I s

13: Compute the sample mean

X ¯ I s (N max, I s)

14: end for

15: Let

I

denote the set of alternatives containing all the best alternatives produced by

m

processors. Select the alternative with the largest sample mean

X ¯ I s (N max, I s)

for

I s ∈ I

1	S Andradóttir, S H Kim (2010). Fully sequential procedures for comparing constrained systems via simulation. Naval Research Logistics, 57(5): 403–421
2	E A Applegate, G Feldman, S R Hunter, R Pasupathy (2020). Multi-objective ranking and selection: Optimal sampling laws and tractable approximations via SCORE. Journal of Simulation, 14(1): 21–40 https://doi.org/10.1080/17477778.2019.1633891
3	S Banerjee (1961). On confidence interval for two-means problem based on separate estimates of variances and tabulated values of t-table. Sankhya Series A, 23(4): 359–378
4	D Batur, F Choobineh (2012). Stochastic dominance based comparison for system selection. European Journal of Operational Research, 220(3): 661–672 https://doi.org/10.1016/j.ejor.2012.02.018
5	D Batur, L Wang, F F Choobineh (2018). Methods for systems selection based on sequential mean-variance analysis. INFORMS Journal on Computing, 30(4): 724–738 https://doi.org/10.1287/ijoc.2018.0808
6	R E Bechhofer (1954). A single-sample multiple decision procedure for ranking means of normal populations with known variances. Annals of Mathematical Statistics, 25(1): 16–39 https://doi.org/10.1214/aoms/1177728845
7	R E Bechhofer, T J Santner, D M Goldsman (1995). Design and Analysis of Experiments for Statistical Selection, Screening, Multiple Comparisons. Hoboken, NJ: Wiley
8	R Bellman (1966). Dynamic programming. Science, 153(3731): 34–37 https://doi.org/10.1126/science.153.3731.34 pmid: 17730601
9	D P Bertsekas (1995). Dynamic Programming and Optimal Control, vol. 1. 4th ed. Belmont, MA: Athena Scientific
10	J Boesel, B L Nelson, S H Kim (2003). Using ranking and selection to “clean up” after simulation optimization. Operations Research, 51(5): 814–825 https://doi.org/10.1287/opre.51.5.814.16751
11	J Branke, S E Chick, C Schmidt (2007). Selecting a selection procedure. Management Science, 53(12): 1916–1932 https://doi.org/10.1287/mnsc.1070.0721
12	J Branke, W Zhang (2015). A new myopic sequential sampling algorithm for multi-objective problems. In: Proceedings of the Winter Simulation Conference. IEEE, 3589–3598
13	J Branke, W Zhang, Y Tao (2016). Multiobjective ranking and selection based on hypervolume. In: Proceedings of the Winter Simulation Conference. IEEE, 859–870
14	S Bubeck, N Cesa-Bianchi (2012). Regret analysis of stochastic and nonstochastic multi-armed bandit problems. arXiv Preprint. arXiv:1204.5721
15	C H Chen (1996). A lower bound for the correct subset-selection probability and its application to discrete-event system simulations. IEEE Transactions on Automatic Control, 41(8): 1227–1231 https://doi.org/10.1109/9.533692
16	C H Chen, J Lin, E Yücesan, S E Chick (2000). Simulation budget allocation for further enhancing the efficiency of ordinal optimization. Discrete Event Dynamic Systems, 10(3): 251–270 https://doi.org/10.1023/A:1008349927281
17	E J Chen (2005). Using parallel and distributed computing to increase the capability of selection procedures. In: Proceedings of the Winter Simulation Conference. IEEE, 723–731
18	S E Chick (2006). Subjective probability and Bayesian methodology. In: Henderson S G, Nelson B L, eds. Handbooks in Operations Research and Management Science: Simulation, Chapter 9. Amsterdam, Chapter 9: Elsevier, 13: 225–257
19	S E Chick, J Branke, C Schmidt (2010). Sequential sampling to myopically maximize the expected value of information. INFORMS Journal on Computing, 22(1): 71–80 https://doi.org/10.1287/ijoc.1090.0327
20	S E Chick, K Inoue (2001). New two-stage and sequential procedures for selecting the best simulated system. Operations Research, 49(5): 732–743 https://doi.org/10.1287/opre.49.5.732.10615
21	G M Clark, W Yang (1986). A bonferroni selection procedure when using common random numbers with unknown variances. In: Proceedings of the Winter Simulation Conference. IEEE, 313–315
22	J Dean, S Ghemawat (2008). MapReduce: Simplified data processing on large clusters. Communications of the ACM, 51(1): 107–113 https://doi.org/10.1145/1327452.1327492
23	D J Eckman, S G Henderson (2018a). Guarantees on the probability of good selection. In: Proceedings of the Winter Simulation Conference. IEEE, 351–365
24	D J Eckman, S G Henderson (2018b). Reusing search data in ranking and selection: What could possibly go wrong? ACM Transactions on Modeling and Computer Simulation, 28(3): 1–15
25	D J Eckman, S G Henderson (2020). Fixed-confidence, fixed-tolerance guarantees for selection-of-the-best procedures. ACM Transactions on Modeling and Computer Simulation, 31(2): 7
26	E Even-Dar, S Mannor, Y Mansour (2002). PAC bounds for multi-armed bandit and Markov decision processes. In: COLT '02: Proceedings of the 15th Annual Conference on Computational Learning Theory. Springer-Verlag, 255–270
27	E Even-Dar, S Mannor, Y Mansour (2006). Action elimination and stopping conditions for the multi-armed bandit and reinforcement learning problems. Journal of Machine Learning Research, 7(39): 1079–1105
28	V Fabian (1974). Note on Anderson’s sequential procedures with triangular boundary. Annals of Statistics, 2(1): 170–176 https://doi.org/10.1214/aos/1176342622
29	W W Fan, L J Hong, B L Nelson (2016). Indifference-zone-free selection of the best. Operations Research, 64(6): 1499–1514 https://doi.org/10.1287/opre.2016.1530
30	W W Fan, L J Hong, X W Zhang (2013). Robust selection of the best. In: Proceedings of the Winter Simulation Conference. IEEE, 868–876
31	W W Fan, L J Hong, X W Zhang (2020). Distributionally robust selection of the best. Management Science, 66(1): 190–208 https://doi.org/10.1287/mnsc.2018.3213
32	G Feldman, S R Hunter (2018). SCORE allocations for bi-objective ranking and selection. ACM Transactions on Modeling and Computer Simulation, 28(1): 1–28 https://doi.org/10.1145/3158666
33	P I Frazier (2014). A fully sequential elimination procedure for indifference-zone ranking and selection with tight bounds on probability of correct selection. Operations Research, 62(4): 926–942 https://doi.org/10.1287/opre.2014.1282
34	P I Frazier, W B Powell, S Dayanik (2008). A knowledge-gradient policy for sequential information collection. SIAM Journal on Control and Optimization, 47(5): 2410–2439 https://doi.org/10.1137/070693424
35	P I Frazier, W B Powell, S Dayanik (2009). The knowledge-gradient policy for correlated normal beliefs. INFORMS Journal on Computing, 21(4): 599–613 https://doi.org/10.1287/ijoc.1080.0314
36	M C Fu, S G Henderson (2017). History of seeking better solutions, aka simulation optimization. In: Proceedings of the Winter Simulation Conference. IEEE, 131–157
37	M C Fu, J Q Hu, C H Chen, X Xiong (2007). Simulation allocation for determining the best design in the presence of correlated sampling. INFORMS Journal on Computing, 19(1): 101–111 https://doi.org/10.1287/ijoc.1050.0141
38	V Gabillon, M Ghavamzadeh, A Lazaric (2012). Best arm identification: A unified approach to fixed budget and fixed confidence. In: Pereira F, Burges C J C, Bottou L, Weinberger K Q, eds. Advances in Neural Information Processing Systems. Minneapolis, MN: Curran Associates, Inc., 25: 3212–3220
39	F Gao, S Gao, H Xiao, Z Shi (2019a). Advancing constrained ranking and selection with regression in partitioned domains. IEEE Transactions on Automation Science and Engineering, 16(1): 382–391 https://doi.org/10.1109/TASE.2018.2811809
40	S Gao, W Chen, L Shi (2017a). A new budget allocation framework for the expected opportunity cost. Operations Research, 65(3): 787–803 https://doi.org/10.1287/opre.2016.1581
41	S Gao, J Du, C H Chen (2019b). Selecting the optimal system design under covariates. In: 15th International Conference on Automation Science and Engineering (CASE). IEEE, 547–552
42	S Gao, H Xiao, E Zhou, W Chen (2017b). Robust ranking and selection with optimal computing budget allocation. Automatica, 81: 30–36 https://doi.org/10.1016/j.automatica.2017.03.019
43	P Glynn, S Juneja (2004). A large deviations perspective on ordinal optimization. In: Proceedings of the Winter Simulation Conference. IEEE, 577–585
44	P Glynn, S Juneja (2018). Selecting the best system and multi-armed bandits. arXiv Preprint. arXiv:1507.04564
45	D Goldsman, B L Nelson, J Banks (1998). Comparing systems via simulation. In: Banks J, ed. Handbook of Simulation: Principles, Methodology, Advances, Applications, Practice. Hoboken, NJ: Wiley, 273–306
46	W Gropp, E Lusk, A Skjellum (1999). Using MPI: Portable Parallel Programming with the Message-Passing Interface, vol. 1. Cambridge, MA: MIT press
47	S Gupta (1956). On a Decision Rule for a Problem in Ranking Means. Dissertation for the Doctorol Degree. Chapel Hill, NC: University of North Carolina
48	D He, S E Chick, C H Chen (2007). Opportunity cost and OCBA selection procedures in ordinal optimization for a fixed number of alternative systems. IEEE Transactions on Systems, Man and Cybernetics: Part C, Applications and Reviews, 37(5): 951–961 https://doi.org/10.1109/TSMCC.2007.900656
49	D He, L H Lee, C H Chen, M C Fu, S Wasserkrug (2010). Simulation optimization using the cross-entropy method with optimal computing budget allocation. ACM Transactions on Modeling and Computer Simulation, 20(1): 1–22 https://doi.org/10.1145/1667072.1667076
50	C M Healey, S Andradóttir, S H Kim (2014). Selection procedures for simulations with multiple constraints under independent and correlated sampling. ACM Transactions on Modeling and Computer Simulation, 24(3): 14 https://doi.org/10.1145/2567921
51	C M Healey, S Andradóttir, S H Kim (2015). A minimal switching procedure for constrained ranking and selection under independent or common random numbers. IIE Transactions, 47(11): 1170–1184 https://doi.org/10.1080/0740817X.2015.1009198
52	L J Hong (2006). Fully sequential indifference-zone selection procedures with variance-dependent sampling. Naval Research Logistics, 53(5): 464–476 https://doi.org/10.1002/nav.20155
53	L J Hong, J Luo, B L Nelson (2015a). Chance constrained selection of the best. INFORMS Journal on Computing, 27(2): 317–334 https://doi.org/10.1287/ijoc.2014.0628
54	L J Hong, B L Nelson (2005). The tradeoff between sampling and switching: New sequential procedures for indifference-zone selection. IIE Transactions, 37(7): 623–634 https://doi.org/10.1080/07408170590948486
55	L J Hong, B L Nelson (2007a). A framework for locally convergent random-search algorithms for discrete optimization via simulation. ACM Transactions on Modeling and Computer Simulation, 17(4): 19 https://doi.org/10.1145/1276927.1276932
56	L J Hong, B L Nelson (2007b). Selecting the best system when systems are revealed sequentially. IIE Transactions, 39(7): 723–734 https://doi.org/10.1080/07408170600838415
57	L J Hong, B L Nelson (2009). A brief introduction to optimization via simulation. In: Proceedings of the Winter Simulation Conference. IEEE, 75–85
58	L J Hong, B L Nelson, J Xu (2015b). Discrete optimization via simulation. In: Fu M C, ed. Handbook of Simulation Optimization. Berlin: Springer
59	R Hu, M Ludkovski (2017). Sequential design for ranking response surfaces. SIAM/ASA Journal on Uncertainty Quantification, 5(1): 212–239
60	S R Hunter, E A Applegate, V Arora, B Chong, K Cooper, O Rincón-Guevara, C Vivas-Valencia (2019). An introduction to multiobjective simulation optimization. ACM Transactions on Modeling and Computer Simulation, 29(1): 1–36 https://doi.org/10.1145/3299872
61	S R Hunter, B L Nelson (2017). Parallel ranking and selection. In: Tolk A, Fowler J, Shao G, Yücesan E, eds. Advances in Modeling and Simulation. Berlin: Springer, 249–275
62	S R Hunter, R Pasupathy (2013). Optimal sampling laws for stochastically constrained simulation optimization on finite sets. INFORMS Journal on Computing, 25(3): 527–542 https://doi.org/10.1287/ijoc.1120.0519
63	K Jamieson, M Malloy, R Nowak, S Bubeck (2014). lil’UCB: An optimal exploration algorithm for multi-armed bandits. In: Conference on Learning Theory, 423–439
64	B Kamiński, P Szufel (2018). On parallel policies for ranking and selection problems. Journal of Applied Statistics, 45(9): 1690–1713 https://doi.org/10.1080/02664763.2017.1390555
65	S C Kao, T L Lai (1980). Sequential selection procedures based on confidence sequences for normal populations. Communications in Statistics: Theory and Methods, 9(16): 1657–1676 https://doi.org/10.1080/03610928008827990
66	E Kaufmann, S Kalyanakrishnan (2013). Information complexity in bandit subset selection. Journal of Machine Learning Research, 30: 228–251
67	S H Kim, B L Nelson (2001). A fully sequential procedure for indifference-zone selection in simulation. ACM Transactions on Modeling and Computer Simulation, 11(3): 251–273 https://doi.org/10.1145/502109.502111
68	S H Kim, B L Nelson (2006a). On the asymptotic validity of fully sequential selection procedures for steady-state simulation. Operations Research, 54(3): 475–488 https://doi.org/10.1287/opre.1060.0281
69	S H Kim, B L Nelson (2006b). Selecting the best system. In: Henderson S G, Nelson B L, eds. Handbooks in Operations Research and Management Science: Simulation. Amsterdam: Elsevier, 501–534
70	L H Lee, E P Chew, S Teng, D Goldsman (2010). Finding the non-dominated Pareto set for multi-objective simulation models. IIE Transactions, 42(9): 656–674 https://doi.org/10.1080/07408171003705367
71	L H Lee, N A Pujowidianto, L W Li, C H Chen, C M Yap (2012). Approximate simulation budget allocation for selecting the best design in the presence of stochastic constraints. IEEE Transactions on Automatic Control, 57(11): 2940–2945 https://doi.org/10.1109/TAC.2012.2195931
72	X Li, X Zhang, Z Zheng (2018). Data-driven ranking and selection: High-dimensional covariates and general dependence. In: Proceedings of the Winter Simulation Conference. IEEE, 1933–1944
73	J Luo, L J Hong (2011). Large-scale ranking and selection using cloud computing. In: Proceedings of the Winter Simulation Conference. IEEE, 4051–4061
74	J Luo, L J Hong, B L Nelson, Y Wu (2015). Fully sequential procedures for large-scale ranking-and-selection problems in parallel computing environments. Operations Research, 63(5): 1177–1194 https://doi.org/10.1287/opre.2015.1413
75	S Ma (2018). Sequential Ranking and Selection Procedures and Sample Complexity. Dissertation for the Doctorol Degree. New York: Cornell University
76	S Ma, S G Henderson (2017). An efficient fully sequential selection procedure guaranteeing probably approximately correct selection. In: Proceedings of the Winter Simulation Conference. IEEE, 2225–2236
77	B L Nelson, F J Matejcik (1995). Using common random numbers for indifference-zone selection and multiple comparisons in simulation. Management Science, 41(12): 1935–1945 https://doi.org/10.1287/mnsc.41.12.1935
78	B L Nelson, J Swann, D Goldsman, W Song (2001). Simple procedures for selecting the best simulated system when the number of alternatives is large. Operations Research, 49(6): 950–963 https://doi.org/10.1287/opre.49.6.950.10019
79	E C Ni, D F Ciocan, S G Henderson, S R Hunter (2015). Comparing message passing interface and MapReduce for large-scale parallel ranking and selection. In: Proceedings of the Winter Simulation Conference. IEEE, 3858–3867
80	E C Ni, D F Ciocan, S G Henderson, S R Hunter (2017). Efficient ranking and selection in parallel computing environments. Operations Research, 65(3): 821–836 https://doi.org/10.1287/opre.2016.1577
81	E C Ni, S R Hunter, S G Henderson (2013). Ranking and selection in a high performance computing environment. In: Proceedings of the Winter Simulation Conference. IEEE, 833–845
82	E C Ni, S R Hunter, S G Henderson (2014). A comparison of two parallel ranking and selection procedures. In: Proceedings of the Winter Simulation Conference. IEEE, 3761–3772
83	R Pasupathy, S R Hunter, N A Pujowidianto, L H Lee, C H Chen (2015). Stochastically constrained ranking and selection via SCORE. ACM Transactions on Modeling and Computer Simulation, 25(1): 1–26 https://doi.org/10.1145/2630066
84	E Paulson (1949). A multiple decision procedure for certain problems in the analysis of variance. Annals of Mathematical Statistics, 20(1): 95–98 https://doi.org/10.1214/aoms/1177730094
85	E Paulson (1964). A sequential procedure for selecting the population with the largest mean from k. Annals of Mathematical Statistics, 35(1): 174–180 https://doi.org/10.1214/aoms/1177703739
86	M Pearce, J Branke (2017). Efficient expected improvement estimation for continuous multiple ranking and selection. In: Proceedings of the Winter Simulation Conference. IEEE, 2161–2172
87	L Pei, B L Nelson, S Hunter (2018). A new framework for parallel ranking & selection using an adaptive standard. In: Proceedings of the Winter Simulation Conference. IEEE, 2201–2212
88	Y Peng, E K Chong, C H Chen, M C Fu (2018). Ranking and selection as stochastic control. IEEE Transactions on Automatic Control, 63(8): 2359–2373 https://doi.org/10.1109/TAC.2018.2797188
89	Y Peng, M C Fu (2017). Myopic allocation policy with asymptotically optimal sampling rate. IEEE Transactions on Automatic Control, 62(4): 2041–2047 https://doi.org/10.1109/TAC.2016.2592378
90	J Pichitlamken, B L Nelson, L J Hong (2006). A sequential procedure for neighborhood selection-of-the-best in optimization via simulation. European Journal of Operational Research, 173(1): 283–298 https://doi.org/10.1016/j.ejor.2004.12.010
91	Y Rinott (1978). On two-stage selection procedures and related probability-inequalities. Communications in Statistics: Theory and Methods, 7(8): 799–811 https://doi.org/10.1080/03610927808827671
92	I O Ryzhov (2016). On the convergence rates of expected improvement methods. Operations Research, 64(6): 1515–1528 https://doi.org/10.1287/opre.2016.1494
93	H Shen, L J Hong, X Zhang (2017). Ranking and selection with covariates. In: Proceedings of the Winter Simulation Conference. IEEE, 169
94	H Shen, L J Hong, X Zhang (2019). Ranking and selection with covariates for personalized decision making. arXiv Preprint. arXiv:1710.02642
95	E Song, B L Nelson, L J Hong (2015). Input uncertainty and indifference-zone ranking & selection. In: Proceedings of the Winter Simulation Conference. IEEE, 414–424
96	S C Tsai, J Luo, C C Sung (2017). Combined variance reduction techniques in fully sequential selection procedures. Naval Research Logistics, 64(6): 502–527 https://doi.org/10.1002/nav.21770
97	S C Tsai, B L Nelson (2009). Fully sequential selection procedures with control variates. IIE Transactions, 42(1): 71–82 https://doi.org/10.1080/07408170903228942
98	A Wald (1945). Sequential tests of statistical hypotheses. Annals of Mathematical Statistics, 16(2): 117–186 https://doi.org/10.1214/aoms/1177731118
99	A Wald (1947). Sequential Analysis. New York: John Wiley & Sons
100	W Wang, H Wan (2017). Sequential probability ratio test for multiple-objective ranking and selection. In: Proceedings of the Winter Simulation Conference. IEEE, 1998–2009
101	R R Wilcox (1984). A table for Rinott’s selection procedure. Journal of Quality Technology, 16(2): 97–100 https://doi.org/10.1080/00224065.1984.11978896
102	D Wu, E Zhou (2019). Fixed confidence ranking and selection under input uncertainty. In: Proceedings of the Winter Simulation Conference. IEEE, 3717–3727
103	J Xie, P Frazier, S E Chick (2016). Bayesian optimization via simulation with pairwise sampling and correlated prior beliefs. Operations Research, 64(2): 542–559 https://doi.org/10.1287/opre.2016.1480
104	E Yücesan, Y C Luo, C H Chen, I Lee (2001). Distributed web-based simulation experiments for optimization. Simulation Practice and Theory, 9(1–2): 73–90 https://doi.org/10.1016/S0928-4869(01)00037-4
105	M Zaharia, M Chowdhury, M J Franklin, S Shenker, I Stoica (2010). Spark: Cluster computing with working sets. In: HotCloud'10: Proceedings of the 2nd USENIX Conference on Hot Topics in Cloud Computing. Berkeley, CA, 10
106	S Zhang, J Xu, L H Lee, E P Chew, W P Wong, C H Chen (2017). Optimal computing budget allocation for particle swarm optimization in stochastic optimization. IEEE Transactions on Evolutionary Computation, 21(2): 206–219 https://doi.org/10.1109/TEVC.2016.2592185 pmid: 29170617
107	X Zhang, H Shen, L J Hong, L Ding (2020). Knowledge gradient for selection with covariates: Consistency and computation. arXiv Preprint. arXiv:1906.05098
108	Y Zhong, L J Hong (2020). Knockout-tournament procedures for large-scale ranking and selection in parallel computing environments. Operations Research, to appear
109	Y Zhong, S Liu, J Luo, L J Hong (2020). Speeding up Paulson’s procedure for large-scale problems using parallel computing. INFORMS Journal on Computing, in press, doi: 10.1287/ijoc.2020. 1054

[1]	Shu CHEN, Menghan SHANG, Jianping WANG. Evacuation strategies for vertical ship lift during initial fire: Integrated application of stairs and elevators[J]. Front. Eng, 2021, 8(1): 135-147.
[2]	Ming LU, Nicolas DIAZ, Monjurul HASAN. Proposing a “lean and green” framework for equipment cost analysis in construction[J]. Front. Eng, 2019, 6(3): 384-394.
[3]	Jide SUN, Mian ZHENG, Martin SKITMORE, Bo XIA, Xincheng WANG. Industry effect of job hopping: an agent-based simulation of Chinese construction workers[J]. Front. Eng, 2019, 6(2): 249-261.
[4]	Bin YU, Sijia REN, Enze WU, Yifan ZHOU, Yunpeng WANG. Optimization of urban bus operation frequency under common route condition with rail transit[J]. Front. Eng, 2017, 4(4): 451-462.
[5]	SangHyun LEE. Applying system dynamics to strategic decision making in construction[J]. Front. Eng, 2017, 4(1): 35-40.
[6]	Omar M. Basha,Li Weng,Zhuo-wu Men,Wayne Xu,Badie I. Morsi. NICE’s Indirect Coal-to-Liquid Process for Producing Clean Transportation Fuels Using Fischer-Tropsch Synthesis[J]. Front. Eng, 2016, 3(4): 362-376.
[7]	Han-peng Zhang,Yi Liao,Hui-xia Luo. Two-echelon Emergency Response Problem and Simulation Considering Secondary Disasters[J]. Front. Eng, 2014, 1(3): 318-321.

Viewed

Full text

Abstract

Cited

Shared

Discussed