Open and real-world human-AI coordination by heterogeneous training with communication
Cong GUAN1,2, Ke XUE1,2, Chunpeng FAN3, Feng CHEN1,2, Lichao ZHANG3, Lei YUAN1,2,3, Chao QIAN1,3, Yang YU1,2,3()
1. National Key Laboratory for Novel Software Technology, Nanjing University, Nanjing 210023, China 2. School of Artificial Intelligence, Nanjing University, Nanjing 210023, China 3. Polixir Technologies, Nanjing 211106, China
Human-AI coordination aims to develop AI agents capable of effectively coordinating with human partners, making it a crucial aspect of cooperative multi-agent reinforcement learning (MARL). Achieving satisfying performance of AI agents poses a long-standing challenge. Recently, ah-hoc teamwork and zero-shot coordination have shown promising advancements in open-world settings, requiring agents to coordinate efficiently with a range of unseen human partners. However, these methods usually assume an overly idealistic scenario by assuming homogeneity between the agent and the partner, which deviates from real-world conditions. To facilitate the practical deployment and application of human-AI coordination in open and real-world environments, we propose the first benchmark for open and real-world human-AI coordination (ORC) called ORCBench. ORCBench includes widely used human-AI coordination environments. Notably, within the context of real-world scenarios, ORCBench considers heterogeneity between AI agents and partners, encompassing variations in capabilities and observations, which aligns more closely with real-world applications. Furthermore, we introduce a framework known as Heterogeneous training with Communication (HeteC) for ORC. HeteC builds upon a heterogeneous training framework and enhances partner population diversity by using mixed partner training and frozen historical partners. Additionally, HeteC incorporates a communication module that enables human partners to communicate with AI agents, mitigating the adverse effects of partially observable environments. Through a series of experiments, we demonstrate the effectiveness of HeteC in improving coordination performance. Our contribution serves as an initial but important step towards addressing the challenges of ORC.
. [J]. Frontiers of Computer Science, 2025, 19(4): 194314.
Cong GUAN, Ke XUE, Chunpeng FAN, Feng CHEN, Lichao ZHANG, Lei YUAN, Chao QIAN, Yang YU. Open and real-world human-AI coordination by heterogeneous training with communication. Front. Comput. Sci., 2025, 19(4): 194314.
G, Klein D D, Woods J M, Bradshaw R R, Hoffman P J Feltovich . Ten challenges for making automation a “team player” in joint human-agent activity. IEEE Intelligent Systems, 2004, 19( 6): 91–95
2
A, Dafoe Y, Bachrach G, Hadfield E, Horvitz K, Larson T Graepel . Cooperative AI: machines must learn to find common ground. Nature, 2021, 593( 7857): 33–36
3
P, Hernandez-Leal B, Kartal M E Taylor . A survey and critique of multiagent deep reinforcement learning. Autonomous Agents and Multi-Agent Systems, 2019, 33( 6): 750–797
4
W, Du S F Ding . A survey on multi-agent deep reinforcement learning: From the perspective of challenges and applications. Artificial Intelligence Review, 2021, 54( 5): 3215–3238
5
A, Oroojlooy D Hajinezhad . A review of cooperative multi-agent deep reinforcement learning. Applied Intelligence, 2023, 53( 11): 13677–13722
6
R, Lowe Y, Wu A, Tamar J, Harb P, Abbeel I Mordatch . Multi-agent actor-critic for mixed cooperative-competitive environments. In: Proceedings of the 31st International Conference on Neural Information Processing Systems. 2017, 6382−6393
7
P, Sunehag G, Lever A, Gruslys W M, Czarnecki V, Zambaldi M, Jaderberg M, Lanctot N, Sonnerat J Z, Leibo K, Tuyls T Graepel . Value-decomposition networks for cooperative multi-agent learning based on team reward. In: Proceedings of the 17th International Conference on Autonomous Agents and MultiAgent Systems. 2018, 2085−2087
8
T, Rashid M, Samvelyan C, Schroeder G, Farquhar J, Foerster S Whiteson . QMIX: monotonic value function factorisation for deep multi-agent reinforcement learning. In: Proceedings of the 35th International Conference on Machine Learning. 2018, 4295−4304
9
C, Yu A, Velu E, Vinitsky J, Gao Y, Wang A M, Bayen Y Wu . The surprising effectiveness of PPO in cooperative multi-agent games. In: Proceedings of the 36th Conference on Neural Information Processing Systems Datasets and Benchmarks Track. 2022, 24611−24624
10
R, Gorsane O, Mahjoub Kock R J, De R, Dubb S, Singh A Pretorius . Towards a standardised performance evaluation protocol for cooperative marl. In: Proceedings of the 36th Conference on Neural Information Processing Systems, 2022, 5510−5521
11
H, Hu A, Lerer A, Peysakhovich J Foerster . “Other-play” for zero-shot coordination. In: Proceedings of the 37th International Conference on Machine Learning. 2020, 409
12
M, Carroll R, Shah M K, Ho T, Griffiths S A, Seshia P, Abbeel A Dragan . On the utility of learning about humans for human-AI coordination. In: Proceedings of the 33rd International Conference on Neural Information Processing Systems. 2019, 465
13
L, Yuan L, Li Z, Zhang F, Chen T, Zhang C, Guan Y, Yu Z H Zhou . Learning to coordinate with anyone. In: Proceedings of the 5th International Conference on Distributed Artificial Intelligence, 2023, 4
14
Z H Zhou . Open-environment machine learning. National Science Review, 2022, 9( 8): nwac123
15
X, Liu J, Liang D Y, Liu R, Chen S M Yuan . Weapon-target assignment in unreliable peer-to-peer architecture based on adapted artificial bee colony algorithm. Frontiers of Computer Science, 2022, 16( 1): 161103
L, Yuan Z, Zhang L, Li C, Guan Y Yu . A survey of progress on cooperative multi-agent reinforcement learning in open environment. 2023, arXiv preprint arXiv: 2312.01058
18
P, Stone G A, Kaminka S, Kraus J S Rosenschein . Ad hoc autonomous agent teams: Collaboration without pre-coordination. In: Proceedings of the 24th AAAI Conference on Artificial Intelligence. 2010, 1504−1509
19
R, Mirsky I, Carlucho A, Rahman E, Fosong W, Macke M, Sridharan P, Stone S V Albrecht . A survey of ad Hoc teamwork research. In: Proceedings of the 19th European Conference on Multi-Agent Systems. 2022, 275−293
20
A, Lupu B, Cui H, Hu J Foerster . Trajectory diversity for zero-shot coordination. In: Proceedings of the 38th International Conference on Machine Learning. 2021, 7204−7213
21
D J, Strouse K R, McKee M, Botvinick E, Hughes R Everett . Collaborating with humans without human data. In: Proceedings of the 35th Conference on Neural Information Processing Systems. 2021, 14502−14515
22
R, Zhao J, Song Y, Yuan H, Hu Y, Gao Y, Wu Z, Sun W Yang . Maximum entropy population-based training for zero-shot human-AI coordination. In: Proceedings of the 37th AAAI Conference on Artificial Intelligence. 2023, 689
23
C, Yu J, Gao W, Liu B, Xu H, Tang J, Yang Y, Wang Y Wu . Learning zero-shot cooperation with humans, assuming humans are biased. In: Proceedings of the 11th International Conference on Learning Representations. 2023
24
X, Wang S, Zhang W, Zhang W, Dong J, Chen Y, Wen W Zhang . Quantifying zero-shot coordination capability with behavior preferring partners. In: Proceedings of the 12th International Conference on Learning Representations. 2024
25
S, Kapetanakis D Kudenko . Reinforcement learning of coordination in heterogeneous cooperative multi-agent systems. In: Proceedings of the 3rd International Joint Conference on Autonomous Agents and Multiagent Systems. 2004, 1258−1259
26
C, Wang C, Pérez-D’Arpino D, Xu F F, Li K, Liu S Savarese . Co-GAIL: Learning diverse strategies for human-robot collaboration. In: Proceedings of the 5th Conference on Robot Learning. 2022, 1279−1290
27
K, Xue Y, Wang C, Guan L, Yuan H, Fu Q, Fu C, Qian Y Yu . Heterogeneous multi-agent zero-shot coordination by coevolution. 2022, arXiv preprint arXiv: 2208.04957
28
C, Cabrera A, Paleyes P, Thodoroff N D Lawrence . Real-world machine learning systems: a survey from a data-oriented architecture perspective. 2023, arXiv preprint arXiv: 2302.04810
29
T H, Davenport R Ronanki . Artificial intelligence for the real world. Harvard Business Review, 2018, 96(1): 108−116
30
M C, Fontaine Y C, Hsu Y, Zhang B, Tjanaka S Nikolaidis . On the importance of environments in human-robot coordination. In: Proceedings of the 17th Robotics: Science and Systems 2021. 2021
31
L, Busoniu R, Babuska Schutter B De . A comprehensive survey of multiagent reinforcement learning. IEEE Transactions on Systems, Man, and Cybernetics, Part C (Applications and Reviews), 2008, 38( 2): 156–172
32
K, Zhang Z, Yang T Başar . Multi-agent reinforcement learning: a selective overview of theories and algorithms. In: Vamvoudakis K G, Wan Y, Lewis F L, Cansever D, eds. Handbook of Reinforcement Learning and Control. Cham: Springer, 2021, 321−384
33
G, Sartoretti J, Kerr Y, Shi G, Wagner T K S, Kumar S, Koenig H Choset . Primal: pathfinding via reinforcement and imitation multi-agent learning. IEEE Robotics and Automation Letters, 2019, 4( 3): 2378–2385
34
J, Wang W, Xu Y, Gu W, Song T C Green . Multi-agent reinforcement learning for active voltage control on power distribution networks. In: Proceedings of the 35th Conference on Advances in Neural Information Processing Systems. 2021, 3271−3284
35
K, Xue J, Xu L, Yuan M, Li C, Qian Z, Zhang Y Yu . Multi-agent dynamic algorithm configuration. In: Proceedings of the 36th Conference on Advances in Neural Information Processing Systems. 2022, 20147−20161
36
M, Wen J G, Kuba R, Lin W, Zhang Y, Wen J, Wang Y Yang . Multi-agent reinforcement learning is a sequence modeling problem. In: Proceedings of the 36th Conference on Neural Information Processing Systems. 2022, 16509−16521
37
M, Samvelyan T, Rashid Witt C S, De G, Farquhar N, Nardelli T G J, Rudner C, Hung P H S, Torr J N, Foerster S Whiteson . The starcraft multi-agent challenge. In: Proceedings of the 18th International Conference on Autonomous Agents and MultiAgent Systems. 2019, 2186−2188
38
N, Bard J N, Foerster S, Chandar N, Burch M, Lanctot H F, Song E, Parisotto V, Dumoulin S, Moitra E, Hughes I, Dunning S, Mourad H, Larochelle M G, Bellemare M Bowling . The hanabi challenge: A new frontier for AI research. Artificial Intelligence, 2020, 280: 103216
39
C, Zhu M, Dastani S Wang . A survey of multi-agent reinforcement learning with communication. 2022, arXiv preprint arXiv: 2203.08975
40
F, Zhang C, Jia Y C, Li L, Yuan Y, Yu Z Zhang . Discovering generalizable multi-agent coordination skills from multi-task offline data. In: Proceedings of the 11th International Conference on Learning Representations. 2023
41
X, Wang Z, Zhang W Zhang . Model-based multi-agent reinforcement learning: Recent progress and prospects. 2022, arXiv preprint arXiv: 2203.10603
42
J, Guo Y, Chen Y, Hao Z, Yin Y, Yu S Li . Towards comprehensive testing on the robustness of cooperative multi-agent reinforcement learning. In: Proceedings of 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW). 2022
43
L, Yuan Z, Zhang K, Xue H, Yin F, Chen C, Guan L, Li C, Qian Y Yu . Robust multi-agent coordination via evolutionary generation of auxiliary adversarial attackers. In: Proceedings of the 37th AAAI Conference on Artificial Intelligence. 2023, 1319
44
J N, Foerster Y M, Assael Freitas N, De S Whiteson . Learning to communicate with deep multi-agent reinforcement learning. In: Proceedings of the 30th International Conference on Neural Information Processing Systems. 2016, 2145−2153
45
S, Sukhbaatar A, Szlam R Fergus . Learning multiagent communication with backpropagation. In: Proceedings of the 30th International Conference on Neural Information Processing Systems. 2016, 2252−2260
46
Z, Ding T, Huang Z Lu . Learning individually inferred communication for multi-agent cooperation. In: Proceedings of the 34th International Conference on Neural Information Processing Systems. 2020, 1851
47
H, Mao Z, Zhang Z, Xiao Z, Gong Y Ni . Learning agent communication under limited bandwidth by message pruning. In: Proceedings of the 34th AAAI Conference on Artificial Intelligence. 2020, 5142−5149
48
L, Yuan J, Wang F, Zhang C, Wang Z, Zhang Y, Yu C Zhang . Multi-agent incentive communication via decentralized teammate modeling. In: Proceedings of the 36th AAAI Conference on Artificial Intelligence. 2022, 9466−9474
49
S Q, Zhang Q, Zhang J Lin . Efficient communication in multi-agent reinforcement learning via variance based control. In: Proceedings of the 33rd International Conference on Neural Information Processing Systems. 2019, 291
50
S Q, Zhang Q, Zhang J Lin . Succinct and robust multi-agent communication with temporal message control. In: Proceedings of the 34th International Conference on Neural Information Processing Systems. 2020, 1449
51
C, Guan F, Chen L, Yuan C, Wang H, Yin Z, Zhang Y Yu . Efficient multi-agent communication via self-supervised information aggregation. In: Proceedings of the 36th Conference on Neural Information Processing Systems. 2022, 1020−1033
52
A, Das T, Gervet J, Romoff D, Batra D, Parikh M, Rabbat J Pineau . TarMAC: Targeted multi-agent communication. In: Proceedings of the 36th International Conference on Machine Learning. 2019, 1538−1546
53
C, Guan F, Chen L, Yuan Z, Zhang Y Yu . Efficient communication via self-supervised information aggregation for online and offline multi-agent reinforcement learning. 2023, arXiv preprint arXiv: 2302.09605
54
L, Yuan T, Jiang L, Li F, Chen Z, Zhang Y Yu . Robust multi-agent communication via multi-view message certification. 2023, arXiv preprint arXiv: 2305.13936
55
L, Yuan F, Chen Z, Zhang Y Yu . Communication-robust multi-agent learning by adaptable auxiliary multi-agent adversary generation. Frontiers of Computer Science, 2024, 18( 6): 186331
56
J, Gwak J, Jung R, Oh M, Park M A K, Rakhimov J Ahn . A review of intelligent self-driving vehicle software research. KSII Transactions on Internet and Information Systems (TIIS), 2019, 13( 11): 5299–5320
57
O M, Andrychowicz B, Baker M, Chociej R, Józefowicz B, McGrew J, Pachocki A, Petron M, Plappert G, Powell A, Ray J, Schneider S, Sidor J, Tobin P, Welinder L L, Weng W Zaremba . Learning dexterous in-hand manipulation. The International Journal of Robotics Research, 2020, 39( 1): 3–20
58
D C Engelbart . Augmenting human intellect: a conceptual framework. Stanford Research Institute, 2023
59
S, Carter M Nielsen . Using artificial intelligence to augment human intelligence. Distill, 2017, 2( 12): e9
60
H, Hu A, Lerer B, Cui L, Pineda N, Brown J N Foerster . Off-belief learning. In: Proceedings of the 38th International Conference on Machine Learning. 2021, 4369−4379
61
J, Treutlein M, Dennis C, Oesterheld J Foerster . A new formalism, method and open issues for zero-shot coordination. In: Proceedings of the 38th International Conference on Machine Learning. 2021, 10413−10423
62
Y, Li S, Zhang J, Sun Y, Du Y, Wen X, Wang W Pan . Cooperative open-ended learning framework for zero-shot coordination. In: Proceedings of the 40th International Conference on Machine Learning. 2023, 844
63
F A, Oliehoek C Amato . A Concise Introduction to Decentralized POMDPs. Cham: Springer, 2016
64
W, Xue W, Qiu B, An Z, Rabinovich S, Obraztsova C K Yeo . Mis-spoke or mis-lead: Achieving robustness in multi-agent communicative reinforcement learning. In: Proceedings of the 21st International Conference on Autonomous Agents and Multiagent Systems. 2022, 1418−1426
65
D, Silver T, Hubert J, Schrittwieser I, Antonoglou M, Lai A, Guez M, Lanctot L, Sifre D, Kumaran T, Graepel T, Lillicrap K, Simonyan D Hassabis . Mastering chess and shogi by self-play with a general reinforcement learning algorithm. 2017, arXiv preprint arXiv: 1712.01815
66
G Tesauro . TD-gammon, a self-teaching backgammon program, achieves master-level play. Neural Computation, 1994, 6( 2): 215–219
67
M, Jaderberg V, Dalibard S, Osindero W M, Czarnecki J, Donahue A, Razavi O, Vinyals T, Green I, Dunning K, Simonyan C, Fernando K Kavukcuoglu . Population based training of neural networks. 2017, arXiv preprint arXiv: 1711.09846
68
K, Lucas R E Allen . Any-play: an intrinsic augmentation for zero shot coordination. In: Proceedings of the 21st International Conference on Autonomous Agents and Multiagent Systems. 2022, 853–861
69
W U, Mondal M, Agarwal V, Aggarwal S V Ukkusuri . On the approximation of cooperative heterogeneous multi-agent reinforcement learning (MARL) using mean field control (MFC). Journal of Machine Learning Research, 2022, 23( 1): 129
70
J G, Kuba X, Feng S, Ding H, Dong J, Wang Y Yang . Heterogeneous-agent mirror learning: A continuum of solutions to cooperative MARL. 2022, arXiv preprint arXiv: 2208.01682
71
R, Charakorn P, Manoonpong N Dilokthanakul . Generating diverse cooperative agents by learning incompatible policies. In: Proceedings of the 11th International Conference on Learning Representations. 2023
72
X, Lou J, Guo J, Zhang J, Wang K, Huang Y Du . PECAN: leveraging policy ensemble for context-aware zero-shot human-AI coordination. In: Proceedings of the 22nd International Conference on Autonomous Agents and Multiagent Systems. 2023, 679−688
73
S, Zheng A, Trott S, Srinivasa N, Naik M, Gruesbeck D C, Parkes R Socher . The AI economist: Improving equality and productivity with AI-Driven tax policies. 2020, arXiv preprint arXiv: 2004. 13332
74
T Bäck . Evolutionary Algorithms in Theory and Practice: Evolution Strategies, Evolutionary Programming, Genetic Algorithms. New York: Oxford University Press, 1996
75
H, Hao X, Zhang A Zhou . Enhancing SAEAs with unevaluated solutions: A case study of relation model for expensive optimization. Science China Information Sciences, 2024, 67( 2): 120103
76
Y, Wang K, Xue C Qian . Evolutionary diversity optimization with clustering-based selection for reinforcement learning. In: Proceedings of the 10th International Conference on Learning Representations. 2022
77
J Demšar . Statistical comparisons of classifiers over multiple data sets. The Journal of Machine Learning Research, 2006, 7: 1–30
