A comprehensive survey on graph neural network accelerators

doi:10.1007/s11704-023-3307-2

Frontiers of Computer Science

2025, Vol. 19

Issue (2): 192104 https://doi.org/10.1007/s11704-023-3307-2

本期目录

A comprehensive survey on graph neural network accelerators

Jingyu LIU^1,², Shi CHEN^1,², Li SHEN^1,²(

)

¹. School of Computer, National University of Defense Technology, Changsha 410073, China
². Key Laboratory of Advanced Microprocessor Chips and Systems, Changsha 410073, China

全文: PDF(16089 KB) HTML

Abstract：

Deep learning has gained superior accuracy on Euclidean structure data in neural networks. As a result, non-Euclidean structure data, such as graph data, has more sophisticated structural information, which can be applied in neural networks as well to address more complex and practical problems. However, actual graph data obeys a power-law distribution, so the adjacent matrix of a graph is random and sparse. Graph processing accelerator (GPA) is designed to handle the problems above. However, graph computing only processes 1-dimensional data. In graph neural networks (GNNs), graph data is multi-dimensional. Consequently, GNNs include the execution processes of both traditional graph processing and neural network, which have irregular memory access and regular computation, respectively. To obtain more information in graph data and require better model generalization ability, the layers of GNN are deeper, so the overhead of memory access and computation is considerable. At present, GNN accelerators are designed to deal with this issue. In this paper, we conduct a systematic survey regarding the design and implementation of GNN accelerators. Specifically, we review the challenges faced by GNN accelerators, and existing related works in detail to process them. Finally, we evaluate previous works and propose future directions in this booming field.

Key words： graph neural network accelerators graph convolutional networks design space exploration deep learning domain-specific architecture

收稿日期: 2023-04-17 出版日期: 2024-03-18

Corresponding Author(s): Li SHEN

引用本文:

. [J]. Frontiers of Computer Science, 2025, 19(2): 192104.
Jingyu LIU, Shi CHEN, Li SHEN. A comprehensive survey on graph neural network accelerators. Front. Comput. Sci., 2025, 19(2): 192104.

链接本文:

https://academic.hep.com.cn/fcs/CN/10.1007/s11704-023-3307-2
https://academic.hep.com.cn/fcs/CN/Y2025/V19/I2/192104

Fig.1

Tab.1

Fig.2

Tab.2

Tab.3

Fig.3

Fig.4

Fig.5

Fig.6

Fig.7

Fig.8

Fig.9

Fig.10

Fig.11

Fig.12

Fig.13

Fig.14

Fig.15

Fig.16

Fig.17

Fig.18

Fig.19

Fig.20

Fig.21

Fig.22

Fig.23

Fig.24

Fig.25

Fig.26

Fig.27

Fig.28

Fig.29

Fig.30

Notations	Descriptions
$G$	A graph
$\| ? \|$	The length of a set
$V$	The set of nodes in a graph
$?$	Element-wise product
$A T$	Transpose of adjacent matrix $A$
$v i$	A node $v i ∈$ $V$
$E$	The set of edges in a graph
$N (v)$	the neighbors of node $v$
$D v$	The degrees of a node $v$
$e i j$	An edge $e i j ∈$ $E$
$N$	The number of nodes, $N = \| V \|$
$M$	The number of edges, $M = \| E \|$
$D$	The dimension of a node vector
$X$ $∈ R N × D$	The feature matrix of a graph
$x$ $∈ R N$	The feature vector of a graph when $D$ equals 1
$X i ∈ R D$	The feature vector of the node $v i$

Models	Combine	Aggregate
GCN	$h v l ? 1$	$∑ c u l$
GIN	$M L P l (h v l ? 1, W l)$	$(1 + ?) ? c v l + ∑ c u l$
GraphSAGE	$h v l ? 1$	$M e a n (c u l)$
GAT	$h v l ? 1$	$∑ α v, u l ? c u l$

1	Cao S, Lu W, Xu Q. Deep neural networks for learning graph representations. In: Proceedings of the 30th AAAI Conference on Artificial Intelligence. 2016, 1145−1152
2	P, Velickovic G, Cucurull A, Casanova A, Romero P, Liò Y Bengio . Graph attention networks. In: Proceedings of the 6th International Conference on Learning Representations. 2018
3	J, You R, Ying X, Ren W L, Hamilton J Leskovec . GraphRNN: a deep generative model for graphs. 2018, arXiv preprint arXiv: 1802.08773
4	K, Xu W, Hu J, Leskovec S Jegelka . How powerful are graph neural networks? In: Proceedings of the 7th International Conference on Learning Representations. 2019
5	Z, Wu S, Pan F, Chen G, Long C, Zhang P S Yu . A comprehensive survey on graph neural networks. IEEE Transactions on Neural Networks and Learning Systems, 2021, 32( 1): 4–24
6	W L, Hamilton Z, Ying J Leskovec . Inductive representation learning on large graphs. In: Proceedings of the 31st International Conference on Neural Information Processing Systems. 2017, 1025−1035
7	R, Ying R, He K, Chen P, Eksombatchai W L, Hamilton J Leskovec . Graph convolutional neural networks for web-scale recommender systems. In: Proceedings of the 24th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining. 2018, 974−983
8	T N, Kipf M Welling . Semi-supervised classification with graph convolutional networks. In: Proceedings of the 5th International Conference on Learning Representations. 2017
9	H, Gao Z, Wang S Ji . Large-scale learnable graph convolutional networks. In: Proceedings of the 24th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining. 2018, 1416−1424
10	Li R, Wang S, Zhu F, Huang J. Adaptive graph convolutional neural networks. In: Proceedings of the 32nd AAAI Conference on Artificial Intelligence, (AAAI 18), the 30th Innovative Applications of Artificial Intelligence (IAAI-18), and the 8th AAAI Symposium on Educational Advances in Artificial Intelligence (EAAI 18). 2018, 434
11	Xu K, Hu W, Leskovec J, Jegelka S. How powerful are graph neural networks? 2018, arXiv preprint arXiv: 1810.00826
12	Zhang M, Cui Z, Neumann M, Chen Y. An end-to-end deep learning architecture for graph classification. In: Proceedings of the 32nd AAAI Conference on Artificial Intelligence, (AAAI-18), the 30th Innovative Applications of Artificial Intelligence (IAAI-18), and the 8th AAAI Symposium on Educational Advances in Artificial Intelligence (EAAI-18). 2018, 4438−4445
13	Yin L, Wang J, Zheng H. Exploring architecture, dataflow, and sparsity for gcn accelerators: a holistic framework. In: Proceedings of the Great Lakes Symposium on VLSI 2023. 2023, 489–495
14	Garg R, Qin E, Munoz-Matrinez F, Guirado R, Jain A, Abadal S, Abellan J L, Acacio M E, Alarcon E, Rajamanickam S, Krishna T. Understanding the design space of sparse/dense multiphase gnn dataflows on spatial accelerators. In: Proceedings of IEEE International Parallel and Distributed Processing Symposium. 2022, 571–582
15	T, Hamaguchi H, Oiwa M, Shimbo Y Matsumoto . Knowledge transfer for out-of-knowledge-base entities: a graph neural network approach. In: Proceedings of the 26th International Joint Conference on Artificial Intelligence. 2017, 1802−1808
16	M, Schlichtkrull T N, Kipf P, Bloem den Berg R, van I, Titov M Welling . Modeling relational data with graph convolutional networks. In: Proceedings of the 15th International Conference on the Semantic Web. 2018, 593−607
17	J, Zhou G, Cui S, Hu Z, Zhang C, Yang Z, Liu L, Wang C, Li M Sun . Graph neural networks: a review of methods and applications. AI Open, 2020, 1: 57–81
18	L, Ma Z, Yang Y, Miao J, Xue M, Wu L, Zhou Y Dai . NeuGraph: parallel deep neural network computation on large graphs. In: Proceedings of 2019 USENIX Annual Technical Conference. 2019, 443−458
19	M, Yan Z, Chen L, Deng X, Ye Z, Zhang D, Fan Y Xie . Characterizing and understanding GCNs on GPU. IEEE Computer Architecture Letters, 2020, 19( 1): 22–25
20	Yang J, Tang D, Song X, Wang L, Yin Q, Chen R, Yu W, Zhou J. GNNLab: a factored system for sample based GNN training over GPUs. In: Proceedings of the 17th European Conference on Computer Systems. 2022, 417−434
21	Wang L, Yin Q, Tian C, Yang J, Chen R, Yu W, Yao Z, Zhou J. FlexGraph: a flexible and efficient distributed framework for GNN training. In: Proceedings of the 16th European Conference on Computer Systems. 2021, 67−82
22	Tailor S A, Fernández-Marqués J, Lane N D. Degree-quant: quantization-aware training for graph neural networks. In: Proceedings of the 9th International Conference on Learning Representations. 2021
23	B, Feng Y, Wang X, Li S, Yang X, Peng Y Ding . SGQuant: squeezing the last bit on graph neural networks with specialized quantization. In: Proceedings of the 32nd IEEE International Conference on Tools with Artificial Intelligence. 2020, 1044−1052
24	Y, Wang B, Feng Y Ding . QGTC: accelerating quantized graph neural networks via GPU tensor core. In: Proceedings of the 27th ACM SIGPLAN Symposium on Principles and Practice of Parallel Programming. 2022, 107−119
25	Y, Wang B, Feng G, Li S, Li L, Deng Y, Xie Y Ding . GNNAdvisor: an adaptive and efficient runtime system for GNN acceleration on GPUs. In: Proceedings of the 15th USENIX Symposium on Operating Systems Design and Implementation. 2021, 515−531
26	T, Geng A, Li R, Shi C, Wu T, Wang Y, Li P, Haghi A, Tumeo S, Che S, Reinhardt M C Herbordt . AWB-GCN: a graph convolutional network accelerator with runtime workload rebalancing. In: Proceedings of the 53rd Annual IEEE/ACM International Symposium on Microarchitecture. 2020, 922−936
27	Y, Huang L, Zheng P, Yao Q, Wang X, Liao H, Jin J Xue . Accelerating graph convolutional networks using crossbar-based processing-in-memory architectures. In: Proceedings of IEEE International Symposium on High-Performance Computer Architecture. 2022, 1029−1042
28	J, Li A, Louri A, Karanth R C Bunescu . GCNAX: a flexible and energy-efficient accelerator for graph convolutional neural networks. In: Proceedings of IEEE International Symposium on High-Performance Computer Architecture. 2021, 775−788
29	H, You T, Geng Y, Zhang A, Li Y Lin . GCoD: graph convolutional network acceleration via dedicated algorithm and accelerator Co-design. In: Proceedings of IEEE International Symposium on High-Performance Computer Architecture. 2022, 460−474
30	T, Geng C, Wu Y, Zhang C, Tan C, Xie H, You M, Herbordt Y, Lin A Li . I-GCN: a graph convolutional network accelerator with runtime locality enhancement through islandization. In: Proceedings of the 54th Annual IEEE/ACM International Symposium on Microarchitecture. 2021, 1051−1063
31	Z, Gong H, Ji Y, Yao C W, Fletcher C J, Hughes J Torrellas . Graphite: optimizing graph neural networks on CPUs through cooperative software-hardware techniques. In: Proceedings of the 49th Annual International Symposium on Computer Architecture. 2022, 916−931
32	R, Hwang M, Kang J, Lee D, Kam Y, Lee M Rhu . GROW: a row-stationary sparse-dense GEMM accelerator for memory-efficient graph convolutional neural networks. In: Proceedings of IEEE International Symposium on High-Performance Computer Architecture. 2023, 42−55
33	M, Yoo J, Song J, Lee N, Kim Y, Kim J Lee . SGCN: exploiting compressed-sparse features in deep graph convolutional network accelerators. In: Proceedings of IEEE International Symposium on High-Performance Computer Architecture. 2023, 1−14
34	T, Yang D, Li F, Ma Z, Song Y, Zhao J, Zhang F, Liu L Jiang . PASGCN: an ReRAM-based PIM design for GCN with adaptively sparsified graphs. IEEE Transactions on Computer-Aided Design of Integrated Circuits and Systems, 2023, 42( 1): 150–163
35	S, Xu Z, Shao C, Yang X, Liao H Jin . Accelerating backward aggregation in GCN training with execution path preparing on GPUs. IEEE Transactions on Parallel and Distributed Systems, 2022, 33( 12): 4891–4902
36	C, Gui L, Zheng B, He C, Liu X, Chen X, Liao H Jin . A survey on graph processing accelerators: challenges and opportunities. Journal of Computer Science and Technology, 2019, 34( 2): 339–371
37	A, Roy I, Mihailovic W Zwaenepoel . X-stream: edge-centric graph processing using streaming partitions. In: Proceedings of the 24th Symposium on Operating Systems Principles. 2013, 472−488
38	B, Perozzi R, Al-Rfou S Skiena . DeepWalk: online learning of social representations. In: Proceedings of the 20th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. 2014, 701−710
39	M M, Ozdal S, Yesil T, Kim A, Ayupov J, Greth S, Burns O Ozturk . Energy efficient architecture for graph analytics accelerators. In: Proceedings of the 43rd ACM/IEEE Annual International Symposium on Computer Architecture. 2016, 166−177
40	M, Zhang Y, Zhuo C, Wang M, Gao Y, Wu K, Chen C, Kozyrakis X Qian . GraphP: reducing communication for PIM-based graph processing with efficient data partition. In: Proceedings of IEEE International Symposium on High Performance Computer Architecture. 2018, 544−557
41	L, Song Y, Zhuo X, Qian H, Li Y Chen . GraphR: accelerating graph processing using ReRAM. In: Proceedings of IEEE International Symposium on High Performance Computer Architecture. 2018, 531−543
42	C, Xie L, Yan W J, Li Z Zhang . Distributed power-law graph computing: theoretical and empirical analysis. In: Proceedings of the 27th International Conference on Neural Information Processing Systems. 2014, 1673−1681
43	J E, Gonzalez Y, Low H, Gu D, Bickson C Guestrin . PowerGraph: distributed graph-parallel computation on natural graphs. In: Proceedings of the 10th USENIX Conference on Operating Systems Design and Implementation. 2012, 17−30
44	S, Heidari Y, Simmhan R N, Calheiros R Buyya . Scalable graph processing frameworks: a taxonomy and open challenges. ACM Computing Surveys, 2019, 51( 3): 60
45	T J, Ham L, Wu N, Sundaram N, Satish M Martonosi . Graphicionado: a high-performance and energy-efficient accelerator for graph analytics. In: Proceedings of the 49th Annual IEEE/ACM International Symposium on Microarchitecture. 2016, 1−13
46	M, Yan X, Hu S, Li A, Basak H, Li X, Ma I, Akgun Y, Feng P, Gu L, Deng X, Ye Z, Zhang D, Fan Y Xie . Alleviating irregularity in graph analytics acceleration: a hardware/software Co-design approach. In: Proceedings of the 52nd Annual IEEE/ACM International Symposium on Microarchitecture. 2019, 615−628
47	S, Zhang Z, Qin Y, Yang L, Shen Z Wang . Transparent partial page migration between CPU and GPU. Frontiers of Computer Science, 2020, 14( 3): 143101
48	S, Fan J, Fei L Shen . Accelerating deep learning with a parallel mechanism using CPU + MIC. International Journal of Parallel Programming, 2018, 46( 4): 660–673
49	M, Yan L, Deng X, Hu L, Liang Y, Feng X, Ye Z, Zhang D, Fan Y Xie . HyGCN: a GCN accelerator with hybrid architecture. In: Proceedings of IEEE International Symposium on High Performance Computer Architecture. 2020, 15−29
50	T, Yang D, Li Y, Han Y, Zhao F, Liu X, Liang Z, He L Jiang . PIMGCN: a ReRAM-based PIM design for graph convolutional network acceleration. In: Proceedings of the 58th ACM/IEEE Design Automation Conference. 2021, 583−588
51	H Yang . AliGraph: a comprehensive graph neural network platform. In: Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining. 2019, 3165−3166
52	H, Zeng V K Prasanna . GraphACT: accelerating GCN training on CPU-FPGA heterogeneous platforms. In: Proceedings of 2020 ACM/SIGDA International Symposium on Field-Programmable Gate Arrays. 2020, 255−265
53	Kung H T. Why systolic architectures? Computer, 1982, 15(1): 37−46
54	S, Liang Y, Wang C, Liu L, He H, Li D, Xu X Li . EnGN: a high-throughput and energy-efficient accelerator for large graph neural networks. IEEE Transactions on Computers, 2021, 70( 9): 1511–1525
55	A, Auten M, Tomei R Kumar . Hardware acceleration of graph neural networks. In: Proceedings of the 57th ACM/IEEE Design Automation Conference. 2020, 1−6
56	S, Liang C, Liu Y, Wang H, Li X Li . DeepBurning-GL: an automated framework for generating graph neural network accelerators. In: Proceedings of IEEE/ACM International Conference on Computer Aided Design. 2020, 72
57	X, Song T, Zhi Z, Fan Z, Zhang X, Zeng W, Li X, Hu Z, Du Q, Guo Y Chen . Cambricon-G: a polyvalent energy-efficient accelerator for dynamic graph neural networks. IEEE Transactions on Computer-Aided Design of Integrated Circuits and Systems, 2022, 41( 1): 116–128
58	Zhou Z, Shi B, Zhang Z, Guan Y, Sun G, Luo G. BlockGNN: towards efficient GNN acceleration using block-circulant weight matrices. In: Proceedings of the 58th ACM/IEEE Design Automation Conference. 2021, 1009−1014
59	A I, Arka J R, Doppa P P, Pande B K, Joardar K Chakrabarty . RegraphX: NoC-enabled 3D heterogeneous ReRAM architecture for training graph neural networks. In: Proceedings of Design, Automation & Test in Europe Conference & Exhibition. 2021, 1667−1672
60	K, Kiningham P, Levis C Ré . GRIP: a graph neural network accelerator architecture. IEEE Transactions on Computers, 2023, 72( 4): 914–925
61	C, Chen K, Li Y, Li X Zou . ReGNN: a redundancy-eliminated graph neural networks accelerator. In: Proceedings of IEEE International Symposium on High-Performance Computer Architecture. 2022, 429−443
62	C, Liu H, Liu H, Jin X, Liao Y, Zhang Z, Duan J, Xu H Li . ReGNN: a ReRAM-based heterogeneous architecture for general graph neural networks. In: Proceedings of the 59th ACM/IEEE Design Automation Conference. 2022, 469−474
63	Y, Lee J, Chung M Rhu . SmartsAGE: training large-scale graph neural networks using in-storage processing architectures. In: Proceedings of the 49th Annual International Symposium on Computer Architecture. 2022, 932−945
64	Q, Sun Y, Liu H, Yang R, Zhang M, Dun M, Li X, Liu W, Xiao Y, Li Z, Luan D Qian . CoGNN: efficient scheduling for concurrent GNN training on GPUs. In: Proceedings of International Conference on High Performance Computing, Networking, Storage and Analysis. 2022, 39
65	R, Sarkar S, Abi-Karam Y, He L, Sathidevi C Hao . FlowGNN: a dataflow architecture for real-time workload-agnostic graph neural network inference. In: Proceedings of the 29th IEEE International Symposium on High-Performance Computer Architecture. 2023, 1099−1112
66	S, Yun B, Kim J, Park H, Nam J H, Ahn E Lee . GraNDe: near-data processing architecture with adaptive matrix mapping for graph convolutional networks. IEEE Computer Architecture Letters, 2022, 21( 2): 45–48
67	F G Gustavson . Two fast algorithms for sparse matrices: multiplication and permuted transposition. ACM Transactions on Mathematical Software, 1978, 4( 3): 250–269

Viewed

Full text

Abstract

Cited

Shared

Discussed