Adapting to the stream: an instance-attention GNN method for irregular multivariate time series data
Kun HAN1, Abigail M Y KOAY2, Ryan K L KO1, Weitong CHEN3(), Miao XU1()
. School of Electrical Engineering and Computer Science, The University of Queensland,Brisbane Queensland 4072, Australia . School of Science, Technology and Engineering, University of the Sunshine Coast, Queensland 4556, Australia . School of Computer and Mathematical Sciences, The University of Adelaide, Adelaide, South Australia 5005, Australia
Multivariate time series (MTS) data are vital for various applications, particularly in machine learning tasks. However, challenges such as sensor failures can result in irregular and misaligned data with missing values, thereby complicating their analysis. While recent advancements use graph neural networks (GNNs) to manage these Irregular Multivariate Time Series (IMTS) data, they generally require a reliable graph structure, either pre-existing or inferred from adequate data to properly capture node correlations. This poses a challenge in applications where IMTS data are often streamed and waiting for future data to estimate a suitable graph structure becomes impractical. To overcome this, we introduce a dynamic GNN model suited for streaming characteristics of IMTS data, incorporating an instance-attention mechanism that dynamically learns and updates graph edge weights for real-time analysis. We also tailor strategies for high-frequency and low-frequency data to enhance prediction accuracy. Empirical results on real-world datasets demonstrate the superiority of our proposed model in both classification and imputation tasks.
Kun HAN,Abigail M Y KOAY,Ryan K L KO, et al. Adapting to the stream: an instance-attention GNN method for irregular multivariate time series data[J]. Front. Comput. Sci.,
2025, 19(8): 198340.
Fig.1 Comparison of regular multivariate time series and irregular multivariate time series. Different colors represent different variables in the MTS. Solid circles denote observed data points, while hollow circles indicate missing data points
Fig.2 Framework of DynIMTS. The model is a recurrent structure based on a spatial-temporal encoder and consists of three main components: embedding learning, spatial-temporal learning, and graph learning. The denotes samples in a Batch, and and is the corresponding graph structure. The final learned sensor embedding is used for classification
AQI-36
AQI
MAE
MSE
MRE
MAE
MSE
MRE
GRIN (physical)
15.8 ± 0.6
792.0 ± 93.9
0.5 ± 0.0
17.4 ± 0.3
935.3 ± 34.3
0.5 ± 0.0
RITS
16.8 ± 1.7
872.6 ± 184.5
0.7 ± 0.1
22.8 ± 0.7
1437.7 ± 99.3
0.6 ± 0.0
GRIN (all-ones)
15.7 ± 0.5
710.7 ± 45.7
0.6 ± 0.0
19.8 ± 0.3
1089.0 ± 11.0
0.6 ± 0.0
DynIMTS
14.9 ± 1.1
673.0 ± 149.7
0.5 ± 0.0
18.3 ± 0.3
997.8 ± 70.4
0.5 ± 0.0
Tab.1 Comparative performance of imputation models on AQI and AQI-36 datasets (mean ± std)
Fig.3 Evaluation of imputation methods at 20% and 80% missing data ratios for exchange rates dataset. (a) MAE by missing ratio and method; (b) MSE by missing ratio and method; (c) MRE by missing ratio and method
Fig.4 Visualization of the imputation results produced by our model with a 20% missing data ratio on the Exchange-rate dataset across eight dimensions. The black lines represent the ground truth values, while the orange lines represent the predicted values. (a) Dimension 1; (b) Dimension 2; (c) Dimension 3; (d) Dimension 4; (e) Dimension 5; (f) Dimension 6; (g) Dimension 7; (h) Dimension 8
P12
P19
AUROC
AUPRC
AUROC
AUPRC
RITS
71.2 ± 2.0
16.4 ± 2.7
86.7 ± 0.7
47.2 ± 3.1
Raindrop
60.9 ± 5.3
14.7 ± 2.0
74.1 ± 0.8
33.6 ± 1.7
GRIN (all-ones)
70.8 ± 2.0
12.5 ± 2.8
87.1 ± 1.1
48.4 ± 2.0
DynIMTS
75.1 ± 1.6
25.2 ± 3.0
87.1 ± 1.1
48.4 ± 1.9
Tab.2 Evaluation on P12 and P19 datasets using AUROC% and AUPRC% (mean ± std)
Missing ratio/%
Models
PAM
AWR
10
RITS
87.8 ± 0.6
83.8 ± 2.8
10
Raindrop
75.5 ± 1.7
52.5 ± 4.8
10
GRIN (all-ones)
97.5 ± 0.6
79.1 ± 4.0
10
DynIMTS
97.4 ± 0.5
84.4 ± 3.7
20
RITS
79.1 ± 0.7
73.2 ± 16.8
20
Raindrop
75.5 ± 1.7
47.5 ± 5.8
20
GRIN (all-ones)
97.6 ± 0.2
80.0 ± 3.6
20
DynIMTS
98.3 ± 0.2
83.5 ± 3.2
30
RITS
73.5± 0.8
69.2 ± 14.5
30
Raindrop
70.4 ± 0.5
47.7 ± 6.6
30
GRIN (all-ones)
95.9 ± 1.2
77.4 ± 3.2
30
DynIMTS
97.2 ± 0.7
77.2 ± 5.7
40
RITS
71.9 ± 0.9
56.0 ± 24.6
40
Raindrop
64.0 ± 1.4
50.6 ± 5.2
40
GRIN (all-ones)
93.5 ± 0.6
76.5 ± 5.9
40
DynIMTS
94.0 ± 0.5
78.6 ± 2.2
50
RITS
61.1 ± 1.9
59.0 ± 14.0
50
Raindrop
58.3 ± 2.0
34.6 ± 3.8
50
GRIN (all-ones)
93.5 ± 0.6
76.5 ± 4.3
50
DynIMTS
95.8 ± 0.5
78.8 ± 6.6
Tab.3 Accuracy% (mean ± std) on PAM and AWR datasets with various missing rates
P12
P19
AUROC
AUPRC
AUROC
AUPRC
w/o graph
57.9 ± 1.3
8.6 ± 0.7
81.1 ± 1.6
39.1 ± 2.9
w graph normalisation
59.3 ± 5.1
9.7 ± 1.6
83.4 ± 2.1
42.1 ± 6.4
w/o instance attention
69.3 ± 8.3
18.7 ± 9.0
85.2 ± 2.4
46.0 ± 5.7
DynIMTS
75.1 ± 1.6
25.2 ± 3.0
87.8 ± 0.5
47.5 ± 3.5
Tab.4 Ablation study for the impact of graph, instance attention and graph normalisation
Fig.5 Visualization of the adjacency matrices of epochs 0, 10 and 20 on the P19 dataset. The darker the colour is, the larger the value on the corresponding point. (a) Graph at epoch 0; (b) graph at epoch 10; (c) graph at epoch 20
1
H I, Fawaz G, Forestier J, Weber L, Idoumghar P A Muller . Deep learning for time series classification: a review. Data Mining and Knowledge Discovery, 2019, 33( 4): 917–963
2
M, Schirmer M, Eltayeb S, Lessmann M Rudolph . Modeling irregular time series with continuous recurrent units. In: Proceedings of the 39th International Conference on Machine Learning. 2022, 19388−19405
3
O B, Sezer M U, Gudelek A M Ozbayoglu . Financial time series forecasting with deep learning: a systematic literature review: 2005–2019. Applied Soft Computing, 2020, 90: 106181
4
C, Feng P Tian . Time series anomaly detection for cyber-physical systems via neural system identification and Bayesian filtering. In: Proceedings of the 27th ACM SIGKDD Conference on Knowledge Discovery & Data Mining. 2021, 2858−2867
5
A P, Mathur N O Tippenhauer . SWaT: a water treatment testbed for research and training on ICS security. In: Proceedings of 2016 International Workshop on Cyber-physical Systems for Smart Water Networks. 2016, 31−36
6
C M, Ahmed V R, Palleti A P Mathur . WADI: a water distribution testbed for research in the design of secure cyber physical systems. In: Proceedings of the 3rd International Workshop on Cyber-Physical Systems for Smart Water Networks. 2017, 25−28
7
D, Cao Y, Wang J, Duan C, Zhang X, Zhu C, Huang Y, Tong B, Xu J, Bai J, Tong Q Zhang . Spectral temporal graph neural network for multivariate time-series forecasting. In: Proceedings of the 34th International Conference on Neural Information Processing Systems. 2020, 1491
8
Y, Yehuda D, Freedman K Radinsky . Self-supervised classification of clinical multivariate time series using time series dynamics. In: Proceedings of the 29th ACM SIGKDD Conference on Knowledge Discovery and Data Mining. 2023, 5416−5427
9
G, Li B, Choi J, Xu S S, Bhowmick K P, Chun G L H Wong . ShapeNet: a shapelet-neural network approach for multivariate time series classification. In: Proceedings of the 35th AAAI Conference on Artificial Intelligence. 2021, 8375−8383
10
Q, Tan M, Ye B, Yang S, Liu A J, Ma T C F, Yip G L H, Wong P C Yuen . DATA-GRU: dual-attention time-aware gated recurrent unit for irregular multivariate time series. In: Proceedings of the 34th AAAI Conference on Artificial Intelligence. 2020, 930−937
11
P, Kidger J, Morrill J, Foster T Lyons . Neural controlled differential equations for irregular time series. In: Proceedings of the 34th International Conference on Neural Information Processing Systems. 2020, 562
12
H, Zhou S, Zhang J, Peng S, Zhang J, Li H, Xiong W Zhang . Informer: beyond efficient transformer for long sequence time-series forecasting. In: Proceedings of the 35th AAAI Conference on Artificial Intelligence. 2021, 11106−11115
13
Y, Nie N H, Nguyen P, Sinthong J Kalagnanam . A time series is worth 64 words: long-term forecasting with transformers. In: Proceedings of the 11th International Conference on Learning Representations. 2023
14
M T, Bahadori Y Liu . Granger causality analysis in irregular time series. In: Proceedings of the 12th SIAM International Conference on Data Mining. 2012, 660−671
15
M, Schulz K Stattegger . Spectrum: spectral analysis of unevenly spaced paleoclimatic time series. Computers & Geosciences, 1997, 23( 9): 929–945
16
K, Rehfeld N, Marwan J, Heitzig J Kurths . Comparison of correlation analysis techniques for irregularly sampled time series. Nonlinear Processes in Geophysics, 2011, 18( 3): 389–404
17
P, Babu P Stoica . Spectral analysis of nonuniformly sampled data - a review. Digital Signal Processing, 2010, 20( 2): 359–378
18
P, Stoica J, Li H He . Spectral analysis of nonuniformly sampled data: a new approach versus the periodogram. IEEE Transactions on Signal Processing, 2009, 57( 3): 843–858
19
S C X, Li B Marlin . Classification of sparse and irregularly sampled time series with mixtures of expected Gaussian kernels and random features. In: Proceedings of the 31st Conference on Uncertainty in Artificial Intelligence. 2015, 484−493
20
S C X, Li B Marlin . A scalable end-to-end Gaussian process adapter for irregularly sampled time series classification. In: Proceedings of the 30th International Conference on Neural Information Processing Systems. 2016, 1804−1812
21
Z, Che S, Purushotham K, Cho D, Sontag Y Liu . Recurrent neural networks for multivariate time series with missing values. Scientific Reports, 2018, 8( 1): 6085
22
Y, Rubanova R T Q, Chen D Duvenaud . Latent ODEs for irregularly-sampled time series. In: Proceedings of the 33rd International Conference on Neural Information Processing Systems. 2019, 478
23
Z, Wang Y, Zhang A, Jiang J, Zhang Z, Li J, Gao K, Li C, Lu Z Ren . Improving irregularly sampled time series learning with time-aware dual-attention memory-augmented networks. In: Proceedings of the 30th ACM International Conference on Information & Knowledge Management. 2021, 3523−3527
24
S N, Shukla B M Marlin . Multi-time attention networks for irregularly sampled time series. In: Proceedings of the 9th International Conference on Learning Representations. 2021
25
S, Tipirneni C K Reddy . Self-supervised transformer for sparse and irregularly sampled multivariate clinical time-series. ACM Transactions on Knowledge Discovery from Data (TKDD), 2022, 16( 6): 105
26
J, Zhang S, Zheng W, Cao J, Bian J Li . Warpformer: a multi-scale modeling approach for irregular clinical time series. In: Proceedings of the 29th ACM SIGKDD Conference on Knowledge Discovery and Data Mining. 2023, 3273−3285
27
Y, Chen K, Ren Y, Wang Y, Fang W, Sun D Li . ContiFormer: continuous-time transformer for irregular time series modeling. In: Proceedings of the 37th International Conference on Neural Information Processing Systems. 2023, 2042
28
C, Sun S, Hong M, Song Y H, Chou Y, Sun D, Cai H Li . TE-ESN: time encoding echo state network for prediction based on irregularly sampled time series data. In: Proceedings of the 30th International Joint Conference on Artificial Intelligence. 2021, 3010−3016
29
A, Cini I, Marisca C Alippi . Filling the G_ap_s: multivariate time series imputation by graph neural networks. In: Proceedings of the 10th International Conference on Learning Representations. 2022
30
Y, Wei J, Peng T, He C, Xu J, Zhang S, Pan S Chen . Compatible transformer for irregularly sampled multivariate time series. In: Proceedings of 2023 IEEE International Conference on Data Mining. 2023, 1409−1414
31
X, Zhang M, Zeman T, Tsiligkaridis M Zitnik . Graph-guided network for irregularly sampled multivariate time series. In: Proceedings of the 10th International Conference on Learning Representations. 2022
32
Z, Wang T, Jiang Z, Xu J, Zhang J Gao . Irregularly sampled multivariate time series classification: a graph learning approach. IEEE Intelligent Systems, 2023, 38( 3): 3–11
33
Z, Wang T, Jiang Z, Xu J, Gao O, Wu K, Yan J Zhang . Uncovering multivariate structural dependency for analyzing irregularly sampled time series. In: Proceedings of European Conference on Machine Learning and Knowledge Discovery in Databases: Research Track. 2023, 238−254
34
Y, Liu Q, Liu J W, Zhang H, Feng Z, Wang Z, Zhou W Chen . Multivariate time-series forecasting with temporal polynomial graph neural networks. In: Proceedings of the 36th International Conference on Neural Information Processing Systems. 2024, 1411
35
de Barros M, Rodrigues T L, Rissi E F, Cabrera E A, Tannuri E S, Gomi R A, Barreira A H R Costa . Embracing data irregularities in multivariate time series with recurrent and graph neural networks. In: Proceedings of the 12th Brazilian Conference on Intelligent Systems. 2023, 3−17
36
W, Cao D, Wang J, Li H, Zhou L, Li Y Li . BRITS: bidirectional recurrent imputation for time series. In: Proceedings of the 32nd International Conference on Neural Information Processing Systems. 2018, 6776−6786
37
Y, Luo X, Cai Y, Zhang J, Xu X Yuan . Multivariate time series imputation with generative adversarial networks. In: Proceedings of the 32nd International Conference on Neural Information Processing Systems. 2018, 1603−1614
38
A, Deng B Hooi . Graph neural network-based anomaly detection in multivariate time series. In: Proceedings of the 35th AAAI Conference on Artificial Intelligence. 2021, 4027−4035
39
Y, Cui K, Zheng D, Cui J, Xie L, Deng F, Huang X Zhou . METRO: a generic graph neural network framework for multivariate time series forecasting. Proceedings of the VLDB Endowment, 2021, 15( 2): 224–236
40
T, Ma P, Ferber S, Huo J, Chen M Katz . Online planner selection with graph neural networks and adaptive scheduling. In: Proceedings of the 34th AAAI Conference on Artificial Intelligence. 2020, 5077−5084
41
W, Luo H, Zhang X, Yang L, Bo X, Yang Z, Li X, Qie J Ye . Dynamic heterogeneous graph neural network for real-time event prediction. In: Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining. 2020, 3213−3223
42
L, Wu H, Lin C, Tan Z, Gao S Z Li . Self-supervised learning on graphs: contrastive, generative, or predictive. IEEE Transactions on Knowledge and Data Engineering, 2023, 35( 4): 4216–4235
43
Y, Wang Y, Xu J, Yang M, Wu X, Li L, Xie Z Chen . Graph-aware contrasting for multivariate time-series classification. In: Proceedings of the 38th AAAI Conference on Artificial Intelligence. 2024, 15725−15734
44
J, Cheng M, Li J, Li F Tsung . Wiener graph deconvolutional network improves graph self-supervised learning. In: Proceedings of the 37th AAAI Conference on Artificial Intelligence. 2023, 7131−7139
45
J, Ji J, Wang C, Huang J, Wu B, Xu Z, Wu J, Zhang Y Zheng . Spatio-temporal self-supervised learning for traffic flow prediction. In: Proceedings of the 37th AAAI Conference on Artificial Intelligence. 2023, 4356−4364
46
L, Xia C, Huang C, Huang K, Lin T, Yu B Kao . Automated self-supervised learning for recommendation. In: Proceedings of the ACM Web Conference 2023. 2023, 992−1002
47
C, Wei J, Liang D, Liu F Wang . Contrastive graph structure learning via information bottleneck for recommendation. In: Proceedings of the 36th International Conference on Neural Information Processing Systems. 2022, 1484
48
Zheng Y, Yi X, Li M, Li R, Shan Z, Chang E, Li T. Forecasting fine-grained air quality based on big data. In: Proceedings of the 21st ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. 2015, 2267−2276
49
X, Yi Y, Zheng J, Zhang T Li . ST-MVL: filling missing values in geo-sensory time series data. In: Proceedings of the 25th International Joint Conference on Artificial Intelligence. 2016, 2704−2710
50
G, Lai W C, Chang Y, Yang H Liu . Modeling long-and short-term temporal patterns with deep neural networks. In: Proceedings of the 41st International ACM SIGIR Conference on Research & Development in Information Retrieval. 2018, 95−104
51
A L, Goldberger L A, Amaral L, Glass J M, Hausdorff P C, Ivanov R G, Mark J E, Mietus G B, Moody C K, Peng H E Stanley . Physiobank, physiotoolkit, and physionet: components of a new research resource for complex physiologic signals. Circulation, 2000, 101( 23): e215–e220
52
M A, Reyna C S, Josef R, Jeter S P, Shashikumar M B, Westover S, Nemati G D, Clifford A Sharma . Early prediction of sepsis from clinical data: the physionet/computing in cardiology challenge 2019. Critical Care Medicine, 2020, 48( 2): 210–217
53
A, Reiss D Stricker . Introducing a new benchmarked dataset for activity monitoring. In: Proceedings of the 16th International Symposium on Wearable Computers. 2012, 108−109
54
M, Shokoohi-Yekta B, Hu H, Jin J, Wang E Keogh . Generalizing DTW to the multi-dimensional case requires an adaptive approach. Data Mining and Knowledge Discovery, 2017, 31( 1): 1–31
55
A, Bagnall H A, Dau J, Lines M, Flynn J, Large A, Bostrom P, Southam E Keogh . The UEA multivariate time series classification archive, 2018. 2018, arXiv preprint arXiv: 1811.00075
56
D J, Hand R J Till . A simple generalisation of the area under the ROC curve for multiple class classification problems. Machine Learning, 2001, 45( 2): 171–186
57
Q, Qi Y, Luo Z, Xu S, Ji T Yang . Stochastic optimization of areas under precision-recall curves with provable convergence. In: Proceedings of the 35th International Conference on Neural Information Processing Systems. 2021, 135
