Joint fuzzy background and adaptive foreground model for moving target detection

doi:10.1007/s11704-022-2099-0

Front. Comput. Sci.

2024, Vol. 18

Issue (2) : 182306 https://doi.org/10.1007/s11704-022-2099-0

Artificial Intelligence

Joint fuzzy background and adaptive foreground model for moving target detection

Dawei ZHANG¹, Peng WANG¹, Yongfeng DONG¹, Linhao LI¹(

), Xin LI²

¹. School of Artificial Intelligence, Hebei University of Technology, Tianjin 300401, China
². Lane Department of Computer Science and Electrical Engineering, West Virginia University, Morgantown, WV 26506-6109, USA

Download: PDF(12779 KB) HTML
Export: BibTeX | EndNote | Reference Manager | ProCite | RefWorks

Abstract

Moving target detection is one of the most basic tasks in computer vision. In conventional wisdom, the problem is solved by iterative optimization under either Matrix Decomposition (MD) or Matrix Factorization (MF) framework. MD utilizes foreground information to facilitate background recovery. MF uses noise-based weights to fine-tune the background. So both noise and foreground information contribute to the recovery of the background. To jointly exploit their advantages, inspired by two framework complementary characteristics, we propose to simultaneously exploit the advantages of these two optimizing approaches in a unified framework called Joint Matrix Decomposition and Factorization (JMDF). To improve background extraction, a fuzzy factorization is designed. The fuzzy membership of the background/foreground association is calculated during the factorization process to distinguish their contributions of both to background estimation. To describe the spatio-temporal continuity of foreground more accurately, we propose to incorporate the first order temporal difference into the group sparsity constraint adaptively. The temporal constraint is adjusted adaptively. Both foreground and the background are jointly estimated through an effective alternate optimization process, and the noise can be modeled with the specific probability distribution. The experimental results of vast real videos illustrate the effectiveness of our method. Compared with the current state-of-the-art technology, our method can usually form the clearer background and extract the more accurate foreground. Anti-noise experiments show the noise robustness of our method.

Keywords matrix decomposition matrix factorization generalized sparsity noise modeling

Corresponding Author(s): Linhao LI

Just Accepted Date: 19 December 2022 Issue Date: 30 March 2023

Cite this article:

Dawei ZHANG,Peng WANG,Yongfeng DONG, et al. Joint fuzzy background and adaptive foreground model for moving target detection[J]. Front. Comput. Sci., 2024, 18(2): 182306.

URL:

https://academic.hep.com.cn/fcs/EN/10.1007/s11704-022-2099-0
https://academic.hep.com.cn/fcs/EN/Y2024/V18/I2/182306

Fig.1 Comparison of the proposed combined methodology and two conventional frameworks. (a) Matrix decomposition (MD) way; (b) matrix factorization (MF) way; (c) the proposed JMDF: we simultaneously take FG model, BG model, and noise model into account by extending the fuzzy membership degree

Fig.2 Flowchart of the proposed framework. First, the video BG is modeled by fuzzy factorization. Second, background subtraction is conducted, and then the Spatio-temporal constraints are applied to obtain the FG component. After that, the Gaussian noise in residual and the FG component update the membership degree together. Finally, the above process is iterated until convergence

Fig.3 Background membership degree in iterative process. The leftmost image is the original video frame. For else images, from left to right, there are the FG masks and background membership heat maps in the fifth, seventh and ninth iterations

Fig.4 On the left, the matrix represents the image to be processed, where the label is the index of each pixel. The sub-graph on the right top shows the conventional neighborhood of pixels. The right bottom illustrates some potential group neighborhood patterns located on the different positions of the image under different circular neighborhood radius

Fig.5

λ

value in three types of scenes. (a) Weak dynamic BG; (b) strong interference; (c) static BG

Tab.1 Comparison of foreground detection results on i2r dataset

Fig.6 Foreground detection results on the I2R dataset. From left to right, the video sequences selected in turn are “WaterSurface”, “Fountain”, “Curtain”, “Campus”, “Escalator”, “Hall”, “Bootstrap”,“ShopMall” and “Lobby”

Sequence	Video	PCP [9]	DECOLOR [15]	E-LSD [43]	ROUTE [12]	OMoGMF+TV [6]	GSTO [62]	JMDF01	JMDF02	JMDF03
Baseline	highway	0.791	0.881	0.963	0.752	0.935	0.937	0.982	0.992	0.979
	office	0.643	0.843	0.924	0.882	0.845	0.906	0.876	0.886	0.903
	pedestrians	0.952	0.701	0.990	0.953	0.982	0.943	0.987	0.994	0.984
	PETS2006	0.690	0.908	0.812	0.675	0.806	0.842	0.861	0.873	0.833
	Average	0.769	0.833	0.922	0.816	0.892	0.907	0.927	0.936	0.925
BadWeather	skating	0.652	0.927	0.921	0.832	0.901	0.945	0.961	0.963	0.953
	snowFall	0.626	0.908	0.795	0.742	0.813	0.846	0.935	0.940	0.932
	blizzard	0.883	0.851	0.873	0.902	0.867	0.918	0.942	0.942	0.941
	wetSnow	0.608	0.904	0.861	0.632	0.822	0.861	0.886	0.884	0.880
	Average	0.692	0.898	0.863	0.777	0.851	0.893	0.931	0.932	0.927
LowFramerate	port_0_17fps	0.062	0.041	0.454	0.052	0.382	0.251	0.314	0.394	0.442
	tramCrossroad_1fps	0.756	0.782	0.864	0.761	0.854	0.832	0.844	0.843	0.842
	tunnelExit_0_35fps	0.583	0.678	0.622	0.545	0.559	0.742	0.783	0.761	0.734
	turnpike_0_5fps	0.752	0.721	0.897	0.752	0.883	0.798	0.895	0.901	0.903
	Average	0.541	0.556	0.709	0.528	0.670	0.656	0.709	0.725	0.730
Turbulence	turbulence0	0.692	0.344	0.754	0.632	0.728	0.581	0.826	0.882	0.883
	turbulence1	0.382	0.421	0.693	0.543	0.798	0.715	0.859	0.870	0.852
	turbulence2	0.043	0.472	0.987	0.512	0.993	0.871	0.982	0.984	0.973
	turbulence3	0.845	0.702	0.936	0.859	0.926	0.860	0.941	0.954	0.960
	Average	0.491	0.482	0.843	0.637	0.861	0.757	0.902	0.923	0.917
Thermal	corridor	0.402	0.973	0.881	0.771	0.962	0.923	0.991	0.993	0.992
	diningRoom	0.464	0.911	0.724	0.616	0.608	0.905	0.772	0.793	0.784
	lakeSide	0.673	0.722	0.435	0.684	0.322	0.776	0.680	0.662	0.604
	library	0.467	0.536	0.971	0.885	0.983	0.952	0.985	0.991	0.989
	park	0.618	0.814	0.727	0.634	0.617	0.859	0.856	0.851	0.853
	Average	0.525	0.791	0.748	0.718	0.698	0.883	0.857	0.858	0.844
IntermittentObjectMotion	abandonedBox	0.710	0.721	0.932	0.823	0.924	0.906	0.904	0.904	0.882
	parking	0.622	0.211	0.382	0.383	0.274	0.801	0.768	0.767	0.624
	sofa	0.634	0.732	0.703	0.632	0.690	0.693	0.691	0.698	0.696
	streetLight	0.421	0.643	0.614	0.440	0.591	0.633	0.631	0.653	0.614
	tramstop	0.242	0.350	0.352	0.243	0.336	0.376	0.347	0.364	0.353
	winterDriveway	0.371	0.808	0.766	0.773	0.646	0.795	0.784	0.785	0.784
	Average	0.500	0.578	0.625	0.549	0.577	0.701	0.688	0.695	0.659
DynamicBackground	boats	0.426	0.903	0.931	0.463	0.907	0.905	0.902	0.916	0.954
	canoe	0.121	0.264	0.822	0.434	0.801	0.778	0.887	0.933	0.923
	fall	0.445	0.707	0.701	0.363	0.566	0.828	0.679	0.764	0.842
	fountain01	0.041	0.024	0.081	0.053	0.042	0.184	0.171	0.220	0.171
	fountain02	0.722	0.726	0.727	0.744	0.805	0.824	0.854	0.832	0.784
	overpass	0.492	0.845	0.793	0.711	0.802	0.872	0.858	0.875	0.871
	Average	0.375	0.578	0.676	0.461	0.654	0.732	0.725	0.762	0.758
Overall average		0.556	0.674	0.769	0.641	0.743	0.790	0.820	0.833	0.823

Tab.2 Comparison of foreground detection results on cdnet2014 dataset

Fig.7 Foreground detection results on the CDnet2014 dataset. From left to right, the video sequences selected in turn are “baseline”, “badWeather”, “lowFramerate”, “turbulence”, “thermal”, “intermittentObjectMotion”, and “dynamicBackground”

Fig.8 Robustness to different noises. The source video frame and the result from our JMDF are displayed in the lower right corner of the figure. We add Gaussian noise, speckle noise, salt and pepper noise, and Poisson noise to the source video, respectively. Except for Poisson noise, we fix the noise mean to 0, and change the noise variance, whose values are shown in the first row of each sub-figure. Then, we plot the noisy frame examples in the second row and the results from JMDF in the third row. As the variance increases, the video gradually becomes blurred, which increases the difficulty of the detection task. Our method achieves good performance even in videos with large noise variance

1	B, Garcia-Garcia T, Bouwmans A J R Silva . Background subtraction in real applications: challenges, current models and future directions. Computer Science Review, 2020, 35: 100204
2	C, Stauffer W E L Grimson . Adaptive background mixture models for real-time tracking. In: Proceedings of 1999 IEEE Computer Society Conference on Computer Vision and Pattern Recognition. 1999, 246−252
3	D, Weinland E Boyer . Action recognition using exemplar-based embedding. In: Proceedings of 2008 IEEE Conference on Computer Vision and Pattern Recognition. 2008, 1−7
4	Z, Lin H Zhang . Low-Rank Models in Visual Analysis: Theories, Algorithms, and Applications. Sea Harbor Drive Orlando: Academic Press, 2017
5	N, Vaswani T, Bouwmans S, Javed P Narayanamurthy . Robust subspace learning: robust PCA, robust subspace tracking, and robust subspace recovery. IEEE Signal Processing Magazine, 2018, 35( 4): 32–55
6	H, Yong D, Meng W, Zuo L Zhang . Robust online matrix factorization for dynamic background subtraction. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2018, 40( 7): 1726–1740
7	Z Zivkovic . Improved adaptive Gaussian mixture model for background subtraction. In: Proceedings of the 17th International Conference on Pattern Recognition. 2004, 28−31
8	A, Elgammal D, Harwood L Davis . Non-parametric model for background subtraction. In: Proceedings of the 6th European Conference on Computer Vision. 2000, 751−767
9	Candès E J, J X, Li Y, Ma J Wright . Robust principal component analysis?. Journal of the ACM, 2011, 58( 3): 11
10	J, Chen J Yang . Robust subspace segmentation via low-rank representation. IEEE Transactions on Cybernetics, 2014, 44( 8): 1432–1445
11	X, Liu G, Zhao J, Yao C Qi . Background subtraction based on low-rank and structured sparse decomposition. IEEE Transactions on Image Processing, 2015, 24( 8): 2502–2514
12	X, Guo Z Lin . Low-rank matrix recovery via robust outlier estimation. IEEE Transactions on Image Processing, 2018, 27( 11): 5316–5327
13	S, Javed A, Mahmood S, Al-Maadeed T, Bouwmans S K Jung . Moving object detection in complex scene using spatiotemporal structured-sparse RPCA. IEEE Transactions on Image Processing, 2019, 28( 2): 1007–1022
14	S, Boyd N, Parikh E, Chu B, Peleato J Eckstein . Distributed optimization and statistical learning via the alternating direction method of multipliers. Foundations and Trends® in Machine Learning, 2011, 3( 1): 1–122
15	X, Zhou C, Yang W Yu . Moving object detection by detecting contiguous outliers in the low-rank representation. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2013, 35( 3): 597–610
16	X, Cao L, Yang X Guo . Total variation regularized RPCA for irregularly moving object detection under dynamic background. IEEE Transactions on Cybernetics, 2016, 46( 4): 1014–1027
17	H B, Xie C, Li R Y D, Xu K Mengersen . Robust kernelized Bayesian matrix factorization for video background/foreground separation. In: Proceedings of the 5th International Conference on Machine Learning, Optimization, and Data Science. 2019, 484−495
18	D, Meng La Torre F De . Robust matrix factorization with unknown noise. In: Proceedings of 2013 IEEE International Conference on Computer Vision. 2013, 1337−1344
19	X, Cao Y, Chen Q, Zhao D, Meng Y, Wang D, Wang Z Xu . Low-rank matrix factorization under general mixture noise distributions. In: Proceedings of 2015 IEEE International Conference on Computer Vision. 2015, 1493−1501
20	Y, Chu X, Wu T, Liu J Liu . A basis-background subtraction method using non-negative matrix factorization. In: Proceedings of SPIE 7546, Second International Conference on Digital Image Processing. 2010, 75461A
21	L, Li Q, Hu X Li . Moving object detection in video via hierarchical modeling and alternating optimization. IEEE Transactions on Image Processing, 2019, 28( 4): 2021–2036
22	L, Li W, Huang I Y H, Gu Q Tian . Statistical modeling of complex backgrounds for foreground object detection. IEEE Transactions on Image Processing, 2004, 13( 11): 1459–1472
23	S, Brutzer B, Höferlin G Heidemann . Evaluation of background subtraction techniques for video surveillance. In: Proceedings of CVPR 2011. 2011, 1937−1944
24	T Bouwmans . Recent advanced statistical background modeling for foreground detection - a systematic survey. Recent Patents on Computer Science, 2011, 4( 3): 147–176
25	W, Nebili B, Farou H Seridi . Using resources competition and memory cell development to select the best GMM for background subtraction. International Journal of Strategic Information Technology and Applications, 2019, 10( 2): 21–43
26	M, Chen X, Wei Q, Yang Q, Li G, Wang M H Yang . Spatiotemporal GMM for background subtraction with superpixel hierarchy. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2018, 40( 6): 1518–1525
27	T, Elguebaly N Bouguila . Background subtraction using finite mixtures of asymmetric Gaussian distributions and shadow detection. Machine Vision and Applications, 2014, 25( 5): 1145–1162
28	Y, Yang D, Han J, Ding Y Yang . An improved ViBe for video moving object detection based on evidential reasoning. In: Proceedings of 2016 IEEE International Conference on Multisensor Fusion and Integration for Intelligent Systems. 2016, 26−31
29	G, Ramírez-Alonso M I Chacón-Murguía . Auto-adaptive parallel SOM architecture with a modular analysis for dynamic object segmentation in videos. Neurocomputing, 2016, 175: 990–1000
30	H C, Wang Y C, Lai W H, Cheng C Y, Cheng K L Hua . Background extraction based on joint gaussian conditional random fields. IEEE Transactions on Circuits and Systems for Video Technology, 2018, 28( 11): 3127–3140
31	Y, Xie S, Gu Y, Liu W, Zuo W, Zhang L Zhang . Weighted schatten p-norm minimization for image denoising and background subtraction. IEEE Transactions on Image Processing, 2016, 25( 10): 4842–4857
32	W, Hu Y, Yang W, Zhang Y Xie . Moving object detection using tensor-based low-rank and saliently fused-sparse decomposition. IEEE Transactions on Image Processing, 2017, 26( 2): 724–737
33	L, Yin A, Parekh I Selesnick . Stable principal component pursuit via convex analysis. IEEE Transactions on Signal Processing, 2019, 67( 10): 2595–2607
34	J, Feng H, Xu S Yan . Online robust PCA via stochastic optimization. In: Proceedings of the 26th International Conference on Neural Information Processing Systems. 2013, 404−412
35	K, Guo L, Liu X, Xu D, Xu D Tao . GoDec+: fast and robust low-rank matrix decomposition based on maximum correntropy. IEEE Transactions on Neural Networks and Learning Systems, 2018, 29( 6): 2323–2336
36	H, Cao X, Shang Y, Wang M, Song S, Chen C I Chang . GO decomposition (GoDec) approach to finding low rank and sparsity matrices for hyperspectral target detection. In: Proceedings of 2020 IEEE International Geoscience and Remote Sensing Symposium. 2020, 2807−2810
37	M, Shakeri H Zhang . COROLA: a sequential solution to moving object detection using low-rank approximation. Computer Vision and Image Understanding, 2016, 146: 27–39
38	S, Javed S H, Oh A, Sobral T, Bouwmans S K Jung . OR-PCA with MRF for robust foreground detection in highly dynamic backgrounds. In: Proceedings of the 12th Asian Conference on Computer Vision. 2014, 284−299
39	S, Javed S H, Oh T, Bouwmans S K Jung . Robust background subtraction to global illumination changes via multiple features-based online robust principal components analysis with Markov random field. Journal of Electronic Imaging, 2015, 24( 4): 043011
40	C, Li X, Wang L, Zhang J, Tang H, Wu L Lin . Weighted low-rank decomposition for robust grayscale-thermal foreground detection. IEEE Transactions on Circuits and Systems for Video Technology, 2017, 27( 4): 725–738
41	L, Zhu Y, Hao Y Song . L1/2 norm and spatial continuity regularized low-rank approximation for moving object detection in dynamic background. IEEE Signal Processing Letters, 2018, 25( 1): 15–19
42	Y, Xu Z, Wu J, Chanussot M D, Mura A L, Bertozzi Z Wei . Low-rank decomposition and total variation regularization of hyperspectral video sequences. IEEE Transactions on Geoscience and Remote Sensing, 2018, 56( 3): 1680–1694
43	J, Zhang X, Jia J Hu . Error bounded foreground and background modeling for moving object detection in satellite videos. IEEE Transactions on Geoscience and Remote Sensing, 2020, 58( 4): 2659–2669
44	M, Wu Y, Sun R, Hang Q, Liu G Liu . Multi-component group sparse RPCA model for motion object detection under complex dynamic background. Neurocomputing, 2018, 314: 120–131
45	X, Liu G Zhao . Background subtraction using multi-channel fused lasso. Electronic Imaging, 2019, 2019( 11): 269
46	B, Xin Y, Tian Y, Wang W Gao . Background subtraction via generalized fused lasso foreground modeling. In: Proceedings of 2015 IEEE conference on Computer Vision and Pattern Recognition. 2015, 4676−4684
47	S, Javed S H, Oh A, Sobral T, Bouwmans S K Jung . Background subtraction via superpixel-based online matrix decomposition with structured foreground constraints. In: Proceedings of 2015 IEEE International Conference on Computer Vision Workshop. 2015, 930−938
48	X, Ye J, Yang X, Sun K, Li C, Hou Y Wang . Foreground-background separation from video clips via motion-assisted matrix restoration. IEEE Transactions on Circuits and Systems for Video Technology, 2015, 25( 11): 1721–1734
49	X, Cao Q, Zhao D, Meng Y, Chen Z Xu . Robust low-rank matrix factorization under general mixture noise distributions. IEEE Transactions on Image Processing, 2016, 25( 10): 4677–4690
50	Q, Liu X Li . Efficient low-rank matrix factorization based on ℓ1,ε-norm for online background subtraction. IEEE Transactions on Circuits and Systems for Video Technology, 2022, 32( 7): 4900–4904
51	Z, Zhu Y, Meng D, Kong X, Zhang Y, Guo Y Zhao . To see in the dark: N2DGAN for background modeling in nighttime scene. IEEE Transactions on Circuits and Systems for Video Technology, 2021, 31( 2): 492–502
52	P W, Patil S Murala . MSFgNet: a novel compact end-to-end deep network for moving object detection. IEEE Transactions on Intelligent Transportation Systems, 2019, 20( 11): 4066–4077
53	C, Zhao A Basu . Dynamic deep pixel distribution learning for background subtraction. IEEE Transactions on Circuits and Systems for Video Technology, 2020, 30( 11): 4192–4206
54	B, Hou Y, Liu N Ling . A super-fast deep network for moving object detection. In: Proceedings of 2020 IEEE International Symposium on Circuits and Systems. 2020, 1−5
55	P W, Patil S, Murala A, Dhall S Chaudhary . MsEDNet: multi-scale deep saliency learning for moving object detection. In: Proceedings of 2018 IEEE International Conference on Systems, Man, and Cybernetics. 2018, 1670−1675
56	L F, Yan X Y Tu . Background modeling based on chebyshev approximation. Journal of System Simulation, 2008, 20(4): 944−946, 1001
57	A, Stagliano N, Noceti A, Verri F Odone . Online space-variant background modeling with sparse coding. IEEE Transactions on Image Processing, 2015, 24( 8): 2415–2428
58	S, Messelodi M C, Modena N, Segata M Zanin . A Kalman filter based background updating algorithm robust to sharp illumination changes. In: Proceedings of the 13th International Conference on Image Analysis and Processing. 2005, 163−170
59	J H, Giraldo S, Javed M, Sultana S K, Jung T Bouwmans . The emerging field of graph signal processing for moving object segmentation. In: Proceedings of the 27th International Workshop on Frontiers of Computer Vision. 2021, 31−45
60	G Pólya . Isoperimetric Inequalities in Mathematical Physics. Princeton: Princeton University Press, 1951
61	A M Tekalp . Digital Video Processing. 2nd ed. Upper Saddle River: Prentice Hall Press, 2015
62	L, Li Z, Wang Q, Hu Y Dong . Adaptive nonconvex sparsity based background subtraction for intelligent video surveillance. IEEE Transactions on Industrial Informatics, 2021, 17( 6): 4168–4178
63	S, Javed P, Narayanamurthy T, Bouwmans N Vaswani . Robust PCA and robust subspace tracking: a comparative evaluation. In: Proceedings of 2018 IEEE Statistical Signal Processing Workshop. 2018, 836−840
64	T, Bouwmans N S, Aybat E H Zahzah . Handbook of Robust Low-Rank and Sparse Matrix Decomposition: Applications in Image and Video Processing. Boca Raton: CRC Press, 2016

[1]

FCS-22099-OF-DZ_suppl_1

Download

[1]	Lele HUANG, Huifang MA, Xiangchun HE, Liang CHANG. Multi-affect(ed): improving recommendation with similarity-enhanced user reliability and influence propagation[J]. Front. Comput. Sci., 2021, 15(5): 155331-.
[2]	Yan-Ping SUN, Min-Ling ZHANG. Compositional metric learning for multi-label classification[J]. Front. Comput. Sci., 2021, 15(5): 155320-.
[3]	Ning LIU, Zhongpai GAO, Jia WANG, Guangtao ZHAI. Psycho-visual modulation based information display: introduction and survey[J]. Front. Comput. Sci., 2021, 15(3): 153703-.
[4]	Tao LIAN, Lin DU, Mingfu ZHAO, Chaoran CUI, Zhumin CHEN, Jun MA. Evaluating and improving the interpretability of item embeddings using item-tag relevance information[J]. Front. Comput. Sci., 2020, 14(3): 143603-.
[5]	Ming HE, Hao GUO, Guangyi LV, Le WU, Yong GE, Enhong CHEN, Haiping MA. Leveraging proficiency and preference for online Karaoke recommendation[J]. Front. Comput. Sci., 2020, 14(2): 273-290.
[6]	Liang SUN, Hongwei GE, Wenjing KANG. Non-negative matrix factorization based modeling and training algorithm for multi-label learning[J]. Front. Comput. Sci., 2019, 13(6): 1243-1254.
[7]	Dakun LIU,Xiaoyang TAN. Max-margin non-negative matrix factorization with flexible spatial constraints based on factor analysis[J]. Front. Comput. Sci., 2016, 10(2): 302-316.
[8]	Richong ZHANG,Han BAO,Hailong SUN,Yanghao WANG,Xudong LIU. Recommender systems based on ranking performance optimization[J]. Front. Comput. Sci., 2016, 10(2): 270-280.
[9]	Jiliang TANG, Xufei WANG, Huiji GAO, Xia HU, Huan LIU. Enriching short text representation in microblog for clustering[J]. Front Comput Sci, 2012, 6(1): 88-101.

Viewed

Full text

Abstract

Cited

Shared

Discussed