A new automatic convolutional neural network based on deep reinforcement learning for fault diagnosis

doi:10.1007/s11465-022-0673-7

Front. Mech. Eng.

2022, Vol. 17

Issue (2) : 17 https://doi.org/10.1007/s11465-022-0673-7

RESEARCH ARTICLE

A new automatic convolutional neural network based on deep reinforcement learning for fault diagnosis

Long WEN¹, You WANG¹, Xinyu LI²

¹. School of Mechanical Engineering and Electronic Information, China University of Geosciences, Wuhan 430074, China
². State Key Laboratory of Digital Manufacturing Equipment and Technology, Huazhong University of Science and Technology, Wuhan 430074, China

Download: PDF(2174 KB) HTML
Export: BibTeX | EndNote | Reference Manager | ProCite | RefWorks

Abstract

Convolutional neural network (CNN) has achieved remarkable applications in fault diagnosis. However, the tuning aiming at obtaining the well-trained CNN model is mainly manual search. Tuning requires considerable experiences on the knowledge on CNN training and fault diagnosis, and is always time consuming and labor intensive, making the automatic hyper parameter optimization (HPO) of CNN models essential. To solve this problem, this paper proposes a novel automatic CNN (ACNN) for fault diagnosis, which can automatically tune its three key hyper parameters, namely, learning rate, batch size, and L2-regulation. First, a new deep reinforcement learning (DRL) is developed, and it constructs an agent aiming at controlling these three hyper parameters along with the training of CNN models online. Second, a new structure of DRL is designed by combining deep deterministic policy gradient and long short-term memory, which takes the training loss of CNN models as its input and can output the adjustment on these three hyper parameters. Third, a new training method for ACNN is designed to enhance its stability. Two famous bearing datasets are selected to evaluate the performance of ACNN. It is compared with four commonly used HPO methods, namely, random search, Bayesian optimization, tree Parzen estimator, and sequential model-based algorithm configuration. ACNN is also compared with other published machine learning (ML) and deep learning (DL) methods. The results show that ACNN outperforms these HPO and ML/DL methods, validating its potential in fault diagnosis.

Keywords deep reinforcement learning hyper parameter optimization convolutional neural network fault diagnosis

Corresponding Author(s): Xinyu LI

About author:

Tongcan Cui and Yizhe Hou contributed equally to this work.

Just Accepted Date: 02 April 2022 Issue Date: 27 June 2022

Cite this article:

Long WEN,You WANG,Xinyu LI. A new automatic convolutional neural network based on deep reinforcement learning for fault diagnosis[J]. Front. Mech. Eng., 2022, 17(2): 17.

URL:

https://academic.hep.com.cn/fme/EN/10.1007/s11465-022-0673-7
https://academic.hep.com.cn/fme/EN/Y2022/V17/I2/17

Fig.1 Detail structure of LSTM?DDPG: (a) actor network, (b) critic network.

Tab.1 Hyper parameter range for ACNN

Tab.2 Comparison results of ACNN with HPO methods using DenseNet-201 in Case 1

Tab.3 Comparison results of ACNN with HPO methods using other backbones in Case 1

Fig.2 Online curves of ACNN in Case 1: (a) learning rate curve, (b) batch size curve, (c) L2-regularization curve, and (d) reward curve.

Tab.4 Comparison results of ACNN with ML and DL in Case 1

Tab.5 Comparison results of ACNN with HPO methods using DenseNet-201 in Case 2

Tab.6 Comparison results of ACNN with HPO methods using other backbones in Case 2

Fig.3 Online curves of ACNN in Case 2: (a) learning rate curve, (b) batch size curve, (c) L2-regularization curve, and (d) reward curve.

Tab.7 Comparison results of ACNN with ML and DL in Case 2

Abbreviations
ACNN	Automatic convolutional neural network
ANN	Artificial neural network
AVG	Average prediction accuracy
AWMS-CNN	Adaptive weighted multiscale convolutional neural network
BO	Bayesian optimization
CNN	Convolutional neural network
DAN	Domain adaption network
DDPG	Deep deterministic policy gradient
DL	Deep learning
DRL	Deep reinforcement learning
ELM	Extreme learning machine
FC	Fully connected
GS	Grid search
HCNN	Hierarchical convolutional neural network
HPO	Hyper parameter optimization
ICN	CNN based on a capsule network with an inception block
IF	Inner race fault
LSTM	Long short-term memory
ML	Machine learning
MSCNN	Multiscale convolutional neural network
NAS	Neural architecture search
OF	Outer race fault
RF	Roller fault
RL	Reinforcement learning
RS	Random search
SMAC	Sequential model-based algorithm configuration
SVM	Support vector machine
TPE	Tree Parzen estimator
VI-CNN	CNN based on vibration image
WDCNN	Deep convolutional neural networks with wide first-layer kernels
WMSCCN	Wide convolution and multiscale convolution
Variables
a	Action of DDPG algorithm
a_t	Current action
b_t	Current batch size
b_max, b_min	The upper and lower boundaries for batch size
$E Π$	Expected value under policy $Π$
f	Fourier frequency in short-time Fourier transform
l_t	Current L2-regulation value
l_max, l_min	The upper and lower boundaries for L2-regulation value
lr_max, lr_min	The upper and lower boundaries for learning rate, respectivley
lr_t	Current learning rate
loss_t	Loss value of the CNN model at time step t
L	Training loss of critic network
M	Number of the sequencing training loss
n	Number of samples in the experience storage D
p	Transition probability function
$Q Π$ (s, a)	Q-value function under policy $Π$
r	Reward of DDPG algorithm
s	State of DDPG algorithm
s_t	State at time step t
STFT	Short-time Fourier transform formulation
t	Time step
w(t)	Window function
y_t	Actual Q-value
α	Factor to control the degree of soft updating
γ	Discount factor
$Π$	Policy of the agent to choose the action
$θ μ$	Online actor network
$θ μ ′$	Target actor network
$ω Q$	Online critic network
$ω Q ′$	Target critic network

1	X Zhang , T Huang , B Wu , Y M Hu , S Huang , Q Zhou , X Zhang . Multi-model ensemble deep learning method for intelligent fault diagnosis with high-dimensional samples. Frontiers of Mechanical Engineering, 2021, 16( 2): 340– 352 https://doi.org/10.1007/s11465-021-0629-3
2	X F Chen , S B Wang , B J Qiao , Q Chen . Basic research on machinery fault diagnostics: past, present, and future trends. Frontiers of Mechanical Engineering, 2018, 13( 2): 264– 291 https://doi.org/10.1007/s11465-018-0472-3
3	Y G Lei , B Yang , X W Jiang , F Jia , N P Li , A K Nandi . Applications of machine learning to machine fault diagnosis: a review and roadmap. Mechanical Systems and Signal Processing, 2020, 138 : 106587 https://doi.org/10.1016/j.ymssp.2019.106587
4	A G Nath , S S Udmale , S K Singh . Role of artificial intelligence in rotor fault diagnosis: a comprehensive review. Artificial Intelligence Review, 2021, 54 : 2609– 2668 https://doi.org/10.1007/s10462-020-09910-w
5	J L Wang , C Q Xu , L Dai , J Zhang , R Y Zhong . An unequal deep learning approach for 3-D point cloud segmentation. IEEE Transactions on Industrial Informatics, 2021, 17( 12): 7913– 7922 https://doi.org/10.1109/TII.2020.3044106
6	Z Y Chen , A Mauricio , W H Li , K Gryllias . A deep learning method for bearing fault diagnosis based on cyclic spectral coherence and convolutional neural networks. Mechanical Systems and Signal Processing, 2020, 140 : 106683 https://doi.org/10.1016/j.ymssp.2020.106683
7	D H Wolpert , W G Macready . No free lunch theorems for optimization. IEEE Transactions on Evolutionary Computation, 1997, 1( 1): 67– 82 https://doi.org/10.1109/4235.585893
8	D H Wolpert . The supervised learning no-free-lunch theorems. In: Roy R, Köppen M, Ovaska S, Furuhashi T, Hoffmann F, eds. Soft Computing and Industry. London: Springer, 2002, 25– 42 https://doi.org/10.1007/978-1-4471-0123-9_3
9	F Hutter L Kotthoff J Vanschoren. Automated Machine Learning: Methods, Systems, Challenges. Cham: Springer, 2019
10	L Wen , X Y Li , L Gao . A new reinforcement learning based learning rate scheduler for convolutional neural network in fault classification. IEEE Transactions on Industrial Electronics, 2021, 68( 12): 12890– 12900 https://doi.org/10.1109/TIE.2020.3044808
11	L Wen , X C Ye , L Gao . A new automatic machine learning based hyperparameter optimization for workpiece quality prediction. Measurement and Control, 2020, 53( 7−8): 1088– 1098 https://doi.org/10.1177%2F0020294020932347
12	M Feurer K Eggensperger S Falkner M Lindauer F Hutter. Practical automated machine learning for the AutoML challenge 2018. In: Proceedings of International Workshop on Automatic Machine Learning at ICML. 2018, 1189‒ 1232
13	F X He , T L Liu , D C Tao . Control batch size and learning rate to generalize well: theoretical and empirical evidence. In: Proceedings of the 33rd Conference on Neural Information Processing Systems (NeurIPS). Vancouver, 2019, 1143– 1152
14	Li Y Z, Wei C, Ma T Y. Towards explaining the regularization effect of initial large learning rate in training neural networks. In: Proceedings of the 33rd International Conference on Neural Information Processing Systems. Vancouver, 2019, 11674–11685
15	P Zhou , Z K Peng , S Q Chen , Y Yang , W M Zhang . Non-stationary signal analysis based on general parameterized time-frequency transform and its application in the feature extraction of a rotary machine. Frontiers of Mechanical Engineering, 2018, 13( 2): 292– 300 https://doi.org/10.1007/s11465-017-0443-0
16	J L Wang , C Q Xu , Z L Yang , J Zhang , X O Li . Deformable convolutional networks for efficient mixed-type wafer defect pattern recognition. IEEE Transactions on Semiconductor Manufacturing, 2020, 33( 4): 587– 596 https://doi.org/10.1109/TSM.2020.3020985
17	G W Xu , M Liu , Z F Jiang , W M Shen , C X Huang . Online fault diagnosis method based on transfer convolutional neural networks. IEEE Transactions on Instrumentation and Measurement, 2020, 69( 2): 509– 520 https://doi.org/10.1109/TIM.2019.2902003
18	Li Z X, Zheng T S, Wang Y, Cao Z, Guo Z Q, Fu H Y. A novel method for imbalanced fault diagnosis of rotating machinery based on generative adversarial networks. IEEE Transactions on Instrumentation and Measurement, 2021, 70: 3500417
19	Chen J B, Huang R Y, Zhao K, Wang W, Liu L C, Li W H. Multiscale convolutional neural network with feature alignment for bearing fault diagnosis. IEEE Transactions on Instrumentation and Measurement, 2021, 70: 3517010
20	J Y Jiao , M Zhao , J Lin , K X Liang . A comprehensive review on convolutional neural network in machine fault diagnosis. Neurocomputing, 2020, 417 : 36– 63 https://doi.org/10.1016/j.neucom.2020.07.088
21	Y Yao , S Zhang , S X Yang , G Gui . Learning attention representation with a multi-scale CNN for gear fault diagnosis under different working conditions. Sensors, 2020, 20( 4): 1233 https://doi.org/10.3390/s20041233
22	S Li , H Q Wang , L Y Song , P X Wang , L L Cui , T J Lin . An adaptive data fusion strategy for fault diagnosis based on the convolutional neural network. Measurement, 2020, 165 : 108122 https://doi.org/10.1016/j.measurement.2020.108122
23	D Kolar , D Lisjak , M Pająk , D Pavković . Fault diagnosis of rotary machines using deep convolutional neural network with wide three axis vibration signal input. Sensors, 2020, 20( 14): 4017 https://doi.org/10.3390/s20144017
24	R X Wang , H K Jiang , X Q Li , S W Liu . A reinforcement neural architecture search method for rolling bearing fault diagnosis. Measurement, 2020, 154 : 107417 https://doi.org/10.1016/j.measurement.2019.107417
25	K Y Zhang , J L Chen , S L He , E Y Xu , F D Li , Z T Zhou . Differentiable neural architecture search augmented with pruning and multi-objective optimization for time-efficient intelligent fault diagnosis of machinery. Mechanical Systems and Signal Processing, 2021, 158 : 107773 https://doi.org/10.1016/j.ymssp.2021.107773
26	D Cabrera , A Guamán , S H Zhang , M Cerrada , R V Sánchez , J Cevallos , J Y Long , C Li . Bayesian approach and time series dimensionality reduction to LSTM-based model-building for fault diagnosis of a reciprocating compressor. Neurocomputing, 2020, 380 : 51– 66 https://doi.org/10.1016/j.neucom.2019.11.006
27	L Li , K Jamieson , G DeSalvo , A Rostamizadeh , A Talwalkar . Hyperband: a novel bandit-based approach to hyperparameter optimization. The Journal of Machine Learning Research, 2018, 18( 1): 6765– 6816 https://doi.org/10.48550/arXiv.1603.06560
28	H Li , Q Zhang , X R Qin , Y T Sun . Raw vibration signal pattern recognition with automatic hyper-parameter-optimized convolutional neural network for bearing fault diagnosis. Proceedings of the Institution of Mechanical Engineers, Part C: Journal of Mechanical Engineering Science, 2020, 234( 1): 343– 360 https://doi.org/10.1177/0954406219875756
29	J Y Long , S H Zhang , C Li . Evolving deep echo state networks for intelligent fault diagnosis. IEEE Transactions on Industrial Informatics, 2020, 16( 7): 4928– 4937 https://doi.org/10.1109/TII.2019.2938884
30	J H Han , D J Choi , S U Park , S K Hong . Hyperparameter optimization using a genetic algorithm considering verification time in a convolutional neural network. Journal of Electrical Engineering & Technology, 2020, 15( 2): 721– 726 https://doi.org/10.1007/s42835-020-00343-7
31	J A Wei , H S Huang , L G Yao , Y Hu , Q S Fan , D Huang . New imbalanced fault diagnosis framework based on cluster-MWMOTE and MFO-optimized LS-SVM using limited and complex bearing data. Engineering Applications of Artificial Intelligence, 2020, 96 : 103966 https://doi.org/10.1016/j.engappai.2020.103966
32	S Hansen. Using deep Q-learning to control optimization hyperparameters. 2016, arXiv:1602.04062
33	Z Z Zhang , J L Chen , Z B Chen , W P Li . Asynchronous episodic deep deterministic policy gradient: toward continuous control in computationally complex environments. IEEE Transactions on Cybernetics, 2021, 51( 2): 604– 613 https://doi.org/10.1109/TCYB.2019.2939174
34	Z Y Zhu , G L Peng , Y H Chen , H J Gao . A convolutional neural network based on a capsule network with strong generalization for bearing fault diagnosis. Neurocomputing, 2019, 323 : 62– 75 https://doi.org/10.1016/j.neucom.2018.09.050
35	Y Wang , D J Ning , S L Feng . A novel capsule network based on wide convolution and multi-scale convolution for fault diagnosis. Applied Sciences, 2020, 10( 10): 3659 https://doi.org/10.3390/app10103659
36	L Wen , X Y Li , L Gao . A new two-level hierarchical diagnosis network based on convolutional neural network. IEEE Transactions on Instrumentation and Measurement, 2020, 69( 2): 330– 338 https://doi.org/10.1109/TIM.2019.2896370
37	W Zhang , G L Peng , C H Li , Y H Chen , Z J Zhang . A new deep learning model for fault diagnosis with good anti-noise and domain adaptation ability on raw vibration signals. Sensors, 2017, 17( 2): 425 https://doi.org/10.3390/s17020425
38	D T Hoang , H J Kang . Rolling element bearing fault diagnosis using convolutional neural network and vibration image. Cognitive Systems Research, 2019, 53 : 42– 50 https://doi.org/10.1016/j.cogsys.2018.03.002
39	G Q Jiang , H B He , J Yan , P Xie . Multiscale convolutional neural networks for fault diagnosis of wind turbine gearbox. IEEE Transactions on Industrial Electronics, 2019, 66( 4): 3196– 3207 https://doi.org/10.1109/TIE.2018.2844805
40	H H Qiao , T Y Wang , P Wang , L Zhang , M D Xu . An adaptive weighted multiscale convolutional neural network for rotating machinery fault diagnosis under variable operating conditions. IEEE Access: Practical Innovations, Open Solutions, 2019, 7 : 118954– 118964 https://doi.org/10.1109/ACCESS.2019.2936625
41	Y Song , Y B Li , L Jia , M K Qiu . Retraining strategy-based domain adaption network for intelligent fault diagnosis. IEEE Transactions on Industrial Informatics, 2020, 16( 9): 6163– 6171 https://doi.org/10.1109/TII.2019.2950667

[1]	Xiaoluo YU, Yifan HUANGFU, Yang YANG, Minggang DU, Qingbo HE, Zhike PENG. Gear fault diagnosis using gear meshing stiffness identified by gearbox housing vibration signals[J]. Front. Mech. Eng., 2022, 17(4): 57-.
[2]	Qun CHAO, Haohan GAO, Jianfeng TAO, Chengliang LIU, Yuanhang WANG, Jian ZHOU. Fault diagnosis of axial piston pumps with multi-sensor data and convolutional neural network[J]. Front. Mech. Eng., 2022, 17(3): 36-.
[3]	Yuan YIN, Weifeng HUANG, Decai LI, Qiang HE, Xiangfeng LIU, Ying LIU. Method for solving the nonlinear inverse problem in gas face seal diagnosis based on surrogate models[J]. Front. Mech. Eng., 2022, 17(3): 33-.
[4]	Long BAI, Fei XU, Xiao CHEN, Xin SU, Fuyao LAI, Jianfeng XU. A hybrid deep learning model for robust prediction of the dimensional accuracy in precision milling of thin-walled structural components[J]. Front. Mech. Eng., 2022, 17(3): 32-.
[5]	Jie LIU, Kaibo ZHOU, Chaoying YANG, Guoliang LU. Imbalanced fault diagnosis of rotating machinery using autoencoder-based SuperGraph feature learning[J]. Front. Mech. Eng., 2021, 16(4): 829-839.
[6]	Xu WANG, Hongyang GU, Tianyang WANG, Wei ZHANG, Aihua LI, Fulei CHU. Deep convolutional tree-inspired network: a decision-tree-structured neural network for hierarchical fault diagnosis of bearings[J]. Front. Mech. Eng., 2021, 16(4): 814-828.
[7]	Xin ZHANG, Tao HUANG, Bo WU, Youmin HU, Shuai HUANG, Quan ZHOU, Xi ZHANG. Multi-model ensemble deep learning method for intelligent fault diagnosis with high-dimensional samples[J]. Front. Mech. Eng., 2021, 16(2): 340-352.
[8]	Liang XUE, Jie LIU, Guilin WEN, Hongxin WANG. Efficient, high-resolution topology optimization method based on convolutional neural networks[J]. Front. Mech. Eng., 2021, 16(1): 80-96.
[9]	Peng ZHOU, Zhike PENG, Shiqian CHEN, Yang YANG, Wenming ZHANG. Non-stationary signal analysis based on general parameterized time--frequency transform and its application in the feature extraction of a rotary machine[J]. Front. Mech. Eng., 2018, 13(2): 292-300.
[10]	Xuefeng CHEN, Shibin WANG, Baijie QIAO, Qiang CHEN. Basic research on machinery fault diagnostics: Past, present, and future trends[J]. Front. Mech. Eng., 2018, 13(2): 264-291.
[11]	Shoudao HUANG, Xuan WU, Xiao LIU, Jian GAO, Yunze HE. Overview of condition monitoring and operation control of electric power conversion systems in direct-drive wind turbines under faults[J]. Front. Mech. Eng., 2017, 12(3): 281-302.
[12]	Diego CABRERA,Fernando SANCHO,René-Vinicio SÁNCHEZ,Grover ZURITA,Mariela CERRADA,Chuan LI,Rafael E. VÁSQUEZ. Fault diagnosis of spur gearbox based on random forest and wavelet packet decomposition[J]. Front. Mech. Eng., 2015, 10(3): 277-286.
[13]	Houjun SU, Tielin SHI, Fei CHEN, Shuhong HUANG. New method of fault diagnosis of rotating machinery based on distance of information entropy[J]. Front Mech Eng, 2011, 6(2): 249-253.
[14]	Lixin GAO, Lijuan WU, Yan WANG, Houpei WEI, Hui YE. Intelligent fault diagnostic system based on RBR for the gearbox of rolling mills[J]. Front Mech Eng Chin, 2010, 5(4): 483-490.
[15]	Shaohong WANG, Tao CHEN, Jianghong SUN. Design and realization of a remote monitoring and diagnosis and prediction system for large rotating machinery[J]. Front Mech Eng Chin, 2010, 5(2): 165-170.

Viewed

Full text

Abstract

Cited

Shared

Discussed