Comparative analysis of twelve transfer learning models for the prediction and crack detection in concrete dams, based on borehole images
Umer Sadiq KHAN1,2, Muhammad ISHFAQUE3(), Saif Ur Rehman KHAN4, Fang Xu1,2, Lerui CHEN5, Yi LEI6
. School of Computer and Information Science, Hubei Engineering University, Xiaogan 432000, China . Institute for AI Industrial Technology Research, Hubei Engineering University, Xiaogan 432000, China . College of Water Conservancy and Hydropower Engineering, Hohai University, Nanjing 210098, China . School of Computer Science and Engineering, Central South University, Changsha 410083, China . College of Aviation, Zhongyuan University of Technology, Zhengzhou 451191, China . School of Civil Engineering, Central South University, Changsha 410083, China
Disaster-resilient dams require accurate crack detection, but machine learning methods cannot capture dam structural reaction temporal patterns and dependencies. This research uses deep learning, convolutional neural networks, and transfer learning to improve dam crack detection. Twelve deep-learning models are trained on 192 crack images. This research aims to provide up-to-date detecting techniques to solve dam crack problems. The finding shows that the EfficientNetB0 model performed better than others in classifying borehole concrete crack surface tiles and normal (undamaged) surface tiles with 91% accuracy. The study’s pre-trained designs help to identify and to determine the specific locations of cracks.
Just Accepted Date: 09 July 2024Online First Date: 24 September 2024Issue Date: 29 October 2024
Cite this article:
Umer Sadiq KHAN,Muhammad ISHFAQUE,Saif Ur Rehman KHAN, et al. Comparative analysis of twelve transfer learning models for the prediction and crack detection in concrete dams, based on borehole images[J]. Front. Struct. Civ. Eng.,
2024, 18(10): 1507-1523.
Tab.1 Pillow Dam Borehole CCTV images Data set summary
Fig.2 The Borehole subsurface CCTV imaging tool for investigation inside the Pillow Dam gallery.
Fig.3 An example data set obtained from borehole CCTV imaging, utilized for pre-processing purposes, including categories such as normal and borehole concrete wall crack sections.
Dataset name
Borehole image
Section count
Tiles
Borehole-CCTV image dataset of pillow dam, China
normal (undamage) image
117
1835
crack image
75
1763
total = 2 category image dataset
192
3598
Tab.2 Data set preprocessing summary
Fig.4 Show the proposed methodology architecture for CCTV borehole images for crack predictions.
Fig.5 Confusion matrix generated through 12 state-of-the-art models for the undamaged surface vs predicted crack. (a) Dense121; (b) DenseNet169; (c) DenseNet201; (d) EfficientNetB0; (e) EfficientNetB2; (f) InceptionV3; (g) InceptionResNetV2; (h) MobileNet; (i) ResNet101; (j) VGG16; (k) VGG19; (l) Xception.
Fig.6 Visual representation of undamaged surface vs. predicted subsurface crack on the 12 state-of-the-art models: (a) DenseNet169; (b) DenseNet121; (c) DenseNet201; (d) EfficientNetB0; (e) EfficientNetB2; (f) InceptionV3; (g) InceptionResNetV2; (h) MobileNet; (i) ResNet101; (j) VGG16; (k) VGG19; (l) Xception.
Fig.7 Comparison results based on precision, recall, F1 score, and accuracy of the 12 DL models.
Fig.8 Training average accuracy comparison between models.
Fig.9 Training accuracy comparison between models.
Fig.10 Training loss comparison between models.
Fig.11 Training average loss comparison between models.
1
Y Li, T Bao, B Xu, X Shu, Y Zhou, Y Du, R Wang, K Zhang. A deep residual neural network framework with transfer learning for concrete dams patch-level crack classification and weakly-supervised localization. Measurement, 2022, 188: 110641 https://doi.org/10.1016/j.measurement.2021.110641
2
M Mulligan, Soesbergen A van, L Sáenz. GOODD, a global dataset of more than 38,000 georeferenced dams. Scientific Data, 2020, 7(1): 31 https://doi.org/10.1038/s41597-020-0362-5
3
Technical Committee EmbankmentsDamsSlopes. Remote Sensing for Monitoring Embankments, Dams, and Slopes: Recent Advances. Reston, VA: American Society of Civil Engineers, 2021
4
Y Li, T Bao, X Shu, Z Chen, Z Gao, K Zhang. A hybrid model integrating principal component analysis, fuzzy C-means, and Gaussian process regression for dam deformation prediction. Arabian Journal for Science and Engineering, 2021, 46(5): 4293–4306 https://doi.org/10.1007/s13369-020-04923-7
5
D Feng, M Q Feng. Computer vision for SHM of civil infrastructure: From dynamic response measurement to damage detection—A review. Engineering Structures, 2018, 156: 105–117 https://doi.org/10.1016/j.engstruct.2017.11.018
6
H Kim, E Ahn, M Shin, S H Sim. Crack and noncrack classification from concrete surface images using machine learning. Structural Health Monitoring, 2019, 18(3): 725–738 https://doi.org/10.1177/1475921718768747
7
B Chen, H Zhang, G Wang, J Huo, Y Li, L Li. Automatic concrete infrastructure crack semantic segmentation using deep learning. Automation in Construction, 2023, 152: 104950 https://doi.org/10.1016/j.autcon.2023.104950
8
C Z Dong, F N Catbas. A review of computer vision-based structural health monitoring at local and global levels. Structural Health Monitoring, 2021, 20(2): 692–743 https://doi.org/10.1177/1475921720935585
9
Q Meng, H Xue, H Song, X Zhuang, T Rabczuk. Rigid-block DEM modeling of mesoscale fracture behavior of concrete with random aggregates. Journal of Engineering Mechanics, 2023, 149(2): 04022114 https://doi.org/10.1061/JENMDT.EMENG-6784
10
M Ishfaque, Q Dai, N Haq, K Jadoon, S M Shahzad, H T Janjuhah. Use of recurrent neural network with long short-term memory for seepage prediction at Tarbela Dam, KP, Pakistan. Energies, 2022, 15(9): 3123 https://doi.org/10.3390/en15093123
11
L M Mauludin, B A Budiman, S P Santosa, X Zhuang, T Rabczuk. Numerical modeling of microcrack behavior in encapsulation-based self-healing concrete under uniaxial tension. Journal of Mechanical Science and Technology, 2020, 34(5): 1847–1853 https://doi.org/10.1007/s12206-020-0405-z
12
L M Mauludin, X Zhuang, T Rabczuk. Computational modeling of fracture in encapsulation-based self-healing concrete using cohesive elements. Composite Structures, 2018, 196: 63–75 https://doi.org/10.1016/j.compstruct.2018.04.066
13
M S Quayum, X Zhuang, T Rabczuk. Computational model generation and RVE design of self-healing concrete. Frontiers of Structural and Civil Engineering, 2015, 9(4): 383–396 https://doi.org/10.1007/s11709-015-0320-z
14
H Zhu, X Wu, Y Luo, Y Jia, C Wang, Z Fang, X Zhuang, S Zhou. Prediction of early compressive strength of ultrahigh-performance concrete using machine learning methods. International Journal of Computational Methods, 2023, 20(8): 2141023 https://doi.org/10.1142/S0219876221410231
15
J S Chen, K Y Lin, S Y Young. Effects of crack width and permeability on moisture-induced damage of pavements. Journal of Materials in Civil Engineering, 2004, 16(3): 276–282 https://doi.org/10.1061/(ASCE)0899-1561(2004)16:3(276
16
S U R Khan, M Zhao, S Asif. Hybrid-NET: A fusion of DenseNet169 and advanced machine learning classifiers for enhanced brain tumor diagnosis. International Journal of Imaging Systems and Technology, 2024, 34(1): e22975
17
M A Al-Masni, D H Kim, T S Kim. Multiple skin lesions diagnostics via integrated deep convolutional networks for segmentation and classification. Computer Methods and Programs in Biomedicine, 2020, 190: 105351 https://doi.org/10.1016/j.cmpb.2020.105351
18
G HuangZ LiuL van der Maaten. Densely connected convolutional networks. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. Honolulu: IEEE Computer Society, 2017, 4700–4708
19
A G HowardM ZhuB Chen. Mobilenets: Efficient convolutional neural networks for mobile vision applications. 2017, arXiv: 1704.04861
20
K HeX ZhangS Ren. Deep residual learning for image recognition. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. Las Vegas, NV: IEEE Computer Society, 2016, 770–778
21
F Chollet. Deep learning with separable convolutions. 2016, arXiv: 1610.2357
22
N H Quang. Automatic skin lesion analysis towards melanoma detection. In: Proceedings of 2017 21st Asia Pacific Symposium on Intelligent and Evolutionary Systems (IES). Hanoi: Springer, 2017, 106–111
23
A H ShahinA KamalM A Elattar. Deep ensemble learning for skin lesion classification from dermoscopic images. In: Prodeeings of 2018 9th Cairo International Biomedical Engineering Conference (CIBEC). Cairo: IEEE, 2018, 150–153
24
C SzegedyS IoffeV Vanhoucke. Inception-V4, inception-ResNet and the impact of residual connections on learning. In: Proceedings of the AAAI Conference on Artificial Intelligence. San Francisco, CA: AAAI Press, 2017
25
M TanQ Le. Efficientnet: Rethinking model scaling for convolutional neural networks. In: Proceedings of International Conference on Machine Learning. Beach, CA: PMLR, 2019, 6105–6114
26
M ChhabraR Kumar. A smart healthcare system based on classifier DenseNet 121 model to detect multiple diseases. In: Proceedings of Second MRCN 2021. Singapore: Springer Nature Singapore, 2022, 297–312
27
B A Ashqar, S S Abu-Naser. Identifying images of invasive hydrangea using pre-trained deep convolutional neural networks. International Journal of Academic Engineering Research, 2019, 3(3): 28–36 https://doi.org/10.33832/ijca.2019.12.4.02
28
M Wang, E Wang, X Liu, Z Wang, C Wang. Influence of neural network structure on rock intelligent classification based on structural and tectonic features of rocks. Rock Mechanics and Rock Engineering, 2022, 55(9): 5415–5432 https://doi.org/10.1007/s00603-022-02907-7
29
Y Li, P Wang, Q Feng, X Ji, D Jin, J Gong. Landslide detection based on shipborne images and deep learning models: A case study in the three gorges reservoir area in China. Landslides, 2023, 20(3): 547–558 https://doi.org/10.1007/s10346-022-01997-2
30
E TolstayaA Egorov. Segmentation of seismic images. In: Processing of 13th International Conference in Computer and Graphic Visualization, Computer Vision and Image Processing. Hangzhou: IEEE, 2021, 564–570
H Zhang, G Yang, H Li, W Du, J Wang. Pixel-wise detection algorithm for crack structural reconstruction based on rock CT images. Automation in Construction, 2023, 152: 104895 https://doi.org/10.1016/j.autcon.2023.104895
33
Y Li, T Bao, X Huang, H Chen, B Xu, X Shu, Y Zhou, Q Cao, J Tu, R Wang, K Zhang. Underwater crack pixel-wise identification and quantification for dams via lightweight semantic segmentation and transfer learning. Automation in Construction, 2022, 144: 104600 https://doi.org/10.1016/j.autcon.2022.104600
34
R StrickerM EisenbachM Sesselmann. Improving visual road condition assessment by extensive experiments on the extended gaps dataset. In: Proceedings of 2019 International Joint Conference on Neural Networks (IJCNN). Budapest: IEEE, 2019, 1–8
35
Z Li. Global sensitivity analysis of the static performance of concrete gravity dam from the viewpoint of structural health monitoring. Archives of Computational Methods in Engineering, 2021, 28(3): 1611–1646 https://doi.org/10.1007/s11831-020-09434-0
36
J Dong, N Wang, H Fang, Q Hu, C Zhang, B Ma, D Ma, H Hu. Innovative method for pavement multiple damages segmentation and measurement by the Road-Seg-CapsNet of feature fusion. Construction and Building Materials, 2022, 324: 126719 https://doi.org/10.1016/j.conbuildmat.2022.126719
37
D Kang, S S Benipal, D L Gopal, Y J Cha. Hybrid pixel-level concrete crack segmentation and quantification across complex backgrounds using deep learning. Automation in Construction, 2020, 118: 103291 https://doi.org/10.1016/j.autcon.2020.103291
38
Y Xiang, J B Sheng, L Wang, Y B Cai, Y Meng, W Cai. Research progresses on equipment technologies used in safety inspection, repair, and reinforcement for deepwater dams. Science China. Technological Sciences, 2022, 65(5): 1059–1071 https://doi.org/10.1007/s11431-021-1958-y
39
K Kov’ari, G Peter. Continuous strain monitoring in the rock foundation of a large gravity dam. Rock Mechanics and Rock Engineering, 1983, 16(3): 157–171 https://doi.org/10.1007/BF01033277
40
Y Li, T Bao, J Gong, X Shu, K Zhang. The prediction of dam displacement time series using STL, extra-trees, and stacked LSTM neural network. IEEE Access: Practical Innovations, Open Solutions, 2020, 8: 94440–94452 https://doi.org/10.1109/ACCESS.2020.2995592
41
W Wang, M Wang, H Li, H Zhao, K Wang, C He, J Wang, S Zheng, J Chen. Pavement crack image acquisition methods and crack extraction algorithms: A review. Journal of Traffic and Transportation Engineering, 2019, 6(6): 535–556 https://doi.org/10.1016/j.jtte.2019.10.001
42
Y Li, T Bao, Z Gao, X Shu, K Zhang, L Xie, Z Zhang. A new dam structural response estimation paradigm powered by deep learning and transfer learning techniques. Structural Health Monitoring, 2022, 21(3): 770–787 https://doi.org/10.1177/14759217211009780
43
Y Li, T Bao, X Shu, Z Gao, J Gong, K Zhang. Data-driven crack behavior anomaly identification method for concrete dams in long-term service using offline and online change point detection. Journal of Civil Structural Health Monitoring, 2021, 11(5): 1449–1460 https://doi.org/10.1007/s13349-021-00520-w
44
M Ishfaque, S Salman, K Z Jadoon, A A K Danish, K U Bangash, Q W Dai. Understanding the effect of hydro-climatological parameters on dam seepage using shapley additive explanation (SHAP): A case study of earth-fill Tarbela dam, Pakistan. Water, 2022, 14(17): 2598 https://doi.org/10.3390/w14172598
45
T Yamaguchi, S Hashimoto. Fast crack detection method for large-size concrete surface images using percolation-based image processing. Machine Vision and Applications, 2010, 21(5): 797–809 https://doi.org/10.1007/s00138-009-0189-8
46
P Chun, S Izumi, T Yamane. Automatic detection method of cracks from concrete surface imagery using two-step light gradient boosting machine. Computer-Aided Civil and Infrastructure Engineering, 2021, 36(1): 61–72 https://doi.org/10.1111/mice.12564
47
Y A Hsieh, Y J Tsai. Machine learning for crack detection: Review and model performance comparison. Journal of Computing in Civil Engineering, 2020, 34(5): 04020038 https://doi.org/10.1061/(ASCE)CP.1943-5487.0000918
48
V Shirhatti, A Borthakur, S Ray. Effect of reference scheme on power and phase of the local field potential. Neural Computation, 2016, 28(5): 882–913 https://doi.org/10.1162/NECO_a_00827
49
C S N Pathirage, J Li, L Li, H Hao, W Liu, P Ni. Structural damage identification based on autoencoder neural networks and deep learning. Engineering Structures, 2018, 172: 13–28 https://doi.org/10.1016/j.engstruct.2018.05.109
50
F Ni, J Zhang, Z Chen. Pixel-level crack delineation in images with convolutional feature fusion. Structural Control and Health Monitoring, 2019, 26(1): e2286 https://doi.org/10.1002/stc.2286
51
B Kim, S Cho. Image-based concrete crack assessment using mask and region-based convolutional neural network. Structural Control and Health Monitoring, 2019, 26(8): e2381 https://doi.org/10.1002/stc.2381
52
J Huyan, W Li, S Tighe, Z Xu, J Zhai. CrackU-net: A novel deep convolutional neural network for pixelwise pavement crack detection. Structural Control and Health Monitoring, 2020, 27(8): e2551 https://doi.org/10.1002/stc.2551
53
C Zhao, D Ding, Z Du, Y Shi, G Su, S Yu. Analysis of perception accuracy of roadside millimeter-wave radar for traffic risk assessment and early warning systems. International Journal of Environmental Research and Public Health, 2023, 20(1): 879 https://doi.org/10.3390/ijerph20010879
54
C Zhao, A Song, Y Zhu, S Jiang, F Liao, Y Du. Data-driven indoor positioning correction for infrastructure-enabled autonomous driving systems: A lifelong framework. IEEE Transactions on Intelligent Transportation Systems, 2023, 24(4): 3908–3921 https://doi.org/10.1109/TITS.2022.3233563
55
C Zhao, A Song, Y Du, B Yang. TrajGAT: A map-embedded graph attention network for real-time vehicle trajectory imputation of roadside perception. Transportation Research Part C, Emerging Technologies, 2022, 142: 103787 https://doi.org/10.1016/j.trc.2022.103787
56
Y J Cha, W Choi, G Suh, S Mahmoudkhani, O Büyüköztürk. Autonomous structural visual inspection using region-based deep learning for detecting multiple damage types. Computer-Aided Civil and Infrastructure Engineering, 2018, 33(9): 731–747 https://doi.org/10.1111/mice.12334
57
J Huyan, W Li, S Tighe, J Zhai, Z Xu, Y Chen. Detection of sealed and unsealed cracks with complex backgrounds using deep convolutional neural network. Automation in Construction, 2019, 107: 102946 https://doi.org/10.1016/j.autcon.2019.102946
58
Y Du, N Pan, Z Xu, F Deng, Y Shen, H Kang. Pavement distress detection and classification based on YOLO network. International Journal of Pavement Engineering, 2021, 22(13): 1659–1672 https://doi.org/10.1080/10298436.2020.1714047
59
J Liang, B Chen, C Shao, J Li, B Wu. Time reverse modeling of damage detection in underwater concrete beams using piezoelectric intelligent modules. Sensors, 2020, 20(24): 7318 https://doi.org/10.3390/s20247318
60
J Chen, F Xiong, Y Zhu, H Yan. A crack detection method for underwater concrete structures using sensing-heating system with porous casing. Measurement, 2021, 168: 108332 https://doi.org/10.1016/j.measurement.2020.108332
61
J Si, W Xiong, D Zhong, A Yan, P Wang, Z Liu. Piezoelectric-based damage-depth monitoring method for underwater energy-relief blasting technique. Journal of Civil Structural Health Monitoring, 2021, 11(2): 251–264 https://doi.org/10.1007/s13349-020-00451-y
62
J KönigM D JenkinsP Barrie. A convolutional neural network for pavement surface crack segmentation using residual connections and attention gating. In: Proceedings of 2019 IEEE International Conference on Image Processing (ICIP). Taipei, China: IEEE, 2019, 1460–1464
63
J Zhang, J Zhang. An improved nondestructive semantic segmentation method for concrete dam surface crack images with high resolution. Mathematical Problems in Engineering, 2020, 2020: 1–14 https://doi.org/10.1155/2020/5054740
64
C V Dung, H Sekiya, S Hirano, T Okatani, C Miki. A vision-based method for crack detection in gusset plate welded joints of steel bridges using deep convolutional neural networks. Automation in Construction, 2019, 102: 217–229 https://doi.org/10.1016/j.autcon.2019.02.013
65
D Sward, J Monk, N Barrett. A systematic review of remotely operated vehicle surveys for visually assessing fish assemblages. Frontiers in Marine Science, 2019, 6: 134 https://doi.org/10.3389/fmars.2019.00134
66
R Capocci, G Dooly, E Omerdić, J Coleman, T Newe, D Toal. Inspection-class remotely operated vehicles—A review. Journal of Marine Science and Engineering, 2017, 5(1): 13 https://doi.org/10.3390/jmse5010013
67
L C Lund-Hansen, T Juul, T D Eskildsen, I Hawes, B Sorrell, C Melvad, K Hancke. A low-cost remotely operated vehicle (ROV) with an optical positioning system for under-ice measurements and sampling. Cold Regions Science and Technology, 2018, 151: 148–155 https://doi.org/10.1016/j.coldregions.2018.03.017
68
H GuoX ZhuangT Rabczuk. A deep collocation method for the bending analysis of Kirchhoff plate. 2021, arXiv: 2102.02617
69
E Samaniego, C Anitescu, S Goswami, V M Nguyen-Thanh, H Guo, K Hamdia, X Zhuang, T Rabczuk. An energy approach to the solution of partial differential equations in computational mechanics via machine learning: Concepts, implementation and applications. Computer Methods in Applied Mechanics and Engineering, 2020, 362: 112790 https://doi.org/10.1016/j.cma.2019.112790
70
X Zhuang, H Guo, N Alajlan, H Zhu, T Rabczuk. Deep autoencoder based energy method for the bending, vibration, and buckling analysis of Kirchhoff plates with transfer learning. European Journal of Mechanics-A/Solids, 2021, 87: 104225 https://doi.org/10.1016/j.euromechsol.2021.104225
71
H Guo, X Zhuang, P Chen, N Alajlan, T Rabczuk. Stochastic deep collocation method based on neural architecture search and transfer learning for heterogeneous porous media. Engineering with Computers, 2022, 38(6): 5173–5198 https://doi.org/10.1007/s00366-021-01586-2
72
H Guo, X Zhuang, X Fu. Physics-informed deep learning for three-dimensional transient heat transfer analysis of functionally graded materials. Computational Mechanics, 2023, 72(3): 1–12
73
H Guo, X Zhuang, N Alajlan, T Rabczuk. Physics-informed deep learning for melting heat transfer analysis with model-based transfer learning. Computers & Mathematics with Applications, 2023, 143: 303–317 https://doi.org/10.1016/j.camwa.2023.05.014
74
A KwasigrochA MikołajczykM Grochowski. Deep neural networks approach to skin lesions classification—A comparative analysis. In: Proceedings of 2017 22nd International Conference on Methods and Models in Automation and Robotics (MMAR). Miedzyzdroje: IEEE, 2017, 1069–1074
75
G Bonaccorso. Machine Learning Algorithms. London: Packt Publishing Ltd., 2017
76
T SaitoM Rehmsmeier. Basic evaluation measures from the confusion matrix. 2017. Available at the website of Classeval
