Please wait a minute...
Shopping Cart Email List Chinese
Advanced Search Fig/Tab
Frontiers of Mechanical Engineering

ISSN 2095-0233

ISSN 2095-0241(Online)

CN 11-5984/TH

Postal Subscription Code 80-975

2018 Impact Factor: 0.989

Featured Articles  |  Most Downloaded  |  Most Reading
Online Submission Subscribe to Our RSS Content Feed
Front. Mech. Eng.    2024, Vol. 19 Issue (3) : 21    https://doi.org/10.1007/s11465-024-0793-3
VMMAO-YOLO: an ultra-lightweight and scale-aware detector for real-time defect detection of avionics thermistor wire solder joints
Xiaoqi YANG1, Xingyue LIU1(), Qian WU1, Guojun WEN1, Shuang MEI1, Guanglan LIAO2, Tielin SHI2
1. School of Mechanical Engineering and Electronic Information, China University of Geosciences, Wuhan 430074, China
2. State Key Laboratory of Digital Manufacturing Equipment and Technology, Huazhong University of Science and Technology, Wuhan 430074, China
 Download: PDF(6508 KB)   HTML
 Export: BibTeX | EndNote | Reference Manager | ProCite | RefWorks
Abstract

The quality of the exposed avionics solder joints has a significant impact on the stable operation of the in-orbit spacecrafts. Nevertheless, the previously reported inspection methods for multi-scale solder joint defects generally suffer low accuracy and slow detection speed. Herein, a novel real-time detector VMMAO-YOLO is demonstrated based on variable multi-scale concurrency and multi-depth aggregation network (VMMANet) backbone and “one-stop” global information gather-distribute (OS-GD) module. Combined with infrared thermography technology, it can achieve fast and high-precision detection of both internal and external solder joint defects. Specifically, VMMANet is designed for efficient multi-scale feature extraction, which mainly comprises variable multi-scale feature concurrency (VMC) and multi-depth feature aggregation-alignment (MAA) modules. VMC can extract multi-scale features via multiple fix-sized and deformable convolutions, while MAA can aggregate and align multi-depth features on the same order for feature inference. This allows the low-level features with more spatial details to be transmitted in depth-wise, enabling the deeper network to selectively utilize the preceding inference information. The VMMANet replaces inefficient high-density deep convolution by increasing the width of intermediate feature levels, leading to a salient decline in parameters. The OS-GD is developed for efficacious feature extraction, aggregation and distribution, further enhancing the global information gather and deployment capability of the network. On a self-made solder joint image data set, the VMMAO-YOLO achieves a mean average precision mAP@0.5 of 91.6%, surpassing all the mainstream YOLO-series models. Moreover, the VMMAO-YOLO has a body size of merely 19.3 MB and a detection speed up to 119 frame per second, far superior to the prevalent YOLO-series detectors.
Keywords defect detection of solder joints      VMMAO-YOLO      ultra-lightweight and high-performance      multi-scale feature extraction      VMC and MAA modules      OS-GD     
Corresponding Author(s): Xingyue LIU   
Issue Date: 01 July 2024
 Cite this article:   
Xiaoqi YANG,Xingyue LIU,Qian WU, et al. VMMAO-YOLO: an ultra-lightweight and scale-aware detector for real-time defect detection of avionics thermistor wire solder joints[J]. Front. Mech. Eng., 2024, 19(3): 21.
 URL:  
https://academic.hep.com.cn/fme/EN/10.1007/s11465-024-0793-3
https://academic.hep.com.cn/fme/EN/Y2024/V19/I3/21
Fig.1  Main kinds of defects in avionics thermistor wire solder joints: (a) break, (b) multi-tin, (c) pinhole, and (d) leak-soldering.
Fig.2  Overall framework of the as-designed VMMAO-YOLO.
Fig.3  Structural schematics of the (a) VMC and (b) MAA modules.
Fig.4  Configuration of the OS-GD module.
Fig.5  Infrared image collection of avionics thermistor wire solder joints: (a) optical image of the solder joints; (b) home-made automatic equipment with functions of both soldering and infrared thermography; (c) infrared image of the solder joint samples.
Fig.6  Distribution of four-type solder joint defect images in each dataset.
Detector mAP@0.5 mAP@0.5:0.95 Model size/MB FLOPs/G FPS (bs = 8)
YOLOv5x 89.4% 57.2% 165 203.80 69
YOLOv6l 88.7% 56.7% 114 150.51 86
YOLOv7x 79.6% 49.4% 135 188.00 61
YOLOv8x 87.3% 57.2% 130 257.40 72
This work 91.6% 58.4% 19.3 36.40 119
Tab.1  Detection performance comparison of VMMAO-YOLO and YOLO-series modelsa)
Fig.7  FPS–mAP scatterplots of the tested detectors.
Detector Break Multi-tin Pinhole Leak-soldering All
YOLOv5x 91.1% 93.6% 73.4% 99.5% 89.4%
YOLOv6l 90.7% 94.3% 69.9% 99.9% 88.7%
YOLOv7x 87.2% 94.7% 42.8% 93.5% 79.6%
YOLOv8x 91.6% 97.1% 61.2% 99.3% 87.3%
This work 91.7% 96.8% 78.6% 99.5% 91.6%
Tab.2  Detailed detection accuracy of the detectors for each type of solder joint defect
Fig.8  Infrared solder joint: (a) original and (b) tagged images. Visual defect inspection results of (c) YOLOv5x, (d) YOLOv6l, (e) YOLOv7x, (f) YOLOv8x, and (g) VMMAO-YOLO.
VMMANet OS-GD mAP@0.5 mAP@0.5:0.95 Model size/MB FLOPs/G FPS
– – 89.4% 57.2% 165 203.8 69
√ – 90.6% 57.8% 6.12 16.8 142
– √ 91.1% 58.0% 223 249.3 57
√ √ 91.6% 58.4% 19.3 36.4 119
Tab.3  Ablation experiments of VMMANet and OS-GD
Fig.9  Variation curves of (a) mAP@0.5, (b) mAP@0.5:0.95, (c) precision, and (d) recall of the different models in the ablation experiments.
Detector mAP@0.5 mAP@0.5:0.95 Model size/MB FLOPs/G FPS
YOLOv5x 89.4% 57.2% 165 203.8 69
YOLOv5x + CBAM 90.2% 57.3% 165 204.3 71
YOLOv5x + GD 90.1% 57.3% 132 187.6 70
YOLOv5x + VMMANet 90.6% 57.8% 6.12 16.8 142
YOLOv5x + OS-GD 91.1% 58.0% 223 249.3 57
YOLOv5x + VMMANet + OS-GD 91.6% 58.4% 19.3 36.4 119
Tab.4  Comparison experiments of VMMANet and OS-GD modules with other similar modules
Fig.10  Variation curves of (a) mAP@0.5 and (b) mAP@0.5:0.95 of the detectors during training.
Fig.11  (a) Original and (b) tagged images of the solder joints. Visual heat maps of the testing models with (c) CBAM, (d) GD, and (e) OS-GD modules processed by Grad CAM. Grad CAM: gradient-weighted class activation mapping.
Abbreviations
ACAS Module for aligning spatial and channel scales
AKConv A convolution method with variable parameters and convolution kernel shape
AP Average precision
AS Align space module
CFE Channel feature extraction module
Conv Convolutional layer
IOU Intersection over union
mAP Mean average precision
OS-GD One-stop global information gather-distribute
P Precision
R Recall
SCConv Convolution method of space and channel reconstruction
SFE Spatial feature extraction model
WMMANet Variable multi-scale concurrency and multi-depth aggregation network
Variables
CF The channel attention maps
Ff Feature tensor F from f locations or operations
N Number of classes
S The area formed by the real box
S′ The area formed by the prediction box
SF The spatial attention maps
x The abscissa of point x on a feature layer space
y The ordinate of point y on a feature layer space
z The k channel in the feature layer
  
1 Bian Y C, Fu G H, Hou Q S, Sun B, Liao G L, Han H D. Using improved YOLOv5s for defect detection of thermistor wire solder joints based on infrared thermography. In: Proceedings of the 5th International Conference on Automation, Control and Robots. Nanning: IEEE, 2021, 29–32
2 T Y FangJ S AnQ ChenY Z HeH J Wang X G Zhang. Progress and comparison in nondestructive detection, imaging and recognition technology for defects of wafers, chips and solder joints. Nondestructive Testing and Evaluation, 2023 (in press)
3 W T Dai, A Mujeeb, M Erdt, A Sourin. Soldering defect detection in automatic optical inspection. Advanced Engineering Informatics, 2020, 43: 101004
https://doi.org/10.1016/j.aei.2019.101004
4 X Zhang, T Huang, B Wu, Y M Hu, S Huang, Q Zhou, X Zhang. Multi-model ensemble deep learning method for intelligent fault diagnosis with high-dimensional samples. Frontiers of Mechanical Engineering, 2021, 16(2): 340–352
https://doi.org/10.1007/s11465-021-0629-3
5 S Q Ren, K M He, R Girshick, J Sun. Faster R-CNN: towards real-time object detection with region proposal networks. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2017, 39(6): 1137–1149
https://doi.org/10.1109/TPAMI.2016.2577031
6 K M HeG GkioxariP DollárR Girshick. Mask R-CNN. In: Proceedings of 2017 IEEE International Conference on Computer Vision. Venice: IEEE, 2017, 2980–2988
7 Redmon J, Divvala S, Girshick R, Farhadi A. You only look once: unified, real-time object detection. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. Las Vegas: IEEE, 2016, 779–788
8 Redmon J, Farhadi A. YOLOv3: an incremental improvement. 2018, arXiv:1804.02767
9 Zhu X K, Lyu S C, Wang X, Zhao Q. TPH-YOLOv5: improved YOLOv5 based on transformer prediction head for object detection on drone-captured scenarios. In: Proceedings of the IEEE/CVF International Conference on Computer Vision Workshops. Montreal: IEEE, 2021, 2778–2788
10 Wang C Y, Bochkovskiy A, Liao H Y M. YOLOv7: trainable bag-of-freebies sets new state-of-the-art for real-time object detectors. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. Vancouver: IEEE, 2023, 7464–7475
11 Aboah A, Wang B, Bagci U, Adu-Gyamfi Y. Real-time multi-class helmet violation detection using few-shot data sampling technique and YOLOv8. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops. Vancouver: IEEE, 2023, 5350–5358
12 Fan F L, Wang B Y, Zhu G L, Wu J H. Efficient faster R-CNN: used in PCB solder joint defects and components detection. In: Proceedings of the 4th IEEE International Conference on Computer and Communication Engineering Technology. Beijing: IEEE, 2021, 1–5
13 K H Zhang, H K Shen. Solder joint defect detection in the connectors using improved faster-RCNN algorithm. Applied Sciences, 2021, 11(2): 576
https://doi.org/10.3390/app11020576
14 C Kim, S Hwang, H Sohn. Weld crack detection and quantification using laser thermography, mask R-CNN, and CycleGAN. Automation in Construction, 2022, 143: 104568
https://doi.org/10.1016/j.autcon.2022.104568
15 W P Liu, S Q Shan, H Y Chen, R Wang, J M Sun, Z K Zhou. X-ray weld defect detection based on AF-RCNN. Welding in the World, 2022, 66(6): 1165–1177
https://doi.org/10.1007/s40194-022-01281-w
16 C Ji, H B Wang, H H Li. Defects detection in weld joints based on visual attention and deep learning. NDT & E International, 2023, 133: 102764
https://doi.org/10.1016/j.ndteint.2022.102764
17 Cherkasov N, Ivanov M, Ulanov A. Weld surface defect detection based on a laser scanning system and YOLOv5. In: Proceedings of the International Conference on Industrial Engineering, Applications and Manufacturing. Sochi: IEEE, 2023, 851–855
18 L Yang, Y H Liu, J Z Peng. An automatic detection and identification method of welded joints based on deep neural network. IEEE Access: Practical Innovations, Open Solutions, 2019, 7: 164952–164961
https://doi.org/10.1109/ACCESS.2019.2953313
19 Zuo Y L, Wang J T, Song J L. Application of YOLO object detection network in weld surface defect detection. In: Proceedings of the 11th IEEE Annual International Conference on CYBER Technology in Automation, Control, and Intelligent Systems. Jiaxing: IEEE, 2021, 704–710
20 S D Liao, C Y Huang, Y Liang, H Q Zhang, S F Liu. Solder joint defect inspection method based on ConvNeXt-YOLOX. IEEE Transactions on Components, Packaging, and Manufacturing Technology, 2022, 12(11): 1890–1898
https://doi.org/10.1109/TCPMT.2022.3224997
21 M Y Liu, Y P Chen, J M Xie, L He, Y Zhang. LF-YOLO: A lighter and faster YOLO for weld defect detection of X-ray image. IEEE Sensors Journal, 2023, 23(7): 7430–7439
https://doi.org/10.1109/JSEN.2023.3247006
22 P Klco, D Koniar, L Hargas, K Pociskova Dimova, M Chnapko. Quality inspection of specific electronic boards by deep neural networks. Scientific Reports, 2023, 13(1): 20657
https://doi.org/10.1038/s41598-023-47958-0
23 G Q Wang, C Z Zhang, M S Chen, Y C Lin, X H Tan, P Liang, Y X Kang, W D Zeng, Q Wang. Yolo-MSAPF: Multiscale alignment fusion with parallel feature filtering model for high accuracy Weld Defect Detection. IEEE Transactions on Instrumentation and Measurement, 2023, 72: 1–14
https://doi.org/10.1109/TIM.2023.3302372
24 J E Kwon, J H Park, J H Kim, Y H Lee, S I Cho. Context and scale-aware YOLO for welding defect detection. NDT & E International, 2023, 139: 102919
https://doi.org/10.1016/j.ndteint.2023.102919
25 C Ajmi, J Zapata, S Elferchichi, K Laabidi. Advanced faster-RCNN model for automated recognition and detection of weld defects on limited X-ray image dataset. Journal of Nondestructive Evaluation, 2024, 43(1): 14
https://doi.org/10.1007/s10921-023-01032-x
26 X Y Zhou, Y Xue, G Y Tian, Z Liu. Thermal analysis of solder joint based on eddy current pulsed thermography. IEEE Transactions on Components, Packaging, and Manufacturing Technology, 2017, 7(7): 1111–1118
https://doi.org/10.1109/TCPMT.2017.2696159
27 A BochkovskiyC Y WangH Y M Liao. YOLOv4: optimal speed and accuracy of object detection. 2020, arXiv:2004.10934
28 Liu S, Qi L, Qin H F, Shi J P, Jia J Y. Path aggregation network for instance segmentation. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. Salt Lake City: IEEE, 2018, 8759–8768
29 Z GeS T Liu F WangZ M LiJ Sun. YOLOX: exceeding YOLO Series in 2021. 2021, arXiv:2107.08430
30 G JocherA ChaurasiaA StokenJ BorovecY NanoCode012K KwonT MichaelJ C Xie Y F FangC ImyhxyA LornaD ZengZ Q Wong C VJ Montes V WangP FatiA NadarD LaughingM UnglvKitDeM SonckTkianaiYxnong Skalski HoganNairStrobelJain. Ultralytics/yolov5: v7.0-YOLOv5 SOTA realtime instance segmentation. Zenodo, 2022
31 C Y LiL LiH L JiangK H WengY F Geng L LiZ D Ke Q Y LiM ChengW Q NieY D LiB Zhang Y F LiangL Y ZhouX M XuX X ChuX M Wei X L Wei. YOLOv6: a single-stage object detection framework for industrial applications. 2022, arXiv:2209.02976
32 Jocher G, Chaurasia A, Qiu J. Ultralytics yolov8. 2023, available at Ultralytics
33 Ding X H, Zhang X Y, Ma N N, Han J G, Ding G G, Sun J. RepVGG: making VGG-style ConvNets great again. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. Nashville: IEEE, 2021, 13728–13737
34 Hu J, Shen L, Sun G. Squeeze-and-excitation networks. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. Salt Lake City: IEEE, 2018, 7132–7141
35 S WooJ Park J Y LeeI S Kweon. CBAM: convolutional block attention module. In: Ferrari V, Hebert M, Sminchisescu C, Weiss Y, eds. In: Ferrari V, Hebert M, Sminchisescu C, Weiss Y, eds. Cham: Springer, 2018, 3–19
36 Y C LiuZ R ShaoN Hoffmann. Global Attention Mechanism: retain Information to Enhance Channel-Spatial Interactions. 2021, arXiv:2112.05561
37 C C WangW HeY NieJ Y GuoC J Liu K HanY H Wang. Gold-YOLO: Efficient Object Detector via Gather-and-Distribute Mechanism. 2023, arXiv:2309.11331
38 Szegedy C, Liu W, Jia Y Q, Sermanet P, Reed S, Anguelov D, Erhan D, Vanhoucke V, Rabinovich A. Going deeper with convolutions. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. Boston: IEEE, 2015, 1–9
39 W Y LvS L XuY ZhaoG Z WangJ M Wei C CuiY N Du Q Q DangY Liu. DETRs Beat YOLOs on Real-time Object Detection. 2023, arXiv:2304.08069
40 X ZhangY Z SongT T SongD G YangY C Ye J ZhouL M Zhang. AKConv: Convolutional Kernel with Arbitrary Sampled Shapes and Arbitrary Number of Parameters. 2023, arXiv:2311.111587
41 Li J F, Wen Y, He L H. SCConv: spatial and channel reconstruction convolution for feature redundancy. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. Vancouver: IEEE, 2023, 6153–6162
42 He K M, Zhang X Y, Ren S Q, Sun J. Deep residual learning for image recognition. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. Las Vegas: IEEE, 2016, 770–778
43 Z Li, C Y Lang, J H Liew, Y D Li, Q B Hou, J S Feng. Cross-layer feature pyramid network for salient object detection. IEEE Transactions on Image Processing, 2021, 30: 4587–4598
https://doi.org/10.1109/TIP.2021.3072811
Viewed
Full text


Abstract

Cited
  Shared   
  Discussed