Detection and removal of excess materials in aircraft wings using continuum robot end-effectors

doi:10.1007/s11465-024-0806-2

Front. Mech. Eng.

2024, Vol. 19

Issue (5) : 36 https://doi.org/10.1007/s11465-024-0806-2

Detection and removal of excess materials in aircraft wings using continuum robot end-effectors

Xiujie CAO, Jingjun YU, Siqi TANG, Junhao SUI, Xu PEI(

)

School of Mechanical Engineering and Automation, Beihang University, Beijing 100191, China

Download: PDF(8726 KB) HTML
Export: BibTeX | EndNote | Reference Manager | ProCite | RefWorks

Abstract

Excess materials are left inside aircraft wings due to manual operation errors, and the removal of excess materials is very crucial. To increase removal efficiency, a continuum robot (CR) with a removal end-effector and a stereo camera is used to remove excess objects. The size and weight characteristics of excess materials in aircraft wings are analyzed. A novel negative pressure end-effector and a two-finger gripper are designed based on the CR. The negative pressure end-effector aims to remove nuts, small rivets, and small volumes of aluminum shavings. A two-finger gripper is designed to remove large volumes of aluminum shavings. A stereo camera is used to achieve automatic detection and localization of excess materials. Due to poor lighting conditions in the aircraft wing compartment, supplementary lighting devices are used to improve environmental lighting. Then, You Only Look Once (YOLO) v5 is used to classify and detect excess objects, and two training data sets of excess objects in two wings are constructed. Due to the limited texture features inside the aircraft wings, this paper adopts an image-matching method based on the results of YOLO v5 detection. This matching method avoids the performance instability problem based on Oriented Fast and Rotated BRIEF feature point matching. Experimental verification reveals that the detection accuracy of each type of excess exceeds 90%, and the visual localization error is less than 2 mm for four types of excess objects. Results show the two end-effectors can work well for the task of removing excess material from the aircraft wings using a CR.

Keywords end-effectors continuum robot visual detection and localization removal of excess materials gripper

Corresponding Author(s): Xu PEI

Issue Date: 29 October 2024

Cite this article:

Xiujie CAO,Jingjun YU,Siqi TANG, et al. Detection and removal of excess materials in aircraft wings using continuum robot end-effectors[J]. Front. Mech. Eng., 2024, 19(5): 36.

URL:

https://academic.hep.com.cn/fme/EN/10.1007/s11465-024-0806-2
https://academic.hep.com.cn/fme/EN/Y2024/V19/I5/36

Fig.1 Excess material in aircraft wings.

Fig.2 Possible positions of excess materials.

Fig.3 Aircraft wing: (a) commercial aircraft wing and (b) fighter jet wing.

Tab.1 Characteristic parameters of the continuum robot

Fig.4 Continuum robot.

Fig.5 Negative pressure removal end-effector.

Fig.6 Detailed description of the negative pressure end-effector: (a) 3D model of the end-effector, (b) half-section view of the end-effector, (c) air source and the air pipe, (d) front view of the end-effector, and (e) side views of the negative pressure end-effector.

Fig.7 Two-finger gripper: (a) removal of a nut by a two-finger gripper, (b) closer look of the two-finger gripper, (c) soft material attached to the two-finger gripper, and (d) removal of an aluminum shaving by the two-finger gripper.

Fig.8 Quick change schematic diagram: (a) quick change description, (b) moving direction, and (c) negative pressure suction end-effector after quick change.

Fig.9 Visual part and coordinate illustration: (a) side view of end-effector, (b) front view of end-effector, and (c) coordinates description.

Fig.10 Data set labels.

Fig.11 Background misdetection illustration: (a) rivet toward the cameras and (b) correct bolts in the aircraft wing.

Fig.12 Light change.

Fig.13 Single batch during training: images captured inside the (a) commercial aircraft wing and (b) fighter aircraft wing.

Fig.14 Training results: (a) precision curve and (b) precision–recall curve. mAP: mean average precision.

Parameter	Value
Intrinsic matrix of the left camera	$[490.3570 − 0.9222 313.2289 0 490.3525 231.3771 001]$
Radial distortion of the left camera	$[0.0983 − 0.1114 0.0342]$
Tangential distortion of the left camera	$[− 0.0048 − 0.0024]$
Intrinsic matrix of the right camera	$[479.6835 0.2407 316.1833 0 478.6164 227.3088 001]$
Radial distortion of the right camera	$[0.0554 − 0.0862 − 0.1362]$
Tangential distortion of the right camera	$[0.0027 − 0.0039]$
Extrinsic matrix T_e	$[0.9869 − 0.0862 − 0.1362 − 5.4362 0.0923 0.9950 0.0391 1.3210 0.1322 − 0.0512 0.9899 − 2.6899 0001]$

Tab.2 Calibration results of stereo cameras

Fig.15 Schematic diagram of a stereo camera.

Fig.16 Stereo camera rectification: (a) input images and (b) rectification results.

Fig.17 Stereo rectification diagram: (a) rotation to parallel planes and (b) rotation to coplanar images.

Fig.18 Image matching using ORB descriptors: (a) images captured by the left and right cameras and (b) feature point matching results.

Fig.19 YOLO detection results: (a) false detection, (b) low-light detection, and (c) enhanced lighting detection.

Fig.20 Detection and localization of excess materials.

Fig.21 (a–l) Suction of the negative pressure end-effector.

Fig.22 (a–l) Two-finger gripper clamping.

1	T F Long, Y Li, J Chen. Productivity prediction in aircraft final assembly lines: comparisons and insights in different productivity ranges. Journal of Manufacturing Systems, 2022, 62: 377–389 https://doi.org/10.1016/j.jmsy.2021.12.010
2	Q J Zhao, Y H Kong, S J Sheng, J J Zhu. Redundant object detection method for civil aircraft assembly based on machine vision and smart glasses. Measurement Science & Technology, 2022, 33(10): 105011 https://doi.org/10.1088/1361-6501/ac7cbd
3	Y D V Yasuda, F A M Cappabianco, L E G Martins, J A B Gripp. Aircraft visual inspection: a systematic literature review. Computers in Industry, 2022, 141: 103695 https://doi.org/10.1016/j.compind.2022.103695
4	V Tzitzilonis, K Malandrakis, L Zanotti Fragonara, J A Gonzalez Domingo, N P Avdelidis, A Tsourdos, K Forster. Inspection of aircraft wing panels using unmanned aerial vehicles. Sensors, 2019, 19(8): 1824 https://doi.org/10.3390/s19081824
5	D Axinte, X Dong, D Palmer, A Rushworth, S C Guzman, A Olarra, I Arizaga, E Gomez-Acedo, K Txoperena, K Pfeiffer, F Messmer, M Gruhler, J Kell. MiRoR—miniaturized robotic systems for holistic in-situ repair and maintenance works in restrained and hazardous environments. IEEE/ASME Transactions on Mechatronics, 2018, 23(2): 978–981 https://doi.org/10.1109/TMECH.2018.2800285
6	J M Guo, J W Zhang, D Wu, Y H Gai, K Chen. An algorithm based on bidirectional searching and geometric constrained sampling for automatic manipulation planning in aircraft cable assembly. Journal of Manufacturing Systems, 2020, 57: 158–168 https://doi.org/10.1016/j.jmsy.2020.08.015
7	B Mei, W D Zhu. Accurate positioning of a drilling and riveting cell for aircraft assembly. Robotics and Computer-Integrated Manufacturing, 2021, 69: 102112 https://doi.org/10.1016/j.rcim.2020.102112
8	I Shafi, M F Mazhar, A Fatima, R M Alvarez, Y Miró, J C M Espinosa, I Ashraf. Deep learning-based real time defect detection for optimization of aircraft manufacturing and control performance. Drones, 2023, 7(1): 31 https://doi.org/10.3390/drones7010031
9	M F Wang, D Palmer, X Dong, D Alatorre, D Axinte, A Norton. Design and development of a slender dual-structure continuum robot for in-situ aeroengine repair. In: Proceedings of the IEEE/RSJ International Conference on Intelligent Robots and Systems. Madrid: IEEE, 2018, 5648–5653
10	G X Li, J J Yu, Y C Tang, J Pan, S G Cao, X Pei. Design and modeling of continuum robot based on virtual-center of motion mechanism. Frontiers of Mechanical Engineering, 2023, 18(2): 23 https://doi.org/10.1007/s11465-022-0739-6
11	M Jha, N R Chauhan. A review on snake-like continuum robots for medical surgeries. IOP Conference Series: Materials Science and Engineering, 2019, 691(1): 012093 https://doi.org/10.1088/1757-899X%2F691%2F1%2F012093
12	M Russo, S M H Sadati, X Dong, A Mohammad, I D Walker, C Bergeles, K Xu, D A Axinte. Continuum robots: an overview. Advanced Intelligent Systems, 2023, 5(5): 2370020 https://doi.org/10.1002/aisy.202370020
13	M F Wang, X Dong, W M Ba, A Mohammad, D Axinte, A Norton. Design, modelling and validation of a novel extra slender continuum robot for in-situ inspection and repair in aeroengine. Robotics and Computer-Integrated Manufacturing, 2021, 67: 102054 https://doi.org/10.1016/j.rcim.2020.102054
14	G C Niu, J K Wang, K L Xu. Model analysis for a continuum aircraft fuel tank inspection robot based on the Rzeppa universal joint. Advances in Mechanical Engineering, 2018, 10(5): 1687814018778229
15	M Russo, L Raimondi, X Dong, D Axinte, J Kell. Task-oriented optimal dimensional synthesis of robotic manipulators with limited mobility. Robotics and Computer-Integrated Manufacturing, 2021, 69: 102096 https://doi.org/10.1016/j.rcim.2020.102096
16	X Dong, M F Wang, A Mohammad, W M Ba, M Russo, A Norton, J Kell, D Axinte. Continuum robots collaborate for safe manipulation of high-temperature flame to enable repairs in challenging environments. IEEE/ASME Transactions on Mechatronics, 2022, 27(5): 4217–4220 https://doi.org/10.1109/TMECH.2021.3138222
17	D A Troncoso, J A Robles-Linares, M Russo, M A Elbanna, S Wild, X Dong, A Mohammad, J Kell, A D Norton, D Axinte. A continuum robot for remote applications: from industrial to medical surgery with slender continuum robots. IEEE Robotics & Automation Magazine, 2023, 30(3): 94–105 https://doi.org/10.1109/MRA.2022.3223220
18	S Norouzi-Ghazbi, A Mehrkish. H. Fallah M, Janabi-Sharifi F. Constrained visual predictive control of tendon-driven continuum robots. Robotics and Autonomous Systems, 2021, 145: 103856 https://doi.org/10.1016/j.robot.2021.103856
19	S Norouzi-Ghazbi, A Mehrkish, F Janabi-Sharifi. Jacobian formulation for two classes of cooperative continuum robots. In: Proceedings of the Canadian Society for Mechanical Engineering International Congress 2020. Charlottetown: Progress in Canadian Mechanical Engineering, 2020, 3–8
20	X Dong, M Raffles, S Cobos-Guzman, D Axinte, J Kell. A novel continuum robot using twin-pivot compliant joints: design, modeling, and validation. Journal of Mechanisms and Robotics, 2016, 8(2): 021010 https://doi.org/10.1115/1.4031340
21	K Oliver-Butler, J Till, C Rucker. Continuum robot stiffness under external loads and prescribed tendon displacements. IEEE Transactions on Robotics, 2019, 35(2): 403–419 https://doi.org/10.1109/TRO.2018.2885923
22	J Hernandez, M S H Sunny, J Sanjuan, I Rulik, M I I Zarif, S I Ahamed, H U Ahmed, M H Rahman. Current designs of robotic arm grippers: a comprehensive systematic review. Robotics, 2023, 12(1): 5 https://doi.org/10.3390/robotics12010005
23	B H Zhang, Y X Xie, J Zhou, K Wang, Z Zhang. State-of-the-art robotic grippers, grasping and control strategies, as well as their applications in agricultural robots: a review. Computers and Electronics in Agriculture, 2020, 177: 105694 https://doi.org/10.1016/j.compag.2020.105694
24	Z Samadikhoshkho, K Zareinia, F Janabi-Sharifi. A brief review on robotic grippers classifications. In: Proceedings of the IEEE Canadian Conference of Electrical and Computer Engineering. Edmonton: IEEE, 2019, 1–4
25	C Firth, K Dunn, M H Haeusler, Y Sun. Anthropomorphic soft robotic end-effector for use with collaborative robots in the construction industry. Automation in Construction, 2022, 138: 104218 https://doi.org/10.1016/j.autcon.2022.104218
26	F Correa. Integrating industry 4.0 associated technologies into automated and traditional construction. In: Proceedings of the 37th International Symposium on Automation and Robotics in Construction. Kitakyushu: The International Association for Automation And Robotics in Construction, 2020, 285–292
27	P Ramon Soria, B C Arrue, A Ollero. Detection, location and grasping objects using a stereo sensor on UAV in outdoor environments. Sensors, 2017, 17(1): 103 https://doi.org/10.3390/s17010103
28	M Alhammad, N P Avdelidis, S Deane, C Ibarra-Castanedo, S Pant, P Nooralishahi, M Ahmadi, M Genest, A Zolotas, L, Valdes J, Maldague X P V Zanotti-Fragonara. Diagnosis of composite materials in aircraft applications: towards a UAV-based active thermography inspection approach. In: Zalameda J N, Mendioroz A, eds. Thermosense: Thermal Infrared Applications XLIII. Florida: SPIE, 2021, 1174306
29	Z X Zou, K Y Chen, Z W Shi, Y H Guo, J P Ye. Object detection in 20 years: a survey. Proceedings of the IEEE, 2023, 111(3): 257–276 https://doi.org/10.1109/JPROC.2023.3238524
30	J Redmon, S Divvala, R Girshick, A Farhadi. You only look once: unified, real-time object detection. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. Las Vegas: IEEE, 2016, 779–788
31	A Nazir, M A Wani. You only look once - object detection models: a review. In: Proceedings of the 10th International Conference on Computing for Sustainable Global Development. New Delhi: IEEE, 2023, 1088–1095
32	J R Chang, Y S Chen. Pyramid stereo matching network. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. Salt Lake City: IEEE, 2018, 5410–5418
33	M P Mathew, T Y Mahesh. Leaf-based disease detection in bell pepper plant using YOLO v5. Signal, Image and Video Processing, 2022, 16(3): 841–847 https://doi.org/10.1007/s11760-021-02024-y
34	A Fusiello, E Trucco, A Verri. A compact algorithm for rectification of stereo pairs. Machine Vision and Applications, 2000, 12(1): 16–22 https://doi.org/10.1007/s001380050120
35	C Grana, D Borghesani, M Manfredi, R Cucchiara. A fast approach for integrating ORB descriptors in the bag of words model. In: Proceedings of the Multimedia Content and Mobile Devices. Burlingame: SPIE, 2013, 866709

[1]	Tingke WU, Zhuyong LIU, Ziqi MA, Boyang WANG, Daolin MA, Hexi YU. Bionic soft robotic gripper with feedback control for adaptive grasping and capturing applications[J]. Front. Mech. Eng., 2024, 19(1): 8-.
[2]	Zheshuai YANG, Laihao YANG, Yu SUN, Xuefeng CHEN. *Comprehensive kinetostatic modeling and morphology characterization of cable-driven continuum robots for in-situ* aero-engine maintenance**[J]. Front. Mech. Eng., 2023, 18(3): 40-.
[3]	Guoxin LI, Jingjun YU, Yichao TANG, Jie PAN, Shengge CAO, Xu PEI. Design and modeling of continuum robot based on virtual-center of motion mechanism[J]. Front. Mech. Eng., 2023, 18(2): 23-.
[4]	Jie PAN, Jingjun YU, Xu PEI. A novel shape memory alloy actuated soft gripper imitated hand behavior[J]. Front. Mech. Eng., 2022, 17(4): 44-.
[5]	Kun XU, Peijin ZI, Xilun DING. Learning from biological attachment devices: applications of bioinspired reversible adhesive methods in robotics[J]. Front. Mech. Eng., 2022, 17(3): 43-.
[6]	Lin WANG, Yuefa FANG, Luquan LI. Design and analysis of the gripper mechanism based on generalized parallel mechanisms with configurable moving platform[J]. Front. Mech. Eng., 2021, 16(4): 765-781.
[7]	Hanmin PENG, Yao HUI, Qingjun DING, Huafeng LI, Chunsheng ZHAO, . IPMC gripper static analysis based on finite element analysis[J]. Front. Mech. Eng., 2010, 5(2): 204-211.

Viewed

Full text

Abstract

Cited

Shared

Discussed