. Faculty of Building and Industrial Construction, Hanoi University of Civil Engineering, Hanoi 11600, Vietnam . Research Group of Development and application of advanced materials and modern technologies in construction, Hanoi University of Civil Engineering, Hanoi 11600, Vietnam
Collecting and analyzing vibration signals from structures under time-varying excitations is a non-destructive structural health monitoring approach that can provide meaningful information about the structures’ safety without interrupting their normal operations. This paper develops a novel framework using prompt engineering for seamlessly integrating users’ domain knowledge about vibration signals with the advanced inference ability of well-trained large language models (LLMs) to accurately identify the actual states of structures. The proposed framework involves formulating collected data into a standardized form, utilizing various prompts to gain useful insights into the dynamic characteristics of vibration signals, and implementing an in-house program with the help of LLMs to perform damage detection. The advantages, as well as limitations, of the proposed method are qualitatively and quantitatively assessed through two realistic case studies from literature, demonstrating that the present method is a new way to quickly construct practical and reliable structural health monitoring applications without requiring advanced programming/mathematical skills or obscure specialized programs.
Just Accepted Date: 02 August 2024Online First Date: 01 November 2024Issue Date: 28 November 2024
Cite this article:
Truong-Thang NGUYEN,Viet-Hung DANG,Thanh-Tung PHAM. Development of an automatic and knowledge-infused framework for structural health monitoring based on prompt engineering[J]. Front. Struct. Civ. Eng.,
2024, 18(11): 1752-1774.
Fig.1 Graphical representation of prompt engineering with LLM.
Fig.2 Typical elements of a prompt.
Fig.3 Example of a structured prompt.
Fig.4 Schematic representation of different prompt strategies.
Fig.5 Working flow of the SHM framework with LLM and prompt engineering.
Fig.6 Graphical representation of the ASCE benchmark steel frame structure.
Fig.7 Example of vibration signals corresponding to nine structural states.
Fig.8 Prompt and LLM’s corresponding responses for analyzing raw vibration signals using the GPT 3.5 model.
Fig.9 Prompt and LLM’s corresponding responses for feature extractions using the GPT 3.5 model.
Fig.10 Prompt and LLM’s corresponding responses for classifying structural states using extracted features using the GPT 3.5 model.
Class
Precision
Recall
F1-score
0
1.00
1.00
1.00
1
1.00
1.00
1.00
2
1.00
1.00
1.00
3
1.00
1.00
1.00
4
1.00
1.00
1.00
5
1.00
1.00
1.00
6
1.00
1.00
1.00
7
0.95
0.95
0.95
8
0.97
0.97
0.97
Tab.1 Number of trainable parameters of the proposed framework
Fig.11 Prompt and LLM’s corresponding responses for building a ML-based classifier using the GPT 3.5 model.
Fig.12 Prompt and LLM’s corresponding responses for fine-tuning a ML-based classifier for the ASCE benchmark data set.
State
Structural state description
0
healthy
1
reducing 50% stiffness of one column on the first floor
2
reducing 50% stiffness of two columns on the first floor
3
reducing 50% stiffness of one column on the third floor
4
reducing 50% stiffness of two columns on the third floor
5
reducing 50% stiffness of two columns on the second floor
6
reducing 50% stiffness of one column on the second floor
Tab.2 Investigated structural states of the three-story steel frame
Fig.13 Schematic representation of the three-story steel frame structure at the Alamos laboratory [35].
Fig.14 Prompt and LLM’s corresponding responses for analyzing raw vibration signals of the laboratory frame structure using the GPT 3.5 model.
Fig.15 Prompt and LLM’s corresponding responses for classifying structural states using extracted features of the laboratory frame structure using the GPT 3.5 model.
Fig.16 Prompt and LLM’s corresponding responses for classifying structural states using extracted features of the laboratory frame structure using the LLAMA 2 model.
Class
Precision
Recall
F1-score
0
0.89
0.85
0.87
1
0.89
0.87
0.88
2
0.92
0.97
0.94
3
0.88
0.90
0.89
4
0.96
0.96
0.96
5
0.90
0.90
0.90
6
0.86
0.86
0.86
Tab.3 Results of structural damage detection using the classifier provided by LLM (LLAMA2) on the laboratory frame structure
Fig.17 Prompt and LLM’s corresponding responses for building a ML-based classifier using the LLAMA 2 model.
Fig.18 Prompt and LLM’s corresponding responses for fine-tuning a ML-based classifier.
Fig.19 Detailed of structural damage detection results via confusion matrix.
Fig.20 Prompt and code snippet derived from LLM responses for plotting ROC curves using the GPT 3.5 model.
Fig.21 Detailed of structural damage detection results via ROC curves.
1
M Christ, N Braun, J Neuffer, A W Kempa-Liehr. Time series feature extraction on basis of scalable hypothesis tests (TSfresh––A python package). Neurocomputing, 2018, 307: 72–77 https://doi.org/10.1016/j.neucom.2018.03.067
2
H V Dang, M Raza, H Tran-Ngoc, T Bui-Tien, H X Nguyen. Connection stiffness reduction analysis in steel bridge via deep CNN and modal experimental data. Structural Engineering and Mechanics. International Journal, 2021, 77(4): 495–508
3
H Dang, T T Nguyen. Robust vibration output-only structural health monitoring framework based on multi-modal feature fusion and self-learning. Periodica Polytechnica. Civil Engineering, 2023, 67(2): 416–430
4
V H Dang, H A Pham. Vibration-based building health monitoring using spatio-temporal learning model. Engineering Applications of Artificial Intelligence, 2023, 126: 106858 https://doi.org/10.1016/j.engappai.2023.106858
5
S Lin, H Zheng, B Han, Y Li, C Han, W Li. Comparative performance of eight ensemble learning approaches for the development of models of slope stability prediction. Acta Geotechnica, 2022, 17(4): 1477–1502 https://doi.org/10.1007/s11440-021-01440-1
6
H GuoX ZhuangT Rabczuk. A deep collocation method for the bending analysis of Kirchhoff plate. 2021, arXiv: 2102.02617
7
X Zhuang, H Guo, N Alajlan, H Zhu, T Rabczuk. Deep autoencoder based energy method for the bending, vibration, and buckling analysis of Kirchhoff plates with transfer learning. European Journal of Mechanics. A, Solids, 2021, 87: 104225 https://doi.org/10.1016/j.euromechsol.2021.104225
8
E Samaniego, C Anitescu, S Goswami, V M Nguyen-Thanh, H Guo, K Hamdia, X Zhuang, T Rabczuk. An energy approach to the solution of partial differential equations in computational mechanics via machine learning: Concepts, implementation and applications. Computer Methods in Applied Mechanics and Engineering, 2020, 362: 112790 https://doi.org/10.1016/j.cma.2019.112790
9
H Guo, X Zhuang, P Chen, N Alajlan, T Rabczuk. Stochastic deep collocation method based on neural architecture search and transfer learning for heterogeneous porous media. Engineering with Computers, 2022, 38(6): 5173–5198 https://doi.org/10.1007/s00366-021-01586-2
10
D M Katz, M J Bommarito, S Gao, P Arredondo. GPT-4 passes the bar exam. Philosophical Transactions. Series A, Mathematical, Physical, and Engineering Sciences, 2024, 382: 257572753
11
H Strobelt, A Webson, V Sanh, B Hoover, J Beyer, H Pfster, A M Rush. Interactive and visual prompt engineering for AD-HOC task adaptation with large language models. IEEE Transactions on Visualization and Computer Graphics, 2023, 29(1): 1146–1156
12
G Yong, K Jeon, D Gil, G Lee. Prompt engineering for zero-shot and few-shot defect detection and classifcation using a visual-language pretrained model. Computer-Aided Civil and Infrastructure Engineering, 2023, 38(11): 1536–1554 https://doi.org/10.1111/mice.12954
13
T Lubiana, R Lopes, P Medeiros, J C Silva, A N A Goncalves, V Maracaja-Coutinho, H I Nakaya. Ten quick tips for harnessing the power of ChatGPT in computational biology. PLOS Computational Biology, 2023, 19(9): e1011319
14
K BuschA RochlitzerD SolaH Leopold. Just tell me: Prompt engineering in business process management. In: Proceedings of International Conference on Business Process Modeling, Development and Support. Cham: Springer Cham, 2023, 3–11
15
K Zhou, J Yang, C C Loy, Z Liu. Learning to prompt for visionlanguage models. International Journal of Computer Vision, 2022, 130(9): 2337–2348 https://doi.org/10.1007/s11263-022-01653-1
16
M P PolakD Morgan. Extracting accurate materials data from research papers with conversational language models and prompt engineering––Example of ChatGPT. 2023, arXiv: 2303.05352
17
A W LoM Singh. From ELIZA to ChatGPT: The Evolution of NLP and Financial Applications. Cambridge, MA: MIT Libraries, 2023
18
J WangZ LiuL ZhaoZ WuC MaS YuH DaiQ YangY LiuS Zhang, et al.. Review of large vision models and visual prompt engineering. 2023, arXiv: 2307.00855
19
K Hatakeyama-Sato, N Yamane, Y Igarashi, Y Nabae, T Hayakawa. Prompt engineering of GPT-4 for chemical research: What can/cannot be done. Science and Technology of Advanced Materials: Methods, 2023, 3(1): 226030
20
T F Heston. Prompt engineering for students of medicine and their teachers. 2023, arXiv: 2308.11628
21
J J Zhu, J Jiang, M Yang, Z J Ren. ChatGPT and environmental research. Environmental Science & Technology, 2023, 57(46): 17667–17670
22
A Neagu. How can large language models and prompt engineering be leveraged in computer science education? Thesis for the Master’s Degree. Delft: Delft University of Technology, 2023
23
R Peres, M Schreier, D Schweidel, A Sorescu. On ChatGPT and beyond: How generative artificial intelligence may affect research, teaching, and practice. International Journal of Research in Marketing, 2023, 40(2): 269–275 https://doi.org/10.1016/j.ijresmar.2023.03.001
24
Q XiaT MaekawaT Hara. Unsupervised human activity recognition through two-stage prompting with ChatGPT. 2023, arXiv: 2306.02140
25
A K Chopra. Dynamics of Structures: Theory and Applications to Earthquake Engineering. Upper Saddle River, NJ: Prentice Hall, 2006
26
J Wei, X Wang, D Schuurmans, M Bosma, B Ichter, F Xia, E H Chi, Q V Le, D Zhou. Chain-of-thought prompting elicits reasoning in large language models. Advances in Neural Information Processing Systems, 2022, 35: 24824–24837
27
A VaswaniN ShazeerN ParmarJ UszkoreitL JonesA N GomezL KaiserI Polosukhin. Attention is all you need. Advances in Neural Information Processing Systems, 2017: 6000–6010
28
H W ChungL HouS LongpreB ZophY TayW FedusE LiX WangM DehghaniS Brahma, et al.. Scaling instruction-finetuned language models. 2022, arXiv: 2210.11416
29
A RadfordK NarasimhanT SalimansI Sutskever. Improving language understanding by generative pre-training. 2018. Available at the website of OpenAI
30
H TouvronL MartinK StoneP AlbertA AlmahairiY BabaeiN BashlykovS BatraP BhargavaS Bhosale, et al.. Llama 2: Open foundation and fne-tuned chat models. 2023, arXiv: 2307.09288
31
S BernalJ BeckC Ventura. An experimental benchmark problem in structural health monitoring. In: Proceedings of the Third International Workshop on Structural Health Monitoring. Stanford, CA: IWSHM, 2001
32
X Ye, Y Cao, A Liu, X Wang, Y Zhao, N Hu. Parallel convolutional neural network toward high efficiency and robust structural damage identifcation. Structural Health Monitoring, 2023, 22(6): 3805–3826
33
Y Chi, C Cai, J Ren, Y Xue, N Zhang. Damage location diagnosis of frame structure based on wavelet denoising and convolution neural network implanted with inception module and LSTM. Structural Health Monitoring, 2024, 23(1): 57–76
34
S J DykeD BernalJ BeckC Ventura. Experimental phase II of the structural health monitoring benchmark problem. In: Proceedings of the 16th ASCE Engineering Mechanics Conference. Reston, VA: ASCE, 2003
35
E FigueiredoG ParkJ FigueirasC FarrarK Worden. Structural Health Monitoring Algorithm Comparisons Using Standard Data Sets. Technical Report LA-14393. 2009
36
D V Hung, H M Hung, P H Anh, N T Thang. Structural damage detection using hybrid deep learning algorithm. Journal of Science and Technology in Civil Engineering, 2020, 14(2): 53–64
37
S Das, P Saha, S Patro. Vibration-based damage detection techniques used for health monitoring of structures: A review. Journal of Civil Structural Health Monitoring, 2016, 6(3): 477–507 https://doi.org/10.1007/s13349-016-0168-5
38
A KirillovE MintunN RaviH MaoC RollandL GustafsonT XiaoS WhiteheadA C BergW Y Lo, et al.. Segment anything. In: Proceedings of 2023 IEEE/CVF International Conference on Computer Vision (ICCV). Paris: IEEE, 2023
39
S HorawalavithanaE AytonS SharmaS HowlandM SubramanianS VasquezR CosbeyM GlenskiS Volkova. Foundation models of scientific knowledge for chemistry: Opportunities, challenges and lessons learned. In: Proceedings of BigScience Episode #5––Workshop on Challenges & Perspectives in Creating Large Language Models. Dublin: Association for Computational Linguistics, 2022, 160–172
40
X Si, X Wu, H Sheng, J Zhu, Z Li. SeisCLIP: A seismology foundation model pre-trained by multi-modal data for multi-purpose seismic feature extraction. IEEE Transactions on Geoscience and Remote Sensing, 2024, 62: 1–13 https://doi.org/10.1109/TGRS.2024.3354456
