Image-based fall detection and classification of a user with a walking support system

doi:10.1007/s11465-017-0465-7

Front. Mech. Eng.

2018, Vol. 13

Issue (3) : 427-441 https://doi.org/10.1007/s11465-017-0465-7

RESEARCH ARTICLE

Image-based fall detection and classification of a user with a walking support system

Sajjad TAGHVAEI¹(

), Kazuhiro KOSUGE²

¹. School of Mechanical Engineering, Shiraz University, Shiraz 71936-16548, Iran
². Department of Bioengineering and Robotics, Tohoku University, Sendai 980-8579, Japan

Download: PDF(824 KB) HTML
Export: BibTeX | EndNote | Reference Manager | ProCite | RefWorks

Abstract

The classification of visual human action is important in the development of systems that interact with humans. This study investigates an image-based classification of the human state while using a walking support system to improve the safety and dependability of these systems. We categorize the possible human behavior while utilizing a walker robot into eight states (i.e., sitting, standing, walking, and five falling types), and propose two different methods, namely, normal distribution and hidden Markov models (HMMs), to detect and recognize these states. The visual feature for the state classification is the centroid position of the upper body, which is extracted from the user’s depth images. The first method shows that the centroid position follows a normal distribution while walking, which can be adopted to detect any non-walking state. The second method implements HMMs to detect and recognize these states. We then measure and compare the performance of both methods. The classification results are employed to control the motion of a passive-type walker (called “RT Walker”) by activating its brakes in non-walking states. Thus, the system can be used for sit/stand support and fall prevention. The experiments are performed with four subjects, including an experienced physiotherapist. Results show that the algorithm can be adapted to the new user’s motion pattern within 40 s, with a fall detection rate of 96.25% and state classification rate of 81.0%. The proposed method can be implemented to other abnormality detection/classification applications that employ depth image-sensing devices.

Keywords fall detection walking support hidden Markov model multivariate analysis

Corresponding Author(s): Sajjad TAGHVAEI

Just Accepted Date: 08 June 2017 Online First Date: 30 October 2017 Issue Date: 11 June 2018

Cite this article:

Sajjad TAGHVAEI,Kazuhiro KOSUGE. Image-based fall detection and classification of a user with a walking support system[J]. Front. Mech. Eng., 2018, 13(3): 427-441.

URL:

https://academic.hep.com.cn/fme/EN/10.1007/s11465-017-0465-7
https://academic.hep.com.cn/fme/EN/Y2018/V13/I3/427

Fig.1 (a) Prototype of the RT Walker; (b) the rear wheel and servo brake system [²⁴]

Fig.2 Depth sensor installed on the RT Walker and the attached coordinate system

Fig.3 Human states while using a walker, as demonstrated by an experienced physiotherapist. Fall side is categorized into fall right and fall left. (a) Walk; (b) sit; (c) stand; (d) fall forward; (e) fall down; (f) fall back; (g) fall side

Fig.4 Framework for the state estimation and control of the walker. PDF: Probability distribution function

Fig.5 (a) Color image captured by Kinect; (b) the segmented user’s mask image obtained by distance slicing

Fig.6 Distribution of the data and contours of constant probability for the normal fitted function in the (a) xy, (b) xz, and (c) xyzplanes. x,y,zare in mm

Fig.7 Distribution of the mapped data and contours of constant probability for the normal fitted function in the (a) XY, (b) XZ, and (c) XYZ planes

Tab.1 Characteristics of the experiment subjects.

Fig.8 Experiments with the RT Walker. (a) 0 s; (b) 6.0 s; (c) 7.8 s; (d) 10.0 s; (e) 14.3 s.

Fig.9 Histograms of the mapped data in the X, Y, and Z values for subject B. (a) User B-X; (b) User B-Y; (c) User B-Z

Fig.10 Q-Q plots of the mapped data in the X, Y, and Z values for subject D. (a) User D-X; (b) User D-Y; and (c) User D-Z

Fig.11 (a) Skewness and (b) kurtosis values of the X, Y, and Z data for all four subjects

Fig.12 Upper body centroid position for all subjects during walking

Fig.13 Variations in the (a) skewness and (b) kurtosis values during the system adaptation of subject C

Fig.14 Variations in the distribution probability (

l o g P x y z

) and brake force f_b for User C. (a) Fall forward; (b) fall down; (c) fall right; (d) fall left; (e) fall back

Fig.15 Variations in the

log P (O | λ 1)

and brake force f_b [N] for user A. (a) Fall forward; (b) fall down; (c) fall right; (d) fall left; (e) fall back

Fig.16 Variations in

l o g P (O | λ 1)

(blue line) and state recognition results (black *) for user B. (a) Fall forward; (b) fall down; (c) fall right; (d) fall left; (e) fall back

Tab.2 Confusion matrix

1	Alami R, Albu-Schaeffer A, Bicchi A, et al. Safe and dependable physical human-robot interaction in anthropic domains: State of the art and challenges. In: Proceedings of IEEE/RSJ International Conference on Intelligent Robots and Systems. Beijing: IEEE, 2006 https://doi.org/10.1109/IROS.2006.6936985
2	Bedaf S, Gelderblom G J, De Witte L. Overview and categorization of robots supporting independent living of elderly people: What activities do they support and how far have they developed. Assistive Technology, 2015, 27(2): 88–100 https://doi.org/10.1080/10400435.2014.978916
3	WHO. Good Health Adds Life to Years. Global Brief for World Health Day 2012. 2012.
4	Stevens J A, Thomas K, Teh L, et al. Unintentional fall injuries associated with walkers and canes in older adults treated in US emergency departments. Journal of the American Geriatrics Society, 2009, 57(8): 1464–1469 https://doi.org/10.1111/j.1532-5415.2009.02365.x
5	Noury N, Fleury A, Rumeau P, et al. Fall detection-principles and methods. In: Proceedings of 29th Annual International Conference of the IEEE on Engineering in Medicine and Biology Society. Lyon: IEEE, 2007: https://doi.org/10.1109/IEMBS.2007.4352627
6	Tong L, Song Q, Ge Y, et al. HMM-based human fall detection and prediction method using tri-axial accelerometer. IEEE Sensors Journal, 2013, 13(5): 1849–1856 doi:10.1109/JSEN.2013.2245231
7	Huynh Q T, Nguyen U D, Irazabal L B, et al. Optimization of an accelerometer and gyroscope-based fall detection algorithm. Journal of Sensors, 2015, 2015: 452078
8	Bourke A K, Lyons G M. A threshold-based fall-detection algorithm using a bi-axial gyroscope sensor. Medical Engineering & Physics, 2008, 30(1): 84–90 https://doi.org/10.1016/j.medengphy.2006.12.001
9	Auvinet E, Multon F, Saint-Arnaud A, et al. Fall detection with multiple cameras: An occlusion-resistant method based on 3-D silhouette vertical distribution. IEEE Transactions on Information Technology in Biomedicine, 2011, 15(2): 290–300 https://doi.org/10.1109/TITB.2010.2087385
10	Hirata Y, Hara A, Kosuge K. Motion control of passive intelligent walker using servo brakes. IEEE Transactions on Robotics, 2007, 23(5): 981–990 https://doi.org/10.1109/TRO.2007.906252
11	Taghvaei S, Kosuge K. HMM-based state classification of a user with a walking support system using visual PCA features. Advanced Robotics, 2014, 28(4): 219–230 https://doi.org/10.1080/01691864.2013.869482
12	Stone E E, Skubic M. Fall detection in homes of older adults using the Microsoft Kinect. IEEE Journal of Biomedical and Health Informatics, 2015, 19(1): 290–301 https://doi.org/10.1109/JBHI.2014.2312180
13	Wang Y, Wu K, Ni L M. WiFall: Device-free fall detection by wireless networks. IEEE Transactions on Mobile Computing, 2016, 16(2): 581–594 https://doi.org/10.1109/TMC.2016.2557792
14	Daher M, El Najjar M E B, Khalil M. Automatic fall detection system using sensing floors. International Journal of Computing and Information Sciences, 2016, 12(1): 75–82 https://doi.org/10.21700/ijcis.2016.110
15	Purwar A, Un Jeong D, Chung W Y. Activity monitoring from real-time triaxial accelerometer data using sensor network. In: Proceedings of International Conference on Control, Automation and Systems. Seoul: IEEE, 2007, 2402–2406 https://doi.org/10.1109/ICCAS.2007.4406764
16	Aguilar P A, Boudy J, Istrate D, et al. A dynamic evidential network for fall detection. IEEE Journal of Biomedical and Health Informatics, 2014, 18(4): 1103–1113 https://doi.org/10.1109/JBHI.2013.2283055
17	Shi G, Chan C S, Li W J, et al. Mobile human airbag system for fall protection using MEMS sensors and embedded SVM classifier. IEEE Sensors Journal, 2009, 9(5): 495–503 https://doi.org/10.1109/JSEN.2008.2012212
18	Zhao K, Jia K, Liu P. Fall detection algorithm based on human posture recognition. In: Pan J S, Tsai P W, Huang H C, eds. Advances in Intelligent Information Hiding and Multimedia Signal Processing. Smart Innovation, Systems and Technologies, Vol 64. Cham: Springer, 2017, 119–126
19	Hsieh C Y, Liu K C, Huang C N, et al. Novel hierarchical fall detection algorithm using a multiphase fall model. Sensors (Basel), 2017, 17(2): 307 https://doi.org/10.3390/s17020307
20	Perry J T, Kellog S, Vaidya S M, et al. Survey and evaluation of real-time fall detection approaches. In: Proceedings of 6th International Symposium on High-Capacity Optical Networks and Enabling Technologies. Alexandria: IEEE, 2009, 158–164 https://doi.org/10.1109/HONET.2009.5423081
21	Rougier C, Meunier J, St-Arnaud A, et al. Fall detection from human shape and motion history using video surveillance. In: Proceedings of 21st International Conference on Advanced Information Networking and Applications Workshops. Niagara Falls: IEEE, 2007, 875–880 https://doi.org/10.1109/AINAW.2007.181
22	Taghvaei S, Jahanandish M H, Kosuge K. Autoregressive moving average hidden Markov model for vision-based fall prediction—An application for walker robot. Assistive Technology, 2017, 29(1): 19–27 https://doi.org/10.1080/10400435.2016.1174178
23	Skubic M, Harris B H, Stone E, et al. Testing non-wearable fall detection methods in the homes of older adults. In: IEEE 38th Annual International Conference of Engineering in Medicine and Biology Society. IEEE, 2016, 557–560 https://doi.org/10.1109/EMBC.2016.7590763
24	Hirata Y, Hara A, Kosuge K. Motion control of passive intelligent walker using servo brakes. IEEE Transactions on Robotics, 2007, 23(5): 981–990 https://doi.org/10.1109/TRO.2007.906252
25	Mazhelis O. One-class classifiers: A review and analysis of suitability in the context of mobile-masquerader detection. South African Computer Journal, 2007, 36: 29–48
26	Yamato J, Ohya J, Ishii K. Recognizing human action in time-sequential images using hidden Markov model. In: Proceedings of IEEE Computer Society Conference on Computer Vision and Pattern Recognition. IEEE, 1992
27	Goswami A, Peshkin M A, Colgate J E. Passive robotics: An exploration of mechanical computation. In: Proceedings of IEEE International Conference on Robotics and Automation. Cincinnati: IEEE, 1990, 279–284 https://doi.org/10.1109/ROBOT.1990.125987
28	Hirata Y, Komatsuda S, Kosuge K. Fall prevention control of passive intelligent walker based on human model. In: Proceedings of IEEE/RSJ International Conference on Intelligent Robots and Systems. Nice: IEEE, 2008, 1222–1228 https://doi.org/10.1109/IROS.2008.4651173
29	Gonzalez R C, Woods R E. Image Processing. Digital Image Processing. Upper Saddle River: Addison-Wesley Publishing Co., Inc., 1977
30	Rencher A C. Methods of Multivariate Analysis. 2nd ed. New York: John Wiley & Sons, 2003
31	Rabiner, L, Juang B. An introduction to hidden Markov models. IEEE ASSP Magazine, 1986, 3(1): 4–16 https://doi.org/10.1109/MASSP.1986.1165342

Viewed

Full text

Abstract

Cited

Shared

Discussed