Several studies have delved into the research of how Kinect can be applied to skele-tal tracking. Tao et al (2013), researched on the kinematic validity of Microsoft
Kinect in skeletal tracking for application in virtual limb rehabilitation. The research investigated the extent to which Kinect can be used to track hand position, limbs, ankles and body trunk. For the experiment, cameras were positioned in the range of 1.45m and 1.75m. The goal was the experiment was to determine the extent to which the Kinect sensor can be used for limb rehabilitation through use of preset and repeti-tive tasks. Additionally, the precision of the Kinect sensor is determined and ana-lyzed.
For the experiment, the researchers used Optotrack 3-D motion capture system which was placed on different locations. The participating individuals then performed dif-ferent movements such as; leaning backwards, elbow flexing and trunk leaning. All this was captured by a Kinect sensor placed at a height of 135 cm.
The results obtained from the experiment showed a simultaneous comparison be-tween the sensor result and the motion capture system. The error of the mean squared difference was then obtained. This showed that for the hand movements, the error was 6.3cm and the variable error was at 2.4 cm. The constant error in all positions was found to be less than 9.8cm. For trunk movements, obtained mean error was at 3.9 cm with a variable of 2.5cm.
Below figure shows the constant camera error with respect to the distance;
Figure 49- Tao et al (2013) constant camera error
Overall, the data obtained from the sensor, closely matched that from the Optotrack motion tracker except that of the elbow. The elbow tracking showed varying results because of the limitations of the Kinect sensor in modeling of the elbow.
The research concluded that the appropriate location of the camera with respect to Kinect Skeletal tracking ought to be at 30*30 square and 1.45m and 1.75m from the user, the camera can also vary with a distance of 0.15m to the left or to the right.
Figure 50- Tao et al (2013) variable camera error
In regards to geometric refinement and pose estimation, there was a research conducted by Choe et al (2014), the researcher aim was to improve the accuracy of Kinect camera for depth recognition and image reconstruction. The researchers used a 3-D mesh to optimize the geometric refinement process. The approach the researchers toom does not require additional Kinect camera or complex setup.
Effectively, the researchers were able to utilize shading information to perfect the geometry refinement. The researcher used different lighting conditions to verify the invariability of Kinect IR images.
Figure 51- Choe et al (2014) invariability of IR images and RGB under different lighting conditions
The data capturing process the researchers used consisted of discrete IR shading im-age acquisition. The Kinect fusion was used to obtain the first mesh as shown in fig-ure-53 below.
The Kinect SDK is used to register depth map with a reconstructed surface.
Figure 52- Choe et al (2014) Data capturing system, used to obtain the base mesh
Figure 53- Choe et al (2014) input shading image, projected mesh and depth map
The research demonstrated that the captured IR images do not result in any overlap-ping visible spectrum. The researcher also described a method of radiometrically calibrating the Kinect IR. The research assumes that Lambertian BRDF which made the result erratic.
5 Findings and conclusion
5.1 Findings
Microsoft Kinect presents and important technology that can be used in a wide array of applications including assisting patients in assisted living environments. A lot of research and experiments have been conducted to test the clinical and technical ca-pability of the Kinect in physical therapy and body parts rehabilitation. Documentary analysis conducted in chapter 4 covered some of the biggest and most important ex-periments conducted to assess the technical capabilities of Kinect components such as the IR camera. Increase in research interest in Microsoft signifies its relevance in applications such as use in assisted living environments.
For ease of research, studies of the application of Kinect in Assisted Living environ-ments can be classified into;
1. Experiments that evaluated accuracy, reliability and precision of Microsoft Kinect
2. Experiments that evaluated the application of Kinect in Clinical settings and Smart Home environments
3. Experiments that investigated use of Kinect for Movement Detection Models 4. Experiments that investigated use of Kinect in Skeletal tracking systems Normally, in assisted living environments assessment is done by people who can be doctors, nurses or hospital volunteers. This means that the assessment relies heavily on a human touch which means higher labor costs and low scalability. For instance, an activity like therapy would require a specialised/trained Physical therapist (PT) or Occupational therapist (OCT). Given that these kinds of clinical assessment can be done by people, it is subject to errors and inaccuracies.
To solve this problem, researchers are testing motion sensors. Notably, motion sen-sors have in the past few years received significant interest because of their afforda-bility and practicality. The commonly technologies used for motion sensing are opto-electronics and nonoptoopto-electronics sensors. While optoopto-electronics use markers, nonoptoelctronics do not. In instances where markers are used, they are placed on the bodies of the individuals which are then tracked by a camera sensor. Where markers are not used the sensors apply inertia, mechanical and magnetic techniques to track motion.
Our findings show that Kinect can be used both for optoelectronics and nonoptoelc-tronics experiments. For inertia systems as seen in chapter-4, researchers use sensor fusion algorithms and human skeletal algorithms. Magnetic systems on the other hand use motion capture technologies to transmit and receive signals that can be used
for position, orientation and pose of receiver. In studied experiments, we have seen that the sensors are 6 DoF per receiver which is able to provide 3-D positioning.
Findings also show that Kinect can also be used in collaboration with wearable tech-nologies such as wearable sensors, smart suits and music gloves. These devices to-gether with Kinect sensors are able to follow the user’s motion passively or actively.
Review of visual based motion trackers show that they either use contrast based or depth based imaging. Contrast based systems work by tracking different colour markers attached to the bodies or hands that are being tracked. Depth-sensing sys-tems use depth imagery segmentation and vision algorithms to track and detect hu-man motions.
Importance of the Microsoft Kinect compared to other previous camera
Compared to other cameras, we found that Kinect has a lot of advantages and fea-tures that make it ideal for motion tracking. For instance, Microsoft Kinect provides a Software Development Kit that gives developers important access to body joint positions.
Specification of the Kinect that are ideal for motion tracking include; RGB camera, multi-array microphone, infrared projector and CMOS sensor. According to the ex-periments analysed in chapter 4, it was found that truly, Kinect sensor can handle both depth and infrared streams at 640X480 pixels which can be increased when needed to 1280X1024. The stream supports 8-bit resolution and can accommodate VGA or UYVY colour format.
The senor can be adjusted to near range or default range. At near range people within 0.4-3m are visible while in default range visibility is at 0.8m-2.5m. The microphone is capable of processing 4 channels of 16-but audio at a rate of 16 kHz. The sensor can visualize 6 people but is only capable of tracking 2 people at a time.
Reliability and Accuracy of Kinect
From analyse papers, it’s obvious that a lot of researchers have tried to ascertain the reliability and accuracy of Kinect sensor. Generally, most researchers agree that Ki-nect is good as a motion capture mainly because it’s easily available and affordable.
However, researchers point out that the technology suffers from occlusion. It has been observed that at time, the Kinect sensor would recognise chair legs like they are human legs. This means that for successful tracking, problems brought about by oc-clusion need to be effectively addressed.
Important to note, accuracy tests of the Kinect show that its sensors are accurate enough for use in smart living or assisted living environments. In a trial to test Kinect application in assisted living environments, Dutta (2014) compared Kinect to Vicon
in the tracking of motion. The result of the research showed that in monitoring of elderly falling Kinect was accurate enough to be used. In a different research on the accuracy, Kurillo et al (2013) found that Microsoft Kinect provided more reliability compared to MoCap system. In terms of range of motion measurements, Kinect proved to be a more accurate measure compared to MoCap. This is at the backdrop of research in different areas such as hip abduction, elbow flexing, knee flexing and shoulder abductions. Other researchers like Hawi et al (2014) showed that Kinect had an exemplary test-retest reliability but had low accuracy compared to goniometers.
The most important finding from all the literature sets studied in Chapter-4 was that Kinect can be reliably used as a depth sensor. However, developers should factor in occlusion issues and the noise usually experienced in skeletal tracking. Researchers also agree that to solve most of the challenges presented by Kinect Kalman filters, sensor fusion and calibration should be used.
Findings on Application of Kinect to patients with Neurological Disorders
Key application highlighted in this research is the use of Kinect in assisted living environments. Assisted living environments usually has patients with different needs like those with chronic diseases that require specialised care. Researchers such as Llorens et al (2013) have pioneered research in this area with encouraging findings.
The researcher created a game that promote rehabilitation activities in patients with Neurological Disorders. Clinical tests of the game showed significant improvements in body balance and mobility of the patients.
Research conducted by Exell et al (2014) showed that electrical simulation could be used to rehabilitate a patient’s arm. In conclusion, the researcher insisted that Kinect is accurate enough although it needs further research.
5.2 Conclusions
This paper reviewed different literature and experiment results on Microsoft Kinect in the field for assisted living. First, similar experiments using other technologies were reviewed that aimed to provide solutions in assisted living environments. Limi-tations and errors presented by these systems were analyzed and discussed. Previous systems used in motion tracking were not as effective as Kinect. These systems were only able to track specific body parts like palm, hand and face. The systems were also not interactive and did not provide ease in programming the way the Kinect pro-vides the Software Development Kit (SDK) which propro-vides programmable access to skeletal tracking.
The arrival of the Kinect has ushered a new age for motion sensors. Today, there exist numerous research on the application of Kinect in motion sensing and in smart environments. Kinect has proven to be more accurate, precise and reliable compared to RGB systems. However, Kinect is not without limitation and faults. Issues such as occlusion and noise still exist and require improvement. In assisted living environ-ments, these problems can be significantly reduced by Kalman filtering, calibration and sensor fusion.
Further, this researched discussed evaluation and performance of Kinect in assisted living environments with patients requiring different levels of attention. Studies in this area targeted different monitoring architectures and infrastructure in both home environments and in specialized hospital monitoring environments. Experiments uti-lized different body movements, games, cognitive therapy and exercises. Some ex-periments resulted into successful assessment of falls, movements and even postures.
However, other studies lacked clinical evaluation of the results which raises ques-tions on the effectiveness of the experiments.
In addition, this research compared Kinect with other sensor technologies both as a whole and in the form of component by component. Examples of other analyzed de-vices included Leap motion, Asus Xtion and Intel Creative Cameras. Although the different cameras were suited for different small functions, Kinect proved to be the better option for full body tracking.
The rapid growth in the field of smart environment and assisted living, and the con-tinuous advancement in the field of artificial intelligence, have opened the possibility for many options for further work in this study. A viable option is testing the Mi-crosoft Kinect sensor with the Smart Environment for Assisted Living (SEAL) appli-cations. In this case, Kinect could be the eyes and brains of the SEAL app, helping to monitor the patient in real time. It will also be sending real time updates about the patient into the SEAL app, and activating the SEAL app alarm in cases where by the patient seems to be in danger. The alarm could be calling for help whenever the
pa-tient falls and he/she is not able to help him/herself up, and it could also send mes-sages to the doctors or nurses when no activity (movement, breathing, etc.) is record-ed from patient for a period. Kinect sensor could also be usrecord-ed for other machine learning related studies.
References
Hemant Ghayvat, Jie Liu, Subhas Chandra Mukhopadhyay, and Xiang Gui “Well-ness Sensor Networks: A Proposal and Implementation for Smart Homefor Assisted Living” IEEE SENSORS JOURNAL, VOL. 15, NO. 12, PP 17341-17344 December 2015.
Lin Yang, Longyu Zhang, Haiwei Dong, Abdulhameed Alelaiwi, and Abdulmotaleb El Saddik “Evaluating and Improving the Depth Accuracy of Kinect for Windows v2” IEEE SENSORS JOURNAL, VOL. 15, NO. 8, PP 4275-4277August 2015.
Marek R. Ogiela, Tomasz Hachaj and Katarzyna Koptyra “Effectiveness comparison of Kinect and Kinect 2 for recognition of Oyama karate techniques” 18th Interna-tional Conference on Network-Based Information Systems, 2015.
Teng Deng1, Hui Li1, Jianfei Cai1, Tat-Jen Cham1, Henry Fuchs “Kinect Shadow Detection and Classification” 2013 IEEE International Conference on Computer Vi-sion Workshops
Joshua Fabian, Tyler Young and James C. Peyton Jones “Integration of Microsoft Kinect With Simulink: Real-Time Object Tracking Example” EEE/ASME Transac-tions On Mechatronics, vol. 19, no. 1, February 2014
Zhang, Z., 2012. Microsoft kinect sensor and its effect. IEEE multimedia, 19(2), pp.4-10.
Han, J., Shao, L., Xu, D. and Shotton, J., 2013. Enhanced computer vision with mi-crosoft kinect sensor: A review. IEEE transactions on cybernetics, 43(5), pp.1318-1334.
Lange, B., Chang, C.Y., Suma, E., Newman, B., Rizzo, A.S. and Bolas, M., 2011, August. Development and evaluation of low cost game-based balance rehabilitation tool using the Microsoft Kinect sensor. In Engineering in medicine and biology soci-ety, EMBC, 2011 annual international conference of the IEEE (pp. 1831-1834).
IEEE.
Stowers, J., Hayes, M. and Bainbridge-Smith, A., 2011, April. Altitude control of a quadrotor helicopter using depth map from Microsoft Kinect sensor. In Mechatronics (ICM), 2011 IEEE International Conference on (pp. 358-362). IEEE.
Galna, B., Barry, G., Jackson, D., Mhiripiri, D., Olivier, P. and Rochester, L., 2014.
Accuracy of the Microsoft Kinect sensor for measuring movement in people with Parkinson's disease. Gait & posture, 39(4), pp.1062-1068.
Villaroman, N., Rowe, D. and Swan, B., 2011, October. Teaching natural user inter-action using OpenNI and the Microsoft Kinect sensor. In Proceedings of the 2011 conference on Information technology education (pp. 227-232). ACM.
Azzari, G., Goulden, M.L. and Rusu, R.B., 2013. Rapid characterization of vegeta-tion structure with a microsoft kinect sensor. Sensors, 13(2), pp.2384-2398.
Chang, C.Y., Lange, B., Zhang, M., Koenig, S., Requejo, P., Somboon, N., Sawchuk, A.A. and Rizzo, A.A., 2012, May. Towards pervasive physical rehabilitation using Microsoft Kinect. In Pervasive Computing Technologies for Healthcare (PervasiveHealth), 2012 6th International Conference on (pp. 159-162). IEEE.
Biswas, K.K. and Basu, S.K., 2011, December. Gesture recognition using microsoft kinect®. In Automation, Robotics and Applications (ICARA), 2011 5th International Conference on (pp. 100-103). IEEE.
El-laithy, R.A., Huang, J. and Yeh, M., 2012, April. Study on the use of Microsoft Kinect for robotics applications. In Position Location and Navigation Symposium (PLANS), 2012 IEEE/ION (pp. 1280-1288). IEEE.
Gonzalez-Jorge, H., Riveiro, B., Vazquez-Fernandez, E., Martínez-Sánchez, J. and Arias, P., 2013. Metrological evaluation of microsoft kinect and asus xtion sen-sors. Measurement, 46(6), pp.1800-1806.
Moazzam, I., Kamal, K., Mathavan, S., Usman, S. and Rahman, M., 2013, October.
Metrology and visualization of potholes using the microsoft kinect sensor.
In Intelligent Transportation Systems-(ITSC), 2013 16th International IEEE Confer-ence on (pp. 1284-1291). IEEE.
Kawatsu, C., Li, J. and Chung, C.J., 2013. Development of a fall detection system with Microsoft Kinect. In Robot Intelligence Technology and Applications 2012 (pp.
623-630). Springer Berlin Heidelberg.
Kawatsu, C., Li, J. and Chung, C.J., 2013. Development of a fall detection system with Microsoft Kinect. In Robot Intelligence Technology and Applications 2012 (pp.
623-630). Springer Berlin Heidelberg.
Weerasinghe, I.T., Ruwanpura, J.Y., Boyd, J.E. and Habib, A.F., 2012. Application of Microsoft Kinect sensor for tracking construction workers. In Construction Re-search Congress 2012: Construction Challenges in a Flat World (pp. 858-867).
Araujo, R.M., Graña, G. and Andersson, V., 2013, March. Towards skeleton bio-metric identification using the microsoft kinect sensor. In Proceedings of the 28th Annual ACM Symposium on Applied Computing (pp. 21-26). ACM.
Bevilacqua, V., Nuzzolese, N., Barone, D., Pantaleo, M., Suma, M., D'Ambruoso, D., ... & Stroppa, F. (2014, June). Fall detection in indoor environment with kinect sensor. In Innovations in Intelligent Systems and Applications (INISTA) Proceedings, 2014 IEEE International Symposium on (pp. 319-324). IEEE.
Konstantinidis, E. I., Antoniou, P. E., Bamparopoulos, G., & Bamidis, P. D. (2015).
A lightweight framework for transparent cross platform communication of controller data in ambient assisted living environments. Information Sciences, 300, 124-139.
Anastasiou, D. (2011, May). Gestures in assisted living environments.
In International Gesture Workshop (pp. 1-12). Springer, Berlin, Heidelberg.
Baeg, S. H., Park, J. H., Koh, J., Park, K. W., & Baeg, M. H. (2007, October). Build-ing a smart home environment for service robots based on RFID and sensor net-works. In Control, Automation and Systems, 2007. ICCAS'07. International Confer-ence on (pp. 1078-1082). IEEE.
Hondori, H.M., Khademi, M., Dodakian, L., Cramer, S.C. and Lopes, C.V., 2013. A spatial augmented reality rehab system for post-stroke hand rehabilitation.
In MMVR (pp. 279-285).
Mohamed, A.B.H., Val, T., Andrieux, L. and Kachouri, A., 2013, January. Assisting people with disabilities through Kinect sensors into a smart house. In Computer Med-ical Applications (ICCMA), 2013 International Conference on (pp. 1-5). IEEE.
Chin, L.C., Basah, S.N., Yaacob, S., Din, M.Y. and Juan, Y.E., 2015, March. Accu-racy and reliability of optimum distance for high performance Kinect Sensor.
In Biomedical Engineering (ICoBE), 2015 2nd International Conference on (pp. 1-7). IEEE.
Alexiadis, D.S., Chatzitofis, A., Zioulis, N., Zoidi, O., Louizis, G., Zarpalas, D. and Daras, P., 2017. An Integrated Platform for Live 3D Human Reconstruction and Mo-tion Capturing. IEEE TransacMo-tions on Circuits and Systems for Video Technolo-gy, 27(4), pp.798-813.
Tahavori, F., Alnowami, M., Jones, J., Elangovan, P., Donovan, E. and Wells, K., 2013, October. Assessment of Microsoft Kinect technology (Kinect for Xbox and
Kinect for Windows) for patient monitoring during external beam radiotherapy.
In Nuclear Science Symposium and Medical Imaging Conference (NSS/MIC), 2013 IEEE (pp. 1-5). IEEE.
Sengupta, K. and Ohya, J., 1996, November. Pose estimation of human body part using multiple cameras. In Robot and Human Communication, 1996., 5th IEEE In-ternational Workshop on (pp. 146-151). IEEE.
Tao, G., Archambault, P.S. and Levin, M.F., 2013, August. Evaluation of Kinect skeletal tracking in a virtual reality rehabilitation system for upper limb hemiparesis.
In Virtual Rehabilitation (ICVR), 2013 International Conference on (pp. 164-165).
IEEE.
Choe, G., Park, J., Tai, Y.W. and So Kweon, I., 2014. Exploiting shading cues in kinect ir images for geometry refinement. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (pp. 3922-3929).