Yang, LichaoShan, XiaocaiLv, ChenBrighton, JamesZhao, Yifan2021-08-162021-08-162021-07-28Yang L, Shan X, Lv C, et al., (2022) Learning spatio-temporal representations with a dual-stream 3-D residual network for nondriving activity recognition. IEEE Transactions on Industrial Electronics, Volume 69, Number 7, July 2022, pp. 7405-74140278-0046https://doi.org/10.1109/TIE.2021.3099254https://dspace.lib.cranfield.ac.uk/handle/1826/16995Accurate recognition of non-driving activity (NDA) is important for the design of intelligent Human Machine Interface to achieve a smooth and safe control transition in the conditionally automated driving vehicle. However, some characteristics of such activities like limited-extent movement and similar background pose a challenge to the existing 3D convolutional neural network (CNN) based action recognition methods. In this paper, we propose a dual-stream 3D residual network, named D3D ResNet, to enhance the learning of spatio-temporal representation and improve the activity recognition performance. Specifically, a parallel 2-stream structure is introduced to focus on the learning of short-time spatial representation and small-region temporal representation. A 2-feed driver behaviour monitoring framework is further build to classify 4 types of NDAs and 2 types of driving behaviour based on the drivers head and hand movement. A novel NDA dataset has been constructed for the evaluation, where the proposed D3D ResNet achieves 83.35% average accuracy, at least 5% above three selected state-of-the-art methods. Furthermore, this study investigates the spatio-temporal features learned in the hidden layer through the saliency map, which explains the superiority of the proposed model on the selected NDAs.enAttribution-NonCommercial 4.0 Internationalaction recognitionnon-driving related taskautomated drivingArticle