Reconstructing what you said: Text Inference using Smartphone Motion

Abstract

Smartphones and tablets are becoming ubiquitous within our connected lives and as a result these devices are increasingly being used for more and more sensitive applications, such as banking. The security of the information within these sensitive applications is managed through a variety of different processes, all of which minimise the exposure of this sensitive information to other potentially malicious applications on the device. This paper documents experiments with motion sensors on the device as a side-channel for inferring the text typed into a sensitive application. These sensors are freely accessible without the phone user having to give permission. The research was able to, on average, identify nearly 30% of typed bigrams from unseen words, using a very small volume of training data, less than the size of a tweet. Given the redundancy in language this performance is often enough to understand the phrase being typed. We found that large devices were more vulnerable than small devices, as were users who held the device in one hand whilst typing with fingers. Of those bigrams which were incorrectly identified 60% of the errors involved the space bar and nearly half of the errors are within two keys on the keyboard.