December 12, 2022

Eye-Shield: Real-Time Protection of Mobile Device Screen Information from Shoulder Surfing

Posted on December 12, 2022  •  4 minutes  • 781 words  •

Information

Authors

Brian Tang, Kang G. Shin

Conference

Accepted at 32nd USENIX Security Symposium (2023)

Blog

Intro

People use mobile devices ubiquitously for computing, communication, storage, web browsing, and more. As a result, the information accessed and stored within mobile devices, such as financial and health information, text messages, and emails, can often be sensitive. Despite this, people frequently use their mobile devices in public areas, becoming susceptible to a simple yet effective attack – shoulder surfing. Shoulder surfing occurs when a person near a mobile user peeks at the user’s mobile device, potentially acquiring passcodes, PINs, browsing behavior, or other personal information. We propose a novel solution, called Eye-Shield, to prevent shoulder surfers from accessing/stealing sensitive on-screen information.

Design Overview

Eye-Shield is designed to protect all types of on-screen information in real time, without any serious impediment to users’ interactions with their mobile devices. In particular, Eye-Shield generates images that appear readable and interpretable at close distances, but appear blurry or pixelated at farther distances and wider angles. It is capable of protecting on-screen information from shoulder surfers, operating in real time, and being minimally intrusive to the intended users. Eye-Shield achieves this by leveraging the fact that at a sufficient distance, it is impossible for an optical system to distinguish between two nearby light sources. By applying this theory of resolving power 1, we can construct checkered grids of pixels that can appear individually discernable at a close distance, but appear as a uniform average of the projected colors. The design is inspired by Chen et al.2 Fig. 3 provides an overview of the processing required to protect on-screen information. The main components required for the Eye-Shield algorithm are 1) the original screen/image, 2) a checkered grid mask in the dimensions of the original image, and 3) a blurred or pixelated version of the original image. Then, the protected output image will be computed with Algorithm 1.

Algorithm 1

procedure EYE-SHIELD ALGORITHM(img, grid, targ)
// Where img is the original w × h × 3 image
// where grid is a w × h checkered grid of 1s and 0s
// where targ is the w × h × 3 image, blurred or pixelated
complement = (targ^2) · 2 – img^2
delta = (complement – img2^2) · grid
newimg = sqrt(img^2 + delta)
clip (newimg, 0, 255)

The below image demonstrates how the checkered grid pattern can make a line of text appear similar to the blurred version from far away.

The photos below show this effect from a shoulder surfer’s perspective with a camera using 3x optical zoom.

Evaluation

We first wanted to evaluate how effective our approach was at protecting content such as colored images, videos, text, and mobile UIs. Through user study tests, both in-person and crowdsourced, we find shoulder surfers are only able to recognize 25.00% of images and 18.78% of text. The crowdsourced study indicates that shoulder surfers can only recognize 32.24% – 35.50% of the content within images, videos, and texts. Our experiments indicate also Eye-Shield provides additional protection beyond using solely a privacy film.

Next we evaluated whether Eye-Shield met real-time constraints for its screen hiding algorithms. Eye-Shield is capable of achieving smooth performance (43 FPS) at even the highest screen resolutions (3088 × 1440). At lower resolutions, Eye-Shield can achieve 60+ FPS, the optimal performance for screens with a 60Hz refresh rate. While running, Eye-Shield consumes an acceptable amount of memory and CPU while having a moderate impact on power consumption.

Finally we evaluated the usability cost of applying Eye-Shield to users’ smartphones. We found that users who are bothered and uncomfortable with shoulder surfing are more likely to use Eye-Shield, with users on average reacting positively towards the quality of text and images. The usability of the system was deemed above average, with a SUS score of 68.86. Our interviews with 15 participants provide additional insights into the usability of Eye-Shield, discussing concerns related to eye-strain, activation, and use cases.

Discussion

Based on feedback from our user studies, we found Eye-Shield would be best implemented as an OS-level toggle-able feature that users can manually activate and deactivate depending on the private nature of their information and their public location.

Citation

@inproceedings{tang2023eye,
    title={Eye-Shield: Real-Time Protection of Mobile Device Screen Information from Shoulder Surfing},
    author={Tang, Brian and Shin, Kang G.},
    booktitle={32nd USENIX Security Symposium},
    year={2023}
}

https://osf.io/haxzb/?view_only=8629a44c79d74b71b603be3830f60583

References


  1. Singh, D. (2015). Fundamentals of optics. PHI Learning Pvt. Ltd.. Paper Link  ↩︎

  2. Chen, C. Y., Lin, B. Y., Wang, J., & Shin, K. G. (2019, August). Keep others from peeking at your mobile device screen!. In The 25th Annual International Conference on Mobile Computing and Networking (pp. 1-16). Paper Link  ↩︎

Follow or contact me

I publish and open-source my work. I also occasionally post random thoughts.