Eye-Shield: Real-Time Protection of Mobile Device Screen Information from Shoulder Surfing

August 6, 2023  •  5 min  • 995 words  •

Tags: Mobile Privacy Security Conference HCI

Pdf Cite Code Slides

Information

Authors

Brian Tang, Kang G. Shin

Conference

32nd USENIX Security Symposium (2023)

Demo

Please note that this is just a prototype demo. Certain UIs may require low protection to be legible, and others may require strong protection to be protected from shoulder surfers. The actual performance of the system may also vary depending on several uncontrollable factors such as your screen pixel density, screen brightness, environment lighting, etc. Please also note that there may be small noisy artifacts from video compression that may not be present in a real deployment of Eye-Shield.

For the best (desktop) experience, select a protection mode, click “Play all”, and view your device from various angles and distances. If the videos buffer, wait for them to load, and then click “Reset all” and “Play all”. For mobile viewing, unfortunately you can only view one video at a time.

Blog

Intro

People use mobile devices ubiquitously for computing, communication, storage, web browsing, and more. As a result, the information accessed and stored within mobile devices, such as financial and health information, text messages, and emails, can often be sensitive. Despite this, people frequently use their mobile devices in public areas, becoming susceptible to a simple yet effective attack – shoulder surfing. Shoulder surfing occurs when a person near a mobile user peeks at the user’s mobile device, potentially acquiring passcodes, PINs, browsing behavior, or other personal information. We propose a novel solution, called Eye-Shield, to prevent shoulder surfers from accessing/stealing sensitive on-screen information.

Design Overview

Eye-Shield is designed to protect all types of on-screen information in real time, without any serious impediment to users’ interactions with their mobile devices. In particular, Eye-Shield generates images that appear readable and interpretable at close distances, but appear blurry or pixelated at farther distances and wider angles. It is capable of protecting on-screen information from shoulder surfers, operating in real time, and being minimally intrusive to the intended users. Eye-Shield achieves this by leveraging the fact that at a sufficient distance, it is impossible for an optical system to distinguish between two nearby light sources. By applying this theory of resolving power ¹, we can construct checkered grids of pixels that can appear individually discernable at a close distance, but appear as a uniform average of the projected colors. The design is inspired by Chen et al.² Fig. 3 provides an overview of the processing required to protect on-screen information. The main components required for the Eye-Shield algorithm are 1) the original screen/image, 2) a checkered grid mask in the dimensions of the original image, and 3) a blurred or pixelated version of the original image. Then, the protected output image will be computed with Algorithm 1.

Algorithm 1

procedure EYE-SHIELD ALGORITHM(img, grid, targ)
// Where img is the original w × h × 3 image
// where grid is a w × h checkered grid of 1s and 0s
// where targ is the w × h × 3 image, blurred or pixelated
complement = (targ^2) · 2 – img^2
delta = (complement – img2^2) · grid
newimg = sqrt(img^2 + delta)
clip (newimg, 0, 255)

The below image demonstrates how the checkered grid pattern can make a line of text appear similar to the blurred version from far away.

The photos below show this effect from a shoulder surfer’s perspective with a camera using 3x optical zoom.

Evaluation

We first wanted to evaluate how effective our approach was at protecting content such as colored images, videos, text, and mobile UIs. Through user study tests, both in-person and crowdsourced, we find shoulder surfers are only able to recognize 25.00% of images and 18.78% of text. The crowdsourced study indicates that shoulder surfers can only recognize 32.24% – 35.50% of the content within images, videos, and texts. Our experiments indicate also Eye-Shield provides additional protection beyond using solely a privacy film.

Next we evaluated whether Eye-Shield met real-time constraints for its screen hiding algorithms. Eye-Shield is capable of achieving smooth performance (43 FPS) at even the highest screen resolutions (3088 × 1440). At lower resolutions, Eye-Shield can achieve 60+ FPS, the optimal performance for screens with a 60Hz refresh rate. While running, Eye-Shield consumes an acceptable amount of memory and CPU while having a moderate impact on power consumption.

Finally we evaluated the usability cost of applying Eye-Shield to users’ smartphones. We found that users who are bothered and uncomfortable with shoulder surfing are more likely to use Eye-Shield, with users on average reacting positively towards the quality of text and images. The usability of the system was deemed above average, with a SUS score of 68.86. Our interviews with 15 participants provide additional insights into the usability of Eye-Shield, discussing concerns related to eye-strain, activation, and use cases.

Discussion

Based on feedback from our user studies, we found Eye-Shield would be best implemented as an OS-level toggle-able feature that users can manually activate and deactivate depending on the private nature of their information and their public location.

Citation

@inproceedings{tang2023eye,
    title={Eye-Shield: Real-Time Protection of Mobile Device Screen Information from Shoulder Surfing},
    author={Tang, Brian and Shin, Kang G.},
    booktitle={32nd USENIX Security Symposium},
    year={2023}
}

Relevant Links

• https://osf.io/haxzb/?view_only=8629a44c79d74b71b603be3830f60583
• https://github.com/byron123t/eye-shield
• Please note that the code is currently private, as we have filed a patent for our system (U.S. Pat. App. No. 63/468,650-Conf. #8672). We may consider releasing it as copyleft or release it if no companies license our patent.

References