Modern touchscreens utilize capacitive sensing technology to enable precise and robust multi-touch interaction. However, the broader expressive potential of the human hand remains underutilized, since most existing methods directly filter out larger-area hand–screen contact. This paper introduces Magical Touch, an interaction method based on raw capacitive sensing data. By directly integrating raw touchscreen sensor data into the interaction loop, our method allows users to interact with the screen naturally and efficiently using arbitrary hand gestures on existing touchscreen devices.
To demonstrate the feasibility and expressive capacity of this approach, we implement a physics-based interactive game featuring single-player, multiplayer collaborative, and pressure-sensitive modes. These scenarios showcase how digital objects can respond in real-time to both the geometry and contact intensity of the user's hand. Our results indicate that leveraging raw capacitive data can expand the design space of touchscreen interaction, offering an embodied and continuous interaction paradigm beyond existing fingertip-based approaches.
Photos from CHI 2026, Barcelona.
@inproceedings{10.1145/3772363.3799129,
title = {Magical Touch: Exploring Touchscreen Interaction
Based on Raw Capacitive Sensing Data},
author = {Guo, Yuanlei and Gong, Xizi and Zhang, Yizhong and Zhang, Xiaoyu},
booktitle = {Extended Abstracts of the 2026 CHI Conference on
Human Factors in Computing Systems (CHI EA '26),
April 13--17, 2026, Barcelona, Spain},
year = {2026},
publisher = {ACM},
isbn = {979-8-4007-2281-3/2026/04},
doi = {10.1145/3772363.3799129},
license = {CC BY-NC-ND}
}
We are grateful to Federico Zannier and Jonathan Westhues for their help in developing the firmware for capturing raw touch data on Microsoft Surface, and to Qiaochu Wang for his suggestions to improve the demo design.
We also thank the participants who volunteered for our pilot study and the anonymous reviewers for their feedback and suggestions that helped improve the quality of this paper.
This research is sponsored in part by the CityUHK Start-Up Grant (Project No. 9610708).