Project Theme 1: Pervasive Sensing
Project Theme 1: Pervasive Sensing
NSF I-Corps 2024, Hellman Fellows Award 2023, AiFi Gifts 2022, CITRIS Core Seed Award 2020 / CITRIS Foundry 2023, Academic Senate Faculty Research Grants Program 2020/2021
Embedded Files
Vibrations of Everything
Vibrations of Everything
Physical and physiological information is essential for indoor human monitoring applications. In order to capture such information non-intrusively, we utilize/design sensors that can capture the vibration induced by people/body parts to infer a variance of activities (over different scales, both infrastructure and wearable) in the physical world. We explore new opportunities to utilize complementary information over different scales (on-body, room, city, etc.) to achieve accurate inference.
In-Home Activities of Daily Living (ADL) Monitoring
Latest Paper:
Zhizhang Hu, Yue Zhang, Tong Yu, and Shijia Pan. "VMA: Domain Variance-and Modality-Aware Model Transfer for Fine-Grained Occupant Activity Recognition." In 2022 21st ACM/IEEE International Conference on Information Processing in Sensor Networks (IPSN), pp. 259-270. IEEE, 2022.
Retainer-Formed Sensor Capturing Oral Activities-Induced Vibration
Latest Papers:
Zhizhang Hu, Amir Mohammad Radmehr, Yue Zhang, Shijia Pan, and Phuc Nguyen. "IOTeeth: Intra-Oral Teeth Sensing System for Dental Occlusal Diseases Recognition." Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies 8, no. 1 (2024): 1-29.
Shijia Pan, Dong Yoon Lee, Jun Ho Lee, and Phuc Nguyen. "TeethVib: Monitoring Teeth Functional Occlusion Through Retainer Vibration Sensing." In 2021 IEEE/ACM Conference on Connected Health: Applications, Systems and Engineering Technologies (CHASE), pp. 92-96. IEEE, 2021.
Repurpose Vibrotactile Intervention for Fine-grained Muscle Profiling
Latest Paper:
Shubham Rohal, Shreya Shriram, V. P. Nguyen, and Shijia Pan. "FinePose: Fine-Grained Postural Muscle Profiling via Haptic Vibration Signals." In Proceedings of the 2022 Workshop on Body-centric Computing Systems, pp. 19-24. 2022.
Award: Best Poster Award IPSN'22, NSF iCorp
🚀 Alumni Spin-off: PosTrue
Structure-Augmented Intelligent Surfaces
Structure-Augmented Intelligent Surfaces
To push the boundary of ambient information inference via surface sensing, we further explore methods and apparatus for augmenting the ability to acquire and recognize human-induced surface deformation by changing the physical properties of the surface.
Augment rigid surface with LEGO® bricks enabled metasurface
Latest Paper:
Yue Zhang, Shikha Patel, Dong Yoon Lee, Paolo Celli, Amelie Bonde, and Shijia Pan. 2023. Poster Abstract: LEVO: LEGO® Bricks Enhanced Single-Point Vibration Sensing for Occupant Monitoring. In The 21st ACM Conference on Embedded Networked Sensor Systems (SenSys ’23), November 12–17, 2023, Istanbul, Turkiye. ACM, New York, NY, USA, 2 pages.
Award: Best Poster Award SenSys'23
Augment elastic surface with Miura-Ori Origami metastructure
Latest Papers:
Shubham Rohal, Dong Yoon Lee, Carlos Ruiz, Joshua Zhang, Jonathon Fagert, Jun Han, and Shijia Pan. "Don’t Crosstalk to Me: Origami Structure-Augmented Sensing for Scalable Surface Pressure Monitoring." In Proceedings of the 22nd ACM Conference on Embedded Networked Sensor Systems, pp. 493-506. 2024.
Shubham Rohal, Carlos Ruiz, Yue Zhang, and Shijia Pan. "MOOCA: Muira-Ori Origami-Based Configurable Shelf-Liner for Autonomous Retail." In Proceedings of the 20th ACM Conference on Embedded Networked Sensor Systems, pp. 776-777. 2022.
Patent: "Programmable smart-sensing shelf liner apparatus for autonomous retail." U.S. Patent Application No. 18/487,059.
Award: Best Demo Award SenSys'22
Augment rigid surface with controlled propagation path
Latest Paper:
Yue Zhang, Carlos Ruiz, Shubham Rohal, and Shijia Pan. "CPA: Cyber-Physical Augmentation for Vibration Sensing in Autonomous Retails." In Proceedings of the 24th International Workshop on Mobile Computing Systems and Applications, pp. 8-14. 2023.
Project Theme 2: Ambient Intelligence
Project Theme 2: Ambient Intelligence
ORED Proof of Concept Competition 2025, CAHSI-Google Institutional Research Program 2024, F3 R&D GSR Award 2024, Climate Action Competition Award 2023, CITRIS Core Seed Award 2022
Cross-Modal Association
Cross-Modal Association
Multiple co-located sensing modalities capture the same physical event from different aspects and provide complementary information for inference. However, to accurately associate signals from different modalities capturing the same event is challenging, especially when the modalities are significantly different.
Latest Papers:
Yue Zhang, Zhizhang Hu, Uri Berger, and Shijia Pan. "CMA: Cross-Modal Association Between Wearable and Structural Vibration Signal Segments for Indoor Occupant Sensing." In The 22nd International Conference on Information Processing in Sensor Networks, pp. 96-109. 2023.
Yue Zhang, Zhizhang Hu, Uri Berger, and Shijia Pan. "Poster Abstract: Integrating On-and Off-body Sensing for Young Adults Failure to Launch (FTL) Behavior Profiling." In Proceedings of the 22nd International Conference on Information Processing in Sensor Networks, pp. 320-321. 2023.
Yue Zhang, Zhizhang Hu, Susu Xu, and Shijia Pan. "AutoQual: Task-Oriented Structural Vibration Sensing Quality Assessment Leveraging Co-located Mobile Sensing Context." CCF Transactions on Pervasive Computing and Interaction 3 (2021): 378-396.
Award: Best Poster Award Runner-Up IPSN'23
Adaptation in Sensing
Adaptation in Sensing
Data acquisition quality directly affects the information representative and the model accuracy in applications of real-world deployed cyber-physical systems. We combine physical and data-driven knowledge to design metrics and methods to assess the signal and dataset quality for particular sensing tasks.
Latest Papers:
Shubham Rohal, Yue Zhang, Carlos Ruiz, and Shijia Pan. "AutoLoc: Autonomous Sensor Location Configuration via Cross-Modal Sensing." Frontiers in Big Data 5 (2022): 835949.
Yu, Tong, Yue Zhang, Zhizhang Hu, Susu Xu, and Shijia Pan. "Vibration-based indoor human sensing quality reinforcement via Thompson Sampling." In Proceedings of the First International Workshop on Cyber-Physical-Human System Design and Implementation, pp. 33-38. 2021.
Lixing He, Carlos Ruiz, Mostafa Mirshekari, and Shijia Pan. "SCSV2: physics-informed self-configuration sensing through vision and vibration context modeling." In Adjunct Proceedings of the 2020 ACM International Joint Conference on Pervasive and Ubiquitous Computing and Proceedings of the 2020 ACM International Symposium on Wearable Computers, pp. 532-537. 2020.
Adaptation in Learning
Adaptation in Learning
The heterogeneity of cyber-physical systems brings challenges and opportunities for various applications. Our goal is to explore new methods to combine the complementary characteristics of multiple sensing modalities for accurate fine-grained learning with limited (labeled) data.
Latest Paper:
Dong Yoon Lee, Alyssa Weakley, Hui Wei, Blake Brown, Keyana Carrion, and Shijia Pan. "RRAR: Robust Real-World Activity Recognition with Vibration by Scavenging Near-Surface Audio Online." In Proceedings of the 3rd ACM International Workshop on Intelligent Acoustic Systems and Applications, pp. 1-6. 2025.
Zhizhang Hu, Yue Zhang, Tong Yu, and Shijia Pan. "VMA: Domain Variance-and Modality-Aware Model Transfer for Fine-Grained Occupant Activity Recognition." In 2022 21st ACM/IEEE International Conference on Information Processing in Sensor Networks (IPSN), pp. 259-270. IEEE, 2022.
AIoT Enabled Augmented Air Quality Monitoring
AIoT Enabled Augmented Air Quality Monitoring
San Joaquin Valley's (SJV) residents face challenges of air pollutant exposure -- the basin terrain of the SJV traps air pollutants easily. At the same time, the local land use (e.g., industry, agriculture) results in a high variation of pollutant sources of PM2.5. We develop Artificial Intelligence of Things (AIoT) with both advanced sensing and learning algorithms to tackle the challenge with low-cost and scalable solutions.
Latest Papers:
Shangjie Du, Hui Wei, Dong Yoon Lee, Zhizhang Hu, Shijia Pan. 2025. Graph-Based Physics-Guided Urban PM2.5 Air Quality Imputation with Constrained Monitoring Data. ACM Transactions on Sensor Networks (TOSN).
Shangjie Du, Zhizhang Hu, and Shijia Pan. 2024. GraPhy: Graph-Based Physics-Guided Urban Air Quality Modeling for Monitoring-Constrained Regions. In Proceedings of the 11th ACM International Conference on Systems for Energy-Efficient Buildings, Cities, and Transportation (BuildSys ’24), November 7–8, 2024, Hangzhou, China.
Zhizhang Hu∗, Shangjie Du∗, Yuning Chen, Xuan Zhang, Wan Du, Asa Bradman, and Shijia Pan. 2023. Poster Abstract: Enhancing Fault Resilience of Air Quality Monitoring in San Joaquin Valley: A Data Equity Analysis. In Proceedings of the 21st ACM Conference on Embedded Networked Sensor Systems (SenSys ’23), November 12–17, 2023, Istanbul, Turkiye. ACM, New York, NY, USA, 2 pages.
Gang Wang, Shijia Pan, and Susu Xu. "Decoupling the unfairness propagation chain in crowd sensing and learning systems for spatio-temporal urban monitoring." In Proceedings of the 8th ACM International Conference on Systems for Energy-Efficient Buildings, Cities, and Transportation, pp. 200-203. 2021.
Award: Best Paper Award Runner-Up BuildSys'24
Smart Farm Augmented Geospatial Mapping
Smart Farm Augmented Geospatial Mapping
California’s San Joaquin Valley (SJV) is the key contributor to the nation’s agriculture production, with 7984 square miles of prime farmland, farmland of statewide importance, or unique farmland. However, the region faces escalating climate change with frequent events such as wildfires and droughts. These severe climate events directly affect agricultural yields, posing financial challenges to the already disadvantaged communities. A high-granularity and high-resolution multimodal (e.g., soil moisture, temperature, etc.) map of the farmland is essential for enabling precision agriculture control (e.g., irrigation ) and enhancing the farm’s climate resilience.
Latest Paper:
Yue Zhang, Abdias Tellez Benitez, Reza Ehsani, and Shijia Pan. "DaQual: Data Quality Assessment for Tree Trunk Relative Water Content Sensors in a Pomegranate Orchard." In Proceedings of the 20th ACM Conference on Embedded Networked Sensor Systems, pp. 1096-1101. 2022.
Project Theme 3: Ubiquitous Interfaces
Project Theme 3: Ubiquitous Interfaces
Structure-Assisted Surface Tactile Interface
Structure-Assisted Surface Tactile Interface
Coming soon!
IoT Privacy Visualization
IoT Privacy Visualization
Internet of Things (IoT) sensing systems provide a wide range of services to improve convenience and comfort in our daily lives. However, users often have a limited understanding of the privacy characteristics of their sensors. We empower users with proactive privacy protection by enhancing their privacy perception via more intuitive presentations and recommendations.
Latest Paper:
Dipu Ram Roy, Jieqiong Zhao, Shijia Pan, Shiwei Fang. "Towards Understanding User Privacy Concerns of Internet of Things Sensor Data." In Proceedings of the 2nd International Workshop on Security and Privacy of Sensing Systems, pp. 1-5. 2025.
Yue Zhang, Shangjie Du, Jiqing Wen, Robert Likamwa, Shiwei Fang, and Shijia Pan. "Poster: PrivaSee: Augmented Reality-Enabled Privacy Perception Visualization for Internet of Things." In Proceedings of the 22nd Annual International Conference on Mobile Systems, Applications and Services, pp. 694-695. 2024.
Unvoiced Speech (with Stony Brook, UMass)
Unvoiced Speech (with Stony Brook, UMass)
Unvoiced user interfaces address the limitation of voiced user interfaces, such as being prone to error in noisy environments, and compromising privacy in public places. We explore using an IMU pair to capture jaw motions for unvoiced speech recognition.
Latest Paper:
Srivastava, Tanmay, Prerna Khanna, Shijia Pan, Phuc Nguyen, and Shubham Jain. "Unvoiced: Designing an LLM-assisted Unvoiced User Interface using Earables." In Proceedings of the 22nd ACM Conference on Embedded Networked Sensor Systems, pp. 784-798. 2024.
Page updated
Google Sites
Report abuse