Synopsis


This project seeks to build a WiFi-enabled passive continuous wellbeing monitoring framework for activity recognition, gesture recognition, sleep monitoring, and vital signs tracking at home environments. This project pursues the vision of a system that utilizes device-free localization strategies, vital signs monitoring methods, and statistical learning techniques to depict a comprehensive picture of users' wellbeing. Such wellbeing information is further utilized to assist in real-time disease prediction by leveraging today's ever-growing mobile environments. A hierarchical multivariate logistic regression model is developed to effectively mine through health conditions and identify risk factors for certain diseases. Chances of developing specific health problems, such as cardiovascular diseases, is promptly predicted. The project also provides user-centric access control of archived wellbeing monitoring information to ensure data privacy and coping with distrusted servers. During this project reporting period, we focus on the following specific goals:



Task 1: Activity Recognition and Gesture Recognition



Smart Human Dynamics Monitoring Using Existing WiFi Signals

The rapid pace of urbanization and socioeconomic development encourage people to spend more time together and therefore monitoring of human dynamics is of great importance, especially for facilities of elder care and involving multiple activities. Traditional approaches are limited due to their high deployment costs and privacy concerns (e.g., camera-based surveillance or sensor-attachment-based solutions). This project proposes to provide a fine-grained comprehensive view of human dynamics using existing WiFi infrastructures often available in many indoor venues. Our approach is low-cost and device-free, which does not require any active human participation. The proposed system aims to provide smart human dynamics monitoring through participant number estimation, human density estimation and walking speed and direction derivation. A semi-supervised learning approach leveraging the non-linear regression model is developed to significantly reduce training efforts and accommodate different monitoring environments. The system can be utilized to derive participant number and density estimation based on the statistical distribution of Channel State Information (CSI) measurements. In addition, people's walking speed and direction are estimated by using a frequency-based mechanism. Extensive experiments demonstrate that the proposed system can perform fine-grained effective human dynamic monitoring with high accuracy in estimating participants number, density, and walking speed and direction at various indoor environments.

nsf

[SenSys'17] WiFi-Enabled Smart Human Dynamics Monitoring, [pdf]

Xiaonan Guo, Bo Liu, Cong Shi, Hongbo Liu, Yingying Chen, Mooi Choo Chuah,

in Proceedings of the 15th ACM Conference on Embedded Networked Sensor Systems,

Delft, The Netherlands, November 2017.



PPG-based Finger-level Gesture Recognition Leveraging Wearables

We demonstrate that it is possible to leverage the widely deployed PPG sensors in wrist-worn wearable devices to enable finger-level gesture recognition, which could facilitate many emerging human-computer interactions (e.g., sign-language interpretation and virtual reality). While prior solutions in gesture recognition require dedicated devices (e.g., video cameras or IR sensors) or leverage various signals in the environments (e.g., sound, RF or ambient light), this project introduces the first PPG-based gesture recognition system that can differentiate fine-grained hand gestures at finger level using commodity wearables. Our innovative system harnesses the unique blood flow changes in a user’s wrist area to distinguish the user’s finger and hand movements. The insight is that hand gestures involve a series of muscle and tendon movements that compress the arterial geometry with different degrees, resulting in significant motion artifacts to the blood flow with different intensity and time duration. By leveraging the unique characteristics of the motion artifacts to PPG, our system can accurately extract the gesture-related signals from the significant background noise (i.e., pulses), and identify different minute finger-level gestures. Extensive experiments are conducted with over 3600 gestures collected from 10 adults.

nsf

[INFOCOM'18] PPG-based Finger-level Gesture Recognition Leveraging Wearables, [pdf]

Tianming Zhao, Jian Liu, Yan Wang, Hongbo Liu, Yingying Chen,

in Proceedings of IEEE International Conference on Computer Communications,

Honolulu, USA, April 2018.



Device-free Fitness Assistant Using Existing WiFi Infrastructure

There is a growing trend for people to perform regular workouts in home/office environments because work-at-home people or office workers can barely squeeze in time to go to dedicated exercise places (e.g., gym). To provide personalized fitness assistance in home/office environments, traditional solutions, e.g., hiring personal coaches incur extra cost and are not always available, while new trends requiring wearing smart devices around the clock are cumbersome. In order to overcome these limitations, this project proposes to develop a device-free fitness assistant system in home/office environments using existing WiFi infrastructure. The proposed system aims to provide personalized fitness assistance by differentiating individuals, automatically recording fine-grained workout statistics, and assessing workout dynamics. In particular, our system performs individual identification via deep learning techniques on top of workout interpretation. It further assesses the workout by analyzing both short and long-term workout quality and provides workout reviews for users to improve their daily exercises. Additionally, our system adopts a spectrogram-based workout detection algorithm along with a Cumulative Short-Time Energy (CSTE)-based workout segmentation method to ensure its robustness. Extensive experiments involving 20 participants demonstrate the effectiveness of our system.

nsf

[UbiComp'19] Device-free Personalized Fitness Assistant Using WiFi, [pdf]

Xiaonan Guo, Jian Liu, Cong Shi, Hongbo Liu, Yingying Chen, Mooi Choo Chuah,

in PACM on Interactive, Mobile, Wearable, and Ubiquitous Computing, Volume 2, Issue 4, Pages 1-23, 2018. (presented at UbiComp 2019).



Multi-User Tracking and Activity Recognition Using Commodity WiFi

Indoor human tracking and activity recognition is gaining increasing attention and undergoing fast development in a variety of real-world applications, especially in mobile healthcare. This work presents MultiTrack, a commodity WiFi-based human sensing system that can track multiple users and recognize the activities of multiple users performing them simultaneously. Such a system enables easy and large-scale deployment for multi-user tracking and sensing without the need for additional sensors through the use of existing WiFi devices (e.g., desktops, laptops, and smart appliances). The basic idea is to identify and extract the signal reflection corresponding to each individual user with the help of multiple WiFi links and all the available WiFi channels at 5GHz. Given the extracted signal reflection of each user, MultiTrack examines the path of the reflected signals at multiple links to simultaneously track multiple users. It further reconstructs the signal profile of each user as if only a single user has performed activity in the environment to facilitate multi-user activity recognition. We evaluate MultiTrack in different multipath environments with up to 4 users for multi-user tracking and up to 3 users for activity recognition. Experimental results show that our system can achieve decimeter localization accuracy and over 92% activity recognition accuracy under multi-user scenarios.

nsf

[CHI'19] MultiTrack: Multi-User Tracking and Activity Recognition Using Commodity WiFi, [pdf]

Sheng Tan, Linghan Zhang, Zi Wang, Jie Yang,

in Proceedings of the Conference on Human Factors in Computing Systems,

Glasgow, United Kingdom, May 2019.



Studying Acoustic-based Sensing and Applications

With advancements of wireless and sensing technologies, recent studies have demonstrated technical feasibility and effectiveness of using acoustic signals for sensing. In the past decades, low-cost audio infrastructures are widely-deployed and integrated into mobile and Internet of Things (IoT) devices to facilitate a broad array of applications including human activity recognition, tracking, localization, and security monitoring. The technology underpinning these applications lies in the analysis of propagation properties of acoustic signals (e.g., reflection, diffraction, and scattering) when they encounter human bodies. As a result, these applications serve as the foundation to support various daily functionalities such as safety protection, smart healthcare, and smart appliance interaction. In this project, we provide a comprehensive review on acoustic-based sensing in terms of hardware infrastructure, technical approaches, and its broad applications. We classify various applications and compare different acoustic-based sensing approaches: in recognition and tracking, we review daily activity recognition, human health and behavioral monitoring hand gesture recognition, hand movement tracking, and speech recognition; in localization and navigation, we discuss ranging and direction finding, indoor and outdoor localization, and floor map construction; in security and privacy, we survey user authentication, keystroke snooping attacks, audio adversarial attacks, acoustic vibration attacks, and privacy protection schemes. Lastly we discuss future research directions and limitations of the acoustic-based sensing.

nsf

[Computer Networks] Acoustic-based Sensing and Applications: a Survey, [pdf]

Yang Bai, Li Lu, Jerry Cheng, Jian Liu, Yingying Chen, Jiadi Yu,

in Computer Networks, 2020.



Device-free Eating Monitoring Leveraging Wi-Fi Signals

Eating well plays a key role in people’s overall health and wellbeing. Studies have shown that many health-related problems such as obesity, diabetes, and anemia are closely associated with people’s unhealthy eating habits (e.g., skipping meals, eating irregularly, and overeating). Thus, keeping track of diet is becoming more important. Different from traditional self-report-based approaches, in this project, we develop a device-free eating monitoring system using WiFi-enabled devices (e.g., smartphone or laptop). Our system aims to automatically monitor users’ eating activities by identifying the fine-grained eating motions and detecting the minute movements during chewing and swallowing. In particular, our system distinguishes eating from non-eating activities by using K-means clustering with principal component analysis on the extracted Channel State Information (CSI) from WiFi signals. It further adopts a soft decision-based eating motion classification through identifying the utensils (e.g., using a folk, knife, spoon, or bare hands) in use. Moreover, we propose a minute motion reconstruction method to identify chewing and swallowing by detecting users’ minute facial muscle movements. The derived fine-grained eating monitoring results are beneficial to the understanding of users’ eating behaviors and estimation of food intake types and amounts. Extensive experiments with 20 users show the effectiveness of the proposed system.

nsf

[ICCCN'20] WiEat: Fine-grained Device-free Eating Monitoring Leveraging Wi-Fi Signals, [pdf]

Zhenzhe Lin, Yucheng Xie, Xiaonan Guo, Yanzhi Ren, Yingying Chen, Chen Wang,

in International Conference on Computer Communications and Networks,.

Hawaii, USA, August 2020.



Task 2: Vital Signs Monitoring and User Verification



Measuring Pulse Transit Time Using Smartphone Motion Sensors and Optical Sensors

Blood pressure (BP) variations represent the functioning of the cardiovascular system and could yield early indicators of cardiovascular diseases such as hypertension, hypertensive heart disease, atrial fibrillation, and stroke. Traditional approaches fail to provide continuous monitoring of BP and rely on bulky equipments such as oscillometry, volume clamp, and auscultation. This project proposes a smartphone-only solution for measuring blood pressure in terms of pulse transit time (PTT). An application based on an Android smartphone is developed to collect seismocardiogram (SCG), gyrocardiogram (GCG), and photoplethysmography (PPG) recordings. The system does not need any other external system for measurements, so the total cost and system complexity are minimized. PTT metrics are calculated as the time difference between the aortic valve opening points in the SCG or GCG signals recorded by the accelerometer or gyroscope of a smartphone respectively, and the fiducial points in the PPG signal recorded by a modified optical sensor connected to the audio input. A digital signal processing (DSP) system is developed in a post-processing environment and experimentally validated on ten healthy subjects at rest. Our results indicate that a smartphone-only PTT measurement system is feasible and comparable with stand-alone sensor node systems.

nsf

[EMBC'18] A Low-cost, Smartphone-only Pulse Transit Time Measurement System Using Cardio-mechanical Signals and Optical Sensors, [pdf]

Chenxi Yang, Yudi Dong, Yingying Chen, Negar Tavassolian,

in Proceedings of 40th Annual IEEE International Conference of the Engineering in Medicine and Biology Society,

Honolulu, USA, July 2018.



Semi-black-box Attacks Against Speech Recognition Systems Using Adversarial Samples

As automatic speech recognition (ASR) systems have been integrated into a diverse set of devices around us in recent years, security vulnerabilities of them have become an increasing concern for the public. Existing studies have demonstrated that deep neural networks (DNNs), acting as the computation core of ASR systems, is vulnerable to deliberately designed adversarial attacks. Based on the gradient descent algorithm, existing studies have successfully generated adversarial samples which can disturb ASR systems and produce adversary-expected transcript texts designed by adversaries. Most of these research simulated white-box attacks which require knowledge of all the components in the targeted ASR systems. In this work, we propose the first semi-black-box attack against the ASR system - Kaldi. Requiring only partial information from Kaldi and none from DNN, we can embed malicious commands into a single audio chip based on the gradient-independent genetic algorithm. The crafted audio clip could be recognized as the embedded malicious commands by Kaldi and unnoticeable to humans in the meanwhile. Experiments show that our attack can achieve high attack success rate with unnoticeable perturbations to three types of audio clips (pop music, pure music, and human command) without the need of the underlying DNN model parameters and architecture.

nsf

[DySPAN'19] Semi-black-box Attacks Against Speech Recognition Systems Using Adversarial Samples, [pdf]

Yi Wu, Jian Liu, Yingying Chen, Jerry Cheng,

in Proceedings of the IEEE International Symposium on Dynamic Spectrum Access Networks,

Newark, New Jersey, November 2019.



Continuous Authentication on Wrist-worn Wearables Using PPG-based Biometrics

Traditional one-time user authentication processes might cause friction and unfavorable user experience in many widely-used applications. This is a severe problem in particular for security-sensitive facilities if an adversary could obtain unauthorized privileges after a user’s initial login. Recently, continuous user authentication (CA) has shown its great potential by enabling seamless user authentication with few active participation. In this project, we devise a low-cost system exploiting a user’s pulsatile signals from the photoplethysmography (PPG) sensor in commercial wrist-worn wearables for CA. Compared to existing approaches, our system requires zero user effort and is applicable to practical scenarios with non-clinical PPG measurements having motion artifacts (MA). We explore the uniqueness of the human cardiac system and design an MA filtering method to mitigate the impacts of daily activities. Furthermore, we identify general fiducial features and develop an adaptive classifier using the gradient boosting tree (GBT) method. As a result, our system can authenticate users continuously based on their cardiac characteristics so little training effort is required. Experiments with our wrist-worn PPG sensing platform on 20 participants under practical scenarios demonstrate that our system can achieve a high CA accuracy with a low false detection rate.

nsf

[INFOCOM'20] TrueHeart: Continuous Authentication on Wrist-worn Wearables Using PPG-based Biometrics, [pdf]

Tianming Zhao, Yan Wang, Jian Liu, Yingying Chen, Jerry Cheng, Jiadi Yu,

in Proceedings of IEEE International Conference on Computer Communications,.

Beijing, China, April 2020.



Task 3: Statistical Learning and Disease Prediction



Revealing Demographics and Social Relationships from Surrounding Access Points

While the mobile users enjoy the anytime anywhere Internet access by connecting their mobile devices through Wi-Fi services, the increasing deployment of access points (APs) have raised a number of privacy concerns. This project explores the potential of smartphone privacy leakage caused by surrounding APs, which affect the users’ wellbeing. In particular, we study to what extent the users’ personal information such as demographics and social relationships could be revealed leveraging simple signal information from APs without examining the Wi-Fi traffic. We develop two new mechanisms: the Behavior-based Demographics Inference method differentiates various individual behaviors via the extracted activity features (e.g., activeness and time slots) at each daily place to reveal users’ demographics, whereas the Closeness-based Social Relationships Inference algorithm captures how closely people interact with each other by evaluating their physical closeness and derives fine-grained social relationships. Furthermore, our approach utilizes users’ activities at daily visited places derived from the surrounding APs to infer users’ social interactions and individual behaviors. Extensive experiments are conducted with participants’ real daily life to show the performance of our system in leveraging the simple signal information from surrounding APs to reveal people’s demographics and social relationships.

nsf

[ICDCS'17] Smartphone Privacy Leakage of Social Relationships and Demographics from Surrounding Access Points, [pdf]

Chen Wang, Chuyu Wang, Yingying Chen, Lei Xie, Sanglu Lu,

in Proceedings of IEEE International Conference on Distributed Computing,

Atlanta, USA, June 2017



Sensing Fruit Ripeness Using Wireless Signals

Recent advances in wireless technology have greatly expanded the WiFi usage from providing laptop connectivity to connecting mobile and smart devices to the Internet and home networks. Such an evolution has resulted in the prevalence of WiFi devices, which provides opportunities to extend WiFi’s capabilities beyond communication, particularly in object and human sensing. This work presents FruitSense, a fruit ripeness sensing system that leverages wireless signals to enable non-destructive and low-cost detection of fruit ripeness. Such a system reuses existing WiFi devices in homes without the need for additional sensors. In particular, it uses WiFi signals to sense the physiological changes associated with fruit ripening for detecting the ripeness of fruit. FruitSense leverages the larger bandwidth at 5GHz (i.e., over 600MHz) to extract the multipath-independent signal components to characterize the physiological compounds of the fruit. It then measures the similarity between the extracted features and the ones in ripeness profiles for identifying the ripeness level. We evaluate FruitSense in different multipath environments with two types of fruits (i.e, kiwi, and avocado) under four levels of ripeness. Experimental results show that FruitSense can detect the ripeness levels of fruits with an accuracy of over 90%.

nsf

[ICCCN'18] Sensing Fruit Ripeness Using Wireless Signals, [pdf]

Sheng Tan, Linghan Zhang, Jie Yang,

in International Conference on Computer Communications and Networks,

Hangzhou, China, July 2018.



Identifying Fetal Signals for Activating the NF-κB Pathway in the Placenta

Human parturition is a precisely coordinated process resulting from activation of a series of endocrine and immune responses, and a premature and sustained activation of these responses is hypothesized to lead to cases of preterm birth. In this project, we test the hypothesis that the fetus influences placental function by packaging signaling proteins in exosomes, and that these signals could be a trigger that initiates parturition. By performing proteomics analysis on purified exosomes of fetal cord arterial blood, we identified a total of 328 proteins including C4BPA, a gene controlling the canonical pathway of complement activation, in exosomes isolated from umbilical artery blood. Co-immunoprecipitation and functional assays suggest that C4BPA originating from the fetal lung served as a potent trigger for p100 processing. These results suggest that fetal C4BPA-induced activation of non-canonical NF-κB in human placenta may play a critical role in initiation of human parturition.

nsf

[Nature Scientific Reports] Fetal Lung C4BPA Induces P100 Processing in Human Placenta, [pdf]

MayraCruz Ithier, Nataliya Parobchak, StacyYadava, JerryCheng, BingbingWang, Todd Rosen,

Scientific Reports, Volume 9, Issue 1, Pages 2045-2322, 2019.



MIXP: Efficient Deep Neural Networks Pruning for Further FLOPs Compression via Neuron Bond

Neuron networks pruning is effective in compressing pre-trained CNNs for their deployment on low-end edge devices. However, few works have focused on reducing the computational cost of pruning and inference. We find that existing pruning methods usually remove parameters without fine-grained impact analysis, making it hard to achieve an optimal solution. This work aims to develop a novel mixture pruning mechanism, MIXP, which can effectively reduce the computational cost of CNNs while maintaining a high weight compression ratio and model accuracy. We propose to remove neuron bond that can effectively reduce convolution computations and weight size in CNNs. We also design an influence factor to analyze the importance of neuron bonds and weights in a fine-grained way so that MIXP could achieve precise pruning with few retraining iterations. Experiments with MNIST, CIFAR-10, and ImageNet datasets demonstrate that MIXP could achieve significantly fewer FLOPs and retraining iterations on four widely-used CNNs than existing pruning methods. This work contributed to developing new compression models when handling large amounts of wellbeing monitoring data from WiFi sensing in the cloud or edge server. The newly developed pruning model helped to compute the large-scale wellbeing sensing data in an efficient way.

nsf

[IJCNN '21] MIXP: Efficient Deep Neural Networks Pruning for Further FLOPs Compression via Neuron Bond, [pdf]

Bin Hu, Tianming Zhao, Yucheng Xie, Yan Wang, Xiaonan Guo, Jerry Cheng, Yingying Chen

in Proceedings of International Joint Conference on Neural Networks,

July 2021



Joint Selection in Generalized Linear Mixed Model Analysis: A Confidence Distribution Approach

Generalized linear mixed models (GLMMs) are commonly used to describe relationships between correlated responses and covariates. In this project, we propose a regularized method to select both fixed and random effects in GLMMs. In contrast to using the observed data likelihood functions, we propose to construct the objective functions using the confidence distribution of model parameters based on the joint and separate marginal asymptotic distributions of the fixed effect and random effect parameter estimators to perform effect selections. With a proper choice of regularization parameters in the adaptive LASSO framework, we show the consistency and oracle properties of the proposed regularized estimators. Simulation studies have been conducted to assess the performance of the proposed estimators and demonstrate computational efficiency. Our method has also been applied to two longitudinal cancer studies to identify demographic and clinical factors associated with health outcomes after cancer therapies. This work contributed to the modeling for health status prediction based on the processed sensing data. The linear mixed model analysis helped to describe the relationships between correlated responses (e.g., heart problems) and covariates (e.g., heartbeats and breathing rates).

Joint Selection in Generalized Linear Mixed Model Analysis: A Confidence Distribution Approach [pdf]

Shou-En Lu, Sinae Kim, Jerry Q. Cheng, Changfa Lin, Sharad Goyal, Salma Jabbour,

Biometrics, 2021 (submitted).



Acknowledgment: This material is based upon collaborative projects among Rutgers University, New York Institute of Technology, and Florida State University supported by the National Science Foundation under Grants CNS1826647, CNS1954959, and CNS1514238. Any opinions, findings, and conclusions or recommendations expressed in this material are those of the author(s) and do not necessarily reflect the views of the National Science Foundation.