PD is a progressive neurodegenerative condition that is characterized by fluctuating motor symptoms like tremor, rigidity, and bradykinesia. Accurate and consistent measurement of the ON and OFF medication states is necessary to optimize treatment and improve the quality of life. However, current approaches are based on episodic and subjective clinical assessments or intrusive wearable and video-based systems that raise usability, scalability, and privacy concerns. These challenges highlight the need for a non-intrusive, objective, and privacy-preserving solution that can provide real-time and fine-grained monitoring of Parkinson’s motor symptoms in real-world environments.

The objective of this work is to develop a Multi-Scale Temporal Attention-Transformer Network (MS-TATNet) framework for Parkinson’s motor symptom monitoring using 2D skeleton pose data. Specifically, the framework aims to accurately detect when patients are in the ON state (when medication is effective and symptoms are reduced) and the OFF state (when the effect of medication reduces and symptoms reappear or worsen), while simultaneously estimating the continuous severity of motor symptoms.

The main contributions of the proposed MS-TATNet Framework are as follows:

• Privacy-Preserving MS-TATNet Framework: The work proposes a Multi-Scale Temporal Attention-Transformer Network (MS-TATNet) framework to monitor PD motor symptoms using 2D skeleton pose data, instead of raw video or wearable sensors. This approach allows for privacy-preserving and non-intrusive analysis appropriate for real-world deployment.

The remaining part of this paper is structured as follows. The existing works on PD detection using various approaches are reviewed in Section 2. The suggested MS-TATNet framework is presented in Section 3, the experimental findings are described in Section 4, and the study is concluded in Section 5.

Lin et al. [16] utilized raw kinematic signals from inertial measurement unit sensors to create a model that uses ML to distinguish early-stage PD from essential tremor based on gait and postural transition parameters. Even though the model was highly stable, it was not appropriate for real-time applications or personal usage at home for monitoring PD patients. Davidashvilly et al. [17] created a Deep Neural Network (DNN) for PD patients’ activity recognition using wearable sensor data. The model demonstrated better activity recognition performance using healthy data. However, there was an inconsistency in the activity-matching procedure with the dataset. This was due to the lack of an activity label that matched those in the dataset used in that work. Johnson et al. [18] used a multivariate ML model to remotely screen early-stage PD using a consumer-grade wearable device. This approach successfully generated high-dimensional information from several sensors. But there were problems with the non-demographically matched research group and the non-clinically proven PD diagnosis due to its diminished control over enrollment screening. Rodriguez et al. [19] introduced and applied an ML algorithm to assess the intensity of tremor in free-living PD patients using wearable sensor data with an Inertial Measurement Unit (IMU) securely attached to patient’s wrist and ankles. Even though the model enhanced therapeutic relevance in continuously monitoring PD symptoms, the assessments during unrestrained action in free-living situations continue to be challenging. Hammoud et al. [20] developed a wrist-worn IMU sensor to identify and monitor the development of PD using ML approaches. With both left and right wrist sensors, the model showed better performance with the left hand. Nevertheless, the model could not account for anatomical and physiological aspects to determine why the left-hand sensor performed better.

Brien et al. [21] presented a simple, non-invasive PD classification algorithm using video-based eye tracking and ML methods. While the model attained comparable measures of sensitivity and specificity, it was still challenging to evaluate what is sensitive to PD rather than age, particularly cognitive ratings. Sarapata et al. [22] created a scalable and autonomous video-based human activity identification system for PD motor dysfunction using Spatio-Temporal Graph Convolutional Network (ST-GCN). The system successfully attained reasonable accuracy in activity categorization and frame-by-frame precise annotation for high resolution. However, the model has difficulties to identify identical body postures and movement patterns. Zeng et al. [23] introduced a computerized video-based gait analysis model using a skeleton-silhouette fusion convolution network. The model provides fine-grained extra characteristics for high-resolution gait measurement in addition to accurately predicting the Movement Disorder Society-Unified Parkinson’s Disease Rating Scale (MDS-UPDRS) gait score. However, the model did not include patients with severity score 3 and 4, which limits its applicability to more severe cases. Liu et al. [24] established a global temporal-difference shift network to predict the PD tremors’ MDS-UPDRS score from video. The model demonstrated an increased ability for generalization ability for the most severe score prediction of PD tremor. But the model faced challenges in accurately detecting patients with mild and moderate severity of PD. Gao et al. [25] developed an DL-based model for eye movement analysis-based PD assessment using regular red, green, and blue (RGB)-video data. The model combined the 1D-Convolutional Neural Network with Attention-based Network exhibited superior performance in PD classification. The model had problems with higher interference cost and generalizability issues. Most prior studies use raw video or high-resolution images, raising patient privacy concerns and require large storage or bandwidth. No models jointly perform motor state classification and continuous symptom severity regression on pose data. There is a need for non-invasive, objective, and scalable methods to detect motor symptom fluctuations in real-world settings. Thus, this work develops a real-time AI system that monitors PD motor symptoms using skeleton pose data to detect patients’ medication and its severity.

This work utilizes 2D skeleton pose data from REMAP [26], a human rater-labelled dataset of real-world mobility behavior in PD including Sit-To-Stand (STS) transitions and turns in gait while living in a home environment. These distinct activities are recorded during clinical evaluation as well as during unstructured and unobserved free-living. It involves 24 subjects, twelve participants (mean age 61.25; seven males, five females) have PD, while the remaining twelve (mean age 59.25, three males, nine females) do not have PD. The dataset was collected using wall-mounted Microsoft Kinect cameras in communal rooms of a fully furnished test-bed house, capturing RGB video at 640 $\times$ 480 resolution and 30 frames per second. From these recordings, episodes of STS and turning in gait were extracted and converted into 2D skeleton sequences using pose estimation methods. A range of other annotations that offer extensive details about the actions are shown in Table 1.

The REMAP dataset includes multiple recording sessions for each participant, and each session contains several linked data files rather than a single sample, resulting in a significantly larger number of usable instances. For the STS task, each participant contributes approximately 5–7 linked recording files, with each file containing 150–300 sequential frames representing a complete sit-to-stand movement cycle. The STS metadata file includes fields such as Transition ID, Participant ID, PD or Control, sts_whole_episode_duration, sts_final_attempt_duration, on_or_off_medication, Deep Brain Stimulation (DBS)_state, Clinical_assessment, STS_additional_features, and MDS-UPDRS score 3.9 (arising from chair). Each corresponding linked file contains time-series motion data, where each row represents one frame of the movement via the time(s) column, and skeleton joint positions are stored as coordinates labeled x0, y0, x1, y1 … up to x24, y24, representing all 25 tracked body joints. Most STS clips were 17 seconds long, with 2 seconds/2000 milliseconds included before the transition and a variable amount of data included afterwards to make up the total duration. The skeleton included 25 joints that can be divided into different parts of the body as shown in Fig. 3, in the head (0 nose, 15 and 16 eyes and 17 and 18 ears), trunk (1 neck and 8 mid hip), arms (2 and 5 shoulders, 3 and 6 elbows and 4 and 7 wrists), legs (9 and 12 hips, 10 and 13 knees, 11 and 14 ankles), and feet (19 and 22 big toes, 20 and 23 small toes and 21 and 24 heels).

Fig. 3.

Layout of 2-dimensional skeleton joints used in sit-to-stand data comprises 25 joints.

For turning of gait episodes, the RGB video clips were trimmed to contain the turning action with 6 frames of data/200 milliseconds included both before and after the action itself, comprises 17 body joints. The resulting frame-by-frame skeleton data provided structured representations of mobility actions suitable for quantitative analysis. Each participant provides multiple turning trials, resulting in several linked key-point files per subject. The Turning metadata file includes Turn ID, Participant ID, PD or Control, number_of_turning_steps, turning_angle, type_of_turn, turning_duration, On_or_Off_medication, DBS_state, and clinical_assessment. Together, the repeated STS and turning recordings across all participants create a rich motion-sequence dataset with thousands of frames and a large number of movement samples, enabling detailed analysis despite the limited number of subjects.

The dataset first undergoes column normalization using dictionary mapping and string standardization. Categorical and text values were encoded through rule-based normalization, and missing values were handled thorough imputation. After these preprocessing steps, the dataset was split into 64% training, 20% testing, and 16% validation sets. Feature scaling and normalization were then applied using statistics computed from the training set, ensuring no information leakage into the validation or test sets.

To effectively capture the diverse temporal dynamics of Parkinson’s motor symptoms, this work develops an MS-TCN [27]. The motor fluctuations of PD occur on various temporal scales, while tremors are captured in rapid oscillatory behavior, rigidity and bradykinesia slowly evolve over time. A single receptive field is not suitable for modelling such variability. Therefore, the model uses parallel temporal convolutional branches with multiple dilation factors to extract short, medium, and long-term temporal dependencies. Each stage contains four dilated 1D convolution layers with dilation rates of 1, 2, 4, and 8, enabling the network to capture short-, medium-, and long-range dependencies within the movement sequence. The stages use progressively larger kernel sizes and channel capacities, Stage 1 uses a kernel size of 3 with 64 channels, Stage 2 uses a kernel size of 5 with 96 channels, and Stage 3 uses a kernel size of 5 with 128 channels, resulting in a total of 12 convolutional layers.

Formally, given a skeleton pose sequence $U=\{u_1,u_2,\mathrm{\dots },u_T\}$, where $u_tϵR^d$ signifies the $d$-dimensional skeletal joint features at $t$time step, each branch applies a dilated 1D convolution defined in Eqn. 1.

(1)$$y\left(t\right)={\sum }_{i=0}^{k-1}w(i)\cdot u\left(t-r\cdot i\right)$$

where $k$ is the kernel size, $w(i)$ represents the convolutional weights, and $r$ is the dilation rate. Each branch produces a feature representation $F_s$, $F_m$, and $F_l$, corresponding to short-, medium-, and long-term temporal patterns, respectively. These features are concatenated into one multi-scale feature representation as shown in Eqn. 2.

(2)$$F_{concat}=\left[F_s\parallel F_m\parallel F_l\right]\in {ℝ}^{T\times 288}$$

where $\parallel$ indicates channel-wise concatenation. The multi-scale features obtained through concatenation are passed through SDPA to emphasize significant temporal patterns. The model is trained in a supervised learning framework, where movement patterns of Parkinson’s motor symptoms are defined according to established clinical criteria, and labels for each patient are provided by board-certified neurologists based on standardized clinical assessments. These labels serve as ground truth, enabling the network to learn and classify temporal patterns corresponding to diverse motor symptoms.

To dynamically identify and prioritize the most relevant information of the input sequence, the proposed model incorporates the SDPA mechanism. The SDPA mechanism uses the dot product [28], which is scaled by the square root of the key vector’s dimension to calculate attention scores between a query vector ${\mathbb{Q}}_q$ and a collection of key vectors ${𝕂}_k$ and value vectors ${𝕍}_v$. Given, query $\left({\mathbb{Q}}_q\right)$, key $\left({𝕂}_k\right)$, and value $\left({𝕍}_v\right)$ projections of $F_{concat}$, we can determine the attention score as follows in Eqn. 3.

(3)$$Attention({\mathbb{Q}}_q,{𝕂}_k,{𝕍}_v)=softmax\frac{{\mathbb{Q}}_q{𝕂}_k^T}{\sqrt{d_k}}{𝕍}_v$$

where $d_k$ represents the key vectors’ dimensionality. The softmax function is used to normalize the attention weights to ensure they sum up to one. This mechanism ensures that the network prioritizes the most informative time scales based on the input sequence. This enhances the quality of the representation before it is processed by the Transformer encoder blocks.

The model uses a stacked Transformer encoder with Multi-Head Self-Attention (MHSA) to capture higher-level temporal dependencies [29]. The encoder consists of three identical layers connected sequentially. Each layer processes the output of the previous one using MHSA with four attention heads, followed by a feed-forward network with a hidden dimension of 256. The embedding dimension of each token is 128. The input to the Transformer is a temporal sequence reduced to 3–8 tokens depending on the duration of the movement trial. Learnable positional embeddings of size 128 are added to preserve temporal ordering. By stacking multiple encoder layers, the model gradually extracts more complex and long-range temporal relationships. The final contextualized representation is used as the final output, as depicted in Fig. 4.

Fig. 4.

Architecture of stacked transformer encoder blocks with multi-head self-attention mechanism for advanced temporal dependency modeling.

Temporal pooling is a procedure that is commonly employed in sequence modeling for transforming input sequences of variable-length into smaller fixed size representations. Temporal pooling summarizes information across time, enabling the model to process longer input sequences with fewer computations [30]. In this study, temporal pooling is applied after the transformer encoder blocks to produce compact representations that preserve the most relevant temporal information. This process allows for short-term variability and long-term dependencies, which are encoded by the previous multi-scale and attention mechanisms. The pooled representation provides a stable input for the dual-output prediction heads, the softmax classifier for classify the medication state of the patients, and a regression layer for measuring the severity of the disease.

A softmax classifier is a multi-class generalization of logistic regression [31]. It is applied to normalize the raw output scores (logits) of a neural network into probability values, ensuring that the outputs sum to one and are all lie in the range [0, 1]. The function for a softmax classifier function is defined mathematically as shown in Eqn. 4.

(4)$$G_e\left(v\right)=\frac{e^{v_e}}{{\sum }_z{e}^{v_z}}$$

where $G_e\left(v\right)$ is the output from the softmax activation function, $v_e$ is the element of the input vector $v$. In this framework, the softmax classifier is utilized in the output layer to distinguish between the ON and OFF medication states of patients with PD.

The MS-TATNet framework incorporates a regression layer to estimate the severity of motor symptoms in patients [32]. Formally, the regression head is implemented as a fully connected layer applied to the pooled temporal representation, is shown in Eqn. 5.

(5)$$\widehat{y}=W_{r}r+b_r$$

where $r$ denotes the pooled input feature vector, $W_r$ and $b_r$ represent the regression layer’s weight matrix and bias term, and $\widehat{y}$ is the predicted continuous severity score.

Table 5 indicate that the model performs steadily and consistently across several data splits and has a low standard deviation. These results demonstrate the model’s durability and dependability while validating its strong and reliable performance. These findings suggest that the model is not too reliant on any subset of data. This mitigates concerns regarding overfitting and data uniformity.

In this section, an ablation analysis is performed to assess the impact of MS-TCN, SDPA mechanism, stack transformer encoder with MHSA, and temporal pooling on MS-TATNet model for classifying PD motor symptoms. We compare the model’s performance with and without these techniques and different combinations of these techniques to analyze the impact of the techniques in the MS-TATNet framework. To validate our findings, we employ several evaluation measures, including accuracy, precision, recall, and F1-score, along with R² Score and SD value. The findings, illustrated in Table 6, validate the efficacy of these techniques, resulting in more reliable classification outcomes.

Table 7 summarizes the trade-offs between different input modalities, including RGB video, 3D skeleton, and the 2D skeleton utilized in the MS-TATNet model. It presents identity exposure, data stored, reconstruction risk, hardware requirements, model accuracy, inference latency per sequence, and real-time capability. The result highlight that the 2D skeleton approach achieves the highest accuracy while maintaining very low privacy risk and real-time performance on CPU hardware, whereas RGB video and 3D skeleton modalities which either compromise privacy or require more specialized hardware.

This section evaluates the effectiveness of the MS-TATNet model in comparison to various existing methods for classifying PD.

Table 8 (Ref. [14, 17, 19, 21, 23, 24]) compares the MS-TATNet with existing models, such as Support Vector Machine with Recursive Feature Elimination (SVM-RFE), DNN, Support Vector Machine with Radial Basis Functions (SVM-RBF), ML based approaches, ST-GCN and Global Temporal-difference Shift Network (GTSN). Previous models achieved accuracies ranging from 71–95%, while the MS-TATNet achieved high performance of 99.63% accuracy, 99.50% recall and 99.33% specificity. Overall, this demonstrates the MS-TATNet superior ability to capture motor state fluctuation with both high sensitivity and generalization. This is due to its capability to extract short- and long-range temporal dependencies across skeletal pose data while preserving privacy. This enables robust generalization across patients and relatively more reliable motor state detection of motion-state fluctuations.

Parkinson’s motor symptoms originate from dopaminergic loss in the substantia nigra and abnormal modulation of the basal ganglia–thalamocortical motor circuits. These pathological changes produce bradykinesia, rigidity, impaired turning, and variability in sit-to-stand transitions patterns that are directly captured in the temporal dynamics of the 2D skeleton pose data. The MS-TATNet identifies reduced joint velocity, decreased movement amplitude, hesitation, and tremor-like oscillations, which are biomechanical correlates of these neural circuit abnormalities.

The continuous severity score generated by the regression module shows strong alignment with clinically evaluated components of the MDS-UPDRS Part III motor subscale, including bradykinesia, tremor intensity, gait, and postural control. Higher predicted severity values correspond to physiologically meaningful impairment and reflect dopaminergic ON/OFF fluctuations rather than only a statistical trend. Thus, the model provides a clinically interpretable, objective, and fine-grained digital biomarker that aligns with established PD motor circuit dysfunction.

In addition, the model’s ability to detect continuous fluctuations in severity has direct implications for PD therapy and disease management. Dopaminergic medications such as levodopa produce characteristic pharmacodynamic cycles, including wearing-off, delayed-ON, and dose-failure that manifest as measurable changes in movement amplitude, speed, and tremor patterns. The model captures these transitions, allowing potential real-time monitoring of medication response. Because symptom trajectories vary across early, moderate, and advanced PD, the fine-grained severity output can support longitudinal tracking of neurodegenerative progression. Moreover, such continuous monitoring could aid adaptive therapy strategies, such as optimizing levodopa dosing schedules or informing closed-loop DBS systems with objective, high-frequency motor state information.