Hand Tremor Characterization from a Spatiotemporal Convolutional Representation

Introduction

Parkinson’s disease (PD) is the second most prevalent neurodegenerative disorder, marked by a deficit in dopamine and impacting about 1 % of individuals aged 60 and older ¹. In Colombia, PD has an estimated prevalence of 4.7 per 1000 inhabitants, being more frequent in people over 60 years old ². Physiologically, PD is associated with a progressive loss of dopamine, a neurotransmitter responsible for optimal and synchronized locomotion processes. Due to this dopamine deficit, motor alterations affect coordinated movements and balance. For instance, postural instability, muscle rigidity, slowness of movement, tremors, and voice involvement constitute the typical symptoms of PD. Tremor is a dominant symptom of this disease. It is a rhythmic and involuntary movement caused by a muscle’s reciprocal innervations, which vary in intensity and frequency ³. In approximately 50 % of patients, the cardinal motor is tremor, and only 10 % do not have this symptom. Tremor patterns also vary in frequency and amplitude. Consequently, they can be identified depending on the posture of the limbs and the presence or absence of force. Hence, the detection and characterization of tremors are key to establishing complementary mechanisms to the standardized protocols that allow diagnosing PD.

In clinical practice, resting and postural tremors are commonly evaluated, primarily focusing on amplitude and persistence ⁴. In the resting configuration, the arms are supported by the muscles or a stable surface, and the tremor is observed during relaxation. Alternatively, the arms are held against gravity at a 90° angle from the body in the postural configuration. In this sense, the physical effort caused by maintaining the arms in this unnatural posture guarantees motion exaggeration. The unnatural position of the arms causes this overload; therefore, these schemes include noisy signals limiting the adequate quantification of hand tremor patterns. Every kind of tremor shows a different range of motion frequency, with resting tremors usually between 4 and 6 Hertz. Meanwhile, postural tremors correspond to the broader range, i.e., between 5 and 12 Hertz ⁵. Additionally, the movement frequency can be augmented for stress or anxiety and can reduce involuntary movements.

The quantification of tremors is usually performed using technological tools that integrate electronic devices and portable systems (inertial-type sensors), such as accelerometers, gyroscopes, and magnetometers ⁶. For instance, systems based on inertial sensors placed on the hands and arms quantify velocity and acceleration rates in postural and tremor configurations. Similarly, inertial sensors based on kinematic measures can separate Parkinsonian patterns from control subjects. Complex systems based on the Doppler effect can measure upper limb movements

showing frequencies, tremors, and the direction of signal tremors in the range of 0-8 Hz ⁷. In addition, ⁸ quantified the acceleration data into a deep learning scheme to quantify the tremor severity. However, these systems require fine calibration to reduce the signal noise, but, in some cases, motion pens reduce the movement amplitude and can be challenging to detect slight motions.

Some works have exploited video techniques to quantify tremor features through non-invasive strategies. Specifically, video has been used indirectly to measure tremors. In this kind of method, the participant holds the camera with their hands in a postural position and places it in front of a common background, which helps to characterize kinematic trajectories ⁹. However, this method has several drawbacks: it is highly dependent on maintaining a static position of the arm extended at 90 degrees relative to the trunk, which is challenging for healthy participants, and even more so for patients. These additional movements not associated with tremors make it difficult to distinguish between patients and control subjects.

Another example of video techniques involves recording the hands and identifying landmarks, which allows quantifying kinematic characteristics related to amplitude and frequency ¹⁰. However, this approach is quite sensitive to noise, as the location of points of interest between frames can vary throughout the video without being associated with tremors. These slight errors add noise to subtle tremors. Additionally, the number of landmarks may lead to a simplification of tremor; a reduced number of landmarks can affect the identification of tremor-associated patterns.

Interestingly, ¹¹ proposed a convolutional neural network (CNN) scheme to codify the tremor severity through a wearable wrist device equipped with an accelerometer and a gyroscope. Here, the collected signals are transformed into the frequency domain and then mapped to the deep net. Furthermore, ⁸ demonstrated the feasibility of using acceleration data in three directions to quantify tremor severity. This approach measured the arm movements of 20 people performing daily actions like drinking, extending their arms, touching their noses, stacking glasses, drawing, and writing. All the captured signals were then filtered in order to identify voluntary human movements and the severity of tremor patterns through activity classification models (ACM) and tremor assessment models (TEM). Despite these advantages, all these strategies still rely on physical devices that might introduce difficulties associated with sophisticated calibration processes and external noise sources that may bias the recorded measures. Moreover, sensibility is also compromised due to the anti-natural movement caused by the invasive devices.

This work introduces a convolutional spatiotemporal representation able to encode hand tremor patterns associated with PD, which allows distinguishing between Parkinsonian and control patients. This distinction was made separately for both resting and postural configurations. To this effect, hand video sequences were magnified to highlight associated tremors. The hand tremor patterns were then encoded via a hierarchical architecture involving 3D convolutions and embedded dense vectors with motion information. The results showcase the capabilities of this architecture, which provides indices correlated with PD. In addition, explainability maps were computed to support clinical decisions, enabling the visualization of the regions that, according to the network, are associated with a specific prediction.

Proposed approach

This work quantifies the spatiotemporal information through a 3D convolution strategy to classify Parkinsonian hand tremors in video sequences. Furthermore, as previously mentioned, explainability maps were computed as a visual tool to support clinical decisions within the deep representation strategy. Fig. 1 illustrates the proposed methodology.

Figure 1: Representation of the proposed methodology. The upper part shows the 3D convolutional architecture, where each row represents different inputs in the network, and the bottom part presents the feature maps of the model

Explanability maps

Predictive capabilities in clinical settings face challenges associated with the poor interpretability of output predictions. To overcome such limitation, this work implemented a spatio-temporal gradient-weighted class activation mapping (Grad-CAM) approximation as an alternative visualization tool to support PD predictions (Fig. 2). Particularly, Grad-CAM maps were adapted to the volumetric responses of the different 3D convolutional layers in the proposed architecture. This strategy generates a heat map, highlighting the critical regions in a specific network input by means of the gradient information that flows from a certain output probability to a specific convolutional layer in the proposed CNN. To obtain the discriminant location map regarding the class c and an input video, the gradient of the probability score of that class (y ^c ) is first computed with respect to the A ^k feature maps of a convolutional layer. These backward-flowing gradients are spatially averaged to obtain the weights

Where is the global average pooling, and denotes the gradients via backpropagation. After calculating the weights for the target class c, a weighted combination of activation maps is followed by a rectified linear unit (ReLU). The ReLU is applied to the linear combination since only the features that positively influence the class are of interest, and it highlights the associated regions as follows:

where is the linear combination. Finally, an upsampling guarantees the same resolution as the original image.

Figure 2: Sum-pooling strategy for feature maps in a convolutional layer: a) input image, b) feature maps of a given convolutional layer, c) sum-pooling

Experimental setup

Evaluation and results

The proposed approach aims to achieve proper differentiation between control and PD patient groups regarding hidden tremor patterns in standard video sequences recorded in postural and resting hand configurations. For validation, the videos were projected in two different versions: raw and magnified sequences. The latter were obtained after applying a temporal filter that highlights key frequencies related to PD. Particularly, classical Euler magnification ¹² was employed as a pre-processing stage.

A first validation was carried out with the aim of selecting the best temporal sequence length to discriminate between PD and control patients. Fig. 3 illustrates the model’s performance in terms of accuracy and the F1 score. To perform an ablation study, the video clips were downsampled to F = (8,12,16,24) frames per video. As expected, a low frame representation level led to poor spatiotemporal video patterns. On the other hand, through a medium frame representation ¹², the model achieved the best results, given its compact tremor feature characterization. It is important to note that these videos were recorded within a range of 12-15 seconds and at a typical PD tremor frequency ¹³. Consequently, the proposed approach shows potential in distinguishing between control subjects and patients with mild tremors in the early stages of the disease. The differentiation capabilities of full video frames were limited due to redundancy between frames.

Figure 3: Frame variation for the best architecture configuration. The x-axis shows the frame variation, while the y-axis shows the accuracy values and the F1 score (blue and red lines, respectively)

Secondly, the spatiotemporal network’s contribution was analyzed while considering three and five 3D convolutional layers, varying the number of dense layers and their corresponding number of units. Fig. 4 illustrates the network’s performance concerning accuracy and the F1 score in standard and magnified video sequences for both configurations. As expected, the best model performance was achieved through the deepest configuration (five convolutional layers and two dense layers with 2048 units). This behavior reinforces the hypothesis that hidden hand tremors could be properly coded along the temporal dimension. Nevertheless, it is important to note that a deeper configuration alone is not enough: the 5C+Dense net configuration always reports the worst performance and is even surpassed by the standard 3C setup. In this vein, an additional dense layer offers a suitable spatiotemporal representation.

Figure 4: Model performance with respect to variations in the number of convolutional and dense layers. From left to right, model performance for the resting and postural configurations. From top to bottom, model performance concerning standard and magnified video sequences

Additionally, Table I shows a detailed metric evaluation of the best model configuration for each type of video sequence. As expected, the magnified postural configuration obtained the highest F1, sensitivity, and accuracy scores. In such cases, a major effort of the limbs increases the physical effort required to keep the arms in a fixed position, which contributes to a natural exaggeration of the tremor, which may even reveal patterns that cannot be observed in the resting configuration. Particularly, tremors are more constant in the postural configuration, given the muscular effort required to maintain a posture against gravity. Thus, amplifying these standard movements significantly improves the model’s performance.

Table I: Model performance across different video configurations

Interestingly enough, a perfect precision score was obtained under the resting hand configuration without magnification. This may be associated with the digital noise introduced by the synthetic exaggeration caused by optical magnification, i.e., control movements not associated with Parkinsonian patterns may be exaggerated, resulting in reduced precision and accuracy. Euler magnification is subject to a global constraint that may increase the dynamic noise of video sequences, even surpassing patients’ natural tremor. From a clinical perspective, it is important to use video materials that provide a clearer and less ambiguous signal. In this vein, standard resting videos are less prone to model misinterpretation than magnified ones. In light of the above, the proposed scheme yields promising results in distinguishing PD patients from control subjects, minimizing the false positive rate. It could be regarded as a potential clinical tool to support medical decisions while only requiring standard video sequences. Specifically, it could potentially discriminate between two types of tremors observed in clinical practice, i.e., postural and resting tremors. In this context, a differentiated diagnosis of tremors could support the development of more personalized treatments, thereby improving the patient’s quality of life ¹⁴.

Complementarily, Fig. 5 shows the training adjustment performed for the postural magnified configuration architecture. It can be observed that accuracy increases while losses decrease consistently for all folds until a stable configuration is reached. Also note that, in the first epochs, there are remarkable variations in the results and the stabilization of losses. Nonetheless, as the number of epochs increases, these variations are significantly reduced, showing a greater confidence in the results.

Figure 5: Training for the magnified postural architectures through k-fold cross-validation. This plot shows the mean accuracy and losses values achieved during each epoch, together with their variance (vertical lines).

As a second phase in the validation and evaluation of the proposed methodology, explainability maps were calculated to interpret and support the predictions. Notably, Grad-CAM offers weighted

volumetric visualization across different video regions, with a strong influence on the decision. Fig. 6 shows the Grad-CAM algorithm employed for the same magnified sequences of control and PD patients. Our proposal suggests the palms of the hands as the main regions to achieve control predictions. Even more interesting is the fact that, for the PD patients, the model seems to focus on the boundary regions of the hands, which are prone to slight movements. These regions are associated with a greater perception of tremors (mainly the fingertips). It can thus be assumed that convolutional bank filters computed from 3D convolutional kernels pay attention to the subtle motion of the hand and the arm. In fact, the prediction weighted with such activation shows an association with tremors. In clinical practice, there is a phenomenon called pill-rolling, wherein the patient’s tremor is predominantly observed in the thumb and forefinger ¹⁵. Notably, with an early diagnosis of this condition, the symptoms of PD can be alleviated through medication and physical therapy, helping the patient to regain independence. In this vein, the proposed approach exhibits a robust behavior in codifying hidden hand tremors in regions that are most sensitive to the patient’s movement in both resting and postural configurations.

Figure 6: Heat maps obtained using the Grad-CAM algorithm. From left to right, PD and control subjects. Columns a, b, and c show the input and the response maps of the third and fourth convolutional layers, respectively

Conclusions and future work

Nowadays, PD diagnosis strongly depends on tremor observations based on medical expertise. These observations only consider strong changes in motion, which hinders the monitoring of PD progression given the high variance in the amplitude and frequency associated with the motion of a particular patient ¹⁶. To overcome this issue, this work presented a deep volumetric strategy that represents postural and tremor configurations through video sequences, with the purpose of identifying tremors associated with PD. The strategy included a convolutional architecture that extracts spatiotemporal patterns correlated with tremors and propagates them through different layers until PD and control subjects are distinguished from each other. In addition, explainability maps were computed by backpropagating output gradients into convolutionally learned spatiotemporal maps.

This approach was tested with five patients and five control subjects, for a total of 80 video sequences, showing promising results for predicting PD from video. The two studied configurations were seen to be complementary, demonstrating the importance of using both in clinical practice. Magnified postural patterns can contribute to a greater classification sensitivity, and standard resting videos can increase precision. However, some prediction errors were observed, which may be associated with the high variability in the intensity and frequency of the patients’ tremors. Particularly, in patients with early signs of the disease, the tremors were less pronounced and less consistent throughout the

video, especially in resting configurations. In such cases, processing the video samples and adequately classifying the patients concerning the control population become challenging tasks. Despite these limitations, postural effort was seen as appropriate for indicating abnormal patterns associated with PD. Moreover, processing raw videos of the resting configuration tests was more effective than using optical Eulerian magnification. The proposed 3D convolutional network is able to capture localized tremors without any optical exaggeration, providing an accurate and useful representation for the classification model. This is in line with the preference for standard video configurations in the context of medical diagnostics, where precision and reliability are critical.

The results of the proposed approach are very descriptive, highlighting its potential to support observational analysis by enabling the identification of spatiotemporal regions with stronger associations to tremors. Therefore, the explainability maps of the control subjects show a focus on the palm as opposed to those of the patients, which reveal a focus, for instance, on the thumb and the fingertips. The results suggest a potential link between the pill-rolling PD tremor (a tremor of the thumb and the index finger) and the explainability maps. Our approach was validated by studying a reduced number of subjects, given the difficulties in acquiring data from more patients. The main issue concerning data is the ability to quantify populations with comparable demographic characteristics in order to develop approaches that allow representing and classifying disease patterns.

In light of the above, we propose further validation with larger datasets and patients stratified according to the stages of the disease. This may be useful to identify new motion patterns that contribute to the precision of clinical predictions in relation to PD progression or the effectiveness of a particular treatment. Furthermore, the inclusion of new cardinal symptoms might improve the representation and the tool’s impact on diagnostic support