Abstract

Efficient human–robot collaboration during physical interaction requires estimating the human state for optimal role allocation and load sharing. Machine learning (ML) methods are gaining popularity for estimating the interaction parameters from physiological signals. However, due to individual differences, the ML models might not generalize well to new subjects. In this study, we present a convolution neural network (CNN) model to predict motor control difficulty using surface electromyography (sEMG) from human upper limb during physical human–robot interaction (pHRI) task and present a transfer learning approach to transfer a learned model to new subjects. Twenty-six individuals participated in a pHRI experiment where a subject guides the robot's end-effector with different levels of motor control difficulty. The motor control difficulty is varied by changing the damping parameter of the robot from low to high and constraining the motion to gross and fine movements. A CNN network with raw sEMG as input is used to classify the motor control difficulty. The CNN's transfer learning approach is compared against Riemann geometry-based Procrustes analysis (RPA). With very few labeled samples from new subjects, we demonstrate that the CNN-based transfer learning approach (avg. 69.77%) outperforms the RPA transfer learning (avg. 59.20%). Moreover, we observe that the subject's skill level in the pre-trained model has no significant effect on the transfer learning performance of the new users.

References

1.
Maurtua
,
I.
,
Ibarguren
,
A.
,
Kildal
,
J.
,
Susperregi
,
L.
, and
Sierra
,
B.
,
2017
, “
Human–Robot Collaboration in Industrial Applications: Safety, Interaction and Trust
,”
Int. J. Adv. Robot. Syst.
,
14
(
4
), p.
1729881417716010
.
2.
Vaughan
,
N.
,
Gabrys
,
B.
, and
Dubey
,
V. N.
,
2016
, “
An Overview of Self-Adaptive Technologies Within Virtual Reality Training
,”
Comput. Sci. Rev.
,
22
, pp.
65
87
.
3.
Kim
,
J.
,
Campbell
,
A. S.
,
de Ávila
,
B. E.-F.
, and
Wang
,
J.
,
2019
, “
Wearable Biosensors for Healthcare Monitoring
,”
Nat. Biotechnol.
,
37
(
4
), pp.
389
406
.
4.
Reilly
,
R. B.
, and
Lee
,
T. C.
,
2010
, “
Electrograms (ECG, EEG, EMG, EOG)
,”
Technol. Health Care
,
18
(
6
), pp.
443
458
.
5.
Côté-Allard
,
U.
,
Fall
,
C. L.
,
Drouin
,
A.
,
Campeau-Lecours
,
A.
,
Gosselin
,
C.
,
Glette
,
K.
,
Laviolette
,
F.
, and
Gosselin
,
B.
,
2019
, “
Deep Learning for Electromyographic Hand Gesture Signal Classification Using Transfer Learning
,”
IEEE Trans. Neural Syst. Rehabil. Eng.
,
27
(
4
), pp.
760
771
.
6.
Ameri
,
A.
,
Akhaee
,
M. A.
,
Scheme
,
E.
, and
Englehart
,
K.
,
2019
, “
A Deep Transfer Learning Approach to Reducing the Effect of Electrode Shift in EMG Pattern Recognition-Based Control
,”
IEEE Trans. Neural Syst. Rehabil. Eng.
,
28
(
2
), pp.
370
379
.
7.
Aune
,
T. K.
,
Ingvaldsen
,
R.
, and
Ettema
,
G.
,
2008
, “
Effect of Physical Fatigue on Motor Control at Different Skill Levels
,”
Percept. Motor Skills
,
106
(
2
), pp.
371
386
.
8.
Hogan
,
N.
,
1985
, “
Impedance Control: An Approach to Manipulation: Part I—Theory
,”
J. Dyn. Syst. Meas. Control
,
107
(
1
), pp.
1
7
.
9.
Calanca
,
A.
,
Muradore
,
R.
, and
Fiorini
,
P.
,
2015
, “
A Review of Algorithms for Compliant Control of Stiff and Fixed-Compliance Robots
,”
IEEE/ASME Trans. Mechatron.
,
21
(
2
), pp.
613
624
.
10.
Grafakos
,
S.
,
Dimeas
,
F.
, and
Aspragathos
,
N.
,
2016
, “
Variable Admittance Control in pHRI Using EMG-Based Arm Muscles Co-Activation
,”
2016 IEEE International Conference on Systems, Man, and Cybernetics
,
Budapest, Hungary
,
Oct. 9–12
, IEEE, pp.
001900
001905
.
11.
Gopinathan
,
S.
,
Ötting
,
S. K.
, and
Steil
,
J. J.
,
2017
, “
A User Study on Personalized Stiffness Control and Task Specificity in Physical Human–Robot Interaction
,”
Front. Robot. AI
,
4
, pp.
5
20
.
12.
Bian
,
F.
,
Ren
,
D.
,
Li
,
R.
, and
Liang
,
P.
,
2018
, “
Improving Stability in Physical Human–Robot Interaction by Estimating Human Hand Stiffness and a Vibration Index
,”
Ind. Robot.
,
46
(
4
), pp.
529
540
.
13.
Jujjavarapu
,
S. S.
, and
Esfahani
,
E. T.
,
2019
, “
Stiffness Based Stability Enhancement in Human–Robot Collaboration
,”
International Design Engineering Technical Conferences and Computers and Information in Engineering Conference
,
Anaheim, CA
,
Aug. 18–21
, p. V05AT07A019.
14.
Keemink
,
A. Q.
,
van der
,
K. H.
, and
Stienen
,
A. H.
,
2018
, “
Admittance Control for Physical Human–Robot Interaction
,”
Int. J. Robot. Res.
,
37
(
11
), pp.
1421
1444
.
15.
Memar
,
A. H.
, and
Esfahani
,
E. T.
,
2018
, “
EEG Correlates of Motor Control Difficulty in Physical Human–Robot Interaction: A Frequency Domain Analysis
,”
2018 IEEE Haptics Symposium
,
San Francisco, CA
,
Mar. 25–28
, IEEE, pp.
229
234
.
16.
Novak
,
D.
,
Beyeler
,
B.
,
Omlin
,
X.
, and
Riener
,
R.
,
2014
, “
Workload Estimation in Physical Human–Robot Interaction Using Physiological Measurements
,”
Interact. Comput.
,
27
(
6
), pp.
616
629
.
17.
Nazmi
,
N.
,
Abdul Rahman
,
M.
,
Yamamoto
,
S.-I.
,
Ahmad
,
S.
,
Zamzuri
,
H.
, and
Mazlan
,
S.
,
2016
, “
A Review of Classification Techniques of EMG Signals During Isotonic and Isometric Contractions
,”
Sensors
,
16
(
8
), p.
1304
.
18.
Seashore
,
R. H.
,
1930
, “
Individual Differences in Motor Skills
,”
J. Gen. Psychol.
,
3
(
1
), pp.
38
66
.
19.
Taborri
,
J.
,
Palermo
,
E.
,
Masiello
,
D.
, and
Rossi
,
S.
,
2017
, “
Factorization of EMG Via Muscle Synergies in Walking Task: Evaluation of Intra-Subject and Inter-Subject Variability
,”
2017 IEEE International Instrumentation and Measurement Technology Conference
,
Torino, Italy
,
IEEE
, pp.
1
6
.
20.
Faust
,
O.
,
Hagiwara
,
Y.
,
Hong
,
T. J.
,
Lih
,
O. S.
, and
Acharya
,
U. R.
,
2018
, “
Deep Learning for Healthcare Applications Based on Physiological Signals: A Review
,”
Comput. Methods Programs Biomed.
,
161
, pp.
1
13
.
21.
Gao
,
Y.
,
Hendricks
,
L. A.
,
Kuchenbecker
,
K. J.
, and
Darrell
,
T.
,
2016
, “
Deep Learning for Tactile Understanding From Visual and Haptic Data
,”
2016 IEEE International Conference on Robotics and Automation
,
Stockholm, Sweden
,
May 16–21
,
IEEE
, pp.
536
543
.
22.
Hu
,
Y.
,
Wong
,
Y.
,
Wei
,
W.
,
Du
,
Y.
,
Kankanhalli
,
M.
, and
Geng
,
W.
,
2018
, “
A Novel Attention-Based Hybrid CNN-RNN Architecture for sEMG-Based Gesture Recognition
,”
PLoS One
,
13
(
10
), p.
e0206049
.
23.
Manjunatha
,
H.
,
2021
, “
Addressing Stability, Transferability, and Interpretability Issues in Physical Human–Robot Interaction Using Physiological Data and Deep Learning
,”
PhD thesis
,
State University of New York at Buffalo
.
24.
Tan
,
C.
,
Sun
,
F.
,
Kong
,
T.
,
Zhang
,
W.
,
Yang
,
C.
, and
Liu
,
C.
,
2018
, “
A Survey on Deep Transfer Learning
,”
International Conference on Artificial Neural Networks
,
Rhodes, Greece
,
Oct. 4–7
,
Springer
, pp.
270
279
.
25.
Du
,
Y.
,
Jin
,
W.
,
Wei
,
W.
,
Hu
,
Y.
, and
Geng
,
W.
,
2017
, “
Surface EMG-Based Inter-Session Gesture Recognition Enhanced by Deep Domain Adaptation
,”
Sensors
,
17
(
3
), p.
458
.
26.
Li
,
Q.
,
Zhang
,
A.
,
Li
,
Z.
, and
Wu
,
Y.
,
2021
, “
Improvement of EMG Pattern Recognition Model Performance in Repeated Uses by Combining Feature Selection and Incremental Transfer Learning
,”
Front. Neurorobot.
,
15
, pp.
699174.1
699174.15
.
27.
Xiong
,
D.
,
Zhang
,
D.
,
Zhao
,
X.
, and
Zhao
,
Y.
,
2021
, “
Deep Learning for EMG-Based Human–Machine Interaction: A Review
,”
IEEE/CAA J. Autom. Sin.
,
8
(
3
), pp.
512
533
.
28.
Schirrmeister
,
R. T.
,
Springenberg
,
J. T.
,
Fiederer
,
L. D. J.
,
Glasstetter
,
M.
,
Eggensperger
,
K.
,
Tangermann
,
M.
,
Hutter
,
F.
,
Burgard
,
W.
, and
Ball
,
T.
,
2017
, “
Deep Learning With Convolutional Neural Networks for EEG Decoding and Visualization
,”
Hum. Brain Mapp.
,
38
(
11
), pp.
5391
5420
.
29.
Passalis
,
N.
,
Tefas
,
A.
,
Kanniainen
,
J.
,
Gabbouj
,
M.
, and
Iosifidis
,
A.
,
2020
, “
Deep Adaptive Input Normalization for Time Series Forecasting
,”
IEEE Trans. Neural Netw. Learn. Syst.
,
31
(
9
), pp.
3760
3765
.
30.
Barachant
,
A.
,
Bonnet
,
S.
,
Congedo
,
M.
, and
Jutten
,
C.
,
2011
, “
Multiclass Brain–Computer Interface Classification by Riemannian Geometry
,”
IEEE Trans. Biomed. Eng.
,
59
(
4
), pp.
920
928
.
31.
Pan
,
L.
,
Zhang
,
D.
,
Jiang
,
N.
,
Sheng
,
X.
, and
Zhu
,
X.
,
2015
, “
Improving Robustness Against Electrode Shift of High Density EMG for Myoelectric Control Through Common Spatial Patterns
,”
J. NeuroEng. Rehabil.
,
12
(
1
), p.
110
.
32.
Congedo
,
M.
,
Barachant
,
A.
, and
Bhatia
,
R.
,
2017
, “
Riemannian Geometry for EEG-Based Brain–Computer Interfaces; a Primer and a Review
,”
Brain Comput. Interface
,
4
(
3
), pp.
155
174
.
33.
Barachant
,
A.
,
Carmel
,
J. B.
,
Friel
,
K. M.
, and
Gupta
,
D.
,
2016
, “
Extraction of Motor Patterns From Joint EEG/EMG Recording: A Riemannian Geometry Approach
,”
Sixth International Brain-Computer Interface Meeting: BCI Past, Present, and Future
,
Monterey, CA
,
May 30–June 3
,
p. 181
.
34.
Manjunatha
,
H.
,
Jujjavarapu
,
S. S.
, and
Esfahani
,
E. T.
,
2020
, “
Classification of Motor Control Difficulty Using EMG in Physical Human–Robot Interaction
,”
2020 IEEE International Conference on Systems, Man, and Cybernetics (SMC)
,
Virtual
, pp.
2708
2713
.
35.
Rodrigues
,
P. L. C.
,
Jutten
,
C.
, and
Congedo
,
M.
,
2019
, “
Riemannian Procrustes Analysis: Transfer Learning for Brain–Computer Interfaces
,”
IEEE Trans. Biomed. Eng.
,
66
(
8
), pp.
2390
2401
.
36.
Pan
,
S. J.
, and
Yang
,
Q.
,
2009
, “
A Survey on Transfer Learning
,”
IEEE Trans. Knowl. Data Eng.
,
22
(
10
), pp.
1345
1359
.
37.
Azizpour
,
H.
,
Razavian
,
A. S.
,
Sullivan
,
J.
,
Maki
,
A.
, and
Carlsson
,
S.
,
2015
, “
Factors of Transferability for a Generic Convnet Representation
,”
IEEE Trans. Pattern Anal. Mach. Intell.
,
38
(
9
), pp.
1790
1802
.
38.
Garcia-Gasulla
,
D.
,
Parés
,
F.
,
Vilalta
,
A.
,
Moreno
,
J.
,
Ayguadé
,
E.
,
Labarta
,
J.
,
Cortés
,
U.
, and
Suzumura
,
T.
,
2018
, “
On the Behavior of Convolutional Nets for Feature Extraction
,”
J. Artif. Intell. Res.
,
61
(
1
), pp.
563
592
.
39.
Bird
,
J. J.
,
Kobylarz
,
J.
,
Faria
,
D. R.
,
Ekárt
,
A.
, and
Ribeiro
,
E. P.
,
2020
, “
Cross-Domain MLP and CNN Transfer Learning for Biological Signal Processing: EEG and EMG
,”
IEEE Access
,
8
, pp.
54789
54801
.
40.
Sun
,
T.
,
Hu
,
Q.
,
Libby
,
J.
, and
Atashzar
,
S. F.
,
2022
, “
Deep Heterogeneous Dilation of LSTM for Transient-Phase Gesture Prediction Through High-Density Electromyography: Towards Application in Neurorobotics
,”
IEEE Robot. Autom. Lett.
,
7
(
2
), pp.
2851
2858
.
41.
van der
,
M. L.
, and
Hinton
,
G.
,
2008
, “
Visualizing Data Using t-SNE
,”
J. Mach. Learn. Res.
,
9
, pp.
2579
2605
.
42.
Dimeas
,
F.
, and
Aspragathos
,
N.
,
2016
, “
Online Stability in Human–Robot Cooperation With Admittance Control
,”
IEEE Trans. Haptics
,
9
(
2
), pp.
267
278
.
You do not currently have access to this content.