Multimodal collaborative BCI system based on the improved CSP feature extraction algorithm

Abstract

Background

As a novel approach for people to directly communicate with an external device, the study of brain-computer interfaces (BCIs) has become well-rounded. However, similar to the real-world scenario, where individuals are expected to work in groups, the BCI systems should be able to replicate group attributes.

Methods

We proposed a 4-order cumulants feature extraction method (CUM4-CSP) based on the common spatial patterns (CSP) algorithm. Simulation experiments conducted using motion visual evoked potentials (mVEP) EEG data verified the robustness of the proposed algorithm. In addition, to freely choose paradigms, we adopted the mVEP and steady-state visual evoked potential (SSVEP) paradigms and designed a multimodal collaborative BCI system based on the proposed CUM4-CSP algorithm. The feasibility of the proposed multimodal collaborative system framework was demonstrated using a multiplayer game controlling system that simultaneously facilitates the coordination and competitive control of two users on external devices. To verify the robustness of the proposed scheme, we recruited 30 subjects to conduct online game control experiments, and the results were statistically analyzed.

Results

The simulation results prove that the proposed CUM4-CSP algorithm has good noise immunity. The online experimental results indicate that the subjects could reliably perform the game confrontation operation with the selected BCI paradigm.

Conclusions

The proposed CUM4-CSP algorithm can effectively extract features from EEG data in a noisy environment. Additionally, the proposed scheme may provide a new solution for EEG-based group BCI research.

Keyword

Collaborative brain-computer interface (BCI) ; Motion visual evoked potentials (mVEP) ; Steady-state visual evoked potential (SSVEP) ; Game controlling system

Cite this article

Cunbo LI, Ning LI, Yuan QIU, Yueheng PENG, Yifeng WANG, Lili DENG, Teng MA, Fali LI, Dezhong YAO, Peng XU. Multimodal collaborative BCI system based on the improved CSP feature extraction algorithm. Virtual Reality & Intelligent Hardware, 2022, 4(1): 22-37 DOI:10.1016/j.vrih.2022.01.002

References

1. Piho L, Tjahjadi T. A mutual information based adaptive windowing of informative EEG for emotion recognition. IEEE Transactions on Affective Computing, 2020, 11(4): 722–735 DOI:10.1109/taffc.2018.2840973

2. Long J Y, Li Y Q, Yu T Y, Gu Z H. Target selection with hybrid feature for BCI-based 2-D cursor control. IEEE Transactions on Biomedical Engineering, 2012, 59(1): 132–140 DOI:10.1109/tbme.2011.2167718

3. Khan M A, Das R, Iversen H K, Puthusserypady S. Review on motor imagery based BCI systems for upper limb post-stroke neurorehabilitation: From designing to application. Computers in Biology and Medicine, 2020, 123: 103843 DOI:10.1016/j.compbiomed.2020.103843

4. Farwell L A, Donchin E. Talking off the top of your head: toward a mental prosthesis utilizing event-related brain potentials. Electroencephalography and Clinical Neurophysiology, 1988, 70(6): 510–523 DOI:10.1016/0013-4694(88)90149-6

5. Pfurtscheller G, Neuper C. Motor imagery and direct brain-computer communication. Proceedings of the IEEE, 2001, 89(7): 1123–1134 DOI:10.1109/5.939829

6. Guo F, Hong B, Gao X R, Gao S K. A brain–computer interface using motion-onset visual evoked potential. Journal of Neural Engineering, 2008, 5(4): 477–485 DOI:10.1088/1741-2560/5/4/011

7. Middendorf M, McMillan G, Calhoun G, Jones K S. Brain-computer interfaces based on the steady-state visual-evoked response. IEEE Transactions on Rehabilitation Engineering, 2000, 8(2): 211–214 DOI:10.1109/86.847819

8. Qian K, Nikolov P, Huang D D, Fei D Y, Chen X D, Bai O. A motor imagery-based online interactive brain-controlled switch: Paradigm development and preliminary test. Clinical Neurophysiology, 2010, 121(8): 1304–1313 DOI:10.1016/j.clinph.2010.03.001

9. Zhang Y, Nam C S, Zhou G X, Jin J, Wang X Y, Cichocki A. Temporally constrained sparse group spatial patterns for motor imagery BCI. IEEE Transactions on Cybernetics, 2019, 49(9): 3322–3332 DOI:10.1109/tcyb.2018.2841847

10. Beveridge R, Wilson S, Coyle D. 3D graphics, virtual reality, and motion-onset visual evoked potentials in neurogaming. Progress in Brain Research, 2016, 228: 329–353 DOI:10.1016/bs.pbr.2016.06.006

11. Guo F, Hong B, Gao X R, Gao S K. A brain–computer interface using motion-onset visual evoked potential. Journal of Neural Engineering, 2008, 5(4): 477–485 DOI:10.1088/1741-2560/5/4/011

12. Chen J J, Li Z R, Hong B, Maye A, Engel A K, Zhang D. A single-stimulus, multitarget BCI based on retinotopic mapping of motion-onset VEPs. IEEE Transactions on Biomedical Engineering, 2019, 66(2): 464–470 DOI:10.1109/tbme.2018.2849102

13. Beveridge R, Wilson S, Callaghan M, Coyle D. Neurogaming with motion-onset visual evoked potentials (mVEPs): adults versus teenagers. IEEE Transactions on Neural Systems and Rehabilitation Engineering, 2019, 27(4): 572–581 DOI:10.1109/tnsre.2019.2904260

14. Bakardjian H, Tanaka T, Cichocki A. Optimization of SSVEP brain responses with application to eight-command Brain-Computer Interface. Neuroscience Letters, 2010, 469(1): 34–38 DOI:10.1016/j.neulet.2009.11.039

15. Cecotti H. A self-paced and calibration-less SSVEP-based brain–computer interface speller. IEEE Transactions on Neural Systems and Rehabilitation Engineering, 2010, 18(2): 127–133 DOI:10.1109/tnsre.2009.2039594

16. Pfurtscheller G, Solis-Escalante T, Ortner R, Linortner P, Muller-Putz G R. Self-paced operation of an SSVEP-based orthosis with and without an imagery-based “brain switch: ”A feasibility study towards a hybrid BCI. IEEE Transactions on Neural Systems and Rehabilitation Engineering, 2010, 18(4): 409–414 DOI:10.1109/tnsre.2010.2040837

17. Ortner R, Allison B Z, Korisek G, Gaggl H, Pfurtscheller G. An SSVEP BCI to control a hand orthosis for persons with tetraplegia. IEEE Transactions on Neural Systems and Rehabilitation Engineering, 2011, 19(1): 1–5 DOI:10.1109/tnsre.2010.2076364

18. McFarland D J, Wolpaw J R. EEG-based brain-computer interfaces. Current Opinion in Biomedical Engineering, 2017, 4: 194–200 DOI:10.1016/j.cobme.2017.11.004

19. LaFleur K, Cassady K, Doud A, Shades K, Rogin E, He B. Quadcopter control in three-dimensional space using a noninvasive motor imagery-based brain-computer interface. Journal of Neural Engineering, 2013, 10(4): 046003 DOI:10.1088/1741-2560/10/4/046003

20. Doud A J, Lucas J P, Pisansky M T, He B. Continuous three-dimensional control of a virtual helicopter using a motor imagery based brain-computer interface. PLoS One, 2011, 6(10): e26322 DOI:10.1371/journal.pone.0026322

21. Li Y, Pan J, Wang F, Yu Z. A hybrid BCI system combining P300 and SSVEP and its application to wheelchair control. IEEE Transactions on Bio-Medical Engineering, 2013, 60(11): 3156–3166 DOI:10.1109/tbme.2013.2270283

22. Cecotti H. A self-paced and calibration-less SSVEP-based brain–computer interface speller. IEEE Transactions on Neural Systems and Rehabilitation Engineering, 2010, 18(2): 127–133 DOI:10.1109/tnsre.2009.2039594

23. Pichiorri F, Morone G, Petti M, Toppi J, Pisotta I, Molinari M, Paolucci S, Inghilleri M, Astolfi L, Cincotti F, Mattia D. Brain-computer interface boosts motor imagery practice during stroke recovery. Annals of Neurology, 2015, 77(5): 851–865 DOI:10.1002/ana.24390

24. Ang K K, Chua K S G, Phua K S, Wang C C, Chin Z Y, Kuah C W K, Low W, Guan C T. A randomized controlled trial of EEG-based motor imagery brain-computer interface robotic rehabilitation for stroke. Clinical EEG and Neuroscience, 2015, 46(4): 310–320 DOI:10.1177/1550059414522229

25. Little S, Pogosyan A, Neal S, Zavala B, Zrinzo L, Hariz M, Foltynie T, Limousin P, Ashkan K, FitzGerald J, Green A L, Aziz T Z, Brown P. Adaptive deep brain stimulation in advanced Parkinson disease. Annals of Neurology, 2013, 74(3): 449–457 DOI:10.1002/ana.23951

26. Nijboer F, Sellers E W, Mellinger J, Jordan M A, Matuz T, Furdea A, Halder S, Mochty U, Krusienski D J, Vaughan T M, Wolpaw J R, Birbaumer N, Kübler A. A P300-based brain-computer interface for people with amyotrophic lateral sclerosis. Clinical Neurophysiology, 2008, 119(8): 1909–1916 DOI:10.1016/j.clinph.2008.03.034

27. Yao P, Xu G, Jia L, Duan J, Han C, Tao T, Wang Y, Zhang S. Multiscale noise suppression and feature frequency extraction in SSVEP based on underdamped second-order stochastic resonance. Journal of Neural Engineering, 2019, 16(3): 036032 DOI:10.1088/1741-2552/ab16f9

28. Wu H, Niu Y, Li F, Li Y C, Fu B X, Shi G M, Dong M H. A parallel multiscale filter bank convolutional neural networks for motor imagery EEG classification. Frontiers in Neuroscience, 2019, 13: 1275 DOI:10.3389/fnins.2019.01275

29. Yan W Q, Xu G H, Chen L T, Zheng X W. Steady-state motion visual evoked potential (SSMVEP) enhancement method based on time-frequency image fusion. Computational Intelligence and Neuroscience, 2019, 9439407 DOI:10.1155/2019/9439407

30. Zhou Y J, He S H, Huang Q Y, Li Y Q. A hybrid asynchronous brain-computer interface combining SSVEP and EOG signals. IEEE Transactions on Biomedical Engineering, 2020, 67(10): 2881–2892 DOI:10.1109/tbme.2020.2972747

31. Pfurtscheller G, Allison B Z, Brunner C. The hybrid BCI. Frontiers in Neuroscience, 2010, 4(30): 30–30

32. Ma T, Li H, Deng L L, Yang H, Lv X, Li P Y, Li F L, Zhang R, Liu T J, Yao D Z, Xu P. The hybrid BCI system for movement control by combining motor imagery and moving onset visual evoked potential. Journal of Neural Engineering, 2017, 14(2): 026015 DOI:10.1088/1741-2552/aa5d5f

33. Kevric J, Subasi A. Comparison of signal decomposition methods in classification of EEG signals for motor-imagery BCI system. Biomedical Signal Processing and Control, 2017, 31: 398–406 DOI:10.1016/j.bspc.2016.09.007

34. Xu M P, Han J, Wang Y J, Jung T P, Ming D. Implementing over 100 command codes for a high-speed hybrid brain-computer interface using concurrent P300 and SSVEP features. IEEE Transactions on Biomedical Engineering, 2020, 67(11): 3073–3082 DOI:10.1109/tbme.2020.2975614

35. Lotte F, Guan C T. Regularizing common spatial patterns to improve BCI designs: unified theory and new algorithms. IEEE Transactions on Biomedical Engineering, 2011, 58(2): 355–362 DOI:10.1109/tbme.2010.2082539

36. He S, Tan H, Li Y. EEG- and EOG-based asynchronous hybrid BCI: a system integrating a speller, a web browser, an E-mail client, and a file explorer[J]. IEEE Transactions on Neural Systems and Rehabilitation Engineering, 2019, 99: 1.

37. Cecotti H, Graser A. Convolutional neural networks for P300 detection with application to brain-computer interfaces. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2011, 33(3): 433–445 DOI:10.1109/tpami.2010.125

38. Lotte F, Bougrain L, Cichocki A, Clerc M, Congedo M, Rakotomamonjy A, Yger F. A review of classification algorithms for EEG-based brain-computer interfaces: a 10 year update. Journal of Neural Engineering, 2018, 15(3): 031005 DOI:10.1088/1741-2552/aab2f2

39. Gao Z K, Yuan T, Zhou X J, Ma C, Ma K, Hui P. A deep learning method for improving the classification accuracy of SSMVEP-based BCI. IEEE Transactions on Circuits and Systems II: Express Briefs, 2020, 67(12): 3447–3451 DOI:10.1109/tcsii.2020.2983389

40. Liu D K, Liu C, Hong B. Bi-directional visual motion based BCI speller. In: 2019 9th International IEEE/EMBS Conference on Neural Engineering (NER). San Francisco, CA, USA, IEEE, 2019, 589–592 DOI:10.1109/ner.2019.8717075

41. Allison B Z, Brunner C, Kaiser V, Müller-Putz G R, Neuper C, Pfurtscheller G. Toward a hybrid brain-computer interface based on imagined movement and visual attention. Journal of Neural Engineering, 2010, 7(2): 26007 DOI:10.1088/1741-2560/7/2/026007

42. Ma J X, Zhang Y, Cichocki A, Matsuno F. A novel EOG/EEG hybrid human-machine interface adopting eye movements and ERPs: application to robot control. IEEE Transactions on Biomedical Engineering, 2015, 62(3): 876–889 DOI:10.1109/tbme.2014.2369483

43. Duan L L, Li J, Ji H F, Pang Z L, Zheng X C, Lu R R, Li M Z, Zhuang J. Zero-shot learning for EEG classification in motor imagery-based BCI system. IEEE Transactions on Neural Systems and Rehabilitation Engineering, 2020, 28(11): 2411–2419 DOI:10.1109/tnsre.2020.3027004

44. Dagdevir E, Tokmakci M. Optimization of preprocessing stage in EEG based BCI systems in terms of accuracy and timing cost. Biomedical Signal Processing and Control, 2021, 67: 102548 DOI:10.1016/j.bspc.2021.102548

45. Kaur B, Singh D, Roy P P. EEG based emotion classification mechanism in BCI. Procedia Computer Science, 2018, 132: 752–758 DOI:10.1016/j.procs.2018.05.087

46. Al-Nafjan A, Hosny M, Al-Ohali Y, Al-Wabil A. Review and classification of emotion recognition based on EEG brain-computer interface system research: a systematic review. Applied Sciences, 2017, 7(12): 1239 DOI:10.3390/app7121239

47. Zhu L, Su C W, Cui G C, Zhou C L, Zhang J H, Kong W Z. Idle-state detection in multi-user motor imagery brain computer interface with cross-brain CSP and hyper-brain-network. In: 2019 International Conference on Cyberworlds (CW). Kyoto, Japan, IEEE, 2019, 225–230 DOI:10.1109/cw.2019.00045

48. Zhang J H, Su C W, Zapala D, Zhu L, Cui G C, Kong W Z. A CNN-based Approach for three-class classification of motor imagery EEG data including ‘rest state’ in hybrid multi-user BCI. In: 2019 IEEE International Conference on Bioinformatics and Biomedicine (BIBM). San Diego, CA, USA, IEEE, 2019, 770–773 DOI:10.1109/bibm47256.2019.8983380

49. Sagee G S, Hema S. EEG feature extraction and classification in multiclass multiuser motor imagery brain computer interface using Bayesian Network and ANN. In: 2017 International Conference on Intelligent Computing, Instrumentation and Control Technologies (ICICICT). Kerala, India, IEEE, 2017, 938–943 DOI:10.1109/icicict1.2017.8342691

50. Korczowski L, Congedo M, Jutten C. Single-trial classification of multi-user P300-based Brain-Computer Interface using Riemannian geometry. 2015 37th Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC), 2015, 1769–1772 DOI:10.1109/embc.2015.7318721

51. Beveridge R, Wilson S, Coyle D. 3D graphics, virtual reality, and motion-onset visual evoked potentials in neurogaming. In: Progress in Brain Research. Amsterdam: Elsevier, 2016, 329–353 DOI:10.1016/bs.pbr.2016.06.006

52. Akhtar A, Norton J J S, Kasraie M, Bretl T. Playing checkers with your mind: an interactive multiplayer hardware game platform for brain-computer interfaces. In: 2014 36th Annual International Conference of the IEEE Engineering in Medicine and Biology Society. Chicago, IL, USA, IEEE, 2014, 1650–1653 DOI:10.1109/embc.2014.6943922

53. Bonnet L, Lotte F, Lécuyer A. Two brains, one game: design and evaluation of a multiuser BCI video game based on motor imagery. IEEE Transactions on Computational Intelligence and AI in Games, 2013, 5(2): 185–198 DOI:10.1109/tciaig.2012.2237173

54. Akhtar A, Norton J J S, Kasraie M, Bretl T. Playing checkers with your mind: an interactive multiplayer hardware game platform for brain-computer interfaces. In: 2014 36th Annual International Conference of the IEEE Engineering in Medicine and Biology Society. Chicago, IL, USA, IEEE, 2014, 1650–1653 DOI:10.1109/embc.2014.6943922

55. Homan R W, Herman J, Purdy P. Cerebral location of international 10-20 system electrode placement. Electroencephalography and Clinical Neurophysiology, 1987, 66(4): 376–382 DOI:10.1016/0013-4694(87)90206-9

56. Yu T Y, Xiao J, Wang F Y, Zhang R, Gu Z H, Cichocki A, Li Y Q. Enhanced motor imagery training using a hybrid BCI with feedback. IEEE Transactions on Biomedical Engineering, 2015, 62(7): 1706–1717 DOI:10.1109/tbme.2015.2402283

57. Pradhan A K, Routray A, Biswal B. Higher order statistics-fuzzy integrated scheme for fault classification of a series-compensated transmission line. IEEE Transactions on Power Delivery, 2004, 19(2): 891–893 DOI:10.1109/tpwrd.2003.820413

58. Xu Y Y, Dai S, Wu S Y, Chen J, Fang G Y. Vital sign detection method based on multiple higher order cumulant for ultrawideband radar. IEEE Transactions on Geoscience and Remote Sensing, 2012, 50(4): 1254–1265 DOI:10.1109/tgrs.2011.2164928

59. Martis R J, Acharya U R, Lim C M, Mandana K M, Ray A K, Chakraborty C. Application of higher order cumulant features for cardiac health diagnosis using ecg signals. International Journal of Neural Systems, 2013, 23(4): 1350014 DOI:10.1142/s0129065713500147

60. Tandra R, Sahai A. SNR walls for signal detection. IEEE Journal of Selected Topics in Signal Processing, 2008, 2(1): 4–17 DOI:10.1109/jstsp.2007.914879

61. Esfahani E T, Sundararajan V. Using brain–computer interfaces to detect human satisfaction in human–robot interaction. International Journal of Humanoid Robotics, 2011, 8(1): 87–101 DOI:10.1142/s0219843611002356

62. Atkinson J, Campos D. Improving BCI-based emotion recognition by combining EEG feature selection and kernel classifiers. Expert Systems with Applications, 2016, 47: 35–41 DOI:10.1016/j.eswa.2015.10.049