EEG Emotion Recognition Based on Hybrid Neural Networks

CAI Dongli; ZHONG Qinghua; ZHU Yongsheng; ZHANG Han

doi:10.6054/j.jscnun.2021017

Journal of South China Normal University (Natural Science Edition) > 2021 > 53(1): 109-118. > DOI: 10.6054/j.jscnun.2021017

CAI Dongli, ZHONG Qinghua, ZHU Yongsheng, ZHANG Han. EEG Emotion Recognition Based on Hybrid Neural Networks[J]. Journal of South China Normal University (Natural Science Edition), 2021, 53(1): 109-118. DOI: 10.6054/j.jscnun.2021017

Citation:

PDF (1603 KB)

EEG Emotion Recognition Based on Hybrid Neural Networks

School of Physics and Telecommunications Engineering, South China Normal University, Guangzhou 510006, China

More Information

Received Date: August 01, 2020
Available Online: March 23, 2021

Graphical Abstract

Abstract

Abstract

A hybrid neural network (3DCNN-BLSTM) based on a 3-Dimensional Convolutional Neural Network (3D-CNN) combined with a Bi-directional Long Short-term Memory Neural Network (BLSTM) is proposed to preserve the spatial information of the EEG while taking full advantage of its time-related information. Emotion re-cognition experiments on DEAP and SEED datasets are carried out to evaluate the classification performance of the model. The experiment results show that the 3DCNN-BLSTM model can effectively learn the correlation between EEG multi-channels and time dimension information and improve the performance of emotion classification. The ave-rage accuracy of emotion recognition of arousal and valence in the two-classification experiments on DEAP dataset are 93.56% and 93.21% respectively; the average accuracy of emotion recognition in the four-classification experiments on DEAP dataset is 90.97%; and the average accuracy of emotion recognition in the three-classification experiments on SEED dataset is 98.90%.
- EEG,
- emotion recognition,
- 3D-CNN,
- BLSTM,
- hybrid neural network

FullText(HTML)

References (30)

References

[1]	MEI H, XU X M. EEG based emotion classification using convolutional neural network[C]//Proceedings of the IEEE International Conference on Security, Pattern Analy-sis, and Cybernetics. Shenzhen: IEEE, 2017: 130-135.
[2]	TRIPATHI S, ACHARYA S, SHARMA R D, et al. Using deep and convolutional neural networks for accurate emotion classification on DEAP dataset[C]//Proceedings of the Thirty-First AAAI Conference on Artificial Intelligence. Palo Alto, CA: AAAI Press, 2017: 4746-4752.
[3]	MOON S, JANG S, LEE J, et al. Convolutional neural network approach for Eeg-based emotion recognition using brain connectivity and its spatial information[C]//Proceedings of 2018 IEEE International Conference on Acoustics, Speech and Signal Processing. Calgary, Canada: IEEE, 2018: 2556-2560.
[4]	KWON Y H, SHIN S B, KIM S, et al. Electroencephalogra-phy based fusion two-dimensional (2D)-convolution neural networks (CNN) model for emotion recognition system[J]. Sensors, 2018, 18(5): 1383/1-13. http://www.ncbi.nlm.nih.gov/pmc/articles/PMC5982398/
[5]	KOELSTRA S, MUHL C, SOLEYMANI M, et al. DEAP: a database for emotion analysis using physiological signals[J]. IEEE Transactions on Affective Computing, 2012, 3(1): 18-31. doi: 10.1109/T-AFFC.2011.15
[6]	SALAMA E S, ELKHORIBI R A, SHOMAN M, et al. EEG-based emotion recognition using 3D convolutional neural networks[J]. International Journal of Advanced Computer Science and Applications, 2018, 9(8): 329-337.
[7]	ALHAGRY S, FAHMY A A, ELKHORIBI R A, et al. Emotion recognition based on EEG using LSTM recurrent neural network[J]. International Journal of Advanced Computer Science and Applications, 2017, 8(10): 355-358. http://www.researchgate.net/publication/320802497_Emotion_Recognition_based_on_EEG_using_LSTM_Recurrent_Neural_Network
[8]	XING X F, LI Z Q, XU T Y, et al. SAE+LSTM: a new framework for emotion recognition from multi-channel EEG[J]. Frontiers in Neurorobotics, 2019, 13: 37/1-14. http://www.researchgate.net/publication/333730290_SAELSTM_A_New_Framework_for_Emotion_Recognition_From_Multi-Channel_EEG
[9]	WANG Y X, QIU S, LI J P, et al. EEG-based emotion recognition with similarity learning network[C]//Proceedings of 2019 41st Annual International Conference of the IEEE Engineering in Medicine & Biology Society. Berlin: IEEE, 2019: 1209-1212.
[10]	YANG Y L, WU Q F, QIU M, et al. Emotion recognition from multi-channel EEG through parallel convolutional recurrent neural network[C]//Proceedings of 2018 International Joint Conference on Neural Networks. Rio de Janeiro, Brazil: IEEE, 2018: 1-7.
[11]	SHEN F Y, DAI G J, LIN G, et al. EEG-based emotion recognition using 4D convolutional recurrent neural network[J]. Cognitive Neurodynamics, 2020, 14(6): 815-828. doi: 10.1007/s11571-020-09634-1
[12]	SHEYKHIVAND S, MOUSAVI Z, REZAⅡ T Y, et al. Recognizing emotions evoked by music using CNN-LSTM Networks on EEG Signals[J]. IEEE Access, 2020, 8: 139332-139345. doi: 10.1109/ACCESS.2020.3011882
[13]	YIN Z, ZHAO M Y, WANG Y X, et al. Recognition of emotions using multimodal physiological signals and an ensemble deep learning model[J]. Computer Methods and Programs in Biomedicine, 2017, 140: 93-110. doi: 10.1016/j.cmpb.2016.12.005
[14]	ZHENG W L, LU B L. Investigating critical frequency bands and channels for EEG-based emotion recognition with deep neural networks[J]. IEEE Transactions on Autonomous Mental Development, 2015, 7(3): 162-175. doi: 10.1109/TAMD.2015.2431497
[15]	HOCHREITER S, SCHMIDHUBER J. Long short-term memory[J]. Neural Computation, 1997, 9(8): 1735-1780. doi: 10.1162/neco.1997.9.8.1735
[16]	GRAVES A, SCHMIDHUBER J. Framewise phoneme cla-ssification with bidirectional LSTM and other neural network architectures[J]. Neural networks, 2005, 18(5/6): 602-610. http://www.sciencedirect.com/science/article/pii/S0893608005001206
[17]	MATURANA D, SCHERER S. Voxnet: a 3d convolutional neural network for real-time object recognition[C]//Proceedings of 2015 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS). Hamburg, Germany: IEEE, 2015: 922-928.
[18]	YANG Y L, WU Q F, FU Y, et al. Continuous convolutional neural network with 3d input for EEG-based emotion recognition[C]//International Conference on Neural Information Processing. Cham, Switzerland: Springer, 2018: 433-443.
[19]	CHUNG S Y, YOON H J. Affective classification using Bayesian classifier and supervised learning[C]//Proceedings of 2012 12th International Conference on Control, Automation and Systems. JeJu Island, South Korea: IEEE, 2012: 1768-1771.
[20]	CHAO H, DONG L, LIU Y L, et al. Emotion recognition from multiband EEG signals using CapsNet[J]. Sensors, 2019, 19(9): 2212/1-16. http://www.ncbi.nlm.nih.gov/pubmed/31086110
[21]	ZHUANG N, ZENG Y, TONG L, et al. Emotion recognition from EEG signals using multidimensional information in EMD Domain[J]. BioMed Research International, 2017, 2017: 8317357/1-9. http://www.ncbi.nlm.nih.gov/pubmed/28900626
[22]	YANG H, HAN J, MIN K, et al. A multi-column CNN model for emotion recognition from EEG signals[J]. Sensors, 2019, 19(21): 4736/1-12. http://www.ncbi.nlm.nih.gov/pubmed/31683608
[23]	ZUBAIR M, YOON C. EEG based classification of human emotions using discrete wavelet transform[M]//IT Convergence and Security 2017. Singapore: Springer, 2018: 21-28.
[24]	JADHAV N, MANTHALKAR R, JOSHI Y. Electroencephalography based emotion recognition using gray-level co-occurrence matrix features[C]//RAMAN B, KUMAR S, ROY P, et al, ed. Proceedings of the International Conference on Computer Vision and Image Processing. Singapore: Springer, 2016: 335-343.
[25]	HATAMIKIA S, NASRABADI A M. Recognition of emotional states induced by music videos based on nonlinear feature extraction and SOM classification[C]//Procee-dings of 2014 21th Iranian Conference on Biomedical Engineering (ICBME). Tehran, Iran: IEEE, 2014: 333-337.
[26]	ZHANG X Y, WANG W R, SHEN C Y, et al. Extraction of EEG components based on time - frequency blind source sepa-ration[C]//Proceedings of the International Conference on Intelligent Information Hiding and Multimedia Signal Processing. Cham, Switzerland: Springer, 2017: 3-10.
[27]	MARTÍNEZ-RODRIGO A, GARCÍA-MARTÍNEZ B, ALCARAZ R, et al. Study of electroencephalographic signal regularity for automatic emotion recognition[C]//Proceedings of the International Conference on Ubiquitous Computing and Ambient Intelligence. Cham, Switzerland: Springer, 2017: 766-777.
[28]	LI J P, ZHANG Z X, HE H G. Hierarchical convolutional neural networks for EEG-based emotion recognition[J]. Cognitive Computation, 2018, 10(2): 368-380. doi: 10.1007/s12559-017-9533-x
[29]	魏琛, 陈兰岚, 张傲. 基于集成卷积神经网络的脑电情感识别[J]. 华东理工大学学报(自然科学版), 2019, 45(4): 614-622. https://www.cnki.com.cn/Article/CJFDTOTAL-HLDX201904014.htm WEI C, CHEN L L, ZHANG A. Emotion recognition of EEG based on ensemble convolutional neural networks[J]. Journal of East China University of Science and Technology, 2019, 45(4): 614-622. https://www.cnki.com.cn/Article/CJFDTOTAL-HLDX201904014.htm
[30]	田莉莉, 邹俊忠, 张见, 等. 基于改进的卷积神经网络脑电信号情感识别[J]. 计算机工程与应用, 2019, 55(22): 99-105. https://www.cnki.com.cn/Article/CJFDTOTAL-JSGG201922015.htm TIAN L L, ZOU J Z, ZHANG J, et al. Emotion recognition of EEG signal based on improved convolutional neural network[J]. Computer Engineering and Applications, 2019, 55(22): 99-105. https://www.cnki.com.cn/Article/CJFDTOTAL-JSGG201922015.htm

Cited By

Get Citation

PDF

XML

Article views (1514) PDF downloads (150)

Turn off MathJax

Article Contents

Abstract

References

EEG Emotion Recognition Based on Hybrid Neural Networks

Abstract

References

Catalog

Export File

Citation

Format

Content