logo

SCIENTIA SINICA Informationis, Volume 49 , Issue 9 : 1097-1118(2019) https://doi.org/10.1360/N112018-00337

A review of EEG features for emotion recognition

More info
  • ReceivedDec 24, 2018
  • AcceptedMar 19, 2019
  • PublishedSep 6, 2019

Abstract


Funded by

国家自然科学基金(U1736220,61725204)


References

[1] Huang X T. Introduction to Psychology. Beijing: Peoples Education Press, 1991. Google Scholar

[2] van den Broek E L. Ubiquitous emotion-aware computing. Pers Ubiquit Comput, 2013, 17: 53-67 CrossRef Google Scholar

[3] Posner J, Russell J A, Peterson B S. The circumplex model of affect: an integrative approach to affective neuroscience, cognitive development, and psychopathology.. Develop Psychopathol, 2005, 17 CrossRef PubMed Google Scholar

[4] Lang P J. The emotion probe: Studies of motivation and attention.. Am Psychologist, 1995, 50: 372-385 CrossRef Google Scholar

[5] Zhao G Z, Song J J, Ge Y, et al. Advances in emotion recognition based on physiological big data. J Comput Res Dev, 2016, 53: 80--92. Google Scholar

[6] Alarcao S M, Fonseca M J. Emotions recognition using EEG signals: A survey. IEEE Transactions on Affective Computing, 2017. Google Scholar

[7] Chanel G, Kierkels J J M, Soleymani M. Short-term emotion assessment in a recall paradigm. Int J Human-Comput Studies, 2009, 67: 607-627 CrossRef Google Scholar

[8] Hruby T, Marsalek P. Event-related potentials-the P3 wave. Acta Neurobiol Exp, 2002, 63: 55--63. Google Scholar

[9] Luck S J, Kappenman E S. The Oxford Handbook of Event-Related Potential Components. Oxford: Oxford University Press, 2011. Google Scholar

[10] Lithari C, Frantzidis C A, Papadelis C. Are females more responsive to emotional stimuli? A neurophysiological study across arousal and valence dimensions.. Brain Topogr, 2010, 23: 27-40 CrossRef PubMed Google Scholar

[11] Yazdani A, Lee J S, Ebrahimi T. Implicit emotional tagging of multimedia using EEG signals and brain computer interface. In: Proceedings of the 1st SIGMM Workshop on Social Media, Beijing, 2009. 81--88. Google Scholar

[12] Codispoti M, Ferrari V, Bradley M M. Repetition and event-related potentials: distinguishing early and late processes in affective picture perception.. J Cognitive Neuroscience, 2007, 19: 577-586 CrossRef PubMed Google Scholar

[13] Olofsson J K, Nordin S, Sequeira H. Affective picture processing: an integrative review of ERP findings.. Biol Psychology, 2008, 77: 247-265 CrossRef PubMed Google Scholar

[14] Olofsson J K, Polich J. Affective visual event-related potentials: arousal, repetition, and time-on-task.. Biol Psychology, 2007, 75: 101-108 CrossRef PubMed Google Scholar

[15] Gianotti L R R, Faber P L, Schuler M. First valence, then arousal: the temporal dynamics of brain electric activity evoked by emotional stimuli.. Brain Topogr, 2008, 20: 143-156 CrossRef PubMed Google Scholar

[16] Jiang J F, Zeng Y, Tong L, et al. Single-trial ERP detecting for emotion recognition. In: Proceedings of the 17th IEEE/ACIS International Conference on Software Engineering, Artificial Intelligence, Networking and Parallel/Distributed Computing (SNPD), Shanghai, 2016. 105--108. Google Scholar

[17] Smith N K, Cacioppo J T, Larsen J T. May I have your attention, please: Electrocortical responses to positive and negative stimuli. Neuropsychologia, 2003, 41: 171-183 CrossRef Google Scholar

[18] Kim M K, Kim M, Oh E. A review on the computational methods for emotional state estimation from the human EEG.. Comput Math Methods Med, 2013, 2013(4): 1-13 CrossRef PubMed Google Scholar

[19] Bernat E, Bunce S, Shevrin H. Event-related brain potentials differentiate positive and negative mood adjectives during both supraliminal and subliminal visual processing. Int J PsychoPhysiol, 2001, 42: 11-34 CrossRef Google Scholar

[20] Cuthbert B N, Schupp H T, Bradley M M. Brain potentials in affective picture processing: covariation with autonomic arousal and affective report. Biol Psychology, 2000, 52: 95-111 CrossRef Google Scholar

[21] Nieuwenhuis S, Aston-Jones G, Cohen J D. Decision making, the P3, and the locus coeruleus-norepinephrine system.. Psychological Bull, 2005, 131: 510-532 CrossRef PubMed Google Scholar

[22] Jenke R, Peer A, Buss M. Feature Extraction and Selection for Emotion Recognition from EEG. IEEE Trans Affective Comput, 2014, 5: 327-339 CrossRef Google Scholar

[23] Wang X W, Nie D, Lu B L. EEG-based emotion recognition using frequency domain features and support vector machines. In: Proceedings of International Conference on Neural Information Processing, Berlin, 2011. 734--743. Google Scholar

[24] Bastos-Filho T F, Ferreira A, Atencio A C, et al. Evaluation of feature extraction techniques in emotional state recognition. In: Proceedings of the 4th International Conference on Intelligent Human Computer Interaction (IHCI), Kharagpur, 2012. Google Scholar

[25] Picard R W, Vyzas E, Healey J. Toward machine emotional intelligence: analysis of affective physiological state. IEEE Trans Pattern Anal Machine Intell, 2001, 23: 1175-1191 CrossRef Google Scholar

[26] Kroupi E, Yazdani A, Ebrahimi T. EEG correlates of different emotional states elicited during watching music videos. In: Proceedings of Affective Computing and Intelligent Interaction, Berlin, 2011. 457--466. Google Scholar

[27] Fan C X, Cao L N. Communication principle. Beijing: National Defense Industrial Press, 2001. Google Scholar

[28] Hjorth B. EEG analysis based on time domain properties. Electroencephalography Clin NeuroPhysiol, 1970, 29: 306-310 CrossRef Google Scholar

[29] Petrantonakis P C, Hadjileontiadis L J. Emotion recognition from EEG using higher order crossings.. IEEE Trans Inform Technol Biomed, 2010, 14: 186-197 CrossRef PubMed Google Scholar

[30] Hausdorff J M, Lertratanakul A, Cudkowicz M E. Dynamic markers of altered gait rhythm in amyotrophic lateral sclerosis.. J Appl Physiol, 2000, 88: 2045-2053 CrossRef PubMed Google Scholar

[31] Ansari Asl K, Chanel G, Pun T. A channel selection method for EEG classification in emotion assessment based on synchronization likelihood. In: Proceedings of the 15th European Signal Processing Conference, Poznan, 2007. 1241--1245. Google Scholar

[32] Khosrowabadi R, bin Abdul Rahman A W. Classification of EEG correlates on emotion using features from Gaussian mixtures of EEG spectrogram. In: Proceeding of the 3rd International Conference on Information and Communication Technology for the Moslem World (ICT4M), Jakarta, 2010. 102--107. Google Scholar

[33] Sourina O, Liu Y S. A Fractal-based algorithm of emotion recognition from EEG using arousal-valence model. In: Proceedings of the International Conference on Bio-inspired Systems and Signal Processing (BIOSIGNALS-2011), Rome, 2011. 209--214. Google Scholar

[34] Liu Y, Sourina O. Real-time fractal-based valence level recognition from EEG. In: Proceedings of Transactions on Computational Science XVIII, Berlin, 2013. 101--120. Google Scholar

[35] Conneau A C, Essid S. Assessment of new spectral features for eeg-based emotion recognition. In: Proceedings of IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), Florence, 2014. 4698--4702. Google Scholar

[36] Nie D, Wang X W, Duan R N, et al. A survey on EEG based emotion recognition. Chinese J Biomed Eng, 2012, 31: 595--606. Google Scholar

[37] Zheng J L, Ying Q H, Yang W L. Signal and System. 2nd ed. Beijing: Higher Education Press, 2000. Google Scholar

[38] Sammler D, Grigutsch M, Fritz T. Music and emotion: electrophysiological correlates of the processing of pleasant and unpleasant music.. Psychophysiology, 2007, 44: 293-304 CrossRef PubMed Google Scholar

[39] Davidson R J. What does the prefrontal cortex &. Google Scholar

[40] Yuvaraj R, Murugappan M, Ibrahim N M. Optimal set of EEG features for emotional state classification and trajectory visualization in Parkinson's disease.. Int J PsychoPhysiol, 2014, 94: 482-495 CrossRef PubMed Google Scholar

[41] Aftanas L I, Varlamov A A, Pavlov S V. Affective picture processing: event-related synchronization within individually defined human theta band is modulated by valence dimension.. NeuroSci Lett, 2001, 303: 115-118 CrossRef Google Scholar

[42] Yuvaraj R, Murugappan M, Ibrahim N M. Emotion classification in Parkinson's disease by higher-order spectra and power spectrum features using EEG signals: a comparative study.. J Integr Neurosci, 2014, 13: 89-120 CrossRef PubMed Google Scholar

[43] Keil A, Müller M M, Gruber T. Effects of emotional arousal in the cerebral hemispheres: a study of oscillatory brain activity and event-related potentials. Clin NeuroPhysiol, 2001, 112: 2057-2068 CrossRef Google Scholar

[44] Oude Bos D. EEG-based emotion recognition-the influence of visual and auditory stimuli. Emotion, 2007, 57: 1798--1806. Google Scholar

[45] Balconi M, Lucchiari C. Consciousness and arousal effects on emotional face processing as revealed by brain oscillations. A gamma band analysis.. Int J PsychoPhysiol, 2008, 67: 41-46 CrossRef PubMed Google Scholar

[46] Wei-Long Zheng , Bao-Liang Lu . Investigating Critical Frequency Bands and Channels for EEG-Based Emotion Recognition with Deep Neural Networks. IEEE Trans Auton Mental Dev, 2015, 7: 162-175 CrossRef Google Scholar

[47] Bekkedal M Y V, Rossi Iii J, Panksepp J. Human brain EEG indices of emotions: delineating responses to affective vocalizations by measuring frontal theta event-related synchronization.. NeuroSci BioBehaval Rev, 2011, 35: 1959-1970 CrossRef PubMed Google Scholar

[48] Graimann B, Pfurtscheller G. Quantification and visualization of event-related changes in oscillatory brain activity in the time-frequency domain. Progress in brain research, 2006, 159: 79-97. Google Scholar

[49] Balconi M, Lucchiari C. EEG correlates (event-related desynchronization) of emotional face elaboration: a temporal analysis.. NeuroSci Lett, 2006, 392: 118-123 CrossRef PubMed Google Scholar

[50] Duan R N, Zhu J Y, Lu B L. Differential entropy feature for EEG-based emotion classification. In: Proceedings of the 6th International IEEE/EMBS Conference on Neural Engineering (NER), San Diego, 2013. 81--84. Google Scholar

[51] Shi L C, Jiao Y Y, Lu B L. Differential entropy feature for EEG-based vigilance estimation. In: Proceedings of the 35th Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC), Osaka, 2013. 6627--6630. Google Scholar

[52] Behnam H, Sheikhani A, Mohammadi M R, et al. Analyses of EEG background activity in Autism disorders with fast Fourier transform and short time Fourier measure. In: Proceedings of International Conference on Intelligent and Advanced Systems, Kuala Lumpur, 2007. 1240--1244. Google Scholar

[53] Kiymik M K, Güler I, Dizibüyük A. Comparison of STFT and wavelet transform methods in determining epileptic seizure activity in EEG signals for real-time application.. Comput Biol Med, 2005, 35: 603-616 CrossRef PubMed Google Scholar

[54] Yoon H J, Chung S Y. EEG-based emotion estimation using Bayesian weighted-log-posterior function and perceptron convergence algorithm.. Comput Biol Med, 2013, 43: 2230-2237 CrossRef PubMed Google Scholar

[55] Rozgić V, Vitaladevuni S N, Prasad R. Robust EEG emotion classification using segment level decision fusion. In: Proceedings of IEEE International Conference on Acoustics, Speech and Signal Processing, Vancouver, 2013. 1286--1290. Google Scholar

[56] Akin M. Comparison of wavelet transform and FFT methods in the analysis of EEG signals. J Med Syst, 2002, 26: 241-247 CrossRef Google Scholar

[57] Adeli H, Zhou Z, Dadmehr N. Analysis of EEG records in an epileptic patient using wavelet transform. J NeuroSci Methods, 2003, 123: 69-87 CrossRef Google Scholar

[58] Sun Z, Chang C C. Structural Damage Assessment Based on Wavelet Packet Transform. J Struct Eng, 2002, 128: 1354-1361 CrossRef Google Scholar

[59] Hadjidimitriou S K, Hadjileontiadis L J. Toward an EEG-based recognition of music liking using time-frequency analysis.. IEEE Trans Biomed Eng, 2012, 59: 3498-3510 CrossRef PubMed Google Scholar

[60] Davidson R J, Ekman P, Saron C D. Approach-withdrawal and cerebral asymmetry: Emotional expression and brain physiology: I.. J Personality Social Psychology, 1990, 58: 330-341 CrossRef Google Scholar

[61] Huang D, Guan C, Ang K K, et al. Asymmetric spatial pattern for EEG-based emotion detection. In: Proceedings of International Joint Conference on Neural Networks (IJCNN), Brisbane, 2012. Google Scholar

[62] Takahashi K. Remarks on emotion recognition from bio-potential signals. In: Proceedings of IEEE International Conference on Industrial Technology (IEEE ICIT'04), Hammamet, 2004. 1148--1153. Google Scholar

[63] Davidson R, Fox N. Asymmetrical Brain Activity Discriminates between Positive and Negative Affective Stimuli in Human Infants. Science, 1982, 218: 1235-1237 CrossRef ADS Google Scholar

[64] Blankertz B, Tomioka R, Lemm S. Optimizing Spatial filters for Robust EEG Single-Trial Analysis. IEEE Signal Process Mag, 2008, 25: 41-56 CrossRef ADS Google Scholar

[65] Koelstra S, Yazdani A, Soleymani M, et al. Single trial classification of EEG and peripheral physiological signals for recognition of emotions induced by music videos. In: Proceedings of International Conference on Brain Informatics, Berlin, 2010. 89--100. Google Scholar

[66] Winkler I, Jäger M, Mihajlovic V, et al. Frontal EEG asymmetry based classification of emotional valence using common spatial patterns. World Acad Sci Eng Technol, 2010, 45: 373--378. Google Scholar

[67] Novi Q, Guan C, Dat T H, et al. Sub-band common spatial pattern (SBCSP) for brain-computer interface. In: Proceedings of the 3rd International IEEE/EMBS Conference on Neural Engineering, Kohala Coast, 2007. 204--207. Google Scholar

[68] Ang K K, Chin Z Y, Zhang H, et al. Filter bank common spatial pattern (FBCSP) in brain-computer interface. In: Proceedings of IEEE International Joint Conference on Neural Networks (IEEE World Congress on Computational Intelligence), Hong Kong, 2008. 2390--2397. Google Scholar

[69] Duan R N, Wang X W, Lu B L. EEG-based emotion recognition in listening music by using support vector machine and linear dynamic system. In: Proceedings of International Conference on Neural Information Processing, Berlin, 2012. 468--475. Google Scholar

[70] Sakata O, Shiina T, Saito Y. Multidimensional Directed Information and Its Application. Electron Comm Jpn Pt III, 2002, 85: 45-55 CrossRef Google Scholar

[71] Petrantonakis P C, Hadjileontiadis L J. A novel emotion elicitation index using frontal brain asymmetry for enhanced EEG-based emotion recognition.. IEEE Trans Inform Technol Biomed, 2011, 15: 737-746 CrossRef PubMed Google Scholar

[72] Clemmensen L, Hastie T, Witten D. Sparse Discriminant Analysis. Technometrics, 2011, 53: 406-413 CrossRef Google Scholar

[73] Katsigiannis S, Ramzan N. DREAMER: A Database for Emotion Recognition Through EEG and ECG Signals From Wireless Low-cost Off-the-Shelf Devices.. IEEE J Biomed Health Inform, 2018, 22: 98-107 CrossRef PubMed Google Scholar

[74] Song T F, Zheng W M, Song P, et al. EEG emotion recognition using dynamical graph convolutional neural networks. IEEE Trans Affect Comput, 2018. Google Scholar

[75] Delorme A, Makeig S. EEGLAB: an open source toolbox for analysis of single-trial EEG dynamics including independent component analysis.. J NeuroSci Methods, 2004, 134: 9-21 CrossRef PubMed Google Scholar

[76] Liu Y J, Yu M, Zhao G. Real-Time Movie-Induced Discrete Emotion Recognition from EEG Signals. IEEE Trans Affective Comput, 2018, 9: 550-562 CrossRef Google Scholar

[77] Swami A, Mendel C, Nikias C. Higher-order spectral analysis (HOSA) toolbox. Version, 2000, 2: 3. Google Scholar

[78] Sjöstrand K, Clemmensen L H, Larsen R, et al. SpaSM: a matlab toolbox for sparse statistical modeling. J Stat Soft, 2018, 84: 37. Google Scholar

[79] Zhou Z H. Machine Learning. Beijing: Tsinghua University Press, 2016. Google Scholar

[80] Gross J J, Levenson R W. Emotion elicitation using films. Cognition Emotion, 1995, 9: 87-108 CrossRef Google Scholar

[81] Fredrickson B L. Positive emotions and upward spirals in organizations. Positive Organ Scholarship, 2003, 3: 163--175. Google Scholar

[82] Yu C, Sun K, Zhong M Y, et al. One-dimensional handwriting: inputting letters and words on smart glasses. In: Proceedings of CHI Conference on Human Factors in Computing Systems (CHI'16), San Jose, 2016. 71--82. Google Scholar

  • Figure 1

    (Color online) Valence-arousal dimensional emotion model

  • Figure 2

    (Color online) ERP pattern stimulated by event at 0 ms

  • Figure 3

    (Color online) FFT calculation when $N=8$

  • Figure 4

    (Color online) 30 sampling electrodes of 32-channel NeuroScan Quik-cap

  • Table 1   Feature dimensions on SEED, DREAMER and CAS-THU
    Domain Feature SEED DREAMER CAS-THU
    Time Mean 62 14 14
    Standard deviation 62 14 14
    1-order difference 62 14 14
    Normalized 1-order difference 62 14 14
    2-order difference 62 14 14
    Normalized 2-order difference 62 14 14
    Hjorth-activity 62 14 14
    Hjorth-mobility 62 14 14
    Hjorth-complexity 62 14 14
    Energy 62 14 14
    Power 62 14 14
    HOC 310 70 70
    NSI 62 14 14
    FD 62 14 14
    Time-frequency PSD 310 42 70
    HOS 248 56 56
    DE 310 42 70
    Space DASM 135 21 35
    RASM 135 21 35
    Index 27 7 7
    DCAU 115 6 10
    MDI 27 7 7
    CSP 9
    Total 2423 463 542
  • Table 2   Features whose importance values are of top 10 on 2 or 3 datasets of SEED, DREAMER and CAS-THU when $x=$10, 30 or 50
    $x$ Domain 2 datasets 3 datasets
    10 Time Normalized 1-order difference 1-order difference
    Normalized 2-order difference 2-order difference
    Hjorth-activity Hjorth-complexity
    NSI Hjorth-mobility
    FD
    Space DASM
    RASM
    30 Time Normalized 1-order difference 1-order difference
    Normalized 2-order difference 2-order difference
    Hjorth-mobility Hjorth-complexity
    HOC NSI
    FD
    Space DASM
    50 Time Normalized 1-order difference
    Normalized 2-order difference 1-order difference
    Hjorth-complexity 2-order difference
    Hjorth-mobility NSI
    FD
    Time-frequency DE