An exploratory study of multimodal interaction modeling based on neural computation

This work was supported by National Natural Science Foundation of China (Grant Nos. 61232013, 61422212, 61303162) and National High Technology Research and Development Program of China (Grant Nos. 2015AA020506, 2015AA016305).

References

[1] Jaimes A, Sebe N. Multimodal human-computer interaction: a survey. Comput Vis Image Und, 2007, 108: 116-134 CrossRef Google Scholar

[2] Li M Z, Dai G Z, Dong S H. Software model and interaction algorithm of multimodal interface (in Chinese). Chinese J Comput, 1998, 21: 111-118 Google Scholar

[3] Salvendy G. Handbook of Human Factors and Ergonomics. Hoboken: John Wiley & Sons, 2012. Google Scholar

[4] Ware C. Information Visualization: Perception for Design. San Francisco: Morgan Kaufmann Publishers Incorporated, 2013. Google Scholar

[5] Shneiderman B, Plaisant C, Cohen M, et al. Designing the User Interface: Strategies for Effective Human-Computer Interaction. 5th ed. Upper Saddle River: Addison-Wesley, 2009. Google Scholar

[6] Wang H A, Ma C X. Interactive multi-scale structures for summarizing video content. Sci China Inf Sci, 2013, 56: 052108-118 Google Scholar

[7] Gallace A, Spence C. In Touch With the Future: the Sense of Touch From Cognitive Neuroscience to Virtual Reality. Oxford: Oxford University Press, 2014. Google Scholar

[8] Gallace A, Tan Z H, Spence C. The body surface as a communication system: the state of the art after 50 years. Presence, 2007, 16: 655-676 CrossRef Google Scholar

[9] Bau O, Poupyrev I, Israr A, et al. TeslaTouch: electrovibration for touch surfaces. In: Proceedings of the 23rd Annual ACM Symposium on User Interface Software and Technology. New York: ACM Press, 2010. 283--292. Google Scholar

[10] McGrath B J, Estrada A, Braithwaite M B, et al. Tactile Situation Awareness System Flight Demonstration. Army Aeromedical Resaerch Lab Fort Rucker AL Technical Report USAARL 2004-10. 2004. Google Scholar

[11] Lu L, Tian F, Dai G Z, et al. A study of the multimodal congnition and interaction based on touch, audition and vision (in Chinese). J Comput Aided Design Comput Graph, 2014, 26: 654-661 Google Scholar

[12] McGurk H, MacDonald J. Hearing lips and seeing voices. Nature, 1976, 264, 746--748. Google Scholar

[13] Mayer R E. Multimedia learning. Psychol Learn Motiv, 2002, 41: 85-139 CrossRef Google Scholar

[14] Kieras D E, Meyer D E. An overview of the EPIC architecture for cognition and performance with application to human-computer interaction. Human-Comput Interact, 1997, 12: 391-438 CrossRef Google Scholar

[15] Ohshiro T, Angelaki D E, de Angelis G C. A normalization model of multisensory integration. Nature Neurosci, 2011, 14: 775-782 CrossRef Google Scholar

[16] Carandini M. From circuits to behavior: a bridge too far? Nature Neurosci, 2012, 15: 507--509. Google Scholar

[17] Vroomen J, Keetels M. Perception of intersensory synchrony: a tutorial review. Atte Perce Psycho, 2010, 72: 871-884 CrossRef Google Scholar

[18] Calvert G A, Spence C, Stein B E. The Handbook of Multisensory Processes. Cambridge: MIT Press, 2004. Google Scholar

[19] Ernst M O, Bülthoff H H. Merging the senses into a robust percept. Trends Cogn Sci, 2004, 8: 162-169 CrossRef Google Scholar

[20] Stein B E, Stanford T R, Rowland B A. The neural basis of multisensory integration in the midbrain: its organization and maturation. Hearing Res, 2009, 258: 4-15 CrossRef Google Scholar

[21] Carandini M, Heeger D J. Normalization as a canonical neural computation. Nature Rev Neurosci, 2012, 13: 51-62 CrossRef Google Scholar

[22] Stein B E, Stanford T R. Multisensory integration: current issues from the perspective of the single neuron. Nature Rev Neurosci, 2008, 9: 255-266 Google Scholar

[23] Molholm S, Ritter W, Murray M M, et al. Multisensory auditory-visual interactions during early sensory processing in humans: a high-density electrical mapping study. Cogn Brain Res, 2002, 14: 115-128 CrossRef Google Scholar

[24] Baranek G T. Efficacy of sensory and motor interventions for children with autism. J Autism Develop Disord, 2002, 32: 397-422 CrossRef Google Scholar

[25] Brown S, Shankar R, Smith K. Borderline personality disorder and sensory processing impairment. Progress Neurol Psychiat, 2009, 13: 10-16 CrossRef Google Scholar

[26] Stein B E, Meredith M A. The Merging of the Senses. Cambridge: MIT Press, 1993. Google Scholar

[27] Meredith M A, Stein B E. Spatial determinants of multisensory integration in cat superior colliculus neurons. J Neurophysiol, 1996, 75: 1843-1857 Google Scholar

[28] Meredith M A, Nemitz J W, Stein B E. Determinants of multisensory integration in superior colliculus neurons. I. temporal factors. J Neurosci, 1987, 7: 3215-3229 Google Scholar

[29] Meredith M A, Stein B E. Interactions among converging sensory inputs in the superior colliculus. Science, 1983, 221: 389-391 CrossRef Google Scholar

[30] Spence C, Driver J. Crossmodal Space and Crossmodal Attention. Oxford: Oxford University Press, 2004. Google Scholar

[31] Gillmeister H, Eimer M. Tactile enhancement of auditory detection and perceived loudness. Brain Res, 2007, 1160: 58-68 CrossRef Google Scholar

[32] Heeger D J. Normalization of cell responses in cat striate cortex. Visual Neurosci, 1992, 9: 181-197 CrossRef Google Scholar

[33] Körding K P, Wolpert D M. Bayesian integration in sensorimotor learning. Nature, 2004, 427: 244-247 CrossRef Google Scholar

[34] Ernst M O, Banks M S. Humans integrate visual and haptic information in a statistically optimal fashion. Nature, 2002, 415: 429-433 CrossRef Google Scholar

[35] Sternberg R. Cognitive Psychology. 3rd ed. Carolina: Wadsworth Publishing, 2003. Google Scholar

0

Citations
0

Altmetric
Article metrics

Email
Export

An exploratory study of multimodal interaction modeling based on neural computation

Abstract

Funded by

Acknowledgment

References

0

0

Interesting search

Top 5Related Articles