TOWARDS EMOTION RECOGNITION IN TEXTS –  A SOUND-SYMBOLIC EXPERIMENT

Velina  Slavova

doi:10.5937/IJCRSEE1902041S

Authors

Velina Slavova New Bulgarian University, Department of Computer Science https://orcid.org/0000-0003-3881-8820

DOI:

https://doi.org/10.5937/IJCRSEE1902041S

Keywords:

Emotional sound-symbolism, Text emotion recognition, Emotion and cognition

Abstract

The purpose of this study is to investigate the relationship between the phonetic content of prose texts in English and the emotion that the texts inspire, namely - the effect of vowel-consonant bi-phones on subjects’ evaluation of positive or negative emotional valence when reading. The methodology is based on data from an experiment where the participants, native speakers of three different languages, evaluated the valence invoked in them by one-page texts from English books. The sub-lexical level of the texts was obtained using phonetic transcriptions of the words and their further decomposition into vowel-consonant bi-phones. The statistical investigation relies on density-measures of the investigated bi-phones over each text as a whole. The result shows that there exists a correlation between the obtained sub-lexical representation and the valence perceived by the readers. Concerning the type of the consonants in the bi-phones (abrupt or sonorant), the influence of the abrupt bi-phones is stronger. However, sub-sets of both types of bi-phones showed relatedness with the emotional valence conveyed by the texts. In conclusion, the speech, expressed in written form, is laden with emotional valence even when the words’ lexicological meaning is not taken into consideration and the words are apprehended as mere phonetic constructs. This prompts hypothesizing that words’ semantics itself is partly underpinned by some mental emotion-related level of conceptualization, influenced by sounds. For practical purposes, the result suggests that based on the syllabic content of a text it should be possible to predict the valence that the text would inspire in its readers.

Downloads

Download data is not yet available.

References

Abbassi, E., Kahlaoui, K., Wilson, M. A., & Joanette, Y. (2011). Processing the emotions in words: The complementary contributions of the left and right hemispheres. Cognitive, Affective, & Behavioral Neuroscience, 11(3), 372-385. https://doi.org/10.3758/s13415-011-0034-1

Adelman, J. S., Estes, Z., & Cossu, M. (2018). Emotional sound symbolism: Languages rapidly signal valence via phonemes. Cognition, 175, 122-130. https://doi.org/10.1016/j.cognition.2018.02.007

Aryani, A., Conrad, M., & Jacobs, A. M. (2013). Extracting salient sublexical units from written texts:”Emophon,” a corpus-based approach to phonological iconicity. Frontiers in psychology, 4, 654. https://doi.org/10.3389/fpsyg.2013.00654

Aryani, A., Kraxenberger, M., Ullrich, S., Jacobs, A. M., & Conrad, M. (2016). Measuring the basic affective tone of poems via phonological saliency and iconicity. Psychology of Aesthetics, Creativity, and the Arts, 10(2), 191. https://doi.org/10.1037/aca0000033

Brandt, P. A., & Bennett, A. (2015). ‘It’s Five O’Clock’–Microprosody and Enunciation. Available at SSRN 2566228. Retrieved from: https://papers.ssrn.com/sol3/papers.cfm?abstract_id=2566228

Buechel, S., & Hahn, U. (2017, April). EMOBANK: Studying the impact of annotation perspective and representation format on dimensional emotion analysis. In Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics: Volume 2, Short Papers (pp. 578-585). https://doi.org/10.18653/v1/E17-2092

Burton, N. L. (2015). Heaven and hell: The psychology of the emotions. Kent, UK: Acheron Press.

D’Anselmo, A., Prete, G., Zdybek, P., Tommasi, L., & Brancucci, A. (2019). Guessing meaning from word sounds of unfamiliar languages: a cross-cultural sound symbolism study. Frontiers in Psychology, 10, 593. https://doi.org/10.3389/fpsyg.2019.00593

De Saussure, F. (1916/1983). Course in General Linguistics (trans. Roy Harris). London: Duckworth.

Edmiston, P., Perlman, M., & Lupyan, G. (2018). Repeated imitation makes human vocalizations more word-like. Proceedings of the Royal Society B: Biological Sciences, 285(1874), 20172709. https://doi.org/10.1098/rspb.2017.2709

Ekman, P. (1999). Basic emotions. Hand-book of cognition and emotion, 45-60. https://doi.org/10.1002/0470013494.ch3

Fantz, R. L., & Miranda, S. B. (1975). Newborn Infant Attention to Form of Contour. Child Development, 46(1), 224. https://doi.org/10.2307/1128853

Gökgöz, K. (2018). Syllables in TİD. Dilbilim Araştırmaları Dergisi, 29(1), 29-49. https://doi.org/10.18492/dad.374893

Hamann, S. (2012). Mapping discrete and dimensional emotions onto the brain: controversies and consensus. Trends in Cognitive Sciences, 16(9).http://doi.org/10.1016/j.tics.2012.07.006

Imai, M., & Kita, S. (2014). The sound symbolism bootstrapping hypothesis for language acquisition and language evolution. Philosophical transactions of the Royal Society B: Biological sciences, 369(1651), 20130298. https://doi.org/10.1098/rstb.2013.0298

Itô, J. (2018). Syllable theory in prosodic phonology. Routledge. https://doi.org/10.4324/9780429455124

Jakobson, R. (1960). Linguistics and poetics. In Style in language (pp. 350-377). MA: MIT Press.

Jones, J. M., & Vigliocco, G. (2017). Iconicity in Word Learning: What Can We Learn from Cross- Situational Learning Experiments?. Proceedings of the Annual Meeting of the Cognitive Society 2017. Retrieved from: https://pdfs.semanticscholar.org/ac61/7f1f53f8cc4df37b315edb4672df86294109.pdf

Kawahara, S., & Shinohara, K. (2012). A tripartite trans-modal relationship among sounds, shapes and emotions: A case of abrupt modulation. In Proceedings of the Annual Meeting of the Cognitive Science Society (p. Vol 34, No 34). Retrieved from https://pdfs.semanticscholar.org/3408/1ce795c89940e8fdd2a23b97a6aa834319dc.pdf

Kleinginna, P. R., & Kleinginna, A. M. (1981). A Categorized List of Emotion Definitions, with Suggestions for a Consensual Definition. Motivation and Emotion, 5(4), 981. https://doi.org/10.1007/BF00992553

Köhler, W. (1929). Gestalt Psychology. New York, NY: Liveright.

Leder, H., Tinio, P. P., & Bar, M. (2011). Emotional valence modulates the preference for curved objects. Perception, 40(6), 649-655. https://doi.org/10.1068/p6845

Macoir, J., Hudon, C., Tremblay M-P, Laforce R. Jr & Wilson M. A. (2019), The contribution of semantic memory to the recognition of basic emotions and emotional valence: Evidence from the semantic variant of primary progressive aphasia, Social Neuroscience, https://doi.org/10.1080/17470919.2019.1577295

Maddock, R. J., Garrett, A. S., & Buonocore, M. H. (2003). Posterior cingulate cortex activation by emotional words: fMRI evidence from a valence decision task. Human brain mapping, 18(1), 30-41. https://doi.org/10.1002/hbm.10075

Mandelker, A. (1983). Russian formalism and the objective analysis of sound in poetry. Slav. East Eur. J. 27, 327-338.https://doi.org/10.2307/307860

Marinkovic, K., Dhond, R. P., Dale, A. M., Glessner, M., Carr, V., & Halgren, E. (2003). Spatiotemporal dynamics of modality-specific and supramodal word processing. Neuron, 38(3), 487-497. https://doi.org/10.1016/S0896-6273(03)00197-1

Origlia, A., Cutugno, F., & Galatà, V. (2014). Continuous emotion recognition with phonetic syllables. Speech Communication, 57, 155-169. https://doi.org/10.1016/j.specom.2013.09.012

Perlman, M., Dale, R., & Lupyan, G. (2015). Iconicity can ground the creation of vocal symbols. Royal Society open science, 2(8), 150152. https://doi.org/10.1098/rsos.150152

Russell, J. A. (2003). Core affect and the psychological construction of emotion. Psychological Review, 110(1), 145. http://doi.org/10.1037/0033-295X.110.1.145

Russell, J. A., & Mehrabian, A. (1977). Evidence for a Three-Factor Theory of Emotions. Journal of Research in Personality, 11, 273-294. http://doi.org/10.1016/0092-6566(77)90037-X

Saarimäki, H., Gotsopoulos, A., Jääskeläinen, I. P., Lampinen, J., Vuilleumier, P., Hari, R., ... & Nummenmaa, L. (2015). Discrete neural signatures of basic emotions. Cerebral cortex, 26(6), 2563-2573. https://doi.org/10.1093/cercor/bhv086

Scherer, K. R. (2000). A cross-cultural investigation of emotion inferences from voice and speech: Implications for speech technology. In Sixth International Conference on Spoken Language Processing. Retrieved from: https://www.isca-speech.org/archive/archive_papers/icslp_2000/i00_2379.pdf

Scherer, K. R., Banse, R., & Wallbott, H. G. (2001). Emotion inferences from vocal expression correlate across languages and cultures. Journal of Cross-cultural psychology, 32(1), 76-92. https://doi.org/10.1177/0022022101032001009

Shklovsky, V. (1990). Theory of Prose, transl. B. Sher. (Elmwood Park, IL: Dalkey Archive).

Sidhu, D. M., & Pexman, P. M. (2018). Five mechanisms of sound symbolic association. Psychonomic bulletin & review, 25(5), 1619-1643. https://doi.org/10.3758/s13423-017-1361-1

Trotsky, L. (1957). Literature and Revolution. New York, NY: Russell and Russell.

Ullrich, S., Aryani, A., Kraxenberger, M., Jacobs, A. M., & Conrad, M. (2017). On the relation between the general affective meaning and the basic sublexical, lexical, and inter-lexical features of poetic texts-a case study using 57 poems of HM Enzensberger. Frontiers in psychology, 7, 2073. https://doi.org/10.3389/fpsyg.2016.02073

Van Orden, G. C., Johnston, J. C., & Hale, B. L. (1988). Word identification in reading proceeds from spelling to sound to meaning. Journal of Experimental Psychology: Learning, Memory, and Cognition, 14(3), 371. https://doi.org/10.1037//0278-7393.14.3.371

Winter, B. (2016). Taste and smell words form an affectively loaded and emotionally flexible part of the English lexicon. Language, Cognition and Neuroscience, 31(8), 975-988. https://doi.org/10.1080/23273798.2016.1193619

Winter, B., Perlman, M., Perry, L. K., & Lupyan, G. (2017). Which words are most iconic?. Interaction Studies, 18(3), 443-464. https://doi.org/10.1075/is.18.3.07win

Whissell, C. (1999). Phonosymbolism and the Emotional Nature of Sounds: Evidence of the Preferential Use of Particular Phonemes in Texts of Differing Emotional Tone. Perceptual and Motor Skills, 89(1), 19-48. http://doi.org/10.2466/pms.1999.89.1.19