Using Facial Features to Contextualize Linguistic Analysis in Multimodal Communcation

LIWC Research Series:

This paper demonstrates that analyzing language patterns in light of their associated facial expressions elicits significant differences between deceptive and truthful commu- nication. Facial Action Units (AU) were analyzed in video recordings (1.2M frames) of 151 dyadic conversations following an interrogation protocol, in which one of the participants is known to be either lying or telling the truth. Linguistic features were extracted from the transcripts using Linguistic Inquiry and Word Count (LIWC) dictionary. Our framework extracted facial-feature contexts automatically corresponding to high and low intensities of AU occurrences. This helped us dive deeper into answers corresponding to the video segments where the witnesses kept their eyes wide open (high intensity of AU05– upper lid raise). We found that in these segments, deceivers used significantly fewer ‘Seeing’, ‘Perceptual’ and ‘Cognitive’ words and their answers were significantly shorter than truth-tellers.

Read the paper


M. K. Hasan, T. Sen, Y. Yang, R. A. Baten, K. G. Haut and M. E. Hoque, "LIWC into the Eyes: Using Facial Features to Contextualize Linguistic Analysis in Multimodal Communication," 2019 8th International Conference on Affective Computing and Intelligent Interaction (ACII), 2019, pp. 1-7, doi: 10.1109/ACII.2019.8925467.