Assessment of human response to robot facial expressions through visual evoked potentials

BERT facial expressions

Abstract

Humanoid robots provide a means in which to investigate human social cognition; the processes that affect interaction between individuals. Until recently there has been no investigation into the human physiological response to robot facial expressions and perception. To the author’s knowledge this research is the first study to investigate the human neurophysiological response to hybrid artificial faces from a humanoid robot, rather than image based stimulus.

The ability of the BERT2 robot hybrid face to effectively convey emotion to a human observer, the physiological response to perceived emotional information through modulation of face specific Event Related Potential (ERP) components and the affect of presenting stimuli from a humanoid robot with increased contextual information compared to the same digital image via a computer monitor was investigated. Event Related Potential results demonstrate in both experiments the ’face-specific’ N170 ERP component is detected and artificial expressions of emotion with high correlation ratios are  discriminated. Emotional expressions evoke a larger response over neutral stimuli, with negative evoking an increased amplitude and latency to positive emotions.

Emotion recognition results between ’Laboratory’ (83.9%) and ’Community’ (62.5%) groups re-enforce the requirement for large, diverse population participation in experiments to provide accurate data sets and information for the design of social robots. Pupil dilation is an often overlooked dimension of non-verbal communication that can influence perceived emotion.

The pupil dilation for the ’Neutral’ expression provides a baseline of 25% of the iris diameter, with a ±5%range for emotional expressions. There is a tremendous potential for mutually beneficial future research in this area between psychology, neuroscience and robotics. Reciprocal influences between social cognitive neuroscience and humanoid robotics promise a better understanding of social interactions that will ultimately lead to increasing the social acceptance of future robotic companions.

Aims of the Research

The humanoid robot provides an ideal test bed for mutually beneficial research collaboration into human social characteristics between disciplines such as psychology, neuroscience and social robotics. Through an understanding of human social cognition, we can facilitate natural and efficient Human Robot Interaction (HRI). This research investigated the human response to the digital facial expressions of a humanoid robot and the recognition rates of intended emotion to answer a grand research question;

“How effective is the BERT2 hybrid face in conveying emotion to a human observer?”.

The ability of the BERT2 robot  to effectively convey emotion to a human observer, the physiological response to perceived emotional information, and the affect of presenting stimuli from a humanoid robot, with respect to stimuli from a monitor, would focus this research to provide information to guide design improvements for future social robots. The objectives for meeting the proposed aims are the following:

• Design an experiment to investigate the possible modulation of Event Related Potential (ERP) components due to the perception of digital emotional expressions as stimuli from two presentation sources; a humanoid robot and computer monitor.
• Program software that will interface between existing software used by the BERT2 robot and supplied third party devices and provide the capability to generate, transmit, collect and store data for offline analysis.
• Analyse data to detect Event Related Potential (ERP) components and identify any modulation in the amplitude and/or latency of ERP components, any trends in the data and generate a graphical 3D topography of each emotion.
• Repeat emotion recognition experiment previously conducted to investigate rates of emotion recognition for the modified expressions.
• Investigate the expected pupil dilation for each digital facial expression. Pupil dilation is often over looked in the design of a emotional expressions, but can reveal the emotional state and intention behind an expression.

The outcomes of this project will form a base for further academic research in this field and demonstrate the potential for the further use of robotics as testbeds in future social experiments.

Research Questions

The use of a grand research question is often used to guide the direction of an investigation to meet the aims of the research, for this study the grand question is “How effective is the BERT2 humanoid robot in conveying emotion to a human observer?”.
With an awareness of the current methods into facial expressions and emotion recognition, this research will focus on answering a number of secondary or sub-questions;

• Can the digital expressions from the BERT2 robot evoke distinct Event Related Potential (ERP) components?
• What extent are the implied emotions recognisable from the EEG data?
• Does presenting stimuli from different sources (a hybrid robot face or a computer monitor) modulate ERP components?
• What are the emotion recognition rates for each digital expression?
• Is there a distinction between expected pupil dilation for digital expressions?

Event Related Potential Experiment

This experiment sought to examine the ability of the digital hybrid face to effectively convey emotion to a human observer by monitoring the perception of the facial expression. Through the use of EEG techniques to record the human physiological response and modulation of event related potentials, the perception of emotional expressions and the variance due to the context presentation source of stimuli from a computer monitor and the BERT2 humanoid robot can be investigated.

Participants Care was taken to ensure the participants were naive to the research questions as psychology provides an extensive literature regarding how context, manipulated by an emotionally-laden task, can bias information processing in a subsequent task on emotional faces. As participants were unaware as to the research questions and asked only to observe presented stimuli the affect of cognition or directed attention as the source of the observed modulation is unlikely. Only the ’Community’ group took part in the ERP experiment and were pleased with the implicit paradigm as some participants mentioned that the experience of the experiment was overwhelming and outside their comfort zone. Participants mentioned that they might not be able to handle anything but the simplest of paradigm tasks. This is of interest for future experimental designs that could fail to collect accurate information due to confusion on the part of the volunteer.

Emotion ERP results generated by TOPOGS software

N170 It has been demonstrated that the presented stimuli elicit a N170 physiological response that is modulated by the expressions from the digital face. A distinctive N170 component was identified in the grand-average epochs from all subjects, for all emotions, from both sources presented at the centre of the subject field of view. The facial expressions used as the stimuli have a high correlation ratio and therefore low interstimulus perceptual variance but could modulate the ERP response through the perceived face diagram, demonstrating that modulation of ERP components are not due to perceived variance. This is counter to the findings of Thierry(2007) and the opinion that N170 modulation is due to the variation between stimuli rather than any face-specific process. This research will add to the body of research supporting the face specific nature of the N170 ERP component. These results are consistent with previous studies into the human response to conveyed emotion that both pleasant and unpleasant expressions evoked a larger N170 ERP response than neutral and that this difference activity is located on the right occipital-parietal brain area .

Human perception of robot emotion is subject to increased latency and muted amplitude relative to a human face.These results are in line with other trials conducted with human images, though our results for the robot facial expressions are modulated with a lower amplitude and increased latency, congruent with a recent similar study. The latency of the N170 ’face specific’ ERP is increased, with a muted amplitude as the presented stimuli deviates from the ’ideal’ stimuli; a human face. This trend demonstrated by distorted or rotated human faces with or without contextual information was extended by Dubal who found that images of robot expressions are encoded as early as human faces but evoke a later and muted response. This study demonstrates that this trend of increased latency and reduced amplitude of N170 ERP components continues with physical humanoid robots. This work confirms the trend and includes human physiological responses to facial expressions from physical humanoid robots. Consistent with the results from studies, both pleasant or unpleasant expressions evoked a larger N170 response ERP than neutral and that this difference activity is located on the right lateral occipital-parietal area. Positive emotions were evoked earlier than negative, which follows current literature.

Comparison of N170 ERP latency (mS) between stimuli sources

Delay between sources An average trend delay of 7ms occurred in physiological responses between the two sources of computer monitor and the hybrid face. The latency of the N170 face specific ERP seems to increase along with a muted amplitude as the presented stimuli deviates from a ’human face’ stimuli. Seyama(2009) has proposed that the human visual process is inefficient when dealing with artificial stimuli and therefore the artificial stimuli presented from a humanoid robot with further ’non-human’ factors could cause an increase in cognitive requirements.

It is speculated that this might be due to the increase in configural information due to the ’framing’ attribuate of the head.
This trend demonstrated by distorted or rotated human faces with or without contextual information was extended by Dubal(2010) who found that robot expressions are encoded as early as human faces but evoke a later and muted response . This work confirms the trend and provides human physiological responses to artificial facial expressions from physical robots. A delay as the volunteer’s attention is refocused to the hybrid face is also a possible cause for the increased latency in perception of humaniod expression. While a fixation cross was presented on the monitor, subjective comments from participants noted that focusing attention was easier and more natural for the hybrid face. It was noted from participants that the human focus was ’drawn automatically’ to the correct location as the molded representation of the ’ears’ helped to set the configuration parameters. Subject responses to the interaction with the BERT2 face were positive with people quickly accepting and becoming familiar with the digital facial expressions. One participant commented that like a human disfiguration, the focus of attention moves beyond physical characteristics. While the physiological differences between the two sources may be put down to the perceptual salience of stimuli, a constant modulation of the P100 ERP component would have been expected if difference between source luminance was a significant variable. Stimuli from the robot include additional contextual information (head size, shape, colour, configuration with internal features, etc) that provide similar perception stimuli, but evoke additional neuron clusters that allow particular internal neurological decision processes to reject the artificial face from later processing leading to reduced late positive potentials.

Emotion Recognition

Facial expressions are the most efficient method in which to communicate how we feel, though accurately recognising faces can be difficult due to the variety and complexity of the human face; it has more than forty-four muscles that produce the numerous subtle expressions.
To examine the subjective cognitive element of emotion recognition from the BERT2 hybrid face, participants were asked to match emotions to digital expressions. Recognition rates from all data sets demonstrate that while those unfamiliar with technology will achieve lower overall recognition rates from robot facial expressions, exposed to the BERT2 robot for the first time members of the community can still perceive a wide range of implied emotions. The post experiment recognition rates were lower for the ’Community’ participation group than those found by Bazo, but this was predicted as the ’Laboratory’ participants were drawn from a sample with a greater mean and age range. The ’Laboratory’ group of participants drawn from research students from the Bristol Robotics Laboratory achieved the highest recognition rates. This variability in recognition rates underlines the need to take additional care when selecting the participants for and the conclusions drawn from such experiments. The variability in emotion recognition rates highlights the motivations to acquire positivistic measurements, rather than relying solely upon qualitative data.

Subjective comments from the ’Community’ set demonstrated that overall many found it difficult to recognise some of the more subtle intended emotion from the digital facial expressions. Only minor modifications were made to the configuration strings for each emotion to increase the pupil diameter, so comparison between current and previous results is permitted. The digital expressions are after all cartoon characterisation of each emotion using basic structural components to create basic emotions from few features. There was discussion and confusion as to the exact emotion implied by the ’disgust’ facial expression as Bazo had noted previously, participants from both sets commented on the ’quizzical’,’confused’ and ’suspicious’ nature of the expression.

From photos taken of ’Community’ participants recreating their subjective facial expressions (they could not verify their expression with any mirror) for the range of emotions a contrast to the BERT2 expressions can be made. The similarity and subtlety between ’Stern’ and ’Angry’ expressions from these photos demonstrate emotional content contained within the slightest variation within the human expressive range. Human ’Neutral’ expressions are varied due to the individual nature of the human face. High emotional arousal states such as Happy, Surprise and Anger are easily recognisable due to the extreme motion and movement of the facial features, results from the emotion recognition experiment concur. Negative emotions with low arousal (minimal facial movement) and low valence such as Sad, Afraid and Tired, are often too similar for accurate recognition rates. Similarly, the emotions for Stern, Angry and Tired have repeatedly been confused with each other. For these reasons even human to human emotion recognition trials rarely get near prefect results.

Both groups had accurately chose ’disgust’ correctly, with ’Laboratory’ (78.6%) and ’Community’ (55.2%) data sets out performing previous results from Bazo (18.4%). This may have been due to a process of elimination that the forced-choice has artificially inflated the result of both groups to an overall 66.9% recognition rate. By allowing an ’unknown’ catagory a more accurate reflection of emotion recognition rates could be assessed along with expressions that fail significantly to convey emotion.

The stimuli were eight colour computer rendered expressions with high correlation coefficients depicting eight emotions. With such high correlation and the small number of stimuli presented, the participant would be required to use an amount of effort to remain focused on the task of discrimination within the forced-paradigm. Subjects noted that facial expression began ’to blur into one’ and confirmed that once they had selected an emotion for an expression, they would simply re-select that emotion if they recognised the configuration. This is one reason for poor recognition rates for particular emotions. The participant might initally mistake the emotions ’stern’ with ’anger’ and visa versa, resulting in a session of poor emotion recognition. The ’Laboratory’ group achieved the highest overall recognition rate of 83.9% against a result of 68.5% from Bazo, with the ’Community’ group gaining an overall recognition rate of 62.5%. Both the additional and common group data set scored consistently with lower variability than results from Bazo. Both groups scored higher recognition rates for ’Static’ or ’Realism’ (a static image combined with small random movements), rather than the ’Animation’ or ’Both’ effects. This result is not in line with the literature that states animating the facial expression through transition stages should increase the success rate of recognition. Facial expression alone without context is not sufficient to effectively communicate the range of human emotion.

This experiment sought to provide information to achieve the most efficient base or static expression before complicating the system with additional variables. There are numerous psychology and neuroscience papers describing the effect of body language on emotion cognition, and while social robotics improves interaction methods of humanoids, there has been no work into combining these fields to provide mutually beneficial future research.

Pupil Dilation

This experiment investigated the pupil dilation of the hybrid digital face, to quantify a relationship between and provide empirical values for pupil dilation to emotional expression. Trends between expected pupil dilation and face diagrams could provide a potential relationship for design considerations in future non-verbal communication and affective computing within social robots.

The common trend of positive and negative emotions evoking larger modulations relative to a neutral facial expression follows in the expected pupil dilation for emotions. Partala (2002) suggested that the autonomic nervous system is sensitive to highly arousing emotional stimulation and showed that pupil size was significantly larger after both negative and positive, than neutral stimulation. Future design of social robots will have to provide the appearance of human-like physiological responses (voluntary and involuntary) if a level of social resonance is to be achieved.

The results from this experiment provide group-average estimations of pupil dilation to robot facial expression that will help improve emotion recognition rates in future experiments. Using these results, the variation in pupil size can be limited to
avoid an unnatural appearance and evoke ’eerie’ feelings described by the ’Uncanny Valley’. One qualitative statement from a participant mentioned that “it was surprisingly obvious when the pupil size was the right or wrong size”. They noted that they were not aware how dramatic the effect of pupil size was on the overall demeanor and interpreted emotion of the robot face. The iris diameter was constant throughout the experiment, but varying the size of the iris may impact the expected dilation of the pupils for different emotional expressions. A similar experiment should be conducted to set the diameter of digital iris and other variables rather than the initial subjective set up saved in the default configuration file. The current eye colour of the digital face, provides a definite contrast between the light blue iris and black pupil, that even with an increase pupil dilation, gives the appearance of a cold stare for some facial expressions. Initial third party observations noted that with the original pupil size, even the happy emotion expression appeared to give the robot a disturbing ’maniac-like’ stare. A darker eye colour would reduce the contrast, but as the hybrid digital face projects its image rather than rely on ambient light within the external environment to highlight the face, the BERT2 face will require techniques to better reflect a human response to given environmental conditions. The use of integrated cameras will allow the system to tailor the luminance of the digital face according to the environment.

Contributions to knowledge

R. Craig, R. Vaidyanathan, C. James, and C. Melhuish. Assessment of human response to robot facial expressions through visual evoked potentials. Proceedings IEEE-RAS International Conference on Humanoid Robots, 2010.

IEEE Xplore

By: admin

1 Comments

Saturday, 18th 18 Dec 2010

Filed under Research