Sentimenti. Good research provides good tools to analyse emotions

Sentimenti = emotions. We have already written about how to analyse emotions correctly when we have a way to automatically measure or classify them. Today we will tell you how we collected data to create Sentimenti tools. So, once again, we’re writing about how to do something right – to research the emotional meaning of texts.

The text was written on the occasion of a speech at the GHOST Day machine learning conference. Here you can review the presentation shown there, on polish.

Sentimenti. What emotions?

In order to train machine learning algorithms and automatically indicate the emotions expressed in the text, we first had to ask people how they feel. Such a simple question had to be broken down into several components.

First, which emotions? How do we know how many are there, how they differ, what number of categories will be optimal? This was handled by emotion specialists, people from the LOBI team. In psychology there are several models of emotions, from very simple to complex and multidimensional. We decided on two, which we currently call simply models of sentiment and emotion.

According to the sentiment model proposed in the 1977 article by Russell and Mehrabian, every emotion can be described on two axes: positive-negative and high-low agitation. As far as the model of emotions is concerned, as we have written many times, this Plutchika won. Apart from scientific reasons, he was supported by the practical ones – part of the Polish Slavic network has already been described emotionally according to him. Thanks to the use of the same model (in the corrected translation, which we also had to work on) we could compare the results of our research with the expert annotations available in this resource. This is one of our tests of the accuracy of results.

What words?

We know how to annotate, but what? Our first test was to focus on the emotional meaning of words. We decided to compare our results with the words from the WordNet, and additionally with the NAWL (formerly created by our colleagues from the LOBI). Therefore, some part of the words must coincide with these resources.

We have strived to create a list of 30,000 words or meanings – after all, we have many ambiguous words, whose emotional overtones also change depending on the context. For example, depression or lowering is something completely different from mood disorders. We have decided that we can list a maximum of three meanings of one word, and to show the participants which of these meanings we will show each one in a short phrase: depressed terrain, depression treatment.

Thanks to the WordNet annotation project (our people are also working on it), we knew that about 27% of Polish words have some emotional meaning. From our point of view, they are more interesting than neutral, so they took precedence. Besides, we controlled our list of words in terms of word frequency (to have more of these frequent, but also an appropriate percentage of rare).

Sentimenti project. What about the participants?

We wanted to be able to say something about the overtones of texts written in Polish. For this to happen, we had to find out how an average, typical user of the language understands them. According to the rules of scientific art, we had to examine a representative group of Poles – one whose structure corresponds to the structure of the population in terms of age, education and other important characteristics.

Such a survey cannot be done simply through the Internet, in social media, catching people on the street. We needed professionals, so we used the services of a nationwide research panel. Our inquiry was won by Ariadna. This company found the participants for us, but we also controlled the course of the survey. We had to provide an appropriate number of grades for each word (minimum 50 people), give survey participants identifiers (to be able to compare their answers with data on age, place of residence and others). 20 thousand people participated in the study.

How to ask about the overtones of words?

Sentiment and emotional arousal are usually described on a scale. On the other hand, models of emotions are rather categorical – they ask if there is emotion or not. Unless we are talking about Plutchik’s model, which immediately assumes that the intensity of emotions can change: from annoyance through anger to rage, for example. In our model we still wanted to add a zero point, no emotion from the anger spectrum.

After many analyses and pilots we created our tool for annotating emotions, thanks to which every meaning of a word written in a phrase could be assessed on scales which are operationalisations of sentiment models and basic emotions. The participants of the study pointed out the overtones of the word through the interface shown below. Previously, they had also received an accurate, simple written instruction, which they could return to at any time during the study.

skale sentymentu i emocji w badaniu Sentimenti; autorskie narzędzie

A significant stage in the construction of the study was to determine how many words each participant should see. We couldn’t take too much time for them, for example, because people read the text less carefully and give careless answers. The ideal arrangement turned out to be 150 words (phrases) in three rounds. The breaks between the rounds could last even several days – for us, the time spent on each phrase counted (too short and too long had to be rejected from the results).

Further than words

Our next study was supposed to be about the overtones of texts. Linguists do not know from today on that from the meaning of a word to the meaning of the whole text leads a winding road. The grammar and layout of the text also express emotions.

The project assumed that we would examine opinions – for example about hotels and doctors. It was not difficult to collect opinions, additionally they could be divided into positive and negative based on the accompanying stars or point scores. The challenge was to select domain-neutral texts. We enriched our corpus with shorter forms, sentences and phrases, coming from Paralingua corpus or our pilot studies, among others. Slightly later, our colleagues working in the WordNet team also created a corpus of opinions (annotated emotionally not by the participants of the research, but by linguists).

To make the results comparable to those of the words, we examined the overtones of the texts with the participation of a representative group of people on the same scales as the words. This study involved 2,000 people and 7,000 texts and phrases. Each participant read 50 texts, and each text was rated by at least 25 people.

First people, then AI

Currently, our Sentimenti tools for classifying texts by emotion and sentiment achieve high accuracy for every emotion. The most modern, fancy neural networks or other algorithms cannot do this without good data. We were able to teach our artificial intelligence of emotions only because we collected a well constructed database of emotional words and texts evaluated by a representative group of Polish language users.

We have already written about how we constructed and taught algorithms for automatic analysis of emotions, both on the blog and in a scientific publication. A part of our word database (about 20%) will be published as a corpus accompanying the publication describing in detail the data collection and analysis. This means that this resource will be available for scientists from all over the world who want to study emotions in Polish. We want the database to be as interactive as NAWL’s list of affective words, with its own dedicated page.