Emotion Recognition by Audio Signals as one of the Ways to Combat Phone Fraud

Nikitin Petr Vladimirovich ORCID: 0000-0001-8866-5610 PhD in Pedagogy Associate Professor, Department of Data Analysis and Machine Learning 125993, Russia, Moscow, 4th veshnyakovsky str., 4, office building 2 pvnikitin@fa.ru Other publications by this author     Osipov Aleksei Viktorovich PhD in Physics and Mathematics Associate Professor, Department of Information Security, Federal State Educational Budgetary Institution of Higher Education "Financial University under the Government of the Russian Federation" 125167, Russia, Moscow, 4th veshnyakovsky str., 4, office building 2 avosipov@fa.ru Pleshakova Ekaterina Sergeevna PhD in Technical Science Associate Professor, Department of Information Security, Federal State Educational Budgetary Institution of Higher Education "Financial University under the Government of the Russian Federation" 125167, Russia, Moscow, 4th veshnyakovsky str., 4, office building 2 espleshakova@fa.ru Korchagin Sergei Alekseevich PhD in Physics and Mathematics Deputy Dean of the Faculty of Information Technology, Federal State Educational Budgetary Institution of Higher Education "Financial University under the Government of the Russian Federation" 125167, Russia, Moscow, 4th veshnyakovsky str., 4, office building 2 sakorchagin@fa.ru Gorokhova Rimma Ivanovna PhD in Pedagogy Associate Professor, Department of Data Analysis and Machine Learning 125167, Russia, Moscow, 4th veshnyakovsky str., 4, office building 2 rigorokhova@fa.ru Other publications by this author

Gataullin Sergei Timurovich PhD in Economics Deputy Dean of the Faculty of Information Technology, Federal State Educational Budgetary Institution of Higher Education "Financial University under the Government of the Russian Federation" 125167, Russia, Moscow, 4th veshnyakovsky str., 4, office building 2 stgataullin@fa.ru

The article was prepared as part of the state assignment of the Government of the Russian Federation to the Financial University for 2022 on the topic "Models and methods of text recognition in anti-telephone fraud systems" (VTK-GZ-PI-30-2022)

Introduction. The development of mobile communications is directly related to the development of information technology. Opportunities that provide a person with means of communication contribute to improving the quality of life. The use of mobile communication is expanding every year, becoming an integral part of the lives of almost all people. This gives rise to an increase in illegal actions using means of communication. Telephone fraud is especially widespread. In the Criminal Code of the Russian Federation, article 159 defines fraud as "theft of someone else's property or acquisition of the right to someone else's property by deception or abuse of trust." Thus, the type of fraud in question is essentially aimed at using information and telecommunication technologies to obtain personal funds of citizens or bank card data. In this case, the increasingly widespread possibilities of electronic payments become a way to receive other people's money. According to Article 159.3 of the Criminal Code of the Russian Federation, "electronic means of payment is a means and (or) a method that allows the client of a money transfer operator to make, certify and transmit orders for the purpose of transferring funds within the framework of the forms of non-cash payments used using information and communication technologies, electronic media, including payment cards, and also other technical devices". It becomes clear that it is enough for fraudsters to get confidential information from the victim, then the commission of the crime becomes irreversible. That is why the issues related to the prevention of telephone fraud are given so much attention in various studies.

Special attention in modern research is paid to the psychological portrait of people exposed to telephone fraud. The research of N. V. Meshkova, V. T. Kudryavtseva, S. N. Enikolopova ^[1] examines potential victims of telephone fraudsters from the perspective of their psychological portrait. The implementation of telephone fraud begins with a call and develops further depending on the behavior of the victim: as far as possible to manipulate her, impose their own rules of behavior. The authors of the article identified the factors of people used by scammers: "a high level of claims, dissatisfaction with their social status, an inflated standard of consumption and lack of communication, excessive credulity, not critical, superstitious behavior" ^[1].

In A. A. Pudovkin's dissertation research, the analysis of the personal characteristics of a fraudster and victims of fraud was carried out. A modern fraudster who commits telephone fraud is characterized as an "intellectual criminal" who has a high level of education, is competent in the professional field related to telecommunications technologies and psychology, has acting skills, the ability to inspire confidence in strangers, and the skills to perform risky actions. The study showed the importance of studying the properties of a person who becomes a victim of a crime and becomes involved in complex schemes of criminals. In this regard, A. A. Pudovkin determines that the most susceptible are people, on the one hand, who possess such traits as gambling and adventurism, on the other hand, people who do not have critical thinking, therefore gullible and naive. Another characteristic is the thirst for easy money and pathological greed (Pudovkin, A. A. Criminal law and criminological features of fraud: dis. ... cand. jurid. sciences': 12.00.08 St. Petersburg, 2007 144 p. RGB OD, 61:07-12/949).

The properties of the victim personality, the most vulnerable to violence, are also considered in the work of O. A. Klachkova. In the conducted research, a system of psychological properties characterizing such personalities is determined and their qualities are determined. The author first of all points out the lack of attention and the desire to be noticed, to be able to surprise others. A special place among the qualities of people susceptible to fraud is carelessness ^[2].

The analysis of the presented studies shows that telephone fraud has two opposite sides. On the one hand, there are scammers with their own characteristic features, on the other hand, there are individuals subjected to fraud. And in order to achieve results, both components must coincide, that is, there must be a person ready to believe in a specific fraudulent action and perform the action imposed on him, that is, to become a victim. I. G. Moiseeva considered the problem of telephone fraud from the perspective of psychological analysis of fraudulent actions through electronic means of payment. The author identifies the types of fraud committed on the basis of the use of a number of means of information and telecommunication technologies. New technologies and their active implementation give a huge scope for use in the commission of crimes: from mobile communications to electronic payments and plastic bank cards ^[3].

Obtaining a technology that will allow identifying a fraudster by identifying emotions in accordance with the general model of fraudulent actions will help prevent the actions of the victim.

In the study of A. A. Romanova and V. A. Mashlyakevich considered the methods of using mobile communication tools to carry out the most unthinkable crimes to obtain funds when making transactions during purchase and sale. The authors have defined algorithms for committing frauds using mobile communications ^[4]. Since the actions of fraudsters according to different schemes are quite variable and are, to some extent, of a nature depending on the specific victim and the specific situation, in this case it is possible to determine with the help of his emotions whether the event that is taking place is a stage in the commission of a crime.

Not only users, but also banks are concerned about phone fraud. This direction is constantly developing with the development of information technology and is causing more and more problems in the financial sector. The authors E. V. Barasheva, D. A. Stepanenko conducted a historical study of crimes involving the use of information and communication technologies in the banking sector. ^[5] This study showed that fraud has its own history of development and is becoming more sophisticated. A. A. Ivanova, V. V. Mishchenko also consider the relevance of problems directly related to financial activities and methods of combating fraud in the financial sphere. Researchers see a decrease in the statistics of financial crimes in the need to increase, first of all, the financial literacy of citizens, as well as in establishing the regulatory framework for combating them ^[6]. I. V. Sukhorukova pays special attention to the relevance of problems directly related to the financial activities of Sberbank. The study determined that cyber attacks are carried out on the bank constantly and highlighted the most commonly used methods of fraud and theft from bank cards ^[7].

The questions of predicting emotions are the subject of research in a variety of fields. The prediction of emotions, age, and origin based on vocal data was considered in the article by A. Anuchitanukul, L. Specia ^[8]. Burst2Vec's adversarial multitasking approach uses pre-trained speech representations to capture acoustic information from raw signals and includes the concept of eliminating model bias through adversarial learning. The authors achieved a 30% increase in the productivity of the model application in the course of the study. The possibilities of applying the investigated approach show the possibilities of multitasking learning (MTL) for recognizing various components of the caller on the phone, including emotions, in the fight against telephone fraud.

The research of A. I. Ivanov and I. A. Kubasov ^[9] is devoted to the need to prevent telephone fraud by identifying the voice characteristics of fraudsters. The study is based on the creation of a database of voices and the transition to automated procedures for detecting telephone fraudsters based on the analysis of their voice portraits. The authors propose the use of artificial intelligence and for this purpose build the following algorithm: automatic marking of a voice message into frames; automated formation of evidence images by documenting the voice, which is carried out by keywords and allows you to determine the scope of the fraudster; consideration of the tonality of sounds; the use of various speech recognition methods ^[9]. The authors propose a technological method that can serve as the basis for the use of artificial neural networks and their training to determine the main key elements of the portrait of a fraudster by his voice and timbre of sound. This study examines the basics for the use of artificial intelligence to recognize fraudsters.

The use of a voice identification system as an additional user protection is considered in the study of M. A. Maslov and V. A. Kostikova. The article highlights the advantages and disadvantages of using voice biometrics to recognize the calls of intruders. The authors propose the construction of a voice identification system based on an acoustic voice model, a linguistic language model, a semantic model and a semantic model ^[10].

Thus, research on combating telephone fraud is constantly being conducted in different directions. Emotion recognition based on the received voice signal can serve as a way to combat intruders. It becomes possible to stop criminal actions at the very beginning if modern methods are used more and more widely. Among such methods, the most relevant is the use of predictive analytics and machine learning methods.

Data and methods. Let's consider algorithms for recognizing emotions by the timbre of the voice by means of machine learning. The evidence of the effectiveness of the use of machine learning and deep learning in determining the authors were considered in studies ^[11-16].

The first thing to do is to find suitable datasets. Currently, there is not much data with voice messages describing emotions in the public domain. The authors found three large datasets that can be used in the study. But for our research, we will limit ourselves to two.

The first TESS dataset (https://www.kaggle.com/ejlok1/toronto-emotional-speech-set-tess ). It contains 2800 WAV audio tracks. Note that this dataset is voiced only by female voices and is marked up by 7 emotions: anger, disgust, fear, happiness, sadness, surprise, neutral emotion.

The second SAVEE dataset (https://www.kaggle.com/barelydedicated/savee-database ). The dataset is voiced by male voices and marked up according to the same 7 emotions. It contains 3360 WAV audio tracks.

Since there may be both men and women among the scammers (among the victims), in order to increase the efficiency of the system being developed, the authors combined the TESS and SAVEE datasets into one dataset. Thus, the final dataset will be a dataset consisting of 6160 wav audio tracks, voiced by both male and female voices and marked up by 7 emotions: anger, disgust, fear, happiness, sadness, surprise, neutral emotion. Figure 1 shows the distribution of classes in the final dataset.

Figure 1. The distribution of classes in the dataset shows that all emotions are approximately equal, there is a slight predominance of zero emotions, but everything is within acceptable limits.

The main stage for solving the subsequent classification problem is the stage of converting audio files into a numeric format. Note that this transformation should take place quickly and at the same time the information should not lose its informativeness. The authors of the study came to the conclusion that it is most optimal to convert audio files to mel-cepstral coefficients (MFCC).

The mathematical transformations of sound in MFCC are as follows (for example, the word "one"):

1. It is necessary to apply the Fourier transform to obtain the spectrum of the audio signal (Fig. 2); Figure 2.

A temporary representation of the word "one" and its spectrum after the Fourier transform.

2. Using windows (weight functions) uniformly located on the chalk axis, we project the spectrum obtained in the previous step onto the chalk scale and transfer this resulting graph to the frequency scale (Fig. 3). Figure 3. Projection of the window function onto the frequency scale, so that the windows are more accurately concentrated at low frequencies, because that's where it's most difficult for us to distinguish sounds and that's where we need the maximum amount of information from the audio signal.

3. Find the amount of signal energy that is in each window by multiplying the signal spectrum vectors by the window function (formula 1). (1)4.                

The next step is to square the results obtained, then take the logarithm and use the discrete cosine transform (formula 2).

As a result, we get the result we need (Fig. 4).Figure 4. MFCC coefficients All transformations were performed using the Python programming language.

Figure 5 shows the transformation of a data set into chalk-kepstral coefficients.

Figure 5. Conversion of audio data As a result, we have obtained a small set of values that can replace thousands of values of the importance of a speech signal or spectrograms when determining speech in full.

This significantly increases the speed of data processing, practically without sacrificing their informativeness. The received data sets can be viewed at the following link: https://drive.google.com/drive/folders/1WQflU_1ZYsO4EuCJx9SRB5lUGKDtgEOL ?usp= sharing.

In the second stage, we will use machine and deep learning methods to solve the classification problem. In our case, for the task of determining emotions.

The following machine learning methods were used to solve the classification problem: logistic regression; randomForest, gradient boosting ^[17-19]. The ROC-AUC curve was taken as a learning metric.

The randomForest model showed itself best. The results of the model's training are shown in Figure 6.Figure 6. The quality of the model's training from Figure 6 shows that the model guesses emotions 4 and 5 the worst of all. These emotions can be very important in cases of telephone fraud.

Therefore, it is necessary to improve the results of the model.

Consider neural networks for this.

The authors tested several neural network architectures: multilayer neural networks, convolutional neural networks (CNN), recurrent neural networks (LSTM) ^[20-22]. The best results were achieved when using CNN, with the architecture shown in Figure 7.

Figure 7. CNN architecture for recognizing emotions by voice timbre, the neural network achieved the best results at a learning rate of lr = 0.01 after 400 training epochs.

The accuracy reached 98% for each of the emotions (Fig.8).

Figure 8. Results of neural network training To use this model, a script was implemented that receives an audio stream from an input device, calculates MFCC for fixed fragments of this stream and predicts one of the seven emotions using the developed model.

Note that the model is lightweight enough to carry out classification in real time. In the script using the resulting model on a real audio stream from the input device, the PyAudio library was used. The results were comparable.

But, nevertheless, I would like to note that due to technical limitations and the lack of real data, the operation of the system was not tested properly (for example, in a telephone conversation). Moreover, the training took place on recordings with English-speaking speech, this can give unexpected results when working with other languages. But these remarks do not lower the quality of the study.

Conclusions. The authors have developed and implemented a neural network model for determining emotions by the timbre of the voice in real time. The results of the study showed that machine learning and deep learning technologies can be used as a way to combat telephone fraud. Mobile operators or banks have big data with fraudulent conversations, conversations of victims. From this data, you can get marked-up datasets and using the described technologies to fight fraud. It is especially interesting to consider this study in conjunction with the technologies already used to detect fraudulent conversations, for example, by stop words. Then the multimodal technology will be more effective and will help to avoid fraudulent actions to a greater extent.