Analysis of modern SOTA-architectures of artificial neural networks for solving problems of image classification and object detection

Korchagin Valeriy Dmitrievich ORCID: 0009-0003-1773-0085 Postgraduate student, Department of Advanced Engineering School of Chemical Engineering and Mechanical Engineering, Federal State Budgetary Educational Institution of Higher Education "Russian Chemical-Technological University named after D.I. Mendeleev" 127253, Russia, Moscow, Pskovskaya str., 12, room 1, sq. 159 valerak249@gmail.com

Introduction

Currently, there is an active popularization of research in the field of development and integration of AI technologies in various spheres of human activity. Artificial neural networks (hereinafter – INS) are becoming a fundamental tool that has a revolutionary impact on many aspects of human life. They have the ability to learn from vast amounts of data, recognize complex images, analyze and formulate forecasts that previously seemed unattainable and provide the opportunity to optimize the performance of everyday tasks by partially or fully automating certain operations. Every year, INS become more powerful and complex, opening up new opportunities in areas such as CV, NLP (from English Natural Language Processing, NLP - natural language processing. Terminology), offline navigation, forecasting and much more. Improving the efficiency of the neural network model is becoming a priority area of research in various ML fields, allowing similar results to be achieved with more economical consumption of computing resources.

The purpose of this article is to conduct a study of modern INS architectures used to solve problems of object classification and detection. The main focus of the research is aimed at identifying the most effective methods in the INS structure by comparative analysis of their advantages and disadvantages.

The article will review the historical development of recent architectural achievements, provide a detailed description of the key aspects underlying modern approaches in the field of CV, review current trends and challenges faced by researchers and engineers working with AI technologies, and present comparative graphs demonstrating the main performance indicators of modern INS models. The results obtained will allow us to identify the common advantages and disadvantages of modern approaches to the construction of neural networks, to determine the prospects and directions of future research.

As it is known ^[1,2,3], the internal structure of the INS is a complex mathematical model containing many variable parameters and dynamically changing computational blocks. Regardless of the depth of the model and the complexity of the approaches used, the general architectural representation of the INS can be illustrated as follows (Fig. 1).

Fig. 1 – Diagram of the neural network architecture in general form

where x and y are neurons or features of the input layer, ? is the summator of the weighting coefficients obtained from the input set of features, the graph reflects the activation function to maintain the linearity or nonlinearity of the output results

In addition, INS are classified according to various criteria, including the number of computational layers and the type of building internal connections between neurons: single-layer, multi-layer and feedback. However, regardless of the class used, the basic structural elements inherent in any INS model are preserved.

Any differentiable solution can be used as an activation function ^[4,5]. The determination of the most appropriate dependence is carried out by a data engineer based on: a deep analysis of the subject area and the choice of an activation function with an optimal distribution character, existing experience in solving a similar set of tasks, a selection method to achieve the most accurate indicators of model convergence.

After selecting the activation function, the computational processing stage follows. The input data is fed to the neural network in the form of a tensor containing a certain set of parameters ^[6]. Further, depending on the current stage of model construction, either unidirectional direct propagation of the error occurs, if the model is ready for use in real tasks, or bidirectional (forward and reverse) propagation of the error, if the model is found in the learning process.

In the process of direct propagation, the sum of the input signals is calculated taking into account the weighting coefficients for each connection, then an activation function is applied to obtain the output values.

In the process of learning the model, the final result is analyzed in accordance with the chosen teaching method - with a teacher, without a teacher, or with reinforcement. The differences between the methods are in the way to get an answer about the correctness of the received prediction.

The purpose of INS training is to minimize the loss function by updating the weight coefficients in the direction of the anti-gradient at each step of training using variable gradient descent algorithms.

The gradient value is calculated by determining the partial derivatives of the weighting coefficients.

The learning process by updating the weighting coefficients in accordance with the established step in the direction of the anti–gradient vector is called backpropagation (from the English backpropagation, backprop is the reverse propagation of the error. Terminology) and continues until the threshold value of the accuracy value is reached, or until the specified number of training epochs has passed.

The main difference between more complex models lies in the scale of the input parameter space used, which allows you to create more complex dependencies, aggregating a more extensive set of features and contributing to the processing of both structured and unstructured data of a larger volume.

In addition, modern progressive INS models use a multi–layered backbone structure (from the English backbone - skeleton, spine. Terminology), which allows you to identify both general and more local dependencies within a single dataset. For example, in the context of image recognition, a multi-layered architecture can be used to highlight and concatenate various image patterns.

Advanced INS have a number of common problems, the solution of which contributes to the dissemination of research and the integration of new approaches in the field of ML and AI technologies. These include:

– A large number of input parameters. In the context of image processing and analysis, there is a directly proportional increase in the size of the model, depending on the resolution and the number of color channels. As a result, the use of the basic INS architecture for the analysis of modern high-quality images becomes inapplicable due to a significant increase in both training time and processing of the input data set;

– Retraining. It represents an alternative problem related to the size of the model. An increase in the number of neurons in the INS contributes not only to an increase in the time duration of learning, but also to an improvement in the ability of the model to memorize signs. When training on a limited amount of data, the INS can achieve the minimum values of the loss function, which indicates an ideal prediction within the training sample. However, when testing on a real set of test data, the network may show a significant increase in the proportion of erroneous values compared to the results on the training sample. This phenomenon characterizes the unsuitability of the obtained model for the effective solution of the task, since, instead of generalizing the dependencies in the data, a specific training set is memorized, including both structured and “clean” data, and random, noisy or abnormal samples. As a result, the model may demonstrate a weak ability to predict reliable results in practice;

– Attenuation of gradients. It is a common phenomenon in the context of INS training. This problem is related to the fact that the values of the partial derivatives of the loss function according to the parameters of the model. Calculated in the backprop process, over time, can take on anti-gradient values at the error level. As a result, the INS demonstrates a sharp decrease in the speed of learning and updating weight coefficients, which also characterizes the unsuitability of the model for use in real tasks.

The solution of the presented ensemble of problems contributes to the development of more flexible and efficient architectural approaches that allow achieving similar or higher performance with a more economical consumption of computing resources. Therefore, the study of modern INS architectures is an urgent task, which allows us to obtain information about the most successful approaches used in practice.

Materials and methods of research

To analyze the latest advances in machine learning architectures, it is necessary to familiarize yourself with the comparative graph of the most effective models used in classification and detection tasks in recent years. When selecting the most effective models, only the final iterations of State-Of-The-Art (hereinafter referred to as SOTA) architectural solutions were taken into account. The analysis is based on the results obtained using the following datasets: ImageNet, CIFAR-10, CIFAR-100, MS COCO-minimal. Information about the used datasets was taken from the open research portal PapersWithCode.

To carry out the analysis, a set of five SOTA models from each dataset was formed. Figure 2 shows the summary performance indicators of the INS models, divided into groups depending on the dataset used and the metric used.

Fig. 2. Summary results of the effectiveness of the INS models

A comparative analysis of the effectiveness of the INS will be based on the following set of basic criteria:

– Classification accuracy: the ability of the model to correctly classify the data presented to it;

– Model size: the number of parameters in the INS architecture that affect its size and the ability to deploy in conditions of limited computing resources;

– Adaptability: the ability of the model to produce the correct result when introducing a new, erroneous or modified set of input data. This parameter will be evaluated based on information about the size of the training sample and the accuracy of the classification.

Also, when evaluating the effectiveness of the most successful approaches, the following set of parameters will be implicitly taken into account:

– Learning and prediction speed: It is determined by the time spent on training the model and making predictions. It is an important factor when analyzing a larger data set or solving problems in real time;

– Resistance to retraining: an indicator reflecting the ability of the INS to generalize the data of the training sample;

– Versatility: the ability of the model to work with various types of tasks, such as classification, detection, clustering, and others;

– Model complexity: This criterion is related to the number of layers and neurons in the architecture, affecting learning and performance.

Conducting an actual comparison of these parameters is not possible due to the lack of recorded information in most existing solutions.

The aggregation of the above criteria forms a comprehensive assessment of the effectiveness of the INS, which is important when choosing a model for solving a specific range of tasks.

This article provides an analysis of the most effective approaches demonstrating a significant increase in efficiency compared to previous architectures based on a study of the relevant literature ^[7-33]. Let's consider the structural components of each of the most effective solutions.

SENet^[7] is an innovative solution in which the SE block (from the English Squeeze and Excitement, SE - compression and excitation) was developed in order to establish clear links between feature channels based on global spatial information. The squeeze block is formed by applying the GAP operation (from the English Global Average Pooling, GAP – global average pooling) to the original feature map, which creates a one-dimensional vector containing the average statistical values for each channel. 

To use the information obtained at the compression stage, an Excitement block is used, the essence of which is to use the gating mechanism (from the English gating – gating, transmission. Terminology) through the sigmoids function to highlight the most significant element in the intermediate dataset. To limit the complexity of the model, the parameterization of the gating mechanism is performed using a bottleneck (from the English bottleneck – a bottleneck, a narrow neck. Terminology), consisting of two FC layers (from the English Fully Connected, FC – fully connected. Terminology), separated from each other by the activation function of ReLU, which is a layer with a dimensionality-lowering coefficient. 

Additional methods for improving accuracy are ShakeShake ^[8] and Cutout^[9] regularization. Regularization is the addition of additional information to the model in order to prevent overfitting and reduce the impact of "noise". It is achieved by adding penalties or constraints to the loss function of the underlying model.

According to the results of the study ^[7], the integration of the developed SE block into the ResNet architecture ^[10] allows to increase the accuracy of the ANN, on average, by 10%, depending on the depth of the model with a relatively small increase in computational costs. In addition, the authors argue that the selection of the reducing coefficient should be carried out manually, based on the requirements of the basic architecture, since, depending on the value of the coefficient, the number of parameters decreases, which may affect the accuracy of the model, as evidenced by the comparison results using the example of the SE-ResNet-50 model ^[7]. It is noteworthy that the computational complexity of the deepest ResNet-152 model using the SE block has a value of 11.32GFLOPs, while the result of the original model is 11.3GFLOPs.

EfficientNet ^[11] (abbreviated EffNet) is an architectural embodiment of a deep neural network based on the technology of generative construction of an adaptive architecture similar to that used in MnasNet. The key idea and difference between EffNet and MnasNet is that, according to the authors ^[11], the dimension of the convolutional layer extracting features from the image has a linear dependence on the size of the input image, which requires changing the depth, width of the convolutional layer or image resolution to extract more "subtle" patterns. Unlike traditional methods ^[11], the author's approach consists in the combined application of all traditional methods.

Additionally, it was found that changing the hyperparameters of the model has a heterogeneous effect on computational complexity. Increasing the depth of the model by adding new convolutional layers has a linear dependence of computational complexity, while increasing the filter width or image resolution has a quadratic effect. 

Based on the results of 2017 research by Google specialists ^[12], an approach was applied to the construction of recurrent neural networks to create an autonomously generated model. The principle of operation of the MnasNet architecture is to predict the most effective block of width, depth or stride for a given tensor (Compound Scaling method), build a model and then train with reinforcement by maximizing the reward value. Based on the classification results, the error gradient is projected into the initial building block of the INS and repeated prediction is made. The authors of the study ^[12] note that iterative repetition continued until a given number of learning epochs were reached, but this approach can be repeated an unlimited number of times. Using the concept of multi-purpose search, optimizing both accuracy and computational complexity, the proprietary basic architecture of EffNet-B0 was developed. 

The main element of the entire architecture is an inverted bottleneck block, called MBConv in combination with the previously mentioned SE block concept ^[7].

The scaling of the INS takes place in 2 stages:

– Stage 1. The authors fix the parameter ? = 1, it is assumed that there is a twofold excess of free resources, after which a search is performed on a small network ?, ?, ? based on expressions 7 and 9. For EffNet-B0, the best coefficient values were ? = 1.2, ? = 1.1, ? = 1.15, taking into account the following limitation ?*? ²*? ² ?2.

– Stage 2. The variables ?, ?, ? are fixed as constants, after which the network is scaled according to various parameters using expression 9. Thus, EffNet modifications from B1 to B7 were obtained

It is important to note that using the Compound Scaling method is effective only for basic models with a small number of parameters. Increasing the size of the model significantly reduces efficiency and slows down the process of finding optimal solutions ^[11].

The general idea of sequentially generating the building blocks of the INS model is called AutoML. However, the main limitation of this approach is the use of predefined convolutional operations, relationships, and block structures. Therefore, full automation of the process of searching for the optimal architecture is not possible within the AutoML approach.

ViT. Visual Transformer architecture ^[13]. Since a distinctive feature of all ViT models is the complete uniformity of the structure, with the exception of the size of the receptive field, the analysis can be produced for all existing models simultaneously.

The idea of ViT arose after the success of the Transformer architecture in NLP tasks. Transformer ^[14] is based on the encoder-decoder structure used to transform sequences. Basically, to solve NLP tasks, the transformer uses the mechanism of multi-head attention (from the English Multi–Head Attention, hereinafter referred to as MHA). This mechanism is based on the method of scaled product of scalars (multiplicative) (from the English Scaled dot-product attention), which is a weighted sum of matching a query with key-value pairs. 

NLP uses the results of word tokenization as input parameters. Instead of using a single attention function with all keys, values and queries of the model dimension, the authors ^[17] decided to apply linear projection of keys and values h times with learned projections of dimensions D _k, D _k and D _v. On each projection, an attention function is performed in parallel, the result of which is the output values of dimension D _v.MHA allows you to use information from different subspaces at different positions.

In addition to the layers of attention, the FC layer of direct propagation is applied piecemeal to each position. 

To generate a probabilistic distribution of the subsequent token based on the decoder output, a linear transformation operation is used in combination with a softmax function.

Since the use of explicit tokenization in CV tasks is impossible due to the high demands on computing resources due to a large number of parameters and the quadratic dependence of the operating time of the attention mechanism on input data, the ViT architecture uses a combined approach based on patching (from the English Patch – patch. Terminology) and PE (from the English Position Embedding, PE – embedding a position).

The patching mechanism is performed by splitting a complete set of pixels into small groups known as patches. The resulting set of tokens is fed into the standard encoder of the Transformer architecture ^[14] together with the class-token, which is used for the final classification of the image followed by a block of the direct distribution network.

Swin-V2. As previously noted, the attention mechanism has a quadratic dependence on the input set of tokens, which calls into question the applicability of ViT for segmentation or detection tasks due to the need to use patches of a smaller dimension, which will immediately have a significant negative impact on network performance.

The Swin architecture ^[15,16] represents the next stage in the development of visual transformers for the tasks described above. A set of patches of dimension 4*4 is received at the input, after which several layers of Batch Merging (hereinafter referred to as PM) and Swin Transformer Block (hereinafter referred to as ST) are sequentially applied through a linear transformation. PM performs the operation of concatenating the features of neighboring tokens in a 2*2 dimension filter, obtaining a higher-level representation of the image. Thus, after each stage, a feature map of various hierarchical representations is formed, after which the ST block is applied. A study of the structure of the ST block shows ^[15,16] that it reflects the classical encoder-decoder architecture ^[14], but differs in the use of skip-connection operations characteristic of ResNet and the modified attention function, which is a key element of the Swin architecture.

The authors ^[15,16] decided to consider attention not with all tokens existing in the set, but only with those that are in a certain area of fixed size ((Window) Multi-Head Self Attention, hereinafter – W-MSA) 

Thus, attention now functions in linear hw time. However, the use of this method is accompanied by a decrease in the representative capacity of the network due to the lack of communication between different tokens. To correct this problem, the authors added an additional layer with a diagonal offset of the receptive window after each block of W-MSA. This restored the interaction between tokens while maintaining linear computational complexity.

Due to the increase in the number of computational operations, when using this approach, the authors proposed conducting a preliminary cyclic shift in order to exclude interaction between non-adjacent tokens by applying a mask. This approach allows you to save the number of attention operations when applying the image padding operation.

In addition to introducing a new Attention calculation method, the authors introduced a new position embedding calculation by adding a trainable relative position bias matrix. 

As a result of the synthesis of RA and W-MSA, it became possible to build an architecture that allows extracting features at different spatial scales and successfully using Swin as a backbone in segmentation and detection tasks.

CoAtNet-7. The Convolutional Attention Network ^[17] is the result of a combination of advanced CNN and ViT approaches for solving CV problems. By combining spatial convolution and the self-attention mechanism, as well as sequentially overlaying layers of convolution and attention, an architecture has been developed that exhibits high performance on both large and small datasets.

CoAtNet introduces a new structure of relative attention (from the English relative attention, hereinafter – RA), which integrates the concept of relative weights or offsets into the process of encoding and decoding data. Instead of assigning fixed weights to each element, RA allows the model to take into account the relative positions of elements in the sequence when calculating weights. This is especially useful when the importance of the elements depends on their relative location rather than absolute. However, using RA to the original image requires significant computational costs.

The authors ^[17] conducted a study of two approaches to solving this problem:

– Using a 16*16 convolutional stem block, followed by L-transformer blocks with RA;

– The use of a multi-stage network with gradual pooling. 4 network configurations were selected for the study, consisting of a C-convolution and a T-transformer.

The assessment was based on two criteria:

– Generalization: the value of the minimum gap between the indicators for the training and validation sample;

– Capacity: Best performance in the same number of epochs.

The result of the author's research was the conclusion that the use of C-C-T-T construction is the most effective among the studied.

Results

For a comprehensive assessment of the effectiveness of the available solutions, it was necessary to analyze the dependence of the accuracy of the model on the number of parameters and the size of the training sample. To do this, we used data from previously described models trained on the ImageNet dataset. In the absence of data for a specific model, it was excluded from the resulting sample. A graphical representation of the normalized results of the study is illustrated in Figure 3.

Fig. 3 – Graph of the normalized ratio of parameters

Based on the analysis of the information provided, it can be argued that convolutional architectures currently have higher efficiency compared to hybrid or ViT solutions. From the point of view of the ratio of accuracy, the size of the training sample and the number of parameters, the most effective architecture is EfficientNet from the CNN family. It shows a high ability to generalize data in classification tasks with a small training sample and a number of model parameters. In addition, modern solutions based on the ViT architecture or using a hybrid approach require more data to achieve record-breaking or CNN-like results. However, the study ^[13] indicates a high learning rate of ViT solutions due to the ability of patches to cover the entire receptive field of the image simultaneously, in contrast to the sequential comparison of output feature maps in CNN architectures. According to the results of the study ^[13], the use of CNN and ViT hybridization allows, on average, to achieve greater classification accuracy with equal computational complexity, as evidenced by the illustration in Figure 4.

Figure 4 - Comparison of computational complexity of CNN, ViT and hybrid approach

Based on the analysis of the above factors, it can be concluded that currently the most effective building blocks for building an INS architecture are: AutoML + Compound Scaling (EffNet, CNN), W-MSA (Swin-V2, ViT), Squeeze and Excitement (SENet, CNN), Skip-connection (residual blocks, ResNet, CNN).

Further research is aimed at improving existing architectural solutions by developing a hybrid CNN-ViT architecture in order to reduce the dependence of the learning rate and accuracy of the model on the size of the training sample. This result can be achieved through the integrated application of AutoML, Compound Scaling, and EffNet backbones technologies, or by developing the latest approach to extracting features from the input dataset.

The obtained research results can be used to select the optimal architecture for building an INS that solves specific business problems or to develop an innovative model that takes into account all the above-mentioned performance criteria.