Development of an automated system for testing a cloud service for deploying virtual machines using modern monitoring tools

Marchenko Andrei ORCID: 0009-0001-9276-3907 Cloud Software Architect, Intel Corporation 193318, Russia, Saint Petersburg region, Saint Petersburg, Badaeva str., 6k1 mar4enko.ag@gmail.com

Shchemelinin Dmitry ORCID: 0000-0003-3032-130X Doctor of Technical Science Vice President, Intel Corporation 195251, Russia, Saint Petersburg, Politechnicheskaya str., 29 dshchmel@gmail.com

Introduction

Cloud providers provide various types of services, such as hosting virtual machines, data storage, databases, analytics and much more. As a rule, access to these services is carried out both through the web interface and through the API (English Application Programming Interface) – a set of ready-made interfaces for software integration with services through various protocols. Monitoring the availability and performance of cloud systems and services is one of the most important aspects of ensuring their stable operation, and also allows you to identify and fix problems that arise during operation. Specialized software is used to monitor the availability of cloud services. One of the popular open source solutions is the Prometheus service ^[1]. This monitoring system allows you to collect and store data on performance and availability in the form of time series – a sequence of values that change over time. The collected data can then be analyzed using various methods, such as visualization, statistical analysis and machine learning.
In this study, a new method was proposed for testing and monitoring a cloud service that provides the user with the opportunity to rent virtual machines in the cloud, as well as choose one of the proposed types of virtual machine with a given amount of RAM, processors and storage. This type of service is very popular both for enterprises and for individual users and allows you to save significant resources by providing computing power on request.

The object of scientific research

The object of the study is Intel's internal cloud platform ^[2], which allows developers and researchers to use the resources of computing systems to test and optimize their applications and algorithms. The platform also provides extensive infrastructure management capabilities for computing, as well as efficient resource management and is used to perform various tasks such as big data processing and machine learning. Intel is one of the world's leading manufacturers of electronic devices and computer components: microprocessors and system logic sets (chipsets) for client computing systems and data centers, chips for artificial intelligence systems and the Internet of Things, non-volatile memory.
One of the key services of the cloud platform is VMaaS (English Virtual Machine as a Service), which provides a virtual infrastructure. Having studied the statistics of virtual machine reservations in the internal cloud (Figure 1), we can conclude that the demand for virtual infrastructure is constantly growing, and consequently, the requirements for stability and reliability of the service are increasing.

Figure 1. The number of virtual machine reservations in the VMaaS system from 06.01.2023 to 01.04.2023

The purpose of the study

The subject of this study is the development of a method and system for testing the availability of a virtual machine deployment service in the cloud based on Prometheus and Selenium ^[3], characterized by the following scientific novelty:

• Implementation of the Virtual Machine deployment testing service in the cloud
• Integration of Prometheus and Selenium tools with the deployment service and definition of scenarios for monitoring and testing cloud resources
• A new method for collecting and analyzing availability data using Prometheus and Selenium, including identifying performance issues and optimizing the operation of the virtual machine deployment service in the cloud

Research Methods

The main functional requirement for the testing subsystem is the ability to create and delete a virtual machine with the specified characteristics – memory, processor, disk space.

The user interface was selected to interact with the system. This allowed not only to ensure maximum compliance of the test scenario with the behavior of the end user in the system, but also to organize the collection of metrics for analyzing the quality of services that provide functionality, the purpose of which is to identify potential problems before users start reporting them.

The following functional requirements for the developed subsystem were formulated:

• Implementation of the cloud system login
• Interface navigation, localization and selection of a virtual machine deployment service
• Creating and deleting a VM
• Checking the availability status of a virtual machine
• Calculation of key metrics of the service:
  • Tracking login time
  • Tracking the creation time of a virtual machine
  • Tracking the running time of the entire script

When choosing key metrics, the need for further forecasting of service availability using mathematical models and event forecasting algorithms to achieve business targets was taken into account ^{[4, 5, 6]}.
Non-functional requirements were also considered. The task of creating a virtual machine using the user interface is a process with a large number of steps, each of which must be considered from the point of view of potential errors and their appropriate handling, including ensuring reliable and stable behavior of the system in case the task unexpectedly fails with an error. Consequently, the following non-functional requirements were formulated:

• Start the script with the specified frequency and according to the schedule
• Automatic repetition of an unsuccessful operation during script execution. Provide for the configuration of the waiting interval between repetitions, and the number of repetitions
• Global time limit for the entire scenario. If the script does not fit within the allotted limit, it must be completed forcibly

To implement the test scenario, taking into account the formulated functional and non-functional requirements, the Python programming language and the selenium-python library ^[7] based on Selenium WebDriver were chosen. The Python language is very common ^[8] and allows you to implement applications of various levels of complexity in a short time. The simple and clear syntax, the absence of additional performance requirements, as well as the ease of debugging and testing makes it an ideal candidate for this study. Selenium WebDriver ^[9] together with the selenium-python library is widely used for testing web applications and allows you to emulate user interaction with the browser.

Figure 2. System architecture

Based on the formulated functional and non-functional requirements, the corresponding architecture was developed (Fig. 2). Kubernetes, an open source software for automating the deployment, scaling and management of containerized applications, as well as a special type of Kubernetes resource – a periodic task (English Cronjob), was chosen as the platform for implementation.  This type of resource allows you to schedule and launch tasks at a certain time, control the schedule and frequency, manage re-launches in case of errors, and also set a global limit on task completion. The Selenium Grid subsystem was used to interact with the user interface of the virtual machine deployment cloud service ^[10]. This component is a platform for remote script execution using WebDriver. It provides parallel launch of test scripts by executing them on different instances of remote browsers. VMaaS is a cloud service that provides a platform for deploying virtual machines. Prometheus was chosen as the main system for collecting and storing metrics, which is also responsible for processing metrics and generating notifications. Alertmanager – processes notifications generated by Prometheus and forwards them to VictorOps. This component is responsible for managing and handling incidents in real time and allows you to notify the appropriate teams in case of problems. Prometheus Pushgateway is an intermediate layer between short-lived tasks and the main Prometheus server. This component is necessary, since a feature of Prometheus' work is periodic polling of targets to collect metrics at a short interval - pull model, while tasks are run on their own schedule and for effective collection of metrics it is necessary to implement a push model. The task in progress sends an HTTP (English Hypertext transfer protocol) request to Prometheus Pushgateway, which temporarily stores metrics and is the target object for Prometheus. Grafana service was used together with Prometheus to create and configure graphs, charts and other visualization panels of metrics ^[11].
To assess the availability of the virtual machine deployment service, key metrics were selected. The user's login time was chosen for the main metric. This is the time during which the user is authenticated and authorized using the registered credentials. The measurement was carried out from the moment the credential entry form appeared, until the moment when the user was shown the main screen with the ability to start a virtual machine. To calculate time metrics, it was suggested to use the following time_monitor function.

The use of this function for the login process is as follows:

The next key metric was the launch time of the virtual machine (English launch instance). It determines the allowable limit during which we wait for the virtual machine to be ready.

If this limit is exceeded, the script will fail with an error.

The last key metric was the total running time of the virtual machine creation script, which includes processing expected exceptional situations, repeating operations, waiting for the machine to be ready and then deleting it. The removal of the machine is an integral part of the completion of the test scenario, since all resources allocated during operation must be released.
The virtual machine startup time metric was analyzed and a threshold value was determined, which was used in Prometheus to create notifications. 
To choose the optimal method for determining the threshold value, it is necessary to check whether the distribution is normal. To do this, it was proposed to apply the Shapiro-Wilk test ^[12]. The verification is based on the verification of the null hypothesis – the data is distributed normally. An alternative hypothesis is that the data does not have a normal distribution. The result of the Shapiro-Wilk criterion is the value of statistics, as well as the value of p-value. In most cases, 0.05 is taken as the threshold value. If the p-value is less than 0.05, the null hypothesis is rejected. The python SciPy library was used for implementation. An array of metrics of the VM startup time from Prometheus for the last 7 days was used as input data.

As a result, a low p-value of 2.098965063816978e-31 was obtained, on the basis of which the null hypothesis was rejected and it was concluded that the data do not have a normal distribution. To calculate the threshold value, the following method was used – the sum of the median and three absolute deviations ^[13]

Threshold = Median + 3 * MAD

Using the Python Numpy library, the median of the data was calculated and the average absolute deviation was calculated.

As a result, a median of 131.5, an average absolute deviation of 12.5 and a threshold value of 169 were calculated.

Results and discussion. A system has been developed that allows you to collect various key parameters of the virtual machine deployment cloud service, based on a real scenario simulating the user's work.

Figure 3. Virtual machine startup time

After analyzing the collected data (Fig. 3) by summing the median and three absolute deviations, a threshold value of 169 was calculated, which can be used in Prometheus in an expression describing a potential problem in the virtual machine deployment service. Exceeding this threshold signals a significant deviation from the expected values, and requires immediate escalation and technical investigation of the causes by the profile support team of the service ^[14].

Conclusions

The main result of the work is the developed system for testing the availability of virtual machines and the method of integration into the monitoring system of Intel's internal cloud service. The main test case was proposed and implemented - the creation and removal of a virtual machine using a graphical user interface, the criteria for triggering triggers were determined. During the operation of cloud environments with the help of this system, problem areas were identified in the architecture of the virtual machine creation service, which made it possible to optimize the system operation in a timely manner. By analyzing the collected data, problems with the performance of the web interface were identified and eliminated, which significantly improved the user experience in using this interface. The developed system allows you to register deviations in the behavior of the service by a number of key parameters, such as the login time and the startup time of the virtual machine. Thanks to this, it was possible to significantly reduce the response time to emerging incidents. The use of free and open source technologies, such as Prometheus, Selenium, contributes to increasing the efficiency and reducing the company's costs for maintaining cloud services. The described method is an effective way to test cloud services, and can also be used to analyze and improve reliability and availability – the most important criteria for business applications running 24/7