Aspects of autonomous drive control using NVIDIA Jetson Nano microcomputer

The article describes the training process and experiments regarding autonomous movement by the autonomous car Waveshare JetRacer AI. The central unit responsible for controlling the vehicle’s systems, i.e. the steering servo and the DC motors used for the drive, is the NVIDIA Jetson Nano embedded device. The application of the IMX219 camera module for data acquisition and training of a neural network models on microcomputer and their use for the implementation of autonomous driving are described.


I. INTRODUCTION
HE autonomous movement of vehicles is a task that requires a careful analysis of the environment. For this purpose, vision or sensory systems are used. They play the role of a source of data about the environment in specific assistance systems, e.g., active cruise control, emergency braking, environment mapping, digital side and rear view mirrors, parking assist and lane assist [1]- [3]. Due to the advancement of such systems and their continuous development, it is possible that they will soon allow the achievement of higher levels of driving automation, ultimately leading to the mass production of fully autonomous vehicles.
There are six levels of vehicle driving autonomy defined by SAE International in the SAE J3016 standard, which are numbered from 0 to 5 [4]: " Level 0: No Automation " Level 1: Driver Assistance " Level 2: Partial Automation " Level 3: Conditional Automation " Level 4: High Automation " Level 5: Full Automation. At level 0 there is no automatic vehicle control, but the system may issue warnings. In the next stage, it is defined that the driver must be ready to take control of the vehicle at any time.
The automated system may be equipped with features such as Adaptive Cruise Control (ACC), Parking Assistance with automated steering, and Lane Keeping Assistance (LKA) Type II in any combination. The second level of partial automation obliges the driver to detect objects and events and to be ready for response if automated system fails to react properly. A computer executes accelerating, braking, and This research was funded partly by the 2022 subvention and partly with the SMART4ALL EU Horizon 2020 project, Grant Agreement No 872614.
steering, but can be immediately deactivated while driver takes over the control. Level 3 allows the driver to safely turn attention from driving tasks within known, limited environments (e.g. freeways). But if any warning or alert occurs, the driver must intervene. In the 4th level the vehicle is controlled by an automated system but only when the conditions in the environment allow the enabling system by the driver. When running, there is no need to pay attention to the driver while driving. The last level requires only to set the destination and press the start button. The automatic system will drive to any location where it is legal to drive [5].
Real-time processing of data received from vision modules or sensors and sending feedback control signals requires the use of computers with high computing power. Due to the dynamic development of embedded devices and increasing their efficiency, it is possible to use them for tasks that require real-time calculations [6]. Examples of such modules are microcomputers from the NVIDIA Jetson family. They are equipped with powerful GPUs that allow performing complex calculations using models of neural networks.  The purpose of processing the captured video sequences is to obtain parameters that allow precise control of the steering system of the vehicle (Fig. 1) minicomputer and a Waveshare IMX219 camera (Fig. 2). Various models of neural networks were trained and tested for the autonomous driving of a vehicle along a designated route. The software allows to stream the image from the car's onboard camera and enables the user to communicate with the environment by controlling the displayed widgets with the use of a mouse or an external controller.

II. WAVESHARE JETRACER AI HARDWARE BASED ON NVIDIA JETSON MICROCOMPUTER
Waveshare JetRacer AI is a platform that allows to independently construct a four-wheeled mobile vehicle. The kit consists of mechanical elements of the chassis, bumpers, camera arm, and wheels. The steering system is controlled by the MG996R servo with a torque of 9 kg/cm. The drive is provided by two 37-520 DC motors with a reduction rate of 1:10 and idle speed of 740 RPM. The 12.6V power supply is provided by the use of 3 Li-Ion 18650 batteries connected in series. Wireless communication, which allows remote programming and car control, is carried out using the AC8265 WiFi module with two antennas. The main PCB is equipped with a 0.91= OLED display with a resolution of 128×32 pixels. It shows the parameters for the use of computer resources and the IP address used to establish wireless communication. The manufacturer also includes a universal gamepad that can be used to control the car [7].
The central unit responsible for controlling the steering and drive systems as well as processing data from the camera is the NVIDIA Jetson Nano B01 microcomputer. Its most important technical specifications [8] are presented in Table  I. This microcomputer was designed mainly for use in tasks related to artificial intelligence. It allows running many neural networks, which can perform processes such as image classification or object detection. Their simultaneous operation while maintaining appropriate performance is ensured by the CUDA architecture, which allows for the performance of complex and computationally expensive operations, such as matrix calculations or 3D rendering using the potential of the CPU and GPU [9]. Image recording is possible by equipping the vehicle with a Waveshare IMX219 camera module 160 degree FoV [10]. Its resolution is 8 megapixels and the lens is wide-angle 160 degrees. This shows the significant extension of the perspective in comparison to 62.2 degrees in the commonly used Raspberry Pi Camera v2. The Waveshare IMX219 camera is also characterized by an aperture of f2.35 and a focal length of 3.15mm. It allows to take pictures with a maximum resolution of 3280×2464 pixels and video recording with a resolution of 1080p and frequency of 30 frames per second. The camera module itself (without dedicated PCB) is compatible with both the Raspberry Pi Camera v2 PCB and the NVIDIA Jetson Nano minicomputer used by connecting via the CSI connector.
Waveshare JetRacer AI car programming can be performed through a wireless connection to the NVIDIA Jetson Nano microcomputer. For this purpose, the JupyterLab environment is used, which is installed by default in the software package provided by the manufacturer. The system also has implemented libraries necessary to support neural networks and video sequence processing operations, such as: OpenCV, Tensorflow, NumPy, PyTorch, or NVIDIA TensorRT [11].

III. DATA COLLECTION FOR NEURAL NETWORK TRAINING
The task for the Waveshare JetRacer AI vehicle is to follow an oval track, on which the edges are marked with yellow lines and the central axis with a white dotted line [12]. Based on the image from the camera, which is sent to the car's main computer, the neural network algorithm decides to correct the direction of the vehicle's movement by changing the position of the steering servo. In the presented study, movement at variable speed is not assumed. Its value is manually controlled by the program widget, but it is possible to extend the algorithm functionality of its autonomous selection in the future.
The implementation of the Waveshare JetRacer AI car autonomous movement algorithm consists of several stages. The first one is the hardware configuration of the device, i.e. the appropriate mechanical setting of the steering system and empirical determination of its correction coefficients and the parameters of the drive system. They depend on the design of a specific car. After establishing communication with the computer via the wireless network, it is possible to adjust them through the widgets included in the program. Then, in the next step, for convenience of use, it is worth connecting the wireless controller to the computer to be able to remotely control the vehicle. For this purpose, can be used the gamepad provided by the manufacturer or any other controller such as the Xbox 360 Controller.
In the next stage, the camera widget is initialized to show a preview on one screen and on the second one to precisely define the coordinates of the direction vector, according to which the car should move autonomously. These values can be set using the widget sliders, or after appropriate code modification with a wireless controller. The user interface and the actual vehicle position are shown in Figs. 3 and 4.  Having prepared all the described tools and the vehicle on the track in the target environment, the neural network model training can be started. This process should begin with collecting training and validation data. It involves the car going over the track several times, but not continuously and recording the video sequence, but by stopping, changing the position, and taking pictures.
The key information, apart from the camera frame itself, is also the coordinates of the vector of the direction in which the car should move. They are saved in the file name along with a unique identifier for the image. Setting the vector position is a task that must be performed manually using the interface sliders and preview on the widget, or with the use of a controller.
It is required to prepare the dataset in such a way that it contains data allowing to choose the optimal path for the vehicle, but also boundary conditions, e.g. car on the edge of the track for the case to return to the correct trajectory. The larger the dataset that defines how a car behaves, the greater the chance of more precise autonomous driving.

AUTONOMOUS VEHICLE MOVEMENT
The key element of the software that allows the implementation of the task of autonomous vehicle movement is the appropriate selection of the neural network model. The concept of the structure of the implemented solution is presented in Fig. 5.
The algorithm for autonomous driving of the Waveshare JetRacer AI car was based on the ResNet18 neural network. This architecture was chosen for its effectiveness at the level of other solutions such as VGG. The advantage of ResNet solutions, however, is the small size of the model at the level of 22.7 MB [13]. The architecture of this convolutional neural network is 18 layers deep (17 convolutional and one connected) [14]. The shape of its last layer has been changed by the Linear function from the PyTorch library, and finally it has the same number of results as the number of classes in the dataset. The Linear PyTorch function creates a singlelayer unidirectional feedforward network in which data only flows from input to output. An important aspect of this type of solution is backpropagation, which consists of improving at each step of the training process due to the correction of weights based on the estimation of the error made by neurons during training. The Adam algorithm was responsible for optimizing the network model with such a structure. The set of collected data (600 frames) was divided as follows: 80% of the training images and 20% of the validation data. To optimize the performance of the software, it was decided to convert the PyTorch model to the TensorRT model, which enables more than three times the increase in the number of processed frames per second, from 29.4 to 90.2 FPS [15].  50 and 100. The mean values of the loss function and training times are summarized in Table II.
The average values obtained for the loss function show the high efficiency of the neural network model training process. As expected, the dependence of the decrease in the function value with the increase in the number of epochs can be noticed. The results for the validation data are slightly lower than for the training images, but these are average values, so there is no risk of overfitting phenomenon. The analysis of the complete input data confirmed that the structure of the network, the ratio of the number of training and validation images, as well as the selection of network parameters made it possible to achieve results that allow testing under real conditions.
Due to the use of the neural network model, when the vehicle is moving, an image is taken from the camera, which, after analysis, returns the value of the function responsible for steering the torsion of the front axle of the platform, depending on the model's prediction. In addition, parameters such as steering gain and steering bias have also been implemented, which have to be set manually because they are the result of the physical imperfections of the vehicle's design and the servo used for the steering system.
Depending on the number of epochs implemented during the neural network training process, the following observations were made during real tests. The car, using a model trained for 30 epochs, tended to cross the orange lines at the exit of the curve and cut them off. This problem was regular and repetitive, since crossings of the track limits occurred more or less at the same points on the route. This phenomenon was eliminated for the model with 50 epochs. The vehicle was already moving inside the orange lines, in accordance with the planned trajectory, but on straight sections its path deviated from the central axis marked by a white dotted line. This effect was completely eliminated for a model trained with 100 epochs. The car was driven without problems along the set trajectory, not exceeding the limits of the track and keeping its position in the axis of the route.

V. CONCLUSIONS
The tested model of the Waveshare JetRacer AI vehicle allowed for the implementation of an algorithm for the task of autonomous driving along a designated route. Due to the use of the NVIDIA Jetson Nano B01 microcomputer supporting the GPU-based CUDA architecture, it was possible to run a complex model of the neural network. It processes the data obtained from the Waveshare IMX219 160 degree FoV camera, and then sends the control signals to the servo responsible for the steering system.
Preparation of training data for a dataset that reflects the test environment, selection of appropriate parameters of the neural network, and optimization of its performance allow for obtaining a reliable algorithm for autonomous driving. It should be noted that it does not require powerful computing power from efficient graphics cards dedicated to PC devices, but can also be successfully used on embedded devices [16].
The conducted experiments confirm the correct operation of both the hardware and the software part. After an appropriate selection of the number of epochs during the neural network model training process, it is possible to eliminate undesirable behavior of the vehicle, such as going beyond the boundaries of the route or deviation from the set movement trajectory. The trajectory of the movement of Waveshare JetRacer AI model based on the algorithm analyzed is stable.
Further research will allow the system to be expanded with additional sensor modules for more precise environment monitoring (including obstacles detection) using data fusion techniques [17].