AI Camera Module for Smart Vision

RZ/V2L AI Applications is a collection of applications running on the Renesas RZ/V2L vision AI chip. It is available on Renesas' GitHub pages.

Learn more: AI Applications and AI SDK on RZ/V series

Transcript

This video is a tutorial for RZ/V2L AI Applications and AI SDK (Software Development Kit). RZ/V2L AI Applications is a collection of applications running on Renesas vision AI chip RZ/V2L. It is available on Renesas' GitHub pages.

Customers will receive the following benefits by using the AI applications. 

• You can evaluate an AI application and use it in your product for free. 
• From various AI applications that Renesas has prepared, you can select an application that matches your business purposes. 
• AI applications have been adjusted to fit the usage situation. It can be used as is without any modification.

AI applications can be quickly and easily run on the RZ/V2L evaluation board by using the AI SDK, which can be obtained free of charge from the Renesas website. The AI SDK includes the software shown in this figure. AI Applications use a USB/MIPI camera with Video for Linux 2 (V4L2). For more information about the software, please refer to the RZ/V2L AI SDK Release Note.

The goals of this tutorial are building an AI Development Environment with RZ/V2L AI SDK and running AI applications prepared on Renesas GitHub Pages. You can achieve these goals by following the get started on Renesas GitHub Pages.

From here, an overview of this tutorial is explained. 
Step 1 describes how to obtain the RZ V2L Evaluation Board kit. 
Step 2 describes the necessary equipment and software. 
In step 3, AI SDK is obtained from the Renesas website and saved on the Ubuntu PC. 
In step 4, AI SDK is extracted. 
In step 5, a Docker image is generated from the Dockerfile included in the AI SDK. The Docker container is built from the docker image. 
In step 6, the application source files obtained from GitHub are built in the container. 
In the first half of step 7, the bootloader files included in the AI SDK are copied from the Ubuntu PC to the Windows PC, and those are written to the RZ/V2L evaluation board using the Windows PC. 
In the second half of step 7, two partitions are created on the microSD card. One is used for the Linux kernel image and the Linux device tree file to boot Linux on the RZ/V2L evaluation board. The other is used for the Linux root file system. 
In step 8, the built application file and the AI model-related files are copied to the Linux root file system. 
In step 9, environment variables are set to the RZ/V2L evaluation board using a Windows PC. Linux system starts up on the RZ/V2L evaluation board using the prepared microSD card. 
Step 10 shows how to run AI Applications on the RZ/V2L evaluation board.

For a better understanding of this tutorial, please refer to the documentation as you follow the instructions in this video. This tutorial is based on version 1.00 of AI SDK. Please refer to the documentation shown above when the version changes.

This step describes how to obtain an evaluation board. 
Click on the "Get RZ/V2L EVK" button.
Click on the cart symbol. The distributors selling the RZ/V2L Evaluation Board Kit and the remaining stock will be displayed. 
Select the distributor and purchase the RZ/V2L Evaluation Board Kit.

This step describes how to obtain the necessary environment. This picture shows all the necessary equipment to run the AI applications.

MIPI Camera Module, micro USB cable, and RZ/V2L Evaluation Board are included in the RZ/V2L Evaluation Board Kit. After obtaining the RZ/V2L Evaluation Board Kit, please prepare the following items.

USB Hub, USB Keyboard, USB Mouse and USB Camera. The USB Camera is optional and is only required for specific applications.

USB Type-C Cable, AC Adaptor and micro HDMI Cable. Linux PC, MicroSD Card and SD Card Reader. Please refer to the documentation (https://renesas-rz.github.io/rzv_ai_sdk/1.00/getting_started.html) for the required Ubuntu version. Linux PC must have internet access.

HDMI Monitor and Windows PC. Windows PC also requires internet access.

This is the list of all necessary equipment explained. Please prepare them.

To run AI applications, the following software must be installed on your Ubuntu PC. The Docker Engine is used to build a development environment for AI applications. Git is used to copy AI applications stored on GitHub.

First, install the Docker engine on your Ubuntu PC. 
Click on "Docker" link. Type "ubuntu" in the search window and select "Install Docker Engine on Ubuntu". 
Install Docker Engine following the content of "Set up and install Docker Engine from Docker’s apt repository".

Then, install git on your Ubuntu PC.
Run the "sudo apt-get update" command in the terminal.
Install git.
Set up your username and email address to complete step 2.

This step describes how to obtain RZ/V2L AI SDK. In this step, the AI SDK is downloaded to the Ubuntu PC from the Renesas website. 
In the documentation step 3, click on the "Download Link" button. 
Click on the account symbol and then click "Log In". 
Enter your email address and password to log in. 
Click on the RZ/V2L AI Software Development Kit. 
Click on the Software License Agreement. 
Check the software license agreement. 
Click on the "Accept and download" button. The download will begin. The AI SDK file size is about 10 GB, which will take time to download.

This step describes how to extract RZ/V2L AI SDK package. In this step, the AI SDK zip file is extracted. 
First, create a working directory. 
Then, register the working directory path as an environment variable. 
Move to the working directory. 
Extract the RZ/V2L AI SDK zip file. 
If all directories and files are generated as shown in the log, the AI SDK zip file has been properly extracted.

This step describes how to set up RZ/V2L AI SDK. In this step, a Docker image is generated from the Dockerfile, and then a Docker container is built from the Docker image. 
Move to the ai_sdk_setup directory. 
Create the Docker image named rzv2l_ai_sdk_image. 
Create a new directory named "data" under ai_sdk_setup directory. 
Create the Docker container named rzv2l_ai_sdk_container. 
When the container is created, the prompt will change. 
In the container, copy the tvm runtime (libtvm_runtime.so) to the data directory (/drp-ai_tvm/data). 
Exit the container.

This step describes how to build RZ/V2L AI Application. In this step, the application source file obtained from GitHub is built in the docker container. The build method presented here is only one example. For details on how to build each AI application, please refer to the documentation (https://renesas-rz.github.io/rzv_ai_sdk/1.00/getting_started.html). AI applications are uploaded to GitHub. In this tutorial, R01_object_detection will be the example application, like a "Hello World". 
Move to the data directory in the host Linux environment. 
Clone the specified repository from GitHub. 
Start the container. 
Then bash is opened to run commands in the container and the prompt changes. 
Register the path to the cloned repository as an environment variable. 
Move to the directory where the source code of the Application is stored. Create the directory for building source code. 
Move to "build" directory. Build the application. 
If a file named object_detection is created, the application build has succeeded. Exit the container to complete step 6.

This step describes how to set up the RZ/V2L Evaluation Board Kit. In the first half of this step, the bootloader files are copied from the Ubuntu PC to the Windows PC, and those are written to the RZ/V2L evaluation board via serial communication. To run AI applications, the following software must be installed on your Windows PC. These software are used for the serial communication between Windows PC and RZ/V2L Evaluation Board. The evaluation board is equipped with a USB UART IC made by FTDI, so the driver software made by FTDI is required. In this tutorial, Tera Term is an example of terminal software. 
In the documentation step 7, click on the address of the FTDI chip. 
Click "here" and the Windows driver installer is downloaded. 
Click on the Installation Guides page. 
Click the Windows 10/11 installation guide. 
Follow the installation guide to install the Windows driver. 
To install Tera term, download it from this website.

In preparation for writing Bootloaders to the evaluation board, copy the three files in the bootloader directory (${WORK}/board_setup/bootloader) to your Windows PC. The process of writing bootloaders needs to be done only once. To write bootloaders, Windows PC and RZ/V2L Evaluation Board need to be connected as shown in this figure.

Connect a Windows PC to the evaluation board using the micro USB Cable. Set SW11 as shown in the screen.

Before connecting the USB type-C cable, the resulting connection should be as shown in the figure. 
Supply power to the board by connecting the USB Type-C cable. 
When the two green LEDs light up, press and hold the red power switch for one second. The LED in the red frame lights up. 
Start tera term on your Windows PC. 
Open the New Connection window and select "Serial" Communication and "USB Serial Port". Click the "OK" button. 
Open the Terminal setup window, check the new line conditions, and click the "OK" button. 
Open the Serial port setup and connection window, verify the serial communication settings, and click the "New setting" button. 
Press the blue reset button. 
On Tera Term, "please send!" message appears. 
Open the Send file window, select the Flash Writer file (*.mot), and click the "Open" button. 
Type "XLS2" command. 
Type the value "11E00". 
Then, type the value "00000". 
When "please send!" message appears, open the Send file window, select bl2_bp-smarc-rzv2l_pmic.srec, and click the "Open" button. 
Type "y" to clear the data. 
Type "XLS2" command to write the next file. 
Type the value "00000". 
Then, type the value "1D200". 
When the "please send!" message appears, open the Send file window, select fip-smarc-rzv2l_pmic.srec, and click the "Open" button. 
Type "y" to clear the data. 
Disconnect serial communication and exit teraterm.

After writing bootloaders, the board must be rebooted. Press and hold the red power switch for 2 seconds. When the LED in the red frame turns off, disconnect the USB type-c cable from the evaluation board. Then remove the micro USB Cable.

In the second part of this step, the microSD card is prepared to boot Linux on the RZ/V2L. Two partitions are created on the microSD card. One is used for the Linux kernel image and the Linux device tree file to boot Linux on the RZ/V2L evaluation board. The other is used for the Linux root file system.

Prepare the microSD card with at least 4 GB of free space. The process described in this step will erase all contents stored on your micro SD card. 
Run the "lsblk" command before inserting the microSD card into the Linux PC. 
Check the command result and then, insert the microSD card into the Linux PC. 
Run the "lsblk" command again. 
Compare the two results to see the device name of the microSD card. 
Run the "df" command to see if the partitions on the microSD card are automatically mounted. If any partitions are automatically mounted, unmount them. 
Run the "fdisk" command to create a partition. 
Enter "o" to create a new DOS disklabel. 
Type "n" to create a new partition. 
Type "p" for partition type. 
Press ENTER key for partition number. 
Press ENTER key for first sector. 
Enter "+500M" for last sector. 
Type "Y" to remove the signature. 
Type "n" to create the second partition. 
Type "p" for partition type. 
Press ENTER key for partition number. 
Press ENTER key for first sector. 
Press ENTER key for last sector. 
Type "Y" to remove the signature. 
Type "p" to display partition information. 
Type "t" to change the partition type. 
Type "1" for partition number. 
Type "b" to specify the type. 
Type "w" to write the partition information and finish fdisk command. 
Run "partprobe" command to reflect the partition updates. 
Display the microSD partition information. 
Format the first partition as FAT32. 
Format the second partition as ext4. 
"Run the "df" command to check if the two partitions are created successfully as shown in the console. 
Mount the microSD card partition 1 to write the necessary files. 
Copy the Linux Kernel Image to the microSD card partition 1. 
Copy the Linux Device Tree File to the microSD card partition 1. 
Run "sync" command. 
Unmount the microSD card partition 1. 
Mount the microSD card partition 2 to write the necessary files. 
Extract the Linux root filesystem to partition 2. 
Copy the tvm runtime to the specified location. 
Run "sync" command. 
Unmount the microSD card partition 2.

Step 8 describes how to deploy the application to the board. In this step, the built application file in step 6 and the object files of the pre-trained AI model are copied to the Linux root file system. At the end of this step, the folder structure of the root file system of the micro SD will look like this figure. 
Insert the micro SD card prepared in step 7 into the Linux PC. 
Mount the partition 2. 
Create an application directory named tvm in the root file system. 
Start the container to copy the necessary files for the AI Applications to the execution environment. 
Set the environment variable. 
Move to the yolov3_onnx directory. 
Download the specified file from GitHub. 
Rename it to deploy.so. 
Exit the container. 
Copy the necessary files to the application directory. 
After copying, verify that the required files are in the proper location. 
Run "sync" command. 
Unmount partition 2. 
Eject the microSD card.

This step describes how to boot the RZ/V2L Evaluation Board Kit. In this step, environment variables are set to the RZ/V2L evaluation board using a Windows PC. Linux system starts up on the RZ/V2L evaluation board using the prepared microSD card.

The process of setting environment variables in U-boot command mode needs to be done only once. If you have done this process, you can skip it.

To set boot parameters, the equipment must be connected to the board as shown in this figure. 
Insert the microSD card prepared in step 8 to the evaluation board. 
Set SW11 as shown on the screen. 
Connect the Windows PC to the evaluation board using the micro USB Cable. 
Connect the Google Coral camera to the evaluation board. The blue part of the cable should be on the upper side. 
Connect an HDMI monitor to the evaluation board using the micro HDMI cable. 
Before connecting the USB type-C cable, the resulting connection should be as shown in the figure. 
Supply power to the board by connecting the USB Type-C cable. 
When the two green LEDs light up, press and hold the red power switch for one second. The LED in the red frame lights up. 
Start Tera Term. 
Configure Tera Term as described in step 7 for serial communication between the Windows PC and the board. 
Press the blue reset button. 
Press Enter key before the countdown reaches zero to enter the U-boot command mode. If you missed it, press the reset button again. 
In U-boot command mode, set the environment variables. Please refer to the documentation and copy & paste the commands. 
When the boot process is complete, a login message will be displayed. " 
Input the following information to verify login. 
After confirmation, shutdown is required to reconnect the cables. 
Enter the Shutdown command. 
Verify that the console displays "Power down". 
Press and hold the red power switch for 2 seconds. When the LED in the red frame turns off, disconnect the USB type-c cable from the evaluation board.

Step 10 describes how to run the application. In this step, the AI application stored on the microSD card is run on the RZ/V2L evaluation board. To run the AI application, the board connection must be changed as shown in this figure. 
Disconnect the micro USB Cable from the evaluation board. 
Connect a USB hub. 
Before connecting the USB type-C cable, the resulting connection should be as shown in the figure. 
Power the board again by connecting the USB type-C cable. 
When the two green LEDs light up, press and hold the red power switch for one second. The LED in the red frame lights up. 
On the HDMI monitor, Yocto Linux starts up. From this point, the screen displays the HDMI monitor until the shutdown process starts. 
Click the icon at the top-left corner to open the terminal. Note that the keyboard connected to the RZ/V2L evaluation board is recognized as an English keyboard even if it is a Japanese keyboard. 
Move to tvm directory. 
Change the permission of the object detection application executable file. 
Run the application. 
The object detection application starts. 
The display shows the image captured by the camera and AI results. 
To exit the application, press the super key (Windows key) and the tab key simultaneously to go back to terminal, then press enter key.

This step describes how to shut down the RZ/V2L Evaluation Board Kit. 
Enter the Shutdown command in the terminal. 
Verify that the "Power down" message appears. 
Press and hold the red power switch for 2 seconds. When the LED in the red frame turns off, disconnect the USB type-c cable from the evaluation board. 
Then, disconnect everything connected to the board.

The next time you boot the RZ/V2L Evaluation Board, you do not need to write bootloaders (in step 7) and set environment variables in U-boot command mode (in step 9). Please start from the state shown in this figure.

For more information, please visit Renesas GitHub Pages.