基于Qualcomm® Robotics RB5边缘侧AI应用,目标跟踪检测-CSDN.NET

CSDN首页> 业界

订阅业界RSS

基于Qualcomm® Robotics RB5边缘侧AI应用,目标跟踪检测

发表于 2023-12-08 11:26:54

行人跟踪

此示例使用ByteTrack模型实现了人员跟踪，将通过分配唯一的跟踪器ID来跟踪帧中的人员。示例代码

软件基于高通Snapdragon Neural Processing Engine （以下简称SNPE）开发，硬件基于高通RB5，示例程序可以快速的适配到任何SNPE的硬件。

完整的示例代码代码可以在这里获得

https://github.com/quic/sample-apps-for-robotics-platforms/tree/master/RB5/linux_kernel_5_x/AI-ML-apps/AI_Tracker_Solution

259045792-8a2f6988

此解决方案所需硬件如下：

An Ubuntu 20.04 PC
Qualcomm Robotics RB5 Development kit: https://developer.qualcomm.com/qualcomm-robotics-rb5-kit编辑https://developer.qualcomm.com/qualcomm-robotics-rb5-kit
A USB camera
A display monitor

1．前提条件

Ubuntu 20.04， Python 3.6.9
Qualcomm Robotics RB5 Development kit: https://developer.qualcomm.com/qualcomm-robotics-rb5-kit

2． x86 Host 安装和NN模型编译

应用程序中使用的模型需要把预训练好的权重文件编译换为DLC文件以在应用程序中运行。

2.1 Qualcomm® Neural Processing SDK 安装

SDK 下载Qualcomm Neural Processing SDK for AI Tools & Resources Archive - Qualcomm Developer Network.

SNPE SDK提供了用于模型转换（onnx到dlc）、模型量化的工具。请参阅SDK中详细文档中给出的步骤进行安装. Snapdragon Neural Processing Engine SDK: Main Page

安装 SDK

SDK安装步骤参考 https://developer.qualcomm.com/docs/snpe/setup.html

安装完成后设置环境变量

export SNPE_ROOT=<path to snpe folder>/snpe-1.68.0.3932
export ANDROID_NDK_ROOT=<path to android ndk folder>

从pip查找onnx安装路径

python3.6 -m pip show onnx

在上述命令的输出中查找“Location:”行

export ONNX_DIR=<path from Location line>/onnx

为SDK设置onnx环境

cd $SNPE_ROOT
source bin/envsetup.sh -o $ONNX_DIR

2.2 准备Bytetick模型

按照 ModelPreperation.md 中的步骤导出和编译模型。

Download pre-trained bytetrack_s_mot17.pth.tar

https://drive.google.com/file/d/1uSmhXzyV1Zvb4TJJCzpsZOIcw7CCJLxj/view?usp=sharing

**2.3 *.pth 文件转换为 ONNX**

建议安装Python 3.6.9版本。

sudo apt install python3.6 python3.6-venv build-essential make python3-dev python3.6-dev protobuf-compiler libprotobuf-dev

下载ByteTrack源代码

cd /home/
https://github.com/ifzhang/ByteTrack.git 
cd ByteTrack
pip3 install -r requirements.txt
python3 setup.py develop
pip3 install cython
pip3 install 'git+https://github.com/cocodataset/cocoapi.git#subdirectory=PythonAPI'
pip3 install cython_bbox

导出到ONNX：

python3 tools/export_onnx.py --output-name bytetrack_s.onnx -f exps/example/mot/yolox_s_mix_det.py -c pretrained/bytetrack_s_mot17.pth.tar

要检查输出层和输出张量节点，请在Netron应用程序中打开模型 Netron.，然后单击图像中提到的输出层

将Onnx模型转换为DLC

Convert to DLC

snpe-onnx-to-dlc -i bytetrack_s.onnx --out_node <output name 1> --out_node <output name 2> --out_node <output name 3>

与上面的屏幕截图相对应的示例：

snpe-onnx-to-dlc -i bytetrack_s.onnx --out_node output

使用以下命令执行模型量化

snpe-dlc-quantize --input_dlc=bytetrack_s.dlc --input_list=inputlist.txt --output_dlc=bytetrack_s_quant.dlc

请参考模型目录中给出的inputlist.txt文件，文件保存的是模型实际输入的张量的raw文件路径，更多关于高通模型量化工具使用参考Snapdragon Neural Processing Engine SDK: Quantized vs Non-Quantized Models

3. 安装RTSP视频流服务

示例程序使用live555 RTSP软件来模拟IP摄像头的视频流。

3.1 准备人员视频

如果测试视频是mp4、mkv或其他格式，请准备一个测试视频。需要转换为H264原始视频。以下步骤演示如何将MP4转换为H264原始视频。

sudo apt install ffmpeg

ffmpeg -i /home/ByteTrack/videos/palace.mp4 -f h264 -vcodec /home/ByteTrack/videos/palace.264

3.2 Live555服务器安装

wget http://www.live555.com/liveMedia/public/live555-latest.tar.gz
tar -zxvf live555-latest.tar.gz
cd live/
./genMakefiles linux-64bit
make -j4
cd ..
copy test video to mediaServer folder

cp /home/ByteTrack/videos/palace.264 ./live/mediaServer
cd ./live/mediaServer
./live555MediaServer
rtsp://192.168.4.111:8554/ is the rtsp url.

"192.168.4.111" is RTSP server IP address
8554 is the default port
is the video file name under mediaServer folder In this case, url "rtsp://192.168.4.111:8554/Big_Buck_Bunny_1080_10s_1MB.264" is the video address.

3.3 验证live555服务器

在Windows桌面上下载并安装VLC媒体播放器VLC: Official site - Free multimedia solutions for all OS! - VideoLAN

Launch VLC player, choose "Media->Open Network Stream" input RTSP url rtsp://192.168.4.111:8554/Big_Buck_Bunny_1080_10s_1MB.264

单击播放以测试live555服务器是否正常工作。注意：确保网络地址可访问

如何配置、构建和运行此人员跟踪应用程序

1. 使用如下命令克隆存储库

adb shell
cd /home/
apt install git
git clone https://github.com/quic/sample-apps-for-robotics-platforms.git
cd sample-apps-for-robotics-platforms/RB5/linux_kernel_5_x/AI-ML-apps/AI_Tracker_Solution/

2. 更新应用程序配置

该应用程序包括多种解决方案，如人员入侵检测、头盔检测、火灾检测和裂缝检测。所有解决方案的配置在data/config.json文件中进行了描述。应更新此配置文件以选择所需的解决方案、模型配置和输入/输出流。应用程序可以将rtsp/camera流作为输入，并将输出转储到mp4或hdmi监视器。

Table 1-1 show all the configuration items:

输入配置

key	Value	Description
input-config-name	string	Name of the input config
stream-type	string	Input stream type camera or rtsp
stream-width	int	Width of the input stream
stream-height	int	Height of the input stream
SkipFrame	int	model input height
camera-url	string	rtsp stream path if the input stream is rtsp

模型配置

key	Value	Description
model-name	string	Name of the model
model-path	string	Path of the dlc file
label-path	string	Path of the label file
runtime	string	SNPE Runtime (GPU, CPU, DSP)
nms-threshold	float	NMS threshold
conf-threshold	float	Confidence threshold
labels	int	Number of labels + 5
input-layers	string	Name of the input layers
output-layers	string	Name of the output layers
output-tensors	string	Name of the output tensors

解决方案配置

key	Value	Description
solution-name	string	Name of the Solution
model-name	string	Name of the model configuration to be used
input-config-name	string	Name of the Input configuration to be used
Enable	bool	1 to Enable and 0 to Disable the solution
output-type	string	Filesink to save the output in mp4
		Wayland if display the output on hdmi monitor
output-path	string	Path of the output, Enabled if output type
		is filesink

Example 1: 从相机获取输入流并在hdmi监视器上输出的配置

{
    "input-configs":[
        {
            "input-config-name":"camera",
            "stream-type":"camera",
            "stream-width":1280,
            "stream-height":720,
            "SkipFrame":1,
            "fps-n":30,
            "fps-d":1
        },
 ],
     "model-configs":[
        {
            "model-name":"ByteTrack",
            "model-path":"../models/bytetrack_s_quant.dlc",
            "label-path":"../data/coco_label.txt",
            "runtime":"DSP",
            "labels":85,
            "grids":25200,
            "nms-threshold":0.5,
            "conf-threshold":0.5,
            "input-layers":[
                "images"
            ],
            "output-layers":[
            "/head/Transpose"
            ],
            "output-tensors":[
                "output"
            ],
            "global-threshold":0.2
        },
 ],
     "solution-configs":[
        {
            "solution-name":"People-Tracking-1",
            "model-name":"ByteTrack",
            "input-config-name":"camera",
            "Enable":1,
            "output-type":"wayland",
        },
        {
            "solution-name":"People-Tracking-2",
            "model-name":"ByteTrack",
            "input-config-name":"rtsp4",
            "Enable":0,
            "output-type":"filesink",
            "output-path":"/root/tracking.mp4"
        }
 ]
 }

3. 模型集成

将model推送到应用程序中的model目录中，并更新config.json文件。更新输出层和输出张量。要检查输出层和输出张量节点，请在Netron应用程序Netron中打开模型，然后单击图像中提到的输出层。

 "model-configs":[
      {
          "model-name":"ByteTrack", --> Add model name here. It should match with the model name in solution config
          "model-type":"bytetrack", --> It should be yolov5 for YoloV5 model.
          "model-path":"../models/bytetrack_s_quant1.dlc", --> Path of the quantized model
          "label-path":"../data/label.txt", --> Path to the label file
          "runtime":"DSP", 
          "labels":6, --> Update label here.
          "grids":25200,
          "nms-threshold":0.5,
          "conf-threshold":0.4,
          "input-layers":[
              "images" --> Open the model in netron.app and get the input-layers names.
          ],
          "output-layers":[ --> Refer the steps given above to know the output-layers and output-tensors
              "/head/Transpose",
          ],
          "output-tensors":[
              "output",
          ],
          "global-threshold":0.2
      },

4. 在RB5设备上下载和部署高通SNPE SDK库的步骤

Download SDK from https://developer.qualcomm.com/software/qualcomm-neural-processing-sdk/tools-archive.

Windows

cd snpe-1.68.0\snpe-1.68.0.3932
adb push lib\aarch64-ubuntu-gcc7.5\. /usr/lib/
adb push lib\aarch64-ubuntu-gcc7.5\libsnpe_dsp_domains_v2.so /usr/lib/rfsa/adsp/
adb push lib\dsp\. /usr/lib/rfsa/adsp/
adb push bin\aarch64-ubuntu-gcc7.5\snpe-net-run /usr/bin/

Linux

cd snpe-1.68.0/snpe-1.68.0.3932/
adb push lib/aarch64-ubuntu-gcc7.5/* /usr/lib/
adb push lib/aarch64-ubuntu-gcc7.5/libsnpe_dsp_domains_v2.so /usr/lib/rfsa/adsp/
adb push lib/dsp/* /usr/lib/rfsa/adsp/
adb push bin/aarch64-ubuntu-gcc7.5/snpe-net-run /usr/bin/

Verify SDK version

adb shell
chmod +x /usr/bin/snpe-net-run
snpe-net-run --version

5. 执行应用程序的步骤

5.1 Installing OpenCV 4.5.5

Download OpenCV 4.5.5 source code from https://codeload.github.com/opencv/opencv/tar.gz/refs/tags/4.5.5

adb shell
wget https://codeload.github.com/opencv/opencv/tar.gz/refs/tags/4.5.5 -O opencv-4.5.5.tar.gz
tar  -zxvf opencv-4.5.5.tar.gz
cd ./opencv-4.5.5

安装依赖项

apt install build-essential cmake unzip git pkg-config
apt install libjpeg-dev libpng-dev libtiff-dev
apt-get install libjsoncpp-dev libjson-glib-dev libgflags-dev libgstreamer1.0-dev libgstreamer-plugins-base1.0-dev libgstreamer-plugins-bad1.0-dev gstreamer1.0-plugins-base gstreamer1.0-plugins-good gstreamer1.0-plugins-bad gstreamer1.0-plugins-ugly gstreamer1.0-libav gstreamer1.0-tools gstreamer1.0-x gstreamer1.0-alsa gstreamer1.0-gl gstreamer1.0-gtk3 gstreamer1.0-qt5 gstreamer1.0-pulseaudio
apt install libjasper-dev
apt-get install libeigen3-dev

如果您收到一个关于libjasper-dev丢失的错误，请按照以下说明进行操作：

wget http://ports.ubuntu.com/ubuntu-ports/pool/main/j/jasper/libjasper-dev_1.900.1-debian1-2.4ubuntu1.3_arm64.deb
dpkg -i libjasper-dev_1.900.1-debian1-2.4ubuntu1.3_arm64.deb

wget http://ports.ubuntu.com/ubuntu-ports/pool/main/j/jasper/libjasper1_1.900.1-debian1-2.4ubuntu1.3_arm64.deb
dpkg -i libjasper1_1.900.1-debian1-2.4ubuntu1.3_arm64.deb

否则（如果安装了libjasper-dev），继续进行：

apt install libavcodec-dev libavformat-dev libswscale-dev libv4l-dev
apt install libxvidcore-dev libx264-dev

OpenCV的highgui模块依赖于GTK库进行GUI操作。安装GTK命令：

apt install libgtk-3-dev

安装Python 3头文件和库：

apt install libatlas-base-dev gfortran
apt install python3.6-dev

构建和安装：

mkdir build && cd build

cmake -D CMAKE_BUILD_TYPE=RELEASE \
      -D CMAKE_INSTALL_PREFIX=/usr/local/opencv4.5 \
      -D OPENCV_ENABLE_NONFREE=ON \
      -D OPENCV_GENERATE_PKGCONFIG=YES \
      -D WITH_QT=ON \
      -D WITH_OPENGL=ON \
      -D BUILD_EXAMPLES=OFF \
      -D INSTALL_PYTHON_EXAMPLES=OFF \
      ..
make -j8
make install

5.2 构建应用程序

adb shell
cd /home/sample-apps-for-robotics-platforms/RB5/linux_kernel_5_x/AI-ML-apps/AI_Tracker_Solution/
mkdir build 
cd build
cmake -DSNPE_SDK_BASE_DIR=<SDK Directory Path>/snpe-1.68.0.3932 ..
make

5.3 运行应用程序

在监视器上显示输出。请通过HDMI电缆将显示器连接到设备。按照以下说明启用weston：

export XDG_RUNTIME_DIR="/run/user/root"
cd build
./out/main -c ../data/config.json

验证结果

如果输出类型为filesink，请检查“输出路径”的目录是否为filesink。或者，请检查与HDMI连接的显示器的输出。

作者：高通工程师，廖洋洋

CSDN官方微信

扫描二维码,向CSDN吐槽

微信号：CSDNnews

程序员移动端【订阅下载】

微博关注