精华内容
下载资源
问答
  • OpenPose 基于OpenCV DNN 的手部关键点检测

    万次阅读 热门讨论 2019-05-30 21:38:35
    原文:OpenPose 基于OpenCV DNN 的手部关键点检测 - AIUAI 手部关键点检测,旨在找出给定图片中手指上的关节点及指尖关节点. 其类似于面部关键点检测(Facial Landmark Detection) 和人体关键点检测(Human Body Pose ...

    原文:OpenPose 基于OpenCV DNN 的手部关键点检测 - AIUAI

    手部关键点检测,旨在找出给定图片中手指上的关节点及指尖关节点. 其类似于面部关键点检测(Facial Landmark Detection) 和人体关键点检测(Human Body Pose Estimation);不同之处是,整个手部是作为一个目标物体的.

    OpenPose 基于OpenCV DNN 的单人姿态估计 - AIUAI

    OpenPose 基于OpenCV DNN 的多人姿态估计 - AIUAI

    在这里插入图片描述

    手部关键点检测的应用场景包括:

    [1] - 手势识别

    [2] - 手语识别与理解

    [3] - 手部的行为识别等.

    1. Background

    主要是基于 CMU Perceptual Computing Lab 开源的手部关键点检测模型.

    Github - OpenPose

    手部关键点检测器的实现主要是基于论文:Hand Keypoint Detection in Single Images using Multiview Bootstrapping - CVPR2017.

    其中,如下:

    在这里插入图片描述

    论文中,首先采用少量标注的人手部关键点图像数据集,训练类似于人体姿态关键点所使用的 CPM - Convolutional Pose Machines 网络,以得到手部关键点的粗略估计. 采用了 31个 HD 高清摄像头 从不同的视角对人手部进行拍摄.

    然后,将拍摄图像送入手部关键点检测器,以初步得到许多粗略的关键点检测结果. 一旦有了同一手部的不同视角的关键点,则构建关键点测量(Keypoint triangulation),以得到关键点的 3D 位置. 关键点的 3D 位置被从 3D 重新投影到每一幅不同视角的 2D 图片,并采用 2D 图像和关键点,进一步训练网络.,以鲁棒的预测手部关键点位置. 这对于关键点难以预测的图片而言是尤其重要的. 采用这种方式,通过少量几次迭代,即可得到较为准确的手部关键点检测器.

    总而言之,关键点检测器和多视角图像(multi-view images) 一起构建了较为准确的手部关键点检测模型. 采用的检测网络类似于人体关键点中所用的网络结构. 进度提升的主要因素是采用了多视角图片标注图片数据集.

    手部关键点检测模型共输出 22 个关键点,其中包括手部的 21 个点,第 22 个点表示背景. 如图:

    Github 项目 - OpenPose 关键点输出格式 - AIUAI

    在这里插入图片描述

    2. 具体实现

    2.1. 手部关键点检测模型下载

    [1] - hand/pose_deploy.prototxt

    [2] - hand/pose_iter_102000.caffemodel

    2.2. Python 实现

    #!/usr/bin/python3
    #!--*-- coding: utf-8 --*--
    from __future__ import division
    import os
    import cv2
    import time
    import numpy as np
    import matplotlib.pyplot as plt
    
    
    class general_pose_model(object):
        def __init__(self, modelpath):
            self.num_points = 22
            self.point_pairs = [[0,1],[1,2],[2,3],[3,4],
                                [0,5],[5,6],[6,7],[7,8],
                                [0,9],[9,10],[10,11],[11,12],
                                [0,13],[13,14],[14,15],[15,16],
                                [0,17],[17,18],[18,19],[19,20]]
            # self.inWidth = 368
            self.inHeight = 368
            self.threshold = 0.1
            self.hand_net = self.get_hand_model(modelpath)
    
    
        def get_hand_model(self, modelpath):
    
            prototxt   = os.path.join(modelpath, "hand/pose_deploy.prototxt")
            caffemodel = os.path.join(modelpath, "hand/pose_iter_102000.caffemodel")
            hand_model = cv2.dnn.readNetFromCaffe(prototxt, caffemodel)
    
            return hand_model
    
    
        def predict(self, imgfile):
            img_cv2 = cv2.imread(imgfile)
            img_height, img_width, _ = img_cv2.shape
            aspect_ratio = img_width / img_height
    
            inWidth = int(((aspect_ratio * self.inHeight) * 8) // 8)
            inpBlob = cv2.dnn.blobFromImage(img_cv2, 1.0 / 255, (inWidth, self.inHeight), (0, 0, 0), swapRB=False, crop=False)
    
            self.hand_net.setInput(inpBlob)
    
            output = self.hand_net.forward()
    
            # vis heatmaps
            self.vis_heatmaps(imgfile, output)
    
            #
            points = []
            for idx in range(self.num_points):
                probMap = output[0, idx, :, :] # confidence map.
                probMap = cv2.resize(probMap, (img_width, img_height))
    
                # Find global maxima of the probMap.
                minVal, prob, minLoc, point = cv2.minMaxLoc(probMap)
    
                if prob > self.threshold:
                    points.append((int(point[0]), int(point[1])))
                else:
                    points.append(None)
    
            return points
    
    
        def vis_heatmaps(self, imgfile, net_outputs):
            img_cv2 = cv2.imread(imgfile)
            plt.figure(figsize=[10, 10])
    
            for pdx in range(self.num_points):
                probMap = net_outputs[0, pdx, :, :]
                probMap = cv2.resize(probMap, (img_cv2.shape[1], img_cv2.shape[0]))
                plt.subplot(5, 5, pdx+1)
                plt.imshow(cv2.cvtColor(img_cv2, cv2.COLOR_BGR2RGB))
                plt.imshow(probMap, alpha=0.6)
                plt.colorbar()
                plt.axis("off")
            plt.show()
    
    
        def vis_pose(self, imgfile, points):
            img_cv2 = cv2.imread(imgfile)
            img_cv2_copy = np.copy(img_cv2)
            for idx in range(len(points)):
                if points[idx]:
                    cv2.circle(img_cv2_copy, points[idx], 8, (0, 255, 255), thickness=-1,
                               lineType=cv2.FILLED)
                    cv2.putText(img_cv2_copy, "{}".format(idx), points[idx], cv2.FONT_HERSHEY_SIMPLEX,
                                1, (0, 0, 255), 2, lineType=cv2.LINE_AA)
    
            # Draw Skeleton
            for pair in self.point_pairs:
                partA = pair[0]
                partB = pair[1]
    
                if points[partA] and points[partB]:
                    cv2.line(img_cv2, points[partA], points[partB], (0, 255, 255), 3)
                    cv2.circle(img_cv2, points[partA], 8, (0, 0, 255), thickness=-1, lineType=cv2.FILLED)
    
            plt.figure(figsize=[10, 10])
            plt.subplot(1, 2, 1)
            plt.imshow(cv2.cvtColor(img_cv2, cv2.COLOR_BGR2RGB))
            plt.axis("off")
            plt.subplot(1, 2, 2)
            plt.imshow(cv2.cvtColor(img_cv2_copy, cv2.COLOR_BGR2RGB))
            plt.axis("off")
            plt.show()
    
    
    if __name__ == '__main__':
        print("[INFO]Pose estimation.")
    
        imgs_path = "/path/to/hand"
        img_files = [os.path.join(imgs_path, img_file) for img_file in os.listdir(imgs_path)]
    
        #
        start = time.time()
        modelpath = "/path/to/hand_models"
        pose_model = general_pose_model(modelpath)
        print("[INFO]Model loads time: ", time.time() - start)
    
        for img_file in img_files:
            start = time.time()
            res_points = pose_model.predict(img_file)
            print("[INFO]Model predicts time: ", time.time() - start)
            pose_model.vis_pose(img_file, res_points)
    
        print("[INFO]Done.")
    

    示例如图:

    在这里插入图片描述

    各手部关键点的 heatmap:

    在这里插入图片描述

    2.3. C++ 实现

    #include <opencv2/dnn.hpp>
    #include <opencv2/imgproc.hpp>
    #include <opencv2/highgui.hpp>
    #include <iostream>
    
    using namespace std;
    using namespace cv;
    using namespace cv::dnn;
    
    
    const int POSE_PAIRS[20][2] =
    {
        {0,1}, {1,2}, {2,3}, {3,4},         // thumb
        {0,5}, {5,6}, {6,7}, {7,8},         // index
        {0,9}, {9,10}, {10,11}, {11,12},    // middle
        {0,13}, {13,14}, {14,15}, {15,16},  // ring
        {0,17}, {17,18}, {18,19}, {19,20}   // small
    };
    
    string protoFile = "hand/pose_deploy.prototxt";
    string weightsFile = "hand/pose_iter_102000.caffemodel";
    
    int nPoints = 22;
    
    int main(int argc, char **argv)
    {
        cout << "USAGE : ./handPoseImage <imageFile> " << endl;
    
        string imageFile = "right-frontal.jpg";
        // Take arguments from commmand line
        if (argc == 2)
        {
          imageFile = argv[1];
        }
    
        float thresh = 0.01;
    
        Mat frame = imread(imageFile);
        Mat frameCopy = frame.clone();
        int frameWidth = frame.cols;
        int frameHeight = frame.rows;
    
        float aspect_ratio = frameWidth/(float)frameHeight;
        int inHeight = 368;
        int inWidth = (int(aspect_ratio*inHeight) * 8) / 8;
    
        cout << "inWidth = " << inWidth << " ; inHeight = " << inHeight << endl;
    
        double t = (double) cv::getTickCount();
        Net net = readNetFromCaffe(protoFile, weightsFile);
    
        Mat inpBlob = blobFromImage(frame, 1.0 / 255, Size(inWidth, inHeight), Scalar(0, 0, 0), false, false);
    
        net.setInput(inpBlob);
    
        Mat output = net.forward();
    
        int H = output.size[2];
        int W = output.size[3];
    
        // find the position of the body parts
        vector<Point> points(nPoints);
        for (int n=0; n < nPoints; n++)
        {
            // Probability map of corresponding body's part.
            Mat probMap(H, W, CV_32F, output.ptr(0,n));
            resize(probMap, probMap, Size(frameWidth, frameHeight));
    
            Point maxLoc;
            double prob;
            minMaxLoc(probMap, 0, &prob, 0, &maxLoc);
            if (prob > thresh)
            {
                circle(frameCopy, cv::Point((int)maxLoc.x, (int)maxLoc.y), 8, Scalar(0,255,255), -1);
                cv::putText(frameCopy, cv::format("%d", n), cv::Point((int)maxLoc.x, (int)maxLoc.y), cv::FONT_HERSHEY_COMPLEX, 1, cv::Scalar(0, 0, 255), 2);
    
            }
            points[n] = maxLoc;
        }
    
        int nPairs = sizeof(POSE_PAIRS)/sizeof(POSE_PAIRS[0]);
    
        for (int n = 0; n < nPairs; n++)
        {
            // lookup 2 connected body/hand parts
            Point2f partA = points[POSE_PAIRS[n][0]];
            Point2f partB = points[POSE_PAIRS[n][1]];
    
            if (partA.x<=0 || partA.y<=0 || partB.x<=0 || partB.y<=0)
                continue;
    
            line(frame, partA, partB, Scalar(0,255,255), 8);
            circle(frame, partA, 8, Scalar(0,0,255), -1);
            circle(frame, partB, 8, Scalar(0,0,255), -1);
        }
    
        t = ((double)cv::getTickCount() - t)/cv::getTickFrequency();
        cout << "Time Taken = " << t << endl;
        imshow("Output-Keypoints", frameCopy);
        imshow("Output-Skeleton", frame);
        imwrite("Output-Skeleton.jpg", frame);
    
        waitKey();
    
        return 0;
    }
    

    3. 参考

    [1] - Hand Keypoint Detection using Deep Learning and OpenCV - 2018.10.08

    [2] - Github - spmallick/learnopencv/HandPose

    [3] - Stackoverflow - how-can-i-use-smoothing-techniques-to-remove-jitter-in-pose-estimation - 关键点平滑

    展开全文
  • 目录PaddleHub手部关键点检测hand_pose_localization一、安装新版Hub二、定义待预测照片三、API预测四、加载预训练模型并预测五、实现效果 PaddleHub手部关键点检测 hand_pose_localization openpose 手部关键点检测...

    PaddleHub手部关键点检测

    hand_pose_localization

    openpose 手部关键点检测模型。

    NOTE: 如果您在本地运行该项目示例,需要首先安装PaddleHub。如果您在线运行,需要首先fork该项目示例。之后按照该示例操作即可。

    hand_pose_localization模型链接:https://www.paddlepaddle.org.cn/hubdetail?name=hand_pose_localization&en_category=KeyPointDetection

    环境:PaddlePaddle2.0.0rc PaddleHub2.0.0b1 hand_pose_localization 1.0.1(最新版)

    一、安装新版Hub

    !pip install paddlehub==2.0.0b1 -i https://pypi.tuna.tsinghua.edu.cn/simple
    

    二、定义待预测照片

    以本示例中文件夹下shou.jpg为待预测图片

    image = "shou.jpg"
    

    三、API预测

    def keypoint_detection(
        self,
        images=None,
        paths=None,
        batch_size=1,
        output_dir='output',
        visualization=False
    )
    

    预测API,识别出人体手部关键点。

    参数

    • images (list[numpy.ndarray]): 图片数据,ndarray.shape 为 [H, W, C], 默认设为 None;
    • paths (list[str]): 图片的路径, 默认设为 None;
    • batch_size (int): batch 的大小,默认设为 1;
    • visualization (bool): 是否将识别结果保存为图片文件,默认设为 False;
    • output_dir (str): 图片的保存路径,默认设为 output。

    返回

    • res (list[list[listint]]): 每张图片识别到的21个手部关键点组成的列表,每个关键点的格式为x, y,若有关键点未识别到则为None

    四、加载预训练模型并预测

    import cv2
    import paddlehub as hub
    
    # use_gpu:是否使用GPU进行预测
    model = hub.Module(name='hand_pose_localization', use_gpu=False)
    
    # 调用关键点检测API
    result = model.keypoint_detection(images=[cv2.imread(image)],visualization=True)
    
    # or
    # result = model.keypoint_detection(paths=['/PATH/TO/IMAGE'])
    
    # 打印预测结果
    print(result)
    

    五、实现效果

    点我进入项目

    展开全文
  • 手部关键点检测,旨在找出给定图片中手指上的关节点及指尖关节点. 其类似于面部关键点检测(Facial Landmark Detection) 和人体关键点检测(手部关键点检测的应用场景包括:[1] - 手势识别[2] - 手语识别与理解[3] - ...

    手部关键点检测,旨在找出给定图片中手指上的关节点及指尖关节点. 其类似于面部关键点检测(Facial Landmark Detection) 和人体关键点检测(

    手部关键点检测的应用场景包括:

    [1] - 手势识别

    [2] - 手语识别与理解

    [3] - 手部的行为识别等.

    1. Background

    主要是基于 CMU Perceptual Computing Lab 开源的手部关键点检测模型.

    其中,如下:

    论文中,首先采用少量标注的人手部关键点图像数据集,训练类似于人体姿态关键点所使用的 CPM - 31个 HD 高清摄像头 从不同的视角对人手部进行拍摄.

    然后,将拍摄图像送入手部关键点检测器,以初步得到许多粗略的关键点检测结果. 一旦有了同一手部的不同视角的关键点,则构建关键点测量(Keypoint triangulation),以得到关键点的 3D 位置. 关键点的 3D 位置被从 3D 重新投影到每一幅不同视角的 2D 图片,并采用 2D 图像和关键点,进一步训练网络.,以鲁棒的预测手部关键点位置. 这对于关键点难以预测的图片而言是尤其重要的. 采用这种方式,通过少量几次迭代,即可得到较为准确的手部关键点检测器.

    总而言之,关键点检测器和多视角图像(multi-view images) 一起构建了较为准确的手部关键点检测模型. 采用的检测网络类似于人体关键点中所用的网络结构. 进度提升的主要因素是采用了多视角图片标注图片数据集.

    手部关键点检测模型共输出 22 个关键点,其中包括手部的 21 个点,第 22 个点表示背景. 如图:

    2. 具体实现

    2.1. 手部关键点检测模型下载

    2.2. Python 实现#!/usr/bin/python3

    #!--*-- coding: utf-8 --*--

    from __future__ import division

    import os

    import cv2

    import time

    import numpy as np

    import matplotlib.pyplot as plt

    class general_pose_model(object):

    def __init__(self, modelpath):

    self.num_points = 22

    self.point_pairs = [[0,1],[1,2],[2,3],[3,4],

    [0,5],[5,6],[6,7],[7,8],

    [0,9],[9,10],[10,11],[11,12],

    [0,13],[13,14],[14,15],[15,16],

    [0,17],[17,18],[18,19],[19,20]]

    # self.inWidth = 368

    self.inHeight = 368

    self.threshold = 0.1

    self.hand_net = self.get_hand_model(modelpath)

    def get_hand_model(self, modelpath):

    prototxt = os.path.join(modelpath, "hand/pose_deploy.prototxt")

    caffemodel = os.path.join(modelpath, "hand/pose_iter_102000.caffemodel")

    hand_model = cv2.dnn.readNetFromCaffe(prototxt, caffemodel)

    return hand_model

    def predict(self, imgfile):

    img_cv2 = cv2.imread(imgfile)

    img_height, img_width, _ = img_cv2.shape

    aspect_ratio = img_width / img_height

    inWidth = int(((aspect_ratio * self.inHeight) * 8) // 8)

    inpBlob = cv2.dnn.blobFromImage(img_cv2, 1.0 / 255, (inWidth, self.inHeight), (0, 0, 0), swapRB=False, crop=False)

    self.hand_net.setInput(inpBlob)

    output = self.hand_net.forward()

    # vis heatmaps

    self.vis_heatmaps(imgfile, output)

    #

    points = []

    for idx in range(self.num_points):

    probMap = output[0, idx, :, :] # confidence map.

    probMap = cv2.resize(probMap, (img_width, img_height))

    # Find global maxima of the probMap.

    minVal, prob, minLoc, point = cv2.minMaxLoc(probMap)

    if prob > self.threshold:

    points.append((int(point[0]), int(point[1])))

    else:

    points.append(None)

    return points

    def vis_heatmaps(self, imgfile, net_outputs):

    img_cv2 = cv2.imread(imgfile)

    plt.figure(figsize=[10, 10])

    for pdx in range(self.num_points):

    probMap = net_outputs[0, pdx, :, :]

    probMap = cv2.resize(probMap, (img_cv2.shape[1], img_cv2.shape[0]))

    plt.subplot(5, 5, pdx+1)

    plt.imshow(cv2.cvtColor(img_cv2, cv2.COLOR_BGR2RGB))

    plt.imshow(probMap, alpha=0.6)

    plt.colorbar()

    plt.axis("off")

    plt.show()

    def vis_pose(self, imgfile, points):

    img_cv2 = cv2.imread(imgfile)

    img_cv2_copy = np.copy(img_cv2)

    for idx in range(len(points)):

    if points[idx]:

    cv2.circle(img_cv2_copy, points[idx], 8, (0, 255, 255), thickness=-1,

    lineType=cv2.FILLED)

    cv2.putText(img_cv2_copy, "{}".format(idx), points[idx], cv2.FONT_HERSHEY_SIMPLEX,

    1, (0, 0, 255), 2, lineType=cv2.LINE_AA)

    # Draw Skeleton

    for pair in self.point_pairs:

    partA = pair[0]

    partB = pair[1]

    if points[partA] and points[partB]:

    cv2.line(img_cv2, points[partA], points[partB], (0, 255, 255), 3)

    cv2.circle(img_cv2, points[partA], 8, (0, 0, 255), thickness=-1, lineType=cv2.FILLED)

    plt.figure(figsize=[10, 10])

    plt.subplot(1, 2, 1)

    plt.imshow(cv2.cvtColor(img_cv2, cv2.COLOR_BGR2RGB))

    plt.axis("off")

    plt.subplot(1, 2, 2)

    plt.imshow(cv2.cvtColor(img_cv2_copy, cv2.COLOR_BGR2RGB))

    plt.axis("off")

    plt.show()

    if __name__ == '__main__':

    print("[INFO]Pose estimation.")

    imgs_path = "/path/to/hand"

    img_files = [os.path.join(imgs_path, img_file) for img_file in os.listdir(imgs_path)]

    #

    start = time.time()

    modelpath = "/path/to/hand_models"

    pose_model = general_pose_model(modelpath)

    print("[INFO]Model loads time: ", time.time() - start)

    for img_file in img_files:

    start = time.time()

    res_points = pose_model.predict(img_file)

    print("[INFO]Model predicts time: ", time.time() - start)

    pose_model.vis_pose(img_file, res_points)

    print("[INFO]Done.")

    示例如图:

    各手部关键点的 heatmap:

    2.3. C++ 实现#include

    #include

    #include

    #include

    using namespace std;

    using namespace cv;

    using namespace cv::dnn;

    const int POSE_PAIRS[20][2] =

    {

    {0,1}, {1,2}, {2,3}, {3,4}, // thumb

    {0,5}, {5,6}, {6,7}, {7,8}, // index

    {0,9}, {9,10}, {10,11}, {11,12}, // middle

    {0,13}, {13,14}, {14,15}, {15,16}, // ring

    {0,17}, {17,18}, {18,19}, {19,20} // small

    };

    string protoFile = "hand/pose_deploy.prototxt";

    string weightsFile = "hand/pose_iter_102000.caffemodel";

    int nPoints = 22;

    int main(int argc, char **argv)

    {

    cout << "USAGE : ./handPoseImage " << endl;

    string imageFile = "right-frontal.jpg";

    // Take arguments from commmand line

    if (argc == 2)

    {

    imageFile = argv[1];

    }

    float thresh = 0.01;

    Mat frame = imread(imageFile);

    Mat frameCopy = frame.clone();

    int frameWidth = frame.cols;

    int frameHeight = frame.rows;

    float aspect_ratio = frameWidth/(float)frameHeight;

    int inHeight = 368;

    int inWidth = (int(aspect_ratio*inHeight) * 8) / 8;

    cout << "inWidth = " << inWidth << " ; inHeight = " << inHeight << endl;

    double t = (double) cv::getTickCount();

    Net net = readNetFromCaffe(protoFile, weightsFile);

    Mat inpBlob = blobFromImage(frame, 1.0 / 255, Size(inWidth, inHeight), Scalar(0, 0, 0), false, false);

    net.setInput(inpBlob);

    Mat output = net.forward();

    int H = output.size[2];

    int W = output.size[3];

    // find the position of the body parts

    vector points(nPoints);

    for (int n=0; n < nPoints; n++)

    {

    // Probability map of corresponding body's part.

    Mat probMap(H, W, CV_32F, output.ptr(0,n));

    resize(probMap, probMap, Size(frameWidth, frameHeight));

    Point maxLoc;

    double prob;

    minMaxLoc(probMap, 0, &prob, 0, &maxLoc);

    if (prob > thresh)

    {

    circle(frameCopy, cv::Point((int)maxLoc.x, (int)maxLoc.y), 8, Scalar(0,255,255), -1);

    cv::putText(frameCopy, cv::format("%d", n), cv::Point((int)maxLoc.x, (int)maxLoc.y), cv::FONT_HERSHEY_COMPLEX, 1, cv::Scalar(0, 0, 255), 2);

    }

    points[n] = maxLoc;

    }

    int nPairs = sizeof(POSE_PAIRS)/sizeof(POSE_PAIRS[0]);

    for (int n = 0; n < nPairs; n++)

    {

    // lookup 2 connected body/hand parts

    Point2f partA = points[POSE_PAIRS[n][0]];

    Point2f partB = points[POSE_PAIRS[n][1]];

    if (partA.x<=0 || partA.y<=0 || partB.x<=0 || partB.y<=0)

    continue;

    line(frame, partA, partB, Scalar(0,255,255), 8);

    circle(frame, partA, 8, Scalar(0,0,255), -1);

    circle(frame, partB, 8, Scalar(0,0,255), -1);

    }

    t = ((double)cv::getTickCount() - t)/cv::getTickFrequency();

    cout << "Time Taken = " << t << endl;

    imshow("Output-Keypoints", frameCopy);

    imshow("Output-Skeleton", frame);

    imwrite("Output-Skeleton.jpg", frame);

    waitKey();

    return 0;

    }

    3. 参考

    展开全文
  • 手部关键点检测是在手指上找到关节以及在给定图像中找到指尖的过程。它类似于在脸部(面部关键点检测)或身体(人体姿势估计)上找到关键点。但是手部检测不同的地方在于,我们将整个手部视为一个对象。 美国卡耐基...

    目录

    1 背景

    2 实现

    3. 结果和代码

    4 参考


    手部关键点检测是在手指上找到关节以及在给定图像中找到指尖的过程。它类似于在脸部(面部关键点检测)或身体(人体姿势估计)上找到关键点。但是手部检测不同的地方在于,我们将整个手部视为一个对象。

    美国卡耐基梅隆大学智能感知实验室(CMU Perceptual Computing Lab)发布了手的关键点检测模型。详情见:

    https://arxiv.org/pdf/1704.07809.pdf

    我们将在本文介绍如何调用该模型。

    1 背景

    上图出自上面说的论文

    他们从一小组标记的手部图像开始,并使用神经网络(卷积姿势分析机https://arxiv.org/pdf/1602.00134.pdf

    )来粗略估计手部关键点。他们设置了一个多视图系统可以从31个高清摄像头获取来自不同视点或角度的图像。

    他们将这些图像传递通过检测器,以获得许多粗略的关键点预测。一旦从不同视图获得同一手的检测到的关键点,就会执行关键点三角测量以获得关键点的3D位置。关键点的3D位置用于通过从3D到2D的重投影来稳健地预测关键点。这对于难以预测关键点的图像尤其重要。通过这种方式,他们可以在几次迭代中获得更好的检测器。

    总之,他们使用关键点检测器和多视图图像来提出改进的检测器。改进的主要来源是标记的图像集的多视图图像。

    该模型产生22个关键点。手有21个关键点(0到20号关键点),而第22个关键点代表背景。关键点位置如下图所示:

    2 实现

    从此链接下载该模型:

    http://posefs1.perception.cs.cmu.edu/OpenPose/models/hand/pose_iter_102000.caffemodel

    这是一个caffe模型。

    模型读取预测代码和其他caffe模型一样,如下所示:

    	//模型文件位置
    	string protoFile = "./model/pose_deploy.prototxt";
    	string weightsFile = "./model/pose_iter_102000.caffemodel";
    
    	// read image 读取图像
    	string imageFile = "./image/hand.jpg";
    	Mat frame = imread(imageFile);
    	if (frame.empty())
    	{
    		cout << "check image" << endl;
    		return 0;
    	}
    	//复制图像
    	Mat frameCopy = frame.clone();
    	//读取图像长宽
    	int frameWidth = frame.cols;
    	int frameHeight = frame.rows;
    
    	float thresh = 0.01;
    
    	//原图宽高比
    	float aspect_ratio = frameWidth / (float)frameHeight;
    	int inHeight = 368;
    	//缩放图像
    	int inWidth = (int(aspect_ratio*inHeight) * 8) / 8;
    
    	cout << "inWidth = " << inWidth << " ; inHeight = " << inHeight << endl;
    
    	double t = (double)cv::getTickCount();
    	//调用caffe模型
    	Net net = readNetFromCaffe(protoFile, weightsFile);
    	Mat inpBlob = blobFromImage(frame, 1.0 / 255, Size(inWidth, inHeight), Scalar(0, 0, 0), false, false);
    	net.setInput(inpBlob);
    	Mat output = net.forward();
    
    	int H = output.size[2];
    	int W = output.size[3];

    输出有22个矩阵,每个矩阵是关键点的概率图。为了找到确切的关键点,首先,我们将概率图缩放到原始图像的大小。然后通过查找概率图的最大值来找到关键点的位置。这是使用OpenCV中的minmaxLoc函数完成的。我们绘制检测到的点以及图像上的编号。我们将使用检测到的点来获取关键点形成的骨架并将其绘制在图像上。画骨架代码如下:

    	// find the position of the body parts 找到各点的位置
    	vector<Point> points(nPoints);
    	for (int n = 0; n < nPoints; n++)
    	{
    		// Probability map of corresponding body's part. 第一个特征点的预测矩阵
    		Mat probMap(H, W, CV_32F, output.ptr(0, n));
    		//放大预测矩阵
    		resize(probMap, probMap, Size(frameWidth, frameHeight));
    
    		Point maxLoc;
    		double prob;
    		//寻找预测矩阵,最大值概率以及最大值的坐标位置
    		minMaxLoc(probMap, 0, &prob, 0, &maxLoc);
    		if (prob > thresh)
    		{
    			//画图
    			circle(frameCopy, cv::Point((int)maxLoc.x, (int)maxLoc.y), 8, Scalar(0, 255, 255), -1);
    			cv::putText(frameCopy, cv::format("%d", n), cv::Point((int)maxLoc.x, (int)maxLoc.y), cv::FONT_HERSHEY_COMPLEX, 1, cv::Scalar(0, 0, 255), 2);
    		}
    		//保存特征点的坐标
    		points[n] = maxLoc;
    	}
    
    	//获取要画的骨架线个数
    	int nPairs = sizeof(POSE_PAIRS) / sizeof(POSE_PAIRS[0]);
    
    	//连接点,画骨架
    	for (int n = 0; n < nPairs; n++)
    	{
    		// lookup 2 connected body/hand parts
    		Point2f partA = points[POSE_PAIRS[n][0]];
    		Point2f partB = points[POSE_PAIRS[n][1]];
    
    		if (partA.x <= 0 || partA.y <= 0 || partB.x <= 0 || partB.y <= 0)
    			continue;
    
    		//画骨条线
    		line(frame, partA, partB, Scalar(0, 255, 255), 8);
    		circle(frame, partA, 8, Scalar(0, 0, 255), -1);
    		circle(frame, partB, 8, Scalar(0, 0, 255), -1);
    	}
    

    结果如下: 

    3. 结果和代码

    需要注意的一点是,检测器需要手周围的边界框来预测关键点。因此,为了获得更好的效果,手应靠近相机,反正总而言之手的位置要清楚,在屏幕中央。现在的深度学习只能这样。精度不怎么高,只能在特定场合下使用,就是先确定关键点,然后训练模型,基于统计进行检测。

    代码见:

    https://github.com/luohenyueji/OpenCV-Practical-Exercise

    C++代码:

    // HandPoints_detection.cpp : 此文件包含 "main" 函数。程序执行将在此处开始并结束。
    //
    
    #include "pch.h"
    #include <iostream>
    #include <opencv2/opencv.hpp>
    
    using namespace std;
    using namespace cv;
    using namespace cv::dnn;
    
    //各个部位连接线坐标,比如(0,1)表示第0特征点和第1特征点连接线为拇指
    const int POSE_PAIRS[20][2] =
    {
    	{0,1}, {1,2}, {2,3}, {3,4},         // thumb
    	{0,5}, {5,6}, {6,7}, {7,8},         // index
    	{0,9}, {9,10}, {10,11}, {11,12},    // middle
    	{0,13}, {13,14}, {14,15}, {15,16},  // ring
    	{0,17}, {17,18}, {18,19}, {19,20}   // small
    };
    
    int nPoints = 22;
    
    int main()
    {
    	//模型文件位置
    	string protoFile = "./model/pose_deploy.prototxt";
    	string weightsFile = "./model/pose_iter_102000.caffemodel";
    
    	// read image 读取图像
    	string imageFile = "./image/hand.jpg";
    	Mat frame = imread(imageFile);
    	if (frame.empty())
    	{
    		cout << "check image" << endl;
    		return 0;
    	}
    	//复制图像
    	Mat frameCopy = frame.clone();
    	//读取图像长宽
    	int frameWidth = frame.cols;
    	int frameHeight = frame.rows;
    
    	float thresh = 0.01;
    
    	//原图宽高比
    	float aspect_ratio = frameWidth / (float)frameHeight;
    	int inHeight = 368;
    	//缩放图像
    	int inWidth = (int(aspect_ratio*inHeight) * 8) / 8;
    
    	cout << "inWidth = " << inWidth << " ; inHeight = " << inHeight << endl;
    
    	double t = (double)cv::getTickCount();
    	//调用caffe模型
    	Net net = readNetFromCaffe(protoFile, weightsFile);
    	Mat inpBlob = blobFromImage(frame, 1.0 / 255, Size(inWidth, inHeight), Scalar(0, 0, 0), false, false);
    	net.setInput(inpBlob);
    	Mat output = net.forward();
    
    	int H = output.size[2];
    	int W = output.size[3];
    
    	// find the position of the body parts 找到各点的位置
    	vector<Point> points(nPoints);
    	for (int n = 0; n < nPoints; n++)
    	{
    		// Probability map of corresponding body's part. 第一个特征点的预测矩阵
    		Mat probMap(H, W, CV_32F, output.ptr(0, n));
    		//放大预测矩阵
    		resize(probMap, probMap, Size(frameWidth, frameHeight));
    
    		Point maxLoc;
    		double prob;
    		//寻找预测矩阵,最大值概率以及最大值的坐标位置
    		minMaxLoc(probMap, 0, &prob, 0, &maxLoc);
    		if (prob > thresh)
    		{
    			//画图
    			circle(frameCopy, cv::Point((int)maxLoc.x, (int)maxLoc.y), 8, Scalar(0, 255, 255), -1);
    			cv::putText(frameCopy, cv::format("%d", n), cv::Point((int)maxLoc.x, (int)maxLoc.y), cv::FONT_HERSHEY_COMPLEX, 1, cv::Scalar(0, 0, 255), 2);
    		}
    		//保存特征点的坐标
    		points[n] = maxLoc;
    	}
    
    	//获取要画的骨架线个数
    	int nPairs = sizeof(POSE_PAIRS) / sizeof(POSE_PAIRS[0]);
    
    	//连接点,画骨架
    	for (int n = 0; n < nPairs; n++)
    	{
    		// lookup 2 connected body/hand parts
    		Point2f partA = points[POSE_PAIRS[n][0]];
    		Point2f partB = points[POSE_PAIRS[n][1]];
    
    		if (partA.x <= 0 || partA.y <= 0 || partB.x <= 0 || partB.y <= 0)
    			continue;
    
    		//画骨条线
    		line(frame, partA, partB, Scalar(0, 255, 255), 8);
    		circle(frame, partA, 8, Scalar(0, 0, 255), -1);
    		circle(frame, partB, 8, Scalar(0, 0, 255), -1);
    	}
    
    	//计算运行时间
    	t = ((double)cv::getTickCount() - t) / cv::getTickFrequency();
    	cout << "Time Taken = " << t << endl;
    	imshow("Output-Keypoints", frameCopy);
    	imshow("Output-Skeleton", frame);
    	imwrite("Output-Skeleton.jpg", frame);
    
    	waitKey();
    
    	return 0;
    }

    python代码:

    from __future__ import division
    import cv2
    import time
    import numpy as np
    
    protoFile = "./model/pose_deploy.prototxt"
    weightsFile = "./model/pose_iter_102000.caffemodel"
    nPoints = 22
    POSE_PAIRS = [ [0,1],[1,2],[2,3],[3,4],[0,5],[5,6],[6,7],[7,8],[0,9],[9,10],[10,11],[11,12],[0,13],[13,14],[14,15],[15,16],[0,17],[17,18],[18,19],[19,20] ]
    net = cv2.dnn.readNetFromCaffe(protoFile, weightsFile)
    
    frame = cv2.imread("./image/hand.jpg")
    frameCopy = np.copy(frame)
    frameWidth = frame.shape[1]
    frameHeight = frame.shape[0]
    aspect_ratio = frameWidth/frameHeight
    
    threshold = 0.1
    
    t = time.time()
    # input image dimensions for the network
    inHeight = 368
    inWidth = int(((aspect_ratio*inHeight)*8)//8)
    inpBlob = cv2.dnn.blobFromImage(frame, 1.0 / 255, (inWidth, inHeight), (0, 0, 0), swapRB=False, crop=False)
    
    net.setInput(inpBlob)
    
    output = net.forward()
    print("time taken by network : {:.3f}".format(time.time() - t))
    
    # Empty list to store the detected keypoints
    points = []
    
    for i in range(nPoints):
        # confidence map of corresponding body's part.
        probMap = output[0, i, :, :]
        probMap = cv2.resize(probMap, (frameWidth, frameHeight))
    
        # Find global maxima of the probMap.
        minVal, prob, minLoc, point = cv2.minMaxLoc(probMap)
    
        if prob > threshold :
            cv2.circle(frameCopy, (int(point[0]), int(point[1])), 8, (0, 255, 255), thickness=-1, lineType=cv2.FILLED)
            cv2.putText(frameCopy, "{}".format(i), (int(point[0]), int(point[1])), cv2.FONT_HERSHEY_SIMPLEX, 1, (0, 0, 255), 2, lineType=cv2.LINE_AA)
    
            # Add the point to the list if the probability is greater than the threshold
            points.append((int(point[0]), int(point[1])))
        else :
            points.append(None)
    
    # Draw Skeleton
    for pair in POSE_PAIRS:
        partA = pair[0]
        partB = pair[1]
    
        if points[partA] and points[partB]:
            cv2.line(frame, points[partA], points[partB], (0, 255, 255), 2)
            cv2.circle(frame, points[partA], 8, (0, 0, 255), thickness=-1, lineType=cv2.FILLED)
            cv2.circle(frame, points[partB], 8, (0, 0, 255), thickness=-1, lineType=cv2.FILLED)
    
    
    cv2.imshow('Output-Keypoints', frameCopy)
    cv2.imshow('Output-Skeleton', frame)
    
    
    cv2.imwrite('Output-Keypoints.jpg', frameCopy)
    cv2.imwrite('Output-Skeleton.jpg', frame)
    
    print("Total time taken : {:.3f}".format(time.time() - t))
    
    cv2.waitKey(0)
    

    4 参考

    手部特征点识别

    https://www.learnopencv.com/hand-keypoint-detection-using-deep-learning-and-opencv/

    其他身体特征点识别,一样的套路

    https://www.learnopencv.com/deep-learning-based-human-pose-estimation-using-opencv-cpp-python/

    展开全文
  • 疯狂购物车-快速集成华为HMS ML Kit 手部关键点检测轻松操控你的购物车 前言 双十一即将来临不知道各位的购物是不是已经塞满了东西呢?小编也想将自己的购物车塞得满满的,奈何钱包不想,于是就只能通过游戏虚拟购物...
  • 原标题:OpenCV手部关键点检测(手势识别)代码示例(欢迎关注“我爱计算机视觉”公众号,一个有价值有深度的公众号~)前几日分享了learnopencv.com博主Satya Mallick发表的关于OpenCV Mask RCNN实例分割的博文(详见:...
  • 原标题:OpenCV手部关键点检测(手势识别)代码示例(欢迎关注“我爱计算机视觉”公众号,一个有价值有深度的公众号~)前几日分享了learnopencv.com博主Satya Mallick发表的关于OpenCV Mask RCNN实例分割的博文(详见:...
  • caffee:https:https://github.com/CMU-Perceptual-Computing-Lab/openpose添加链接描述 这是openpose的原始版本,也是最好用的 keras版本:...同样很流畅,与caffe...
  • (欢迎关注“我爱计算机视觉”公众号,一个有价值有深度的公众号~)前几日分享了learnopencv.com博主Satya ...昨日Satya Mallick又发表了使用OpenCV调用OpenPose工程中的手部关键点检测(hand pose estimation)模型的文...
  • 点击我爱计算机视觉标星,更快获取CVML新技术前几日分享了learnopencv.com博主Satya Mallick发表的关于OpenCV Mask RCNN实例分割的博文(详见:...昨日Satya Mallick又发表了使用OpenCV调用OpenPose工程中的手部关...
  • 昨日Satya Mallick又发表了使用OpenCV调用OpenPose工程中的手部关键点检测(hand pose estimation)模型的文章,对于想要使用手部关键点检测做手势识别、手语识别、抽烟检测等工程开发的朋友来说这是一个非常简单的...
  • 今天就通过一个实例:使用飞桨X2Paddle将Caffe框架训练的OpenPose 手部关键点检测模型的迁移至PaddlePaddle框架上,并实现推理部署,介绍一下如何使用模型转换工具来解决深度学习模型跨框架迁移的问题。 本文包含了...
  • 今天就通过一个实例:使用飞桨X2Paddle将Caffe框架训练的OpenPose 手部关键点检测模型的迁移至PaddlePaddle框架上,并实现推理部署,介绍一下如何使用模型转换工具来解决深度学习模型跨框架迁移的问题。 本文包含了...
  • 今天就通过一个实例:使用飞桨X2Paddle将Caffe框架训练的OpenPose 手部关键点检测模型的迁移至PaddlePaddle框架上,并实现推理部署,介绍一下如何使用模型转换工具来解决深度学习模型跨框架迁移的问题。 本文包含了...
  • 基于深度学习的手部21类关键点检测

    万次阅读 多人点赞 2021-05-03 19:49:12
    基于深度学习的手部21类关键点检测 开发环境 * Python 3.7 * PyTorch >= 1.5.1 * opencv-python 数据源 普通USB彩色(RGB)网络摄像头 已经存储下来的视频或者图片 数据集 本项目数据集百度网盘下载地址:...
  • 目录PaddleHub检测肢体和手部关键点openpose_hands_estimationopenpose_body_estimation一、安装新版Hub二、定义待预测照片并安装模型三、API预测四、命令行预测五、实现效果 PaddleHub检测肢体和手部关键点 ...
  • OpenCV基于OpenPose的手部关键点检测概述✔️ 手部关键点检测,旨在找出给定图片中手指上的关节点及指尖关节点, 其中手部关键点检测的应用场景主要包括:手势识别手语识别与理解手部的行为识别✔️ Opencv的DNN手部...

空空如也

空空如也

1 2 3 4 5
收藏数 83
精华内容 33
关键字:

手部关键点检测