基于 OpenVINO, yolov5 推理

OpenVINO 是英特尔开发的一款功能强大的深度学习工具包，可实现跨多个硬件平台的优化神经网络推理。在本文中，我们讨论了 OpenVINO 的特性和优势，以及它如何与领先的计算机视觉平台 Viso Suite 集成，以构建和交付可扩展的应用程序。

?🇶🇦 什么是 OpenVINO？

OpenVINO 是英特尔开发的跨平台深度学习工具包。该名称代表“开放式视觉推理和神经网络优化”。OpenVINO 专注于通过面向英特尔硬件平台的一次写入、随处部署的方法优化神经网络推理。

该工具包在 Apache License 2.0 版下免费使用，有两个版本：

● OpenVINO 工具包，由开源社区支持
● Intel Distribution of OpenVINO toolkit，由 Intel 支持。

使用 OpenVINO 工具包，软件开发人员可以通过与应用逻辑集成的高级 C++ 推理引擎 API选择模型并部署预训练的深度学习模型（YOLO v3、ResNet 50等）。

因此，OpenVINO 提供集成功能来加快应用程序和解决方案的开发，这些应用程序和解决方案使用计算机视觉、自动语音识别、自然语言处理(NLP)、推荐系统、机器学习等解决多项任务。

---

🚢 安装 OpenVINO

1. 安装地址： https://www.intel.com/content/www/us/en/developer/tools/openvino-toolkit/download.html?VERSION=v_2022_3_1&ENVIRONMENT=RUNTIME&OP_SYSTEM=WINDOWS&DISTRIBUTION=ARCHIVE
2. 这里我下载的是 2022.3.1 Runtime OpenVINO Archives 版本.
3. 然后拷贝到 C:\Program Files (x86)\Intel 下。
4. 使用 python 安装 openvino-env (别的根据需求来安装)，详细安装步骤可参考: https://docs.openvino.ai/2023.2/openvino_docs_install_guides_installing_openvino_from_archive_windows.html

   pip install -r tools\requirements.txt -i https://pypi.tuna.tsinghua.edu.cn/simple
   pip install -r tools\requirements_onnx.txt -i https://pypi.tuna.tsinghua.edu.cn/simple
   

5. 使用 mo -h 来验证是否安装正确。

---

🚀 Export Model

• pt convert to onnx

  python export.py --weights ./weights/yolov5s.pt --img 640 --batch 1 --include onnx
  

• onnx convert to openvino

  # 这里我将归一化参数写入模型中了, 这样前处理可以不用做归一化处理了
  # data_type 32: FP32 16: FP16
  mo --input_model ./weights/yolov5s.onnx --output_dir ./weights --data_type FP16 --mean_values [0,0,0] --scale_values [255,255,255]
  

• pt convert to openvino

  python export.py --weights ./weights/yolov5s.pt --img 640 --batch 1 --include openvino
  

---

🍋? Yolov5

• 获取当前支持的所有的AI硬件推理设备, 如果输出没有设备，则可尝试更新显卡驱动. https://www.intel.com/content/www/us/en/support/products/80939/graphics.html

  #include <openvino/openvino.hpp>
  #include <iostream>
  #include <vector>
  int main() {
      ov::Core ie;
      //获取当前支持的所有的AI硬件推理设备
      std::vector<std::string> devices = ie.get_available_devices();
      for (int i = 0; i < devices.size(); i++) {
          std::cout << devices[i] << std::endl;
      }
      system("pause");
      return 0;
  }
  
  // ----------------------- output -----------------------
  CPU
  GNA
  GPU
  

• 推理

  1. 代码来自这位博主, 详细过程可参考博主博客 https://blog.csdn.net/weixin_45650500/article/details/134455535
  2. 如果感觉初始化时间等待过长，可尝试添加缓存，则第一次加载过后，再次初始化会很快. core.set_property(ov::cache_dir("cache"));

  #include <opencv2/dnn.hpp>
  #include <openvino/openvino.hpp>
  #include <opencv2/opencv.hpp>
  
  using namespace std;
  
  const float SCORE_THRESHOLD = 0.2;
  const float NMS_THRESHOLD = 0.4;
  const float CONFIDENCE_THRESHOLD = 0.4
  
  struct Detection
  {
      int class_id;
      float confidence;
      cv::Rect box;
  };
  struct Resize
  {
      cv::Mat resized_image;
      int dw;
      int dh;
  };
  Resize resize_and_pad(cv::Mat& img, cv::Size new_shape) {
      float width = img.cols;
      float height = img.rows;
      float r = float(new_shape.width / max(width, height));
      int new_unpadW = int(round(width * r));
      int new_unpadH = int(round(height * r));
      Resize resize;
      cv::resize(img, resize.resized_image, cv::Size(new_unpadW, new_unpadH), 0, 0, cv::INTER_AREA);
  
      resize.dw = new_shape.width - new_unpadW;
      resize.dh = new_shape.height - new_unpadH;
      cv::Scalar color = cv::Scalar(100, 100, 100);
      cv::copyMakeBorder(resize.resized_image, resize.resized_image, 0, resize.dh, 0, resize.dw, cv::BORDER_CONSTANT, color);
  
      return resize;
  }
  
  int main() {
  
      // Step 1. Initialize OpenVINO Runtime core
      ov::Core core;
      core.set_property(ov::cache_dir("cache"));
      // Step 2. Read a model
      std::shared_ptr<ov::Model> model = core.read_model("model/yolov5s.xml", "model/yolov5s.bin");
      //此处需要自行修改xml和bin的路径
  
      // Step 3. Read input image
      // 图像路径  
      cv::Mat img = cv::imread("images/bus.jpg");
      // resize image
      Resize res = resize_and_pad(img, cv::Size(640, 640));
  
      // Step 4. Inizialize Preprocessing for the model
      ov::preprocess::PrePostProcessor ppp = ov::preprocess::PrePostProcessor(model);
      // Specify input image format
      ppp.input().tensor().set_element_type(ov::element::u8).set_layout("NHWC").set_color_format(ov::preprocess::ColorFormat::BGR);
      // Specify preprocess pipeline to input image without resizing
      //ppp.input().preprocess().convert_element_type(ov::element::f32).convert_color(ov::preprocess::ColorFormat::RGB).scale({ 255., 255., 255. });
      ppp.input().preprocess().convert_element_type(ov::element::f32).convert_color(ov::preprocess::ColorFormat::RGB);
      //  Specify model's input layout
      ppp.input().model().set_layout("NCHW");
      // Specify output results format
      ppp.output().tensor().set_element_type(ov::element::f32);
      // Embed above steps in the graph
      model = ppp.build();
      ov::CompiledModel compiled_model = core.compile_model(model, "GPU");
  
      // Step 5. Create tensor from image
      float* input_data = (float*)res.resized_image.data;
      ov::Tensor input_tensor = ov::Tensor(compiled_model.input().get_element_type(), compiled_model.input().get_shape(), input_data);
  
      // Step 6. Create an infer request for model inference 
      ov::InferRequest infer_request = compiled_model.create_infer_request();
      infer_request.set_input_tensor(input_tensor);
      //增加计时器统计推理时间
      double start = clock();
      infer_request.infer();
      double end = clock();
      double last = start - end;
      cout << "Detect Time" << last << "ms" << endl;
      
      //Step 7. Retrieve inference results 
      const ov::Tensor& output_tensor = infer_request.get_output_tensor();
      ov::Shape output_shape = output_tensor.get_shape();
      float* detections = output_tensor.data<float>();
  
      // Step 8. Postprocessing including NMS  
      std::vector<cv::Rect> boxes;
      vector<int> class_ids;
      vector<float> confidences;
  
      for (int i = 0; i < output_shape[1]; i++) {
          float* detection = &detections[i * output_shape[2]];
  
          float confidence = detection[4];
          if (confidence >= CONFIDENCE_THRESHOLD) {
              float* classes_scores = &detection[5];
              cv::Mat scores(1, output_shape[2] - 5, CV_32FC1, classes_scores);
              cv::Point class_id;
              double max_class_score;
              cv::minMaxLoc(scores, 0, &max_class_score, 0, &class_id);
  
              if (max_class_score > SCORE_THRESHOLD) {
                  confidences.push_back(confidence);
                  class_ids.push_back(class_id.x);
                  float x = detection[0];
                  float y = detection[1];
                  float w = detection[2];
                  float h = detection[3];
                  float xmin = x - (w / 2);
                  float ymin = y - (h / 2);
                  boxes.push_back(cv::Rect(xmin, ymin, w, h));
              }
          }
      }
      std::vector<int> nms_result;
      cv::dnn::NMSBoxes(boxes, confidences, SCORE_THRESHOLD, NMS_THRESHOLD, nms_result);
      std::vector<Detection> output;
      for (int i = 0; i < nms_result.size(); i++)
      {
          Detection result;
          int idx = nms_result[i];
          result.class_id = class_ids[idx];
          result.confidence = confidences[idx];
          result.box = boxes[idx];
          output.push_back(result);
      }
  
      // Step 9. Print results and save Figure with detections
      for (int i = 0; i < output.size(); i++)
      {
          auto detection = output[i];
          auto box = detection.box;
          auto classId = detection.class_id;
          auto confidence = detection.confidence;
          float rx = (float)img.cols / (float)(res.resized_image.cols - res.dw);
          float ry = (float)img.rows / (float)(res.resized_image.rows - res.dh);
          box.x = rx * box.x;
          box.y = ry * box.y;
          box.width = rx * box.width;
          box.height = ry * box.height;
          cout << "Bbox" << i + 1 << ": Class: " << classId << " "
              << "Confidence: " << confidence << " Scaled coords: [ "
              << "cx: " << (float)(box.x + (box.width / 2)) / img.cols << ", "
              << "cy: " << (float)(box.y + (box.height / 2)) / img.rows << ", "
              << "w: " << (float)box.width / img.cols << ", "
              << "h: " << (float)box.height / img.rows << " ]" << endl;
          float xmax = box.x + box.width;
          float ymax = box.y + box.height;
          cv::rectangle(img, cv::Point(box.x, box.y), cv::Point(xmax, ymax), cv::Scalar(0, 255, 0), 3);
          cv::rectangle(img, cv::Point(box.x, box.y - 20), cv::Point(xmax, box.y), cv::Scalar(0, 255, 0), cv::FILLED);
          cv::putText(img, std::to_string(classId), cv::Point(box.x, box.y - 5), cv::FONT_HERSHEY_SIMPLEX, 0.5, cv::Scalar(0, 0, 0));
      }
      //显示具体结果
      cv::namedWindow("ImageWindow", cv::WINDOW_NORMAL);
      cv::resizeWindow("ImageWindow", 800, 600);
      cv::imshow("ImageWindow", img);
  
      cv::waitKey(0);
      cv::destroyAllWindows();
      return 0;
  }
  

• 输出结果
  

---

参考

• https://www.intel.com/content/www/us/en/developer/tools/openvino-toolkit/download.html?VERSION=v_2023_2_0&OP_SYSTEM=WINDOWS&DISTRIBUTION=ARCHIVE
• https://docs.openvino.ai/2023.2/openvino_docs_install_guides_installing_openvino_from_archive_windows.html
• https://blog.csdn.net/weixin_45650500/article/details/134455535