diff --git a/README.md b/README.md index 9eefabb..45b7e9f 100644 --- a/README.md +++ b/README.md @@ -1,40 +1,45 @@ -# Plumerai People Detection OpenCV pipeline example +# Plumerai Video Intelligence OpenCV pipeline example -This is the source code for a simple version of using the Plumerai People Detection library in an end-to-end demo setting. It runs on Linux, uses OpenCV and V4L, and retrieves input data from a webcam (or optionally an RTSP stream) and writes box coordinates to console and to an output video stream displayed on screen. It can be modified as needed for proto-typing with the Plumerai People Detection library. +This is the source code for a simple version of using the Plumerai Video Intelligence library in an end-to-end demo setting. It runs on Linux, uses OpenCV and V4L, and retrieves input data from a webcam (or optionally an RTSP stream) and writes box coordinates to console and to an output video stream displayed on screen. It can be modified as needed for proto-typing with the Plumerai Video Intelligence library. -The Plumerai People Detection library itself is not included: this repository can't be used without access to the library. If you do not have access to the library and would like to evaluate it, then contact us at [plumerai.com/contact_us](https://plumerai.com/contact_us). For more information, see [plumerai.com/people-detection](https://plumerai.com/people-detection). +The Plumerai Video Intelligence library itself is not included: this repository can't be used without access to the library. If you do not have access to the library and would like to evaluate it, then contact us at [plumerai.com/contact_us](https://plumerai.com/contact_us). For more information, see [plumerai.com/people-detection](https://plumerai.com/people-detection). ## Linux system requirements The example application uses [OpenCV](https://opencv.org/). Using the `CAP_V4L` OpenCV API, it connects to the webcam with [V4L (video 4 linux)](https://en.wikipedia.org/wiki/Video4Linux). Most (USB) webcams compatible with V4L should work with this example demo application, as long as they support the Motion-JPEG (MJPEG) input compression format. If you have V4L installed on your system (see below), this can be verified by running `v4l2-ctl --list-formats-ext`. On a Debian-based system (e.g. Ubuntu) the pre-requisites (OpenCV with V4L) can be installed as follows: + ```bash apt install libopencv-dev libopencv-highgui-dev libopencv-videoio-dev libv4l-dev ``` + In case of issues or for other Linux systems, see the [official OpenCV release page](https://opencv.org/releases/). ## Configuring the camera settings First, make sure the camera settings (`camera_height`, `camera_width`, and `camera_id`) are set correctly near the top of the `src/opencv_example.cc` file. -The higher the input resolution, the better the results can become. However, note that this might slow down the entire example application, because the camera capture and video-displaying might take up more resources. The framerate reported is purely for the Plumerai People Detection algorithm itself, and does not count camera capture or displaying of the results. +The higher the input resolution, the better the results can become. However, note that this might slow down the entire example application, because the camera capture and video-displaying might take up more resources. The framerate reported is purely for the Plumerai Video Intelligence algorithm itself, and does not count camera capture or displaying of the results. -Note that the example application itself (the camera capture / video decoding and displaying) is not optimized for speed. Furthermore, the people detection algorithm in its current form is not optimized for speed on x86 systems, only for targets such as Arm Cortex-A. +Note that the example application itself (the camera capture / video decoding and displaying) is not optimized for speed. Furthermore, the video intelligence algorithm in its current form is not optimized for speed on x86 systems, only for targets such as Arm Cortex-A. ## Optional: use an RTSP stream as input -This repository also contains example code to use an RTSP video stream instead of camera data as input. The code changes are minimal, and can be enabled by uncommenting the `#define USE_RTSP_INPUT` macro near the top of the `src/opencv_example.cc` file. Then, a little bit below the RTSP stream settings (`camera_height`, `camera_width`, and `rtsp_url`) can be changed as needed. +This repository also contains example code to use an RTSP video stream instead of camera data as input. The code changes are minimal, and can be enabled by uncommenting the `#define USE_RTSP_INPUT` macro near the top of the `src/opencv_example.cc` file. Then, a little bit below the RTSP stream settings (`camera_height`, `camera_width`, and `rtsp_url`) can be changed as needed. ## Compiling the example application -The OpenCV example can be compiled with a C++11 compiler linking to both OpenCV4 and the Plumerai People Detection library, e.g.: +The OpenCV example can be compiled with a C++11 compiler linking to both OpenCV4 and the Plumerai Video Intelligence library, e.g.: + ```bash -g++ -std=c++11 -O3 src/opencv_example.cc -I/path/to/plumeraipeopledetection/include/ /path/to/plumeraipeopledetection/lib//libplumeraipeopledetection.a `pkg-config --cflags --libs opencv4` -ldl -pthread -o opencv_example +g++ -std=c++11 -O3 src/opencv_example.cc -I/path/to/plumeraivideointelligence/include/ /path/to/plumeraivideointelligence/lib//libplumeraivideointelligence.a `pkg-config --cflags --libs opencv4` -ldl -pthread -o opencv_example ``` + Here, `` above refers to the platform you are on, e.g. `x86_64` or `aarch64`. In case of an older OpenCV version, remove the `4` in the `opencv4` part above. The example app can then be executed without any command-line arguments, e.g.: + ``` ./opencv_example ``` diff --git a/src/opencv_example.cc b/src/opencv_example.cc index e7be110..845e324 100644 --- a/src/opencv_example.cc +++ b/src/opencv_example.cc @@ -1,6 +1,6 @@ -// Copyright (C) 2022 Plumerai Ltd under the Apache 2.0 license. +// Copyright (C) 2024 Plumerai Ltd under the Apache 2.0 license. // -// This code implements a simple demo around the Plumerai People Detection +// This code implements a simple demo around the Plumerai Video Intelligence // library using OpenCV to capture webcam input and display to screen. You can // modify the code as needed for your prototyping use-case, it might not work // out-of-the-box depending on your system and camera used. See the README for @@ -16,14 +16,16 @@ #include "opencv2/highgui/highgui.hpp" #include "opencv2/videoio.hpp" -// The Plumerai People Detection library. If you do not have access to this +// The Plumerai Video Intelligence library. If you do not have access to this // library and would like to evaluate it, then contact us via // https://plumerai.com/contact_us. For more information, see our website // https://plumerai.com/people-detection, and docs.plumerai.com for information // how to use it. -#include "plumerai/people_detection.h" +#include "plumerai/video_intelligence.h" -// Enable this marco to switch the input of this example application from +using namespace plumerai; + +// Enable this macro to switch the input of this example application from // camera input to RTSP stream: // #define USE_RTSP_INPUT @@ -61,11 +63,11 @@ const int colours[num_colours][3] = { {128, 128, 0}, {255, 215, 180}, {0, 0, 128}}; int main() { - // Set-up the Plumerai People Detection algorithm - auto ppd = plumerai::PeopleDetection(height, width); + // Set-up the Plumerai Video Intelligence algorithm + auto pvi = plumerai::VideoIntelligence(height, width); // Set-up the output window - const auto window_text = "Plumerai People Detection"; + const auto window_text = "Plumerai Video Intelligence"; cv::namedWindow(window_text, cv::WINDOW_AUTOSIZE); // Set-up the camera, see the above constants for the settings @@ -94,14 +96,20 @@ int main() { return -1; } - // Process the frame using the Plumerai People Detection library. This + // Process the frame using the Plumerai Video Intelligence library. This // function is timed and framerate in FPS is reported. Note that the - // Plumerai People Detection library is not optimized for all systems, + // Plumerai Video Intelligence library is not optimized for all systems, // see the README for more details. auto start_time = std::chrono::steady_clock::now(); - auto predictions = ppd.process_frame(cv_image.data); + auto error_code = pvi.process_frame( + ImagePointer(cv_image.data)); auto elapsed_time = std::chrono::steady_clock::now() - start_time; + if (error_code != ErrorCode::SUCCESS) { + printf("Error: Could not process the frame.\n"); + return -1; + } + // Report the time and framerate the above function took auto time_ms = std::chrono::duration(elapsed_time).count(); @@ -109,12 +117,14 @@ int main() { printf("Processing took %.1f ms (%d fps)\n", time_ms, fps); // Process the resulting bounding-boxes if there are any + std::vector predictions; + error_code = pvi.object_detection().get_detections(predictions); + if (error_code != ErrorCode::SUCCESS) { + printf("Error: Could not get detections.\n"); + return -1; + } for (auto &p : predictions) { - // Filter out low-confidence predictions, adjust the threshold as needed - if (p.confidence < 0.5f) { - continue; - } - + // Convert the relative bounding-box coordinates to the image size auto x_min = std::max(0, std::min(static_cast(p.x_min * width), width - 1)); auto y_min = diff --git a/testing/fake_libplumerai.cc b/testing/fake_libplumerai.cc index 759a852..f3b54c9 100644 --- a/testing/fake_libplumerai.cc +++ b/testing/fake_libplumerai.cc @@ -1,20 +1,20 @@ -// Copyright (C) 2022 Plumerai Ltd under the Apache 2.0 license. +// Copyright (C) 2024 Plumerai Ltd under the Apache 2.0 license. // -// Implements a fake version of the Plumerai People Detection library such that -// we can test linking and running the application without needing access to the -// actual library. This file does not implement the regular Plumerai People -// Detection library header, do not use it outside of (automated) testing. -#include "plumerai/people_detection.h" +// Implements a fake version of the Plumerai Video Intelligence library such +// that we can test linking and running the application without needing access +// to the actual library. This file does not implement the regular Plumerai +// Video Intelligence library header, do not use it outside of (automated) +// testing. +#include "plumerai/video_intelligence.h" namespace plumerai { -PeopleDetection::PeopleDetection(int height, int width) {} -PeopleDetection::~PeopleDetection() {} +VideoIntelligence::VideoIntelligence(int height, int width) {} +VideoIntelligence::~VideoIntelligence() {} -std::vector PeopleDetection::process_frame( - const std::uint8_t *image_data, float delta_t) { - // This does nothing: it returns an empty vector - return std::vector(); +ErrorCode ObjectDetection::get_detections(std::vector& results) { + results.clear(); + return ErrorCode::SUCCESS; } } // namespace plumerai diff --git a/testing/plumerai/people_detection.h b/testing/plumerai/people_detection.h deleted file mode 100644 index 9bf99af..0000000 --- a/testing/plumerai/people_detection.h +++ /dev/null @@ -1,33 +0,0 @@ -// Copyright (C) 2022 Plumerai Ltd under the Apache 2.0 license. -// -// Implements a fake version of the Plumerai People Detection header such that -// we can test linking and running the application without needing access to the -// actual library. This file is not the regular Plumerai People Detection -// header, do not use it outside of (automated) testing. -#pragma once - -#include -#include - -typedef struct BoxPrediction { - float y_min; - float x_min; - float y_max; - float x_max; - float confidence; - unsigned int id; - unsigned int class_id; -} BoxPrediction; - -namespace plumerai { - -class PeopleDetection { - public: - PeopleDetection(int height, int width); - ~PeopleDetection(); - - std::vector process_frame(const std::uint8_t *image_data, - float delta_t = 0.f); -}; - -} // namespace plumerai diff --git a/testing/plumerai/video_intelligence.h b/testing/plumerai/video_intelligence.h new file mode 100644 index 0000000..51d6b42 --- /dev/null +++ b/testing/plumerai/video_intelligence.h @@ -0,0 +1,52 @@ +// Copyright (C) 2024 Plumerai Ltd under the Apache 2.0 license. +// +// Implements a fake version of the Plumerai Video Intelligence header such that +// we can test linking and running the application without needing access to the +// actual library. This file is not the regular Plumerai Video Intelligence +// header, do not use it outside of (automated) testing. +#pragma once + +#include +#include + +typedef struct BoxPrediction { + float y_min; + float x_min; + float y_max; + float x_max; + float confidence; + unsigned int id; + unsigned int class_id; +} BoxPrediction; + +namespace plumerai { + +enum class ErrorCode { SUCCESS = 0 }; + +enum class ImageFormat { PACKED_RGB888 }; + +template +struct ImagePointer { + ImagePointer(const std::uint8_t* _data) {} +}; + +class ObjectDetection { + public: + ErrorCode get_detections(std::vector& results); +}; + +class VideoIntelligence { + public: + VideoIntelligence(int height, int width); + ~VideoIntelligence(); + + template + ErrorCode process_frame(const ImagePointer image_data, + float delta_t = 0.f) { + return ErrorCode::SUCCESS; + } + + ObjectDetection object_detection() const { return ObjectDetection(); } +}; + +} // namespace plumerai