Getting Started with VisionFlow#

We have explained how to add VisionFlow as a dependency in your project in Installing . Now, we will quickly demonstrate the usage of the main functional interfaces in VisionFlow through a simple example.

This example will include a complete workflow from creating a project, adding tools, adding data and annotations, training the model, to deploying the model. If you are not interested in how to create a project and train a model through the VisionFlow interface, but only want to know how to deploy a pre-trained model exported as a model file, then you can start reading from the Loading Exported Model section.

Initializing#

The VisionFlow library has some global settings that need to be specified at runtime and must be set before calling any other interfaces. These settings include:

Logging output settings: By setting the logging output options, you can customize how VisionFlow’s logs are output, such as specifying a file path, a callback function, standard output/terminal, or MSVC debugger.
Language preference: VisionFlow supports multiple languages. By default, the language preference is automatically set to match your operating system’s language during initialization. If you wish to use a different language, you can specify it through initialization parameters.

Here is an example of initializing VisionFlow:

C++

#include <iostream>
#include "visionflow/visionflow.hpp"

namespace vflow = visionflow;

void my_logger_output(int log_level, const char *content, size_t len) {
    if (log_level > 2) {
        std::cout << std::string(content, len) << std::endl;
    }
}

int main(int /*argc*/, char ** /*argv*/) try {

    vflow::InitOptions opts;

    // Set the log output file
    opts.logger.file_sink = "visionflow.log";
    // Set whether to output the logs to the standard output terminal
    opts.logger.stdout_sink = true;
    // You can customize the handling of VisionFlow's log output by setting a log output callback function
    // opts.logger.func_sink = my_logger_output;

    // Set the language to Chinese
    opts.language = "zh_CN";

    vflow::initialize(opts);

    return 0;
}

Python

To be completed

To be completed

Creating Project#

Project is the fundamental unit in VisionFlow for managing data and processing workflows. We must first create a Project before proceeding with any subsequent steps.

C++

vflow::ProjectDescriptor desc;

desc.workspace_token = "D:/the/path/to/workspace";
desc.project_name = "my_first_project";

auto out_desc = vflow::Project::Create(desc);

std::cout << "My first VisionFlow project created at: "
          << out_desc.created_time << std::endl;
std::cout << "The VisionFlow version which the project created by: "
          << out_desc.sdk_version << std::endl;

// And then you can open the project with the desc or the output_desc
auto project = vflow::Project::Open(desc);

std::cout << "My first VisionFlow project name is: "
          << project->descriptor().project_name << std::endl;

Python

To be completed

To be completed

Adding Tools#

VisionFlow provides a variety of tools that you can freely combine and connect to accomplish different processing workflows. You can find all the tools provided by VisionFlow in the Tools Usage and Graph.

In this example, we will add an Input tool and a Segmentation tool, and we connect the Segmentation tool to the output of the Input tool:

C++

std::string input_id = project->add_tool("Input");
std::string segmentation_id = project->add_tool("Segmentation");

bool connected = project->auto_connect(input_id, segmentation_id);
if (!connected) {
    std::cerr << "auto_connect failed, Please use Project::connect instead."
              << std::endl;
}

Python

input_id = project.add_tool("Input")
segmentation_id = project.add_tool("Segmentation")

connected = project.auto_connect(input_id, segmentation_id)
if not connected:
    print("auto_connect failed, Please use Project::connect instead.")

To be completed

Importing Images#

In VisionFlow, we use SampleSet to manage the data within a project. All management of Sample data within the Project is done through SampleSet or PropertySet. A project can have multiple SampleSet, and typically, it includes a main dataset used for training and validating models. However, you can also create additional SampleSet as needed. Adding image data to a project means adding the image data to a dataset within the project. Here is a example to get the SampleSet or PropertySet from the project:

C++

auto sample_set = project->main_sample_set();
auto input_image_set = sample_set.property_set({input_id, "image"});

// Or you can get the read only sample set and read only property set:
// auto ro_sample_set = project->readonly_main_sample_set();
// auto ro_input_image_set = sample_set.readonly_property_set({input_id, "image"});

Python

To be completed

To be completed

There are some different way to import image into the project. If you want to import a series of image files into the project, you can use the visionflow::helper::InputHelper utility to accomplish this. Here’s an example:

C++

vflow::helper::InputHelper input_helper(project.get());

for (int i = 0; i < 100; ++i) {
    input_helper.add_image("D:/path/to/image/" + std::to_string(i) + ".png");
}
// Import and commit the images
auto receipts = input_helper.commit();

// Some image may be improted failed, you can get the reason from the receipts:
size_t success_count = 0;
for (const auto &receipt : receipts) {
    if (receipt.is_success) {
        std::cout << "Failed to import image " << receipt.image_files[0]
                  << "as: " << receipt.error_message << std::endl;
    }else {
        success_count ++;
    }
}

// And then you can get the sample from the sample set:
assert(sample_set.size() == success_count);

Python

To be completed

To be completed

In some scenarios, your image may not exist in the file system but is already loaded into memory. In such cases, you can use the function visionflow::helper::add_image_to_sample() to quickly add the image into a sample set:

C++

// We have a image loaded into memory:
auto image = vflow::Image::FromFile("D:/path/to/image/0.png");

// Create a sample template from the sample set:
auto sample = sample_set.create_empty_sample();
// Add the the image into sample:
vflow::helper::add_image_to_sample(sample, image, input_id, true);
// Add the sample to sample set:
sample_set.add(sample);

Python

To be completed

To be completed

Note

The documentation of the visionflow::helper::add_image_to_sample() The function provides a detailed explanation of its function, which can help you understand how it is implemented in detail. If you want to gain a deeper understanding of VisionFlow’s data management mechanism, it is strongly recommended to carefully read the documentation.

Adding Label#

Before introducing how to add annotations to a dataset, let’s briefly explain how to read and write data in PropertySet. A PropertySet is a data container that can be iteratively accessed. Each PropertySet corresponds to a data node in the processing workflow and contains the data of all samples in your dataset for that node. Taking the example of the image property set we added upon, we can access the data within it as follows (You can refer to the documentation of visionflow::data::PropertySet for more information about the interfaces related to PropertySet.):

C++

// 获取属性集
auto image_set = sample_set.property_set({input_id, "image"});
// 遍历属性集中的数据
for (const auto &[sample_id, image_prop] : image_set) {
    if (image_prop) { // 如果某个样本在此数据节点上的数据不存在，则会返回空指针.
        image_prop->as<vflow::props::Image>().image().show(10);
    } else {
        auto modify_image = vflow::Image::FromFile("D:/path/to/image/0.png");
        vflow::props::Image new_prop(modify_image);
        // 更新属性集中的数据
        image_set.update(sample_id, new_prop);
    }
}

Python

To be completed

To be completed

Through this way, we can add all the views generated during image import to the training set (This is necessary for subsequent training, as only views added to the training set will participate in the training process.) :

C++

auto views_set = sample_set.property_set({input_id, "views"});

for (auto [sample_id, views_prop] : views_set) {
    if (views_prop) {
        for (auto &[view_id, view] : views_prop->as<vflow::props::ViewList>()) {
            view.set_split_tag(vflow::kTrain);
        }
        views_set.update(sample_id, *views_prop);
    }
}

Python

To be completed

To be completed

After understanding the functionality and usage of attribute sets, adding annotations is essentially updating our annotation data to the property set corresponding to the annotation data node:

C++

auto label_set = sample_set.property_set({segmentation_id, "truth"});

for (const auto &[sample_id, old_label] : label_set) {
    // 你可以使用你自定义的函数替换下面的流程，例如：
    // auto new_label = show_and_modify_label(image_prop, old_label);
    if (old_label) {
        auto new_label = old_label->as<vflow::props::PolygonRegionList>();
        for (auto &[region_id, region] : new_label) {
            region.set_name("MyClass");
        }
        label_set.update(sample_id, new_label);
    }
}

Python

To be completed

To be completed

Setting Training Parameters#

Different tools require different sets of parameters at different stages. For the segmentation tool in our example, before training, we need to set the following parameter groups:

C++

// 设置标注的类别清单
vflow::param::LabelClasses classes;
classes.add("MyClass");
project->set_param({segmentation_id, "classes"}, classes);

// 设置用于训练的图像颜色
vflow::param::BaseColor color;
color.set_color(vflow::param::kGray);
project->set_param({segmentation_id, "base_color"}, color);

// 设置训练参数
vflow::param::SegmentationTrainingParameters train_param;
train_param.set_epoch(10)
    .get_augmentations()
    .get_geometry_augmentation()
    .set_flip_horizontal(true)
    .set_flip_vertical(true);
project->set_param({segmentation_id, "trainer.args"}, train_param);

Python

To be completed

To be completed

Training and Configuration Parameters#

Once all the data is prepared, training a model is very simple:

C++

// 创建训练器的策略，你可以通过阅读有关类型的详细接口文档
// 详细了解这些参数的作用
vflow::runtime::StrategyOptions strategy;
strategy.allow_auto_update = true;
strategy.allow_auto_update_rely_on_prop = true;
strategy.ignore_update_time_requirement = true;
// 你可以通过设置自定义的回调函数来接收训练进度信息
strategy.call_back = nullptr;

// 创建训练执行器
auto trainer = project->create_config_runtime({segmentation_id, "trainer"}, strategy);

// 为训练执行器创建数据服务
auto data_server = vflow::adapt(project.get(), trainer);

// 初始化和执行训练，训练成功后模型会自动保存到工程中
trainer.initialize(data_server);
trainer.execute(data_server);

Python

To be completed

To be completed

Exporting the Model#

After training the model and configuring the parameters, you may want to deploy your model to other hosts. One way to do this is by directly copying the project to another host and using the copied project for deploying your detection workflow. However, since the project contains all the data used for training and validating the model, its size can be quite large, making direct copying inconvenient. To facilitate deployment, you can export all the models and configured parameters into a standalone file as shown below:

C++

project->export_model("D:/path/to/save/my_first_project.vfmodel");

Python

project.export_model("D:/path/to/save/my_first_project.vfmodel")

To be completed

If everything is successful, you should now be able to find the my_first_project.vfmodel file in the D:/path/to/save/ directory. You can copy this file to the machine where you need to deploy the detection workflow and proceed with the following deployment steps.

Loading Exported Model#

Before loading the exported model, make sure that the VisionFlow dependency libraries have been initialized. For the detailed process of initializing the dependency libraries, please refer to the Initializing section.

After that, you can open your exported model as shown in the code below:

C++

vflow::Model model("D:/path/to/my_first_project.vfmodel");

Python

model = vflow.Model("D:/path/to/my_first_project.vfmodel")

To be completed

Setting Inference Parameters#

We provide some convenient interfaces that allow you to read and modify certain parameters within the model during the deployment phase. Although in practical scenarios, the parameters that can be modified during the deployment phase are limited, we still allow the modification of any parameter within the model through the interface. After opening the model, you can modify the parameters by reading them from the model, making the necessary changes, and then saving them back to the model as shown below:

C++

// 根据你的工程中的工具的名称读取相应工具中参数
std::string segmentation_id = "Segmentation";
auto filter_param = model.get_param({segmentation_id, "filter.args"});
if (!filter_param) {
    std::cerr << "Filter parameter for Segmentation not exist." << std::endl;
    exit(-1);
}

// 在这里，我们修改了类别名为"MyClass"的缺陷的过滤参数.
filter_param->as<vflow::param::PolygonsFilterParameters>()
    .get_class_thresholds("MyClass")
    .set_enable(true)
    .set_area_range({100, 50000});

// 然后, 你可以将修改后的参数重新保存到模型中
model.set_param({segmentation_id, "filter.args"}, *filter_param);

Python

To be completed

To be completed

Warning

The parameters set using the model.set_param(***) interface are only effective within the currently opened model. After you close this model and reopen it, these modifications will be lost. If you want to permanently save these modifications, you need to create a backup of the opened model as shown in the code below:

C++

model.resave_to("D:/other/path/model_2.vfmodel");

Python

model.resave_to("D:/other/path/model_2.vfmodel")

To be completed

Inference#

Before starting the inference, we need to create a runtime that can execute based on the model data. There are various strategies for creating the runtime, but here, for simplicity, we will directly use the default parameters to run all the tools in the model:

C++

vflow::runtime::AllTools strategy;
// 若你的模型中存在前面的模型或参数晚于后面的模型或参数更新的情况
// 而你又确信这不是一种错误那么使用此选项忽略这些问题
strategy.options.ignore_update_time_requirement = true;
// 如果的相机采集图像的流程没有注册到VisionFlow中，
// 那么你的流程图中对应的节点是虚拟节点，需启用此选项
strategy.options.allow_unrealized_oper = true;
auto runtime = model.create_runtime(strategy);

Python

To be completed

To be completed

After creating the runtime, we can use it to detect the images we want to detect. The runtime requires a Sample to store the input images, intermediate results, and the final output of the entire detection process. Therefore, we need to create a Sample and add our image to it as shown below, and then we can execute our vision processing flow on this sample:

C++

std::string input_id = "Input"; // The id of the Input tool

auto sample = runtime.create_sample();
auto image = vflow::Image::FromFile("D:/path/to/image.png");
vflow::helper::add_image_to_sample(sample, image, input_id);

// Then we can execute the processing flow on the sample
runtime.execute(sample);

Python

To be completed

To be completed

After execution, we can retrieve the detection results from the sample for each tool and process these results according to our requirements(The ID of each intermediate or final result can be obtained by referring to the detailed workflow diagram of each tool.):

C++

auto segment_pred = sample.get({segmentation_id, "pred"});
std::cout << "Result: " << segment_pred->to_json().to_string() << std::endl;

const auto &segment_pred = result->as<vflow::props::PolygonRegionList>();
for (const auto &[id, region] : segment_pred) {
    std::cout << "Found a defect: " << id
              << ", name: " << region.name()
              << ", area: " << region.area() << std::endl;
}

// Draw the defects on the image and show the image:
vflow::img::draw(image, segment_pred.to_multi_polygons(), {10, 10, 240}, 2);
image.show();

Python

To be completed

To be completed

Full Example#

You can get the full online inference example below:

C++

Online Inference Usage#

#include <exception>
#include <iostream>
#include <string>

#include "visionflow/visionflow.hpp"

namespace vflow = visionflow;

void my_logger_output(int log_level, const char *content, size_t len) {
  if (log_level > 2) {
    std::cout << std::string(content, len) << std::endl;
  }
}

int main(int /*argc*/, char ** /*argv*/) try {

  vflow::InitOptions opts;

  // 设置日志输出文件
  opts.logger.file_sink = "visionflow.log";
  // 设置是否将日志输出到标准输出终端
  opts.logger.stdout_sink = true;
  // 你可以通过设置日志输出的回调函数自定义处理VisionFlow输出的日志
  // opts.logger.func_sink = my_logger_output;

  // 设置语言为中文
  opts.language = "zh_CN";

  vflow::initialize(opts);

  vflow::Model model("D:/path/to/save/my_first_project.vfmodel");

  // 根据你的工程中的工具的名称读取相应工具中参数
  std::string segmentation_id = "Segmentation";
  auto filter_param = model.get_param({segmentation_id, "filter.args"});
  if (!filter_param) {
    std::cerr << "Filter parameter for Segmentation not exist." << std::endl;
    exit(-1);
  }

  // 在这里，我们修改了类别名为"MyClass"的缺陷的过滤参数.
  filter_param->as<vflow::param::PolygonsFilterParameters>()
      .get_class_thresholds("MyClass")
      .set_enable(true)
      .set_area_range({0, 500000});

  // 然后, 你可以将修改后的参数重新保存到模型中
  model.set_param({segmentation_id, "filter.args"}, *filter_param);

  model.resave_to("D:/other/path/model_2.vfmodel");

  // 在开始推理之前，我们需要根据模型数据创建出一个可以执行的运行时，创建运行时有很多不同的策略，
  // 在这里，为了简单起见，我们直接使用默认参数运行模型中所有的工具：
  vflow::runtime::AllTools strategy;
  // 若你的模型中存在前面的模型或参数晚于后面的模型或参数更新的情况
  // 而你又确信这不是一种错误那么使用此选项忽略这些问题
  strategy.options.ignore_update_time_requirement = true;
  // 如果的相机采集图像的流程没有注册到VisionFlow中，
  // 那么你的流程图中对应的节点是虚拟节点，需启用此选项
  strategy.options.allow_unrealized_oper = true;
  auto runtime = model.create_runtime(strategy);

  // 在创建出运行时之后，我们就可以使用运行时来检测我们需要检测的图像了。
  // 运行时需要一个样本来保存整个检测流程的输入图像，中间结果和最终的
  // 输出，因此，我们需要像下面这样，先创建出一个样本，并将我们的图像添加到样本中：
  std::string input_id = "Input";
  while (true) {
    auto sample = runtime.create_sample();
    auto image = vflow::Image::FromFile("D:/path/to/image.png");
    vflow::helper::add_image_to_sample(sample, image, input_id);
    runtime.execute(sample);

    auto result = sample.get({segmentation_id, "pred"});

    std::cout << "Result: " << result->to_json().to_string() << std::endl;

    const auto &segment_pred = result->as<vflow::props::PolygonRegionList>();
    for (const auto &[id, region] : segment_pred) {
      std::cout << "Found a defect: " << id << ", name: " << region.name()
                << ", area: " << region.area() << std::endl;
    }

    // Draw the defects on the image and show the image:
    vflow::img::draw(image, segment_pred.to_multi_polygons(), {10, 10, 240}, 2);
    image.show();
  }

  return 0;

} catch (const std::exception &ex) {
  std::cerr << "Unexpected Escaped Exception: " << ex.what();
  return -1;
}

Python

To be completed

To be completed