# ViDiELClassify3D

The ViDiELClassify3D function Functions are tools that are available in Spreadsheet for processing and analyzing acquisitions or other results. You can add functions to your Spreadsheet job to create tool chains and produce results for specific applications. automatically classifies point cloud A point cloud is a 3D structure that consists of the data points collected by a 3D vision system during acquisition. projection images into the defined classes based on an initial training image set. To use this function, you must define the classes, load an initial set of point clouds into the function, and then individually label a subset of the training images to ensure that the function recognizes the features on the projection images and classifies them correctly.

The following example shows how the function classifies objects in a point cloud. The input image contains a point cloud of croissants arranged on a tray. The output image shows the croissants classified into different classes based on their features.

Input Point Cloud | Output Point Cloud

## Getting Started

For a visual overview of the setup and training process, watch the video on using the VidiELClassify3D function on the Cognex Support Youtube channel.

## Using VidiELClassify3D

To use the ViDiELClassify3D function:

1. Drag and drop the ViDELClassify3D function into the Spreadsheet.

   The property sheet pops up:

2. Set up the Region of Interest (ROI).

   Use any of the following methods:

   - Double-click the Box3D parameter to set up the ROI in the acquisition Acquisition is the process or result of the vision system acquiring a new image. display.

   - Expand the Box3D parameter and define the box parameters manually.

   - Double-click the External Box parameter and select a Box3D function cell in Spreadsheet.

   - If the part tends to move in the scanning region, set up a fixture A fixture is a coordinate location on the image that keeps the tool in the place determined by the fixture. using the Fixture3D parameter.

3. In the General tab, set the Projection Image A projection image is the 2D projection of a 3D point cloud acquisition. Resolution in a way that makes the feature of interest easy to see. The feature of interest is the defect or any feature that you want the function to classify.

4. In the Projection tab of the property sheet, select the Projection Mode, Feature Width, and Feature Height that you want the function to use for training. For more information, see Projection Parameters.

   Note: Click OK to apply the changes. This closes the property sheet. Open the property sheet again to see the training images with the updated projection mode in the Train and Images tabs of the property sheet.

5. In the Train tab of the property sheet, add the classes you want the function to use.

6. Click OK to close the property sheet.

7. In the Spreadsheet, check the checkbox in the Collect Samples cell of the function. This allows the function to load images for training.

8. Load images into the function either by double-clicking the desired images in the Filmstrip, or by clicking the Play button in the Filmstrip controls to load all of the images in the Filmstrip.
9. Once you have loaded the training images, uncheck the checkbox in the Collect Samples cell of the function.

10. Double-click the ViDiELClassify3D cell in the Spreadsheet to re-open the property sheet.

11. Label at least five images for each class.

    Use any of the following methods:

    - In the Train tab, label the images one-by-one according to the desired class.

    - In the Images tab, label multiple images at the same time:

      1. Press Ctrl and click to select multiple images.

      2. Drag and drop the images into the Labeled Images column of the desired class.

    Note: Once you have started labeling, you cannot change the ROI.
12. Review the Model Health indicator at the top of the table to check the performance of the function.

    - The Model Health indicates how confident the function is in predicting the class of new, unseen images. Labeling images increases the Model Health percentage. Higher percentages indicate greater confidence.

    - When you collect more images from the Filmstrip, the function starts to predict the class of unlabeled images.

    - Continue labeling images until the Model Health is sufficiently high. Aim for 100%, if possible, but it might be difficult if you have highly variable images. You must decide what Model Health is acceptable for your specific application.

      Tip: Add new images to the Filmstrip during labeling to validate how accurately the function predicts classes.

13. Optional: After labeling, click the Auto-Tune button in the Projection tab to automatically select the best projection settings for your application. For more information, see Auto-Tune option in Projection Parameters.

    Note: The selected projection mode is not visible in the property sheet until you close and reopen the property sheet.
14. Click OK to finalize the function settings.

Inputs

Setting | Description
PointCloud A point cloud is a 3D structure that consists of the data points collected by a 3D vision system during acquisition. | Reference to a cell containing a point cloud. The default value is $A$0.
Fixture3D | Specifies the point cloud coordinate system in which the 3D fixture is defined. It is offset from the PointCloud origin in the following format with the following arguments: {X, Y, Z, Rotation, Tilt, TiltDirection}
X | Offset on the X-axis from the PointCloud origin.
Y | Offset on the Y-axis from the PointCloud origin.
Z | Offset on the Z-axis from the PointCloud origin.
Rotation | Specifies the in-plane rotation in degrees around the PointCloud Z-axis. The value range is -180 to 180. The tool normalizes external references with values outside the allowed range to the allowed range.
Tilt | Specifies the tilt angle in degrees from the PointCloud Z-axis after rotation. The value range is 0-180. External references with values outside the allowed range result in an error.
Tilt Direction | Specifies the direction of tilt in degrees from the rotated PointCloud X-axis of the tool, around the PointCloud Z-axis of the tool. The value range is -180 to 180. The tool normalizes external references with values outside the allowed range to the allowed range.
Note: The Fixture3D parameter is only available if you set External Fixture to 0. Note: You can use the Fixture3D parameter to set the ROI fixture.
External Fixture | Specifies a reference to a Fixture3D structure defined in another cell. The function uses the referenced cell as the fixture of the function. Set to 0 (OFF) to use the internal fixture of the function instead.
Box3D | Specifies the dimensions, offset and angle for the 3D rectangle of the function. It is offset from the PointCloud origin in the following format with the following arguments: {SizeX, SizeY, X, Y, Z, Rotation, Tilt, TiltDirection}
SizeX | The size of the box along the X-axis of the PointCloud. The value range is 0.001 to inf., and the default value is 100.
SizeY | The size of the box along the Y-axis of the PointCloud. The value range is 0.001 to inf., and the default value is 100.
SizeZ | The size of the box along the Z-axis of the PointCloud. The value range is 0.001 to inf., and the default value is 100.
X | Offset on the X-axis from the PointCloud origin. The value range is -inf. to inf., and the default value is 0.
Y | Offset on the Y-axis from the PointCloud origin. The value range is -inf. to inf., and the default value is 0.
Z | Offset on the Z-axis from the PointCloud origin. The value range is -inf. to inf., and the default value is 0.
Rotation | Specifies the in-plane rotation of the box (in degrees) about the PointCloud Z-axis of the tool. The value range is -360 to 360. External references with values outside the allowed range will be normalized to the allowed range.
Tilt | Specifies the tilt angle of the box (in degrees) from the PointCloud Z-axis of the tool. The value range is 0 - 180. External references with values outside the allowed range will result in an error.
Tilt Direction | Specifies the tilt direction of the box (in degrees) from the rotated PointCloud X-axis of the tool, about the PointCloud Z-axis of the tool. The value range is -360 to 360. External references with values outside the allowed range will be normalized to the allowed range.
Note: This parameter is available only if External Box is set to 0.
External Box | Specifies a reference to a Box3D structure defined in another cell to be used with the function. Set to 0 (OFF) to use the internal Box3D structure of the function.
Projection Image Resolution | Specifies the resolution of the 2D projection image generated from the 3D point cloud within the ROI. The resolution determines the level of detail in the 2D projection image. This parameter affects tool performance in the following ways: Higher resolution captures finer surface details. The right resolution improves accuracy by better representing relevant features. Higher resolution requires more processing time, and it might be unsuitable for fast-moving production lines. The following settings are available: Low: Generates low-resolution images. Suitable for large, simple features or when speed is more important than detail. Medium: Balances detail and speed. Recommended for most use cases. High: Generates a high-resolution image. Best for detecting small features but can increase processing time. Each setting doubles the image resolution of the previous level. These are fixed resolution levels.
Collect Samples | Enables or disables sample collection for training.
Unclassified Detection | Enables or disables detection of unclassified results. When enabled, flags images that do not meet the confidence threshold for any defined class.
Confidence Threshold | Specifies the minimum confidence score for valid classification. The available range is 0–100. The default value is 70. A higher threshold results in stricter class assignment.
Show | Select the type of graphic to overlay on the image.

Projection Parameters

Projection is the process of converting 3D point cloud data into a 2D projection image. This is necessary because the tool works on 2D images, not raw 3D point cloud data.

The tool uses different projection modes to transform point clouds into projection images. Each mode highlights specific types of features or surface characteristics. Choosing the right projection is important because it determines how features are visualized in the projection image. See the following examples:

- Use Height Map for flat surfaces with height differences.

- Use Shape Change for detecting edges.

- Use Flex Inspect for inspecting small defects on smooth surfaces.

The Feature Width and Feature Height parameters determine the scale of the feature of interest used when generating the projection image. A low value for these parameters makes fine features more visible in the projection image. A high value smooths out fine features and makes coarse features more visible.

These parameters do not apply equally to all projection modes. The following table summarizes which parameters apply to which projection mode:

Projection Mode | Feature Width | Feature Height
Height Map |  |
Automatic Height Map |  |
Shape Change |  |
Vertical Shape Change |  |
Flex Inspect |  |
Curve Inspect |  |
Greyscale |  |

Setting | Description
--- | ---
Projection Mode | Specifies the mode used to create the 2D projection image from the point cloud.
Height Map | Creates three-dimensional height maps. This projection mode collects 3D data points within the ROI, and transforms them into a 2D height data representation, where the assigned height value for each pixel signifies the average distance between the points within the corresponding XY cell and the bottom of the ROI box. The projection mode fixes minor irregularities in the height data and directly maps it to an 8-bit image. In this mapped image, pixel values range from 0 (bottom of the box) to 255 (top of the box). This projection mode uses Feature Width to fill holes of missing pixels when data is missing in the point cloud. If there is no missing data, Feature Width has no effect on the resulting projection image. The following image shows a projection image formed using the Height Map projection mode: Example: Calculating Projection Image Range Values from ROI The following example shows how the function calculates the projection image range based on the point cloud and the ROI:
Element | Description
Blue shape | Point cloud
Black rectangle | ROI box
255 | Maximum pixel value, top of the ROI box
0 | Minimum pixel value, bottom of the ROI box
Automatic Height Map | Creates three-dimensional height maps, automatically selecting the height range. This projection mode collects 3D data points within the ROI, and transforms them into a 2D height data representation, similar to the Height Map projection mode. This mode automatically sets the height range for optimal results. This projection mode uses Feature Width to fill holes of missing pixels when data is missing in the point cloud. If there is no missing data, Feature Width has no effect on the resulting projection image. The following image shows a projection image formed using the Automatic Height Map projection mode: Example: Calculating Projection Image Range Values from Point Cloud The following example shows how the function calculates the projection image range based on the point cloud and the ROI:
Element | Description
Blue shape | Point cloud
Black rectangle | ROI box
Grey area | Ignored area
255 | Maximum pixel value, topmost cloud point within ROI box
0 | Minimum pixel value, bottommost cloud point within ROI box
Shape Change | Identifies 3D depth along the XYZ axes, and visualizes 3D structural variations along the X, Y, and Z axes. This projection mode collects 3D data points within the ROI, and projects them into a 2D data representation. Changes are color coded relative to the X, Y, and Z axes in order to highlight changes in the orientation of surface normals. Estimated kernel A kernel is a rectangular array of numbers that defines the processing neighborhood for operations performed on each pixel location in an image. sizes for surface normals are set based on the Feature Width parameter. This projection mode amplifies edges but is less sensitive to height changes. The following image shows a projection image formed using the Shape Change projection mode:
Vertical Shape Change | Identifies 3D depth along the Z-axis. This projection mode is designed to identify and visualize 3D structural variations along the Z-axis. This projection mode collects 3D data points within the ROI, and projects them into a 2D data representation, highlighting changes in the orientation of surface normals relative to the Z axis. Estimated kernel sizes for surface normals are set based on the Feature Width parameter. This projection mode amplifies edges but is less sensitive to height changes. The following image shows a projection image formed using the Vertical Shape Change projection mode:
Flex Inspect | Creates projection images that identify features and anomalies on a smooth freeform surface, where the coverage of features is expected to be a small portion of the entire surface. If the features of interest do not have a sufficiently large and clear background surface area, projection images can have bad contrast between features and surrounding areas. This projection mode estimates the shape of the local background as a freeform surface and shows height changes relative to the local background shape. This mode computes the average height around a pixel according to the Feature Width parameter, and overlays the difference between pixel height and the computed average height onto the projection according to the Feature Height parameter. The following image shows a projection image formed using the Flex Inspect projection mode:
Curve Inspect | Creates projection images that identify features and anomalies on a smooth curved surface, where the coverage of features is expected to be a small portion of the entire surface. If the features of interest do not have a sufficiently large and clear background surface area, projection images can have bad contrast between features and surrounding areas. This projection mode estimates the shape of the local background as a curved surface and shows height changes relative to the local background shape. This mode computes a curved surface model based on pixel neighborhoods according to the Feature Width parameter, and overlays the difference between pixel height and the surface model onto the projection according to the Feature Height parameter. The following image shows a projection image formed using the Curve Inspect projection mode:
Greyscale | Identifies overall 2D surface flaws. This projection mode is a direct mapping technique from the height data addressed to an 8-bit image. In this mapping, the value of each pixel in the resulting image corresponds directly to the intensity of the original 3D points. The following image shows a projection image formed using the Greyscale projection mode:
Feature Width | Specifies the expected width of a feature of interest in millimeters. This parameter determines the estimated kernel sizes when converting a 3D point cloud into a 2D projection image. Best Practices for Setting Feature Width Set this parameter according to the scale of the feature of interest for your particular task. For example, if the feature is 5 mm, set the parameter to 5. Use the following guidelines when setting this parameter: First, ensure that the Projection Image Resolution parameter is set in a way that makes the feature easy to see. Setting a lower value makes fine features more visible in the projection image. This allows the function to inspect or work with them as needed. Setting a higher value smooths out fine features and makes coarser features sharper in the projection image. This allows the function to focus on a larger level of detail by hiding noise and small features that are not relevant to the task. If the value is too low for the current task, noise and irrelevant features might decrease the accuracy of the function. This issue can occur especially when using higher Projection Image Resolution settings. If the value is too high for the current task, the feature of interest might not be visible in the generated projection image, or not be sharp enough. This can prevent the tool from identifying the feature correctly. Incorrect settings can cause the function to miss or misclassify features. Note: This parameter applies to every projection mode, except for Greyscale.
Feature Height | Specifies the expected height of a feature of interest in millimeters. This parameter determines the pixel height when converting a 3D point cloud into a 2D projection image. Best Practices for Setting Feature Height Set this parameter according to the scale of the feature of interest for your particular task. For example, if the feature is 5 mm, set the parameter to 5. Use the following guidelines when setting this parameter: First, ensure that the Projection Image Resolution parameter is set in a way that makes the feature easy to see. Setting a lower value makes fine features more visible in the projection image. This allows the function to inspect or work with them as needed. Setting a higher value smooths out fine features and makes coarser features sharper in the projection image. This allows the function to focus on a larger level of detail by hiding noise and small features that are not relevant to the task. If the value is too low for the current task, noise and irrelevant features might decrease the accuracy of the function. This issue can occur especially when using higher Projection Image Resolution settings. If the value is too high for the current task, the feature of interest might not be visible in the generated projection image, or not be sharp enough. This can prevent the tool from identifying the feature correctly. Incorrect settings can cause the function to miss or misclassify features. Note: This parameter applies only to the Flex Inspect and Curve Inspect projection modes.
Auto-Tune | Selects the projection settings for your application based on the training image analysis. When this option is enabled, the function tests multiple projection parameter combinations. The function automatically selects the best scoring settings. Note: This button is disabled until you have labeled a sufficient number of images.

Outputs

Output | Description
Unlabeled | The number of loaded images that are unlabeled.
Health | The Model Health of the function.
Passed | Boolean value indicating the success of the function.
Num Class | The number of defined classes.
Index | The index of the defined classes.
Label | The name of the defined classes.
Num Images | The number of classified images in the defined classes.
Score | The overall confidence score for the defined classes.

Vision Data Access Functions

Function | Description
--- | ---
GetClassLabel(ViDiELClassify3D, [Index], [Mode]) | Returns the classification label.
GetIsUnclassified | Sets the Unclassified Detection Mode.
GetModelHealth(ViDiELClassify3D) | Returns the health of a model between 0 and 100. The Model Health indicates how confident the function is in predicting the class of new, unseen images. Labeling images increases the Model Health percentage. Higher percentages indicate greater confidence.
GetNClasses(ViDiELClassify3D) | Returns the number of classes.
GetNImages(ViDiELClassify3D,[Index], [Mode]) | Returns the number of images.
GetPassed(ViDiELClassify3D,[Index]) | Returns the pass/fail status. Returns 1 on pass and returns 0 on fail.
GetScore(ViDiELClassify3D,[Index1],[Index2]) | Returns a score value between 0 and 100.