# Review Parameter Search Results

After the Parameter Search utility has completed the computation of the grid of parameter settings, you can review the results to determine the best combinations for your application. There are 2 key components to review the results: the Results table and the Graphs.

### Review Results

You can investigate the results of parameter search on the Results page once the search is done. The search results are displayed in the result table. Each row of this table describes the values of each parameter combination that were tested, and the test results (F-score, AUC, and others), which are different by tool type. For each row of this table, you can project a Database Overview that formulates the search results with a confusion matrix.

CAUTION: Before you click Home to go back to Home screen, click Save to save the current parameter search results. Going back to Home screen will reset all the results.

Statistics on Result Table

The result table describes the result statistics per each tool type:

Red Analyze | Green Classify | Blue Locate and Blue Read
--- | --- | ---
Precision (Confusion Matrix) | F-Score | Found (Features)
Recall (Confusion Matrix) | F-Score (avg) | Precision (Features)
F-Score (Confusion Matrix) | AUC | Recall (Features)
Precision (Region Area Metrics) | Speed [ms] | F-Score (Features)
Recall (Region Area Metrics) |  | Found (Models)
F-Score (Region Area Metrics) | Precision (Models)
AUC | Recall (Models)
Speed [ms] | F-Score (Models)
| Feature Position Distance
Feature Size Difference
Feature Angle Difference
Speed [ms]

Search Results for Red Analyze

The search results of a Red Analyze tool includes Region Area Metrics (Precision, Recall, F-score calculated in pixel level, only for Red Analyze Supervised, Red Analyze High Detail, Red Analyze High Detail Quick) and the metrics of Confusion Matrix (Precision, Recall, F-score calculated from confusion matrix items). See Red Analyze Tool – Statistics – Region Area Metrics and Red Analyze Tool – Statistics – Confusion Matrix for the interpretation of the search results.

Search Results for Green Classify

The search results of a Green Classify tool includes the classification performance metrics including F-score (avg) and F-score. See Green Classify Tool – Statistics for the details. F-score (avg) is the average over the F-scores calculated for each class.

Search Results for Blue Locate/Read

The search results of a Blue Locate/Read tool includes Feature Metrics (the Precision, Recall, F-score calculated per feature) and Model Metrics (the Precision, Recall, F-score calculated per model). See Blue Locate Tool – Statistics and Blue Read Tool – Statistics for the details.

There are other Blue Locate/Read result metrics that are currently only shown in the Parameter Search Utility.

- Feature Size Difference

- Feature Angle Difference

- Feature Position Distance

For Blue Locate/Read, a found feature is said to be correctly found (marked) if there is a labeled feature relatively close. For each correctly found feature, you can compute the position distance, angle difference, and size difference by comparing that correctly found feature to the labeled feature. The reported Feature Position Distance, Feature Angle Difference, and Feature Size Difference in result table are “average feature position distance,” “average angle difference,” and “average size difference” (root-mean-square averages), respectively, calculated over all correctly found features.

Metrics | Description
Feature Size Difference | The difference of Feature Size between the correctly found feature and the labeled feature
Feature Angle Difference | The difference of Feature Angle between the correctly found feature and the labeled feature
Feature Position Distance | The difference of Feature Position between the correctly found feature and the labeled feature

Steps

1. When you click a row in the result table, a row is selected and the Database Overview with a Confusion Matrix corresponds to the search results of this row will show up in the leftmost side at the bottom of the screen.

   - The Database Overview is much the same as the one in Cognex Deep Learning Studio. See Statistics for interpreting the results in Database Overview.

   - For Red Analyze tools, Count drop-down options (Views, Untrained Views, Regions, Untrained Regions) can be applied to project the Database Overview by each aspect. See Red Analyze Tool – Statistics – Confusion Matrix

2. Click the Pin to pin down the Database Overview for a row. 2 Database Overviews can be fixed at maximum simultaneously. Click icon to release the Database Overview from the panel below.

   - The Database Overview of every selected row will be pegged down at the leftmost side of the panel below, and 1 or more pinned down Database Overviews will be pegged on its right.

   - If you click icon on each pegged Database Overview, it will be released from the panel.

3. When a row in the result table is selected, click Apply To Workspace to apply the selected parameter set and the trained tool (neural network model) in the loaded workspace. The tool parameter values and the tool itself in the currently loaded workspace will be replaced with those in the selected row in the result table.

   After it is applied in the loaded workspace, you now can access the tool and its training results in VisionPro Deep Learning. In Cognex Deep Learning Studio, open the workspace and process the tool and you will get the same results that were produced in Parameter Search.

   Note: As the previously saved Parameter Search Data XML file does not contain the trained networks (the trained tools), you cannot use Apply To Workspace with the loaded Parameter Search Data XML file. To apply the previously saved search results to your workspace, you must save the trained networks by enabling Save Tools checkbox in Search page to save tool files that contain the trained networks before beginning a search.

   Example: Sorting a Red Analyze Focused Supervised Result by F-Score (Region Area Metric) and apply the best result to the current tool, along with the parameter combination (0% Luminance, 10% Scale, 20 Epoch Count) and the trained neural network that produced this result.

   Example: Launch VisionPro Deep Learning, load the workspace and the tool, and process the tool to review the applied result and the parameter combination.

4. Click Save to export the search configuration and search results as Parameter Search Data XML file. Click Load to import a previously configured and saved Parameter Search Data XML file to the Results page. It can be used as the basis for a new search.

   - Note that the search result table is saved with the Parameter Search Data XML file while the trained networks are not saved with it.

5. If you previously enabled Save Tools checkbox in Search page before you start the search, the tools containing the search results are saved after the search. Each saved tool holds the trained neural network model corresponding to each set of parameter values tested.

   - For example, if you have 10 sets of parameters (10 rows in result table), 10 tools will be saved as 10 files to the directory containing the Parameter Search Data XML file.

### Explore Graphs

On the Graphs page, the search results are displayed by a result table, which is a little different from the one in the Results page. This table describes the same results, but these results are served using some statistical measures which are calculated over all trials of searching. If the Trial in Search page was set to more than 1, the columns will display the minimum, maximum, mean, standard deviation, median, median absolute deviation of the chosen metric at the Metric drop-down list over all trials, given these values of parameters. For example, let us assume the following search configuration is given:

- Tool: Red Analyze Focused Supervised

- Parameter Set: Luminance, Scale, Epoch Count

- Parameter Range

  - Luminance: 0%, 5%

  - Scale: 0%, 10%

  - Epoch Count: 10, 20

- Trial: 3

If the Luminance, Scale, Epoch Count values at the first row of the result table are {Luminance: 0%, Scale 10%, Epoch Count: 10} and if you set the performance metric as F-Score of Confusion Matrix at the Metric drop-down list, the result table will display the minimum, maximum, mean, standard deviation, median, median absolute deviation of F-Score of Confusion Matrix over 3 trials when Luminance is 0%, Scale is 10%, and Epoch Count is 10, at its first low. See Red Analyze Tool – Statistics – Confusion Matrix for the details of F-Score of Confusion Matrix.

CAUTION: Before you click Home to go back to Home screen, click Save to save the current parameter search results and graphs. Going back to Home screen will reset all the results and graphs.

In addition to this, 2D contour plots that describe 3 or more variables (dimensions) and depict the search results are given. Especially, these graphs illustrate the relationships between the chosen parameters in terms of the statistical search results over all trials.

To plot the search results on the graph, you first set the x-axis and y-axis of the graph from the parameters used in the search. Set the desired parameters as x-axis and y axis, respectively, using the drop-down lists for each parameter.

Once the 2 axes are set, you are ready to see how the value of a performance metric variates by the values of 2 axes. The numeric values plotted on the graph are basically the values of each performance metric turning up as you scroll through the Metric drop-down list. Plus, in Measure drop-down list, you need to select a statistical measure which is calculated over all trials for the selected metric. The list of the supported measures in the Measure drop-down list is as follows:

- Mean: the mean of the metric over all search trials

- STD: the standard deviation of the metric over all search trials

- MAD: the median absolute deviation of the metric over all search trials

- Median: the median of the metric over all search trials

For example, if you set Luminance as x-axis, set Scale as y-axis, select F-Score in Metric, and select MAD in Measure, the MAD values of F-Score over all trials will be plotted on the graph following the changes of Luminance (x-axis) and Scale (y-axis) values.

If there are 3 or more parameters, the parameters for x-axis and y-axis can be swapped over among them. The values of parameters other than x-axis and y-axis can also be projected on the graph by selecting the specific values for each parameter's drop-down list.

Note: The elements showing up in Metric drop-down list are different by each tool type (Red Analyze, Green Classify, Blue Locate/Read).

Steps

1. Select F-Score in the Metric drop-down list, and this will show the result table for F-Score just below. The columns display the values of parameters and the mean, standard deviation, median, median absolute deviation, minimum, and maximum of F-Score over all trials, given these values of parameters.

   - For Red Analyze tools if you chose Confusion Matrix metrics in Metric drop-down list, Count drop-down (Views, Untrained Views, Regions, Untrained Regions) can be applied to project the graph by each aspect. See Red Analyze Tool – Statistics – Confusion Matrix

   - For Blue Locate/Read tools if you chose Precision, Recall, and F-Score in Metric drop-down list, Count drop-down (Features, Model) can be applied to project the Precision, Recall, F-Score for Features or the Precision, Recall, F-Score for Models on the graph. See Blue Locate Tool – Statistics and Blue Read Tool – Statistics for the details.

2. Provide a measure (mean, standard deviation, median, or median absolute deviation) at the Measure drop-down list, and set the parameters for the x-axis and y-axis.

   - If you previously set more than 1 trial at Trials bar on Search page, there will be many columns where correspond to the F-Scores of each trial (Trial 1, Trial 2, Trial 3, ... , Trial N). The chosen measure for a metric is calculated over all trials.
   - Note that when it was tested with only 1 Trial, the results will not be returned for STD, MAD, Min or Max.

3. Set the values for the parameters other than x-axis and y-axis at each drop-down list. Then, the corresponding contour plot will show up.

4. If you have finished investigating the result table and the graphs, and found the one best set of parameter values, you can go back to the Results page, select a row that describes the best set found in the result table, and click Apply To Workspace to apply the selected set and the trained tool (neural network model) in the loaded workspace. The tool parameter values and the tool itself in the currently loaded workspace will be replaced with those in the selected row in the result table.