# GPU Allocation

GPUs are allocated to tools during training Training is the process that your tool, which is a neural network, is learning about the features (pixels) based on the labels you made. For example, a tool will learn the defect/normal pixels in each image based on the defect/normal labels you drew. The goal of the tool Training is learning enough to give the correct inspection results of whether an unseen image is defective or not. The key to training is to ensure that you include all possible variations within your training set, and that your images are accurately labeled. Training times vary by the application, tool setup and the GPU in the PC being used to train the network. and processing within a stream. When a GPU is allocated to a tool, other tools cannot use that GPU.

GPU allocation determines how tools access available GPUs, which tools run in parallel, and which tools must wait in a queue. The allocation behavior depends on the tool type (Standard or Legacy) and on whether the operation is training or processing.

## Standard Tools

A Standard tool occupies a GPU until the training or processing of all the images in the tool is completed.

Tools waiting for training or processing are handled using a first-in, first-out queue. If there are N Standard tools and one GPU, the tools are trained sequentially.

For a chain of Standard tools, the same GPU allocation principle applies, except that the upstream tool is trained first, followed by the downstream tools.

During processing, the upstream tool is processed first. The downstream tools are then processed sequentially. Processing of a downstream tool does not begin until all images in the preceding tool have been processed.

## Legacy Tools

Training Legacy a tool occupies one GPU until training of all images in the tool is complete. Tools waiting for training are handled in a first-in, first-out queue. If there are N Legacy tools and one GPU, the tools are trained sequentially.

Processing a Legacy tool occupies one GPU only for the duration of processing a single image. During processing, Legacy tools also use a first-in, first-out queue. Because processing a single image typically takes very little time, GPU allocation rotates among tools as each image is processed.

When a new tool processing is started, it is added to the queue and occupies a GPU only until processing of one image is complete. The tool then re-enters the queue to process the next image. This continues until all images in the tool have been processed.

For a chain of Legacy tools, the same GPU allocation principle applies, except that the upstream tool is always trained first, followed by downstream tools.

After the processing of the upstream tool finishes, the downstream tools are processed one after the other on a per-image basis. The utilization ratio of each GPU varies with the image size. GPU allocation is balanced as evenly as possible.

## Using Standard and Legacy Tools in a Stream

When your stream contains both Standard and Legacy tools, Standard tools are processed first, followed by Legacy tools.

Standard tools are allocated to a GPU before Legacy tools. Legacy tools attempt to use GPUs that are not allocated to Standard tools.

Note: Do not disable Optimized GPU Memory when training multiple Standard and Legacy tools within a stream. Doing so can significantly slow down processing, regardless of the number of GPUs you use.