# GPU Allocation

There is a slight but important difference between the allocation logic of GPU resources of Standard modes.

When your stream has both Standard type and Legacy type tools, Standard type has the priority of processing compared to Legacy type. Standard tools lock the GPU before Legacy type. Then, Legacy type tools try to occupy GPUs that are not locked by Standard tools.

Note: Do not enable Optimized GPU Memory when training Training is the process that your tool, which is a neural network, is learning about the features (pixels) based on the labels you made. For example, a tool will learn the defect/normal pixels in each image based on the defect/normal labels you drew. The goal of the tool Training is learning enough to give the correct inspection results of whether an unseen image is defective or not. The key to training is to ensure that you include all possible variations within your training set, and that your images are accurately labeled. Training times vary by the application, tool setup and the GPU in the PC being used to train the network. multiple tools of Standard and Legacy type within a stream, because it causes significantly slower processing regardless of the number of GPUs you use.

## Standard type

For Standard type tools, a GPU resource is locked on a tool basis:

- Training one tool (all images to be trained in a tool) occupies one GPU

- Processing one tool (all images to be processed in a tool) occupies one GPU

This means that when a Standard type tool occupies a GPU device for training or processing, this GPU is locked. Other tools cannot use it until the Standard type tool finishes its training or processing job.

If there are N Standard type tools and 1 GPU, these N Standard type tools sequentially occupy 1 GPU (first in first out). Once a tool occupies a GPU for its training or processing job, all the other tools wait in a queue until the running job is finished. If you execute a new training or processing job, this job is added to the end of the queue.

## Tool Chain of Standard type tools

For a chain of Standard type tools, the above principle of GPU allocation applies in the same way except that the parent tool (the upstream tool) is always trained first and then the child tool (the downstream tool). For example, see the figure below:

When processing, the parent is processed first, and since the processing of a Standard tool occupies one GPU per tool, one child is processed first when there is a single GPU resource. The other child is processed after the processing of this child tool (all images to be processed in this child tool) is done.

### Legacy type tool

For Legacy type tools, a GPU is locked on a tool basis for a training job but on an image basis for a processing job:

- Training one tool (all images to be trained in a tool) occupies one GPU.

- Processing one image (a single image to be processed from any tool) occupies one GPU.

This means that when a Legacy type tool occupies a GPU for training, this GPU is locked so that other tools cannot use it until the Legacy type tool finishes its training job.

If there are N Legacy type tools and 1 GPU, these N Legacy type tools occupy one GPU sequentially (first in first out). Once a tool occupies a GPU for a training job, all the other tools wait in a queue until the running job is finished. If you execute a new training job, this job is added to the end of the queue.

However, when a Legacy type tool occupies a GPU for processing, the GPU is locked only until it finishes processing on an image. Other tools can occupy this GPU (mostly the tool that has been waiting up front in a queue) right away as soon as it finishes processing for an image, which normally takes very little time.

Processing Legacy type tools uses a first in first out queue. Once a tool occupies a GPU for a processing job, all the other tools wait in a queue until the processing of an image is finished. But as the processing time for an image is generally very short, the other Legacy type tools occupy the GPU in rotation.

- If you execute a new training job, it is added to the queue and sooner or later might occupy a GPU for long until the training job for all its images is completed.

- If you execute a new processing job, it is added to the queue and might occupy a GPU only until the processing of an image of this tool is done. Then, the processing job occupies the same GPU again, or another GPU for the processing of another image of this tool (another processing job). This is repeated until all the images that belong to the tool are done processing.

### Tool Chain of Legacy Type Tools

For a chain of Legacy type tools, the above principle of GPU allocation applies in the same way except that the parent tool (the upstream tool) is always trained first and then the child tool (the downstream tool) is trained. For example, see the figure below:

When processing, the parent is processed first, but since the processing of a Legacy tool occupies a GPU on an image basis, the two children tools can be processed simultaneously.

Note: For the processing, the utilization ratio of each GPU differs by the size of each image. The allocation between GPUs is figured out as equally as possible.