# GPU Allocation

There is a slight but important difference between the allocation logic of GPU resources of High Detail modes.

When your stream has both High Detail Mode and Focused mode tools, High Detail mode has the priority of processing compared to Focused mode. High Detail tools lock the GPU before Focused mode. Then, Focused mode tools try to occupy GPUs that are not locked by High Detail tools.

Note: It is not recommended to enable Optimized GPU Memory (Help - Compute Devices) for training multiple tools of High Detail and Focused mode within a stream. Using multiple modes in the same stream while Optimized GPU Memory is enabled can result in significantly slower processing, regardless of the number of GPUs you use.

## High Detail Mode

For High Detail mode tools, a GPU resource is locked on a tool basis:

- Training one tool (all images to be trained in a tool) occupies one GPU

- Processing one tool (all images to be processed in a tool) occupies one GPU

This means that when a High Detail mode tool occupies a GPU device for training or processing, this GPU is locked. Other tools cannot use it until the High Detail mode tool finishes its training or processing job.

If there are N High Detail mode tools and 1 GPU, these N High Detail mode tools sequentially occupy 1 GPU (first in first out). Once a tool occupies a GPU for its training or processing job, all the other tools wait in a queue until the running job is finished. If you execute a new training or processing job, this job is added to the end of the queue.

## Tool Chain of High Detail modes

For a chain of High Detail modes tools, the above principle of GPU allocation applies in the same way except that the parent tool (the upstream tool) is always trained first and then the child tool (the downstream tool). For example, see the figure below:

When processing, the parent is processed first, and since the processing of a High Detail tool occupies one GPU per tool, one child is processed first when there is a single GPU resource. The other child is processed after the processing of this child tool (all images to be processed in this child tool) is done.

### Focused Mode

For Focused mode tools, a GPU is locked on a tool basis for a training job but on an image basis for a processing job:

- Training one tool (all images to be trained in a tool) occupies one GPU.

- Processing one image (a single image to be processed from any tool) occupies one GPU.

This means that when a Focused mode tool occupies a GPU for training, this GPU is locked so that other tools cannot use it until the Focused mode tool finishes its training job.

If there are N Focused mode tools and 1 GPU, these N Focused mode tools occupy one GPU sequentially (first in first out). Once a tool occupies a GPU for a training job, all the other tools wait in a queue until the running job is finished. If you execute a new training job, this job is added to the end of the queue.

However, when a Focused mode tool occupies a GPU for processing, the GPU is locked only until it finishes processing on an image. Other tools can occupy this GPU (mostly the tool that has been waiting up front in a queue) right away as soon as it finishes processing for an image, which normally takes very little time.

Processing Focused mode tools uses a first in first out queue. Once a tool occupies a GPU for a processing job, all the other tools wait in a queue until the processing of an image is finished. But as the processing time for an image is generally very short, the other Focused mode tools occupy the GPU in rotation.

- If you execute a new training job, it is added to the queue and sooner or later might occupy a GPU for long until the training job for all its images is completed.

- If you execute a new processing job, it is added to the queue and might occupy a GPU only until the processing of an image of this tool is done. Then, the processing job occupies the same GPU again, or another GPU for the processing of another image of this tool (another processing job). This is repeated until all the images that belong to the tool are done processing.

### Tool Chain of Focused Modes

For a chain of Focused mode tools, the above principle of GPU allocation applies in the same way except that the parent tool (the upstream tool) is always trained first and then the child tool (the downstream tool) is trained. For example, see the figure below:

When processing, the parent is processed first, but since the processing of a Focused tool occupies a GPU on an image basis, the two children tools can be processed simultaneously.

Note: For the processing, the utilization ratio of each GPU differs by the size of each image. The allocation between GPUs is figured out as equally as possible.