# Runtime API Processing Guide

Focused modes support parallel processing as their processings are done on an image basis. You can leverage this power of parallel processing when you organize your code workflow for processing Focused tools in a runtime workspace with API. It is highly recommended to see GPU Allocation before reading this page for more details about parallel processing of tools.

### Objects in Runtime Workspace

When you execute processing tasks with VisionPro Deep Learning API in a runtime environment, you are going to handle objects like Threads, GPUs, Samples, etc. for organizing API code workflow.

Objects | Description/Note
Workspace | Thread-Safe
Stream | Thread-Safe
Tool | Thread-Safe
Sample | A container object that handles an image Not Thread-Safe
Image | An object that has the actual image data to be processed Not Thread-Safe
Buffer | An object that receives the result Not Thread-Safe
Thread |

### 2 Way Strategies

When multiple GPU devices are initialized, you have 2 options for invoking processing by calling API.

Note: The application of the following workflow strategies is limited to the processing of Focused mode tools. See GPU Allocation for more details.

##### Providing "Empty List" as a Device List Argument

if you use an empty list for the "device list" argument when calling API, the processing is executed on any available (unoccupied, released) GPU. This means that you are not choosing the specific GPUs to process specific tools. This strategy is recommended when you:

- simply require parallel processing itself without any other purpose:

- simply want to test whether your parallel processing code is working or not.

- want your code implementation to be executed always and only in a parallel way in whatever situation you belong to.

- process 2 tools whose expected time for processing are similar to each other, which means that they are good to go for parallel processing without designating specific GPUs

- process 1 tool in parallel, which means that you want to exploit multiple GPUs to boost processing by parallel processing.

For explanation, it is good to take a view on a simple case that you need to process 1 Focused mode tool with 2 Threads and 2 GPUs.

If you provide an empty list as the device list argument and execute processing this tool in both 2 Threads (Thread 0 and Thread 1), one of the Threads, say Thread 0, occupies GPU 0 only for a while to process an image.

When GPU 0 is occupied by Thread 0, GPU 0 is locked so Thread 1 access GPU 1 and occupy it for processing another image from this tool.

After Thread 0 finishes the processing of an image with GPU 0, it releases GPU 0 and GPU 0 can be occupied by Thread 1 if the processing job of Thread 1 was waiting in the waiting list (Queue). In this way, this single tool can be processed in parallel with 2 GPUs and 2 Threads.

#####

##### Providing "Non-Empty List" as a Device List Argument

if you use a non-empty list for the "device list" argument when calling API, the processing is executed on the available (unoccupied, released) GPUs that are in the provided "device list". This means that you are choosing a list of one or more GPUs to process specific tools. This strategy is recommended when you:

- process 2 tools whose expected time for processing are deviating a lot from each other, which means that one of them could be the bottleneck against one another in parallel processing if this tool did not have designated GPU(s) that is(are) solely allocated to this tool.

- want to control which GPU(s) should be used for processing specific tools and which should not.

For explanation, it is good to take a view on a simple case that you need to process 1 Focused mode tool with 2 Threads and 2 GPUs.

If you provide a list that contains GPU 0 for Thread 0 and a list that contains GPU 1 for Thread 1, each, as the device list argument and execute processing this tool in both 2 Threads (Thread 0 and Thread 1), GPU 0 is solely allocated to Thread 0 and GPU 1 is to Thread 1.

Thread 0 uses only GPU 0 until it finishes its processing jobs for a given set of images, and it is same to Thread 1 and GPU 1.

For another case, this time you have to process 2 Focused mode tools (A, B) with 2 Threads (Thread 0 for tool A, Thread 1 for tool B) and 3 GPUs (GPU 0, GPU 1, GPU 2). Tool B takes much more time for processing, since the number of its images to be processed is significantly larger than those of tool A or the resolution of the image data of tool B is much higher than that of tool A.

Even with this circumstance, you are good to go for parallel processing provided with an empty device list. The total processing time (the time required to finish processing of both A and B) would remain the same compared to when a non-empty device list is provided.

But the problem of this case is that the processing of tool B could slow down the processing of tool A, since for the most part 3 GPUs are going to be occupied by Thread 1 because of tool B's high workload and since the processing of Focused tool is done on image.

Therefore, if you want this parallel processing of tool A and B without significantly slowing down the processing of tool A, you can provide a device list, [GPU 1] or [GPU 1, GPU 2], for Thread 1 to make sure these GPUs are dedicated to the processing of tool B. You can either also provide a device list or an empty list for Thread 0. This will prevent tool B from being the bottleneck against tool A in parallel processing.