Background Task Processing

Introduction

In order to build responsive Toolkit apps, work needs to be scheduled to run in background threads. The BackgroundTaskManager handles generic background processing and is used by many of the Toolkit frameworks.

When using a ShotgunModel, for example, the model will internally create a ShotgunDataRetriever to handle background communication with Shotgun. The ShotgunDataRetriever in turn uses threads to handle its background processing and uses this BackgroundTaskManager in order to schedule the background processing.

A centralized thread pool

By default, each Shotgun Model will have its own background task manager that runs Shotgun queries in the background. If your app uses a lot of different models or data retrievers, this becomes hard to maintain and may lead to unpredictable states due to the fact that the Shotgun API isn’t thread safe.

Note

We recommend that apps using several Shotgun Models or Data retrievers use a single BackgroundTaskManager for handling of all its background processing.

In these situations, you can maintain a single BackgroundTaskManager for your app and supply it to your ShotgunModel and ShotgunDataRetriever instances when creating them. This allows for a setup where all threaded work is handled by a single thread pool and allows for efficient control and prioritization of the work that needs to be carried out.

Here is an example of how the Toolkit apps typically set this up:

# import the task manager
task_manager = sgtk.platform.import_framework("tk-framework-shotgunutils", "task_manager")


class AppDialog(QtGui.QWidget):
    """
    App main dialog
    """
    def __init__(self, parent):
        # in your main dialog init, create a background task manager
        self._task_manager = task_manager.BackgroundTaskManager(parent=self,
                                                                start_processing=True,
                                                                max_threads=2)

        # create models and request that they use the task manager
        self._model_a = shotgun_model.SimpleShotgunModel(parent=self,
                                                         bg_task_manager=self._task_manager)
        self._model_b = shotgun_model.SimpleShotgunModel(parent=self,
                                                         bg_task_manager=self._task_manager)


    def closeEvent(self, event):
        # gracefully close down threadpool
        self._task_manager.shut_down()

        # okay to close dialog
        event.accept()

Using the Task Manager directly

The task manager isn’t just a controllable building block used by the internal Toolkit libraries; you can also use it directly to control background work that your app is doing.

Simply register a task that you want it to perform and it will queue it up and execute it once a worker becomes available. You can control how many working threads you want the task manager to run in parallel and tasks can easily be prioritized, grouped and organized in a hierarchical fashion.

Class BackgroundTaskManager

Note

Import the module into your Toolkit App using the following statement:

task_manager = sgtk.platform.import_framework("tk-framework-shotgunutils", "task_manager")

class task_manager.BackgroundTaskManager(parent, start_processing=False, max_threads=8)[source]

Main task manager class. Manages a queue of tasks running them asynchronously through a pool of worker threads.

The BackgroundTaskManager class itself is reentrant but not thread-safe so its methods should only be called from the thread it is created in. Typically this would be the main thread of the application.

Signal task_completed(uid, group, result):

Emitted when a task has been completed. The uid parameter holds the unique id associated with the task, the group is the group that the task is associated with and the result is the data returned by the task.

Signal task_failed(uid, group, message, traceback_str):

Emitted when a task fails for some reason. The uid parameter holds the unique id associated with the task, the group is the group that the task is associated with, the message is a short error message and the traceback_str holds a full traceback.

Signal task_group_finished(group):

Emitted when all tasks in a group have finished. The group is the group that has completed.

Parameters:

parent (QWidget) – The parent QObject for this instance
start_processing – If True then processing of tasks will start immediately
max_threads – The maximum number of threads the task manager will use at any time.

next_group_id()[source]

Return the next available group id

Returns:: A unique group id to be used for tasks that belong to the same group.

start_processing()[source]: Start processing of tasks

pause_processing()[source]: Pause processing of tasks - any currently running tasks will complete as normal.

shut_down()[source]: Shut down the task manager. This clears the task queue and gracefully stops all running threads. Completion/failure of any currently running tasks will be ignored.

add_task(cbl, priority=None, group=None, upstream_task_ids=None, task_args=None, task_kwargs=None)[source]

Add a new task to the queue. A task is a callable method/class together with any arguments that should be passed to the callable when it is called.

Parameters:

cbl – The callable function/class to call when executing the task
priority – The priority this task should be run with. Tasks with higher priority are run first.
group – The group this task belongs to. Task groups can be used to simplify task management (e.g. stop a whole group, be notified when a group is complete)
upstream_task_ids – A list of any upstream tasks that should be completed before this task is run. The results from any upstream tasks are appended to the kwargs for this task.
task_args – A list of unnamed parameters to be passed to the callable when running the task
task_kwargs – A dictionary of named parameters to be passed to the callable when running the task

Returns:

A unique id representing the task.

add_pass_through_task(priority=None, group=None, upstream_task_ids=None, task_kwargs=None)[source]

Add a pass-through task to the queue. A pass-through task doesn’t perform any work but can be useful when synchronising other tasks (e.g. pulling the results from multiple upstream tasks into a single task)

Parameters:

priority – The priority this task should be run with. Tasks with higher priority are run first.
group – The group this task belongs to. Task groups can be used to simplify task management (e.g. stop a whole group, be notified when a group is complete). A group is expressed as a string, for example ‘thumbnails’, ‘IO’ or ‘shotgun’.
upstream_task_ids – A list of any upstream tasks that should be completed before this task is run. The results from any upstream tasks are appended to the kwargs for this task.
task_kwargs – A dictionary of named parameters that will be appended to the result of the pass-through task.

Returns:

A unique id representing the task.

stop_task(task_id, stop_upstream=True, stop_downstream=True)[source]

Stop the specified task from running. If the task is already running then it will complete but the completion/failure signal will be ignored.

Parameters:

task_id – The id of the task to stop
stop_upstream – If true then all upstream tasks will also be stopped
stop_downstream – If true then all downstream tasks will also be stopped

stop_task_group(group, stop_upstream=True, stop_downstream=True)[source]

Stop all tasks in the specified group from running. If any tasks are already running then they will complete but their completion/failure signals will be ignored.

Parameters:

group – The task group to stop
stop_upstream – If true then all upstream tasks will also be stopped
stop_downstream – If true then all downstream tasks will also be stopped

stop_all_tasks()[source]: Stop all currently queued or running tasks. If any tasks are already running then they will complete but their completion/failure signals will be ignored.

blockSignals(self, b: bool) → bool

childEvent(self, event: PySide2.QtCore.QChildEvent) → None

children(self) → List[PySide2.QtCore.QObject]

static connect(arg__1: PySide2.QtCore.QObject, arg__2: bytes, arg__3: Callable, type: PySide2.QtCore.Qt.ConnectionType = PySide2.QtCore.Qt.ConnectionType.AutoConnection) → bool
static connect(self, arg__1: bytes, arg__2: Callable, type: PySide2.QtCore.Qt.ConnectionType = PySide2.QtCore.Qt.ConnectionType.AutoConnection) → bool
static connect(self, arg__1: bytes, arg__2: PySide2.QtCore.QObject, arg__3: bytes, type: PySide2.QtCore.Qt.ConnectionType = PySide2.QtCore.Qt.ConnectionType.AutoConnection) → bool
static connect(self, sender: PySide2.QtCore.QObject, signal: bytes, member: bytes, type: PySide2.QtCore.Qt.ConnectionType = PySide2.QtCore.Qt.ConnectionType.AutoConnection) → PySide2.QtCore.QMetaObject.Connection
static connect(sender: PySide2.QtCore.QObject, signal: PySide2.QtCore.QMetaMethod, receiver: PySide2.QtCore.QObject, method: PySide2.QtCore.QMetaMethod, type: PySide2.QtCore.Qt.ConnectionType = PySide2.QtCore.Qt.ConnectionType.AutoConnection) → PySide2.QtCore.QMetaObject.Connection
static connect(sender: PySide2.QtCore.QObject, signal: bytes, receiver: PySide2.QtCore.QObject, member: bytes, type: PySide2.QtCore.Qt.ConnectionType = PySide2.QtCore.Qt.ConnectionType.AutoConnection) → PySide2.QtCore.QMetaObject.Connection

connectNotify(self, signal: PySide2.QtCore.QMetaMethod) → None

customEvent(self, event: PySide2.QtCore.QEvent) → None

deleteLater(self) → None

static disconnect(arg__1: PySide2.QtCore.QMetaObject.Connection) → bool
static disconnect(arg__1: PySide2.QtCore.QObject, arg__2: bytes, arg__3: Callable) → bool
static disconnect(self, arg__1: bytes, arg__2: Callable) → bool
static disconnect(self, receiver: PySide2.QtCore.QObject, member: Optional[bytes] = None) → bool
static disconnect(self, signal: bytes, receiver: PySide2.QtCore.QObject, member: bytes) → bool
static disconnect(sender: PySide2.QtCore.QObject, signal: PySide2.QtCore.QMetaMethod, receiver: PySide2.QtCore.QObject, member: PySide2.QtCore.QMetaMethod) → bool
static disconnect(sender: PySide2.QtCore.QObject, signal: bytes, receiver: PySide2.QtCore.QObject, member: bytes) → bool

disconnectNotify(self, signal: PySide2.QtCore.QMetaMethod) → None

dumpObjectInfo(self) → None

dumpObjectTree(self) → None

dynamicPropertyNames(self) → List[PySide2.QtCore.QByteArray]

emit(self, arg__1: bytes, *args: None) → bool

event(self, event: PySide2.QtCore.QEvent) → bool

eventFilter(self, watched: PySide2.QtCore.QObject, event: PySide2.QtCore.QEvent) → bool

findChild(self, arg__1: type, arg__2: str = '') → object

findChildren(self, arg__1: type, arg__2: PySide2.QtCore.QRegExp) → Iterable
findChildren(self, arg__1: type, arg__2: PySide2.QtCore.QRegularExpression) → Iterable
findChildren(self, arg__1: type, arg__2: str = '') → Iterable

inherits(self, classname: bytes) → bool

installEventFilter(self, filterObj: PySide2.QtCore.QObject) → None

isSignalConnected(self, signal: PySide2.QtCore.QMetaMethod) → bool

isWidgetType(self) → bool

isWindowType(self) → bool

killTimer(self, id: int) → None

metaObject(self) → PySide2.QtCore.QMetaObject

moveToThread(self, thread: PySide2.QtCore.QThread) → None

objectName(self) → str

parent(self) → PySide2.QtCore.QObject

property(self, name: bytes) → Any

receivers(self, signal: bytes) → int

static registerUserData() → int

removeEventFilter(self, obj: PySide2.QtCore.QObject) → None

sender(self) → PySide2.QtCore.QObject

senderSignalIndex(self) → int

setObjectName(self, name: str) → None

setParent(self, parent: PySide2.QtCore.QObject) → None

setProperty(self, name: bytes, value: Any) → bool

signalsBlocked(self) → bool

startTimer(self, interval: int, timerType: PySide2.QtCore.Qt.TimerType = PySide2.QtCore.Qt.TimerType.CoarseTimer) → int

thread(self) → PySide2.QtCore.QThread

timerEvent(self, event: PySide2.QtCore.QTimerEvent) → None

tr(self, arg__1: bytes, arg__2: bytes = b'', arg__3: int = -1) → str