Kaa Internals

Module Importing

In large and complex Python libraries, it is often difficult to avoid circular importing. For example, main, which provides functions to manage the kaa mainloop, requires thread to ensure that kaa.main.stop() runs in the main thread. thread requires async so that ThreadInProgress can subclass InProgress. async in turn requires main in order to run the main loop for kaa.InProgress.wait().

Import cycles can generally be handled in several different ways:

1. Consolidate all code involved in import cycles into the same file to eliminate the circular dependencies altogether.
2. Use only absolute imports (import foo.bar) and never relative imports (from foo import bar), and move imports that generate import cycles to the bottom of the file.
3. Import at function invocation time, just before the module is needed. This will work for most scenarios, but not when you want to subclass a class defined in another module.

The first option is usually the simplest and most reliable, but for complex dependencies can involve merging considerable amounts of code, which makes maintenance more complicated. In the case of Kaa, it would involve merging all the code related to signals, threads, timers, coroutines, generators, async (InProgress) and the mainloop. This would result in a monstrous, nearly 5000 line file.

The second option would require explicit absolute importing of all modules inside the framework (import kaa.async) which would preclude being able to import directly from the source tree, which is useful for debugging and bootstrapping the installation. It also splits up imports, burying the circular imports at the bottom of the file, and relies on proper comments to explain what’s happening. Managing circular dependencies this way has shown to be fragile.

The third option sounds tempting if it’s possible (i.e. the circular dependency is truly only needed at runtime, not at compile time). However this can be volatile and can result in deadlocks in certain circumstances. Generally we have found that importing a module that hasn’t been imported already from the main thread should be avoided. However if the module is already imported, it is safe to import it from a thread.

The solution chosen for Kaa is a combination of the first and third options. Namely:

• most of the nastier circular dependencies have been avoided by moving the Object, Signal, Signals classes, and the code used to manage the thread notifier pipe and callback queuing for mainthread invocation (CoreThreading) into one file core.py
• the remaining (three) circular dependencies are resolved by performing run-time imports; the lazy importing code has been modified to load the required modules up-front to prevent (or at least mitigate) these modules from being imported for the first time inside a thread.

The current circular dependencies (which are being handled by importing at function invocation time) are:

• core requires async (for Signal.__inprogress__() and Signals.any() and .all()); async requires core (for everything).
• async requires timer (for InProgress.timeout()); timer requires thread (for threaded decorator); thread needs async (so that ThreadInProgress can subclass InProgress).
• async requires main (InProgress.wait() calls main.loop()); main requires thread (for threaded decorator, and thread.killall()); thread requires async (for the reason explained above).

There are a number of “core” modules that must be imported as a group to ensure that none of the modules imported at function invocation time are imported for the first time in a thread. These are core, nf_wrapper, async, thread, timer, and main. The (crude) diagram below documents the existing dependencies for these modules, showing the first level of dependencies for each. A . denotes that there are no further dependencies (or the lower level dependencies are considered resolved):

   weakref -> .
     utils -> .
  strutils -> .
  callable -> utils -> .
nf_wrapper -> utils -> .
           -> callable -> .
      core -> utils -> .
           -> callable -> .
           -> nf_wrapper -> .
           -> [imports async at runtime]
     async -> utils -> .
           -> callable -> .
           -> core -> .
           -> [imports timer at runtime]
           -> [imports main at runtime]
    thread -> utils -> .
           -> callable -> .
           -> core -> .
           -> async -> .
     timer -> weakref -> .
           -> utils -> .
           -> nf_wrapper -> .
           -> core -> .
           -> thread -> .
      main -> nf_wrapper -> .
           -> core -> .
           -> timer -> .
           -> thread -> .

As the dependency graph shows, it is sufficient to import main to cause the six “core” modules to be imported. The _LazyProxy class in __init__.py will implicitly import main when almost any kaa object is referenced, which should make it extremely unlikely that either async or main should ever be imported for the first time within a thread.