User Guide

Introduction

Plugins are arbitrary callables. They can declare other plugins as requirements while operating under certain guarantees:

A plugin can be loaded (i.e. called) exactly once until it is unloaded.
A plugin’s dependencies will be loaded before.
A plugin’s loaded dependents will be reloaded after.
When a plugin is unloaded, its loaded dependents will be unloaded before.

This paradigm naturally puts an emphasis on the structure of packages and applications and less on its orchestration. This allows consumers of applications to easily swap or add plugins while guaranteeing conformity to API contracts.

Install

The package is currently not available on pypi pending a PEP 541 request

The package can be configured as a Github dependency in a requirements.txt

pyplugin @ git+https://github.com/pyplugin/pyplugin@main

or to pin to a tag

pyplugin @ git+https://github.com/pyplugin/pyplugin@v0.1

Defining a Plugin

The fastest way to define a plugin is by using the built-in decorator:

from pyplugin import plugin

@plugin
def db_client(uri):
    return db_library.connect(uri)

We may also use the Plugin class:

from pyplugin import Plugin

my_plugin = Plugin(my_func)

We can also extend the Plugin class and use the built-in decorator for ease of use:

from pyplugin import Plugin, plugin

@plugin(cls=Plugin)
def db_client(uri):
    return db_library.connect(uri)

Naming & Registering

Plugins are globally registered under their fully-qualified dot-delimited package-name with using __name__. To change this, you may pass in name:

@plugin(name="my_plugin")
def db_client(uri):
    return db_library.connect(uri)

Plugins are automatically registered globally under their name. To register or unregister plugins, there are the register() and unregister() functions.

Anonymous Plugins

It may also be desirable to anonymize plugins so that they are not automatically registered upon definition:

@plugin(anonymous=True)
def db_client(uri):
    return db_library.connect(uri)

This may be useful if you want to defer registering a plugin without having to unregister it first:

@plugin
def db_client(uri):
    return db_library.connect(uri)

@plugin(anonymous=True)
def replacement_for_db_client():
    return foo

register(
    replacement_for_db_client,
    name="special_name",
)

Passing Plugin Instance

It may be beneficial in some cases to have the plugin passed in to the load callable, this is possible with the bind argument:

@plugin(bind=True)
def db_client(self, uri):
    assert self.get_full_name() == "db_client"
    return db_library.connect(uri)

Plugin Type Definition

By default, plugins will cache the return value type in the type attribute, including a is_class_type attribute, which means that the plugin returned a class type and type is that class.

You can explicitly set the type when defining a plugin:

@plugin(type=DatabaseClient)
def db_client(uri):
    return db_library.connect(uri)

You may also choose for your plugins to error if the return value does not match the type using the enforce_type argument (default: False).

See Settings for changing these settings (infer_type and enforce_type).

Requirements

We can define prerequisite plugins that will and must be loaded before loading:

from pyplugin import plugin

@plugin
def db_client(uri):
    return db_library.connect(uri)

@plugin(requires="db_client")
def db_writer(db_client):
    def func(doc):
        return db_client.insert_one(doc)
    return func

Now to load db_writer, db_client must be passed in or loaded (or it will attempt to load).

Defining Requirements

The requires parameter can be in a few different forms:

str: This will call lookup_plugin() before loading to find the dependency.
Plugin: This will explicitly pin a dependency to a specific plugin.
tuple: A tuple where the first element is 1 or 2 and the second element is the keyword arg we will pass to the plugin.
PluginRequirement: Which is a dataclass with two elements described in 3.
Iterable: An iterable of any of the above.

Dynamic Requirements

It is possible to load a plugin from within another plugin. By default, this will mark the loaded plugin as a requirement of the calling plugin as if it was defined in requires. For example:

@plugin
def upstream(arg=4):
    return arg

@plugin
def dyn_plugin():
    return upstream()

assert dyn_plugin() == 4
upstream(arg=5)
assert dyn_plugin.instance == 5

Reloading upstream with arg=5 also reloaded dyn_plugin.

See Settings dynamic_requirements for more.

Replacing Plugins

This requirement framework allows consumers of this library an opportunity to swap db_client with a custom user-defined implementation. For example:

# User Code (Option 1)
from my_library import db_client

db_client.load(uri="mongodb://localhost:27018")

# User Code (Option 2)
from my_library import db_client
from pyplugin import plugin, register

@plugin
class DictDB(dict):
    def insert_one(doc):
        self[doc["_id"]] = doc

# Replace (Option 1)
replace_registered_plugin("db_client", DictDB)

# Replace (Option 2)
db_client.replace_with(DictDB)

Now whenever db_writer is used, it will use the new DictDB.

See replace_registered_plugin() and replace_with() for more.

Note: The replace_with() method by default will keep the type of the original plugin (changed with the replace_type argument).

Loading a Plugin

You can load the plugin by simply calling it:

client = db_client()

or by explicitly calling the load() method:

client = db_client.load()

Plugin load can be broken down into the following steps:

Find, resolve, and cross-correlate dependencies
Load dependencies
Resolve any load conflicts (e.g. unload this plugin first then continue on)
Call the underlying callable and any callbacks in order
Reload loaded dependents

Find, Resolve, and Cross-Correlate Dependencies

Before loading, all dependencies defined in requirements will be resolved. If the dependency is a str, then lookup_plugin() will be used which will first check if there’s a registered plugin with the same name, then it will optionally attempt to import the name and register the plugin automatically. If Dynamic Requirements are enabled, this will also be handled.

Afterward the resolved dependency will be added to the dependencies map (which maps kwarg to plugin). In addition, we will append this plugin to each dependency’s dependents list.

Load Dependencies

In this step, the calling arguments are inspected. For each keyword argument which are the keys of the dependencies map, if the keyword is not in the varkwargs used to load the plugin, it will attempt to load the mapped plugin (without any arguments).

Resolve Load Conflicts

If the plugin is already loaded, and the arguments are different, this is considered a load conflict. You can pass in conflict_strategy to load() to resolve this which can be one of “keep_existing”, “replace”, “force”, or “error” (default: “replace”).

keep_existing: Ignore the load request
replace: First call unload() before attempting to load
force: Like replace but also will apply if load_args and load_kwargs match.
error: raises PluginLoadError.

Call Underlying Callable & Callbacks

The plugin’s underlying load_callable is then passed the arguments and keyword arguments and the return value is then saved.

Any callbacks defined in callbacks will be called in order.

Reload Loaded Dependents

The plugin’s loaded dependents are then reloaded with the new return value of the plugin.

Unloading a Plugin

We can define an unload operation upon definition:

from pyplugin import plugin

@plugin(
    unload_callable=lambda instance: instance.disconnect()
)
def db_client(uri):
    return db_library.connect(uri)

Now if we call the unload() method, the unload_callable will be called. Before a plugin is unloaded, any dependent plugins are unloaded first.

Similarly, plugin unload can be broken down into the following:

Resolve any unload conflicts (e.g. ignore and return if already unloaded)
Unload dependents
Call underlying unload callable

Resolve Unload Conflicts

If a plugin is already unloaded and unload() is called you may choose to pass conflict_strategy which can be one of “ignore” or “error”.

Unload Dependents

Before unloading, a plugin’s dependents (as appears in dependents), will first be unloaded. (This guarantees that a plugin’s requirements are up to date, and a plugin’s state fully encapsulates its consumers.)

Call Underlying Unload Callable

Finally, the loaded instance is passed into the underlying load_callable and returned.

Plugin Group

Use the PluginGroup class to load a group of plugins together:

from pyplugin import plugin, PluginGroup

@plugin
def plugin1():
    return 1

@plugin
def plugin2():
    return 2

# Option 1
my_group = PluginGroup(name="my_group")
my_group.add(plugin1)
my_group.add(plugin2)

# Option 2
my_group = PluginGroup(name="my_group", plugins=[plugin1, plugin2])
# or
my_group = PluginGroup(name="my_group", plugins=["plugin1", "plugin2"])

The group itself is a Plugin and its return type is a list of the loaded instances in the same group order:

assert my_group() == [1, 2]

The group also implements Sequence:

assert len(my_group) == 2

for plugin in my_group:
    ...

my_group.remove(plugin1)

Group Membership by Name

In the above example, we are able to add plugins to the group by name. Like plugin requirements, this will perform a lookup when it is time to load / unload. Unlike requirements, this will not “resolve” and cache to a specific Plugin upon first load but will remain dynamic.

Group Requirements

Like other plugins, groups can declare dependencies that must be loaded before attempting to load. If possible, these will be passed to the plugins in the group, being ignored if not declared in their signature (see safe_args in the load() method).

Overriding Load Order / Arguments

Plugin Groups can also be initialized with a load_callable like another plugin but in a special form:

from pyplugin import group

@group
def loader(plugins, *args, **kwargs):
    print("Before loading")
    yield plugins, args, kwargs
    print("After loading")

The yield statement allows to dynamically change the load order in addition to providing new load arguments. Similarly, this applies to the unload_callable being passed in plugins, instances, args, kwargs for unloading.

Group Type

By default, the type of the group is inferred from the element of the first registered / loaded plugin.

By setting enforce_type, this allows you to define a group that will only accept certain plugins.