Turning your plugins into a Python package to share

If you'd like to contribute your plugin to the community, this is done easily by creating a Python package and uploading to PyPI. Let's walk through this!

Choosing a plugin name

Your plugin name should be all lowercase, and have underscores divide any words. Example: pireactor_relay_plugin is fine, but pioreactor-relay-plugin is not, nor is Pioreactor-Relay-Plugin.

However, your distribution package name should be lowercase and have dashes instead of underscores (it's a Python thing: I agree, this is confusing, I'll walk you through it). You can just replace any underscores with dashes: so our example distribution package name is pioreactor-relay-plugin.

Later in these steps, we'll reference these names as follows:

PLUGIN_NAME: should be the name in all lowercase with underscores. Example: pioreactor_relay_plugin
DISTRIBUTION-NAME: should be the name in all lowercase with dashes. Example: pioreactor-relay-plugin

Organizing your files

tip

Note that the way files are organized may depend on if your plugin is an automation or a background job. Plugins can install both automations and jobs.

Consider an example plugin: a plugin named pioreactor_relay_plugin that implements a background job which just turns on or off a PWM channel. Follow the file organization here: CamDavidsonPilon/pioreactor-relay-plugin.

Here's a general directory outline of how your files should be organized for a job:

📁 <DISTRIBUTION-NAME (with dashes)>
├─ 📁 <PLUGIN_NAME>
│  ├─ 📁 ui
│  │  ├─ 📁 contrib
│  │  │  ├─ 📁 jobs
│  │  │  │  ├─ 📝 <PLUGIN_NAME>.yaml
│  ├─ 📝 __init__.py
│  ├─ 📝 additional_config.ini
│  ├─ 📝 my_plugin_code.py
├─ 📝 LICENSE.txt
├─ 📝 MANIFEST.in
├─ 📝 README.md
├─ 📝 setup.py

The directory outline is very similar for an automation plugin — the only difference is the location of the .yaml file.

📁 <DISTRIBUTION-NAME with dashes>
├─ 📁 <PLUGIN_NAME>
│  ├─ 📁 ui
│  │  ├─ 📁 contrib
│  │  │  ├─ 📁 automations
│  │  │  │  ├─ 📁 <AUTOMATION TYPE (one of {dosing, led, temperature})>
│  │  │  │  │  ├─ 📝 <PLUGIN_NAME>.yaml
│  ├─ 📝 __init__.py
│  ├─ 📝 additional_config.ini
│  ├─ 📝 my_plugin_code.py
├─ 📝 LICENSE.txt
├─ 📝 README.md
├─ 📝 MANIFEST.in
├─ 📝 setup.py

Start by creating a new folder for your plugin, equal to the distribution name (the name with dashes). In our case, we named it pioreactor-relay-plugin. This top level folder will contain five important parts:

1. The subfolder `<PLUGIN_NAME>` containing your plugin's code

Within the top level directory, we created a sub-directory called <PLUGIN_NAME>.

2. A license text file, named `LICENSE.txt`

A common license for software is the MIT license.

3. A MANIFEST file, named `MANIFEST.in`

When creating a Python package, there's a default set of files that are included. To assure that our additional configuration and yaml files are included, create a MANIFEST.in file and paste the following:

recursive-include <PLUGIN_NAME>/ui/ *.yaml
include <PLUGIN_NAME>/additional_config.ini

4. A `README.md`

Write a few notes with general information on your plugin to guide users. Call out any additional installation steps, or hardware required. This is a markdown document.

5. A Python `setup.py` file

Create a Python file called setup.py and copy & paste the following. Make changes based on your own plugin information.

# -*- coding: utf-8 -*-
from setuptools import setup, find_packages

setup(
    name="<DISTRIBUTION-NAME (with dashes)>",
    version="<VERSION>",
    license_files = ('LICENSE.txt',),
    description="<DESCRIPTION OF PLUGIN>",
    long_description=open("README.md").read(),
    long_description_content_type="text/markdown",
    author_email="<EMAIL>",
    author="<NAME>",
    url="<A HOMEPAGE>",
    packages=find_packages(),
    include_package_data=True,
    install_requires=[], # PROVIDE OTHER PYTHON REQUIREMENTS, ex: "pioreactor>=23.6.0", "numpy>=1.0"
    entry_points={
        "pioreactor.plugins": "<PLUGIN_NAME> = <PLUGIN_NAME>"
    },
)

Contents of the subfolder, `<PLUGIN_NAME>`

1. Your plugins Python files

This Python file contains the core code for your plugin. If your plugin is implementing a background job, then there should be a function decorated with @click.command at the bottom of the file. See example here. For discovery reasons, this function's name should start with click_.

2. A Python `init.py` file

If implementing an automation: Import the Class of your automation file:

from <SUBFOLDER CONTAINING PLUGIN>.<PYTHON FILE NAME> import <CLASS NAME>

If implementing a job: This will contain an import statement such as the following:
```
from <PLUGIN_NAME>.<PYTHON FILE NAME> import <PLUGIN CLICK FUNCTION>
```
This imports the function within our plugin file that executes our plugin action.
Example for the relay plugin:
```
from pioreactor_relay_plugin.relay import click_relay
```
where click_relay is the function decorated with @click.command.

3. Optional: A configuration file, named `additional_config.ini`

This configuration file will contain additional configs that we want to add to our list of existing Configurations on the Pioreactor web interface. This file will be merged with the existing config.ini when the plugin is installed.

tip

A convention we've tried to follow is to use the section name convention of [<job_name>.config] or [<automation_name>.config] in the configuration files. For example, our relay job has [relay.config] in its additional_config.ini and settings under it.

4. Optional: Adding details for the UI

If implementing a job:

Within <PLUGIN_NAME> folder, create subfolders named ui/contrib/jobs. For a job, create a .yaml file that looks like the following format. The name of the yaml can be anything, but convention is to use the <job_name>.yaml:

---
display_name:  # human readable name
job_name: # `job_name` as defined in your Python file
display: # bool; true to display on the /Pioreactors card
source: # name of your plugin
description: # description of what your plugin does
published_settings:
  - key:  # as defined in Python
    unit: # unit (ml, lux, AU, etc.)
    label: # human readable name
    description: # description of your setting
    type:  # one of numeric, boolean, string, json
    default: # provide a default value, often null
    display: # bool; true to display on the /Pioreactors card
  - key: ...
    unit: ...
  ...

There are lots of examples of job yaml files here.

If implementing an automation:

In the case of creating an automation plugin, create subfolder(s) with ui/contrib/automations/<AUTOMATION TYPE>, where AUTOMATION TYPE is one of dosing, led, or temperature depending on your automation type. Create a yaml file with the following convention. The name of the yaml file can be anything, but by convention it's <automation_name>.yaml.

---
display_name:  # human readable name
automation_name: # automation name as defined in your Python files
source: # name of your plugin
description: # description of your plugin
fields:
  - key:  # as defined in Python
    unit: # unit of your key
    label: # human readable name
    description: # description of your key
  - key: ...
    unit: ...

There are lots of examples of automation yaml files here.

5. Optional: adding tables to the SQL store

You can also add a file called additional_sql.sql that will run against the SQLite database. For example, a CO₂ sensor may want to create a new table in the database to store its sensor data. It's additional_sql.sql may look like:

CREATE TABLE IF NOT EXISTS co2_readings (
    experiment               TEXT NOT NULL,
    pioreactor_unit          TEXT NOT NULL,
    timestamp                TEXT NOT NULL,
    co2_reading_ppm          REAL
);

You also need to tell Pioreactor software how to populate this table from your source of data. Include the following in your code such that it executes when the plugin is loaded:

a parser function that accepts a MQTT topic and payload, and returns a dictionary that maps to the new tables schema.
a TopicToParserToTable object is created with the MQTT topics to listen to, the parser, and the table name to load to. This TopicToParserToTable is provided to register_source_to_sink.
Example below for a CO2 sensor:

...
from pioreactor.background_jobs.leader.mqtt_to_db_streaming import produce_metadata
from pioreactor.background_jobs.leader.mqtt_to_db_streaming import register_source_to_sink
from pioreactor.background_jobs.leader.mqtt_to_db_streaming import TopicToParserToTable

...

def parser(topic, payload) -> dict:
    metadata = produce_metadata(topic)
    return {
        "experiment": metadata.experiment,
        "pioreactor_unit": metadata.pioreactor_unit,
        "timestamp": timing.current_utc_timestamp(),
        "co2_reading_ppm": float(payload),
    }


register_source_to_sink(
    TopicToParserToTable(
        ["pioreactor/+/+/scd_reading/co2", "pioreactor/+/+/co2_reading/co2"],
        parser,
        "co2_readings",
    )
)

...

note

You also need to add the following to your MANIFEST.in:

...
include <PLUGIN_NAME>/additional_sql.sql

See an example plugin that uses this idea here.

6. Optional: adding a custom chart to the UI

To add a chart that display real-time and historical data (provided by MQTT and SQL store respectively), you can do the following:

In a new folder named ui/contrib/charts in your project, add a YAML file as described here. The name of the file can by the chart_key field, append with .yaml.
In your additional_config.ini, add a new entry to be merged:

[ui.overview.charts]
<your chart key>=1

info

You'll need to make sure your database table has the necessary fields. See details under data_source in the docs here.

See an example plugin that uses this idea here.

7. Optional: adding post_install and pre_uninstall bash scripts.

If your plugin needs to edit the operating system, you can include either of the following files: post_install.sh, or pre_uninstall.sh. We have used these files to enable systemd services such that the job being installed will start at startup. Example post_install.sh and pre_uninstall.sh.

You'll need to append to your MANIFEST.in, too:

include <PLUGIN_NAME>/post_install.sh
include <PLUGIN_NAME>/pre_uninstall.sh

For an example, see the logs2slack plugin's repository.

8. Optional: only installing on leader Pioreactor.

Some plugins may only want to be installed on the leader. For example, if they are changing the UI or database. To signal that a plugin should only be installed on the leader (and no-op on workers), add an empty file named LEADER_ONLY to the folder.

You'll need to append to your MANIFEST.in, too:

include <PLUGIN_NAME>/LEADER_ONLY

For an example, see the logs2slack plugin's repository.

Create a Python package on PyPI

Create an account on https://pypi.org/. Make sure to verify your email.

On your command line, type the following:

pip install twine
pip install --upgrade build
python -m build --wheel     

This creates a .whl file in a dist/ folder (that will be created). Next, we upload the whl file to PyPI:

twine upload dist/<THE .WHL FILE>

You will then be prompted for a username and password. Use the credentials for your PyPI account. Then, your package is uploaded and viewable at the link provided in the output!

Note

Before you build a new wheel, it's good practice to clean up your previous build.
This can be done using python setup.py clean --all on the command line.

Installing your Python package on your cluster

A plugin can be installed through the command line on a leader using pio:

pio install-plugin <DISTRIBUTION-NAME OR PLUGIN_NAME>

To install a given plugin on the leader and all workers connected to the leader in a cluster, pios install-plugin can be used.

pios install-plugin <DISTRIBUTION-NAME OR PLUGIN_NAME>

To give your plugin further reach, we've provided an option to add it to the web interface. You will need to edit the plugins.json file within our Pioreactor repository, list-of-plugins.

There are two ways to do this:

Create an issue to have us edit the plugins.json file for you.
Fork from our repository to edit the plugins.json file, then create a pull request.

In both cases, we will evaluate your plugin to ensure code quality and all requirements are met.

Once your plugin is accepted, it will appear on the Plugins tab on the Pioreactor web interface. Users in the community can now easily click Install to download your plugin onto their Pioreactors!

Choosing a plugin name​

Organizing your files​

1. The subfolder <PLUGIN_NAME> containing your plugin's code​

2. A license text file, named LICENSE.txt​

3. A MANIFEST file, named MANIFEST.in​

4. A README.md​

5. A Python setup.py file​

Contents of the subfolder, <PLUGIN_NAME>​

1. Your plugins Python files​

2. A Python __init__.py file​

3. Optional: A configuration file, named additional_config.ini​

4. Optional: Adding details for the UI​

If implementing a job:​

If implementing an automation:​

5. Optional: adding tables to the SQL store​

6. Optional: adding a custom chart to the UI​

7. Optional: adding post_install and pre_uninstall bash scripts.​

8. Optional: only installing on leader Pioreactor.​

Create a Python package on PyPI​

Installing your Python package on your cluster​

Sharing your plugin with the community​