Adding exportable datasets
You can allow users to export your data from the UI's Export Data page.
Exportable dataset YAML files go on the leader. They describe data that the Pioreactor UI can export from the leader's database. They do not start the job that produces the data.
To add an exportable dataset, add a YAML file on the leader in ~/.pioreactor/plugins/exportable_datasets with the following information:
dataset_name: some_unique_dataset_name
default_order_by: timestamp # for example
description: A lovely description which shows up in the UI
display_name: A lovely name which shows up in the UI
has_experiment: true # does your SQL table have an experiment column.?
has_unit: true # does your SQL table have an pioreactor_unit column.?
source: app
table: the_target_table # optional: see note below
timestamp_columns:
- timestamp
always_partition_by_unit: false
column_descriptions: # optional, included in the exported schema.json
timestamp: UTC timestamp for this reading.
reading: Sensor reading after calibration.
column_units: # optional, included in the exported schema.json
reading: AU
query: SELECT * FROM the_target_table WHERE reading < 4 AND ... # optional: see note below
Either table or query is required. If you provide a table, the query field is ignored. The usecase for query is when you need to join multiple tables, use WHERE clauses, or do some other complex SQL operation.
After adding this file, visit the Export Data page in the UI.
Each export archive contains a top-level manifest.json and a schema.json inside each dataset folder. Add column_descriptions and column_units for columns whose meaning or measurement unit is not obvious; these fields make plugin datasets self-describing for people and downstream tools. Keys should match the column names returned by your table or query.