Skip to content

The components of Orion

Orion is a collection of services that form a dedicated orchestration environment:

  • the database is the persistent metadata store that holds flow and task run history, along with references to data produced by tasks and flows
  • the webserver serves a REST API backed by FastAPI that receives information emitted from workflows and additionally responds with orchestration instructions
  • a collection of services primarily focused on managing deployments
  • the UI is the flexible control plane for monitoring, configuring and analyzing your Prefect workflows

All at once

Below we break down each component of Orion individually; to quickly spin up all services simultaneously you can use the simple prefect orion start CLI command which should produce output similar to this:

$ prefect orion start
Starting Orion API...
INFO:     Started server process [36974]
INFO:     Waiting for application startup.
2021-09-29 13:05:44,361 - prefect.orion - INFO - Agent service scheduled to start in-app
2021-09-29 13:05:44,362 - prefect.orion - INFO - Scheduler service scheduled to start in-app
2021-09-29 13:05:44,362 - prefect.orion - INFO - MarkLateRuns service scheduled to start in-app
2021-09-29 13:05:44,385 - prefect.orion.services.marklateruns - DEBUG - About to run MarkLateRuns...
INFO:     Application startup complete.
INFO:     Uvicorn running on http://127.0.0.1:4200 (Press CTRL+C to quit)

The database

The database is the persistent layer that powers many of the features and functionality of Orion. Currently Orion supports the following databases:

  • SQLite: the default in Orion; best for lightweight single server deployments and requires essentially no setup
  • PostgreSQL: best for connecting to external databases but does require additional setup (such as Docker)

Whenever you first install Orion, your database will be located at ~/.prefect/orion.db. To configure this location, you can specify a connection URL with the PREFECT_ORION_DATABASE_CONNECTION_URL environment variable:

export PREFECT_ORION_DATABASE_CONNECTION_URL="sqlite+aiosqlite:////full/path/to/a/location/orion.db"

In-memory databases

One of the benefits of SQLite is in-memory database support; in-memory databases are only supported in Orion for testing purposes and are not compatible with multiprocessing. To use an in-memory sqlite database, set the following environment variable:

export PREFECT_ORION_DATABASE_CONNECTION_URL="sqlite+aiosqlite:///file::memory:?cache=shared&uri=true&check_same_thread=false"

Migrations

Recall that Orion is available as a technical preivew; this means that aspects of Orion's schema are still under active development and therefore upgrades should be considered destructive. As it nears official release, database migration guides and tooling will be available and documented.

If at any point in your testing you'd like to reset your database, you can run the prefect orion reset-db CLI command to do so. This will completely clear all data and reapply the schema.

The webserver

The Orion webserver can be stood up with a single CLI command:

$ prefect orion start --no-services
Starting Orion API...
INFO:     Started server process [84081]
INFO:     Waiting for application startup.
INFO:     Application startup complete.
INFO:     Uvicorn running on http://127.0.0.1:4200 (Press CTRL+C to quit)

There are numerous ways to begin exploring the API:

During normal operation is it not expected that you will need to interact with the API directly, as this is handled automatically for you within the Python client and the UI. In general it is expected that most users will spin up everything all at once with prefect orion start.

Orion services

Orion also ships with a collection of services that are also run with prefect orion start:

  • the Orion agent is a lightweight process responsible for submitting both scheduled and manually triggered deployments as subprocesses
  • the Orion scheduler prepares and creates runs for any scheduled deployments
  • the Orion MarkLateRuns service which updates late runs by placing them in a Late state

The UI

The Orion Dashboard comes prepackaged with the API when you serve it; by default it can be found at http://127.0.0.1:4200/:

The dashboard allows you to track and manage your flows, runs and deployments and additionally allows you to filter by names, tags, and other pieces of metadata to quickly find the information you are looking for.

Additional Reading

To learn more about the concepts presented here, check out the following resources: