Skip to content

Flow orchestration with Prefect

Up to this point, we've demonstrated running Prefect flows and tasks in a local environment using the ephemeral Prefect API. As you've seen, it's possible to run flexible, sophisticated workflows in this way without any further configuration.

Many users find running flows locally is useful for development, testing, and one-off or occasional workflow execution.

Where you begin leveraging the full power of Prefect is when you begin using Prefect for flow orchestration — building, running, and monitoring your workflows at scale.

Orchestration with Prefect helps you schedule and deploy workflows that run in the environments best suited to their execution. Orchestration helps you prevent and recover from failures, and view and manage the status of workflows, making your workflow:

  • Resilient
  • Observable
  • Configurable
  • Interactive
  • Automated

Prefect orchestration components

Designing workflows with Prefect starts with a few basic building blocks that you've already seen: flows and tasks.

Creating and running orchestrated workflows takes advantage of some additional Prefect components.

  • Prefect API server and orchestration engine receives state information from workflows and provides flow run instructions for executing deployments.
  • Prefect database provides a persistent metadata store that holds flow and task run history.
  • Prefect UI provides a control plane for monitoring, configuring, analyzing, and even creating ad-hoc runs of your workflows.
  • Storage for flow and task data lets you configure a persistent store for flow code and flow and task results.
  • Work queues and agents bridge the Prefect orchestration engine with a your execution environments, organizing work that agents can pick up to execute.

These Prefect components and services enable you to form what we call a dedicated orchestration environment. The same components and services enable you to orchestrate flows with either the open-source Prefect Orion API server and orchestration engine or Prefect Cloud.

Let's take a closer look at each component.

Prefect API server

The Prefect API server and orchestration engine is the central component of your orchestration environment.

Without you having to configure or run anything other than your flow code, the ephemeral Prefect API keeps track of the state of your Prefect flow and task runs.

When you run a Prefect API server instance with prefect orion start, the Prefect Orion orchestration engine keeps track of the state of your Prefect flow and task runs, and also lets you:

  • Create and update deployments
  • Execute scheduled flow runs for deployments automatically
  • Execute ad hoc flow runs
  • Configure and manage work queues
  • View logs generated by flow and task runs

...on top of monitoring the state of your flows and tasks.

If your execution environment is logged into Prefect Cloud, Prefect's orchestration-as-a-service platform provides all the capabilities of the Prefect Orion orchestration engine in a hosted manner.

Running the Prefect server

To take full advantage of the Prefect Orion orchestration engine and API server, you can spin up an instance at any time with the prefect orion start CLI command:

$ prefect orion start
Starting...

 ___ ___ ___ ___ ___ ___ _____    ___  ___ ___ ___  _  _
| _ \ _ \ __| __| __/ __|_   _|  / _ \| _ \_ _/ _ \| \| |
|  _/   / _|| _|| _| (__  | |   | (_) |   /| | (_) | .` |
|_| |_|_\___|_| |___\___| |_|    \___/|_|_\___\___/|_|\_|

Configure Prefect to communicate with the server with:

    prefect config set PREFECT_API_URL=http://127.0.0.1:4200/api

Check out the dashboard at http://127.0.0.1:4200

When the Prefect API server is running (either in a local environment or using Prefect Cloud), you can create and run orchestrated workflows including:

During normal operation, we don't expect that most users will need to interact with the Prefect API directly, as this is handled for you automatically by the Prefect Python client and the Prefect UI. Most users will spin up everything all at once with prefect orion start.

There are numerous ways to begin exploring the API:

To stop an instance of the Orion API server, simply CTRL+C to end the process in your terminal, or close the terminal session.

Scheduled flow runs require an Prefect API service

If you create deployments that have schedules, the scheduled flow runs will only attempt to start if the Prefect API server is running or your execution environment is logged into Prefect Cloud. The ephemeral Prefect API does not start scheduled flow runs.

Prefect database

The Prefect database persists data used by many features to orchestrate and track the state of your flow runs, including:

  • Flow and task state
  • Run history and logs
  • Deployments
  • Flow and task run concurrency limits
  • Storage locations for flow and task results
  • Work queue configuration and status

Currently, Prefect supports configuring the following for use as the database:

  • SQLite
  • PostgreSQL

A local SQLite database is the default for Prefect, and a local SQLite database is configured on installation. We recommend SQLite for lightweight, single-server deployments. SQLite requires essentially no setup.

PostgreSQL is good for connecting to external databases, but does require additional setup (such as Docker).

Prefect Cloud provides its own hosted database.

Configuring the Prefect database

Prefect creates a SQLite database, but you can configure your own database.

When you first install Prefect, your database will be located at ~/.prefect/orion.db. To configure this location, you can specify a connection URL with the PREFECT_ORION_DATABASE_CONNECTION_URL environment variable:

$ export PREFECT_ORION_DATABASE_CONNECTION_URL="sqlite+aiosqlite:////full/path/to/a/location/orion.db"
If at any point in your testing you'd like to reset your database, run the prefect orion database reset CLI command:

$ prefect orion database reset

This will completely clear all data and reapply the schema.

See the Database documentation for further details on choosing and configuring the Prefect database.

Prefect UI and dashboard

The Prefect UI and dashboard comes prepackaged with the API when you serve it. By default it can be found at http://127.0.0.1:4200/:

Prefect Orion UI dashboard.

The UI enables you to track and manage your flows, runs, and deployments and additionally allows you to filter by names, tags, and other metadata to quickly find the information you are looking for.

The UI displays many useful insights about your flow runs, including:

  • Flow run summaries
  • Deployed flow details
  • Scheduled flow runs
  • Warnings for late or failed runs
  • Task run details
  • Radar flow and task dependency visualizer
  • Logs

You can also use the Prefect UI to create ad hoc flow runs from deployments, configure and manage work queues, and more.

See the Prefect UI & Cloud documentation for more information about using the Prefect UI.

Storage for flow and task data

Prefect lets you configure separate storage to persist flow code, task results, and flow results.

If you don't configure other storage, Prefect uses temporary local storage. Temporary local storage works fine for many local flow runs, but if you run flows using Docker or Kubernetes, you must set up remote storage.

Prefect currently supports AWS S3, Azure Blob Storage, and Google Cloud Storage.

To learn more about storage configuration, see the Storage documentation. The Running flows in Docker and Running flows in Kubernetes tutorials provide a brief examples of configuring remote storage.

Work queues and agents

Work queues and agents bridge the Prefect orchestration engine and API with your local execution environments.

  • Work queues are configured on the server. They contain the logic about which flows to run and how to run them.
  • Agents run in a local execution environment. They pick up work from queues and execute the flows.

There is no default global work queue or agent, so to orchestrate flow runs you need to configure at least one work queue and agent.

You can create work queues:

Agents are configured to pull work from a specific work queue. You'll use the CLI to start an agent in your execution environment. If you configure work queues in the Prefect UI, the work queue panel provides the CLI command: you can simply copy the entire command and run it in your execution environment.

The Deployments tutorial walks through the steps for configuring a work queue and starting an agent.

Next steps

Continue on to the Deployments tutorial to start seeing flow orchestration with Prefect in action.