Deploying flows to work pools and workers

In this guide, we will dive deep into configuring a deployment for a worker and work pool using prefect deploy.

Deployments overview

All Prefect flow runs are tracked by the API. The API does not require prior registration of flows. With Prefect, you can call a flow locally or on a remote environment and it will be tracked.

Creating a deployment for a Prefect workflow means packaging workflow code, settings, and infrastructure configuration so that the workflow can be managed via the Prefect API and run remotely by a Prefect agent.

The following diagram provides a high-level overview of the conceptual elements involved in defining a deployment and executing a flow run based on that deployment.

%%{
  init: {
    'theme': 'base',
    'themeVariables': {
      'fontSize': '19px'
    }
  }
}%%

flowchart LR
    F("<div style='margin: 5px 10px 5px 5px;'>Flow Code</div>"):::yellow -.-> A("<div style='margin: 5px 10px 5px 5px;'>Deployment Definition</div>"):::gold
    subgraph Server ["<div style='width: 150px; text-align: center; margin-top: 5px;'>Prefect API</div>"]
        D("<div style='margin: 5px 10px 5px 5px;'>Deployment</div>"):::green
    end
    subgraph Remote Storage ["<div style='width: 160px; text-align: center; margin-top: 5px;'>Remote Storage</div>"]
        B("<div style='margin: 5px 6px 5px 5px;'>Flow</div>"):::yellow
    end
    subgraph Infrastructure ["<div style='width: 150px; text-align: center; margin-top: 5px;'>Infrastructure</div>"]
        G("<div style='margin: 5px 10px 5px 5px;'>Flow Run</div>"):::blue
    end

    A --> D
    D --> E("<div style='margin: 5px 10px 5px 5px;'>Worker</div>"):::red
    B -.-> E
    A -.-> B
    E -.-> G

    classDef gold fill:goldenrod,stroke:goldenrod,stroke-width:4px
    classDef yellow fill:gold,stroke:gold,stroke-width:4px
    classDef gray fill:lightgray,stroke:lightgray,stroke-width:4px
    classDef blue fill:blue,stroke:blue,stroke-width:4px,color:white
    classDef green fill:green,stroke:green,stroke-width:4px,color:white
    classDef red fill:red,stroke:red,stroke-width:4px,color:white
    classDef dkgray fill:darkgray,stroke:darkgray,stroke-width:4px,color:white

When creating a deployment, we must answer two basic questions:

• What instructions does a worker need to set up an execution environment for our flow? For example, a flow may have Python requirements, unique Kubernetes settings, or Docker networking configuration.
• How should the flow code be accessed?

A deployment additionally enables you to:

• Schedule flow runs.
• Specify event triggers for flow runs.
• Assign a work queue name to delegate deployment flow runs to work queues.
• Assign one or more tags to organize your deployments and flow runs. You can use those tags as filters in the Prefect UI.
• Assign custom parameter values for flow runs based on the deployment.
• Create ad-hoc flow runs from the API or Prefect UI.
• Upload flow files to a defined storage location for retrieval at run time.
• Specify run time infrastructure for flow runs, such as Docker or Kubernetes configuration.

Managing deployments

You can manage one or more flow deployments by simply adding prefect.yaml file to your working directory:

• prefect.yaml: a YAML file describing base settings for your deployments, procedural steps for preparing deployments, as well as instructions for preparing the execution environment for a deployment run

You can initialize your deployment configuration, which creates the prefect.yaml file, by running the CLI command prefect init in any directory or repository that stores your flow code.

Deployment configuration recipes

Prefect ships with many off-the-shelf "recipes" that allow you to get started with more structure within your prefect.yaml file; run prefect init to be prompted with available recipes in your installation. You can provide a recipe name in your initialization command with the --recipe flag, otherwise Prefect will attempt to guess an appropriate recipe based on the structure of your working directory (for example if you initialize within a git repository, Prefect will use the git recipe).

The prefect.yaml file contains deployment configuration for deployments created from this file, default instructions for how to build and push any necessary code artifacts (such as Docker images), and default instructions for pulling a deployment in remote execution environments (e.g., cloning a GitHub repository).

Any deployment configuration can be overridden via options available on the prefect deploy CLI command when creating a deployment.

The base structure for prefect.yaml is as follows:

# generic metadata
prefect-version: null
name: null

# preparation steps
build: null
push: null

# runtime steps
pull: null

# deployment configurations
deployments:
  - # base metadata
    name: null
    version: null
    tags: []
    description: null
    schedule: null

    # flow-specific fields
    entrypoint: null
    parameters: {}

    # infra-specific fields
    work_pool:
      name: null
      work_queue_name: null
      job_variables: {}

The metadata fields are always pre-populated for you. These fields are for bookkeeping purposes only. The other sections are pre-populated based on recipe; if no recipe is provided, Prefect will attempt to guess an appropriate one based on local configuration.

You can create deployments via the CLI command prefect deploy without ever needing to alter the deployments section of your prefect.yaml file — the prefect deploy command will help in deployment creation via interactive prompts. The prefect.yaml file facilitates version-controlling your deployment configuration and managing multiple deployments.

Deployment actions

Deployment actions defined in your prefect.yaml file control the lifecycle of the creation and execution of your deployments. The three actions available are build, push, and pull. pull is the only required deployment action — it is used to define how Prefect will pull your deployment in remote execution environments.

Each action is defined as a list of steps that are executing in sequence.

Each step has the following format:

section:
  - prefect_package.path.to.importable.step:
      id: "step-id" # optional
      requires: "pip-installable-package-spec" # optional
      kwarg1: value
      kwarg2: more-values

Every step can optionally provide a requires field that Prefect will use to auto-install in the event that the step cannot be found in the current environment. Each step can also specify an id for the step which is used when referencing step outputs in later steps. The additional fields map directly onto Python keyword arguments to the step function. Within a given section, steps always run in the order that they are provided within the prefect.yaml file.

Deployment Instruction Overrides

build, push, and pull sections can all be overridden on a per-deployment basis by defining build, push, and pull fields within a deployment definition in the prefect.yaml file.

The prefect deploy command will use any build, push, or pull instructions provided in a deployment's definition in the prefect.yaml file.

This capability is useful with multiple deployments that require different deployment instructions.

The build action

The build section of prefect.yaml is where any necessary side effects for running your deployments are built - the most common type of side effect produced here is a Docker image. If you initialize with the docker recipe, you will be prompted to provide required information, such as image name and tag:

$ prefect init --recipe docker
>> image_name: < insert image name here >
>> tag: < insert image tag here >

Use --field to avoid the interactive experience

We recommend that you only initialize a recipe when you are first creating your deployment structure, and afterwards store your configuration files within version control. However, sometimes you may need to initialize programmatically and avoid the interactive prompts.
To do so, provide all required fields for your recipe using the --field flag:

$ prefect init --recipe docker \
    --field image_name=my-repo/my-image \
    --field tag=my-tag

build:
- prefect_docker.deployments.steps.build_docker_image:
    requires: prefect-docker>=0.3.0
    image_name: my-repo/my-image
    tag: my-tag
    dockerfile: auto
    push: true

Once you've confirmed that these fields are set to their desired values, this step will automatically build a Docker image with the provided name and tag and push it to the repository referenced by the image name.
As the prefect-docker package documentation notes, this step produces a few fields that can optionally be used in future steps or within prefect.yaml as template values.
It is best practice to use {{ image }} within prefect.yaml (specifically the work pool's job variables section) so that you don't risk having your build step and deployment specification get out of sync with hardcoded values.

Note

Note that in the build step example above, we relied on the prefect-docker package; in cases that deal with external services, additional packages are often required and will be auto-installed for you.

Pass output to downstream steps

Each deployment action can be composed of multiple steps. For example, if you wanted to build a Docker image tagged with the current commit hash, you could use the run_shell_script step and feed the output into the build_docker_image step:

build:
    - prefect.deployments.steps.run_shell_script:
        id: get-commit-hash
        script: git rev-parse --short HEAD
        stream_output: false
    - prefect_docker.deployments.steps.build_docker_image:
        requires: prefect-docker
        image_name: my-image
        image_tag: "{{ get-commit-hash.stdout }}"
        dockerfile: auto

Note that the id field is used in the run_shell_script step so that its output can be referenced in the next step.

The Push Action

The push section is most critical for situations in which code is not stored on persistent filesystems or in version control. In this scenario, code is often pushed and pulled from a Cloud storage bucket of some kind (e.g., S3, GCS, Azure Blobs, etc.). The push section allows users to specify and customize the logic for pushing this code repository to arbitrary remote locations.

For example, a user wishing to store their code in an S3 bucket and rely on default worker settings for its runtime environment could use the s3 recipe:

$ prefect init --recipe s3
>> bucket: < insert bucket name here >

Inspecting our newly created prefect.yaml file we find that the push and pull sections have been templated out for us as follows:

push:
  - prefect_aws.deployments.steps.push_to_s3:
      id: push-code
      requires: prefect-aws>=0.3.0
      bucket: my-bucket
      folder: project-name
      credentials: null

pull:
  - prefect_aws.deployments.steps.pull_from_s3:
      requires: prefect-aws>=0.3.0
      bucket: my-bucket
      folder: "{{ push-code.folder }}"
      credentials: null

The bucket has been populated with our provided value (which also could have been provided with the --field flag); note that the folder property of the push step is a template - the pull_from_s3 step outputs both a bucket value as well as a folder value that can be used to template downstream steps. Doing this helps you keep your steps consistent across edits.

As discussed above, if you are using blocks, the credentials section can be templated with a block reference for secure and dynamic credentials access:

push:
  - prefect_aws.deployments.steps.push_to_s3:
      requires: prefect-aws>=0.3.0
      bucket: my-bucket
      folder: project-name
      credentials: "{{ prefect.blocks.aws-credentials.dev-credentials }}"

Anytime you run prefect deploy, this push section will be executed upon successful completion of your build section. For more information on the mechanics of steps, see below.

The pull action

The pull section is the most important section within the prefect.yaml file. It contains instructions for preparing your flows for a deployment run. These instructions will be executed each time a deployment created within this folder is run via a worker.

There are three main types of steps that typically show up in a pull section:

• set_working_directory: this step simply sets the working directory for the process prior to importing your flow
• git_clone: this step clones the provided repository on the provided branch
• pull_from_{cloud}: this step pulls the working directory from a Cloud storage location (e.g., S3)

Use block and variable references

All block and variable references within your pull step will remain unresolved until runtime and will be pulled each time your deployment is run. This allows you to avoid storing sensitive information insecurely; it also allows you to manage certain types of configuration from the API and UI without having to rebuild your deployment every time.

Below is an example of how to use an existing GitHubCredentials block to clone a private GitHub repository:

pull:
    - prefect.deployments.steps.git_clone:
        repository: https://github.com/org/repo.git
        credentials: "{{ prefect.blocks.github-credentials.my-credentials }}"

Alternatively, you can specify a BitBucketCredentials or GitLabCredentials block to clone from Bitbucket or GitLab. In lieu of a credentials block, you can also provide a GitHub, GitLab, or Bitbucket token directly to the 'access_token` field. You can use a Secret block to do this securely:

pull:
    - prefect.deployments.steps.git_clone:
        repository: https://bitbucket.org/org/repo.git
        access_token: "{{ prefect.blocks.secret.bitbucket-token }}"

Utility steps

Utility steps can be used within a build, push, or pull action to assist in managing the deployment lifecycle:

• run_shell_script allows for the execution of one or more shell commands in a subprocess, and returns the standard output and standard error of the script. This step is useful for scripts that require execution in a specific environment, or those which have specific input and output requirements.

Here is an example of retrieving the short Git commit hash of the current repository to use as a Docker image tag:

build:
    - prefect.deployments.steps.run_shell_script:
        id: get-commit-hash
        script: git rev-parse --short HEAD
        stream_output: false
    - prefect_docker.deployments.steps.build_docker_image:
        requires: prefect-docker>=0.3.0
        image_name: my-image
        tag: "{{ get-commit-hash.stdout }}"
        dockerfile: auto

Provided environment variables are not expanded by default

To expand environment variables in your shell script, set expand_env_vars: true in your run_shell_script step. For example:

- prefect.deployments.steps.run_shell_script:
    id: get-user
    script: echo $USER
    stream_output: true
    expand_env_vars: true

Without expand_env_vars: true, the above step would return a literal string $USER instead of the current user.

• pip_install_requirements installs dependencies from a requirements.txt file within a specified directory.

Below is an example of installing dependencies from a requirements.txt file after cloning:

pull:
    - prefect.deployments.steps.git_clone:
        id: clone-step  # needed in order to be referenced in subsequent steps
        repository: https://github.com/org/repo.git
    - prefect.deployments.steps.pip_install_requirements:
        directory: {{ clone-step.directory }}  # `clone-step` is a user-provided `id` field
        requirements_file: requirements.txt

Below is an example that retrieves an access token from a 3rd party Key Vault and uses it in a private clone step:

pull:
- prefect.deployments.steps.run_shell_script:
    id: get-access-token
    script: az keyvault secret show --name <secret name> --vault-name <secret vault> --query "value" --output tsv
    stream_output: false
- prefect.deployments.steps.git_clone:
    repository: https://bitbucket.org/samples/deployments.git
    branch: master
    access_token: "{{ get-access-token.stdout }}"

You can also run custom steps by packaging them. In the example below, retrieve_secrets is a custom python module that has been packaged into the default working directory of a Docker image (which is /opt/prefect by default). main is the function entry point, which returns an access token (e.g. return {"access_token": access_token}) like the preceding example, but utilizing the Azure Python SDK for retrieval.

- retrieve_secrets.main:
    id: get-access-token
- prefect.deployments.steps.git_clone:
    repository: https://bitbucket.org/samples/deployments.git
    branch: master
    access_token: '{{ get-access-token.access_token }}'

Templating options

Values that you place within your prefect.yaml file can reference dynamic values in several different ways:

• step outputs: every step of both build and push produce named fields such as image_name; you can reference these fields within prefect.yaml and prefect deploy will populate them with each call. References must be enclosed in double brackets and be of the form "{{ field_name }}"
• blocks: Prefect blocks can also be referenced with the special syntax {{ prefect.blocks.block_type.block_slug }}. It is highly recommended that you use block references for any sensitive information (such as a GitHub access token or any credentials) to avoid hardcoding these values in plaintext
• variables: Prefect variables can also be referenced with the special syntax {{ prefect.variables.variable_name }}. Variables can be used to reference non-sensitive, reusable pieces of information such as a default image name or a default work pool name.
• environment variables: you can also reference environment variables with the special syntax {{ $MY_ENV_VAR }}. This is especially useful for referencing environment variables that are set at runtime.

As an example, consider the following prefect.yaml file:

build:
- prefect_docker.deployments.steps.build_docker_image:
    id: build-image
    requires: prefect-docker>=0.3.0
    image_name: my-repo/my-image
    tag: my-tag
    dockerfile: auto
    push: true

deployments:
  - # base metadata
    name: null
    version: "{{ build-image.tag }}"
    tags:
        - "{{ $my_deployment_tag }}"
        - "{{ prefect.variables.some_common_tag }}"
    description: null
    schedule: null

    # flow-specific fields
    entrypoint: null
    parameters: {}

    # infra-specific fields
    work_pool:
        name: "my-k8s-work-pool"
        work_queue_name: null
        job_variables:
            image: "{{ build-image.image }}"
            cluster_config: "{{ prefect.blocks.kubernetes-cluster-config.my-favorite-config }}"

So long as our build steps produce fields called image_name and tag, every time we deploy a new version of our deployment, the {{ build-image.image }} variable will be dynamically populated with the relevant values.

Docker step

The most commonly used build step is prefect_docker.deployments.steps.build_docker_image which produces both the image_name and tag fields.

For an example, check out the deployments tutorial.

Deployment Configurations

Each prefect.yaml file can have multiple deployment configurations that control the behavior of created deployments. These deployments can be managed independently of one another, allowing you to deploy the same flow with different configurations in the same codebase.

Working With Multiple Deployments

Prefect supports multiple deployment declarations within the prefect.yaml file. This method of declaring multiple deployments allows the configuration for all deployments to be version controlled and deployed with a single command.

New deployment declarations can be added to the prefect.yaml file by adding a new entry to the deployments list. Each deployment declaration must have a unique name field which is used to select deployment declarations when using the prefect deploy command.

For example, consider the following prefect.yaml file:

build: ...
push: ...
pull: ...

deployments:
  - name: deployment-1
    entrypoint: flows/hello.py:my_flow
    parameters:
        number: 42,
        message: Don't panic!
    work_pool:
        name: my-process-work-pool
        work_queue_name: primary-queue

  - name: deployment-2
    entrypoint: flows/goodbye.py:my_other_flow
    work_pool:
        name: my-process-work-pool
        work_queue_name: secondary-queue

  - name: deployment-3
    entrypoint: flows/hello.py:yet_another_flow
    work_pool:
        name: my-docker-work-pool
        work_queue_name: tertiary-queue

This file has three deployment declarations, each referencing a different flow. Each deployment declaration has a unique name field and can be deployed individually by using the --name flag when deploying.

For example, to deploy deployment-1 you would run:

$ prefect deploy --name deployment-1

To deploy multiple deployments you can provide multiple --name flags:

$ prefect deploy --name deployment-1 --name deployment-2

To deploy multiple deployments with the same name, you can prefix the deployment name with its flow name:

$ prefect deploy --name my_flow/deployment-1 --name my_other_flow/deployment-1

To deploy all deployments you can use the --all flag:

$ prefect deploy --all

To deploy deployments that match a pattern you can run:

$ prefect deploy -n my-flow/* -n *dev/my-deployment -n dep*prod

The above command will deploy all deployments from the flow my-flow, all flows ending in dev with a deployment named my-deployment, and all deployments starting with dep and ending in prod.

CLI Options When Deploying Multiple Deployments

When deploying more than one deployment with a single prefect deploy command, any additional attributes provided via the CLI will be ignored.

To provide overrides to a deployment via the CLI, you must deploy that deployment individually.

Reusing configuration across deployments

Because a prefect.yaml file is a standard YAML file, you can use YAML aliases to reuse configuration across deployments.

This functionality is useful when multiple deployments need to share the work pool configuration, deployment actions, or other configurations.

You can declare a YAML alias by using the &{alias_name} syntax and insert that alias elsewhere in the file with the *{alias_name} syntax. When aliasing YAML maps, you can also override specific fields of the aliased map by using the <<: *{alias_name} syntax and adding additional fields below.

We recommend adding a definitions section to your prefect.yaml file at the same level as the deployments section to store your aliases.

For example, consider the following prefect.yaml file:

build: ...
push: ...
pull: ...

definitions:
    work_pools:
        my_docker_work_pool: &my_docker_work_pool
            name: my-docker-work-pool
            work_queue_name: default
            job_variables:
                image: "{{ build-image.image }}"
    schedules:
        every_ten_minutes: &every_10_minutes
            interval: 600
    actions:
        docker_build: &docker_build
            - prefect_docker.deployments.steps.build_docker_image: &docker_build_config
                id: build-image
                requires: prefect-docker>=0.3.0
                image_name: my-example-image
                tag: dev
                dockerfile: auto
                push: true

deployments:
  - name: deployment-1
    entrypoint: flows/hello.py:my_flow
    schedule: *every_10_minutes
    parameters:
        number: 42,
        message: Don't panic!
    work_pool: *my_docker_work_pool
    build: *docker_build # Uses the full docker_build action with no overrides

  - name: deployment-2
    entrypoint: flows/goodbye.py:my_other_flow
    work_pool: *my_docker_work_pool
    build:
        - prefect_docker.deployments.steps.build_docker_image:
            <<: *docker_build_config # Uses the docker_build_config alias and overrides the dockerfile field
            dockerfile: Dockerfile.custom

  - name: deployment-3
    entrypoint: flows/hello.py:yet_another_flow
    schedule: *every_10_minutes
    work_pool:
        name: my-process-work-pool
        work_queue_name: primary-queue

In the above example, we are using YAML aliases to reuse work pool, schedule, and build configuration across multiple deployments:

• deployment-1 and deployment-2 are using the same work pool configuration
• deployment-1 and deployment-3 are using the same schedule
• deployment-1 and deployment-2 are using the same build deployment action, but deployment-2 is overriding the dockerfile field to use a custom Dockerfile

Deployment Declaration Reference

Deployment Fields

Below are fields that can be added to each deployment declaration.

Property	Description
name	The name to give to the created deployment. Used with the prefect deploy command to create or update specific deployments.
version	An optional version for the deployment.
tags	A list of strings to assign to the deployment as tags.
description	An optional description for the deployment.
schedule	An optional schedule to assign to the deployment. Fields for this section are documented in the Schedule Fields section.
triggers	An optional array of triggers to assign to the deployment
entrypoint	Required path to the .py file containing the flow you want to deploy (relative to the root directory of your development folder) combined with the name of the flow function. Should be in the format path/to/file.py:flow_function_name.
parameters	Optional default values to provide for the parameters of the deployed flow. Should be an object with key/value pairs.
enforce_parameter_schema	Boolean flag that determines whether the API should validate the parameters passed to a flow run against the parameter schema generated for the deployed flow.
work_pool	Information on where to schedule flow runs for the deployment. Fields for this section are documented in the Work Pool Fields section.

Schedule Fields

Below are fields that can be added to a deployment declaration's schedule section.

Property	Description
interval	Number of seconds indicating the time between flow runs. Cannot be used in conjunction with cron or rrule.
anchor_date	Datetime string indicating the starting or "anchor" date to begin the schedule. If no anchor_date is supplied, the current UTC time is used. Can only be used with interval.
timezone	String name of a time zone, used to enforce localization behaviors like DST boundaries. See the IANA Time Zone Database for valid time zones.
cron	A valid cron string. Cannot be used in conjunction with interval or rrule.
day_or	Boolean indicating how croniter handles day and day_of_week entries. Must be used with cron. Defaults to True.
rrule	String representation of an RRule schedule. See the rrulestr examples for syntax. Cannot be used in conjunction with interval or cron.

For more information about schedules, see the Schedules concept doc.

Work Pool Fields

Below are fields that can be added to a deployment declaration's work_pool section.

Property	Description
name	The name of the work pool to schedule flow runs in for the deployment.
work_queue_name	The name of the work queue within the specified work pool to schedule flow runs in for the deployment. If not provided, the default queue for the specified work pool will be used.
job_variables	Values used to override the default values in the specified work pool's base job template. Maps directly to a created deployments infra_overrides attribute.

Deployment mechanics

Anytime you run prefect deploy in a directory that contains a prefect.yaml file, the following actions are taken in order:

• The prefect.yaml file is loaded. First, the build section is loaded and all variable and block references are resolved. The steps are then run in the order provided.
• Next, the push section is loaded and all variable and block references are resolved; the steps within this section are then run in the order provided
• Next, the pull section is templated with any step outputs but is not run. Note that block references are not hydrated for security purposes - block references are always resolved at runtime
• Next, all variable and block references are resolved with the deployment declaration. All flags provided via the prefect deploy CLI are then overlaid on the values loaded from the file.
• The final step occurs when the fully realized deployment specification is registered with the Prefect API

Deployment Instruction Overrides

The build, push, and pull sections in deployment definitions take precedence over the corresponding sections above them in prefect.yaml.

Each time a step is run, the following actions are taken in order:

• The step's inputs and block / variable references are resolved (see the templating documentation above for more details).
• The step's function is imported; if it cannot be found, the special requires keyword is used to install the necessary packages
• The step's function is called with the resolved inputs.
• The step's output is returned and used to resolve inputs for subsequent steps.