Upgrade from Agents to Workers¶

Upgrading from agents to workers significantly enhances the experience of deploying flows. It simplifies the specification of each flow's infrastructure and runtime environment.

A worker is the fusion of an agent with an infrastructure block. Like agents, workers poll a work pool for flow runs that are scheduled to start. Like infrastructure blocks, workers are typed - they work with only one kind of infrastructure and they specify the default configuration for jobs submitted to that infrastructure.

Accordingly, workers are not a drop-in replacement for agents. Using workers requires deploying flows differently. In particular, deploying a flow with a worker does not involve specifying an infrastructure block. Instead, infrastructure configuration is specified on the work pool and passed to each worker that polls work from that pool.

This guide provides an overview of the differences between agents and workers. It also describes how to upgrade from agents to workers in just a few quick steps.

Enhancements¶

Agents to workers¶

Improved visibility into the status of each worker, including when a worker was started and when it last polled.
Better handling of race conditions for high availability use cases.

Agent infrastructure blocks to work pools¶

Work pools expose a base job template that enables an unprecedented level of customization and governance over workers in that pool.
Push work pools allow for flow execution without the need to host a worker at all.

New `prefect.yaml` file for managing multiple deployments¶

More easily define multiple deployments at once through a prefect.yaml file.
Prefect provides deployment actions that allow you to automatically build images for your flows.
Templating enables dryer deployment definitions.
You can use a deployment creation wizard now! 🧙

What's different¶

Command to build deployments:

prefect deployment build <entrypoint> --> prefect deploy

Prefect will now automatically detect flows in your repo and provide a wizard 🧙 to guide you through setting required attributes for your deployments.
Configuring remote flow code storage:

storage blocks --> pull action

And you can still use an existing storage block as your pull action! The more general pull steps specification allows for explicit customization of the job that defines each run (for example, by adding an additional step that sets a working directory).
Configuring flow run infrastructure:

run-infrastructure blocks --> typed work pool

Default infra config is now set on the typed work pool, and can be overwritten by both individual deployments or individual runs.
Managing multiple deployments:

Create and/or update many deployments at once through a prefect.yaml file.

What's similar¶

Storage blocks can be set as the pull action in a prefect.yaml file.
Infrastructure blocks have similar configuration fields as typed work pools.
Deployment-level infra-overrides operate in much the same way.

infra_override -> job_variable
The process for starting an agent and starting a worker in your environment are virtually identical.

prefect agent start --pool <work pool name> --> prefect worker start --pool <work pool name>

If you use infrastructure-as-code

Notice the command to start a worker is very similar to the command to start an agent. If you previously used terraform, a helm chart, or other infrastructure-as-code methods to start an agent, you should be able to continue using it for a worker, provided all uses of agent are changed to worker.

How to upgrade quickly¶

If you have existing deployments that use infrastructure blocks, you can quickly upgrade them to be compatible with workers by following these steps:

Create a work pool of the same type as the infrastructure block you are currently using.
1. Any work pool infrastructure type other than Prefect Agent will work.
2. Referencing the configuration you've set on the infrastructure block, set similar flow run infrastructure configuration on the work pool.
Start a worker to poll this work pool. You should see the command to start the worker as soon as you save your new work pool.
```
prefect worker start -p <work pool name>
```
Deploy your flow guided by the deployment creation wizard:

Always run prefect deploy commands from the root level of your repo!

With agents you might have had multiple deployment.yaml files, but under worker deployment patterns, each repo will have a single prefect.yaml file located at the root of the repo that contains deployment configuration for all flows in that repo.
```
prefect deploy
```
For step 4, select y on last prompt to save the configuration for the deployment.

Saving the configuration for your deployment will result in a prefect.yaml file populated with your first deployment. You can use this YAML file to edit and define multiple deployments for this repo.
In your prefect.yaml file, configure a pull action referencing whatever configuration you used as your storage block.
Continue to create more deployments that use workers by either adding them to the deployments list in the prefect.yaml file and/or by continuing to use the deployment creation wizard.