Run an Agent with Azure Container Instances¶
Microsoft Azure Container Instances (ACI) provides a convenient and simple service for quickly spinning up a Docker container that can host a Prefect Agent and execute flow runs.
Prerequisites¶
To follow this quickstart, you'll need the following:
- A Prefect Cloud account
- A Prefect Cloud API key (Prefect Cloud organizations may use a service account API key)
- A Microsoft Azure account
- Azure CLI installed and authenticated
Create a resource group¶
Like most Azure resources, ACI applications must live in a resource group. If you don’t already have a resource group you’d like to use, create a new one by running the az group create
command. For example, this example creates a resource group called prefect-agents
in the eastus
region:
az group create --name prefect-agents --location eastus
Feel free to change the group name or location to match your use case. You can also run az account list-locations -o table
to see all available resource group locations for your account.
Create the container instance¶
Prefect provides pre-configured Docker images you can use to quickly stand up a container instance. These Docker images include Python and Prefect. For example, the image prefecthq/prefect:2-python3.10
includes the latest release version of Prefect and Python 3.10.
To create the container instance, use the az container create
command. This example shows the syntax, but you'll need to provide the correct values for [ACCOUNT-ID]
,[WORKSPACE-ID]
, [API-KEY]
, and any dependencies you need to pip install
on the instance. These options are discussed below.
az container create \
--resource-group prefect-agents \
--name prefect-agent-example \
--image prefecthq/prefect:2-python3.10 \
--secure-environment-variables PREFECT_API_URL='https://api.prefect.cloud/api/accounts/[ACCOUNT-ID]/workspaces/[WORKSPACE-ID]' PREFECT_API_KEY='[API-KEY]' \
--command-line "/bin/bash -c 'pip install adlfs s3fs requests pandas; prefect agent start -p default-agent-pool -q test'"
When the container instance is running, go to Prefect Cloud and select the Work Pools page. Select default-agent-pool, then select the Queues tab to see work queues configured on this work pool. When the container instance is running and the agent has started, the test
work queue displays "Healthy" status. This work queue and agent are ready to execute deployments configured to run on the test
queue.
Agents and queues
The agent running in this container instance can now pick up and execute flow runs for any deployment configured to use the test
queue on the default-agent-pool
work pool.
Container create options¶
Let's break down the details of the az container create
command used here.
The az container create command
creates a new ACI container.
--resource-group prefect-agents
tells Azure which resource group the new container is created in. Here, the examples uses the prefect-agents
resource group created earlier.
--name prefect-agent-example
determines the container name you will see in the Azure Portal. You can set any name you’d like here to suit your use case, but container instance names must be unique in your resource group.
--image prefecthq/prefect:2-python3.10
tells ACI which Docker images to run. The script above pulls a public Prefect image from Docker Hub.
You can also build custom images and push them to a public container registry so ACI can access them. Or you can push your image to a private Azure Container Registry and use it to create a container instance.
--secure-environment-variables
sets environment variables that are only visible from inside the container. They do not show up when viewing the container’s metadata. You'll populate these environment variables with a few pieces of information to configure the execution environment of the container instance so it can communicate with your Prefect Cloud workspace:
- A Prefect Cloud [
PREFECT_API_KEY
]/concepts/settings/#prefect_api_key) value specifying the API key used to authenticate with your Prefect Cloud workspace. (Prefect Cloud organizations may use a service account API key.) - The
PREFECT_API_URL
value specifying the API endpoint of your Prefect Cloud workspace.
--command-line
lets you override the container’s normal entry point and run a command instead. The script above uses this section to install the adlfs
pip package so it can read flow code from Azure Blob Storage, along with s3fs
, pandas
, and requests
. It then runs the Prefect agent, in this case using the default work pool and a test
work queue. If you want to use a different work pool or queue, make sure to change these values appropriately.
Create a deployment¶
Following the example of the Flow deployments tutorial, let's create a deployment that can be executed by the agent on this container instance.
In an environment where you have installed Prefect, create a new folder called health_test
, and within it create a new file called health_flow.py
containing the following code.
import prefect
from prefect import task, flow
from prefect import get_run_logger
@task
def say_hi():
logger = get_run_logger()
logger.info("Hello from the Health Check Flow! 👋")
@task
def log_platform_info():
import platform
import sys
from prefect.server.api.server import SERVER_API_VERSION
logger = get_run_logger()
logger.info("Host's network name = %s", platform.node())
logger.info("Python version = %s", platform.python_version())
logger.info("Platform information (instance type) = %s ", platform.platform())
logger.info("OS/Arch = %s/%s", sys.platform, platform.machine())
logger.info("Prefect Version = %s 🚀", prefect.__version__)
logger.info("Prefect API Version = %s", SERVER_API_VERSION)
@flow(name="Health Check Flow")
def health_check_flow():
hi = say_hi()
log_platform_info(wait_for=[hi])
Now create a deployment for this flow script, making sure that it's configured to use the test
queue on the default-agent-pool
work pool.
prefect deployment build --infra process --storage-block azure/flowsville/health_test --name health-test --pool default-agent-pool --work-queue test --apply health_flow.py:health_check_flow
Once created, any flow runs for this deployment will be picked up by the agent running on this container instance.
Infrastructure and storage
This Prefect deployment example was built using the Process
infrastructure type and Azure Blob Storage.
You might wonder why your deployment needs process infrastructure rather than DockerContainer
infrastructure when you are deploying a Docker image to ACI.
A Prefect deployment’s infrastructure type describes how you want Prefect agents to run flows for the deployment. With DockerContainer
infrastructure, the agent will try to use Docker to spin up a new container for each flow run. Since you’ll be starting your own container on ACI, you don’t need Prefect to do it for you. Specifying process infrastructure on the deployment tells Prefect you want to agent to run flows by starting a process in your ACI container.
You can use any storage type as long as you've configured a block for it before creating the deployment.
Cleaning up¶
Note that ACI instances may incur usage charges while running, but must be running for the agent to pick up and execute flow runs.
To stop a container, use the az container stop
command:
az container stop --resource-group prefect-agents --name prefect-agent-example
To delete a container, use the az container delete
command:
az container delete --resource-group prefect-agents --name prefect-agent-example