Skip to content

Getting Started with Prefect Cloud

The following sections will get you set up and using Prefect Cloud, using these steps:

  1. Sign in or register a Prefect Cloud account.
  2. Create workspaces for your account.
  3. Create an API key to authorize a local execution environment.
  4. Configure Orion settings to use Prefect Cloud.
  5. Configure storage.
  6. Run a flow and view flow results in Prefect Cloud.

Sign in or register

To sign in with an existing account or register an account, go to https://app.prefect.cloud/.

You can create an account with:

  • Google account
  • GitHub account
  • Email and password

Create a workspace

A workspace is an isolated environment within Prefect Cloud for your flows and deployments. You can use workspaces to organize or compartmentalize your workflows. For example, you can create separate workspaces to isolate development, staging, and production environments; or to provide separation between different teams.

When you register a new account, you'll be prompted to create a workspace.

Creating a new Prefect Cloud account.

Select Create Workspace. You'll be prompted to provide a name and description for your workspace.

Creating a new workspace in the Cloud UI.

Select Save to create the workspace.

Viewing a workspace dashboard in the Prefect Cloud UI.

If you change your mind, you can select Workspace Settings to modify the workspace details or to delete it.

Editing workspace settings in the Prefect Cloud UI

Workspace Settings also shows you the prefect cloud workspace set Prefect CLI command you can use to sync a local execution environment with the workspace.

$ prefect cloud workspace set --workspace "prefect/workinonit"

Deleting a workspace

Deleting a workspace removes any flows, deployments, and storage created on that workspace.

Create an API key

API keys enable you to authenticate an a local environment to work with Prefect Cloud. See Configure execution environment for details on how API keys are configured in your execution environment.

To create an API key, select the account icon at the bottom-left corner of the UI and select your account name. This displays your account profile.

Select the API Keys tab. This displays a list of previously generated keys and lets you create new API keys or delete keys.

Viewing and editing API keys in the Cloud UI.

Select the + button to create a new API key. You're prompted to provide a name for the key and, optionally, an expiration date. Select Create API Key to generate the key.

Creating an API key in the Cloud UI.

Note that API keys cannot be revealed again in the UI after you generate them, so copy the key to a secure location.

Configure execution environment

Now configure a local execution environment to use Prefect Cloud as the API server for flow runs.

First, Install Prefect in the environment in which you want to execute flow runs.

Next, use the Prefect CLI prefect cloud login command to log into Prefect Cloud from your environment, using the API key generated previously.

$ prefect cloud login --key xxx_XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX

If this is your first time logging in with this API key, you will be prompted to make a new profile as well as select a workspace (you can specify a workspace with the -w or --workspace option).

$ prefect cloud login --key xxx_XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
┏━━━━━━━━━━━━━━━━━━━━━━┓
┃  Select a Workspace: ┃
┡━━━━━━━━━━━━━━━━━━━━━━┩
│ > prefect/workinonit │
└──────────────────────┘
Creating a profile for this Prefect Cloud login. Please specify a profile name: my-cloud-profile
Logged in to Prefect Cloud using profile 'my-cloud-profile'.
Workspace is currently set to 'prefect/workinonit'. The workspace can be changed using `prefect cloud workspace set`.

The command sets PREFECT_API_KEY and PREFECT_API_URL for the new profile.

Now you're ready to run flows locally and have the results displayed in the Prefect Cloud UI.

You can log out of Prefect Cloud by switching to a different profile.

Changing workspaces

If you need to change which workspace you're syncing with, use the prefect cloud workspace set Prefect CLI command while logged in, passing the account handle and workspace name.

$ prefect cloud workspace set --workspace "prefect/workinonit"

If no workspace is provided, you will be prompted to select one.

Manually configuring Cloud settings

Note that you can also manually configure the PREFECT_API_URL and PREFECT_API_KEY settings to interact with Prefect Cloud by using an account ID, workspace ID, and API key.

$ prefect config set PREFECT_API_URL="https://api.prefect.cloud/api/accounts/[ACCOUNT-ID]/workspaces/[WORKSPACE-ID]"
$ prefect config set PREFECT_API_KEY="[API-KEY]"

When you're in a Prefect Cloud workspace, you can copy the PREFECT_API_URL value directly from the page URL.

In this example, we configured PREFECT_API_URL and PREFECT_API_KEY in the default profile. You can use prefect profile CLI commands to create settings profiles for different configurations. For example, you could have a "cloud" profile configured to use the Prefect Cloud API URL and API key, and another "local" profile for local development using a local Prefect API server started with prefect orion start. See Settings for details.

Configure storage

When using Prefect Cloud, we recommend configuring remote storage for persisting flow and task data. See Storage for details.

By default, Prefect uses local file system storage to persist flow code and flow and task results. For local development and testing this may be adequate. Be aware, however, that local storage is not guaranteed to persist data reliably between flow or task runs, particularly when using container-based environments such as Docker or Kubernetes, or running tasks with distributed computing tools like Dask and Ray.

Run a flow with Prefect Cloud

Okay, you're all set to run a local flow with Prefect Cloud. Notice that everything works just like running local flows with the Prefect API server, but because you configured PREFECT_API_URL and PREFECT_API_KEY, your flow runs show up in Prefect Cloud!

In your local environment, where you configured the previous steps, create a file named basic_flow.py with the following contents:

from prefect import flow, get_run_logger

@flow(name="Testing")
def basic_flow():
    logger = get_run_logger()
    logger.warning("The fun is about to begin")

if __name__ == "__main__":
    basic_flow()

Now run basic_flow.py.

$ python basic_flow.py
21:40:12.904 | INFO    | prefect.engine - Created flow run 'perfect-cat' for flow 'Testing'
21:40:12.904 | INFO    | Flow run 'perfect-cat' - Using task runner 'ConcurrentTaskRunner'
21:40:13.176 | WARNING | Flow run 'perfect-cat' - The fun is about to begin
21:40:13.555 | INFO    | Flow run 'perfect-cat' - Finished in state Completed()

Go to the dashboard for your workspace in Prefect Cloud. You'll see the flow run results right there in Prefect Cloud!

Viewing local flow run results in the Cloud UI.

Prefect Cloud now automatically tracks any flow runs in a local execution environment logged into Prefect Cloud.

Run a flow from a deployment

Deployments are flows packaged in a way that let you run them directly from the Prefect Cloud UI, either ad-hoc runs or via a schedule.

To run a flow from a deployment with Prefect Cloud, you'll need to:

  • Create a flow script
  • Create the deployment using the Prefect CLI
  • Start an agent in your execution environment
  • Run your deployment to create a flow run

Create a deployment

Let's go back to your flow code in basic_flow.py. In a terminal, run the prefect deployment build Prefect CLI command to build a manifest JSON file and deployment YAML file that you'll use to create the deployment on Prefect Cloud.

$ prefect deployment build ./basic_flow.py:basic_flow -n test-deployment -q test

What did we do here? Let's break down the command:

  • prefect deployment build is the Prefect CLI command that enables you to prepare the settings for a deployment.
  • ./basic_flow.py:basic_flow specifies the location of the flow script file and the name of the entrypoint flow function, separated by a colon.
  • -n test-deployment is an option to specify a name for the deployment.
  • -q test specifies a work queue for the deployment. The deployment's runs will be sent to any agents monitoring this work queue.

The command outputs basic_flow-deployment.yaml, which contains details about the deployment for this flow.

Create the deployment

In the terminal, use the prefect deployment apply command to apply the settings contained in the manifest and deployment.yaml to create the deployment on Prefect Cloud.

Run the following Prefect CLI command.

$ prefect deployment apply basic_flow-deployment.yaml
Successfully loaded 'test-deployment'
Deployment '66b3fdea-cd3a-4734-b3f2-65f6702ff260' successfully created.

Now your deployment has been created and is ready to orchestrate future Testing flow runs.

To demonstrate that your deployment exists, go back to Prefect Cloud and select the Deployments page. You'll see the 'Testing/Test Deployment' deployment was created.

'Testing/Test Deployment' appears in the Prefect Cloud Deployments page

Create a work queue and agent

Next, we start an agent that can pick up the flow run from your 'Testing/Test Deployment' deployment. Remember that when we created the deployment, it was configured to send work to a work queue called test. This work queue was automatically created when we created the deployment. In Prefect Cloud, you can view your work queues and create new ones manually by selecting the Work Queues page.

In your terminal, run the prefect agent start command, passing a -q test option that tells it to look for work in the test work queue.

$ prefect agent start -q test

Starting agent connected to https://api.prefect.cloud/api/accounts/...

  ___ ___ ___ ___ ___ ___ _____     _   ___ ___ _  _ _____
 | _ \ _ \ __| __| __/ __|_   _|   /_\ / __| __| \| |_   _|
 |  _/   / _|| _|| _| (__  | |    / _ \ (_ | _|| .` | | |
 |_| |_|_\___|_| |___\___| |_|   /_/ \_\___|___|_|\_| |_|


Agent started! Looking for work from queue 'test'...

PREFECT_API_URL setting for agents

PREFECT_API_URL must be set for the environment in which your agent is running.

In this case, we're running the agent in the same environment we logged into Prefect Cloud earlier. However, if you want the agent to communicate with Prefect Cloud from a remote execution environment such as a VM or Docker container, you must configure PREFECT_API_URL in that environment.

Run a flow

Now create a flow run from your deployment. You'll start the flow run from the Prefect Cloud UI, see it run by the agent in your local execution environment, and then see the result back in Prefect Cloud.

Go back to Prefect Cloud and select the Deployments page, then select Test Deployment in the 'Testing/Test Deployment' deployment name. You'll see a page showing details about the deployment.

Overview of the test deployment in Prefect Cloud

To start an ad-hoc flow run, select the Run button from Prefect Cloud.

In the local terminal session where you started the agent, you can see that the agent picks up the flow run and executes it.

$ prefect agent start -q test
Starting agent connected to https://api.prefect.cloud/api/accounts/...

  ___ ___ ___ ___ ___ ___ _____     _   ___ ___ _  _ _____
 | _ \ _ \ __| __| __/ __|_   _|   /_\ / __| __| \| |_   _|
 |  _/   / _|| _|| _| (__  | |    / _ \ (_ | _|| .` | | |
 |_| |_|_\___|_| |___\___| |_|   /_/ \_\___|___|_|\_| |_|


Agent started! Looking for work from queue 'test-queue'...
22:38:06.072 | INFO    | prefect.agent - Submitting flow run '7a9e94d4-7a97-4188-a539-b8594874eb86'
22:38:06.169 | INFO    | prefect.flow_runner.subprocess - Opening subprocess for flow run '7a9e94d4-7a97-4188-a539-b8594874eb86'...
22:38:06.180 | INFO    | prefect.agent - Completed submission of flow run '7a9e94d4-7a97-4188-a539-b8594874eb86'
22:38:09.918 | INFO    | Flow run 'crystal-hog' - Using task runner 'ConcurrentTaskRunner'
22:38:10.225 | WARNING | Flow run 'crystal-hog' - The fun is about to begin
22:38:10.868 | INFO    | Flow run 'crystal-hog' - Finished in state Completed()
22:38:11.797 | INFO    | prefect.flow_runner.subprocess - Subprocess for flow run '7a9e94d4-7a97-4188-a539-b8594874eb86' exited cleanly.

In Prefect Cloud, select the Flow Runs page and notice that your flow run appears on the dashboard. (Your flow run name will be different, but the rest of the details should be similar to what you see here.)

Viewing the flow run based on the test deployment in Prefect Cloud

To learn more, see the Deployments tutorial for a hands-on example.