Skip to content

Frequently Asked Questions

Prefect 2.0

How is Prefect 2.0 licensed?

Prefect 2.0 is licensed under the Apache 2.0 License, an OSI approved open-source license. If you have any questions about licensing, please contact us.

Can I use Prefect 2.0 in production?

Prefect 2.0 is beta software and we do not recommend Orion for production use at this time.

The Orion Engine

Why "Orion"?

As an advanced orchestration engine, Orion practically named itself: ORchestratION.

The word "Prefect" stems from the Latin "praefectus," meaning "one who is in charge." A prefect is an official who oversees a domain and ensures that rules are followed. Similarly, Prefect software ensures that workflows are orchestrated properly.

Curiously enough, Orion is also the home of Ford Prefect, a roving researcher for that wholly remarkable book, The Hitchhiker's Guide to the Galaxy.

Why was Orion created?

Orion has three major objectives:

  • Embracing dynamic, DAG-free workflows
  • An extraordinary developer experience
  • Transparent and observable orchestration rules

As Prefect has matured, so has the modern data stack. The on-demand, dynamic, highly scalable workflows that used to exist principally in the domain of data science and analytics are now prevalent throughout all of data engineering. Few companies have workflows that don’t deal with streaming data, uncertain timing, runtime logic, complex dependencies, versioning, or custom scheduling.

This means that the current generation of workflow managers are built around the wrong abstraction: the directed acyclic graph (DAG). DAGs are an increasingly arcane, constrained way of representing the dynamic, heterogeneous range of modern data and computation patterns.

Furthermore, as workflows have become more complex, it has become even more important to focus on the developer experience of building, testing, and monitoring them. Faced with an explosion of available tools, it is more important than ever for development teams to seek orchestration tools that will be compatible with any code, tools, or services they may require in the future.

And finally, this additional complexity means that providing clear and consistent insight into the behavior of the orchestration engine and any decisions it makes is critically important.

Orion represents a unified solution to these three problems.

Orion is capable of governing any code through a well-defined series of state transitions designed to maximize the user's understanding of what happened during execution. It's popular to describe "workflows as code" or "orchestration as code," but Orion represents "code as code": rather than ask users to change how they work to meet the requirements of the orchestrator, we've defined an orchestrator that adapts to how our users work.

To achieve this, we've leveraged the familiar tools of native Python: first class functions, type annotations, and async support. Users are free to implement as much — or as little — of the Orion engine as is useful for their objectives.

Features

Does Prefect 2.0 support mapping?

Mapping is one of the most popular features in Prefect 1.0, allowing users to tap into their task runner's native fan-out abilities. An equivalent .map() operator will be released for Prefect 2.0 soon. For now, users can take advantage of Prefect's support for native Python to call tasks in loops:

@flow
def my_flow():

    # map over a constant
    for i in range(10):
        my_mapped_task(i)

    # map over a task's output
    l = list_task()
    for i in l.wait().result():
        my_mapped_task_2(i)

Note that when tasks are called on constant values, they cannot detect their upstream edges automatically. In this example, my_mapped_task_2 does not know that it is downstream from list_task(). Prefect will have convenience functions for detecting these associations, and Prefect's .map() operator will automatically track them.

How do I enforce ordering between tasks that don't share data?

To create a dependency between two tasks that do not exchange data, but one needs to wait for the other to finish, use the special wait_for keyword argument:

@task
def task_1():
    pass

@task
def task_2():
    pass

@flow
def my_flow():
    x = task_1()

    # task 2 will wait for task_1 to complete
    y = task_2(wait_for=[x])

When will Prefect 2.0 be released?

The Prefect 2.0 beta period is expected to last for at least the second quarter of 2022.

What external requirements does Prefect have?

Prefect does not have any additional requirements besides those installed by pip install --pre prefect. The entire system, including the UI and services, can be run in a single process via prefect orion start and does not require Docker.

To use PostgreSQL, users must provide the connection string for a running database via the PREFECT_ORION_DATABASE_CONNECTION_URL environment variable.

What databases does Prefect support?

Prefect works with SQLite and PostgreSQL. New Prefect installs default to a SQLite database hosted at ~/.prefect/orion.db.

How do I choose between SQLite and Postgres?

SQLite is appropriate for getting started and exploring Prefect. We have tested it with up to hundreds of thousands of task runs. Many users may be able to stay on SQLite for some time. However, under write-heavy workloads, SQLite performance can begin to suffer. Users running many flows with high degrees of parallelism or concurrency may prefer to start with PostgreSQL.

This answer will be updated with more concrete guidelines in the future.

Relationship with other Prefect products

Is there an upgrade guide from Prefect 1.0 to Prefect 2.0?

As Prefect 2.0 moves closer to release, we will provide upgrade guides and tools to make adoption as easy as possible for Prefect 1.0 users. Given the similarity of the 1.0 and 2.0 client APIs, upgrading should require minimal effort on users' behalf.