Skip to content

prefect.projects.steps.pull

Core set of steps for specifying a Prefect project pull step.

git_clone_project

Clones a git repository into the current working directory.

Parameters:

Name Type Description Default
repository str

the URL of the repository to clone

 required
branch str

the branch to clone; if not provided, the default branch will be used

 None
include_submodules bool

whether to include git submodules when cloning the repository

 False
access_token str

an access token to use for cloning the repository; if not provided the repository will be cloned using the default git credentials

 None

Returns:

Name Type Description
dict dict

a dictionary containing a directory key of the new directory that was created

Raises:

Type Description
subprocess.CalledProcessError

if the git clone command fails for any reason

Examples:

Clone a public repository: 

pull:
    - prefect.projects.steps.git_clone_project:
        repository: https://github.com/PrefectHQ/prefect.git

Clone a branch of a public repository: 

pull:
    - prefect.projects.steps.git_clone_project:
        repository: https://github.com/PrefectHQ/prefect.git
        branch: my-branch

Clone a private repository using an access token: 

pull:
    - prefect.projects.steps.git_clone_project:
        repository: https://github.com/org/repo.git
        access_token: "{{ prefect.blocks.secret.github-access-token }}" # Requires creation of a Secret block
Note that you will need to create a Secret block to store the value of your git credentials. You can also store a username/password combo or token prefix (e.g. x-token-auth) in your secret block. Refer to your git providers documentation for the correct authentication schema.

Clone a repository with submodules: 

pull:
    - prefect.projects.steps.git_clone_project:
        repository: https://github.com/org/repo.git
        include_submodules: true

Clone a repository with an SSH key (note that the SSH key must be added to the worker before executing flows): 

pull:
    - prefect.projects.steps.git_clone_project:
        repository: git@github.com:org/repo.git

Source code in /home/runner/work/docs/docs/prefect_source/src/prefect/projects/steps/pull.py 
 30
 31
 32
 33
 34
 35
 36
 37
 38
 39
 40
 41
 42
 43
 44
 45
 46
 47
 48
 49
 50
 51
 52
 53
 54
 55
 56
 57
 58
 59
 60
 61
 62
 63
 64
 65
 66
 67
 68
 69
 70
 71
 72
 73
 74
 75
 76
 77
 78
 79
 80
 81
 82
 83
 84
 85
 86
 87
 88
 89
 90
 91
 92
 93
 94
 95
 96
 97
 98
 99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
def git_clone_project(
    repository: str,
    branch: Optional[str] = None,
    include_submodules: bool = False,
    access_token: Optional[str] = None,
) -> dict:
    """
    Clones a git repository into the current working directory.

    Args:
        repository (str): the URL of the repository to clone
        branch (str, optional): the branch to clone; if not provided, the default branch will be used
        include_submodules (bool): whether to include git submodules when cloning the repository
        access_token (str, optional): an access token to use for cloning the repository; if not provided
            the repository will be cloned using the default git credentials

    Returns:
        dict: a dictionary containing a `directory` key of the new directory that was created

    Raises:
        subprocess.CalledProcessError: if the git clone command fails for any reason

    Examples:
        Clone a public repository:
        ```yaml
        pull:
            - prefect.projects.steps.git_clone_project:
                repository: https://github.com/PrefectHQ/prefect.git
        ```

        Clone a branch of a public repository:
        ```yaml
        pull:
            - prefect.projects.steps.git_clone_project:
                repository: https://github.com/PrefectHQ/prefect.git
                branch: my-branch
        ```

        Clone a private repository using an access token:
        ```yaml
        pull:
            - prefect.projects.steps.git_clone_project:
                repository: https://github.com/org/repo.git
                access_token: "{{ prefect.blocks.secret.github-access-token }}" # Requires creation of a Secret block
        ```
        Note that you will need to [create a Secret block](/concepts/blocks/#using-existing-block-types) to store the
        value of your git credentials. You can also store a username/password combo or token prefix (e.g. `x-token-auth`)
        in your secret block. Refer to your git providers documentation for the correct authentication schema.

        Clone a repository with submodules:
        ```yaml
        pull:
            - prefect.projects.steps.git_clone_project:
                repository: https://github.com/org/repo.git
                include_submodules: true
        ```

        Clone a repository with an SSH key (note that the SSH key must be added to the worker
        before executing flows):
        ```yaml
        pull:
            - prefect.projects.steps.git_clone_project:
                repository: git@github.com:org/repo.git
        ```
    """
    url_components = urllib.parse.urlparse(repository)
    if url_components.scheme == "https" and access_token is not None:
        updated_components = url_components._replace(
            netloc=f"{access_token}@{url_components.netloc}"
        )
        repository_url = urllib.parse.urlunparse(updated_components)
    else:
        repository_url = repository

    cmd = ["git", "clone", repository_url]
    if branch:
        cmd += ["-b", branch]
    if include_submodules:
        cmd += ["--recurse-submodules"]

    # Limit git history
    cmd += ["--depth", "1"]

    try:
        subprocess.check_call(
            cmd, shell=sys.platform == "win32", stderr=sys.stderr, stdout=sys.stdout
        )
    except subprocess.CalledProcessError as exc:
        # Hide the command used to avoid leaking the access token
        exc_chain = None if access_token else exc
        raise RuntimeError(
            f"Failed to clone repository {repository!r} with exit code"
            f" {exc.returncode}."
        ) from exc_chain

    directory = "/".join(repository.strip().split("/")[-1:]).replace(".git", "")
    projects_logger.info(f"Cloned repository {repository!r} into {directory!r}")
    return {"directory": directory}

set_working_directory

Sets the working directory; works with both absolute and relative paths.

Parameters:

Name Type Description Default
directory str

the directory to set as the working directory

 required

Returns:

Name Type Description
dict dict

a dictionary containing a directory key of the directory that was set
Source code in /home/runner/work/docs/docs/prefect_source/src/prefect/projects/steps/pull.py 
15
16
17
18
19
20
21
22
23
24
25
26
27
def set_working_directory(directory: str) -> dict:
    """
    Sets the working directory; works with both absolute and relative paths.

    Args:
        directory (str): the directory to set as the working directory

    Returns:
        dict: a dictionary containing a `directory` key of the
            directory that was set
    """
    os.chdir(directory)
    return dict(directory=directory)