prefect_dbt.cloud.runs
¶
Module containing tasks and flows for interacting with dbt Cloud job runs
DbtCloudJobRunStatus
¶
Bases: Enum
dbt Cloud Job statuses.
Source code in prefect_dbt/cloud/runs.py
20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 |
|
is_terminal_status_code
classmethod
¶
Returns True if a status code is terminal for a job run. Returns False otherwise.
Source code in prefect_dbt/cloud/runs.py
30 31 32 33 34 35 36 |
|
get_dbt_cloud_run_artifact
async
¶
A task to get an artifact generated for a completed run. The requested artifact is saved to a file in the current working directory.
Parameters:
Name | Type | Description | Default |
---|---|---|---|
dbt_cloud_credentials |
DbtCloudCredentials
|
Credentials for authenticating with dbt Cloud. |
required |
run_id |
int
|
The ID of the run to list run artifacts for. |
required |
path |
str
|
The relative path to the run artifact (e.g. manifest.json, catalog.json, run_results.json) |
required |
step |
Optional[int]
|
The index of the step in the run to query for artifacts. The first step in the run has the index 1. If the step parameter is omitted, then this method will return the artifacts compiled for the last step in the run. |
None
|
Returns:
Type | Description |
---|---|
Union[Dict, str]
|
The contents of the requested manifest. Returns a |
Examples:
Get an artifact of a dbt Cloud job run:
from prefect import flow
from prefect_dbt.cloud import DbtCloudCredentials
from prefect_dbt.cloud.runs import get_dbt_cloud_run_artifact
@flow
def get_artifact_flow():
credentials = DbtCloudCredentials(api_key="my_api_key", account_id=123456789)
return get_dbt_cloud_run_artifact(
dbt_cloud_credentials=credentials,
run_id=42,
path="manifest.json"
)
get_artifact_flow()
Get an artifact of a dbt Cloud job run and write it to a file:
import json
from prefect import flow
from prefect_dbt.cloud import DbtCloudCredentials
from prefect_dbt.cloud.jobs import get_dbt_cloud_run_artifact
@flow
def get_artifact_flow():
credentials = DbtCloudCredentials(api_key="my_api_key", account_id=123456789)
get_run_artifact_result = get_dbt_cloud_run_artifact(
dbt_cloud_credentials=credentials,
run_id=42,
path="manifest.json"
)
with open("manifest.json", "w") as file:
json.dump(get_run_artifact_result, file)
get_artifact_flow()
Source code in prefect_dbt/cloud/runs.py
148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175 176 177 178 179 180 181 182 183 184 185 186 187 188 189 190 191 192 193 194 195 196 197 198 199 200 201 202 203 204 205 206 207 208 209 210 211 212 213 214 215 216 217 218 219 220 221 222 223 224 225 226 227 228 229 230 231 232 233 234 235 236 237 238 |
|
get_dbt_cloud_run_info
async
¶
A task to retrieve information about a dbt Cloud job run.
Parameters:
Name | Type | Description | Default |
---|---|---|---|
dbt_cloud_credentials |
DbtCloudCredentials
|
Credentials for authenticating with dbt Cloud. |
required |
run_id |
int
|
The ID of the job to trigger. |
required |
include_related |
Optional[List[Literal['trigger', 'job', 'debug_logs', 'run_steps']]]
|
List of related fields to pull with the run. Valid values are "trigger", "job", "debug_logs", and "run_steps". If "debug_logs" is not provided in a request, then the included debug logs will be truncated to the last 1,000 lines of the debug log output file. |
None
|
Returns:
Type | Description |
---|---|
Dict
|
The run data returned by the dbt Cloud administrative API. |
Example
Get status of a dbt Cloud job run:
from prefect import flow
from prefect_dbt.cloud import DbtCloudCredentials
from prefect_dbt.cloud.jobs import get_run
@flow
def get_run_flow():
credentials = DbtCloudCredentials(api_key="my_api_key", account_id=123456789)
return get_run(
dbt_cloud_credentials=credentials,
run_id=42
)
get_run_flow()
Source code in prefect_dbt/cloud/runs.py
39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 |
|
list_dbt_cloud_run_artifacts
async
¶
A task to list the artifact files generated for a completed run.
Parameters:
Name | Type | Description | Default |
---|---|---|---|
dbt_cloud_credentials |
DbtCloudCredentials
|
Credentials for authenticating with dbt Cloud. |
required |
run_id |
int
|
The ID of the run to list run artifacts for. |
required |
step |
Optional[int]
|
The index of the step in the run to query for artifacts. The first step in the run has the index 1. If the step parameter is omitted, then this method will return the artifacts compiled for the last step in the run. |
None
|
Returns:
Type | Description |
---|---|
List[str]
|
A list of paths to artifact files that can be used to retrieve the generated artifacts. |
Example
List artifacts of a dbt Cloud job run:
from prefect import flow
from prefect_dbt.cloud import DbtCloudCredentials
from prefect_dbt.cloud.jobs import list_dbt_cloud_run_artifacts
@flow
def list_artifacts_flow():
credentials = DbtCloudCredentials(api_key="my_api_key", account_id=123456789)
return list_dbt_cloud_run_artifacts(
dbt_cloud_credentials=credentials,
run_id=42
)
list_artifacts_flow()
Source code in prefect_dbt/cloud/runs.py
97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 |
|
wait_for_dbt_cloud_job_run
async
¶
Waits for the given dbt Cloud job run to finish running.
Parameters:
Name | Type | Description | Default |
---|---|---|---|
run_id |
int
|
The ID of the run to wait for. |
required |
dbt_cloud_credentials |
DbtCloudCredentials
|
Credentials for authenticating with dbt Cloud. |
required |
max_wait_seconds |
int
|
Maximum number of seconds to wait for job to complete |
900
|
poll_frequency_seconds |
int
|
Number of seconds to wait in between checks for run completion. |
10
|
Raises:
Type | Description |
---|---|
DbtCloudJobRunTimedOut
|
When the elapsed wait time exceeds |
Returns:
Name | Type | Description |
---|---|---|
run_status |
DbtCloudJobRunStatus
|
An enum representing the final dbt Cloud job run status |
run_data |
Dict
|
A dictionary containing information about the run after completion. |
Source code in prefect_dbt/cloud/runs.py
241 242 243 244 245 246 247 248 249 250 251 252 253 254 255 256 257 258 259 260 261 262 263 264 265 266 267 268 269 270 271 272 273 274 275 276 277 278 279 280 281 282 283 284 285 286 287 288 289 290 291 292 293 294 295 296 297 298 299 300 301 |
|