prefect.settings
Prefect settings management.
Each setting is defined as a Setting type. The name of each setting is stylized in all
caps, matching the environment variable that can be used to change the setting.
All settings defined in this file are used to generate a dynamic Pydantic settings class
called Settings. When insantiated, this class will load settings from environment
variables and pull default values from the setting definitions.
The current instance of Settings being used by the application is stored in a
SettingsContext model which allows each instance of the Settings class to be
accessed in an async-safe manner.
Aside from environment variables, we allow settings to be changed during the runtime of the process using profiles. Profiles contain setting overrides that the user may persist without setting environment variables. Profiles are also used internally for managing settings during task run execution where differing settings may be used concurrently in the same process and during testing where we need to override settings to ensure their value is respected as intended.
The SettingsContext is set when the prefect module is imported. This context is
referred to as the "root" settings context for clarity. Generally, this is the only
settings context that will be used. When this context is entered, we will instantiate
a Settings object, loading settings from environment variables and defaults, then we
will load the active profile and use it to override settings. See enter_root_settings_context
for details on determining the active profile.
Another SettingsContext may be entered at any time to change the settings being
used by the code within the context. Generally, users should not use this. Settings
management should be left to Prefect application internals.
Generally, settings should be accessed with SETTING_VARIABLE.value() which will
pull the current Settings instance from the current SettingsContext and retrieve
the value of the relevant setting.
Accessing a setting's value will also call the Setting.value_callback which allows
settings to be dynamically modified on retrieval. This allows us to make settings
dependent on the value of other settings or perform other dynamic effects.
Setting
Setting definition type.
Source code in prefect/settings.py
class Setting(Generic[T]):
"""
Setting definition type.
"""
def __init__(
self,
type: Type[T],
*,
value_callback: Callable[["Settings", T], T] = None,
**kwargs,
) -> None:
self.field: pydantic.fields.FieldInfo = Field(**kwargs)
self.type = type
self.value_callback = value_callback
self.name = None # Will be populated after all settings are defined
self.__doc__ = self.field.description
def value(self) -> T:
"""
Get the current value of a setting.
Example:
```python
from prefect.settings import PREFECT_API_URL
PREFECT_API_URL.value()
```
"""
return self.value_from(get_current_settings())
def value_from(self, settings: "Settings") -> T:
"""
Get the value of a setting from a settings object
Example:
```python
from prefect.settings import get_default_settings
PREFECT_API_URL.value_from(get_default_settings())
```
"""
return settings.value_of(self)
def __repr__(self) -> str:
return f"<{self.name}: {self.type.__name__}>"
def __bool__(self) -> bool:
"""
Returns a truthy check of the current value.
"""
return bool(self.value())
Setting.value
Get the current value of a setting.
Examples:
from prefect.settings import PREFECT_API_URL
PREFECT_API_URL.value()
Source code in prefect/settings.py
def value(self) -> T:
"""
Get the current value of a setting.
Example:
```python
from prefect.settings import PREFECT_API_URL
PREFECT_API_URL.value()
```
"""
return self.value_from(get_current_settings())
Setting.value_from
Get the value of a setting from a settings object
Examples:
from prefect.settings import get_default_settings
PREFECT_API_URL.value_from(get_default_settings())
Source code in prefect/settings.py
def value_from(self, settings: "Settings") -> T:
"""
Get the value of a setting from a settings object
Example:
```python
from prefect.settings import get_default_settings
PREFECT_API_URL.value_from(get_default_settings())
```
"""
return settings.value_of(self)
Settings
pydantic-model
Contains validated Prefect settings.
Settings should be accessed using the relevant Setting object. For example:
from prefect.settings import PREFECT_HOME
PREFECT_HOME.value()
Accessing a setting attribute directly will ignore any value_callback mutations.
This is not recommended:
from prefect.settings import Settings
Settings().PREFECT_PROFILES_PATH # PosixPath('${PREFECT_HOME}/profiles.toml')
Source code in prefect/settings.py
class Settings(SettingsFieldsMixin):
"""
Contains validated Prefect settings.
Settings should be accessed using the relevant `Setting` object. For example:
```python
from prefect.settings import PREFECT_HOME
PREFECT_HOME.value()
```
Accessing a setting attribute directly will ignore any `value_callback` mutations.
This is not recommended:
```python
from prefect.settings import Settings
Settings().PREFECT_PROFILES_PATH # PosixPath('${PREFECT_HOME}/profiles.toml')
```
"""
def value_of(self, setting: Setting[T]) -> T:
"""
Retrieve a setting's value.
"""
value = getattr(self, setting.name)
if setting.value_callback:
value = setting.value_callback(self, value)
return value
@root_validator
def post_root_validators(cls, values):
"""
Add root validation functions for settings here.
"""
# TODO: We could probably register these dynamically but this is the simpler
# approach for now. We can explore more interesting validation features
# in the future.
values = max_log_size_smaller_than_batch_size(values)
return values
def copy_with_update(
self,
updates: Mapping[Setting, Any] = None,
set_defaults: Mapping[Setting, Any] = None,
restore_defaults: Iterable[Setting] = None,
) -> "Settings":
"""
Create a new `Settings` object with validation.
Arguments:
updates: A mapping of settings to new values. Existing values for the
given settings will be overridden.
set_defaults: A mapping of settings to new default values. Existing values for
the given settings will only be overridden if they were not set.
restore_defaults: An iterable of settings to restore to their default values.
Returns:
A new `Settings` object.
"""
updates = updates or {}
set_defaults = set_defaults or {}
restore_defaults = restore_defaults or set()
restore_defaults_names = {setting.name for setting in restore_defaults}
return self.__class__(
**{
**{setting.name: value for setting, value in set_defaults.items()},
**self.dict(exclude_unset=True, exclude=restore_defaults_names),
**{setting.name: value for setting, value in updates.items()},
}
)
def to_environment_variables(
self, include: Iterable[Setting] = None, exclude_unset: bool = False
) -> Dict[str, str]:
"""
Convert the settings object to environment variables.
Note that setting values will not be run through their `value_callback` allowing
dynamic resolution to occur when loaded from the returned environment.
Args:
include_keys: An iterable of settings to include in the return value.
If not set, all settings are used.
exclude_unset: Only include settings that have been set (i.e. the value is
not from the default). If set, unset keys will be dropped even if they
are set in `include_keys`.
Returns:
A dictionary of settings with values cast to strings
"""
include = set(include or SETTING_VARIABLES.values())
if exclude_unset:
set_keys = {
# Collect all of the "set" keys and cast to `Setting` objects
SETTING_VARIABLES[key]
for key in self.dict(exclude_unset=True)
}
include.intersection_update(set_keys)
# Validate the types of items in `include` to prevent exclusion bugs
for key in include:
if not isinstance(key, Setting):
raise TypeError(
"Invalid type {type(key).__name__!r} for key in `include`."
)
env = {
# Use `getattr` instead of `value_of` to avoid value callback resolution
key: getattr(self, key)
for key, setting in SETTING_VARIABLES.items()
if setting in include
}
# Cast to strings and drop null values
return {key: str(value) for key, value in env.items() if value is not None}
class Config:
frozen = True
__reduce__ = reduce_settings
PREFECT_HOME
pydantic-field
Type: Path
Prefect's home directory. Defaults to ~/.prefect. This
directory may be created automatically when required.
PREFECT_DEBUG_MODE
pydantic-field
Type: bool
If True, places the API in debug mode. This may modify
behavior to facilitate debugging, including extra logs and other verbose
assistance. Defaults to False.
PREFECT_TEST_MODE
pydantic-field
Type: bool
If True, places the API in test mode. This may modify
behavior to faciliate testing. Defaults to False.
PREFECT_TEST_SETTING
pydantic-field
Type: Any
This variable only exists to faciliate testing of settings.
If accessed when PREFECT_TEST_MODE is not set, None is returned.
PREFECT_API_URL
pydantic-field
Type: str
If provided, the url of an externally-hosted Orion API.
Defaults to None.
PREFECT_API_KEY
pydantic-field
Type: str
API key used to authenticate against Orion API.
Defaults to None.
PREFECT_CLOUD_URL
pydantic-field
Type: str
API URL for Prefect Cloud
PREFECT_API_REQUEST_TIMEOUT
pydantic-field
Type: float
The default timeout for requests to the API
PREFECT_PROFILES_PATH
pydantic-field
Type: Path
The path to a profiles configuration files.
PREFECT_LOGGING_LEVEL
pydantic-field
Type: str
The default logging level for Prefect loggers. Defaults to "INFO" during normal operation. Is forced to "DEBUG" during debug mode.
PREFECT_LOGGING_SERVER_LEVEL
pydantic-field
Type: str
The default logging level for the Orion API.
PREFECT_LOGGING_SETTINGS_PATH
pydantic-field
Type: Path
The path to a custom YAML logging configuration file. If
no file is found, the default logging.yml is used. Defaults to a logging.yml in the Prefect home directory.
PREFECT_LOGGING_EXTRA_LOGGERS
pydantic-field
Type: str
Additional loggers to attach to Prefect logging at runtime. Values should be comma separated. The handlers attached to the 'prefect' logger will be added to these loggers. Additionally, if the level is not set, it will be set to the same level as the 'prefect' logger.
PREFECT_LOGGING_ORION_ENABLED
pydantic-field
Type: bool
Should logs be sent to Orion? If False, logs sent to the OrionHandler will not be sent to the API.
PREFECT_LOGGING_ORION_BATCH_INTERVAL
pydantic-field
Type: float
The number of seconds between batched writes of logs to Orion.
PREFECT_LOGGING_ORION_BATCH_SIZE
pydantic-field
Type: int
The maximum size in bytes for a batch of logs.
PREFECT_LOGGING_ORION_MAX_LOG_SIZE
pydantic-field
Type: int
The maximum size in bytes for a single log.
PREFECT_AGENT_QUERY_INTERVAL
pydantic-field
Type: float
The agent loop interval, in seconds. Agents will check
for new runs this often. Defaults to 5.
PREFECT_AGENT_PREFETCH_SECONDS
pydantic-field
Type: int
Agents will look for scheduled runs this many seconds in
the future and attempt to run them. This accounts for any additional
infrastructure spin-up time or latency in preparing a flow run. Note
flow runs will not start before their scheduled time, even if they are
prefetched. Defaults to 10.
PREFECT_ORION_DATABASE_CONNECTION_URL
pydantic-field
Type: str
A database connection URL in a SQLAlchemy-compatible
format. Orion currently supports SQLite and Postgres. Note that all
Orion engines must use an async driver - for SQLite, use
sqlite+aiosqlite and for Postgres use postgresql+asyncpg.
SQLite in-memory databases can be used by providing the url
sqlite+aiosqlite:///file::memory:?cache=shared&uri=true&check_same_thread=false,
which will allow the database to be accessed by multiple threads. Note
that in-memory databases can not be accessed from multiple processes and
should only be used for simple tests.
Defaults to a sqlite database stored in the Prefect home directory.
PREFECT_ORION_DATABASE_ECHO
pydantic-field
Type: bool
If True, SQLAlchemy will log all SQL issued to the database. Defaults to False.
PREFECT_ORION_DATABASE_MIGRATE_ON_START
pydantic-field
Type: bool
If True, the database will be upgraded on application creation. If False, the database will need to be upgraded manually.
PREFECT_ORION_DATABASE_TIMEOUT
pydantic-field
Type: float
A statement timeout, in seconds, applied to all database
interactions made by the API. Defaults to 1.
PREFECT_ORION_DATABASE_CONNECTION_TIMEOUT
pydantic-field
Type: float
A connection timeout, in seconds, applied to database
connections. Defaults to 5.
PREFECT_ORION_SERVICES_SCHEDULER_LOOP_SECONDS
pydantic-field
Type: float
The scheduler loop interval, in seconds. This determines
how often the scheduler will attempt to schedule new flow runs, but has
no impact on how quickly either flow runs or task runs are actually
executed. Creating new deployments or schedules will always create new
flow runs optimistically, without waiting for the scheduler. Defaults to
60.
PREFECT_ORION_SERVICES_SCHEDULER_DEPLOYMENT_BATCH_SIZE
pydantic-field
Type: int
The number of deployments the scheduler will attempt to
schedule in a single batch. If there are more deployments than the batch
size, the scheduler immediately attempts to schedule the next batch; it
does not sleep for scheduler_loop_seconds until it has visited every
deployment once. Defaults to 100.
PREFECT_ORION_SERVICES_SCHEDULER_MAX_RUNS
pydantic-field
Type: int
The scheduler will attempt to schedule up to this many
auto-scheduled runs in the future. Note that runs may have fewer than
this many scheduled runs, depending on the value of
scheduler_max_scheduled_time. Defaults to 100.
PREFECT_ORION_SERVICES_SCHEDULER_MAX_SCHEDULED_TIME
pydantic-field
Type: timedelta
The scheduler will create new runs up to this far in the
future. Note that this setting will take precedence over
scheduler_max_runs: if a flow runs once a month and
scheduled_max_scheduled_time is three months, then only three runs will be
scheduled. Defaults to 100 days (8640000 seconds).
PREFECT_ORION_SERVICES_SCHEDULER_INSERT_BATCH_SIZE
pydantic-field
Type: int
The number of flow runs the scheduler will attempt to insert
in one batch across all deployments. If the number of flow runs to
schedule exceeds this amount, the runs will be inserted in batches of this size. Defaults to 500.
PREFECT_ORION_SERVICES_LATE_RUNS_LOOP_SECONDS
pydantic-field
Type: float
The late runs service will look for runs to mark as late
this often. Defaults to 5.
PREFECT_ORION_SERVICES_LATE_RUNS_AFTER_SECONDS
pydantic-field
Type: timedelta
The late runs service will mark runs as late after they
have exceeded their scheduled start time by this many seconds. Defaults
to 5 seconds.
PREFECT_ORION_API_DEFAULT_LIMIT
pydantic-field
Type: int
The default limit applied to queries that can return
multiple objects, such as POST /flow_runs/filter.
PREFECT_ORION_API_HOST
pydantic-field
Type: str
The API's host address (defaults to 127.0.0.1).
PREFECT_ORION_API_PORT
pydantic-field
Type: int
The API's port address (defaults to 4200).
PREFECT_ORION_UI_ENABLED
pydantic-field
Type: bool
Whether or not to serve the Orion UI.
PREFECT_ORION_UI_API_URL
pydantic-field
Type: str
The connection url for communication from the UI to the API.
Defaults to PREFECT_API_URL if set. Otherwise, the default URL is generated from
PREFECT_ORION_API_HOST and PREFECT_ORION_API_PORT. If providing a custom value,
the aforementioned settings may be templated into the given string.
PREFECT_ORION_ANALYTICS_ENABLED
pydantic-field
Type: bool
If True, Orion sends anonymous data (e.g. count of flow runs, package version) to Prefect to help us improve.
PREFECT_ORION_SERVICES_SCHEDULER_ENABLED
pydantic-field
Type: bool
Whether or not to start the scheduling service in the Orion application. If disabled, you will need to run this service separately to schedule runs for deployments.
PREFECT_ORION_SERVICES_LATE_RUNS_ENABLED
pydantic-field
Type: bool
Whether or not to start the late runs service in the Orion application. If disabled, you will need to run this service separately to have runs past their scheduled start time marked as late.
Settings.value_of
Retrieve a setting's value.
Source code in prefect/settings.py
def value_of(self, setting: Setting[T]) -> T:
"""
Retrieve a setting's value.
"""
value = getattr(self, setting.name)
if setting.value_callback:
value = setting.value_callback(self, value)
return value
Settings.post_root_validators
classmethod
Add root validation functions for settings here.
Source code in prefect/settings.py
@root_validator
def post_root_validators(cls, values):
"""
Add root validation functions for settings here.
"""
# TODO: We could probably register these dynamically but this is the simpler
# approach for now. We can explore more interesting validation features
# in the future.
values = max_log_size_smaller_than_batch_size(values)
return values
Settings.to_environment_variables
Convert the settings object to environment variables.
Note that setting values will not be run through their value_callback allowing
dynamic resolution to occur when loaded from the returned environment.
Parameters:
| Name | Description | Default |
|---|---|---|
include_keys |
An iterable of settings to include in the return value. If not set, all settings are used. |
required |
exclude_unset |
Only include settings that have been set (i.e. the value is
not from the default). If set, unset keys will be dropped even if they
are set in bool |
False |
Returns:
| Type | Description |
|---|---|
Dict[str, str] |
A dictionary of settings with values cast to strings |
Source code in prefect/settings.py
def to_environment_variables(
self, include: Iterable[Setting] = None, exclude_unset: bool = False
) -> Dict[str, str]:
"""
Convert the settings object to environment variables.
Note that setting values will not be run through their `value_callback` allowing
dynamic resolution to occur when loaded from the returned environment.
Args:
include_keys: An iterable of settings to include in the return value.
If not set, all settings are used.
exclude_unset: Only include settings that have been set (i.e. the value is
not from the default). If set, unset keys will be dropped even if they
are set in `include_keys`.
Returns:
A dictionary of settings with values cast to strings
"""
include = set(include or SETTING_VARIABLES.values())
if exclude_unset:
set_keys = {
# Collect all of the "set" keys and cast to `Setting` objects
SETTING_VARIABLES[key]
for key in self.dict(exclude_unset=True)
}
include.intersection_update(set_keys)
# Validate the types of items in `include` to prevent exclusion bugs
for key in include:
if not isinstance(key, Setting):
raise TypeError(
"Invalid type {type(key).__name__!r} for key in `include`."
)
env = {
# Use `getattr` instead of `value_of` to avoid value callback resolution
key: getattr(self, key)
for key, setting in SETTING_VARIABLES.items()
if setting in include
}
# Cast to strings and drop null values
return {key: str(value) for key, value in env.items() if value is not None}
Profile
pydantic-model
A user profile containing settings.
Source code in prefect/settings.py
class Profile(pydantic.BaseModel):
"""
A user profile containing settings.
"""
name: str
settings: Dict[Setting, Any] = Field(default_factory=dict)
source: Optional[Path]
@pydantic.validator("settings", pre=True)
def map_names_to_settings(cls, value):
if value is None:
return value
# Cast string setting names to variables
validated = {}
for setting, val in value.items():
if isinstance(setting, str) and setting in SETTING_VARIABLES:
validated[SETTING_VARIABLES[setting]] = val
elif isinstance(setting, Setting):
validated[setting] = val
else:
raise ValueError(f"Unknown setting {setting!r}.")
return validated
class Config:
arbitrary_types_allowed = True
ProfilesCollection
" A utility class for working with a collection of profiles.
Profiles in the collection must have unique names.
The collection may store the name of the active profile.
Source code in prefect/settings.py
class ProfilesCollection:
""" "
A utility class for working with a collection of profiles.
Profiles in the collection must have unique names.
The collection may store the name of the active profile.
"""
def __init__(
self, profiles: Iterable[Profile], active: Optional[str] = None
) -> None:
self.profiles_by_name = {profile.name: profile for profile in profiles}
self.active_name = active
@property
def names(self) -> Set[str]:
"""
Return a set of profile names in this collection.
"""
return set(self.profiles_by_name.keys())
@property
def active_profile(self) -> Optional[Profile]:
"""
Retrieve the active profile in this collection.
"""
if self.active_name is None:
return None
return self[self.active_name]
def set_active(self, name: Optional[str]):
"""
Set the active profile name in the collection.
A null value may be passed to indicate that this collection does not determine
the active profile.
"""
if name is not None and name not in self.names:
raise ValueError(f"Unknown profile name {name!r}.")
self.active_name = name
def update_profile(
self, name: str, settings: Mapping[Union[Dict, str], Any], source: Path = None
) -> Profile:
"""
Add a profile to the collection or update the existing on if the name is already
present in this collection.
If updating an existing profile, the settings will be merged. Settings can
be dropped from the existing profile by setting them to `None` in the new
profile.
Returns the new profile object.
"""
existing = self.profiles_by_name.get(name)
# Convert the input to a `Profile` to cast settings to the correct type
profile = Profile(name=name, settings=settings, source=source)
if existing:
new_settings = {**existing.settings, **profile.settings}
# Drop null keys to restore to default
for key, value in tuple(new_settings.items()):
if value is None:
new_settings.pop(key)
new_profile = Profile(
name=profile.name,
settings=new_settings,
source=source or profile.source,
)
else:
new_profile = profile
self.profiles_by_name[new_profile.name] = new_profile
return new_profile
def add_profile(self, profile: Profile) -> None:
"""
Add a profile to the collection.
If the profile name already exists, an exception will be raised.
"""
if profile.name in self.profiles_by_name:
raise ValueError(
f"Profile name {profile.name!r} already exists in collection."
)
self.profiles_by_name[profile.name] = profile
def remove_profile(self, name: str) -> None:
"""
Remove a profile from the collection.
"""
self.profiles_by_name.pop(name)
def without_profile_source(self, path: Optional[Path]) -> "ProfilesCollection":
"""
Remove profiles that were loaded from a given path.
Returns a new collection.
"""
return ProfilesCollection(
[
profile
for profile in self.profiles_by_name.values()
if profile.source != path
],
active=self.active_name,
)
def to_dict(self):
"""
Convert to a dictionary suitable for writing to disk.
"""
return {
"active": self.active_name,
"profiles": {
profile.name: {
setting.name: value for setting, value in profile.settings.items()
}
for profile in self.profiles_by_name.values()
},
}
def __getitem__(self, name: str) -> Profile:
return self.profiles_by_name[name]
def __iter__(self):
return self.profiles_by_name.__iter__()
def __eq__(self, __o: object) -> bool:
if not isinstance(__o, ProfilesCollection):
return False
return (
self.profiles_by_name == __o.profiles_by_name
and self.active_name == __o.active_name
)
def __repr__(self) -> str:
return f"ProfilesCollection(profiles={list(self.profiles_by_name.values())!r}, active={self.active_name!r})>"
names
property
readonly
Type: Set[str]
Return a set of profile names in this collection.
active_profile
property
readonly
Type: Optional[prefect.settings.Profile]
Retrieve the active profile in this collection.
ProfilesCollection.set_active
Set the active profile name in the collection.
A null value may be passed to indicate that this collection does not determine the active profile.
Source code in prefect/settings.py
def set_active(self, name: Optional[str]):
"""
Set the active profile name in the collection.
A null value may be passed to indicate that this collection does not determine
the active profile.
"""
if name is not None and name not in self.names:
raise ValueError(f"Unknown profile name {name!r}.")
self.active_name = name
ProfilesCollection.update_profile
Add a profile to the collection or update the existing on if the name is already present in this collection.
If updating an existing profile, the settings will be merged. Settings can
be dropped from the existing profile by setting them to None in the new
profile.
Returns the new profile object.
Source code in prefect/settings.py
def update_profile(
self, name: str, settings: Mapping[Union[Dict, str], Any], source: Path = None
) -> Profile:
"""
Add a profile to the collection or update the existing on if the name is already
present in this collection.
If updating an existing profile, the settings will be merged. Settings can
be dropped from the existing profile by setting them to `None` in the new
profile.
Returns the new profile object.
"""
existing = self.profiles_by_name.get(name)
# Convert the input to a `Profile` to cast settings to the correct type
profile = Profile(name=name, settings=settings, source=source)
if existing:
new_settings = {**existing.settings, **profile.settings}
# Drop null keys to restore to default
for key, value in tuple(new_settings.items()):
if value is None:
new_settings.pop(key)
new_profile = Profile(
name=profile.name,
settings=new_settings,
source=source or profile.source,
)
else:
new_profile = profile
self.profiles_by_name[new_profile.name] = new_profile
return new_profile
ProfilesCollection.add_profile
Add a profile to the collection.
If the profile name already exists, an exception will be raised.
Source code in prefect/settings.py
def add_profile(self, profile: Profile) -> None:
"""
Add a profile to the collection.
If the profile name already exists, an exception will be raised.
"""
if profile.name in self.profiles_by_name:
raise ValueError(
f"Profile name {profile.name!r} already exists in collection."
)
self.profiles_by_name[profile.name] = profile
ProfilesCollection.remove_profile
Remove a profile from the collection.
Source code in prefect/settings.py
def remove_profile(self, name: str) -> None:
"""
Remove a profile from the collection.
"""
self.profiles_by_name.pop(name)
ProfilesCollection.without_profile_source
Remove profiles that were loaded from a given path.
Returns a new collection.
Source code in prefect/settings.py
def without_profile_source(self, path: Optional[Path]) -> "ProfilesCollection":
"""
Remove profiles that were loaded from a given path.
Returns a new collection.
"""
return ProfilesCollection(
[
profile
for profile in self.profiles_by_name.values()
if profile.source != path
],
active=self.active_name,
)
get_extra_loggers
value_callback for PREFECT_LOGGING_EXTRA_LOGGERSthat parses the CSV string into a
list and trims whitespace from logger names.
Source code in prefect/settings.py
def get_extra_loggers(_: "Settings", value: str) -> List[str]:
"""
`value_callback` for `PREFECT_LOGGING_EXTRA_LOGGERS`that parses the CSV string into a
list and trims whitespace from logger names.
"""
return [name.strip() for name in value.split(",")] if value else []
debug_mode_log_level
value_callback for PREFECT_LOGGING_LEVEL that overrides the log level to DEBUG
when debug mode is enabled.
Source code in prefect/settings.py
def debug_mode_log_level(settings, value):
"""
`value_callback` for `PREFECT_LOGGING_LEVEL` that overrides the log level to DEBUG
when debug mode is enabled.
"""
if PREFECT_DEBUG_MODE.value_from(settings):
return "DEBUG"
else:
return value
only_return_value_in_test_mode
value_callback for PREFECT_TEST_SETTING that only allows access during test mode
Source code in prefect/settings.py
def only_return_value_in_test_mode(settings, value):
"""
`value_callback` for `PREFECT_TEST_SETTING` that only allows access during test mode
"""
if PREFECT_TEST_MODE.value_from(settings):
return value
else:
return None
default_ui_api_url
value_callback for PREFECT_ORION_UI_API_URL that sets the default value to
PREFECT_API_URL if set otherwise it constructs an API URL from the API settings.
Source code in prefect/settings.py
def default_ui_api_url(settings, value):
"""
`value_callback` for `PREFECT_ORION_UI_API_URL` that sets the default value to
`PREFECT_API_URL` if set otherwise it constructs an API URL from the API settings.
"""
if value is None:
# Set a default value
if PREFECT_API_URL.value_from(settings):
value = "${PREFECT_API_URL}"
else:
value = "http://${PREFECT_ORION_API_HOST}:${PREFECT_ORION_API_PORT}/api"
return template_with_settings(
PREFECT_ORION_API_HOST, PREFECT_ORION_API_PORT, PREFECT_API_URL
)(settings, value)
template_with_settings
Returns a value_callback that will template the given settings into the runtime
value for the setting.
Source code in prefect/settings.py
def template_with_settings(*upstream_settings: Setting) -> Callable[["Settings", T], T]:
"""
Returns a `value_callback` that will template the given settings into the runtime
value for the setting.
"""
def templater(settings, value):
original_type = type(value)
template_values = {
setting.name: setting.value_from(settings) for setting in upstream_settings
}
template = string.Template(str(value))
return original_type(template.substitute(template_values))
return templater
max_log_size_smaller_than_batch_size
Validator for settings asserting the batch size and match log size are compatible
Source code in prefect/settings.py
def max_log_size_smaller_than_batch_size(values):
"""
Validator for settings asserting the batch size and match log size are compatible
"""
if (
values["PREFECT_LOGGING_ORION_BATCH_SIZE"]
< values["PREFECT_LOGGING_ORION_MAX_LOG_SIZE"]
):
raise ValueError(
"`PREFECT_LOGGING_ORION_MAX_LOG_SIZE` cannot be larger than `PREFECT_LOGGING_ORION_BATCH_SIZE`"
)
return values
reduce_settings
Workaround for issues with cloudpickle when using cythonized pydantic which throws exceptions when attempting to pickle the class which has "compiled" validator methods dynamically attached to it.
We cannot define this in the model class because the class is the type that contains unserializable methods.
Note that issue is not specific to the Settings model or its implementation.
Any model using some features of Pydantic (e.g. Path validation) with a Cython
compiled Pydantic installation may encounter pickling issues.
See related issue at https://github.com/cloudpipe/cloudpickle/issues/408
Source code in prefect/settings.py
def reduce_settings(settings):
"""
Workaround for issues with cloudpickle when using cythonized pydantic which
throws exceptions when attempting to pickle the class which has "compiled"
validator methods dynamically attached to it.
We cannot define this in the model class because the class is the type that
contains unserializable methods.
Note that issue is not specific to the `Settings` model or its implementation.
Any model using some features of Pydantic (e.g. `Path` validation) with a Cython
compiled Pydantic installation may encounter pickling issues.
See related issue at https://github.com/cloudpipe/cloudpickle/issues/408
"""
# TODO: Consider moving this to the cloudpickle serializer and applying it to all
# pydantic models
return (
unreduce_settings,
(settings.json(),),
)
unreduce_settings
Helper for restoring settings
Source code in prefect/settings.py
def unreduce_settings(json):
"""Helper for restoring settings"""
return Settings.parse_raw(json)
get_current_settings
Returns a settings object populated with values from the current profile or, if no profile is active, the environment.
Source code in prefect/settings.py
def get_current_settings() -> Settings:
"""
Returns a settings object populated with values from the current profile or, if no
profile is active, the environment.
"""
from prefect.context import SettingsContext
profile = SettingsContext.get()
if profile is not None:
return profile.settings
return get_settings_from_env()
get_settings_from_env
Returns a settings object populated with default values and overrides from environment variables, ignoring any values in profiles.
Calls with the same environment return a cached object instead of reconstructing to avoid validation overhead.
Source code in prefect/settings.py
def get_settings_from_env() -> Settings:
"""
Returns a settings object populated with default values and overrides from
environment variables, ignoring any values in profiles.
Calls with the same environment return a cached object instead of reconstructing
to avoid validation overhead.
"""
# Since os.environ is a Dict[str, str] we can safely hash it by contents, but we
# must be careful to avoid hashing a generator instead of a tuple
cache_key = hash(tuple((key, value) for key, value in os.environ.items()))
if cache_key not in _FROM_ENV_CACHE:
_FROM_ENV_CACHE[cache_key] = Settings()
return _FROM_ENV_CACHE[cache_key]
get_default_settings
Returns a settings object populated with default values, ignoring any overrides from environment variables or profiles.
This is cached since the defaults should not change during the lifetime of the module.
Source code in prefect/settings.py
def get_default_settings() -> Settings:
"""
Returns a settings object populated with default values, ignoring any overrides
from environment variables or profiles.
This is cached since the defaults should not change during the lifetime of the
module.
"""
global _DEFAULTS_CACHE
if not _DEFAULTS_CACHE:
old = os.environ
try:
os.environ = {}
settings = get_settings_from_env()
finally:
os.environ = old
_DEFAULTS_CACHE = settings
return _DEFAULTS_CACHE
temporary_settings
Temporarily override the current settings by entering a new profile.
See Settings.copy_with_update for details on different argument behavior.
Examples:
>>> from prefect.settings import PREFECT_API_URL
>>>
>>> with temporary_settings(updates={PREFECT_API_URL: "foo"}):
>>> assert PREFECT_API_URL.value() == "foo"
>>>
>>> with temporary_settings(set_defaults={PREFECT_API_URL: "bar"}):
>>> assert PREFECT_API_URL.value() == "foo"
>>>
>>> with temporary_settings(restore_defaults={PREFECT_API_URL}):
>>> assert PREFECT_API_URL.value() is None
>>>
>>> with temporary_settings(set_defaults={PREFECT_API_URL: "bar"})
>>> assert PREFECT_API_URL.value() == "bar"
>>> assert PREFECT_API_URL.value() is None
Source code in prefect/settings.py
@contextmanager
def temporary_settings(
updates: Mapping[Setting, Any] = None,
set_defaults: Mapping[Setting, Any] = None,
restore_defaults: Iterable[Setting] = None,
) -> Settings:
"""
Temporarily override the current settings by entering a new profile.
See `Settings.copy_with_update` for details on different argument behavior.
Example:
>>> from prefect.settings import PREFECT_API_URL
>>>
>>> with temporary_settings(updates={PREFECT_API_URL: "foo"}):
>>> assert PREFECT_API_URL.value() == "foo"
>>>
>>> with temporary_settings(set_defaults={PREFECT_API_URL: "bar"}):
>>> assert PREFECT_API_URL.value() == "foo"
>>>
>>> with temporary_settings(restore_defaults={PREFECT_API_URL}):
>>> assert PREFECT_API_URL.value() is None
>>>
>>> with temporary_settings(set_defaults={PREFECT_API_URL: "bar"})
>>> assert PREFECT_API_URL.value() == "bar"
>>> assert PREFECT_API_URL.value() is None
"""
import prefect.context
context = prefect.context.get_settings_context()
new_settings = context.settings.copy_with_update(
updates=updates, set_defaults=set_defaults, restore_defaults=restore_defaults
)
with prefect.context.SettingsContext(
profile=context.profile, settings=new_settings
):
yield new_settings
load_profiles
Load all profiles from the default and current profile paths.
Source code in prefect/settings.py
def load_profiles() -> ProfilesCollection:
"""
Load all profiles from the default and current profile paths.
"""
profiles = _read_profiles_from(DEFAULT_PROFILES_PATH)
user_profiles_path = PREFECT_PROFILES_PATH.value()
if user_profiles_path.exists():
user_profiles = _read_profiles_from(user_profiles_path)
# Merge all of the user profiles with the defaults
for name in user_profiles:
profiles.update_profile(
name,
settings=user_profiles[name].settings,
source=user_profiles[name].source,
)
# Include the active key if set
if user_profiles.active_name:
profiles.set_active(user_profiles.active_name)
return profiles
save_profiles
Writes all non-default profiles to the current profiles path.
Source code in prefect/settings.py
def save_profiles(profiles: ProfilesCollection) -> None:
"""
Writes all non-default profiles to the current profiles path.
"""
profiles_path = PREFECT_PROFILES_PATH.value()
profiles = profiles.without_profile_source(DEFAULT_PROFILES_PATH)
return _write_profiles_to(profiles_path, profiles)
load_profile
Load a single profile by name.
Source code in prefect/settings.py
def load_profile(name: str) -> Profile:
"""
Load a single profile by name.
"""
profiles = load_profiles()
try:
return profiles[name]
except KeyError:
raise ValueError(f"Profile {name!r} not found.")
update_current_profile
Update the persisted data for the profile currently in-use.
If the profile does not exist in the profiles file, it will be created.
Given settings will be merged with the existing settings as described in
ProfilesCollection.update_profile.
Returns:
| Type | Description |
|---|---|
Profile |
The new profile. |
Source code in prefect/settings.py
def update_current_profile(settings: Dict[Union[str, Setting], Any]) -> Profile:
"""
Update the persisted data for the profile currently in-use.
If the profile does not exist in the profiles file, it will be created.
Given settings will be merged with the existing settings as described in
`ProfilesCollection.update_profile`.
Returns:
The new profile.
"""
import prefect.context
current_profile = prefect.context.get_settings_context().profile
if not current_profile:
raise MissingProfileError("No profile is currently in use.")
profiles = load_profiles()
# Ensure the current profile's settings are present
profiles.update_profile(current_profile.name, current_profile.settings)
# Then merge the new settings in
profiles.update_profile(current_profile.name, settings)
save_profiles(profiles)
return profiles[current_profile.name]