Cosmos Config#

This page lists all available Airflow configurations that affect astronomer-cosmos Astronomer Cosmos behavior. They can be set in the airflow.cfg file or using environment variables.

Note

For more information, see Setting Configuration Options.

Sections:

  • [cosmos]

  • [openlineage]

[cosmos]#

cache_dir:

The directory used for caching Cosmos data.

  • Default: {TMPDIR}/cosmos_cache (where {TMPDIR} is the system temporary directory)

  • Environment Variable: AIRFLOW__COSMOS__CACHE_DIR

enable_cache:

Enable or disable caching of Cosmos data.

  • Default: True

  • Environment Variable: AIRFLOW__COSMOS__ENABLE_CACHE

enable_cache_dbt_ls:

Enable or disable caching of the dbt ls command in case using LoadMode.DBT_LS in an Airflow Variable.

  • Default: True

  • Environment Variable: AIRFLOW__COSMOS__ENABLE_CACHE_DBT_LS

enable_cache_partial_parse:

Enable or disable caching of dbt partial parse files in the local disk.

  • Default: True

  • Environment Variable: AIRFLOW__COSMOS__ENABLE_CACHE_PARTIAL_PARSE

enable_cache_package_lockfile:

Enable or disable caching of dbt project package lockfile.

  • Default: True

  • Environment Variable: AIRFLOW__COSMOS__ENABLE_CACHE_PACKAGE_LOCKFILE

propagate_logs:

Whether to propagate logs in the Cosmos module.

  • Default: True

  • Environment Variable: AIRFLOW__COSMOS__PROPAGATE_LOGS

dbt_docs_dir:

The directory path for dbt documentation.

  • Default: None

  • Environment Variable: AIRFLOW__COSMOS__DBT_DOCS_DIR

dbt_docs_conn_id:

The connection ID for dbt documentation.

  • Default: None

  • Environment Variable: AIRFLOW__COSMOS__DBT_DOCS_CONN_ID

enable_cache_profile:

Enable caching for the DBT profile.

  • Default: True

  • Environment Variable: AIRFLOW__COSMOS__ENABLE_CACHE_PROFILE

profile_cache_dir_name:

Folder name to store the DBT cached profiles. This will be a sub-folder of cache_dir

  • Default: profile

  • Environment Variable: AIRFLOW__COSMOS__PROFILE_CACHE_DIR_NAME

remote_cache_dir:

The remote directory to store the dbt cache. Starting with Cosmos 1.6.0, you can store the dbt ls output as cache in a remote location (an alternative to the Variable cache approach released previously since Cosmos 1.5.0) using this configuration. The value for the remote cache directory can be any of the schemes that are supported by the Airflow Object Store feature introduced in Airflow 2.8.0 (e.g. s3://your_s3_bucket/cache_dir/, gs://your_gs_bucket/cache_dir/, abfs://your_azure_container/cache_dir, etc.)

This is an experimental feature available since Cosmos 1.6 to gather user feedback and will be merged into the cache_dir setting in upcoming releases.

  • Default: None

  • Environment Variable: AIRFLOW__COSMOS__REMOTE_CACHE_DIR

remote_cache_dir_conn_id:

The connection ID for the remote cache directory. If this is not set, the default Airflow connection ID identified for the scheme will be used.

  • Default: None

  • Environment Variable: AIRFLOW__COSMOS__REMOTE_CACHE_DIR_CONN_ID

remote_target_path:

(Introduced since Cosmos 1.7.0) The path to the remote target directory. This is the directory designated to remotely copy & store in the files generated and stored by dbt in the dbt project’s target directory. While this remote path is intended to copy files from the dbt project’s target directory, Cosmos currently only supports copying files from the compiled directory within the target folder — and only when the execution mode is set to ExecutionMode.AIRFLOW_ASYNC. Future releases will add support for copying additional files from the target directory. The value for the remote target path can be any of the schemes that are supported by the Airflow Object Store feature introduced in Airflow 2.8.0 (e.g. s3://your_s3_bucket/target_dir/, gs://your_gs_bucket/target_dir/, abfs://your_azure_container/cache_dir, etc.)

  • Default: None

  • Environment Variable: AIRFLOW__COSMOS__REMOTE_TARGET_PATH

remote_target_path_conn_id:

(Introduced since Cosmos 1.7.0) The connection ID for the remote target path. If this is not set, the default Airflow connection ID identified for the scheme will be used.

  • Default: None

  • Environment Variable: AIRFLOW__COSMOS__REMOTE_TARGET_PATH_CONN_ID

[openlineage]#

namespace:

The OpenLineage namespace for tracking lineage.

  • Default: If not configured in Airflow configuration, it falls back to the environment variable OPENLINEAGE_NAMESPACE, otherwise it uses DEFAULT_OPENLINEAGE_NAMESPACE.

  • Environment Variable: AIRFLOW__OPENLINEAGE__NAMESPACE

Note

For more information, see Openlieage Configuration Options.

Environment Variables#

LINEAGE_NAMESPACE:

The OpenLineage namespace for tracking lineage.

  • Default: If not configured in Airflow configuration, it falls back to the environment variable OPENLINEAGE_NAMESPACE, otherwise it uses DEFAULT_OPENLINEAGE_NAMESPACE.