# Table of Contents - [Overview | Dagster Docs](#overview-dagster-docs) - [Search the documentation | Dagster Docs](#search-the-documentation-dagster-docs) - [Unit testing assets and ops | Dagster Docs](#unit-testing-assets-and-ops-dagster-docs) - [Testing partitioned config and jobs | Dagster Docs](#testing-partitioned-config-and-jobs-dagster-docs) - [Libraries | Dagster Docs](#libraries-dagster-docs) - [Creating a multi-asset integration | Dagster Docs](#creating-a-multi-asset-integration-dagster-docs) - [Using Dagster with Airbyte Cloud | Dagster Docs](#using-dagster-with-airbyte-cloud-dagster-docs) - [Dagster & Airbyte | Dagster Docs](#dagster-airbyte-dagster-docs) - [Dagster & Airbyte | Dagster Docs](#dagster-airbyte-dagster-docs) - [Dagster & Anthropic | Dagster Docs](#dagster-anthropic-dagster-docs) - [Dagster & Airlift | Dagster Docs](#dagster-airlift-dagster-docs) - [AWS | Dagster Docs](#aws-dagster-docs) - [Dagster & AWS ECR | Dagster Docs](#dagster-aws-ecr-dagster-docs) - [Dagster & AWS Athena | Dagster Docs](#dagster-aws-athena-dagster-docs) - [Dagster & AWS CloudWatch | Dagster Docs](#dagster-aws-cloudwatch-dagster-docs) - [Dagster & AWS EMR | Dagster Docs](#dagster-aws-emr-dagster-docs) - [Dagster & AWS Glue | Dagster Docs](#dagster-aws-glue-dagster-docs) - [Dagster & AWS Lambda | Dagster Docs](#dagster-aws-lambda-dagster-docs) - [Dagster & AWS Redshift | Dagster Docs](#dagster-aws-redshift-dagster-docs) - [Dagster & AWS Secrets Manager | Dagster Docs](#dagster-aws-secrets-manager-dagster-docs) - [Dagster & AWS S3 | Dagster Docs](#dagster-aws-s3-dagster-docs) - [Dagster & Azure Data Lake Storage Gen 2 | Dagster Docs](#dagster-azure-data-lake-storage-gen-2-dagster-docs) - [Dagster & AWS Systems Parameter Store | Dagster Docs](#dagster-aws-systems-parameter-store-dagster-docs) - [Dagster & Census | Dagster Docs](#dagster-census-dagster-docs) - [Dagster & Chroma | Dagster Docs](#dagster-chroma-dagster-docs) - [Dagster & Cube | Dagster Docs](#dagster-cube-dagster-docs) - [Dagster & Databricks | Dagster Docs](#dagster-databricks-dagster-docs) - [Dagster & Datadog | Dagster Docs](#dagster-datadog-dagster-docs) - [Dagster & dbt | Dagster Docs](#dagster-dbt-dagster-docs) - [Creating a dbt project in a Dagster project | Dagster Docs](#creating-a-dbt-project-in-a-dagster-project-dagster-docs) - [Add a downstream asset | Dagster Docs](#add-a-downstream-asset-dagster-docs) - [Load dbt models as Dagster assets | Dagster Docs](#load-dbt-models-as-dagster-assets-dagster-docs) - [Set up the dbt project | Dagster Docs](#set-up-the-dbt-project-dagster-docs) - [Define assets upstream of your dbt models | Dagster Docs](#define-assets-upstream-of-your-dbt-models-dagster-docs) - [One doc tagged with "other" | Dagster Docs](#one-doc-tagged-with-other-dagster-docs) - [Migration & upgrading | Dagster Docs](#migration-upgrading-dagster-docs) - [3 docs tagged with "monitoring" | Dagster Docs](#3-docs-tagged-with-monitoring-dagster-docs) - [Airflow to Dagster | Dagster Docs](#airflow-to-dagster-dagster-docs) - [Migrate from Dagster OSS to Dagster+ | Dagster Docs](#migrate-from-dagster-oss-to-dagster-dagster-docs) - [4 docs tagged with "reference-architecture" | Dagster Docs](#4-docs-tagged-with-reference-architecture-dagster-docs) - [5 docs tagged with "code-example" | Dagster Docs](#5-docs-tagged-with-code-example-dagster-docs) - [Using Dagster and Airflow together | Dagster Docs](#using-dagster-and-airflow-together-dagster-docs) - [Migrating Airflow operators to Dagster | Dagster Docs](#migrating-airflow-operators-to-dagster-dagster-docs) - [Migrate from Dagster+ Serverless to Hybrid | Dagster Docs](#migrate-from-dagster-serverless-to-hybrid-dagster-docs) - [Migrating an Airflow BashOperator to Dagster | Dagster Docs](#migrating-an-airflow-bashoperator-to-dagster-dagster-docs) - [Migrating an Airflow PythonOperator to Dagster | Dagster Docs](#migrating-an-airflow-pythonoperator-to-dagster-dagster-docs) - [Migrating an Airflow KubernetesPodOperator to Dagster | Dagster Docs](#migrating-an-airflow-kubernetespodoperator-to-dagster-dagster-docs) - [Migrating an Airflow BashOperator (dbt) to Dagster | Dagster Docs](#migrating-an-airflow-bashoperator-dbt-to-dagster-dagster-docs) - [Setup | Dagster Docs](#setup-dagster-docs) - [Observe the Airflow DAG | Dagster Docs](#observe-the-airflow-dag-dagster-docs) - [Federate execution | Dagster Docs](#federate-execution-dagster-docs) - [Setup | Dagster Docs](#setup-dagster-docs) - [Federate execution across Airflow instances with Dagster | Dagster Docs](#federate-execution-across-airflow-instances-with-dagster-dagster-docs) - [Airflow to Dagster migration reference | Dagster Docs](#airflow-to-dagster-migration-reference-dagster-docs) - [Migrate DAG-mapped assets | Dagster Docs](#migrate-dag-mapped-assets-dagster-docs) - [Observe multiple Airflow instances from Dagster | Dagster Docs](#observe-multiple-airflow-instances-from-dagster-dagster-docs) - [Migrate from Airflow to Dagster | Dagster Docs](#migrate-from-airflow-to-dagster-dagster-docs) - [14 docs tagged with "compute" | Dagster Docs](#14-docs-tagged-with-compute-dagster-docs) - [Peer the Airflow instance with a Dagster code location | Dagster Docs](#peer-the-airflow-instance-with-a-dagster-code-location-dagster-docs) - [Setup | Dagster Docs](#setup-dagster-docs) - [Decommission the Airflow DAG | Dagster Docs](#decommission-the-airflow-dag-dagster-docs) - [16 docs tagged with "storage" | Dagster Docs](#16-docs-tagged-with-storage-dagster-docs) - [Migrate from Airflow to Dagster at the task level | Dagster Docs](#migrate-from-airflow-to-dagster-at-the-task-level-dagster-docs) - [Migrate from Airflow to Dagster at the DAG level | Dagster Docs](#migrate-from-airflow-to-dagster-at-the-dag-level-dagster-docs) - [Migrate Airflow tasks | Dagster Docs](#migrate-airflow-tasks-dagster-docs) - [Peer the Airflow instance with a Dagster code location | Dagster Docs](#peer-the-airflow-instance-with-a-dagster-code-location-dagster-docs) - [Observe Airflow tasks | Dagster Docs](#observe-airflow-tasks-dagster-docs) - [15 docs tagged with "ETL" | Dagster Docs](#15-docs-tagged-with-etl-dagster-docs) - [Upgrading Dagster | Dagster Docs](#upgrading-dagster-dagster-docs) - [Dagster & dbt Cloud | Dagster Docs](#dagster-dbt-cloud-dagster-docs) - [Quickstart | Dagster Docs](#quickstart-dagster-docs) - [32 docs tagged with "Community supported" | Dagster Docs](#32-docs-tagged-with-community-supported-dagster-docs) - [Dagster & dbt Cloud (Legacy) | Dagster Docs](#dagster-dbt-cloud-legacy-dagster-docs) - [dagster-deltalake integration reference | Dagster Docs](#dagster-deltalake-integration-reference-dagster-docs) - [48 docs tagged with "Dagster supported" | Dagster Docs](#48-docs-tagged-with-dagster-supported-dagster-docs) - [Dagster & DingTalk | Dagster Docs](#dagster-dingtalk-dagster-docs) - [Dagster & Delta Lake | Dagster Docs](#dagster-delta-lake-dagster-docs) - [Dagster & Embedded ELT | Dagster Docs](#dagster-embedded-elt-dagster-docs) - [Using Delta Lake with Dagster | Dagster Docs](#using-delta-lake-with-dagster-dagster-docs) - [Making a dbt project accessible to Dagster+ Hybrid | Dagster Docs](#making-a-dbt-project-accessible-to-dagster-hybrid-dagster-docs) - [Dagster & DuckDB | Dagster Docs](#dagster-duckdb-dagster-docs) - [Using dbt with Dagster+ | Dagster Docs](#using-dbt-with-dagster-dagster-docs) - [Using Dagster with Fivetran | Dagster Docs](#using-dagster-with-fivetran-dagster-docs) - [dagster-duckdb integration reference | Dagster Docs](#dagster-duckdb-integration-reference-dagster-docs) - [Dagster & HashiCorp Vault | Dagster Docs](#dagster-hashicorp-vault-dagster-docs) - [Dagster & Evidence | Dagster Docs](#dagster-evidence-dagster-docs) - [Dagster & GitHub | Dagster Docs](#dagster-github-dagster-docs) - [Dagster & dlt | Dagster Docs](#dagster-dlt-dagster-docs) - [GCP | Dagster Docs](#gcp-dagster-docs) - [Dagster & HashiCorp | Dagster Docs](#dagster-hashicorp-dagster-docs) - [Dagster & Java | Dagster Docs](#dagster-java-dagster-docs) - [Using DuckDB with Dagster | Dagster Docs](#using-duckdb-with-dagster-dagster-docs) - [Dagster & Hex | Dagster Docs](#dagster-hex-dagster-docs) - [Dagster & Hightouch | Dagster Docs](#dagster-hightouch-dagster-docs) - [Importing a dbt project to Dagster+ Serverless | Dagster Docs](#importing-a-dbt-project-to-dagster-serverless-dagster-docs) - [Dagster & Meltano | Dagster Docs](#dagster-meltano-dagster-docs) - [Dagster & LakeFS | Dagster Docs](#dagster-lakefs-dagster-docs) - [Dagster & GCP Cloud Run | Dagster Docs](#dagster-gcp-cloud-run-dagster-docs) - [Features | Dagster Docs](#features-dagster-docs) - [Quickstart | Dagster Docs](#quickstart-dagster-docs) - [Dagster & Modal | Dagster Docs](#dagster-modal-dagster-docs) - [Dagster & MSSQL Bulk Copy Tool | Dagster Docs](#dagster-mssql-bulk-copy-tool-dagster-docs) - [Dagster & PagerDuty | Dagster Docs](#dagster-pagerduty-dagster-docs) - [Dagster & Not Diamond | Dagster Docs](#dagster-not-diamond-dagster-docs) - [Dagster & Rust | Dagster Docs](#dagster-rust-dagster-docs) - [Dagster & GCP GCS | Dagster Docs](#dagster-gcp-gcs-dagster-docs) - [Dagster & Perian | Dagster Docs](#dagster-perian-dagster-docs) - [Dagster & obstore | Dagster Docs](#dagster-obstore-dagster-docs) - [dagster-dbt integration reference | Dagster Docs](#dagster-dbt-integration-reference-dagster-docs) - [Dagster & Ray | Dagster Docs](#dagster-ray-dagster-docs) - [Dagster & Patito | Dagster Docs](#dagster-patito-dagster-docs) - [Dagster & Polars | Dagster Docs](#dagster-polars-dagster-docs) - [Dagster & Secoda | Dagster Docs](#dagster-secoda-dagster-docs) - [Dagster & Slack | Dagster Docs](#dagster-slack-dagster-docs) - [Dagster & Qdrant | Dagster Docs](#dagster-qdrant-dagster-docs) - [Dagster & Gemini | Dagster Docs](#dagster-gemini-dagster-docs) - [Dagster & GCP BigQuery | Dagster Docs](#dagster-gcp-bigquery-dagster-docs) - [Dagster & Docker | Dagster Docs](#dagster-docker-dagster-docs) - [Dagster & Prometheus | Dagster Docs](#dagster-prometheus-dagster-docs) - [Dagster & TypeScript | Dagster Docs](#dagster-typescript-dagster-docs) - [Dagster & Weights & Biases | Dagster Docs](#dagster-weights-biases-dagster-docs) - [Dagster & Jupyter Notebooks | Dagster Docs](#dagster-jupyter-notebooks-dagster-docs) - [Dagster & SSH/SFTP | Dagster Docs](#dagster-ssh-sftp-dagster-docs) - [Dagster & Iceberg | Dagster Docs](#dagster-iceberg-dagster-docs) - [Dagster & Twilio | Dagster Docs](#dagster-twilio-dagster-docs) - [Dagster & GCP Dataproc | Dagster Docs](#dagster-gcp-dataproc-dagster-docs) - [Dagster & Pandera | Dagster Docs](#dagster-pandera-dagster-docs) - [Dagster & Weaviate | Dagster Docs](#dagster-weaviate-dagster-docs) - [dagstermill integration reference | Dagster Docs](#dagstermill-integration-reference-dagster-docs) - [Usage | Dagster Docs](#usage-dagster-docs) - [Dagster & Microsoft Teams | Dagster Docs](#dagster-microsoft-teams-dagster-docs) - [Dagster & Open Metadata | Dagster Docs](#dagster-open-metadata-dagster-docs) - [Dagster & Power BI | Dagster Docs](#dagster-power-bi-dagster-docs) - [Using Jupyter notebooks with Papermill and Dagster | Dagster Docs](#using-jupyter-notebooks-with-papermill-and-dagster-dagster-docs) - [Dagster & Looker | Dagster Docs](#dagster-looker-dagster-docs) - [Dagster & Pandas | Dagster Docs](#dagster-pandas-dagster-docs) - [BigQuery integration reference | Dagster Docs](#bigquery-integration-reference-dagster-docs) - [Dagster & Kubernetes | Dagster Docs](#dagster-kubernetes-dagster-docs) - [Dagster & Snowflake | Dagster Docs](#dagster-snowflake-dagster-docs) - [Using Google BigQuery with Dagster | Dagster Docs](#using-google-bigquery-with-dagster-dagster-docs) - [Dagster & OpenAI | Dagster Docs](#dagster-openai-dagster-docs) - [Dagster & Sling | Dagster Docs](#dagster-sling-dagster-docs) - [Dagster & Teradata | Dagster Docs](#dagster-teradata-dagster-docs) - [Dagster & Sigma | Dagster Docs](#dagster-sigma-dagster-docs) - [dagster-snowflake integration reference | Dagster Docs](#dagster-snowflake-integration-reference-dagster-docs) - [Using Snowflake with with Dagster I/O managers | Dagster Docs](#using-snowflake-with-with-dagster-i-o-managers-dagster-docs) - [Using Snowflake with Dagster resources | Dagster Docs](#using-snowflake-with-dagster-resources-dagster-docs) - [Dagster & Tableau | Dagster Docs](#dagster-tableau-dagster-docs) - [Dagster & Spark | Dagster Docs](#dagster-spark-dagster-docs) - [Tags | Dagster Docs](#tags-dagster-docs) - [5 docs tagged with "BI" | Dagster Docs](#5-docs-tagged-with-bi-dagster-docs) - [4 docs tagged with "alerting" | Dagster Docs](#4-docs-tagged-with-alerting-dagster-docs) - [6 docs tagged with "metadata" | Dagster Docs](#6-docs-tagged-with-metadata-dagster-docs) - [2 docs tagged with "AI" | Dagster Docs](#2-docs-tagged-with-ai-dagster-docs) --- # Overview | Dagster Docs [Skip to main content](https://docs.dagster.io/#__docusaurus_skipToContent_fallback) Dagster is a data orchestrator built for data engineers, with integrated lineage, observability, a declarative programming model, and best-in-class testability. defs/assets.py import dagster as dg@dg.assetdef hello(context: dg.AssetExecutionContext): context.log.info("Hello!")@dg.asset(deps=[hello])def world(context: dg.AssetExecutionContext): context.log.info("World!") ![Docusaurus themed image](https://docs.dagster.io/img/getting-started/lineage-light.jpg)![Docusaurus themed image](https://docs.dagster.io/img/getting-started/lineage-dark.jpg) Get started[​](https://docs.dagster.io/#get-started "Direct link to Get started") ---------------------------------------------------------------------------------- [![Quickstart](https://docs.dagster.io/img/getting-started/icon-start.svg)\ \ ### Quickstart\ \ Build your first Dagster pipeline in our Quickstart tutorial.](https://docs.dagster.io/getting-started/quickstart) [![Thinking in Assets](https://docs.dagster.io/img/getting-started/icon-assets.svg)\ \ ### Thinking in Assets\ \ New to Dagster? Learn about how thinking in assets can help you manage your data better.](https://docs.dagster.io/guides/build/assets/) [![Dagster Plus](https://docs.dagster.io/img/getting-started/icon-plus.svg)\ \ ### Dagster Plus\ \ Learn about Dagster Plus, our managed offering that includes a hosted Dagster instance and many more features.](https://docs.dagster.io/deployment/dagster-plus) Join the Dagster community[​](https://docs.dagster.io/#join-the-dagster-community "Direct link to Join the Dagster community") ------------------------------------------------------------------------------------------------------------------------------- [![Slack](https://docs.dagster.io/img/getting-started/icon-slack.svg)\ \ ### Slack\ \ Join our Slack community to talk with other Dagster users, use our AI-powered chatbot, and get help with Dagster.](https://dagster.io/slack) [![GitHub](https://docs.dagster.io/img/getting-started/icon-github.svg)\ \ ### GitHub\ \ Star our GitHub repository and follow our development through GitHub Discussions.](https://github.com/dagster-io/dagster) [![Youtube](https://docs.dagster.io/img/getting-started/icon-youtube.svg)\ \ ### Youtube\ \ Watch our latest videos on YouTube.](https://www.youtube.com/@dagsterio) [![Dagster University](https://docs.dagster.io/img/getting-started/icon-education.svg)\ \ ### Dagster University\ \ Learn Dagster through interactive courses and hands-on tutorials.](https://courses.dagster.io/) --- # Search the documentation | Dagster Docs [Skip to main content](https://docs.dagster.io/search#__docusaurus_skipToContent_fallback) Search the documentation ======================== [](https://www.algolia.com/) --- # Unit testing assets and ops | Dagster Docs [Skip to main content](https://docs.dagster.io/guides/test/unit-testing-assets-and-ops#__docusaurus_skipToContent_fallback) On this page Unit testing is essential for ensuring that computations function as intended. In the context of data pipelines, this can be particularly challenging. However, Dagster streamlines the process by enabling direct invocation of computations with specified input values and mocked resources, making it easier to verify that data transformations behave as expected. While unit tests can't fully replace integration tests or manual review, they can catch a variety of errors with a significantly faster feedback loop. This article covers how to write unit tests for assets with a variety of different input requirements. note Before you begin implementing unit tests, note that: * Testing individual assets is generally recommended over unit testing entire jobs. * Unit testing isn't recommended in cases where most of the business logic is encoded in an external system, such as an asset which directly invokes an external Databricks job. * If you want to test your assets at runtime, you can use [asset checks](https://docs.dagster.io/guides/test/asset-checks) to verify the quality of data produced by your pipelines, communicate what the data is expected to do, and more. Unit test examples[​](https://docs.dagster.io/guides/test/unit-testing-assets-and-ops#unit-test-examples "Direct link to Unit test examples") ---------------------------------------------------------------------------------------------------------------------------------------------- ### Assets and ops without arguments[​](https://docs.dagster.io/guides/test/unit-testing-assets-and-ops#no-arguments "Direct link to Assets and ops without arguments") The simplest assets to test are those with no arguments. In these cases, you can directly invoke definitions. src//defs/assets.py import dagster as dg@dg.assetdef loaded_file() -> str: with open("path.txt") as file: return file.read() tests/test\_assets.py def test_loaded_file() -> None: assert loaded_file() == "contents" ### Assets with upstream dependencies[​](https://docs.dagster.io/guides/test/unit-testing-assets-and-ops#upstream-dependencies "Direct link to Assets with upstream dependencies") If an asset has an upstream dependency, you can directly pass a value for that dependency when invoking the definition. src//defs/assets.py import dagster as dg@dg.assetdef loaded_file() -> str: with open("path.txt") as file: return file.read()@dg.assetdef processed_file(loaded_file: str) -> str: return loaded_file.strip() tests/test\_assets.py def test_processed_file() -> None: assert processed_file(" contents ") == "contents" ### Assets with config[​](https://docs.dagster.io/guides/test/unit-testing-assets-and-ops#config "Direct link to Assets with config") If an asset uses config, you can construct an instance of the required config object and pass it in directly. src//defs/assets.py import dagster as dgclass FilepathConfig(dg.Config): path: str@dg.assetdef loaded_file(config: FilepathConfig) -> str: with open(config.path) as file: return file.read() tests/test\_assets.py def test_loaded_file() -> None: assert loaded_file(FilepathConfig(path="path1.txt")) == "contents1" assert loaded_file(FilepathConfig(path="path2.txt")) == "contents2" ### Assets with resources[​](https://docs.dagster.io/guides/test/unit-testing-assets-and-ops#resources "Direct link to Assets with resources") If an asset uses a resource, it can be useful to create a mock instance of the resource to avoid interacting with external services. src//defs/assets.py from dagster_aws.s3 import S3FileHandle, S3FileManagerimport dagster as dg@dg.assetdef loaded_file(file_manager: S3FileManager) -> str: return file_manager.read_data(S3FileHandle("bucket", "path.txt")) tests/test\_assets.py from unittest import mockdef test_file() -> None: mocked_resource = mock.Mock(spec=S3FileManager) mocked_resource.read_data.return_value = "contents" assert loaded_file(mocked_resource) == "contents" assert mocked_resource.read_data.called_once_with( S3FileHandle("bucket", "path.txt") ) ### Assets with context[​](https://docs.dagster.io/guides/test/unit-testing-assets-and-ops#context "Direct link to Assets with context") If an asset uses a `context` argument, you can use `build_asset_context()` to construct a context object. src//defs/assets.py import dagster as dg@dg.asset(partitions_def=dg.DailyPartitionsDefinition("2024-01-01"))def loaded_file(context: dg.AssetExecutionContext) -> str: with open(f"path_{context.partition_key}.txt") as file: return file.read() tests/test\_assets.py def test_loaded_file() -> None: context = dg.build_asset_context(partition_key="2024-08-16") assert loaded_file(context) == "Contents for August 16th, 2024" ### Assets with multiple parameters[​](https://docs.dagster.io/guides/test/unit-testing-assets-and-ops#multiple-parameters "Direct link to Assets with multiple parameters") If an asset has multiple parameters, we recommended using keyword arguments for clarity. src//defs/assets.py import dagster as dgclass SeparatorConfig(dg.Config): separator: str@dg.assetdef processed_file( primary_file: str, secondary_file: str, config: SeparatorConfig) -> str: return f"{primary_file}{config.separator}{secondary_file}" tests/test\_assets.py def test_processed_file() -> None: assert ( processed_file( primary_file="abc", secondary_file="def", config=SeparatorConfig(separator=","), ) == "abc,def" ) Running the tests[​](https://docs.dagster.io/guides/test/unit-testing-assets-and-ops#running-the-tests "Direct link to Running the tests") ------------------------------------------------------------------------------------------------------------------------------------------- Use `pytest` or your test runner of choice to run your unit tests. Navigate to the top-level project directory (the one that contains the tests directory) and run: pytest tests * [Unit test examples](https://docs.dagster.io/guides/test/unit-testing-assets-and-ops#unit-test-examples) * [Assets and ops without arguments](https://docs.dagster.io/guides/test/unit-testing-assets-and-ops#no-arguments) * [Assets with upstream dependencies](https://docs.dagster.io/guides/test/unit-testing-assets-and-ops#upstream-dependencies) * [Assets with config](https://docs.dagster.io/guides/test/unit-testing-assets-and-ops#config) * [Assets with resources](https://docs.dagster.io/guides/test/unit-testing-assets-and-ops#resources) * [Assets with context](https://docs.dagster.io/guides/test/unit-testing-assets-and-ops#context) * [Assets with multiple parameters](https://docs.dagster.io/guides/test/unit-testing-assets-and-ops#multiple-parameters) * [Running the tests](https://docs.dagster.io/guides/test/unit-testing-assets-and-ops#running-the-tests) --- # Testing partitioned config and jobs | Dagster Docs [Skip to main content](https://docs.dagster.io/guides/test/testing-partitioned-config-and-jobs#__docusaurus_skipToContent_fallback) On this page In this article, we'll cover a few ways to test your partitioned config and jobs. note This article assumes familiarity with [partitioned assets](https://docs.dagster.io/guides/build/partitions-and-backfills/partitioning-assets) . Testing partitioned config[​](https://docs.dagster.io/guides/test/testing-partitioned-config-and-jobs#testing-partitioned-config "Direct link to Testing partitioned config") ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ Invoking a [`PartitionedConfig`](https://docs.dagster.io/api/dagster/partitions#dagster.PartitionedConfig) object directly invokes the decorated function. If you want to check whether the generated run config is valid for the config of a job, you can use the [`validate_run_config`](https://docs.dagster.io/api/dagster/execution#dagster.validate_run_config) function. src//defs/ops.py import dagster as dgfrom datetime import datetime@dg.daily_partitioned_config(start_date=datetime(2020, 1, 1))def my_partitioned_config(start: datetime, _end: datetime): return { "ops": { "process_data_for_date": {"config": {"date": start.strftime("%Y-%m-%d")}} } } tests/test\_ops.py import dagster as dgdef test_my_partitioned_config(): # assert that the decorated function returns the expected output run_config = my_partitioned_config(datetime(2020, 1, 3), datetime(2020, 1, 4)) assert run_config == { "ops": {"process_data_for_date": {"config": {"date": "2020-01-03"}}} } # assert that the output of the decorated function is valid configuration for the # partitioned_op_job job assert dg.validate_run_config(partitioned_op_job, run_config) If you want to test that a [`PartitionedConfig`](https://docs.dagster.io/api/dagster/partitions#dagster.PartitionedConfig) creates the partitions you expect, use the `get_partition_keys` or `get_run_config_for_partition_key` functions: src//defs/ops.py import dagster as dg@dg.daily_partitioned_config(start_date=datetime(2020, 1, 1), minute_offset=15)def my_offset_partitioned_config(start: datetime, _end: datetime): return { "ops": { "process_data": { "config": { "start": start.strftime("%Y-%m-%d-%H:%M"), "end": _end.strftime("%Y-%m-%d-%H:%M"), } } } }class ProcessDataConfig(dg.Config): start: str end: str@opdef process_data(context: dg.OpExecutionContext, config: ProcessDataConfig): s = config.start e = config.end context.log.info(f"processing data for {s} - {e}")@job(config=my_offset_partitioned_config)def do_more_stuff_partitioned(): process_data() tests/test\_ops.py def test_my_offset_partitioned_config(): # test that the partition keys are what you expect keys = my_offset_partitioned_config.get_partition_keys() assert keys[0] == "2020-01-01" assert keys[1] == "2020-01-02" # test that the run_config for a partition is valid for partitioned_op_job run_config = my_offset_partitioned_config.get_run_config_for_partition_key(keys[0]) assert dg.validate_run_config(do_more_stuff_partitioned, run_config) # test that the contents of run_config are what you expect assert run_config == { "ops": { "process_data": { "config": {"start": "2020-01-01-00:15", "end": "2020-01-02-00:15"} } } } Testing partitioned jobs[​](https://docs.dagster.io/guides/test/testing-partitioned-config-and-jobs#testing-partitioned-jobs "Direct link to Testing partitioned jobs") ------------------------------------------------------------------------------------------------------------------------------------------------------------------------ To run a partitioned job in-process on a particular partition, supply a value for the `partition_key` argument of [`dagster.JobDefinition.execute_in_process`](https://docs.dagster.io/api/dagster/execution) : tests/test\_ops.py def test_partitioned_op_job(): assert partitioned_op_job.execute_in_process(partition_key="2020-01-01").success * [Testing partitioned config](https://docs.dagster.io/guides/test/testing-partitioned-config-and-jobs#testing-partitioned-config) * [Testing partitioned jobs](https://docs.dagster.io/guides/test/testing-partitioned-config-and-jobs#testing-partitioned-jobs) --- # Libraries | Dagster Docs [Skip to main content](https://docs.dagster.io/integrations/libraries#__docusaurus_skipToContent_fallback) You can integrate Dagster with external services or non-Python languages using our libraries and libraries supported by the community. --- # Creating a multi-asset integration | Dagster Docs [Skip to main content](https://docs.dagster.io/integrations/guides/multi-asset-integration#__docusaurus_skipToContent_fallback) On this page When working in the Dagster ecosystem, you may have noticed that decorators are frequently used. For example, assets, jobs, and ops use decorators. If you have a service that produces many assets, it's possible to define it as a multi-asset decorator-offering a consistent and intuitive developer experience to existing Dagster APIs. In the context of Dagster, decorators are helpful because they often wrap some form of processing. For example, when writing an asset, you define your processing code and then annotate the function with the `asset` decorator /> decorator. Then, the internal Dagster code can register the asset, assign metadata, pass in context data, or perform any other variety of operations that are required to integrate your asset code with the Dagster platform. In this guide, you'll learn how to develop a multi-asset integration for a hypothetical replication tool. note This guide assumes basic familiarity with Dagster and Python decorators. Step 1: Input[​](https://docs.dagster.io/integrations/guides/multi-asset-integration#step-1-input "Direct link to Step 1: Input") ---------------------------------------------------------------------------------------------------------------------------------- For this guide, let's imagine a tool that replicates data between two databases. It's configured using a `replication.yaml` configuration file, in which a user is able to define source and destination databases, along with the tables that they would like to replicate between these systems. connections: source: type: duckdb connection: example.duckdb destination: type: postgres connection: postgresql://postgres:postgres@localhost/postgrestables: - name: users primary_key: id - name: products primary_key: id - name: activity primary_key: id For the integration we're building, we want to provide a multi-asset that encompasses this replication process, and generates an asset for each table being replicated. We will define a dummy function named `replicate` that will mock the replication process, and return a dictionary with the replication status of each table. In the real world, this could be a function in a library, or a call to a command-line tool. import yamlfrom pathlib import Pathfrom typing import Mapping, Iterator, Anydef replicate(replication_configuration_yaml: Path) -> Iterator[Mapping[str, Any]]: data = yaml.safe_load(replication_configuration_yaml.read_text()) for table in data.get("tables"): # < perform replication here, and get status > yield {"table": table.get("name"), "status": "success"} Step 2: Implementation[​](https://docs.dagster.io/integrations/guides/multi-asset-integration#step-2-implementation "Direct link to Step 2: Implementation") ------------------------------------------------------------------------------------------------------------------------------------------------------------- First, let's define a `Project` object that takes in the path of our configuration YAML file. This will allow us to encapsulate the logic that gets metadata and table information from our project configuration. import yamlfrom pathlib import Pathclass ReplicationProject(): def __init__(self, replication_configuration_yaml: str): self.replication_configuration_yaml = replication_configuration_yaml def load(self): return yaml.safe_load(Path(self.replication_configuration_yaml).read_text()) Next, define a function that returns a `multi_asset` function. The `multi_asset` function is a decorator itself, so this allows us to customize the behavior of `multi_asset` and create a new decorator of our own: def custom_replication_assets( *, replication_project: ReplicationProject, name: Optional[str] = None, group_name: Optional[str] = None,) -> Callable[[Callable[..., Any]], AssetsDefinition]: project = replication_project.load() return multi_asset( name=name, group_name=group_name, specs=[ AssetSpec( key=table.get("name"), ) for table in project.get("tables") ], ) Let's review what this code does: * Defines a function that returns a `multi_asset` function * Loads our replication project and iterates over the tables defined in the input YAML file * Uses the tables to create a list of `AssetSpec` objects and passes them to the `specs` parameter, thus defining assets that will be visible in the Dagster UI Next, we'll show you how to perform the execution of the replication function. Recall that decorators allow us to wrap a function that performs some operation. In the case of our `multi_asset`, we defined `AssetSpec` objects for our tables, and the actual processing that takes place will be in the body of the decorated function. In this function, we will perform the replication, and then yield `AssetMaterialization` objects indicating that the replication was successful for a given table. from dagster import AssetExecutionContextreplication_project_path = "replication.yaml"replication_project = ReplicationProject(replication_project_path)@custom_replication_assets( replication_project=replication_project, name="my_custom_replication_assets", group_name="replication",)def my_assets(context: AssetExecutionContext): results = replicate(Path(replication_project_path)) for table in results: if table.get("status") == "SUCCESS": yield AssetMaterialization(asset_key=str(table.get("name")), metadata=table) There are a few limitations to this approach: * **We have not encapsulated the logic for replicating tables.** This means that users who use the `custom_replication_assets` decorator would be responsible for yielding asset materializations themselves. * **Users can't customize the attributes of the asset**. For the first limitation, we can resolve this by refactoring the code in the body of our asset function into a Dagster resource. Step 3: Moving the replication logic into a resource[​](https://docs.dagster.io/integrations/guides/multi-asset-integration#step-3-moving-the-replication-logic-into-a-resource "Direct link to Step 3: Moving the replication logic into a resource") ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Refactoring the replication logic into a resource enables us to support better configuration and re-use of our logic. To accomplish this, we will extend the `ConfigurableResource` object to create a custom resource. Then, we will define a `run` method that will perform the replication operation: from dagster import ConfigurableResourcefrom dagster._annotations import publicclass ReplicationResource(ConfigurableResource): @public def run( self, replication_project: ReplicationProject ) -> Iterator[AssetMaterialization]: results = replicate(Path(replication_project.replication_configuration_yaml)) for table in results: if table.get("status") == "SUCCESS": # NOTE: this assumes that the table name is the same as the asset key yield AssetMaterialization( asset_key=str(table.get("name")), metadata=table ) Now, we can refactor our `custom_replication_assets` instance to use this resource: @custom_replication_assets( replication_project=replication_project, name="my_custom_replication_assets", group_name="replication",)def my_assets(replication_resource: ReplicationProject): replication_resource.run(replication_project) Step 4: Using translators[​](https://docs.dagster.io/integrations/guides/multi-asset-integration#step-4-using-translators "Direct link to Step 4: Using translators") ---------------------------------------------------------------------------------------------------------------------------------------------------------------------- At the end of [Step 2](https://docs.dagster.io/integrations/guides/multi-asset-integration#step-2-implementation) , we mentioned that end users were unable to customize asset attributes, like the asset key, generated by our decorator. Translator classes are the recommended way of defining this logic, and they provide users with the option to override the default methods used to convert a concept from your tool (for example, a table name) to the corresponding concept in Dagster (for example, asset key). To start, we will define a translator method to map the table specification to a Dagster asset key. note In a real world integration, you will want to define methods for all common attributes like dependencies, group names, and metadata. from dagster import AssetKey, _check as checkfrom dataclasses import dataclass@dataclassclass ReplicationTranslator: @public def get_asset_key(self, table_definition: Mapping[str, str]) -> AssetKey: return AssetKey(str(table_definition.get("name"))) Next, we'll update `custom_replication_assets` to use the translator when defining the `key` on the `AssetSpec`. note Note that we took this opportunity to also include the replication project and translator instance on the `AssetSpec` metadata. This is a workaround that we tend to employ in this approach, as it makes it possible to define these objects once and then access them on the context of our asset. def custom_replication_assets( *, replication_project: ReplicationProject, name: Optional[str] = None, group_name: Optional[str] = None, translator: Optional[ReplicationTranslator] = None,) -> Callable[[Callable[..., Any]], AssetsDefinition]: project = replication_project.load() translator = ( check.opt_inst_param(translator, "translator", ReplicationTranslator) or ReplicationTranslator() ) return multi_asset( name=name, group_name=group_name, specs=[ AssetSpec( key=translator.get_asset_key(table), metadata={ "replication_project": project, "replication_translator": translator, }, ) for table in project.get("tables") ], ) Finally, we have to update our resource to use the translator and project provided in the metadata. We are using the `check` method provided by `dagster._check` to ensure that the type of the object is appropriate as we retrieve it from the metadata. Now, we can use the same `translator.get_asset_key` when yielding the asset materialization, thus ensuring that our asset declarations match our asset materializations: class ReplicationResource(ConfigurableResource): @public def run(self, context: AssetExecutionContext) -> Iterator[AssetMaterialization]: metadata_by_key = context.assets_def.metadata_by_key first_asset_metadata = next(iter(metadata_by_key.values())) project = check.inst( first_asset_metadata.get("replication_project"), ReplicationProject, ) translator = check.inst( first_asset_metadata.get("replication_translator"), ReplicationTranslator, ) results = replicate(Path(project.replication_configuration_yaml)) for table in results: if table.get("status") == "SUCCESS": yield AssetMaterialization( asset_key=translator.get_asset_key(table), metadata=table ) Conclusion[​](https://docs.dagster.io/integrations/guides/multi-asset-integration#conclusion "Direct link to Conclusion") -------------------------------------------------------------------------------------------------------------------------- In this guide we walked through how to define a custom multi-asset decorator, a resource for encapsulating tool logic, and a translator for defining the logic to translate a specification to Dagster concepts. Defining integrations with this approach aligns nicely with the overall development paradigm of Dagster, and is suitable for tools that generate many assets. The code in its entirety can be seen below: from collections.abc import Iterator, Mappingfrom dataclasses import dataclassfrom pathlib import Pathfrom typing import Any, Callable, Optionalimport yamlimport dagster as dgimport dagster._check as checkfrom dagster._annotations import publicdef replicate(replication_configuration_yaml: Path) -> Iterator[Mapping[str, Any]]: data = yaml.safe_load(replication_configuration_yaml.read_text()) for table in data.get("tables"): # < perform replication here, and get status > yield {"table": table.get("name"), "status": "success"}class ReplicationProject: def __init__(self, replication_configuration_yaml: str): self.replication_configuration_yaml = replication_configuration_yaml def load(self): return yaml.safe_load(Path(self.replication_configuration_yaml).read_text())class ReplicationResource(dg.ConfigurableResource): @public def run( self, context: dg.AssetExecutionContext ) -> Iterator[dg.AssetMaterialization]: metadata_by_key = context.assets_def.metadata_by_key first_asset_metadata = next(iter(metadata_by_key.values())) project = check.inst( first_asset_metadata.get("replication_project"), ReplicationProject, ) translator = check.inst( first_asset_metadata.get("replication_translator"), ReplicationTranslator, ) results = replicate(Path(project.replication_configuration_yaml)) for table in results: if table.get("status") == "SUCCESS": yield dg.AssetMaterialization( asset_key=translator.get_asset_key(table), metadata=table )@dataclassclass ReplicationTranslator: @public def get_asset_key(self, table_definition: Mapping[str, str]) -> dg.AssetKey: return dg.AssetKey(str(table_definition.get("name")))def custom_replication_assets( *, replication_project: ReplicationProject, name: Optional[str] = None, group_name: Optional[str] = None, translator: Optional[ReplicationTranslator] = None,) -> Callable[[Callable[..., Any]], dg.AssetsDefinition]: project = replication_project.load() translator = ( check.opt_inst_param(translator, "translator", ReplicationTranslator) or ReplicationTranslator() ) return dg.multi_asset( name=name, group_name=group_name, specs=[ dg.AssetSpec( key=translator.get_asset_key(table), metadata={ "replication_project": project, "replication_translator": translator, }, ) for table in project.get("tables") ], ) * [Step 1: Input](https://docs.dagster.io/integrations/guides/multi-asset-integration#step-1-input) * [Step 2: Implementation](https://docs.dagster.io/integrations/guides/multi-asset-integration#step-2-implementation) * [Step 3: Moving the replication logic into a resource](https://docs.dagster.io/integrations/guides/multi-asset-integration#step-3-moving-the-replication-logic-into-a-resource) * [Step 4: Using translators](https://docs.dagster.io/integrations/guides/multi-asset-integration#step-4-using-translators) * [Conclusion](https://docs.dagster.io/integrations/guides/multi-asset-integration#conclusion) --- # Using Dagster with Airbyte Cloud | Dagster Docs [Skip to main content](https://docs.dagster.io/integrations/libraries/airbyte/airbyte-cloud#__docusaurus_skipToContent_fallback) On this page info This feature is considered in a beta stage. It is still being tested and may change. For more information, see the [API lifecycle stages documentation](https://docs.dagster.io/api/api-lifecycle/api-lifecycle-stages) . note If you are just getting started with the Airbyte Cloud integration, we recommend using the new [Airbyte Cloud component](https://docs.dagster.io/guides/build/components/integrations/airbyte-cloud-component-tutorial) . This guide provides instructions for using Dagster with Airbyte Cloud using the `dagster-airbyte` library. Your Airbyte Cloud connection tables can be represented as assets in the Dagster asset graph, allowing you to track lineage and dependencies between Airbyte Cloud assets and data assets you are already modeling in Dagster. You can also use Dagster to orchestrate Airbyte Cloud connections, allowing you to trigger syncs for these on a cadence or based on upstream data changes. What you'll learn[​](https://docs.dagster.io/integrations/libraries/airbyte/airbyte-cloud#what-youll-learn "Direct link to What you'll learn") ----------------------------------------------------------------------------------------------------------------------------------------------- * How to represent Airbyte Cloud assets in the Dagster asset graph, including lineage to other Dagster assets. * How to customize asset definition metadata for these Airbyte Cloud assets. * How to materialize Airbyte Cloud connection tables from Dagster. * How to customize how Airbyte Cloud connection tables are materialized. Prerequisites * The `dagster` and `dagster-airbyte` libraries installed in your environment * Familiarity with asset definitions and the Dagster asset graph * Familiarity with Dagster resources * Familiarity with Airbyte Cloud concepts, like connections and connection tables * An Airbyte Cloud workspace * An Airbyte Cloud client ID and client secret. For more information, see [Configuring API Access](https://docs.airbyte.com/using-airbyte/configuring-api-access) in the Airbyte Cloud REST API documentation. Set up your environment[​](https://docs.dagster.io/integrations/libraries/airbyte/airbyte-cloud#set-up-your-environment "Direct link to Set up your environment") ------------------------------------------------------------------------------------------------------------------------------------------------------------------ To get started, you'll need to install the `dagster` and `dagster-airbyte` Python packages: * uv * pip uv add dagster-airbyte pip install dagster-airbyte Represent Airbyte Cloud assets in the asset graph[​](https://docs.dagster.io/integrations/libraries/airbyte/airbyte-cloud#represent-airbyte-cloud-assets-in-the-asset-graph "Direct link to Represent Airbyte Cloud assets in the asset graph") ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ To load Airbyte Cloud assets into the Dagster asset graph, you must first construct a [`AirbyteCloudWorkspace`](https://docs.dagster.io/api/libraries/dagster-airbyte#dagster_airbyte.AirbyteCloudWorkspace) resource, which allows Dagster to communicate with your Airbyte Cloud workspace. You'll need to supply your workspace ID, client ID and client secret. See [Configuring API Access](https://docs.airbyte.com/using-airbyte/configuring-api-access) in the Airbyte Cloud REST API documentation for more information on how to create your client ID and client secret. Dagster can automatically load all connection tables from your Airbyte Cloud workspace as asset specs. Call the [`load_airbyte_cloud_asset_specs`](https://docs.dagster.io/api/libraries/dagster-airbyte#dagster_airbyte.load_airbyte_cloud_asset_specs) function, which returns list of [`AssetSpec`](https://docs.dagster.io/api/dagster/assets#dagster.AssetSpec) s representing your Airbyte Cloud assets. You can then include these asset specs in your [`Definitions`](https://docs.dagster.io/api/dagster/definitions#dagster.Definitions) object: from dagster_airbyte import AirbyteCloudWorkspace, load_airbyte_cloud_asset_specsimport dagster as dgairbyte_workspace = AirbyteCloudWorkspace( workspace_id=dg.EnvVar("AIRBYTE_CLOUD_WORKSPACE_ID"), client_id=dg.EnvVar("AIRBYTE_CLOUD_CLIENT_ID"), client_secret=dg.EnvVar("AIRBYTE_CLOUD_CLIENT_SECRET"),)airbyte_cloud_specs = load_airbyte_cloud_asset_specs(airbyte_workspace)defs = dg.Definitions(assets=airbyte_cloud_specs) ### Sync and materialize Airbyte Cloud assets[​](https://docs.dagster.io/integrations/libraries/airbyte/airbyte-cloud#sync-and-materialize-airbyte-cloud-assets "Direct link to Sync and materialize Airbyte Cloud assets") You can use Dagster to sync Airbyte Cloud connections and materialize Airbyte Cloud connection tables. You can use the [`build_airbyte_assets_definitions`](https://docs.dagster.io/api/libraries/dagster-airbyte#dagster_airbyte.build_airbyte_assets_definitions) factory to create all assets definitions for your Airbyte Cloud workspace. from dagster_airbyte import AirbyteCloudWorkspace, build_airbyte_assets_definitionsimport dagster as dgairbyte_workspace = AirbyteCloudWorkspace( workspace_id=dg.EnvVar("AIRBYTE_CLOUD_WORKSPACE_ID"), client_id=dg.EnvVar("AIRBYTE_CLOUD_CLIENT_ID"), client_secret=dg.EnvVar("AIRBYTE_CLOUD_CLIENT_SECRET"),)all_airbyte_assets = build_airbyte_assets_definitions(workspace=airbyte_workspace)defs = dg.Definitions( assets=all_airbyte_assets, resources={"airbyte": airbyte_workspace},) ### Customize the materialization of Airbyte Cloud assets[​](https://docs.dagster.io/integrations/libraries/airbyte/airbyte-cloud#customize-the-materialization-of-airbyte-cloud-assets "Direct link to Customize the materialization of Airbyte Cloud assets") If you want to customize the sync of your connections, you can use the [`airbyte_assets`](https://docs.dagster.io/api/libraries/dagster-airbyte#dagster_airbyte.airbyte_assets) decorator to do so. This allows you to execute custom code before and after the call to the Airbyte Cloud sync. from dagster_airbyte import AirbyteCloudWorkspace, airbyte_assetsimport dagster as dgairbyte_workspace = AirbyteCloudWorkspace( workspace_id=dg.EnvVar("AIRBYTE_CLOUD_WORKSPACE_ID"), client_id=dg.EnvVar("AIRBYTE_CLOUD_CLIENT_ID"), client_secret=dg.EnvVar("AIRBYTE_CLOUD_CLIENT_SECRET"),)@airbyte_assets( connection_id="airbyte_connection_id", # Replace with your connection ID workspace=airbyte_workspace, name="airbyte_connection_name", # Replace with your connection name group_name="airbyte_connection_name",)def airbyte_connection_assets( context: dg.AssetExecutionContext, airbyte: AirbyteCloudWorkspace): # Do something before the materialization... yield from airbyte.sync_and_poll(context=context) # Do something after the materialization...defs = dg.Definitions( assets=[airbyte_connection_assets], resources={"airbyte": airbyte_workspace},) ### Customize asset definition metadata for Airbyte Cloud assets[​](https://docs.dagster.io/integrations/libraries/airbyte/airbyte-cloud#customize-asset-definition-metadata-for-airbyte-cloud-assets "Direct link to Customize asset definition metadata for Airbyte Cloud assets") By default, Dagster will generate asset specs for each Airbyte Cloud asset and populate default metadata. You can further customize asset properties by passing an instance of the custom [`DagsterAirbyteTranslator`](https://docs.dagster.io/api/libraries/dagster-airbyte#dagster_airbyte.DagsterAirbyteTranslator) to the [`load_airbyte_cloud_asset_specs`](https://docs.dagster.io/api/libraries/dagster-airbyte#dagster_airbyte.load_airbyte_cloud_asset_specs) function. from dagster_airbyte import ( AirbyteCloudWorkspace, AirbyteConnectionTableProps, DagsterAirbyteTranslator, load_airbyte_cloud_asset_specs,)import dagster as dgairbyte_workspace = AirbyteCloudWorkspace( workspace_id=dg.EnvVar("AIRBYTE_CLOUD_WORKSPACE_ID"), client_id=dg.EnvVar("AIRBYTE_CLOUD_CLIENT_ID"), client_secret=dg.EnvVar("AIRBYTE_CLOUD_CLIENT_SECRET"),)# A translator class lets us customize properties of the built# Airbyte Cloud assets, such as the owners or asset keyclass MyCustomAirbyteTranslator(DagsterAirbyteTranslator): def get_asset_spec(self, props: AirbyteConnectionTableProps) -> dg.AssetSpec: # We create the default asset spec using super() default_spec = super().get_asset_spec(props) # We customize the metadata and asset key prefix for all assets return default_spec.replace_attributes( key=default_spec.key.with_prefix("prefix"), ).merge_attributes(metadata={"custom": "metadata"})airbyte_cloud_specs = load_airbyte_cloud_asset_specs( airbyte_workspace, dagster_airbyte_translator=MyCustomAirbyteTranslator())defs = dg.Definitions(assets=airbyte_cloud_specs) Note that `super()` is called in each of the overridden methods to generate the default asset spec. It is best practice to generate the default asset spec before customizing it. You can pass an instance of the custom [`DagsterAirbyteTranslator`](https://docs.dagster.io/api/libraries/dagster-airbyte#dagster_airbyte.DagsterAirbyteTranslator) to the [`airbyte_assets`](https://docs.dagster.io/api/libraries/dagster-airbyte#dagster_airbyte.airbyte_assets) decorator or the [`build_airbyte_assets_definitions`](https://docs.dagster.io/api/libraries/dagster-airbyte#dagster_airbyte.build_airbyte_assets_definitions) factory. ### Load Airbyte Cloud assets from multiple workspaces[​](https://docs.dagster.io/integrations/libraries/airbyte/airbyte-cloud#load-airbyte-cloud-assets-from-multiple-workspaces "Direct link to Load Airbyte Cloud assets from multiple workspaces") Definitions from multiple Airbyte Cloud workspaces can be combined by instantiating multiple [`AirbyteCloudWorkspace`](https://docs.dagster.io/api/libraries/dagster-airbyte#dagster_airbyte.AirbyteCloudWorkspace) resources and merging their specs. This lets you view all your Airbyte Cloud assets in a single asset graph: from dagster_airbyte import AirbyteCloudWorkspace, load_airbyte_cloud_asset_specsimport dagster as dgsales_airbyte_workspace = AirbyteCloudWorkspace( workspace_id=dg.EnvVar("AIRBYTE_CLOUD_SALES_WORKSPACE_ID"), client_id=dg.EnvVar("AIRBYTE_CLOUD_SALES_CLIENT_ID"), client_secret=dg.EnvVar("AIRBYTE_CLOUD_SALES_CLIENT_SECRET"),)marketing_airbyte_workspace = AirbyteCloudWorkspace( workspace_id=dg.EnvVar("AIRBYTE_CLOUD_MARKETING_WORKSPACE_ID"), client_id=dg.EnvVar("AIRBYTE_CLOUD_MARKETING_CLIENT_ID"), client_secret=dg.EnvVar("AIRBYTE_CLOUD_MARKETING_CLIENT_SECRET"),)sales_airbyte_cloud_specs = load_airbyte_cloud_asset_specs( workspace=sales_airbyte_workspace)marketing_airbyte_cloud_specs = load_airbyte_cloud_asset_specs( workspace=marketing_airbyte_workspace)# Merge the specs into a single set of definitionsdefs = dg.Definitions( assets=[*sales_airbyte_cloud_specs, *marketing_airbyte_cloud_specs],) ### Define upstream dependencies[​](https://docs.dagster.io/integrations/libraries/airbyte/airbyte-cloud#define-upstream-dependencies "Direct link to Define upstream dependencies") By default, Dagster does not set upstream dependencies when generating asset specs for your Airbyte Cloud assets. You can set upstream dependencies on your Airbyte Cloud assets by passing an instance of the custom [`DagsterAirbyteTranslator`](https://docs.dagster.io/api/libraries/dagster-airbyte#dagster_airbyte.DagsterAirbyteTranslator) to the [`load_airbyte_cloud_asset_specs`](https://docs.dagster.io/api/libraries/dagster-airbyte#dagster_airbyte.load_airbyte_cloud_asset_specs) function. class MyCustomAirbyteTranslator(DagsterAirbyteTranslator): def get_asset_spec(self, props: AirbyteConnectionTableProps) -> dg.AssetSpec: # We create the default asset spec using super() default_spec = super().get_asset_spec(props) # We set an upstream dependency for our assets return default_spec.replace_attributes(deps=["my_upstream_asset_key"])airbyte_cloud_specs = load_airbyte_cloud_asset_specs( airbyte_workspace, dagster_airbyte_translator=MyCustomAirbyteTranslator()) Note that `super()` is called in each of the overridden methods to generate the default asset spec. It is best practice to generate the default asset spec before customizing it. You can pass an instance of the custom [`DagsterAirbyteTranslator`](https://docs.dagster.io/api/libraries/dagster-airbyte#dagster_airbyte.DagsterAirbyteTranslator) to the [`airbyte_assets`](https://docs.dagster.io/api/libraries/dagster-airbyte#dagster_airbyte.airbyte_assets) decorator or the [`build_airbyte_assets_definitions`](https://docs.dagster.io/api/libraries/dagster-airbyte#dagster_airbyte.build_airbyte_assets_definitions) factory. ### Define downstream dependencies[​](https://docs.dagster.io/integrations/libraries/airbyte/airbyte-cloud#define-downstream-dependencies "Direct link to Define downstream dependencies") Dagster allows you to define assets that are downstream of specific Airbyte Cloud tables using their asset keys. The asset key for an Airbyte Cloud table can be retrieved using the asset definitions created using the [`airbyte_assets`](https://docs.dagster.io/api/libraries/dagster-airbyte#dagster_airbyte.airbyte_assets) decorator. The below example defines `my_downstream_asset` as a downstream dependency of `my_airbyte_cloud_table`: @airbyte_assets( connection_id="airbyte_connection_id", # Replace with your connection ID workspace=airbyte_workspace,)def airbyte_connection_assets( context: dg.AssetExecutionContext, airbyte: AirbyteCloudWorkspace): ...my_airbyte_cloud_table_asset_key = next( iter( [ spec.key for spec in airbyte_connection_assets.specs if spec.metadata.get("dagster/table_name") == "my_database.my_schema.my_airbyte_cloud_table" ] ))@dg.asset(deps=[my_airbyte_cloud_table_asset_key])def my_downstream_asset(): ... In the downstream asset, you may want direct access to the contents of the Airbyte Cloud table. To do so, you can customize the code within your `@asset`\-decorated function to load upstream data. * [What you'll learn](https://docs.dagster.io/integrations/libraries/airbyte/airbyte-cloud#what-youll-learn) * [Set up your environment](https://docs.dagster.io/integrations/libraries/airbyte/airbyte-cloud#set-up-your-environment) * [Represent Airbyte Cloud assets in the asset graph](https://docs.dagster.io/integrations/libraries/airbyte/airbyte-cloud#represent-airbyte-cloud-assets-in-the-asset-graph) * [Sync and materialize Airbyte Cloud assets](https://docs.dagster.io/integrations/libraries/airbyte/airbyte-cloud#sync-and-materialize-airbyte-cloud-assets) * [Customize the materialization of Airbyte Cloud assets](https://docs.dagster.io/integrations/libraries/airbyte/airbyte-cloud#customize-the-materialization-of-airbyte-cloud-assets) * [Customize asset definition metadata for Airbyte Cloud assets](https://docs.dagster.io/integrations/libraries/airbyte/airbyte-cloud#customize-asset-definition-metadata-for-airbyte-cloud-assets) * [Load Airbyte Cloud assets from multiple workspaces](https://docs.dagster.io/integrations/libraries/airbyte/airbyte-cloud#load-airbyte-cloud-assets-from-multiple-workspaces) * [Define upstream dependencies](https://docs.dagster.io/integrations/libraries/airbyte/airbyte-cloud#define-upstream-dependencies) * [Define downstream dependencies](https://docs.dagster.io/integrations/libraries/airbyte/airbyte-cloud#define-downstream-dependencies) --- # Dagster & Airbyte | Dagster Docs [Skip to main content](https://docs.dagster.io/integrations/libraries/airbyte/airbyte-oss#__docusaurus_skipToContent_fallback) On this page Using this integration, you can trigger Airbyte syncs and orchestrate your Airbyte connections from within Dagster, making it easy to chain an Airbyte sync with upstream or downstream steps in your workflow. Installation[​](https://docs.dagster.io/integrations/libraries/airbyte/airbyte-oss#installation "Direct link to Installation") ------------------------------------------------------------------------------------------------------------------------------- * uv * pip uv add dagster-airbyte pip install dagster-airbyte Example[​](https://docs.dagster.io/integrations/libraries/airbyte/airbyte-oss#example "Direct link to Example") ---------------------------------------------------------------------------------------------------------------- from dagster_airbyte import AirbyteResource, load_assets_from_airbyte_instanceimport dagster as dg# Load all assets from your Airbyte instanceairbyte_assets = load_assets_from_airbyte_instance( # Connect to your OSS Airbyte instance AirbyteResource( host="localhost", port="8000", # If using basic auth, include username and password: username="airbyte", password=dg.EnvVar("AIRBYTE_PASSWORD"), ))defs = dg.Definitions( assets=[airbyte_assets],) About Airbyte[​](https://docs.dagster.io/integrations/libraries/airbyte/airbyte-oss#about-airbyte "Direct link to About Airbyte") ---------------------------------------------------------------------------------------------------------------------------------- **Airbyte** is an open source data integration engine that helps you consolidate your SaaS application and database data into your data warehouses, lakes and databases. * [Installation](https://docs.dagster.io/integrations/libraries/airbyte/airbyte-oss#installation) * [Example](https://docs.dagster.io/integrations/libraries/airbyte/airbyte-oss#example) * [About Airbyte](https://docs.dagster.io/integrations/libraries/airbyte/airbyte-oss#about-airbyte) --- # Dagster & Airbyte | Dagster Docs [Skip to main content](https://docs.dagster.io/integrations/libraries/airbyte#__docusaurus_skipToContent_fallback) --- # Dagster & Anthropic | Dagster Docs [Skip to main content](https://docs.dagster.io/integrations/libraries/anthropic#__docusaurus_skipToContent_fallback) On this page The Anthropic integration allows you to easily interact with the Anthropic REST API using the Anthropic Python API to build AI steps into your Dagster pipelines. You can also log Anthropic API usage metadata in Dagster Insights, giving you detailed observability on API call credit consumption. Installation[​](https://docs.dagster.io/integrations/libraries/anthropic#installation "Direct link to Installation") --------------------------------------------------------------------------------------------------------------------- * uv * pip uv add dagster-anthropic pip install dagster-anthropic Example[​](https://docs.dagster.io/integrations/libraries/anthropic#example "Direct link to Example") ------------------------------------------------------------------------------------------------------ from dagster_anthropic import AnthropicResourceimport dagster as dg@dg.asset(compute_kind="anthropic")def anthropic_asset(context: dg.AssetExecutionContext, anthropic: AnthropicResource): with anthropic.get_client(context) as client: response = client.messages.create( model="claude-3-5-sonnet-20241022", max_tokens=1024, messages=[{"role": "user", "content": "Say this is a test"}], )defs = dg.Definitions( assets=[anthropic_asset], resources={ "anthropic": AnthropicResource(api_key=dg.EnvVar("ANTHROPIC_API_KEY")), },) About Anthropic[​](https://docs.dagster.io/integrations/libraries/anthropic#about-anthropic "Direct link to About Anthropic") ------------------------------------------------------------------------------------------------------------------------------ Anthropic is an AI research company focused on developing safe and ethical AI systems. Their flagship product, Claude, is a language model known for its strong capabilities in analysis, writing, and coding tasks while maintaining high standards of truthfulness and safety. * [Installation](https://docs.dagster.io/integrations/libraries/anthropic#installation) * [Example](https://docs.dagster.io/integrations/libraries/anthropic#example) * [About Anthropic](https://docs.dagster.io/integrations/libraries/anthropic#about-anthropic) --- # Dagster & Airlift | Dagster Docs [Skip to main content](https://docs.dagster.io/integrations/libraries/airlift#__docusaurus_skipToContent_fallback) On this page info This feature is considered in a beta stage. It is still being tested and may change. For more information, see the [API lifecycle stages documentation](https://docs.dagster.io/api/api-lifecycle/api-lifecycle-stages) . note If you are just getting started with Airlift, we recommend using the new [Airlift component](https://docs.dagster.io/migration/airflow-to-dagster/airflow-component-tutorial) . Airlift is a toolkit for integrating Dagster and Airflow. Using [`dagster-airflift`](https://docs.dagster.io/api/libraries/dagster-airlift) , you can: * Observe Airflow instances from within Dagster * Accelerate the migration of Airflow DAGs to Dagster assets with opinionated tooling Compatibility[​](https://docs.dagster.io/integrations/libraries/airlift#compatibility "Direct link to Compatibility") ---------------------------------------------------------------------------------------------------------------------- ### REST API Availability[​](https://docs.dagster.io/integrations/libraries/airlift#rest-api-availability "Direct link to REST API Availability") Airlift depends on the availability of Airflow’s REST API. Airflow’s REST API was made stable in its 2.0 release (Dec 2020) and was introduced experimentally in 1.10 in August 2018. Currently Airflow requires the availability of the REST API. * **OSS:** Stable as of 2.00 * **MWAA** * Note: only available in Airflow 2.4.3 or later on MWAA. * **Cloud Composer:** No limitations as far as we know. * **Astronomer:** No limitations as far as we know. Migrating from Airflow to Dagster[​](https://docs.dagster.io/integrations/libraries/airlift#migrating-from-airflow-to-dagster "Direct link to Migrating from Airflow to Dagster") ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- You can use Airlift to migrate an Airflow DAG to Dagster assets. Airlift enables a migration process that * Can be done task-by-task in any order with minimal coordination * Has task-by-task rollback to reduce risk * Retains Airflow DAG structure and execution history during the migration To get started, see "[Migrate from Airflow to Dagster at the task level](https://docs.dagster.io/migration/airflow-to-dagster/airlift-v1/task-level-migration) ". note If you need to migrate at the DAG level, see "[Migrate from Airflow to Dagster at the DAG level](https://docs.dagster.io/migration/airflow-to-dagster/airlift-v1/dag-level-migration) ". Federating execution between Airflow instances with Dagster[​](https://docs.dagster.io/integrations/libraries/airlift#federating-execution-between-airflow-instances-with-dagster "Direct link to Federating execution between Airflow instances with Dagster") ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- You can use Airlift to observe DAGs from multiple Airflow instances, and federate execution between them using Dagster as a centralized control plane. To get started, see "[Federate execution between Airflow instances with Dagster](https://docs.dagster.io/migration/airflow-to-dagster/airlift-v1/federation) ". Airflow operator migration[​](https://docs.dagster.io/integrations/libraries/airlift#airflow-operator-migration "Direct link to Airflow operator migration") ------------------------------------------------------------------------------------------------------------------------------------------------------------- You can easily migrate common Airflow operators to Dagster. For more information, see "[Airflow operator migration](https://docs.dagster.io/migration/airflow-to-dagster/airflow-operator-migration) ". * [Compatibility](https://docs.dagster.io/integrations/libraries/airlift#compatibility) * [REST API Availability](https://docs.dagster.io/integrations/libraries/airlift#rest-api-availability) * [Migrating from Airflow to Dagster](https://docs.dagster.io/integrations/libraries/airlift#migrating-from-airflow-to-dagster) * [Federating execution between Airflow instances with Dagster](https://docs.dagster.io/integrations/libraries/airlift#federating-execution-between-airflow-instances-with-dagster) * [Airflow operator migration](https://docs.dagster.io/integrations/libraries/airlift#airflow-operator-migration) --- # AWS | Dagster Docs [Skip to main content](https://docs.dagster.io/integrations/libraries/aws#__docusaurus_skipToContent_fallback) --- # Dagster & AWS ECR | Dagster Docs [Skip to main content](https://docs.dagster.io/integrations/libraries/aws/ecr#__docusaurus_skipToContent_fallback) On this page info This feature is considered in a beta stage. It is still being tested and may change. For more information, see the [API lifecycle stages documentation](https://docs.dagster.io/api/api-lifecycle/api-lifecycle-stages) . This integration allows you to connect to AWS Elastic Container Registry (ECR). It provides resources to interact with AWS ECR, enabling you to manage your container images. Using this integration, you can seamlessly integrate AWS ECR into your Dagster pipelines, making it easier to manage and deploy containerized applications. Installation[​](https://docs.dagster.io/integrations/libraries/aws/ecr#installation "Direct link to Installation") ------------------------------------------------------------------------------------------------------------------- * uv * pip uv add dagster-aws pip install dagster-aws Examples[​](https://docs.dagster.io/integrations/libraries/aws/ecr#examples "Direct link to Examples") ------------------------------------------------------------------------------------------------------- from dagster_aws.ecr import ECRPublicResourceimport dagster as dg@dg.assetdef get_ecr_login_password(ecr_public: ECRPublicResource): return ecr_public.get_client().get_login_password()defs = dg.Definitions( assets=[get_ecr_login_password], resources={"ecr_public": ECRPublicResource()},) About AWS ECR[​](https://docs.dagster.io/integrations/libraries/aws/ecr#about-aws-ecr "Direct link to About AWS ECR") ---------------------------------------------------------------------------------------------------------------------- AWS Elastic Container Registry (ECR) is a fully managed Docker container registry that makes it easy for developers to store, manage, and deploy Docker container images. AWS ECR is integrated with Amazon Elastic Kubernetes Service (EKS), simplifying your development to production workflow. With ECR, you can securely store and manage your container images and easily integrate with your existing CI/CD pipelines. AWS ECR provides high availability and scalability, ensuring that your container images are always available when you need them. * [Installation](https://docs.dagster.io/integrations/libraries/aws/ecr#installation) * [Examples](https://docs.dagster.io/integrations/libraries/aws/ecr#examples) * [About AWS ECR](https://docs.dagster.io/integrations/libraries/aws/ecr#about-aws-ecr) --- # Dagster & AWS Athena | Dagster Docs [Skip to main content](https://docs.dagster.io/integrations/libraries/aws/athena#__docusaurus_skipToContent_fallback) On this page This integration allows you to connect to AWS Athena, a serverless interactive query service that makes it easy to analyze data in Amazon S3 using standard SQL. Using this integration, you can issue queries to Athena, fetch results, and handle query execution states within your Dagster pipelines. Installation[​](https://docs.dagster.io/integrations/libraries/aws/athena#installation "Direct link to Installation") ---------------------------------------------------------------------------------------------------------------------- * uv * pip uv add dagster-aws pip install dagster-aws Examples[​](https://docs.dagster.io/integrations/libraries/aws/athena#examples "Direct link to Examples") ---------------------------------------------------------------------------------------------------------- from dagster_aws.athena import AthenaClientResourceimport dagster as dg@dg.assetdef example_athena_asset(athena: AthenaClientResource): return athena.get_client().execute_query("SELECT 1", fetch_results=True)defs = dg.Definitions( assets=[example_athena_asset], resources={"athena": AthenaClientResource()}) About AWS Athena[​](https://docs.dagster.io/integrations/libraries/aws/athena#about-aws-athena "Direct link to About AWS Athena") ---------------------------------------------------------------------------------------------------------------------------------- AWS Athena is a serverless, interactive query service that allows you to analyze data directly in Amazon S3 using standard SQL. Athena is easy to use; point to your data in Amazon S3, define the schema, and start querying using standard SQL. Most results are delivered within seconds. With Athena, there are no infrastructure setups, and you pay only for the queries you run. It scales automatically—executing queries in parallel—so results are fast, even with large datasets and complex queries. * [Installation](https://docs.dagster.io/integrations/libraries/aws/athena#installation) * [Examples](https://docs.dagster.io/integrations/libraries/aws/athena#examples) * [About AWS Athena](https://docs.dagster.io/integrations/libraries/aws/athena#about-aws-athena) --- # Dagster & AWS CloudWatch | Dagster Docs [Skip to main content](https://docs.dagster.io/integrations/libraries/aws/cloudwatch#__docusaurus_skipToContent_fallback) On this page warning This feature is considered deprecated. It is still available, but will be removed in the future, and we recommend avoiding new usage. For more information, see the [API lifecycle stages documentation](https://docs.dagster.io/api/api-lifecycle/api-lifecycle-stages) . This integration allows you to send Dagster logs to AWS CloudWatch, enabling centralized logging and monitoring of your Dagster jobs. By using AWS CloudWatch, you can take advantage of its powerful log management features, such as real-time log monitoring, log retention policies, and alerting capabilities. Using this integration, you can configure your Dagster jobs to log directly to AWS CloudWatch, making it easier to track and debug your workflows. This is particularly useful for production environments where centralized logging is essential for maintaining observability and operational efficiency. Installation[​](https://docs.dagster.io/integrations/libraries/aws/cloudwatch#installation "Direct link to Installation") -------------------------------------------------------------------------------------------------------------------------- * uv * pip uv add dagster-aws pip install dagster-aws Examples[​](https://docs.dagster.io/integrations/libraries/aws/cloudwatch#examples "Direct link to Examples") -------------------------------------------------------------------------------------------------------------- from dagster_aws.cloudwatch import cloudwatch_loggerimport dagster as dg@dg.assetdef my_asset(context: dg.AssetExecutionContext): context.log.info("Hello, CloudWatch!") context.log.error("This is an error") context.log.debug("This is a debug message")defs = dg.Definitions( assets=[my_asset], loggers={ "cloudwatch_logger": cloudwatch_logger, },) About AWS CloudWatch[​](https://docs.dagster.io/integrations/libraries/aws/cloudwatch#about-aws-cloudwatch "Direct link to About AWS CloudWatch") -------------------------------------------------------------------------------------------------------------------------------------------------- AWS CloudWatch is a monitoring and observability service provided by Amazon Web Services (AWS). It allows you to collect, access, and analyze performance and operational data from a variety of AWS resources, applications, and services. With AWS CloudWatch, you can set up alarms, visualize logs and metrics, and gain insights into your infrastructure and applications to ensure they're running smoothly. AWS CloudWatch provides features such as: * Real-time monitoring: Track the performance of your applications and infrastructure in real-time. * Log management: Collect, store, and analyze log data from various sources. * Alarms and notifications: Set up alarms to automatically notify you of potential issues. * Dashboards: Create custom dashboards to visualize metrics and logs. * Integration with other AWS services: Seamlessly integrate with other AWS services for a comprehensive monitoring solution. * [Installation](https://docs.dagster.io/integrations/libraries/aws/cloudwatch#installation) * [Examples](https://docs.dagster.io/integrations/libraries/aws/cloudwatch#examples) * [About AWS CloudWatch](https://docs.dagster.io/integrations/libraries/aws/cloudwatch#about-aws-cloudwatch) --- # Dagster & AWS EMR | Dagster Docs [Skip to main content](https://docs.dagster.io/integrations/libraries/aws/emr#__docusaurus_skipToContent_fallback) On this page The AWS integration provides ways orchestrating data pipelines that leverage AWS services, including AWS EMR (Elastic MapReduce). This integration allows you to run and scale big data workloads using open source tools such as Apache Spark, Hive, Presto, and more. Using this integration, you can: * Seamlessly integrate AWS EMR into your Dagster pipelines. * Utilize EMR for petabyte-scale data processing. * Easily manage and monitor EMR clusters and jobs from within Dagster. * Leverage Dagster's orchestration capabilities to handle complex data workflows involving EMR. Installation[​](https://docs.dagster.io/integrations/libraries/aws/emr#installation "Direct link to Installation") ------------------------------------------------------------------------------------------------------------------- * uv * pip uv add dagster-aws pip install dagster-aws Examples[​](https://docs.dagster.io/integrations/libraries/aws/emr#examples "Direct link to Examples") ------------------------------------------------------------------------------------------------------- from pathlib import Pathfrom typing import Anyfrom dagster_aws.emr import emr_pyspark_step_launcherfrom dagster_aws.s3 import S3Resourcefrom dagster_pyspark import PySparkResourcefrom pyspark.sql import DataFrame, Rowfrom pyspark.sql.types import IntegerType, StringType, StructField, StructTypeimport dagster as dgemr_pyspark = PySparkResource(spark_config={"spark.executor.memory": "2g"})@dg.assetdef people( pyspark: PySparkResource, pyspark_step_launcher: dg.ResourceParam[Any]) -> DataFrame: schema = StructType( [StructField("name", StringType()), StructField("age", IntegerType())] ) rows = [ Row(name="Thom", age=51), Row(name="Jonny", age=48), Row(name="Nigel", age=49), ] return pyspark.spark_session.createDataFrame(rows, schema)@dg.assetdef people_over_50( pyspark_step_launcher: dg.ResourceParam[Any], people: DataFrame) -> DataFrame: return people.filter(people["age"] > 50)defs = dg.Definitions( assets=[people, people_over_50], resources={ "pyspark_step_launcher": emr_pyspark_step_launcher.configured( { "cluster_id": {"env": "EMR_CLUSTER_ID"}, "local_pipeline_package_path": str(Path(__file__).parent), "deploy_local_pipeline_package": True, "region_name": "us-west-1", "staging_bucket": "my_staging_bucket", "wait_for_logs": True, } ), "pyspark": emr_pyspark, "s3": S3Resource(), },) About AWS EMR[​](https://docs.dagster.io/integrations/libraries/aws/emr#about-aws-emr "Direct link to About AWS EMR") ---------------------------------------------------------------------------------------------------------------------- **AWS EMR** (Elastic MapReduce) is a cloud big data platform for processing vast amounts of data using open source tools such as Apache Spark, Apache Hive, Apache HBase, Apache Flink, Apache Hudi, and Presto. It simplifies running big data frameworks, allowing you to process and analyze large datasets quickly and cost-effectively. AWS EMR provides the scalability, flexibility, and reliability needed to handle complex data processing tasks, making it an ideal choice for data engineers and scientists. * [Installation](https://docs.dagster.io/integrations/libraries/aws/emr#installation) * [Examples](https://docs.dagster.io/integrations/libraries/aws/emr#examples) * [About AWS EMR](https://docs.dagster.io/integrations/libraries/aws/emr#about-aws-emr) --- # Dagster & AWS Glue | Dagster Docs [Skip to main content](https://docs.dagster.io/integrations/libraries/aws/glue#__docusaurus_skipToContent_fallback) On this page The AWS integration library provides the PipesGlueClient resource, enabling you to launch AWS Glue jobs directly from Dagster assets and ops. This integration allows you to pass parameters to Glue code while Dagster receives real-time events, such as logs, asset checks, and asset materializations, from the initiated jobs. With minimal code changes required on the job side, this integration is both efficient and easy to implement. Installation[​](https://docs.dagster.io/integrations/libraries/aws/glue#installation "Direct link to Installation") -------------------------------------------------------------------------------------------------------------------- * uv * pip uv add dagster-aws pip install dagster-aws Examples[​](https://docs.dagster.io/integrations/libraries/aws/glue#examples "Direct link to Examples") -------------------------------------------------------------------------------------------------------- import boto3from dagster_aws.pipes import ( PipesGlueClient, PipesS3ContextInjector, PipesS3MessageReader,)import dagster as dg@dg.assetdef glue_pipes_asset( context: dg.AssetExecutionContext, pipes_glue_client: PipesGlueClient): return pipes_glue_client.run( context=context, job_name="Example Job", arguments={"some_parameter_value": "1"}, ).get_materialize_result()defs = dg.Definitions( assets=[glue_pipes_asset], resources={ "pipes_glue_client": PipesGlueClient( client=boto3.client("glue", region_name="us-east-1"), context_injector=PipesS3ContextInjector( client=boto3.client("s3"), bucket="my-bucket", ), message_reader=PipesS3MessageReader( client=boto3.client("s3"), bucket="my-bucket" ), ) },) About AWS Glue[​](https://docs.dagster.io/integrations/libraries/aws/glue#about-aws-glue "Direct link to About AWS Glue") -------------------------------------------------------------------------------------------------------------------------- **AWS Glue** is a fully managed cloud service designed to simplify and automate the process of discovering, preparing, and integrating data for analytics, machine learning, and application development. It supports a wide range of data sources and formats, offering seamless integration with other AWS services. AWS Glue provides the tools to create, run, and manage ETL (Extract, Transform, Load) jobs, making it easier to handle complex data workflows. Its serverless architecture allows for scalability and flexibility, making it a preferred choice for data engineers and analysts who need to process and prepare data efficiently. * [Installation](https://docs.dagster.io/integrations/libraries/aws/glue#installation) * [Examples](https://docs.dagster.io/integrations/libraries/aws/glue#examples) * [About AWS Glue](https://docs.dagster.io/integrations/libraries/aws/glue#about-aws-glue) --- # Dagster & AWS Lambda | Dagster Docs [Skip to main content](https://docs.dagster.io/integrations/libraries/aws/lambda#__docusaurus_skipToContent_fallback) On this page Using this integration, you can leverage AWS Lambda to execute external code as part of your Dagster pipelines. This is particularly useful for running serverless functions that can scale automatically and handle various workloads without the need for managing infrastructure. The PipesLambdaClient class allows you to invoke AWS Lambda functions and stream logs and structured metadata back to Dagster's UI and tools. Installation[​](https://docs.dagster.io/integrations/libraries/aws/lambda#installation "Direct link to Installation") ---------------------------------------------------------------------------------------------------------------------- * uv * pip uv add dagster-aws pip install dagster-aws Examples[​](https://docs.dagster.io/integrations/libraries/aws/lambda#examples "Direct link to Examples") ---------------------------------------------------------------------------------------------------------- import boto3from dagster_aws.pipes import PipesLambdaClientimport dagster as dglambda_client = boto3.client("lambda", region_name="us-west-1")lambda_pipes_client = PipesLambdaClient(client=lambda_client)@dg.assetdef lambda_pipes_asset( context: dg.AssetExecutionContext, lambda_pipes_client: PipesLambdaClient): return lambda_pipes_client.run( context=context, function_name="your_lambda_function_name", event={"key": "value"}, ).get_materialize_result()defs = dg.Definitions( assets=[lambda_pipes_asset], resources={"lambda_pipes_client": lambda_pipes_client},) About AWS Lambda[​](https://docs.dagster.io/integrations/libraries/aws/lambda#about-aws-lambda "Direct link to About AWS Lambda") ---------------------------------------------------------------------------------------------------------------------------------- **AWS Lambda** is a serverless compute service provided by Amazon Web Services (AWS). It allows you to run code without provisioning or managing servers. AWS Lambda automatically scales your application by running code in response to each trigger, such as changes to data in an Amazon S3 bucket or an update to a DynamoDB table. You can use AWS Lambda to extend other AWS services with custom logic, or create your own backend services that operate at AWS scale, performance, and security. * [Installation](https://docs.dagster.io/integrations/libraries/aws/lambda#installation) * [Examples](https://docs.dagster.io/integrations/libraries/aws/lambda#examples) * [About AWS Lambda](https://docs.dagster.io/integrations/libraries/aws/lambda#about-aws-lambda) --- # Dagster & AWS Redshift | Dagster Docs [Skip to main content](https://docs.dagster.io/integrations/libraries/aws/redshift#__docusaurus_skipToContent_fallback) On this page Using this integration, you can connect to an AWS Redshift cluster and issue queries against it directly from your Dagster assets. This allows you to seamlessly integrate Redshift into your data pipelines, leveraging the power of Redshift's data warehousing capabilities within your Dagster workflows. Installation[​](https://docs.dagster.io/integrations/libraries/aws/redshift#installation "Direct link to Installation") ------------------------------------------------------------------------------------------------------------------------ * uv * pip uv add dagster-aws pip install dagster-aws Examples[​](https://docs.dagster.io/integrations/libraries/aws/redshift#examples "Direct link to Examples") ------------------------------------------------------------------------------------------------------------ from dagster_aws.redshift import RedshiftClientResourceimport dagster as dg@dg.assetdef example_redshift_asset(context, redshift: RedshiftClientResource): result = redshift.get_client().execute_query("SELECT 1", fetch_results=True) context.log.info(f"Query result: {result}")redshift_configured = RedshiftClientResource( host="my-redshift-cluster.us-east-1.redshift.amazonaws.com", port=5439, user="dagster", password=dg.EnvVar("DAGSTER_REDSHIFT_PASSWORD"), database="dev",)defs = dg.Definitions( assets=[example_redshift_asset], resources={"redshift": redshift_configured},) About AWS Redshift[​](https://docs.dagster.io/integrations/libraries/aws/redshift#about-aws-redshift "Direct link to About AWS Redshift") ------------------------------------------------------------------------------------------------------------------------------------------ **AWS Redshift** is a fully managed, petabyte-scale data warehouse service in the cloud. You can start with just a few hundred gigabytes of data and scale to a petabyte or more. This enables you to use your data to acquire new insights for your business and customers. Redshift offers fast query performance using SQL-based tools and business intelligence applications, making it a powerful tool for data warehousing and analytics. * [Installation](https://docs.dagster.io/integrations/libraries/aws/redshift#installation) * [Examples](https://docs.dagster.io/integrations/libraries/aws/redshift#examples) * [About AWS Redshift](https://docs.dagster.io/integrations/libraries/aws/redshift#about-aws-redshift) --- # Dagster & AWS Secrets Manager | Dagster Docs [Skip to main content](https://docs.dagster.io/integrations/libraries/aws/secretsmanager#__docusaurus_skipToContent_fallback) On this page info This feature is considered in a beta stage. It is still being tested and may change. For more information, see the [API lifecycle stages documentation](https://docs.dagster.io/api/api-lifecycle/api-lifecycle-stages) . This integration allows you to manage, retrieve, and rotate credentials, API keys, and other secrets using AWS Secrets Manager. Installation[​](https://docs.dagster.io/integrations/libraries/aws/secretsmanager#installation "Direct link to Installation") ------------------------------------------------------------------------------------------------------------------------------ * uv * pip uv add dagster-aws pip install dagster-aws Examples[​](https://docs.dagster.io/integrations/libraries/aws/secretsmanager#examples "Direct link to Examples") ------------------------------------------------------------------------------------------------------------------ from dagster_aws.secretsmanager import ( SecretsManagerResource, SecretsManagerSecretsResource,)import dagster as dg@dg.assetdef my_asset(secretsmanager: SecretsManagerResource): secret_value = secretsmanager.get_client().get_secret_value( SecretId="arn:aws:secretsmanager:region:aws_account_id:secret:appauthexample-AbCdEf" ) return secret_value@dg.assetdef my_other_asset(secrets: SecretsManagerSecretsResource): secret_value = secrets.fetch_secrets().get("my-secret-name") return secret_valuedefs = dg.Definitions( assets=[my_asset, my_other_asset], resources={ "secretsmanager": SecretsManagerResource(region_name="us-west-1"), "secrets": SecretsManagerSecretsResource( region_name="us-west-1", secrets_tag="dagster", ), },) About AWS Secrets Manager[​](https://docs.dagster.io/integrations/libraries/aws/secretsmanager#about-aws-secrets-manager "Direct link to About AWS Secrets Manager") --------------------------------------------------------------------------------------------------------------------------------------------------------------------- **AWS Secrets Manager** helps you protect access to your applications, services, and IT resources without the upfront cost and complexity of managing your own hardware security module infrastructure. With Secrets Manager, you can rotate, manage, and retrieve database credentials, API keys, and other secrets throughout their lifecycle. Users and applications retrieve secrets with a call to Secrets Manager APIs, eliminating the need to hardcode sensitive information in plain text. * [Installation](https://docs.dagster.io/integrations/libraries/aws/secretsmanager#installation) * [Examples](https://docs.dagster.io/integrations/libraries/aws/secretsmanager#examples) * [About AWS Secrets Manager](https://docs.dagster.io/integrations/libraries/aws/secretsmanager#about-aws-secrets-manager) --- # Dagster & AWS S3 | Dagster Docs [Skip to main content](https://docs.dagster.io/integrations/libraries/aws/s3#__docusaurus_skipToContent_fallback) On this page The AWS S3 integration allows data engineers to easily read, and write objects to the durable AWS S3 storage enabling engineers to a resilient storage layer when constructing their pipelines. Installation[​](https://docs.dagster.io/integrations/libraries/aws/s3#installation "Direct link to Installation") ------------------------------------------------------------------------------------------------------------------ * uv * pip uv add dagster-aws pip install dagster-aws Examples[​](https://docs.dagster.io/integrations/libraries/aws/s3#examples "Direct link to Examples") ------------------------------------------------------------------------------------------------------ Here is an example of how to use the `S3Resource` in a Dagster job to interact with AWS S3: import pandas as pdfrom dagster_aws.s3 import S3Resourceimport dagster as dg@dg.assetdef my_s3_asset(s3: S3Resource): df = pd.DataFrame({"column1": [1, 2, 3], "column2": ["A", "B", "C"]}) csv_data = df.to_csv(index=False) s3_client = s3.get_client() s3_client.put_object( Bucket="my-cool-bucket", Key="path/to/my_dataframe.csv", Body=csv_data, )defs = dg.Definitions( assets=[my_s3_asset], resources={"s3": S3Resource(region_name="us-west-2")},) About AWS S3[​](https://docs.dagster.io/integrations/libraries/aws/s3#about-aws-s3 "Direct link to About AWS S3") ------------------------------------------------------------------------------------------------------------------ **AWS S3** is an object storage service that offers industry-leading scalability, data availability, security, and performance. This means customers of all sizes and industries can use it to store and protect any amount of data for a range of use cases, such as data lakes, websites, mobile applications, backup and restore, archive, enterprise applications, IoT devices, and big data analytics. Amazon S3 provides easy-to-use management features so you can organize your data and configure finely tuned access controls to meet your specific business, organizational, and compliance requirements. * [Installation](https://docs.dagster.io/integrations/libraries/aws/s3#installation) * [Examples](https://docs.dagster.io/integrations/libraries/aws/s3#examples) * [About AWS S3](https://docs.dagster.io/integrations/libraries/aws/s3#about-aws-s3) --- # Dagster & Azure Data Lake Storage Gen 2 | Dagster Docs [Skip to main content](https://docs.dagster.io/integrations/libraries/azure-adls2#__docusaurus_skipToContent_fallback) On this page Dagster helps you use Azure Storage Accounts as part of your data pipeline. Azure Data Lake Storage Gen 2 (ADLS2) is our primary focus but we also provide utilities for Azure Blob Storage. Installation[​](https://docs.dagster.io/integrations/libraries/azure-adls2#installation "Direct link to Installation") ----------------------------------------------------------------------------------------------------------------------- * uv * pip uv add dagster-azure pip install dagster-azure Example[​](https://docs.dagster.io/integrations/libraries/azure-adls2#example "Direct link to Example") -------------------------------------------------------------------------------------------------------- import pandas as pdfrom dagster_azure.adls2 import ADLS2Resource, ADLS2SASTokenimport dagster as dg@dg.assetdef example_adls2_asset(adls2: ADLS2Resource): df = pd.DataFrame({"column1": [1, 2, 3], "column2": ["A", "B", "C"]}) csv_data = df.to_csv(index=False) file_client = adls2.adls2_client.get_file_client( "my-file-system", "path/to/my_dataframe.csv" ) file_client.upload_data(csv_data, overwrite=True)defs = dg.Definitions( assets=[example_adls2_asset], resources={ "adls2": ADLS2Resource( storage_account="my_storage_account", credential=ADLS2SASToken(token="my_sas_token"), ) },) In this updated code, we use `ADLS2Resource` directly instead of `adls2_resource`. The configuration is passed to `ADLS2Resource` during its instantiation. About Azure Data Lake Storage Gen 2 (ADLS2)[​](https://docs.dagster.io/integrations/libraries/azure-adls2#about-azure-data-lake-storage-gen-2-adls2 "Direct link to About Azure Data Lake Storage Gen 2 (ADLS2)") ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ **Azure Data Lake Storage Gen 2 (ADLS2)** is a set of capabilities dedicated to big data analytics, built on Azure Blob Storage. ADLS2 combines the scalability, cost-effectiveness, security, and rich capabilities of Azure Blob Storage with a high-performance file system that's built for analytics and is compatible with the Hadoop Distributed File System (HDFS). This makes it an ideal choice for data lakes and big data analytics. * [Installation](https://docs.dagster.io/integrations/libraries/azure-adls2#installation) * [Example](https://docs.dagster.io/integrations/libraries/azure-adls2#example) * [About Azure Data Lake Storage Gen 2 (ADLS2)](https://docs.dagster.io/integrations/libraries/azure-adls2#about-azure-data-lake-storage-gen-2-adls2) --- # Dagster & AWS Systems Parameter Store | Dagster Docs [Skip to main content](https://docs.dagster.io/integrations/libraries/aws/ssm#__docusaurus_skipToContent_fallback) On this page info This feature is considered in a beta stage. It is still being tested and may change. For more information, see the [API lifecycle stages documentation](https://docs.dagster.io/api/api-lifecycle/api-lifecycle-stages) . The Dagster AWS Systems Manager (SSM) Parameter Store integration allows you to manage and retrieve parameters stored in AWS SSM Parameter Store directly within your Dagster pipelines. This integration provides resources to fetch parameters by name, tags, or paths, and optionally set them as environment variables for your operations. Installation[​](https://docs.dagster.io/integrations/libraries/aws/ssm#installation "Direct link to Installation") ------------------------------------------------------------------------------------------------------------------- * uv * pip uv add dagster-aws pip install dagster-aws Examples[​](https://docs.dagster.io/integrations/libraries/aws/ssm#examples "Direct link to Examples") ------------------------------------------------------------------------------------------------------- from dagster_aws.ssm import ParameterStoreResource, ParameterStoreTagimport dagster as dg@dg.assetdef example_parameter_store_asset(parameter_store: ParameterStoreResource): parameter_value = parameter_store.fetch_parameters( parameters=["my-parameter-name"] ).get("my-parameter-name") return parameter_value@dg.assetdef example_parameter_store_asset_with_env(parameter_store: ParameterStoreResource): import os with parameter_store.parameters_in_environment(): return os.getenv("my-other-parameter-name")defs = dg.Definitions( assets=[example_parameter_store_asset, example_parameter_store_asset_with_env], resources={ "parameter_store": ParameterStoreResource( region_name="us-west-1", parameter_tags=[ ParameterStoreTag(key="my-tag-key", values=["my-tag-value"]) ], with_decryption=True, ) },) About AWS Systems Parameter Store[​](https://docs.dagster.io/integrations/libraries/aws/ssm#about-aws-systems-parameter-store "Direct link to About AWS Systems Parameter Store") ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- **AWS Systems Manager Parameter Store** is a secure storage service for configuration data management and secrets management. It allows you to store data such as passwords, database strings, and license codes as parameter values. You can then reference these parameters in your applications or scripts, ensuring that sensitive information isn't hard-coded or exposed in your codebase. AWS Systems Manager Parameter Store integrates with AWS Identity and Access Management (IAM) to control access to parameters, and it supports encryption using AWS Key Management Service (KMS) to protect sensitive data. This service is essential for maintaining secure and manageable configurations across your AWS environment. * [Installation](https://docs.dagster.io/integrations/libraries/aws/ssm#installation) * [Examples](https://docs.dagster.io/integrations/libraries/aws/ssm#examples) * [About AWS Systems Parameter Store](https://docs.dagster.io/integrations/libraries/aws/ssm#about-aws-systems-parameter-store) --- # Dagster & Census | Dagster Docs [Skip to main content](https://docs.dagster.io/integrations/libraries/census#__docusaurus_skipToContent_fallback) On this page With the Census integration you can execute a Census sync and poll until that sync completes, raising an error if it's unsuccessful. Installation[​](https://docs.dagster.io/integrations/libraries/census#installation "Direct link to Installation") ------------------------------------------------------------------------------------------------------------------ * uv * pip uv add dagster-census pip install dagster-census Example[​](https://docs.dagster.io/integrations/libraries/census#example "Direct link to Example") --------------------------------------------------------------------------------------------------- from dagster_census import CensusResourceimport dagster as dg@dg.assetdef census_source(census: CensusResource): census.get_source(source_id=1)defs = dg.Definitions( assets=[census_source], resources={"census": CensusResource(api_key=dg.EnvVar("CENSUS_API_KEY"))},) About Census[​](https://docs.dagster.io/integrations/libraries/census#about-census "Direct link to About Census") ------------------------------------------------------------------------------------------------------------------ **Census** syncs data from your cloud warehouse to the SaaS tools your organization uses. It allows everyone in your organization to take action with good data, no custom scripts or API integrations required. * [Installation](https://docs.dagster.io/integrations/libraries/census#installation) * [Example](https://docs.dagster.io/integrations/libraries/census#example) * [About Census](https://docs.dagster.io/integrations/libraries/census#about-census) --- # Dagster & Chroma | Dagster Docs [Skip to main content](https://docs.dagster.io/integrations/libraries/chroma#__docusaurus_skipToContent_fallback) On this page The Chroma library allows you to easily interact with Chroma's vector database capabilities to build AI-powered data pipelines in Dagster. You can perform vector similarity searches, manage schemas, and handle data operations directly from your Dagster assets. Installation[​](https://docs.dagster.io/integrations/libraries/chroma#installation "Direct link to Installation") ------------------------------------------------------------------------------------------------------------------ * uv * pip uv add dagster-chroma pip install dagster-chroma Example[​](https://docs.dagster.io/integrations/libraries/chroma#example "Direct link to Example") --------------------------------------------------------------------------------------------------- import osfrom dagster_chroma import ChromaResource, HttpConfig, LocalConfigimport dagster as dg@dg.assetdef my_table(chroma: ChromaResource): with chroma.get_client() as chroma_client: collection = chroma_client.create_collection("fruits") collection.add( documents=[ "This is a document about oranges", "This is a document about pineapples", "This is a document about strawberries", "This is a document about cucumbers", ], ids=["oranges", "pineapples", "strawberries", "cucumbers"], ) results = collection.query( query_texts=["hawaii"], n_results=1, )defs = dg.Definitions( assets=[my_table], resources={ "chroma": ChromaResource( connection_config=LocalConfig(persistence_path="./chroma") if os.getenv("DEV") else HttpConfig(host="192.168.0.10", port=8000) ), },) About Chroma[​](https://docs.dagster.io/integrations/libraries/chroma#about-chroma "Direct link to About Chroma") ------------------------------------------------------------------------------------------------------------------ **Chroma** is the open-source AI application database. Chroma makes it easy to build LLM apps by making knowledge, facts, and skills pluggable for LLMs. It provides a simple API for storing and querying embeddings, documents, and metadata. Chroma can be used to build semantic search, question answering, and other AI-powered applications. The database can run embedded in your application or as a separate service. * [Installation](https://docs.dagster.io/integrations/libraries/chroma#installation) * [Example](https://docs.dagster.io/integrations/libraries/chroma#example) * [About Chroma](https://docs.dagster.io/integrations/libraries/chroma#about-chroma) --- # Dagster & Cube | Dagster Docs [Skip to main content](https://docs.dagster.io/integrations/libraries/cube#__docusaurus_skipToContent_fallback) On this page With the Cube integration you can setup Cube and Dagster to work together so that Dagster can push changes from upstream data sources to Cube using its integration API. Installation[​](https://docs.dagster.io/integrations/libraries/cube#installation "Direct link to Installation") ---------------------------------------------------------------------------------------------------------------- * uv * pip uv add dagster-cube pip install dagster-cube Example[​](https://docs.dagster.io/integrations/libraries/cube#example "Direct link to Example") ------------------------------------------------------------------------------------------------- from dagster_cube import CubeResourceimport dagster as dg@dg.assetdef cube_query_workflow(cube: CubeResource): response = cube.make_request( method="POST", endpoint="load", data={"query": {"measures": ["Orders.count"], "dimensions": ["Orders.status"]}}, ) return responsedefs = dg.Definitions( assets=[cube_query_workflow], resources={ "cube": CubeResource( instance_url="https://<>.cubecloudapp.dev/cubejs-api/v1/", api_key=dg.EnvVar("CUBE_API_KEY"), ) },) About Cube[​](https://docs.dagster.io/integrations/libraries/cube#about-cube "Direct link to About Cube") ---------------------------------------------------------------------------------------------------------- **Cube.js** is the semantic layer for building data applications. It helps data engineers and application developers access data from modern data stores, organize it into consistent definitions, and deliver it to every application. * [Installation](https://docs.dagster.io/integrations/libraries/cube#installation) * [Example](https://docs.dagster.io/integrations/libraries/cube#example) * [About Cube](https://docs.dagster.io/integrations/libraries/cube#about-cube) --- # Dagster & Databricks | Dagster Docs [Skip to main content](https://docs.dagster.io/integrations/libraries/databricks#__docusaurus_skipToContent_fallback) On this page The Databricks integration library provides the \`PipesDatabricksClient\` resource, enabling you to launch Databricks jobs directly from Dagster assets and ops. This integration allows you to pass parameters to Databricks code while Dagster receives real-time events, such as logs, asset checks, and asset materializations, from the initiated jobs. With minimal code changes required on the job side, this integration is both efficient and easy to implement. Installation[​](https://docs.dagster.io/integrations/libraries/databricks#installation "Direct link to Installation") ---------------------------------------------------------------------------------------------------------------------- * uv * pip uv add dagster-databricks pip install dagster-databricks Example[​](https://docs.dagster.io/integrations/libraries/databricks#example "Direct link to Example") ------------------------------------------------------------------------------------------------------- import sysfrom dagster_databricks import PipesDatabricksClientfrom databricks.sdk import WorkspaceClientfrom databricks.sdk.service import jobsimport dagster as dg@dg.assetdef databricks_asset( context: dg.AssetExecutionContext, pipes_databricks: PipesDatabricksClient): task = jobs.SubmitTask.from_dict( { # The cluster settings below are somewhat arbitrary. Dagster Pipes is # not dependent on a specific spark version, node type, or number of # workers. "new_cluster": { "spark_version": "12.2.x-scala2.12", "node_type_id": "i3.xlarge", "num_workers": 0, "cluster_log_conf": { "dbfs": {"destination": "dbfs:/cluster-logs-dir-noexist"}, }, }, "libraries": [ # Include the latest published version of dagster-pipes on PyPI # in the task environment {"pypi": {"package": "dagster-pipes"}}, ], "task_key": "some-key", "spark_python_task": { "python_file": "dbfs:/my_python_script.py", # location of target code file "source": jobs.Source.WORKSPACE, }, } ) print("This will be forwarded back to Dagster stdout") print("This will be forwarded back to Dagster stderr", file=sys.stderr) extras = {"some_parameter": 100} return pipes_databricks.run( task=task, context=context, extras=extras, ).get_materialize_result()pipes_databricks_resource = PipesDatabricksClient( client=WorkspaceClient( host="https://.cloud.databricks.com", token="", ))defs = dg.Definitions( assets=[databricks_asset], resources={"pipes_databricks": pipes_databricks_resource}) from dagster_pipes import ( PipesDbfsContextLoader, PipesDbfsMessageWriter, open_dagster_pipes,)# Sets up communication channels and downloads the context data sent from Dagster.# Note that while other `context_loader` and `message_writer` settings are# possible, it is recommended to use `PipesDbfsContextLoader` and# `PipesDbfsMessageWriter` for Databricks.with open_dagster_pipes( context_loader=PipesDbfsContextLoader(), message_writer=PipesDbfsMessageWriter(),) as pipes: # Access the `extras` dict passed when launching the job from Dagster. some_parameter_value = pipes.get_extra("some_parameter") # Stream log message back to Dagster pipes.log.info(f"Using some_parameter value: {some_parameter_value}") # ... your code that computes and persists the asset # Stream asset materialization metadata and data version back to Dagster. # This should be called after you've computed and stored the asset value. We # omit the asset key here because there is only one asset in scope, but for # multi-assets you can pass an `asset_key` parameter. pipes.report_asset_materialization( metadata={ "some_metric": {"raw_value": some_parameter_value + 1, "type": "int"} }, data_version="alpha", ) About Databricks[​](https://docs.dagster.io/integrations/libraries/databricks#about-databricks "Direct link to About Databricks") ---------------------------------------------------------------------------------------------------------------------------------- **Databricks** is a unified data analytics platform that simplifies and accelerates the process of building big data and AI solutions. It integrates seamlessly with Apache Spark and offers support for various data sources and formats. Databricks provides powerful tools to create, run, and manage data pipelines, making it easier to handle complex data engineering tasks. Its collaborative and scalable environment is ideal for data engineers, scientists, and analysts who need to process and analyze large datasets efficiently. * [Installation](https://docs.dagster.io/integrations/libraries/databricks#installation) * [Example](https://docs.dagster.io/integrations/libraries/databricks#example) * [About Databricks](https://docs.dagster.io/integrations/libraries/databricks#about-databricks) --- # Dagster & Datadog | Dagster Docs [Skip to main content](https://docs.dagster.io/integrations/libraries/datadog#__docusaurus_skipToContent_fallback) On this page info This feature is considered in a beta stage. It is still being tested and may change. For more information, see the [API lifecycle stages documentation](https://docs.dagster.io/api/api-lifecycle/api-lifecycle-stages) . While Dagster provides comprehensive monitoring and observability of the pipelines it orchestrates, many teams look to centralize all their monitoring across apps, processes and infrastructure using Datadog's 'Cloud Monitoring as a Service'. The Datadog integration allows you to publish metrics to Datadog from within Dagster ops. No-code Datadog integration Datadog’s new Dagster integration streams event logs to Datadog and includes an out-of-the-box log pipeline and dashboard. In Datadog, navigate to the Dagster integration tile and click Connect Accounts to launch the OAuth flow. Within 10 minutes, the Dagster Overview dashboard starts showing new log events, provided there are any active Dagster jobs emitting events. Get started in [Datadog](https://app.datadoghq.com/account/login) or learn more in their [docs](https://docs.datadoghq.com/integrations/dagster-plus/) . Installation[​](https://docs.dagster.io/integrations/libraries/datadog#installation "Direct link to Installation") ------------------------------------------------------------------------------------------------------------------- * uv * pip uv add dagster-datadog pip install dagster-datadog Example[​](https://docs.dagster.io/integrations/libraries/datadog#example "Direct link to Example") ---------------------------------------------------------------------------------------------------- from dagster_datadog import DatadogResourceimport dagster as dg@dg.assetdef report_to_datadog(datadog: DatadogResource): datadog_client = datadog.get_client() datadog_client.event("Man down!", "This server needs assistance.") datadog_client.gauge("users.online", 1001, tags=["protocol:http"]) datadog_client.increment("page.views")defs = dg.Definitions( assets=[report_to_datadog], resources={ "datadog": DatadogResource( api_key=dg.EnvVar("DATADOG_API_KEY"), app_key=dg.EnvVar("DATADOG_APP_KEY"), ) },) About Datadog[​](https://docs.dagster.io/integrations/libraries/datadog#about-datadog "Direct link to About Datadog") ---------------------------------------------------------------------------------------------------------------------- **Datadog** is an observability service for cloud-scale applications, providing monitoring of servers, databases, tools, and services, through a SaaS-based data analytics platform. * [Installation](https://docs.dagster.io/integrations/libraries/datadog#installation) * [Example](https://docs.dagster.io/integrations/libraries/datadog#example) * [About Datadog](https://docs.dagster.io/integrations/libraries/datadog#about-datadog) --- # Dagster & dbt | Dagster Docs [Skip to main content](https://docs.dagster.io/integrations/libraries/dbt#__docusaurus_skipToContent_fallback) On this page Orchestrate your dbt transformations directly with Dagster. Dagster assets understand dbt at the level of individual dbt models. This means that you can: * Use Dagster's UI or APIs to run subsets of your dbt models, seeds, and snapshots. * Track failures, logs, and run history for individual dbt models, seeds, and snapshots. * Define dependencies between individual dbt models and other data assets. For example, put dbt models after the Fivetran-ingested table that they read from, or put a machine learning after the dbt models that it's trained from. Installation[​](https://docs.dagster.io/integrations/libraries/dbt#installation "Direct link to Installation") --------------------------------------------------------------------------------------------------------------- * uv * pip uv add dagster-dbt pip install dagster-dbt Example[​](https://docs.dagster.io/integrations/libraries/dbt#example "Direct link to Example") ------------------------------------------------------------------------------------------------ from pathlib import Pathfrom dagster_dbt import ( DbtCliResource, DbtProject, build_schedule_from_dbt_selection, dbt_assets,)import dagster as dgRELATIVE_PATH_TO_MY_DBT_PROJECT = "./my_dbt_project"my_project = DbtProject( project_dir=Path(__file__) .joinpath("..", RELATIVE_PATH_TO_MY_DBT_PROJECT) .resolve(),)my_project.prepare_if_dev()@dbt_assets(manifest=my_project.manifest_path)def my_dbt_assets(context: dg.AssetExecutionContext, dbt: DbtCliResource): yield from dbt.cli(["build"], context=context).stream()my_schedule = build_schedule_from_dbt_selection( [my_dbt_assets], job_name="materialize_dbt_models", cron_schedule="0 0 * * *", dbt_select="fqn:*",)defs = dg.Definitions( assets=[my_dbt_assets], schedules=[my_schedule], resources={ "dbt": DbtCliResource(project_dir=my_project), },) About dbt[​](https://docs.dagster.io/integrations/libraries/dbt#about-dbt "Direct link to About dbt") ------------------------------------------------------------------------------------------------------ **dbt** is a SQL-first transformation workflow that lets teams quickly and collaboratively deploy analytics code following software engineering best practices like modularity, portability, CI/CD, and documentation. * [Installation](https://docs.dagster.io/integrations/libraries/dbt#installation) * [Example](https://docs.dagster.io/integrations/libraries/dbt#example) * [About dbt](https://docs.dagster.io/integrations/libraries/dbt#about-dbt) --- # Creating a dbt project in a Dagster project | Dagster Docs [Skip to main content](https://docs.dagster.io/integrations/libraries/dbt/creating-a-dbt-project-in-dagster#__docusaurus_skipToContent_fallback) On this page note Using dbt Cloud? Check out the [Dagster & dbt Cloud documentation](https://docs.dagster.io/integrations/libraries/dbt/dbt-cloud) . In this tutorial, we'll walk you through integrating dbt with Dagster using a smaller version of dbt's example [jaffle shop project](https://github.com/dbt-labs/jaffle_shop) , the [dagster-dbt library](https://docs.dagster.io/api/libraries/dagster-dbt) , and a data warehouse, such as [DuckDB](https://duckdb.org/) . By the end of this tutorial, you'll have your dbt models represented in Dagster along with other [Dagster asset definitions](https://docs.dagster.io/integrations/libraries/dbt/reference#dbt-models-and-dagster-asset-definitions) upstream and downstream of them: ![Asset group with dbt models and Python asset](https://docs.dagster.io/assets/images/asset-graph-materialized-547103d77babca260bec920cc408a975.png) To get there, you will: * [Set up a dbt project](https://docs.dagster.io/integrations/libraries/dbt/creating-a-dbt-project-in-dagster/set-up-dbt-project) * [Load the dbt models into Dagster as assets](https://docs.dagster.io/integrations/libraries/dbt/creating-a-dbt-project-in-dagster/load-dbt-models) * [Create and materialize upstream Dagster assets](https://docs.dagster.io/integrations/libraries/dbt/creating-a-dbt-project-in-dagster/upstream-assets) * [Create and materialize a downstream asset](https://docs.dagster.io/integrations/libraries/dbt/creating-a-dbt-project-in-dagster/downstream-assets) that outputs a plotly chart Prerequisites[​](https://docs.dagster.io/integrations/libraries/dbt/creating-a-dbt-project-in-dagster#prerequisites "Direct link to Prerequisites") ---------------------------------------------------------------------------------------------------------------------------------------------------- To complete this tutorial, you'll need: * **To have [git](https://en.wikipedia.org/wiki/Git) installed**. If it's not installed already (find out by typing `git` in your terminal), you can install it using the [instructions on the git website](https://git-scm.com/book/en/v2/Getting-Started-Installing-Git) . * **To install dbt, Dagster, and the Dagster webserver/UI**. Run the following to install everything using pip: * uv * pip uv add dagster-dbt dbt-duckdb pip install dagster-dbt dbt-duckdb The `dagster-dbt` library installs both `dbt-core` and `dagster` as dependencies. `dbt-duckdb` is installed as you'll be using [DuckDB](https://duckdb.org/) as a database during this tutorial. Refer to the [dbt](https://docs.getdbt.com/dbt-cli/install/overview) and [Dagster](https://docs.dagster.io/getting-started/installation) installation docs for more info. Ready to get started?[​](https://docs.dagster.io/integrations/libraries/dbt/creating-a-dbt-project-in-dagster#ready-to-get-started "Direct link to Ready to get started?") --------------------------------------------------------------------------------------------------------------------------------------------------------------------------- When you've fulfilled all the prerequisites for the tutorial, you can get started by [setting up the dbt project](https://docs.dagster.io/integrations/libraries/dbt/creating-a-dbt-project-in-dagster/set-up-dbt-project) . * [Prerequisites](https://docs.dagster.io/integrations/libraries/dbt/creating-a-dbt-project-in-dagster#prerequisites) * [Ready to get started?](https://docs.dagster.io/integrations/libraries/dbt/creating-a-dbt-project-in-dagster#ready-to-get-started) --- # Add a downstream asset | Dagster Docs [Skip to main content](https://docs.dagster.io/integrations/libraries/dbt/creating-a-dbt-project-in-dagster/downstream-assets#__docusaurus_skipToContent_fallback) On this page By this point, you've [set up a dbt project](https://docs.dagster.io/integrations/libraries/dbt/creating-a-dbt-project-in-dagster/set-up-dbt-project) , [loaded dbt models into Dagster as assets](https://docs.dagster.io/integrations/libraries/dbt/creating-a-dbt-project-in-dagster/load-dbt-models) , and [defined assets upstream of your dbt models](https://docs.dagster.io/integrations/libraries/dbt/creating-a-dbt-project-in-dagster/upstream-assets) . In this step, you'll: * [Install the plotly library](https://docs.dagster.io/integrations/libraries/dbt/creating-a-dbt-project-in-dagster/downstream-assets#step-1-install-the-plotly-library) * [Define a downstream asset that computes a chart using plotly](https://docs.dagster.io/integrations/libraries/dbt/creating-a-dbt-project-in-dagster/downstream-assets#step-2-define-the-order_count_chart-asset) * [Materialize the `order_count_chart` asset](https://docs.dagster.io/integrations/libraries/dbt/creating-a-dbt-project-in-dagster/downstream-assets#step-3-materialize-the-order_count_chart-asset) Step 1: Install the plotly library[​](https://docs.dagster.io/integrations/libraries/dbt/creating-a-dbt-project-in-dagster/downstream-assets#step-1-install-the-plotly-library "Direct link to Step 1: Install the plotly library") ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ * uv * pip uv add plotly pip install plotly Step 2: Define the order\_count\_chart asset[​](https://docs.dagster.io/integrations/libraries/dbt/creating-a-dbt-project-in-dagster/downstream-assets#step-2-define-the-order_count_chart-asset "Direct link to Step 2: Define the order_count_chart asset") -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- You've added upstream assets to your data pipeline, but nothing downstream - until now. In this step, you'll define a Dagster asset called `order_count_chart` that uses the data in the `customers` dbt model to computes a plotly chart of the number of orders per customer. Like the `raw_customers` asset that we added in the [previous section](https://docs.dagster.io/integrations/libraries/dbt/creating-a-dbt-project-in-dagster/upstream-assets#step-2-define-an-upstream-dagster-asset) , we'll put this asset in our `assets.py` file, inside the `jaffle_dagster` directory. To add the `order_count_chart` asset: 1. Replace the imports section with the following: import osimport duckdbimport pandas as pdimport plotly.express as pxfrom dagster import MetadataValue, AssetExecutionContext, assetfrom dagster_dbt import DbtCliResource, dbt_assets, get_asset_key_for_modelfrom .project import jaffle_shop_project This adds an import for plotly, as well as [`get_asset_key_for_model`](https://docs.dagster.io/api/libraries/dagster-dbt#dagster_dbt.get_asset_key_for_model) and [`MetadataValue`](https://docs.dagster.io/api/dagster/metadata#dagster.MetadataValue) , which we'll use in our asset. 2. After your definition of `jaffle_shop_dbt_assets`, add the definition for the `order_count_chart` asset: @asset( compute_kind="python", deps=[get_asset_key_for_model([jaffle_shop_dbt_assets], "customers")],)def order_count_chart(context: AssetExecutionContext): # read the contents of the customers table into a Pandas DataFrame connection = duckdb.connect(os.fspath(duckdb_database_path)) customers = connection.sql("select * from customers").df() # create a plot of number of orders by customer and write it out to an HTML file fig = px.histogram(customers, x="number_of_orders") fig.update_layout(bargap=0.2) save_chart_path = duckdb_database_path.parent.joinpath("order_count_chart.html") fig.write_html(save_chart_path, auto_open=True) # tell Dagster about the location of the HTML file, # so it's easy to access from the Dagster UI context.add_output_metadata( {"plot_url": MetadataValue.url("file://" + os.fspath(save_chart_path))} ) This asset definition looks similar the asset we defined in the previous section. In this case, instead of fetching data from an external source and writing it to DuckDB, it reads data from DuckDB, and then uses it to make a plot. The line `deps=[get_asset_key_for_model([jaffle_shop_dbt_assets], "customers")]` tells Dagster that this asset is downstream of the `customers` dbt model. This dependency will be displayed as such in Dagster's UI. If you launch a run to materialize both of them, Dagster won't run `order_count_chart` until `customers` completes. 3. Add the `order_count_chart` to the `Definitions`: from dagster import Definitionsfrom dagster_dbt import DbtCliResourcefrom .assets import jaffle_shop_dbt_assets, order_count_chart, raw_customersfrom .project import jaffle_shop_projectfrom .schedules import schedulesdefs = Definitions( assets=[raw_customers, jaffle_shop_dbt_assets, order_count_chart], schedules=schedules, resources={ "dbt": DbtCliResource(project_dir=jaffle_shop_project), },) Step 3: Materialize the order\_count\_chart asset[​](https://docs.dagster.io/integrations/libraries/dbt/creating-a-dbt-project-in-dagster/downstream-assets#step-3-materialize-the-order_count_chart-asset "Direct link to Step 3: Materialize the order_count_chart asset") ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- If the Dagster UI is still running from the previous section, click the "Reload Definitions" button in the upper right corner. If you shut it down, then you can launch it again with the same command from the previous section: dagster dev The UI will look like this: ![Asset group with dbt models and Python asset](https://docs.dagster.io/assets/images/asset-graph-70fd0df854d1df5a05e9261fd4d13878.png) A new asset named `order_count_chart` is at the bottom, downstream of the `customers` asset. Click on `order_count_chart` and click **Materialize selected**. That's it! When the run successfully completes, the following chart will automatically open in your browser: ![plotly chart asset displayed in Chrome](https://docs.dagster.io/assets/images/order-count-chart-d6f97631b92487c22d6c9d8cdc1915e3.png) What's next?[​](https://docs.dagster.io/integrations/libraries/dbt/creating-a-dbt-project-in-dagster/downstream-assets#whats-next "Direct link to What's next?") ----------------------------------------------------------------------------------------------------------------------------------------------------------------- That's the end of this tutorial - congratulations! By now, you should have a working dbt and Dagster integration and a handful of materialized Dagster assets. What's next? From here, you can: * Learn more about [asset definitions](https://docs.dagster.io/guides/build/assets) * Learn how to [build jobs that materialize dbt assets](https://docs.dagster.io/integrations/libraries/dbt/reference#scheduling-dbt-jobs) * Get a [deeper understanding of Dagster's dbt integration](https://docs.dagster.io/integrations/libraries/dbt/reference) * Check out the [`dagster-dbt` API docs](https://docs.dagster.io/api/libraries/dagster-dbt) * [Step 1: Install the plotly library](https://docs.dagster.io/integrations/libraries/dbt/creating-a-dbt-project-in-dagster/downstream-assets#step-1-install-the-plotly-library) * [Step 2: Define the order\_count\_chart asset](https://docs.dagster.io/integrations/libraries/dbt/creating-a-dbt-project-in-dagster/downstream-assets#step-2-define-the-order_count_chart-asset) * [Step 3: Materialize the order\_count\_chart asset](https://docs.dagster.io/integrations/libraries/dbt/creating-a-dbt-project-in-dagster/downstream-assets#step-3-materialize-the-order_count_chart-asset) * [What's next?](https://docs.dagster.io/integrations/libraries/dbt/creating-a-dbt-project-in-dagster/downstream-assets#whats-next) --- # Load dbt models as Dagster assets | Dagster Docs [Skip to main content](https://docs.dagster.io/integrations/libraries/dbt/creating-a-dbt-project-in-dagster/load-dbt-models#__docusaurus_skipToContent_fallback) On this page At this point, you should have a [fully-configured dbt project](https://docs.dagster.io/integrations/libraries/dbt/creating-a-dbt-project-in-dagster/set-up-dbt-project) that's ready to work with Dagster. In this section, you'll finally begin integrating dbt with Dagster. To do so, you'll: * [Create a Dagster project that wraps your dbt project](https://docs.dagster.io/integrations/libraries/dbt/creating-a-dbt-project-in-dagster/load-dbt-models#step-1-create-a-dagster-project-that-wraps-your-dbt-project) * [Inspect your Dagster project in Dagster's UI](https://docs.dagster.io/integrations/libraries/dbt/creating-a-dbt-project-in-dagster/load-dbt-models#step-2-inspect-your-dagster-project-in-dagsters-ui) * [Build your dbt models in Dagster](https://docs.dagster.io/integrations/libraries/dbt/creating-a-dbt-project-in-dagster/load-dbt-models#step-3-build-your-dbt-models-in-dagster) * [Understand the Python code in your Dagster project](https://docs.dagster.io/integrations/libraries/dbt/creating-a-dbt-project-in-dagster/load-dbt-models#step-4-understand-the-python-code-in-your-dagster-project) Step 1: Create a Dagster project that wraps your dbt project[​](https://docs.dagster.io/integrations/libraries/dbt/creating-a-dbt-project-in-dagster/load-dbt-models#step-1-create-a-dagster-project-that-wraps-your-dbt-project "Direct link to Step 1: Create a Dagster project that wraps your dbt project") ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- You can create a Dagster project that wraps your dbt project by using the `dagster-dbt` command line interface. Make sure you're in the directory where your `dbt_project.yml` is. If you're continuing from the previous section, then you'll already be in this directory. Then, run: dagster-dbt project scaffold --project-name jaffle_dagster This creates a directory called `jaffle_dagster/` inside the current directory. The `jaffle_dagster/` directory contains a set of files that define a Dagster project. In general, it's up to you where to put your Dagster project. It's most common to put your Dagster project at the root of your git repository. Therefore, in this case, because the `dbt_project.yml` was at the root of the `jaffle_shop` git repository, we created our Dagster project there. **Note**: The `dagster-dbt project scaffold` command creates the Dagster project in whatever directory you run it from. If that's a different directory from where your `dbt_project.yml` lives, then you'll need to provide a value for the `--dbt-project-dir` option so that Dagster knows where to look for your dbt project. Step 2: Inspect your Dagster project in Dagster's UI[​](https://docs.dagster.io/integrations/libraries/dbt/creating-a-dbt-project-in-dagster/load-dbt-models#step-2-inspect-your-dagster-project-in-dagsters-ui "Direct link to Step 2: Inspect your Dagster project in Dagster's UI") --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Now that you have a Dagster project, you can run Dagster's UI to take a look at it. 1. Change directories to the Dagster project directory: cd jaffle_dagster/ 2. To start Dagster's UI, run the following: dagster dev Which will result in output similar to: Serving dagster-webserver on http://127.0.0.1:3000 in process 70635 3. In your browser, navigate to [http://127.0.0.1:3000](http://127.0.0.1:3000/) The page will display the assets: ![Asset graph in Dagster's UI, containing dbt models loaded as Dagster assets](https://docs.dagster.io/assets/images/asset-graph-b448ffe8c111a3e95aa0ff5d374f7ed4.png) Step 3: Build your dbt models in Dagster[​](https://docs.dagster.io/integrations/libraries/dbt/creating-a-dbt-project-in-dagster/load-dbt-models#step-3-build-your-dbt-models-in-dagster "Direct link to Step 3: Build your dbt models in Dagster") ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- You can do more than view your dbt models in Dagster – you can also run them. In Dagster, running a dbt model corresponds to _materializing_ an asset. Materializing an asset means running some computation to update its contents in persistent storage. In this tutorial, that persistent storage is our local DuckDB database. To build your dbt project, i.e. materialize your assets, click the **Materialize all** button near the top right corner of the page. This will launch a run to materialize the assets. When finished, the **Materialized** and **Latest Run** attributes in the asset will be populated: ![Asset graph in Dagster's UI, showing materialized assets](https://docs.dagster.io/assets/images/asset-graph-materialized-2506d4b786513bf20a1d03381fb8a8ce.png) After the run completes, you can: * Click the **asset** to open a sidebar containing info about the asset, including its last materialization stats and a link to view the **Asset details** page * Click the ID of the **Latest Run** in an asset to view the **Run details** page. This page contains detailed info about the run, including timing information, errors, and logs. Step 4: Understand the Python code in your Dagster project[​](https://docs.dagster.io/integrations/libraries/dbt/creating-a-dbt-project-in-dagster/load-dbt-models#step-4-understand-the-python-code-in-your-dagster-project "Direct link to Step 4: Understand the Python code in your Dagster project") ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- You saw how you can create a Dagster project that loads a dbt project. How does this work? Understanding how Dagster loads a dbt project will give you a foundation for customizing how Dagster runs your dbt project, as well as for connecting it to other data assets outside of dbt. The most important file is the Python file that contains the set of definitions for Dagster to load: `jaffle_shop/definitions.py`. Dagster executes the code in this file to find out what assets it should be aware of, as well as details about those assets. For example, when you ran `dagster dev` in the previous step, Dagster executed the code in this file to determine what assets to display in the UI. In our `definitions.py` Python file, we import from `assets.py`, which contains the code to model our dbt models as Dagster assets. To return a Dagster asset for each dbt model, the code in this `assets.py` file needs to know what dbt models you have. It finds out what models you have by reading a file called a `manifest.json`, which is a file that dbt can generate for any dbt project and contains information about every model, seed, snapshot, test, etc. in the project. To retrieve the `manifest.json`, `assets.py` imports from `project.py`, which defines an internal representation of your dbt project. Then, in `assets.py`, the path to the `manifest.json` file can be accessed with `jaffle_shop_project.manifest_path`: from pathlib import Pathfrom dagster_dbt import DbtProjectjaffle_shop_project = DbtProject( project_dir=Path(__file__).joinpath("..", "..", "..").resolve(), packaged_project_dir=Path(__file__).joinpath("..", "..", "dbt-project").resolve(),)# If `dagster dev` is used, the dbt project will be prepared to create the manifest at run time.# Otherwise, we expect a manifest to be present in the project's target directory.jaffle_shop_project.prepare_if_dev() Generating the `manifest.json` file for a dbt project is time-consuming, so it's best to avoid doing so every time this Python module is imported. Thus, in production deployments of Dagster, you'll typically have the CI/CD system that packages up your code generate your `manifest.json`. However, in development, you typically want changes made to files in your dbt project to be immediately reflected in the Dagster UI without needing to regenerate the manifest. `jaffle_shop_project.prepare_if_dev()` helps with this – it re-generates your `manifest.json` at the time Dagster imports your code, _but_ only if it's being imported by the `dagster dev` command. Once you've got a `manifest.json` file, it's time to define your Dagster assets using it. The following code, in your project's `assets.py`, does this: from dagster import AssetExecutionContextfrom dagster_dbt import DbtCliResource, dbt_assetsfrom .project import jaffle_shop_project@dbt_assets(manifest=jaffle_shop_project.manifest_path)def jaffle_shop_dbt_assets(context: AssetExecutionContext, dbt: DbtCliResource): yield from dbt.cli(["build"], context=context).stream() This code might look a bit fancy, because it uses a decorator. Here's a breakdown of what's going on: * It creates a variable named `jaffle_shop_dbt_assets` that holds an object that represents a set of Dagster assets. * These Dagster assets reflect the dbt models described in the manifest file. The manifest file is passed in using the `manifest` argument. * The decorated function defines what should happen when you materialize one of these Dagster assets, e.g. by clicking the **Materialize** button in the UI or materializing it automatically by putting it on a schedule. In this case, it will invoke the `dbt build` command on the selected assets. The `context` parameter that's provided along with `dbt build` carries the selection. If you later want to customize how your dbt models are translated into Dagster assets, you'll do so by editing its definition in `assets.py`. What's next?[​](https://docs.dagster.io/integrations/libraries/dbt/creating-a-dbt-project-in-dagster/load-dbt-models#whats-next "Direct link to What's next?") --------------------------------------------------------------------------------------------------------------------------------------------------------------- At this point, you've loaded your dbt models into Dagster as assets, viewed them in Dagster's asset graph UI, and materialized them. Next, you'll learn how to [add upstream Dagster assets](https://docs.dagster.io/integrations/libraries/dbt/creating-a-dbt-project-in-dagster/upstream-assets) . * [Step 1: Create a Dagster project that wraps your dbt project](https://docs.dagster.io/integrations/libraries/dbt/creating-a-dbt-project-in-dagster/load-dbt-models#step-1-create-a-dagster-project-that-wraps-your-dbt-project) * [Step 2: Inspect your Dagster project in Dagster's UI](https://docs.dagster.io/integrations/libraries/dbt/creating-a-dbt-project-in-dagster/load-dbt-models#step-2-inspect-your-dagster-project-in-dagsters-ui) * [Step 3: Build your dbt models in Dagster](https://docs.dagster.io/integrations/libraries/dbt/creating-a-dbt-project-in-dagster/load-dbt-models#step-3-build-your-dbt-models-in-dagster) * [Step 4: Understand the Python code in your Dagster project](https://docs.dagster.io/integrations/libraries/dbt/creating-a-dbt-project-in-dagster/load-dbt-models#step-4-understand-the-python-code-in-your-dagster-project) * [What's next?](https://docs.dagster.io/integrations/libraries/dbt/creating-a-dbt-project-in-dagster/load-dbt-models#whats-next) --- # Set up the dbt project | Dagster Docs [Skip to main content](https://docs.dagster.io/integrations/libraries/dbt/creating-a-dbt-project-in-dagster/set-up-dbt-project#__docusaurus_skipToContent_fallback) On this page In this part of the tutorial, you will: * [Download a dbt project](https://docs.dagster.io/integrations/libraries/dbt/creating-a-dbt-project-in-dagster/set-up-dbt-project#step-1-download-the-sample-dbt-project) * [Configure your dbt project to run with DuckDB](https://docs.dagster.io/integrations/libraries/dbt/creating-a-dbt-project-in-dagster/set-up-dbt-project#step-2-configure-your-dbt-project-to-run-with-duckdb) * [Build your dbt project](https://docs.dagster.io/integrations/libraries/dbt/creating-a-dbt-project-in-dagster/set-up-dbt-project#step-3-build-your-dbt-project) Step 1: Download the sample dbt project[​](https://docs.dagster.io/integrations/libraries/dbt/creating-a-dbt-project-in-dagster/set-up-dbt-project#step-1-download-the-sample-dbt-project "Direct link to Step 1: Download the sample dbt project") ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Let's get started by downloading a sample dbt project. We'll use the standard dbt [Jaffle Shop](https://github.com/dbt-labs/jaffle_shop) example. 1. First, create a folder that will ultimately contain both your dbt project and Dagster code. mkdir tutorial-dbt-dagster 2. Then, navigate into that folder: cd tutorial-dbt-dagster 3. Finally, download the sample dbt project into that folder. git clone https://github.com/dbt-labs/jaffle_shop.git Step 2: Configure your dbt project to run with DuckDB[​](https://docs.dagster.io/integrations/libraries/dbt/creating-a-dbt-project-in-dagster/set-up-dbt-project#step-2-configure-your-dbt-project-to-run-with-duckdb "Direct link to Step 2: Configure your dbt project to run with DuckDB") ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Running dbt requires a data warehouse to store the tables that are created from the dbt models. For our data warehouse, we'll use DuckDB, because setting it up doesn't require any long-running services or external infrastructure. You'll set up dbt to work with DuckDB by configuring a dbt [profile](https://docs.getdbt.com/docs/core/connect-data-platform/connection-profiles) : 1. Navigate into the `jaffle_shop` folder, which was created when you downloaded the project, inside your `tutorial-dbt-dagster` folder: cd jaffle_shop 2. In this folder, with your text editor of choice, create a file named `profiles.yml` and add the following code to it: jaffle_shop: target: dev outputs: dev: type: duckdb path: tutorial.duckdb threads: 24 Step 3: Build your dbt project[​](https://docs.dagster.io/integrations/libraries/dbt/creating-a-dbt-project-in-dagster/set-up-dbt-project#step-3-build-your-dbt-project "Direct link to Step 3: Build your dbt project") ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- With the profile configured above, your dbt project should now be usable. To test it out, run: dbt build This will run all the models, seeds, and snapshots in the project and store a set of tables in your DuckDB database. note For other dbt projects, you may need to run additional commands before building the project. For instance, a project with [dependencies](https://docs.getdbt.com/docs/collaborate/govern/project-dependencies) will require you to run [`dbt deps`](https://docs.getdbt.com/reference/commands/deps) before building the project. For more information, see the [official dbt Command reference page](https://docs.getdbt.com/reference/dbt-commands) . note What's next?[​](https://docs.dagster.io/integrations/libraries/dbt/creating-a-dbt-project-in-dagster/set-up-dbt-project#whats-next "Direct link to What's next?") ------------------------------------------------------------------------------------------------------------------------------------------------------------------ At this point, you should have a fully-configured dbt project that's ready to work with Dagster. The next step is to [load the dbt models into Dagster as assets](https://docs.dagster.io/integrations/libraries/dbt/creating-a-dbt-project-in-dagster/load-dbt-models) . * [Step 1: Download the sample dbt project](https://docs.dagster.io/integrations/libraries/dbt/creating-a-dbt-project-in-dagster/set-up-dbt-project#step-1-download-the-sample-dbt-project) * [Step 2: Configure your dbt project to run with DuckDB](https://docs.dagster.io/integrations/libraries/dbt/creating-a-dbt-project-in-dagster/set-up-dbt-project#step-2-configure-your-dbt-project-to-run-with-duckdb) * [Step 3: Build your dbt project](https://docs.dagster.io/integrations/libraries/dbt/creating-a-dbt-project-in-dagster/set-up-dbt-project#step-3-build-your-dbt-project) * [What's next?](https://docs.dagster.io/integrations/libraries/dbt/creating-a-dbt-project-in-dagster/set-up-dbt-project#whats-next) --- # Define assets upstream of your dbt models | Dagster Docs [Skip to main content](https://docs.dagster.io/integrations/libraries/dbt/creating-a-dbt-project-in-dagster/upstream-assets#__docusaurus_skipToContent_fallback) On this page By this point, you've [set up a dbt project](https://docs.dagster.io/integrations/libraries/dbt/creating-a-dbt-project-in-dagster/set-up-dbt-project) and [loaded dbt models into Dagster as assets](https://docs.dagster.io/integrations/libraries/dbt/creating-a-dbt-project-in-dagster/load-dbt-models) . However, the tables at the root of the pipeline are static: they're [dbt seeds](https://docs.getdbt.com/docs/build/seeds) , CSVs that are hardcoded into the dbt project. In a more realistic data pipeline, these tables would typically be ingested from some external data source, for example by using a tool like Airbyte or Fivetran, or by Python code. These ingestion steps in the pipeline often don't make sense to define inside dbt, but they often still do make sense to define as Dagster assets. You can think of a Dagster asset definition as a more general version of a dbt model. A dbt model is one kind of asset, but another kind is one that's defined in Python, using Dagster's Python API. The dbt integration reference page includes a [section](https://docs.dagster.io/integrations/libraries/dbt/reference#dbt-models-and-dagster-asset-definitions) that outlines the parallels between dbt models and Dagster asset definitions. In this section, you'll replace the `raw_customers` dbt seed with a Dagster asset that represents it. You'll write Python code that populates this table by fetching data from the web. This will allow you to launch runs that first execute Python code to populate the `raw_customers` table and then invoke dbt to populate the downstream tables. You'll: * [Install the Pandas and DuckDB Python libraries](https://docs.dagster.io/integrations/libraries/dbt/creating-a-dbt-project-in-dagster/upstream-assets#step-1-install-the-pandas-and-duckdb-python-libraries) * [Define an upstream Dagster asset](https://docs.dagster.io/integrations/libraries/dbt/creating-a-dbt-project-in-dagster/upstream-assets#step-2-define-an-upstream-dagster-asset) * [In the dbt project, replace a seed with a source](https://docs.dagster.io/integrations/libraries/dbt/creating-a-dbt-project-in-dagster/upstream-assets#step-3-in-the-dbt-project-replace-a-seed-with-a-source) * [Materialize the assets using the Dagster UI](https://docs.dagster.io/integrations/libraries/dbt/creating-a-dbt-project-in-dagster/upstream-assets#step-4-materialize-the-assets-using-the-dagster-ui) Step 1: Install the Pandas and DuckDB Python libraries[​](https://docs.dagster.io/integrations/libraries/dbt/creating-a-dbt-project-in-dagster/upstream-assets#step-1-install-the-pandas-and-duckdb-python-libraries "Direct link to Step 1: Install the Pandas and DuckDB Python libraries") ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- The Dagster asset that you write will fetch data using [Pandas](https://pandas.pydata.org/) and write it out to your DuckDB warehouse using [DuckDB's Python API](https://duckdb.org/docs/api/python/overview.html) . To use these, you'll need to install them: * uv * pip uv add pandas duckdb pyarrow pip install pandas duckdb pyarrow Step 2: Define an upstream Dagster asset[​](https://docs.dagster.io/integrations/libraries/dbt/creating-a-dbt-project-in-dagster/upstream-assets#step-2-define-an-upstream-dagster-asset "Direct link to Step 2: Define an upstream Dagster asset") ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- To fetch the data the dbt models require, we'll write a Dagster asset for `raw_customers`. We'll put this asset in our `assets.py` file, inside the `jaffle_dagster` directory. This is the file that contains the code that defines our dbt models, which we reviewed at the end of the [last section](https://docs.dagster.io/integrations/libraries/dbt/creating-a-dbt-project-in-dagster/load-dbt-models#step-4-understand-the-python-code-in-your-dagster-project) . Copy and paste this code to overwrite the existing contents of that file: import osimport duckdbimport pandas as pdfrom dagster import AssetExecutionContext, assetfrom dagster_dbt import DbtCliResource, dbt_assetsfrom .project import jaffle_shop_projectduckdb_database_path = jaffle_shop_project.project_dir.joinpath("tutorial.duckdb")@asset(compute_kind="python")def raw_customers(context: AssetExecutionContext) -> None: data = pd.read_csv("https://docs.dagster.io/assets/customers.csv") connection = duckdb.connect(os.fspath(duckdb_database_path)) connection.execute("create schema if not exists jaffle_shop") connection.execute( "create or replace table jaffle_shop.raw_customers as select * from data" ) # Log some metadata about the table we just wrote. It will show up in the UI. context.add_output_metadata({"num_rows": data.shape[0]})@dbt_assets(manifest=jaffle_shop_project.manifest_path)def jaffle_shop_dbt_assets(context: AssetExecutionContext, dbt: DbtCliResource): yield from dbt.cli(["build"], context=context).stream() Let's review the changes we made: 1. At the top, we added imports for `pandas` and `duckdb`, which we use for fetching data into a `DataFrame` and writing it to DuckDB. 2. We added a `duckdb_database_path` variable, which holds the location of our DuckDB database. Remember that DuckDB databases are just regular files on the local filesystem. The path is the same path that we used when we configured our `profiles.yml` file. This variable is used in the implementations of the `raw_customers` asset. 3. We added a definition for the `raw_customers` table by writing a function named `raw_customers` and decorating it with the [`@dg.asset`](https://docs.dagster.io/api/dagster/assets#dagster.asset) decorator. We labeled it with `compute_kind="python"` to indicate in the Dagster UI that this is an asset defined in Python. The implementation inside the function fetches data from the internet and writes it to a table in our DuckDB database. Similar to how running a dbt model executes a select statement, materializing this asset will execute this Python code. Finally, let's update the `assets` argument of our `Definitions` object, in `definitions.py`, to include the new asset we just defined: from dagster import Definitionsfrom dagster_dbt import DbtCliResourcefrom .assets import jaffle_shop_dbt_assets, raw_customersfrom .project import jaffle_shop_projectfrom .schedules import schedulesdefs = Definitions( assets=[raw_customers, jaffle_shop_dbt_assets], schedules=schedules, resources={ "dbt": DbtCliResource(project_dir=jaffle_shop_project), },) Step 3: In the dbt project, replace a seed with a source[​](https://docs.dagster.io/integrations/libraries/dbt/creating-a-dbt-project-in-dagster/upstream-assets#step-3-in-the-dbt-project-replace-a-seed-with-a-source "Direct link to Step 3: In the dbt project, replace a seed with a source") --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 1. Because we're replacing it with a Dagster asset, we no longer need the dbt seed for `raw_customers`, so we can delete it: cd ..rm seeds/raw_customers.csv 2. Instead, we want to tell dbt that `raw_customers` is a table that is defined outside of the dbt project. We can do that by defining it inside a [dbt source](https://docs.getdbt.com/docs/build/sources) . Create a file called `sources.yml` inside the `models/` directory, and put this inside it: version: 2sources: - name: jaffle_shop tables: - name: raw_customers meta: dagster: asset_key: ['raw_customers'] # This metadata specifies the corresponding Dagster asset for this dbt source. This is a standard dbt source definition, with one addition: it includes metadata, under the `meta` property, that specifies the Dagster asset that it corresponds to. When Dagster reads the contents of the dbt project, it reads this metadata and infers the correspondence. For any dbt model that depends on this dbt source, Dagster then knows that the Dagster asset corresponding to the dbt model should depend on the Dagster asset corresponding to the source. 3. Then, update the model that depends on the `raw_customers` seed to instead depend on the source. Replace the contents of `model/staging/stg_customers.sql` with this: with source as ( {#- Use source instead of seed: #} select * from {{ source('jaffle_shop', 'raw_customers') }}),renamed as ( select id as customer_id, first_name, last_name from source)select * from renamed Step 4: Materialize the assets using the Dagster UI[​](https://docs.dagster.io/integrations/libraries/dbt/creating-a-dbt-project-in-dagster/upstream-assets#step-4-materialize-the-assets-using-the-dagster-ui "Direct link to Step 4: Materialize the assets using the Dagster UI") ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- If the Dagster UI is still running from the previous section, click the "Reload Definitions" button in the upper right corner. If you shut it down, then you can launch it again with the same command from the previous section: dagster dev Our `raw_customers` model is now defined as a Python asset. We can also see that assets downstream of this new Python asset, such as `stg_customers` and `customers`, are now marked stale because the code definition of `raw_customers` has changed. ![Asset group with dbt models and Python asset](https://docs.dagster.io/assets/images/asset-graph-9fa4b3cb85499fc0aff64affdd88ecc2.png) Click the **Materialize all** button. This will launch a run with two steps: * Run the `raw_customers` Python function to fetch data and write the `raw_customers` table to DuckDB. * Run all the dbt models using `dbt build`, like in the last section. If you click to view the run, you can see a graphical representation of these steps, along with logs. ![Run page for run with dbt models and Python asset](https://docs.dagster.io/assets/images/run-page-8228150876819c57c648be45e66886be.png) What's next?[​](https://docs.dagster.io/integrations/libraries/dbt/creating-a-dbt-project-in-dagster/upstream-assets#whats-next "Direct link to What's next?") --------------------------------------------------------------------------------------------------------------------------------------------------------------- At this point, you've built and materialized an upstream Dagster asset, providing source data to your dbt models. In the last section of the tutorial, we'll show you how to add a [downstream asset to the pipeline](https://docs.dagster.io/integrations/libraries/dbt/creating-a-dbt-project-in-dagster/downstream-assets) . * [Step 1: Install the Pandas and DuckDB Python libraries](https://docs.dagster.io/integrations/libraries/dbt/creating-a-dbt-project-in-dagster/upstream-assets#step-1-install-the-pandas-and-duckdb-python-libraries) * [Step 2: Define an upstream Dagster asset](https://docs.dagster.io/integrations/libraries/dbt/creating-a-dbt-project-in-dagster/upstream-assets#step-2-define-an-upstream-dagster-asset) * [Step 3: In the dbt project, replace a seed with a source](https://docs.dagster.io/integrations/libraries/dbt/creating-a-dbt-project-in-dagster/upstream-assets#step-3-in-the-dbt-project-replace-a-seed-with-a-source) * [Step 4: Materialize the assets using the Dagster UI](https://docs.dagster.io/integrations/libraries/dbt/creating-a-dbt-project-in-dagster/upstream-assets#step-4-materialize-the-assets-using-the-dagster-ui) * [What's next?](https://docs.dagster.io/integrations/libraries/dbt/creating-a-dbt-project-in-dagster/upstream-assets#whats-next) --- # One doc tagged with "other" | Dagster Docs [Skip to main content](https://docs.dagster.io/tags/integrations/other#__docusaurus_skipToContent_fallback) [Dagster & Airlift\ -----------------](https://docs.dagster.io/integrations/libraries/airlift) Airlift is a toolkit for integrating Dagster and Airflow. --- # Migration & upgrading | Dagster Docs [Skip to main content](https://docs.dagster.io/migration#__docusaurus_skipToContent_fallback) --- # 3 docs tagged with "monitoring" | Dagster Docs [Skip to main content](https://docs.dagster.io/tags/integrations/monitoring#__docusaurus_skipToContent_fallback) [Dagster & AWS CloudWatch\ ------------------------](https://docs.dagster.io/integrations/libraries/aws/cloudwatch) This integration allows you to send Dagster logs to AWS CloudWatch, enabling centralized logging and monitoring of your Dagster jobs. By using AWS CloudWatch, you can take advantage of its powerful log management features, such as real-time log monitoring, log retention policies, and alerting capabilities. [Dagster & Datadog\ -----------------](https://docs.dagster.io/integrations/libraries/datadog) While Dagster provides comprehensive monitoring and observability of the pipelines it orchestrates, many teams look to centralize all their monitoring across apps, processes and infrastructure using Datadog's 'Cloud Monitoring as a Service'. The Datadog integration allows you to publish metrics to Datadog from within Dagster ops. [Dagster & Prometheus\ --------------------](https://docs.dagster.io/integrations/libraries/prometheus) This integration allows you to push metrics to the Prometheus gateway from within a Dagster pipeline. --- # Airflow to Dagster | Dagster Docs [Skip to main content](https://docs.dagster.io/migration/airflow-to-dagster#__docusaurus_skipToContent_fallback) --- # Migrate from Dagster OSS to Dagster+ | Dagster Docs [Skip to main content](https://docs.dagster.io/migration/oss-to-dagster-plus#__docusaurus_skipToContent_fallback) On this page Follow the steps below to migrate from OSS to Dagster+. Step 1: Get started with Dagster+[​](https://docs.dagster.io/migration/oss-to-dagster-plus#step-1-get-started-with-dagster "Direct link to Step 1: Get started with Dagster+") ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- First, you will need to create a Dagster+ organization, choose your deployment type (Hybrid or Serverless), and set up users and authentication. To get started, see the [Dagster+ documentation](https://docs.dagster.io/deployment/dagster-plus/getting-started) . Step 2: Update CI/CD pipeline[​](https://docs.dagster.io/migration/oss-to-dagster-plus#step-2-update-cicd-pipeline "Direct link to Step 2: Update CI/CD pipeline") ------------------------------------------------------------------------------------------------------------------------------------------------------------------- Next, you will need to modify the CI/CD process that deploys your OSS code to follow the Dagster+ deployment pattern. For more information, see the [Dagster+ CI/CD documentation](https://docs.dagster.io/deployment/dagster-plus/ci-cd/ci-cd-in-hybrid) . Step 3: Populate metadata in Dagster+[​](https://docs.dagster.io/migration/oss-to-dagster-plus#step-3-populate-metadata-in-dagster "Direct link to Step 3: Populate metadata in Dagster+") ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- At this point, you should have the same data pipelines in OSS and Dagster+, but the metadata in Dagster+ will be empty. You can either cut over to Dagster+ and start populating metadata after that point, or migrate historical metadata from OSS. ### Option 1: Populate metadata after cutover[​](https://docs.dagster.io/migration/oss-to-dagster-plus#option-1-populate-metadata-after-cutover "Direct link to Option 1: Populate metadata after cutover") If you don't need to migrate historical metadata from your OSS deployment to Dagster+, you can turn off your Dagster OSS deployment and enable the schedules, sensors, and other metadata tracking features in Dagster+. Metadata will start to appear in Dagster+ from that point forward as assets are materialized. ### Option 2: Migrate historical metadata[​](https://docs.dagster.io/migration/oss-to-dagster-plus#option-2-migrate-historical-metadata "Direct link to Option 2: Migrate historical metadata") To migrate historical metadata from your OSS deployment to Dagster+, follow the steps in the [OSS metadata to plus example](https://github.com/dagster-io/dagster/tree/master/examples/oss-metadata-to-plus) . * [Step 1: Get started with Dagster+](https://docs.dagster.io/migration/oss-to-dagster-plus#step-1-get-started-with-dagster) * [Step 2: Update CI/CD pipeline](https://docs.dagster.io/migration/oss-to-dagster-plus#step-2-update-cicd-pipeline) * [Step 3: Populate metadata in Dagster+](https://docs.dagster.io/migration/oss-to-dagster-plus#step-3-populate-metadata-in-dagster) * [Option 1: Populate metadata after cutover](https://docs.dagster.io/migration/oss-to-dagster-plus#option-1-populate-metadata-after-cutover) * [Option 2: Migrate historical metadata](https://docs.dagster.io/migration/oss-to-dagster-plus#option-2-migrate-historical-metadata) --- # 4 docs tagged with "reference-architecture" | Dagster Docs [Skip to main content](https://docs.dagster.io/tags/examples/reference-architecture#__docusaurus_skipToContent_fallback) [BI\ --](https://docs.dagster.io/examples/bi) An event-driven platform that ingests and analyzes data with SQL and Notebooks. [ETL/Reverse ETL\ ---------------](https://docs.dagster.io/examples/etl-reverse-etl) A pipeline that ingests, models, and syncs data between source systems and a warehouse. [Real-time\ ---------](https://docs.dagster.io/examples/real-time) A real-time system that detects abandoned carts and sends notifications to a marketing platform. [Retrieval-Augmented Generation (RAG)\ ------------------------------------](https://docs.dagster.io/examples/retrieval-augmented-generation) A RAG system that indexes data and uses retrieved context to generate responses. --- # 5 docs tagged with "code-example" | Dagster Docs [Skip to main content](https://docs.dagster.io/tags/examples/code-example#__docusaurus_skipToContent_fallback) [Analyzing Bluesky data\ ----------------------](https://docs.dagster.io/examples/bluesky/) Learn how to build an end-to-end analytics pipeline [LLM fine-tuning with OpenAI\ ---------------------------](https://docs.dagster.io/examples/llm-fine-tuning/) Learn how to fine-tune an LLM [Podcast transcription with Modal\ --------------------------------](https://docs.dagster.io/examples/modal/) Learn how to build with Modal [Prompt engineering and Anthropic\ --------------------------------](https://docs.dagster.io/examples/prompt-engineering/) Learn how to do prompt engineering [Retrieval-Augmented Generation (RAG) with Pinecone\ --------------------------------------------------](https://docs.dagster.io/examples/rag/) Learn how to build a RAG system --- # Using Dagster and Airflow together | Dagster Docs [Skip to main content](https://docs.dagster.io/migration/airflow-to-dagster/airflow-component-tutorial#__docusaurus_skipToContent_fallback) On this page info **Dagster Components is currently in Release Candidate status.** APIs are stable, with broader integration coverage and full feature parity in active development and coming soon. Check it out and let us know what you think in the #dg-components channel in the [Dagster Community Slack](https://www.dagster.io/slack) ! The [dagster-airlift](https://docs.dagster.io/integrations/libraries/airlift) library provides an `AirflowInstanceComponent` which can be used to represent Airflow DAGs in Dagster, allowing easy interoperability between Airflow and Dagster. Setup and peering[​](https://docs.dagster.io/migration/airflow-to-dagster/airflow-component-tutorial#setup-and-peering "Direct link to Setup and peering") ----------------------------------------------------------------------------------------------------------------------------------------------------------- ### 1\. Prepare a Dagster project[​](https://docs.dagster.io/migration/airflow-to-dagster/airflow-component-tutorial#1-prepare-a-dagster-project "Direct link to 1. Prepare a Dagster project") To begin, you'll need a Dagster project. You can use an [existing components-ready project](https://docs.dagster.io/guides/build/projects/moving-to-components/migrating-project) or create a new one: uvx -U create-dagster project my-project && cd my-project Activate the project virtual environment: source .venv/bin/activate Finally, add the `dagster-airlift` library to the project: uv add 'dagster-airlift[core]' ### 2\. Scaffold an AirflowInstanceComponent[​](https://docs.dagster.io/migration/airflow-to-dagster/airflow-component-tutorial#2-scaffold-an-airflowinstancecomponent "Direct link to 2. Scaffold an AirflowInstanceComponent") note Currently `dagster-airlift` only supports basic authentication against an Airflow instance. To scaffold a new component in your project, use the `dg scaffold defs` command: dg scaffold defs dagster_airlift.core.components.AirflowInstanceComponent airflow --name my_airflow --auth-type basic_auth Creating defs at /.../my-project/src/my_project/defs/airflow. This will create a component definition file called `defs.yaml` in your project under the `src/my_project/defs/airflow` directory. tree src/my_project/defs src/my_project/defs├── __init__.py└── airflow └── defs.yaml2 directories, 2 files ### 4\. Update `defs.yaml` with Airflow configuration[​](https://docs.dagster.io/migration/airflow-to-dagster/airflow-component-tutorial#4-update-defsyaml-with-airflow-configuration "Direct link to 4-update-defsyaml-with-airflow-configuration") By default, the Airlift component reads values from the environment variables `AIRFLOW_WEBSERVER_URL`, `AIRFLOW_USERNAME`, and `AIRFLOW_PASSWORD`. While you should never include your password directly in this file, you can update `defs.yaml` to add the webserver URL and username: cat src/my_project/defs/airflow/defs.yaml type: dagster_airlift.core.components.AirflowInstanceComponentattributes: name: my_airflow auth: type: basic_auth webserver_url: '{{ env("AIRFLOW_WEBSERVER_URL") }}' username: '{{ env("AIRFLOW_USERNAME") }}' password: '{{ env("AIRFLOW_PASSWORD") }}' Once you have added these values, the following will happen: 1. Dagster will create a sensor called `your_airlift_instance__airflow_monitoring_job_sensor` that is responsible for detecting runs in your Airflow instance and pulling them into Dagster. 2. Your Airflow DAGs will be represented in Dagster in the "Jobs" page, and any jobs pulled from Airflow will be marked with an Airflow icon. 3. Airflow datasets will be represented in Dagster as assets. 4. When an Airflow DAG executes, that run will be represented in Dagster. Mapping Dagster assets to Airflow tasks[​](https://docs.dagster.io/migration/airflow-to-dagster/airflow-component-tutorial#mapping-dagster-assets-to-airflow-tasks "Direct link to Mapping Dagster assets to Airflow tasks") ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Once you have represented your Airflow instance in Dagster using the Airflow instance component, you may want to represent the graph of asset dependencies produced by that DAG as well, which you can do in `defs.yaml`. ### DAG-level mapping[​](https://docs.dagster.io/migration/airflow-to-dagster/airflow-component-tutorial#dag-level-mapping "Direct link to DAG-level mapping") You can manually define which assets are produced by a given Airflow DAG by editing `mappings` in `defs.yaml`: type: dagster_airlift.core.components.AirflowInstanceComponentattributes: name: my_airflow auth: type: basic_auth webserver_url: '{{ env("AIRFLOW_WEBSERVER_URL") }}' username: '{{ env("AIRFLOW_USERNAME") }}' password: '{{ env("AIRFLOW_PASSWORD") }}' mappings: - dag_id: upload_source_data assets: - spec: key: order_data - spec: key: activity_data - spec: key: aggregated_user_data deps: [order_data, activity_data] ### Task-level mapping[​](https://docs.dagster.io/migration/airflow-to-dagster/airflow-component-tutorial#task-level-mapping "Direct link to Task-level mapping") If you have a more specific mapping from a task within the dag to a set of assets, you can also set these mappings at the task level: type: dagster_airlift.core.components.AirflowInstanceComponentattributes: name: my_airflow auth: type: basic_auth webserver_url: '{{ env("AIRFLOW_WEBSERVER_URL") }}' username: '{{ env("AIRFLOW_USERNAME") }}' password: '{{ env("AIRFLOW_PASSWORD") }}' mappings: - dag_id: upload_source_data task_mappings: - task_id: upload_orders assets: - spec: key: order_data - task_id: upload_activity assets: - spec: key: activity_data - task_id: aggregate_user_data assets: - spec: key: aggregated_user_data deps: [order_data, activity_data] * [Setup and peering](https://docs.dagster.io/migration/airflow-to-dagster/airflow-component-tutorial#setup-and-peering) * [1\. Prepare a Dagster project](https://docs.dagster.io/migration/airflow-to-dagster/airflow-component-tutorial#1-prepare-a-dagster-project) * [2\. Scaffold an AirflowInstanceComponent](https://docs.dagster.io/migration/airflow-to-dagster/airflow-component-tutorial#2-scaffold-an-airflowinstancecomponent) * [4\. Update `defs.yaml` with Airflow configuration](https://docs.dagster.io/migration/airflow-to-dagster/airflow-component-tutorial#4-update-defsyaml-with-airflow-configuration) * [Mapping Dagster assets to Airflow tasks](https://docs.dagster.io/migration/airflow-to-dagster/airflow-component-tutorial#mapping-dagster-assets-to-airflow-tasks) * [DAG-level mapping](https://docs.dagster.io/migration/airflow-to-dagster/airflow-component-tutorial#dag-level-mapping) * [Task-level mapping](https://docs.dagster.io/migration/airflow-to-dagster/airflow-component-tutorial#task-level-mapping) --- # Migrating Airflow operators to Dagster | Dagster Docs [Skip to main content](https://docs.dagster.io/migration/airflow-to-dagster/airflow-operator-migration#__docusaurus_skipToContent_fallback) You can easily migrate usage of common Airflow operator types to Dagster. --- # Migrate from Dagster+ Serverless to Hybrid | Dagster Docs [Skip to main content](https://docs.dagster.io/migration/serverless-to-hybrid#__docusaurus_skipToContent_fallback) On this page After utilizing a Dagster+ [Serverless](https://docs.dagster.io/deployment/dagster-plus/serverless) deployment, you may decide to leverage your own infrastructure to execute your code. Transitioning to a Hybrid deployment requires only a few steps and can be done without any loss of execution history or metadata, allowing you to maintain continuity and control over your operations. warning Transitioning from Serverless to Hybrid requires some downtime, as your Dagster+ deployment won't have an agent to execute user code. Prerequisites To follow the steps in this guide, you'll need: * **Organization Admin** permissions in your Dagster+ account Step 1: Deactivate your Serverless agent[​](https://docs.dagster.io/migration/serverless-to-hybrid#step-1-deactivate-your-serverless-agent "Direct link to Step 1: Deactivate your Serverless agent") ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 1. In the Dagster+ UI, navigate to the **Deployment > Agents** page. 2. Click the drop down arrow on the right of the page and select **Switch to Hybrid**. ![PRODUCT NOTE - this arrow drop down is pretty small and easy to confuse with the one in the row for the agent](https://docs.dagster.io/assets/images/switch-agent-to-hybrid-ed23d47cf33e6877dab4a42cf6a1380d.png) It may take a few minutes for the agent to deactivate and be removed from the list of agents. Step 2: Create a Hybrid agent[​](https://docs.dagster.io/migration/serverless-to-hybrid#step-2-create-a-hybrid-agent "Direct link to Step 2: Create a Hybrid agent") --------------------------------------------------------------------------------------------------------------------------------------------------------------------- Next, you'll need to create a Hybrid agent to execute your code. Follow the setup instructions for the agent of your choice: * **[Amazon Web Services (AWS)](https://docs.dagster.io/deployment/dagster-plus/hybrid/amazon-ecs) **, which launches user code as Amazon Elastic Container Service (ECS) tasks. * **[Docker](https://docs.dagster.io/deployment/dagster-plus/hybrid/docker) **, which launches user code in Docker containers on your machine * **[Kubernetes](https://docs.dagster.io/deployment/dagster-plus/hybrid/kubernetes) **, which launches user code on a Kubernetes cluster * **[Local](https://docs.dagster.io/deployment/dagster-plus/hybrid/local) **, which launches user code in operating system subprocesses on your machine Step 3: Confirm successful setup[​](https://docs.dagster.io/migration/serverless-to-hybrid#step-3-confirm-successful-setup "Direct link to Step 3: Confirm successful setup") ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ Once you've set up a Hybrid agent, navigate to the **Deployment > Agents** page in the UI. The new agent should display in the list with a `RUNNING` status: ![Screenshot](https://docs.dagster.io/assets/images/running-agent-6afc43a57399dfc113a85e98c768fb95.png) Next steps[​](https://docs.dagster.io/migration/serverless-to-hybrid#next-steps "Direct link to Next steps") ------------------------------------------------------------------------------------------------------------- * Learn about the configuration options for [dagster.yaml](https://docs.dagster.io/deployment/oss/dagster-yaml) * [Step 1: Deactivate your Serverless agent](https://docs.dagster.io/migration/serverless-to-hybrid#step-1-deactivate-your-serverless-agent) * [Step 2: Create a Hybrid agent](https://docs.dagster.io/migration/serverless-to-hybrid#step-2-create-a-hybrid-agent) * [Step 3: Confirm successful setup](https://docs.dagster.io/migration/serverless-to-hybrid#step-3-confirm-successful-setup) * [Next steps](https://docs.dagster.io/migration/serverless-to-hybrid#next-steps) --- # Migrating an Airflow BashOperator to Dagster | Dagster Docs [Skip to main content](https://docs.dagster.io/migration/airflow-to-dagster/airflow-operator-migration/bash-operator-general#__docusaurus_skipToContent_fallback) On this page In this page, we'll explain migrating an Airflow `BashOperator` to Dagster. note If using the `BashOperator` to execute dbt commands, see "[Migrating an Airflow BashOperator (dbt) to Dagster](https://docs.dagster.io/migration/airflow-to-dagster/airflow-operator-migration/bash-operator-dbt) ". About the Airflow BashOperator[​](https://docs.dagster.io/migration/airflow-to-dagster/airflow-operator-migration/bash-operator-general#about-the-airflow-bashoperator "Direct link to About the Airflow BashOperator") ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ The Airflow `BashOperator` is a common operator used to execute bash commands as part of a data pipeline. from airflow.operators.bash import BashOperatorexecute_script = BashOperator( task_id="execute_script", bash_command="python /path/to/script.py",) The `BashOperator`'s functionality is very general since it can be used to run any bash command, and there exist richer integrations in Dagster for many common BashOperator use cases. We'll explain how 1-1 migration of the BashOperator to execute a bash command in Dagster, and how to use the `dagster-airlift` library to proxy the execution of the original task to Dagster. We'll also provide a reference for richer integrations in Dagster for common BashOperator use cases. Dagster equivalent[​](https://docs.dagster.io/migration/airflow-to-dagster/airflow-operator-migration/bash-operator-general#dagster-equivalent "Direct link to Dagster equivalent") ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ The direct Dagster equivalent to the `BashOperator` is to use the [`PipesSubprocessClient`](https://docs.dagster.io/api/dagster/pipes#dagster.PipesSubprocessClient) to execute a bash command in a subprocess. Migrating the operator[​](https://docs.dagster.io/migration/airflow-to-dagster/airflow-operator-migration/bash-operator-general#migrating-the-operator "Direct link to Migrating the operator") ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ Migrating the operator breaks down into a few steps: 1. Ensure that the resources necessary for your bash command are available to both your Airflow and Dagster deployments. 2. Write an [`asset`](https://docs.dagster.io/api/dagster/assets#dagster.asset) that executes the bash command using the [`PipesSubprocessClient`](https://docs.dagster.io/api/dagster/pipes#dagster.PipesSubprocessClient) . 3. Use `dagster-airlift` to proxy execution of the original task to Dagster. 4. (Optional) Implement a richer integration for common BashOperator use cases. ### Step 1: Ensure shared bash command access[​](https://docs.dagster.io/migration/airflow-to-dagster/airflow-operator-migration/bash-operator-general#step-1-ensure-shared-bash-command-access "Direct link to Step 1: Ensure shared bash command access") First, you'll need to ensure that the bash command you're running is available for use in both your Airflow and Dagster deployments. What this entails will vary depending on the command you're running. For example, if you're running a Python script, it's as simple as ensuring the Python script exists in a shared location accessible to both Airflow and Dagster, and all necessary env vars are set in both environments. ### Step 2: Writing an `@asset`\-decorated function[​](https://docs.dagster.io/migration/airflow-to-dagster/airflow-operator-migration/bash-operator-general#step-2-writing-an-asset-decorated-function "Direct link to step-2-writing-an-asset-decorated-function") You can write a Dagster [`asset`](https://docs.dagster.io/api/dagster/assets#dagster.asset) \-decorated function that runs your bash command. This is quite straightforward using the [`PipesSubprocessClient`](https://docs.dagster.io/api/dagster/pipes#dagster.PipesSubprocessClient) . import dagster as dg@dg.assetdef script_result(context: dg.AssetExecutionContext): return ( dg.PipesSubprocessClient() .run(context=context, command="python /path/to/script.py") .get_results() ) ### Step 3: Using dagster-airlift to proxy execution[​](https://docs.dagster.io/migration/airflow-to-dagster/airflow-operator-migration/bash-operator-general#step-3-using-dagster-airlift-to-proxy-execution "Direct link to Step 3: Using dagster-airlift to proxy execution") Finally, you can use `dagster-airlift` to proxy the execution of the original task to Dagster. For more information, see "[Migrate from Airflow to Dagster at the task level](https://docs.dagster.io/migration/airflow-to-dagster/airlift-v1/task-level-migration) ". ### Step 4: Implementing richer integrations[​](https://docs.dagster.io/migration/airflow-to-dagster/airflow-operator-migration/bash-operator-general#step-4-implementing-richer-integrations "Direct link to Step 4: Implementing richer integrations") For many of the use cases that you might be using the BashOperator for, Dagster might have better options. We'll detail some of those here. #### Running a Python script[​](https://docs.dagster.io/migration/airflow-to-dagster/airflow-operator-migration/bash-operator-general#running-a-python-script "Direct link to Running a Python script") As mentioned above, you can use the [`PipesSubprocessClient`](https://docs.dagster.io/api/dagster/pipes#dagster.PipesSubprocessClient) to run a Python script in a subprocess. But you can also modify this script to send additional information and logging back to Dagster. See the [Dagster Pipes tutorial](https://docs.dagster.io/guides/build/external-pipelines) for more information. #### Running a dbt command[​](https://docs.dagster.io/migration/airflow-to-dagster/airflow-operator-migration/bash-operator-general#running-a-dbt-command "Direct link to Running a dbt command") We have a whole guide for switching from the `BashOperator` to the `dbt` integration in Dagster. For more information, see "[Migrating an Airflow BashOperator (dbt) to Dagster](https://docs.dagster.io/migration/airflow-to-dagster/airflow-operator-migration/bash-operator-dbt) ". #### Running S3 Sync or other AWS CLI commands[​](https://docs.dagster.io/migration/airflow-to-dagster/airflow-operator-migration/bash-operator-general#running-s3-sync-or-other-aws-cli-commands "Direct link to Running S3 Sync or other AWS CLI commands") Dagster has a rich set of integrations for AWS services. For example, you can use the [`s3.S3Resource`](https://docs.dagster.io/api/libraries/dagster-aws#dagster_aws.s3.S3Resource) to interact with S3 directly. * [About the Airflow BashOperator](https://docs.dagster.io/migration/airflow-to-dagster/airflow-operator-migration/bash-operator-general#about-the-airflow-bashoperator) * [Dagster equivalent](https://docs.dagster.io/migration/airflow-to-dagster/airflow-operator-migration/bash-operator-general#dagster-equivalent) * [Migrating the operator](https://docs.dagster.io/migration/airflow-to-dagster/airflow-operator-migration/bash-operator-general#migrating-the-operator) * [Step 1: Ensure shared bash command access](https://docs.dagster.io/migration/airflow-to-dagster/airflow-operator-migration/bash-operator-general#step-1-ensure-shared-bash-command-access) * [Step 2: Writing an `@asset`\-decorated function](https://docs.dagster.io/migration/airflow-to-dagster/airflow-operator-migration/bash-operator-general#step-2-writing-an-asset-decorated-function) * [Step 3: Using dagster-airlift to proxy execution](https://docs.dagster.io/migration/airflow-to-dagster/airflow-operator-migration/bash-operator-general#step-3-using-dagster-airlift-to-proxy-execution) * [Step 4: Implementing richer integrations](https://docs.dagster.io/migration/airflow-to-dagster/airflow-operator-migration/bash-operator-general#step-4-implementing-richer-integrations) * [Running a Python script](https://docs.dagster.io/migration/airflow-to-dagster/airflow-operator-migration/bash-operator-general#running-a-python-script) * [Running a dbt command](https://docs.dagster.io/migration/airflow-to-dagster/airflow-operator-migration/bash-operator-general#running-a-dbt-command) * [Running S3 Sync or other AWS CLI commands](https://docs.dagster.io/migration/airflow-to-dagster/airflow-operator-migration/bash-operator-general#running-s3-sync-or-other-aws-cli-commands) --- # Migrating an Airflow PythonOperator to Dagster | Dagster Docs [Skip to main content](https://docs.dagster.io/migration/airflow-to-dagster/airflow-operator-migration/python-operator#__docusaurus_skipToContent_fallback) On this page In this page, we'll explain migrating an Airflow `PythonOperator` to Dagster. About the Airflow PythonOperator[​](https://docs.dagster.io/migration/airflow-to-dagster/airflow-operator-migration/python-operator#about-the-airflow-pythonoperator "Direct link to About the Airflow PythonOperator") ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ In Airflow, the `PythonOperator` runs arbitrary Python functions. For example, you might have a task that runs a function `write_to_db`, which combs a directory for files, and writes each one to a db table. from pathlib import Pathfrom typing import AnyRAW_DATA_DIR = Path("path")def contents_as_df(path: Path) -> Any: passdef upload_to_db(df: Any): pass# start_opfrom airflow.operators.python import PythonOperatordef write_to_db() -> None: for raw_file in RAW_DATA_DIR.iterdir(): df = contents_as_df(raw_file) upload_to_db(df)PythonOperator(python_callable=write_to_db, task_id="db_upload", dag=...) Dagster equivalent[​](https://docs.dagster.io/migration/airflow-to-dagster/airflow-operator-migration/python-operator#dagster-equivalent "Direct link to Dagster equivalent") ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ The Dagster equivalent is instead to construct a [`asset`](https://docs.dagster.io/api/dagster/assets#dagster.asset) or [`multi_asset`](https://docs.dagster.io/api/dagster/assets#dagster.multi_asset) \-decorated function, which materializes assets corresponding to what your python function is doing. from pathlib import Pathfrom typing import AnyRAW_DATA_DIR = Path("path")TABLE_URI = "blah"def contents_as_df(path: Path) -> Any: passdef upload_to_db(df): pass# start_assetimport dagster as dg@dg.asset(key=TABLE_URI)def write_to_db() -> None: for raw_file in RAW_DATA_DIR.iterdir(): df = contents_as_df(raw_file) upload_to_db(df) Migrating the operator[​](https://docs.dagster.io/migration/airflow-to-dagster/airflow-operator-migration/python-operator#migrating-the-operator "Direct link to Migrating the operator") ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ Migrating the operator breaks down into a few steps: 1. Make a shared library available to both Airflow and Dagster with your python function. 2. Writing an `@asset`\-decorated function which runs the python function shared between both modules. 3. Using `dagster-airlift` to proxy execution of the original task to Dagster. ### Step 1: Building a shared library[​](https://docs.dagster.io/migration/airflow-to-dagster/airflow-operator-migration/python-operator#step-1-building-a-shared-library "Direct link to Step 1: Building a shared library") We recommend a monorepo setup for migration; this allows you to keep all your code in one place and easily share code between Airflow and Dagster, without complex CI/CD coordination. First, we recommend factoring out a shared package to be available to both the Dagster runtime and the Airflow runtime which contains your python function. The process is as follows: 1. Scaffold out a new python project which will contain your shared infrastructure. 2. Ensure that the shared library is available to both your Airflow and Dagster deployments. This can be done by adding an editable requirement to your `setup.py` or `pyproject.toml` file in your Airflow/Dagster package. 3. Include the python dependencies relevant to your particular function in your new package. Write your python function in the shared package, and change your Airflow code to import the function from the shared library. To illustrate what this might look like a bit more; let's say you originally have this project structure in Airflow: airflow_repo/├── airflow_package/│ └── dags/│ └── my_dag.py # Contains your Python function With dag code that looks this: from pathlib import Pathfrom typing import AnyRAW_DATA_DIR = Path("path")def contents_as_df(path: Path) -> Any: passdef upload_to_db(df: Any): pass# start_opfrom airflow.operators.python import PythonOperatordef write_to_db() -> None: for raw_file in RAW_DATA_DIR.iterdir(): df = contents_as_df(raw_file) upload_to_db(df)PythonOperator(python_callable=write_to_db, task_id="db_upload", dag=...) You might create a new top-level package to contain the shared code: airflow_repo/├── airflow_package/│ └── dags/│ └── my_dag.py # Imports the python function from shared module.├── shared-package/│ └── shared_package/│ └── shared_module.py # Contains your Python function And then import the function from the shared package in Airflow: from pathlib import Pathfrom typing import AnyRAW_DATA_DIR = Path("path")def contents_as_df(path: Path) -> Any: passdef upload_to_db(df: Any): pass# start_opfrom airflow.operators.python import PythonOperatordef write_to_db() -> None: for raw_file in RAW_DATA_DIR.iterdir(): df = contents_as_df(raw_file) upload_to_db(df)PythonOperator(python_callable=write_to_db, task_id="db_upload", dag=...)# end_op# start_sharedfrom airflow.operators.python import PythonOperatorfrom shared_module import write_to_dbPythonOperator(python_callable=write_to_db, task_id="db_upload", dag=...) The reason we recommend using a separate `shared` package is to help ensure that there aren't dependency conflicts between Airflow and Dagster as you migrate. Airflow has very complex dependency management, and migrating to Dagster gives you an opportunity to clean up and isolate your dependencies. You can do this with a series of shared packages in the monorepo, which will eventually be isolated code locations in Dagster. ### Step 2: Writing an `@asset`\-decorated function[​](https://docs.dagster.io/migration/airflow-to-dagster/airflow-operator-migration/python-operator#step-2-writing-an-asset-decorated-function "Direct link to step-2-writing-an-asset-decorated-function") Next, you can write a Dagster [`asset`](https://docs.dagster.io/api/dagster/assets#dagster.asset) or [`multi_asset`](https://docs.dagster.io/api/dagster/assets#dagster.multi_asset) \-decorated function that runs your python function. This will generally be pretty straightforward for a `PythonOperator` migration, as you can generally just invoke the shared function into the `asset` function. # start_asset# This would be the python code living in a shared module.from shared_module import my_shared_python_callableimport dagster as dg@dg.assetdef my_shared_asset(): return my_shared_python_callable()# end_asset ### Step 3: Using `dagster-airlift` to proxy execution[​](https://docs.dagster.io/migration/airflow-to-dagster/airflow-operator-migration/python-operator#step-3-using-dagster-airlift-to-proxy-execution "Direct link to step-3-using-dagster-airlift-to-proxy-execution") Finally, you can use `dagster-airlift` to proxy the execution of the original task to Dagster. For more information, see "[Migrate from Airflow to Dagster at the task level](https://docs.dagster.io/migration/airflow-to-dagster/airlift-v1/task-level-migration) ". * [About the Airflow PythonOperator](https://docs.dagster.io/migration/airflow-to-dagster/airflow-operator-migration/python-operator#about-the-airflow-pythonoperator) * [Dagster equivalent](https://docs.dagster.io/migration/airflow-to-dagster/airflow-operator-migration/python-operator#dagster-equivalent) * [Migrating the operator](https://docs.dagster.io/migration/airflow-to-dagster/airflow-operator-migration/python-operator#migrating-the-operator) * [Step 1: Building a shared library](https://docs.dagster.io/migration/airflow-to-dagster/airflow-operator-migration/python-operator#step-1-building-a-shared-library) * [Step 2: Writing an `@asset`\-decorated function](https://docs.dagster.io/migration/airflow-to-dagster/airflow-operator-migration/python-operator#step-2-writing-an-asset-decorated-function) * [Step 3: Using `dagster-airlift` to proxy execution](https://docs.dagster.io/migration/airflow-to-dagster/airflow-operator-migration/python-operator#step-3-using-dagster-airlift-to-proxy-execution) --- # Migrating an Airflow KubernetesPodOperator to Dagster | Dagster Docs [Skip to main content](https://docs.dagster.io/migration/airflow-to-dagster/airflow-operator-migration/kubernetes-pod-operator#__docusaurus_skipToContent_fallback) On this page In this page, we'll explain migrating an Airflow `KubernetesPodOperator` to Dagster. About the Airflow KubernetesPodOperator[​](https://docs.dagster.io/migration/airflow-to-dagster/airflow-operator-migration/kubernetes-pod-operator#about-the-airflow-kubernetespodoperator "Direct link to About the Airflow KubernetesPodOperator") ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- The KubernetesPodOperator in Apache Airflow enables users to execute containerized tasks within Kubernetes pods as part of their data pipelines. from airflow.providers.cncf.kubernetes.operators.pod import KubernetesPodOperatork8s_hello_world = KubernetesPodOperator( task_id="hello_world_task", name="hello-world-pod", image="bash:latest", cmds=["bash", "-cx"], arguments=['echo "Hello World!"'],) Dagster equivalent[​](https://docs.dagster.io/migration/airflow-to-dagster/airflow-operator-migration/kubernetes-pod-operator#dagster-equivalent "Direct link to Dagster equivalent") -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- The Dagster equivalent is to use the [`PipesK8sClient`](https://docs.dagster.io/api/libraries/dagster-k8s#dagster_k8s.PipesK8sClient) to execute a task within a Kubernetes pod. import dagster_k8s as dg_k8simport dagster as dgcontainer_cfg = { "name": "hello-world-pod", "image": "bash:latest", "command": ["bash", "-cx"], "args": ['echo "Hello World!"'],}@dg.assetdef execute_hello_world_task(context: dg.AssetExecutionContext): return ( dg_k8s.PipesK8sClient() .run( context=context, base_pod_meta={"name": "hello-world-pod"}, base_pod_spec={"containers": [container_cfg]}, ) .get_results() ) Migrating the operator[​](https://docs.dagster.io/migration/airflow-to-dagster/airflow-operator-migration/kubernetes-pod-operator#migrating-the-operator "Direct link to Migrating the operator") -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Migrating the operator breaks down into a few steps: 1. Ensure that your Dagster deployment has access to the Kubernetes cluster. 2. Write an [`asset`](https://docs.dagster.io/api/dagster/assets#dagster.asset) that executes the task within a Kubernetes pod using the [`PipesK8sClient`](https://docs.dagster.io/api/libraries/dagster-k8s#dagster_k8s.PipesK8sClient) . 3. Use `dagster-airlift` to proxy execution of the original task to Dagster. ### Step 1: Ensure access to the Kubernetes cluster[​](https://docs.dagster.io/migration/airflow-to-dagster/airflow-operator-migration/kubernetes-pod-operator#step-1-ensure-access-to-the-kubernetes-cluster "Direct link to Step 1: Ensure access to the Kubernetes cluster") First, you need to ensure that your Dagster deployment has access to the Kubernetes cluster where you want to run your tasks. The [`PipesK8sClient`](https://docs.dagster.io/api/libraries/dagster-k8s#dagster_k8s.PipesK8sClient) accepts `kubeconfig` and `kubecontext`, and `env` arguments to configure the Kubernetes client. Here's an example of what this might look like when configuring the client to access an EKS cluster: import dagster_k8s as dg_k8seks_client = dg_k8s.PipesK8sClient( # The client will have automatic access to all # environment variables in the execution context. env={**AWS_CREDENTIALS, "AWS_REGION": "us-west-2"}, kubeconfig_file="path/to/kubeconfig", kube_context="my-eks-cluster",) ### Step 2: Writing an asset that executes the task within a Kubernetes pod[​](https://docs.dagster.io/migration/airflow-to-dagster/airflow-operator-migration/kubernetes-pod-operator#step-2-writing-an-asset-that-executes-the-task-within-a-kubernetes-pod "Direct link to Step 2: Writing an asset that executes the task within a Kubernetes pod") Once you have access to the Kubernetes cluster, you can write an asset that executes the task within a Kubernetes pod using the [`PipesK8sClient`](https://docs.dagster.io/api/libraries/dagster-k8s#dagster_k8s.PipesK8sClient) . In comparison to the KubernetesPodOperator, the PipesK8sClient allows you to define the pod spec directly in your Python code. In the [parameter comparison](https://docs.dagster.io/migration/airflow-to-dagster/airflow-operator-migration/kubernetes-pod-operator#parameter-comparison) section of this doc, you'll find a detailed comparison describing how to map the KubernetesPodOperator parameters to the PipesK8sClient parameters. from dagster import AssetExecutionContext, assetcontainer_cfg = { "name": "hello-world-pod", "image": "bash:latest", "command": ["bash", "-cx"], "args": ['echo "Hello World!"'],}@assetdef execute_hello_world_task(context: AssetExecutionContext): return eks_client.run( context=context, base_pod_meta={"name": "hello-world-pod"}, base_pod_spec={"containers": [container_cfg]}, ).get_results() This is just a snippet of what the PipesK8sClient can do. Take a look at our full guide on the [dagster-k8s PipesK8sClient](https://docs.dagster.io/guides/build/external-pipelines/kubernetes-pipeline) for more information. ### Step 3: Using dagster-airlift to proxy execution[​](https://docs.dagster.io/migration/airflow-to-dagster/airflow-operator-migration/kubernetes-pod-operator#step-3-using-dagster-airlift-to-proxy-execution "Direct link to Step 3: Using dagster-airlift to proxy execution") Finally, you can use `dagster-airlift` to proxy the execution of the original task to Dagster. For more information, see "[Migrate from Airflow to Dagster at the task level](https://docs.dagster.io/migration/airflow-to-dagster/airlift-v1/task-level-migration) ". Parameter comparison[​](https://docs.dagster.io/migration/airflow-to-dagster/airflow-operator-migration/kubernetes-pod-operator#parameter-comparison "Direct link to Parameter comparison") -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Here's a comparison of the parameters between the KubernetesPodOperator and the PipesK8sClient: Directly supported arguments: * in\_cluster (named load\_incluster\_config in the PipesK8sClient) * cluster\_context (named kube\_context in the PipesK8sClient) * config\_file (named kubeconfig\_file in the PipesK8sClient) Many arguments are supported indirectly via the `base_pod_spec` argument. * volumes: Volumes to be used by the Pod (key `volumes`) * affinity: Node affinity/anti-affinity rules for the Pod (key `affinity`) * node\_selector: Node selection constraints for the Pod (key `nodeSelector`) * hostnetwork: Enable host networking for the Pod (key `hostNetwork`) * dns\_config: DNS settings for the Pod (key `dnsConfig`) * dnspolicy: DNS policy for the Pod (key `dnsPolicy`) * hostname: Hostname of the Pod (key `hostname`) * subdomain: Subdomain for the Pod (key `subdomain`) * schedulername: Scheduler to be used for the Pod (key `schedulerName`) * service\_account\_name: Service account to be used by the Pod (key `serviceAccountName`) * priority\_class\_name: Priority class for the Pod (key `priorityClassName`) * security\_context: Security context for the entire Pod (key `securityContext`) * tolerations: Tolerations for the Pod (key `tolerations`) * image\_pull\_secrets: Secrets for pulling container images (key `imagePullSecrets`) * termination\_grace\_period: Grace period for Pod termination (key `terminationGracePeriodSeconds`) * active\_deadline\_seconds: Deadline for the Pod's execution (key `activeDeadlineSeconds`) * host\_aliases: Additional entries for the Pod's /etc/hosts (key `hostAliases`) * init\_containers: Initialization containers for the Pod (key `initContainers`) The following arguments are supported under the nested `containers` key of the `base_pod_spec` argument of the PipesK8sClient: * image: Docker image for the container (key 'image') * cmds: Entrypoint command for the container (key `command`) * arguments: Arguments for the entrypoint command (key `args`) * ports: List of ports to expose from the container (key `ports`) * volume\_mounts: List of volume mounts for the container (key `volumeMounts`) * env\_vars: Environment variables for the container (key `env`) * env\_from: List of sources to populate environment variables (key `envFrom`) * image\_pull\_policy: Policy for pulling the container image (key `imagePullPolicy`) * container\_resources: Resource requirements for the container (key `resources`) * container\_security\_context: Security context for the container (key `securityContext`) * termination\_message\_policy: Policy for the termination message (key `terminationMessagePolicy`) For a full list, see the [kubernetes container spec documentation](https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.26/#container-v1-core) . The following arguments are supported under the `base_pod_meta` argument, which configures the metadata of the pod: * name: `name` * namespace: `namespace` * labels: `labels` * annotations: `annotations` For a full list, see the [kubernetes objectmeta spec documentation](https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.26/#objectmeta-v1-meta) . * [About the Airflow KubernetesPodOperator](https://docs.dagster.io/migration/airflow-to-dagster/airflow-operator-migration/kubernetes-pod-operator#about-the-airflow-kubernetespodoperator) * [Dagster equivalent](https://docs.dagster.io/migration/airflow-to-dagster/airflow-operator-migration/kubernetes-pod-operator#dagster-equivalent) * [Migrating the operator](https://docs.dagster.io/migration/airflow-to-dagster/airflow-operator-migration/kubernetes-pod-operator#migrating-the-operator) * [Step 1: Ensure access to the Kubernetes cluster](https://docs.dagster.io/migration/airflow-to-dagster/airflow-operator-migration/kubernetes-pod-operator#step-1-ensure-access-to-the-kubernetes-cluster) * [Step 2: Writing an asset that executes the task within a Kubernetes pod](https://docs.dagster.io/migration/airflow-to-dagster/airflow-operator-migration/kubernetes-pod-operator#step-2-writing-an-asset-that-executes-the-task-within-a-kubernetes-pod) * [Step 3: Using dagster-airlift to proxy execution](https://docs.dagster.io/migration/airflow-to-dagster/airflow-operator-migration/kubernetes-pod-operator#step-3-using-dagster-airlift-to-proxy-execution) * [Parameter comparison](https://docs.dagster.io/migration/airflow-to-dagster/airflow-operator-migration/kubernetes-pod-operator#parameter-comparison) --- # Migrating an Airflow BashOperator (dbt) to Dagster | Dagster Docs [Skip to main content](https://docs.dagster.io/migration/airflow-to-dagster/airflow-operator-migration/bash-operator-dbt#__docusaurus_skipToContent_fallback) On this page In this page, we'll explain migrating an Airflow `BashOperator` that runs a `dbt` command to Dagster. About the Airflow BashOperator[​](https://docs.dagster.io/migration/airflow-to-dagster/airflow-operator-migration/bash-operator-dbt#about-the-airflow-bashoperator "Direct link to About the Airflow BashOperator") -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- In Airflow, you might have a `BashOperator` that runs a `dbt` command. For example, you might have a task that runs `dbt run` to build your dbt models. from airflow.operators.bash import BashOperatorrun_dbt_model = BashOperator(task_id="build_dbt_models", bash_command="dbt run") Dagster equivalent[​](https://docs.dagster.io/migration/airflow-to-dagster/airflow-operator-migration/bash-operator-dbt#dagster-equivalent "Direct link to Dagster equivalent") -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- The Dagster equivalent is to instead use the `dagster-dbt` library to run commands against your dbt project. Here would be the equivalent code in Dagster: import dagster_dbt as dg_dbtimport dagster as dgproject = dg_dbt.DbtProject(project_dir="path/to/dbt_project")@dg_dbt.dbt_assets(manifest=project.manifest_path)def my_dbt_assets(context: dg.AssetExecutionContext, dbt: dg_dbt.DbtCliResource): yield from dbt.cli(["run"], context=context).stream() Migrating the operator[​](https://docs.dagster.io/migration/airflow-to-dagster/airflow-operator-migration/bash-operator-dbt#migrating-the-operator "Direct link to Migrating the operator") -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Migrating the operator breaks down into a few steps: 1. Making the dbt project available to both your Airflow and Dagster deployments. 2. Writing a @dbt\_asset-decorated function which runs your dbt commands. 3. Using `dagster-airlift` to proxy execution of the original task to Dagster. ### Step 1: Making the dbt project available & building manifest[​](https://docs.dagster.io/migration/airflow-to-dagster/airflow-operator-migration/bash-operator-dbt#step-1-making-the-dbt-project-available--building-manifest "Direct link to Step 1: Making the dbt project available & building manifest") First, you'll need to make the dbt project available to the Dagster runtime and build the manifest. * If you're building your Dagster deployment in a monorepo alongside your dbt and Airflow projects, you can follow this guide: [Monorepo setup](https://docs.dagster.io/integrations/libraries/dbt/reference#deploying-a-dagster-project-with-a-dbt-project) . * If you're deploying within a separate repository, you can follow this guide: [Separate repository setup](https://docs.dagster.io/integrations/libraries/dbt/reference#deploying-a-dbt-project-from-a-separate-git-repository) . ### Step 2: Writing a @dbt\_asset-decorated function[​](https://docs.dagster.io/migration/airflow-to-dagster/airflow-operator-migration/bash-operator-dbt#step-2-writing-a-dbt_asset-decorated-function "Direct link to Step 2: Writing a @dbt_asset-decorated function") Once your dbt project is available, you can write a function that runs your dbt commands using the [`dbt_assets`](https://docs.dagster.io/api/libraries/dagster-dbt#dagster_dbt.dbt_assets) decorator and [`DbtCliResource`](https://docs.dagster.io/api/libraries/dagster-dbt#dagster_dbt.DbtCliResource) . Most dbt CLI commands and flags are supported - to learn more about using `@dbt_assets`, check out the [dagster-dbt quickstart](https://docs.dagster.io/integrations/libraries/dbt/quickstart) and [reference](https://docs.dagster.io/integrations/libraries/dbt/reference) . ### Step 3: Using dagster-airlift to proxy execution[​](https://docs.dagster.io/migration/airflow-to-dagster/airflow-operator-migration/bash-operator-dbt#step-3-using-dagster-airlift-to-proxy-execution "Direct link to Step 3: Using dagster-airlift to proxy execution") Finally, you can use `dagster-airlift` to proxy the execution of the original task to Dagster. For more information, see "[Migrate from Airflow to Dagster at the task level](https://docs.dagster.io/migration/airflow-to-dagster/airlift-v1/task-level-migration) ". * [About the Airflow BashOperator](https://docs.dagster.io/migration/airflow-to-dagster/airflow-operator-migration/bash-operator-dbt#about-the-airflow-bashoperator) * [Dagster equivalent](https://docs.dagster.io/migration/airflow-to-dagster/airflow-operator-migration/bash-operator-dbt#dagster-equivalent) * [Migrating the operator](https://docs.dagster.io/migration/airflow-to-dagster/airflow-operator-migration/bash-operator-dbt#migrating-the-operator) * [Step 1: Making the dbt project available & building manifest](https://docs.dagster.io/migration/airflow-to-dagster/airflow-operator-migration/bash-operator-dbt#step-1-making-the-dbt-project-available--building-manifest) * [Step 2: Writing a @dbt\_asset-decorated function](https://docs.dagster.io/migration/airflow-to-dagster/airflow-operator-migration/bash-operator-dbt#step-2-writing-a-dbt_asset-decorated-function) * [Step 3: Using dagster-airlift to proxy execution](https://docs.dagster.io/migration/airflow-to-dagster/airflow-operator-migration/bash-operator-dbt#step-3-using-dagster-airlift-to-proxy-execution) --- # Setup | Dagster Docs [Skip to main content](https://docs.dagster.io/migration/airflow-to-dagster/airlift-v1/federation/setup#__docusaurus_skipToContent_fallback) On this page note If you are just getting started with Airlift, we recommend using the new [Airlift component](https://docs.dagster.io/migration/airflow-to-dagster/airflow-component-tutorial) . In this step, we'll: * Install the example code and review the project structure * Set up a local environment * Ensure we can run Airflow locally. Install example code[​](https://docs.dagster.io/migration/airflow-to-dagster/airlift-v1/federation/setup#install-example-code "Direct link to Install example code") --------------------------------------------------------------------------------------------------------------------------------------------------------------------- First, create a fresh virtual environment using `uv` and activate it: pip install uvuv venvsource .venv/bin/activate Next, install Dagster and verify that the `dagster` CLI is available: uv pip install dagsterdagster --version Finally, install the tutorial example code: dagster project from-example --name airlift-federation-tutorial --example airlift-federation-tutorial ### Project structure[​](https://docs.dagster.io/migration/airflow-to-dagster/airlift-v1/federation/setup#project-structure "Direct link to Project structure") This tutorial example contains the following files and directories: airlift_federation_tutorial├── constants.py: Contains constant values used throughout both Airflow and Dagster├── dagster_defs: Contains Dagster definitions│ ├── definitions.py: Empty starter file for following along with the tutorial│ └── stages: Contains reference implementations for each stage of the migration process.├── metrics_airflow_dags: Contains the Airflow DAGs for the "downstream" Airflow instance└── warehouse_airflow_dags: Contains the Airflow DAGs for the "upstream" Airflow instance Run Airflow instances locally[​](https://docs.dagster.io/migration/airflow-to-dagster/airlift-v1/federation/setup#run-airflow-instances-locally "Direct link to Run Airflow instances locally") ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ This tutorial involves running two local Airflow instances, which you can do by following commands from the root of the `airlift-federation-tutorial` directory. First, install the required Python packages: make airflow_install Next, scaffold the two Airflow instances required for this tutorial: make airflow_setup Finally, run the two Airflow instances with environment variables set. In one shell, run: make warehouse_airflow_run In a separate shell, run: make metrics_airflow_run This will run two Airflow Web UIs, one for each Airflow instance. You should now be able to access the warehouse Airflow UI at `http://localhost:8081`, with the default username and password set to `admin`. You should be able to see the `load_customers` DAG in the Airflow UI: ![load_customers DAG](https://docs.dagster.io/assets/images/load_customers-23b20cbf736a6315a35d65d949256e34.png) Similarly, you should be able to access the metrics Airflow UI at `http://localhost:8082`, with the default username and password set to `admin`. You should be able to see the `customer_metrics` DAG in the Airflow UI: ![customer_metrics DAG](https://docs.dagster.io/assets/images/customer_metrics-2eebded6aecb23797d3ebc4f6c84226b.png) Next steps[​](https://docs.dagster.io/migration/airflow-to-dagster/airlift-v1/federation/setup#next-steps "Direct link to Next steps") --------------------------------------------------------------------------------------------------------------------------------------- In the next step, "[Observe multiple Airflow instances from Dagster](https://docs.dagster.io/migration/airflow-to-dagster/airlift-v1/federation/observe) ", we'll add asset representations of our DAGs and set up lineage across both Airflow instances. * [Install example code](https://docs.dagster.io/migration/airflow-to-dagster/airlift-v1/federation/setup#install-example-code) * [Project structure](https://docs.dagster.io/migration/airflow-to-dagster/airlift-v1/federation/setup#project-structure) * [Run Airflow instances locally](https://docs.dagster.io/migration/airflow-to-dagster/airlift-v1/federation/setup#run-airflow-instances-locally) * [Next steps](https://docs.dagster.io/migration/airflow-to-dagster/airlift-v1/federation/setup#next-steps) --- # Observe the Airflow DAG | Dagster Docs [Skip to main content](https://docs.dagster.io/migration/airflow-to-dagster/airlift-v1/dag-level-migration/observe#__docusaurus_skipToContent_fallback) On this page note If you are just getting started with Airlift, we recommend using the new [Airlift component](https://docs.dagster.io/migration/airflow-to-dagster/airflow-component-tutorial) . When migrating an entire DAG at once, you must create assets that map to the entire DAG. To do this, you can use [`assets_with_dag_mappings`](https://docs.dagster.io/api/libraries/dagster-airlift#dagster_airlift.core.assets_with_dag_mappings) , which ensures that each mapped asset receives a materialization when the entire DAG completes. For our `rebuild_customers_list` DAG, let's take a look at what the new observation code looks like: import osfrom pathlib import Pathfrom dagster import AssetExecutionContext, AssetSpec, Definitionsfrom dagster_airlift.core import ( AirflowBasicAuthBackend, AirflowInstance, assets_with_dag_mappings, build_defs_from_airflow_instance,)from dagster_dbt import DbtCliResource, DbtProject, dbt_assetsdef dbt_project_path() -> Path: env_val = os.getenv("TUTORIAL_DBT_PROJECT_DIR") assert env_val, "TUTORIAL_DBT_PROJECT_DIR must be set" return Path(env_val)@dbt_assets( manifest=dbt_project_path() / "target" / "manifest.json", project=DbtProject(dbt_project_path()),)def dbt_project_assets(context: AssetExecutionContext, dbt: DbtCliResource): yield from dbt.cli(["build"], context=context).stream()# Instead of mapping assets to individual tasks, we map them to the entire DAG.mapped_assets = assets_with_dag_mappings( dag_mappings={ "rebuild_customers_list": [ AssetSpec(key=["raw_data", "raw_customers"]), dbt_project_assets, AssetSpec(key="customers_csv", deps=["customers"]), ], },)defs = build_defs_from_airflow_instance( airflow_instance=AirflowInstance( auth_backend=AirflowBasicAuthBackend( webserver_url="http://localhost:8080", username="admin", password="admin", ), name="airflow_instance_one", ), defs=Definitions( assets=mapped_assets, resources={"dbt": DbtCliResource(project_dir=dbt_project_path())}, ),) Now, instead of getting a materialization when a particular task completes, each mapped asset will receive a materialization when the entire DAG completes. Next steps[​](https://docs.dagster.io/migration/airflow-to-dagster/airlift-v1/dag-level-migration/observe#next-steps "Direct link to Next steps") -------------------------------------------------------------------------------------------------------------------------------------------------- In the next step, "[Migrate DAG-mapped assets](https://docs.dagster.io/migration/airflow-to-dagster/airlift-v1/dag-level-migration/migrate) ", we will proxy execution for the entire Airflow DAG in Dagster. * [Next steps](https://docs.dagster.io/migration/airflow-to-dagster/airlift-v1/dag-level-migration/observe#next-steps) --- # Federate execution | Dagster Docs [Skip to main content](https://docs.dagster.io/migration/airflow-to-dagster/airlift-v1/federation/federate-execution#__docusaurus_skipToContent_fallback) On this page note If you are just getting started with Airlift, we recommend using the new [Airlift component](https://docs.dagster.io/migration/airflow-to-dagster/airflow-component-tutorial) . In the [previous step](https://docs.dagster.io/migration/airflow-to-dagster/airlift-v1/federation/observe) , we created Dagster asset representations of Airflow DAGs in order to observe the Airflow instances from Dagster, and set up cross-instance lineage for the DAGs. In this step, we'll federate the execution of the DAGs across both Airflow instances by using Dagster's [Declarative Automation](https://docs.dagster.io/guides/automate/declarative-automation) framework. Make the `customer_metrics` DAG executable[​](https://docs.dagster.io/migration/airflow-to-dagster/airlift-v1/federation/federate-execution#make-the-customer_metrics-dag-executable "Direct link to make-the-customer_metrics-dag-executable") ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ To federate execution of the `customer_metrics` Airflow DAG, you first need to make its corresponding asset executable within Dagster. To do this, you can use the [`@multi_asset`](https://docs.dagster.io/api/dagster/assets#dagster.multi_asset) decorator to define how the `customer_metrics` asset should be executed. You'll use the `AirflowInstance` defined earlier to trigger a run of the `customer_metrics` DAG. If the run completes successfully, the asset will be materialized. If the run fails, an exception will be raised: @dg.multi_asset(specs=[customer_metrics_dag_asset])def run_customer_metrics() -> dg.MaterializeResult: run_id = metrics_airflow_instance.trigger_dag("customer_metrics") metrics_airflow_instance.wait_for_run_completion("customer_metrics", run_id) if metrics_airflow_instance.get_run_state("customer_metrics", run_id) == "success": return dg.MaterializeResult(asset_key=customer_metrics_dag_asset.key) else: raise Exception("Dag run failed.") Next, replace the `customer_metrics_dag_asset` in the [`Definitions`](https://docs.dagster.io/api/dagster/definitions#dagster.Definitions) object with the `run_customer_metrics` function: defs = dg.Definitions( assets=[load_customers_dag_asset, run_customer_metrics], sensors=[warehouse_sensor, metrics_sensor],) In the Dagster UI, you should see that the `customer_metrics` asset can now be materialized. Federate execution[​](https://docs.dagster.io/migration/airflow-to-dagster/airlift-v1/federation/federate-execution#federate-execution "Direct link to Federate execution") ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Ultimately, we would like to trigger a run of `customer_metrics` whenever `load_customers` completes successfully. We're already retrieving a materialization when `load_customers` completes, so we can use this to trigger a run of `customer_metrics` by using [Declarative Automation](https://docs.dagster.io/guides/automate/declarative-automation) . First, add an [`AutomationCondition.eager()`](https://docs.dagster.io/api/dagster/assets#dagster.AutomationCondition.eager) to the `customer_metrics_dag_asset`. This will tell Dagster to run the `run_customer_metrics` function whenever the `load_customers` asset is materialized: customer_metrics_dag_asset = customer_metrics_dag_asset.replace_attributes( automation_condition=dg.AutomationCondition.eager(),) Next, create an [`AutomationConditionSensorDefinition`](https://docs.dagster.io/api/dagster/assets#dagster.AutomationConditionSensorDefinition) to set up Declarative Automation: automation_sensor = dg.AutomationConditionSensorDefinition( name="automation_sensor", target="*", default_status=dg.DefaultSensorStatus.RUNNING, minimum_interval_seconds=1,) Add this sensor to the [`Definitions`](https://docs.dagster.io/api/dagster/definitions#dagster.Definitions) object: defs = dg.Definitions( assets=[load_customers_dag_asset, run_customer_metrics], sensors=[warehouse_sensor, metrics_sensor, automation_sensor],) Now the `run_customer_metrics` function will be executed whenever the `load_customers` asset is materialized. You can test this by triggering a run of the `load_customers` DAG in Airflow. When the run completes, you should see a materialization of the `customer_metrics` asset start in the Dagster UI, and eventually a run of the `customer_metrics` DAG in the metrics Airflow instance. Complete code To see what the code should look like after you have completed all the steps above, check out the [example in GitHub](https://github.com/dagster-io/dagster/blob/master/examples/airlift-federation-tutorial/airlift_federation_tutorial/dagster_defs/stages/executable_and_da.py) . Conclusion[​](https://docs.dagster.io/migration/airflow-to-dagster/airlift-v1/federation/federate-execution#conclusion "Direct link to Conclusion") ---------------------------------------------------------------------------------------------------------------------------------------------------- That concludes the tutorial! If you followed all the steps, you should have successfully federated the execution of two DAGs across two Airflow instances using Dagster's Declarative Automation system and set up cross-instance lineage for the DAGs. You can now observe the lineage and execution of both DAGs in the Dagster UI. * [Make the `customer_metrics` DAG executable](https://docs.dagster.io/migration/airflow-to-dagster/airlift-v1/federation/federate-execution#make-the-customer_metrics-dag-executable) * [Federate execution](https://docs.dagster.io/migration/airflow-to-dagster/airlift-v1/federation/federate-execution#federate-execution) * [Conclusion](https://docs.dagster.io/migration/airflow-to-dagster/airlift-v1/federation/federate-execution#conclusion) --- # Setup | Dagster Docs [Skip to main content](https://docs.dagster.io/migration/airflow-to-dagster/airlift-v1/dag-level-migration/setup#__docusaurus_skipToContent_fallback) On this page note If you are just getting started with Airlift, we recommend using the new [Airlift component](https://docs.dagster.io/migration/airflow-to-dagster/airflow-component-tutorial) . In order the complete this tutorial, you'll need to: * Create a virtual environment * Install Dagster and the tutorial example code * Set up a local Airflow instance Create a virtual environment[​](https://docs.dagster.io/migration/airflow-to-dagster/airlift-v1/dag-level-migration/setup#create-a-virtual-environment "Direct link to Create a virtual environment") ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ First, create a fresh virtual environment using `uv` and activate it: pip install uvuv venvsource .venv/bin/activate Install Dagster and the tutorial example code[​](https://docs.dagster.io/migration/airflow-to-dagster/airlift-v1/dag-level-migration/setup#install-dagster-and-the-tutorial-example-code "Direct link to Install Dagster and the tutorial example code") --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Next, install Dagster and verify that the `dagster` CLI is available: uv pip install dagsterdagster --version Finally, install the tutorial example code: dagster project from-example --name airlift-migration-tutorial --example airlift-migration-tutorial ### Project structure[​](https://docs.dagster.io/migration/airflow-to-dagster/airlift-v1/dag-level-migration/setup#project-structure "Direct link to Project structure") The tutorial example contains the following files and directories: tutorial_example├── shared: Contains shared Python & SQL code used Airflow and proxied Dagster code│├── dagster_defs: Contains Dagster definitions│ ├── stages: Contains reference implementations of each stage of the migration process│ ├── definitions.py: Empty starter file for following along with the tutorial│├── airflow_dags: Contains the Airflow DAG and associated files│ ├── proxied_state: Contains migration state files for each DAG, see migration step below│ ├── dags.py: The Airflow DAG definition Set up a local Airflow instance[​](https://docs.dagster.io/migration/airflow-to-dagster/airlift-v1/dag-level-migration/setup#set-up-a-local-airflow-instance "Direct link to Set up a local Airflow instance") --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- This tutorial involves running a local Airflow instance, which you can do by following commands from the root of the `airlift-migration-tutorial` directory. First, install the required Python packages: make airflow_install Next, scaffold the Airflow instance and initialize the `dbt` project: make airflow_setup Finally, run the Airflow instance with environment variables set: make airflow_run This will run the Airflow Web UI in a shell. You should now be able to access the Airflow UI at `http://localhost:8080`, with the default username and password set to `admin`. You should be able to see the `rebuild_customers_list` DAG in the Airflow UI, made up of three tasks: `load_raw_customers`, `run_dbt_model`, and `export_customers`: ![Rebuild customers list DAG](https://docs.dagster.io/assets/images/rebuild_customers_dag-76748681991c12a410926b6d665902d4.png) Next steps[​](https://docs.dagster.io/migration/airflow-to-dagster/airlift-v1/dag-level-migration/setup#next-steps "Direct link to Next steps") ------------------------------------------------------------------------------------------------------------------------------------------------ In the next step, "[Peer your Airflow instance with a Dagster code location](https://docs.dagster.io/migration/airflow-to-dagster/airlift-v1/dag-level-migration/peer) ", we'll peer our Dagster installation with our Airflow instance. * [Create a virtual environment](https://docs.dagster.io/migration/airflow-to-dagster/airlift-v1/dag-level-migration/setup#create-a-virtual-environment) * [Install Dagster and the tutorial example code](https://docs.dagster.io/migration/airflow-to-dagster/airlift-v1/dag-level-migration/setup#install-dagster-and-the-tutorial-example-code) * [Project structure](https://docs.dagster.io/migration/airflow-to-dagster/airlift-v1/dag-level-migration/setup#project-structure) * [Set up a local Airflow instance](https://docs.dagster.io/migration/airflow-to-dagster/airlift-v1/dag-level-migration/setup#set-up-a-local-airflow-instance) * [Next steps](https://docs.dagster.io/migration/airflow-to-dagster/airlift-v1/dag-level-migration/setup#next-steps) --- # Federate execution across Airflow instances with Dagster | Dagster Docs [Skip to main content](https://docs.dagster.io/migration/airflow-to-dagster/airlift-v1/federation#__docusaurus_skipToContent_fallback) On this page note If you are just getting started with Airlift, we recommend using the new [Airlift component](https://docs.dagster.io/migration/airflow-to-dagster/airflow-component-tutorial) . You can use `dagster-airlift` to observe DAGs from multiple Airflow instances and federate execution between them using Dagster as a centralized control plane, all without changing your Airflow code. Tutorial overview[​](https://docs.dagster.io/migration/airflow-to-dagster/airlift-v1/federation#tutorial-overview "Direct link to Tutorial overview") ------------------------------------------------------------------------------------------------------------------------------------------------------ In this tutorial, a data platform team is tasked with managing the following Airflow setup: * An Airflow instance called `warehouse`, run by another team, that contains a DAG called `warehouse.load_customers` that loads customer data into the data warehouse. * An Airflow instance called `metrics`, run by the data platform team, that contains a DAG called `metrics.customer_metrics` that computes metrics on top of the customer data. The data platform team wants to update this setup to only rebuild the `metrics.customer_metrics` DAG when the `warehouse.load_customers` DAG has new data. They can't observe or control this cross-instance dependency in the current setup, so they decide to use `dagster-airlift`. We'll walk you through an example of using `dagster-airlift` to observe the `warehouse` and `metrics` Airflow instances described above, and set up a federated execution controlled by Dagster that only triggers the `metrics.customer_metrics` DAG when the `warehouse.load_customers` DAG has new data, all without requiring any changes to Airflow code. Next steps[​](https://docs.dagster.io/migration/airflow-to-dagster/airlift-v1/federation#next-steps "Direct link to Next steps") --------------------------------------------------------------------------------------------------------------------------------- To get started with this tutorial, follow the [setup steps](https://docs.dagster.io/migration/airflow-to-dagster/airlift-v1/federation/setup) to install the example code, set up a local environment, and run two instances of Airflow locally. * [Tutorial overview](https://docs.dagster.io/migration/airflow-to-dagster/airlift-v1/federation#tutorial-overview) * [Next steps](https://docs.dagster.io/migration/airflow-to-dagster/airlift-v1/federation#next-steps) --- # Airflow to Dagster migration reference | Dagster Docs [Skip to main content](https://docs.dagster.io/migration/airflow-to-dagster/airlift-v1/migration-reference#__docusaurus_skipToContent_fallback) On this page note If you are just getting started with Airlift, we recommend using the new [Airlift component](https://docs.dagster.io/migration/airflow-to-dagster/airflow-component-tutorial) . `dagster-airlift` is a toolkit for observing and migrating Airflow DAGs within Dagster. This reference page provides additional information for working with `dagster-airlift` that is not provided within the migration guides. * [Supporting custom authorization](https://docs.dagster.io/migration/airflow-to-dagster/airlift-v1/migration-reference#supporting-custom-authorization) * [Dagster Plus Authorization](https://docs.dagster.io/migration/airflow-to-dagster/airlift-v1/migration-reference#dagster-authorization) * [Dealing with changing Airflow](https://docs.dagster.io/migration/airflow-to-dagster/airlift-v1/migration-reference#dealing-with-changing-airflow) * [Automating changes to code locations](https://docs.dagster.io/migration/airflow-to-dagster/airlift-v1/migration-reference#automating-changes-to-code-locations) * [Peering to multiple Airflow instances](https://docs.dagster.io/migration/airflow-to-dagster/airlift-v1/migration-reference#peering-to-multiple-airflow-instances) * [Customizing DAG proxying operator](https://docs.dagster.io/migration/airflow-to-dagster/airlift-v1/migration-reference#customizing-dag-proxying-operator) Supporting custom authorization[​](https://docs.dagster.io/migration/airflow-to-dagster/airlift-v1/migration-reference#supporting-custom-authorization "Direct link to Supporting custom authorization") --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- If your Dagster deployment lives behind a custom auth backend, you can customize the Airflow-to-Dagster proxying behavior to authenticate to your backend. `proxying_to_dagster` can take a parameter `dagster_operator_klass`, which allows you to define a custom `BaseProxyTasktoDagsterOperator` class. This allows you to override how a session is created. Let's say for example, your Dagster installation requires an access key to be set whenever a request is made, and that access key is set in an Airflow `Variable` called `my_api_key`. We can create a custom `BaseProxyTasktoDagsterOperator` subclass which will retrieve that variable value and set it on the session, so that any requests to Dagster's graphql API will be made using that api key. from pathlib import Pathimport requestsfrom airflow import DAGfrom airflow.utils.context import Contextfrom dagster_airlift.in_airflow import BaseProxyTaskToDagsterOperator, proxying_to_dagsterfrom dagster_airlift.in_airflow.proxied_state import load_proxied_state_from_yamlclass CustomProxyToDagsterOperator(BaseProxyTaskToDagsterOperator): def get_dagster_session(self, context: Context) -> requests.Session: # pyright: ignore[reportIncompatibleMethodOverride] if "var" not in context: raise ValueError("No variables found in context") api_key = context["var"]["value"].get("my_api_key") session = requests.Session() session.headers.update({"Authorization": f"Bearer {api_key}"}) return session def get_dagster_url(self, context: Context) -> str: # pyright: ignore[reportIncompatibleMethodOverride] return "https://dagster.example.com/"dag = DAG( dag_id="custom_proxy_example",)# At the end of your dag fileproxying_to_dagster( global_vars=globals(), proxied_state=load_proxied_state_from_yaml(Path(__file__).parent / "proxied_state"), build_from_task_fn=CustomProxyToDagsterOperator.build_from_task,) Dagster+ authorization[​](https://docs.dagster.io/migration/airflow-to-dagster/airlift-v1/migration-reference#dagster-authorization "Direct link to Dagster+ authorization") ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------- You can use a custom proxy operator to establish a connection to a Dagster plus deployment. The below example proxies to Dagster Plus using organization name, deployment name, and user token set as Airflow Variables. To set a Dagster+ user token, see "[Managing user tokens in Dagster+](https://docs.dagster.io/deployment/dagster-plus/management/tokens/user-tokens) ". import requestsfrom airflow.utils.context import Contextfrom dagster_airlift.in_airflow import BaseProxyTaskToDagsterOperatorclass DagsterCloudProxyOperator(BaseProxyTaskToDagsterOperator): def get_variable(self, context: Context, var_name: str) -> str: if "var" not in context: raise ValueError("No variables found in context") return context["var"]["value"][var_name] def get_dagster_session(self, context: Context) -> requests.Session: # pyright: ignore[reportIncompatibleMethodOverride] dagster_cloud_user_token = self.get_variable(context, "dagster_cloud_user_token") session = requests.Session() session.headers.update({"Dagster-Cloud-Api-Token": dagster_cloud_user_token}) return session def get_dagster_url(self, context: Context) -> str: # pyright: ignore[reportIncompatibleMethodOverride] org_name = self.get_variable(context, "dagster_plus_organization_name") deployment_name = self.get_variable(context, "dagster_plus_deployment_name") return f"https://{org_name}.dagster.plus/{deployment_name}" Dealing with changing Airflow[​](https://docs.dagster.io/migration/airflow-to-dagster/airlift-v1/migration-reference#dealing-with-changing-airflow "Direct link to Dealing with changing Airflow") --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- In order to make spin-up more efficient, `dagster-airlift` caches the state of the Airflow instance in the dagster database, so that repeat fetches of the code location don't require additional calls to Airflow's rest API. However, this means that the Dagster definitions can potentially fall out of sync with Airflow. Here are a few different ways this can manifest: * A new Airflow DAG is added. The lineage information does not show up for this dag, and materializations are not recorded. * A DAG is removed. The polling sensor begins failing, because there exist assets which expect that DAG to exist. * The task dependency structure within a DAG changes. This may result in `unsynced` statuses in Dagster, or missing materializations. This is not an exhaustive list of problems, but most of the time the tell is that materializations are missing, or assets are missing. When you find yourself in this state, you can force `dagster-airlift` to reload Airflow state by reloading the code location. To do this, go to the `Deployment` tab on the top nav, and click `Redeploy` on the code location relevant to your asset. After some time, the code location should be reloaded with refreshed state from Airflow. Automating changes to code locations[​](https://docs.dagster.io/migration/airflow-to-dagster/airlift-v1/migration-reference#automating-changes-to-code-locations "Direct link to Automating changes to code locations") ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ If changes to your Airflow instance are controlled by a CI/CD process, you can use the Dagster GraphQL client to add a step to automatically trigger a redeploy of the relevant code location. To learn more, see the [Dagster GraphQL client docs](https://docs.dagster.io/guides/operate/graphql/graphql-client) . Peering to multiple Airflow instances[​](https://docs.dagster.io/migration/airflow-to-dagster/airlift-v1/migration-reference#peering-to-multiple-airflow-instances "Direct link to Peering to multiple Airflow instances") --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Airlift supports peering to multiple Airflow instances, as you can invoke `build_defs_from_airflow_instance` multiple times and combine them with `Definitions.merge`: from dagster import Definitionsfrom dagster_airlift.core import AirflowInstance, build_defs_from_airflow_instancedefs = Definitions.merge( build_defs_from_airflow_instance( airflow_instance=AirflowInstance( auth_backend=BasicAuthBackend( webserver_url="http://yourcompany.com/instance_one", username="admin", password="admin", ), name="airflow_instance_one", ) ), build_defs_from_airflow_instance( airflow_instance=AirflowInstance( auth_backend=BasicAuthBackend( webserver_url="http://yourcompany.com/instance_two", username="admin", password="admin", ), name="airflow_instance_two", ) ),) Customizing DAG proxying operator[​](https://docs.dagster.io/migration/airflow-to-dagster/airlift-v1/migration-reference#customizing-dag-proxying-operator "Direct link to Customizing DAG proxying operator") --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Similar to how we can customize the operator we construct on a per-DAG basis, we can customize the operator we construct on a per-DAG basis. We can use the `build_from_dag_fn` argument of `proxying_to_dagster` to provide a custom operator in place of the default. For example, in the following example we can see that the operator is customized to provide an authorization header which authenticates Dagster. from pathlib import Pathimport requestsfrom airflow import DAGfrom airflow.utils.context import Contextfrom dagster_airlift.in_airflow import BaseProxyDAGToDagsterOperator, proxying_to_dagsterfrom dagster_airlift.in_airflow.proxied_state import load_proxied_state_from_yamlclass CustomProxyToDagsterOperator(BaseProxyDAGToDagsterOperator): def get_dagster_session(self, context: Context) -> requests.Session: # pyright: ignore[reportIncompatibleMethodOverride] if "var" not in context: raise ValueError("No variables found in context") api_key = context["var"]["value"].get("my_api_key") session = requests.Session() session.headers.update({"Authorization": f"Bearer {api_key}"}) return session def get_dagster_url(self, context: Context) -> str: # pyright: ignore[reportIncompatibleMethodOverride] return "https://dagster.example.com/" # This method controls how the operator is built from the dag. @classmethod def build_from_dag(cls, dag: DAG): return CustomProxyToDagsterOperator(dag=dag, task_id="OVERRIDDEN")dag = DAG( dag_id="custom_dag_level_proxy_example",)# At the end of your dag fileproxying_to_dagster( global_vars=globals(), proxied_state=load_proxied_state_from_yaml(Path(__file__).parent / "proxied_state"), build_from_dag_fn=CustomProxyToDagsterOperator.build_from_dag,) `BaseProxyDAGToDagsterOperator` has three abstract methods which must be implemented: * `get_dagster_session`, which controls the creation of a valid session to access the Dagster graphql API. * `get_dagster_url`, which retrieves the domain at which the dagster webserver lives. * `build_from_dag`, which controls how the proxying task is constructed from the provided DAG. * [Supporting custom authorization](https://docs.dagster.io/migration/airflow-to-dagster/airlift-v1/migration-reference#supporting-custom-authorization) * [Dagster+ authorization](https://docs.dagster.io/migration/airflow-to-dagster/airlift-v1/migration-reference#dagster-authorization) * [Dealing with changing Airflow](https://docs.dagster.io/migration/airflow-to-dagster/airlift-v1/migration-reference#dealing-with-changing-airflow) * [Automating changes to code locations](https://docs.dagster.io/migration/airflow-to-dagster/airlift-v1/migration-reference#automating-changes-to-code-locations) * [Peering to multiple Airflow instances](https://docs.dagster.io/migration/airflow-to-dagster/airlift-v1/migration-reference#peering-to-multiple-airflow-instances) * [Customizing DAG proxying operator](https://docs.dagster.io/migration/airflow-to-dagster/airlift-v1/migration-reference#customizing-dag-proxying-operator) --- # Migrate DAG-mapped assets | Dagster Docs [Skip to main content](https://docs.dagster.io/migration/airflow-to-dagster/airlift-v1/dag-level-migration/migrate#__docusaurus_skipToContent_fallback) note If you are just getting started with Airlift, we recommend using the new [Airlift component](https://docs.dagster.io/migration/airflow-to-dagster/airflow-component-tutorial) . Previously, we completed the ["observe" stage](https://docs.dagster.io/migration/airflow-to-dagster/airlift-v1/dag-level-migration/observe) of the Airflow DAG-level migration process by encoding the assets that are produced by each task. We also introduced partitioning to those assets. In the [task-level migration step](https://docs.dagster.io/migration/airflow-to-dagster/airlift-v1/task-level-migration/migrate) , we "proxied" execution on a per-task basis through a YAML document. For DAG-mapped assets, execution is proxied on a per-DAG basis. Proxying execution to Dagster will require all assets mapped to that DAG be executable within Dagster. Let's take a look at some fully migrated code mapped to DAGs instead of tasks: import osfrom pathlib import Pathimport dagster as dgimport dagster_airlift.core as dg_airlift_coreimport dagster_dbt as dg_dbt# Code also invoked from Airflowfrom tutorial_example.shared.export_duckdb_to_csv import ExportDuckDbToCsvArgs, export_duckdb_to_csvfrom tutorial_example.shared.load_csv_to_duckdb import LoadCsvToDuckDbArgs, load_csv_to_duckdbdef dbt_project_path() -> Path: env_val = os.getenv("TUTORIAL_DBT_PROJECT_DIR") assert env_val, "TUTORIAL_DBT_PROJECT_DIR must be set" return Path(env_val)def airflow_dags_path() -> Path: return Path(os.environ["TUTORIAL_EXAMPLE_DIR"]) / "tutorial_example" / "airflow_dags"def load_csv_to_duckdb_asset(spec: dg.AssetSpec, args: LoadCsvToDuckDbArgs) -> dg.AssetsDefinition: @dg.multi_asset(name=f"load_{args.table_name}", specs=[spec]) def _multi_asset() -> None: load_csv_to_duckdb(args) return _multi_assetdef export_duckdb_to_csv_defs( spec: dg.AssetSpec, args: ExportDuckDbToCsvArgs) -> dg.AssetsDefinition: @dg.multi_asset(name=f"export_{args.table_name}", specs=[spec]) def _multi_asset() -> None: export_duckdb_to_csv(args) return _multi_asset@dg_dbt.dbt_assets( manifest=dbt_project_path() / "target" / "manifest.json", project=dg_dbt.DbtProject(dbt_project_path()),)def dbt_project_assets(context: dg.AssetExecutionContext, dbt: dg_dbt.DbtCliResource): yield from dbt.cli(["build"], context=context).stream()mapped_assets = dg_airlift_core.assets_with_dag_mappings( dag_mappings={ "rebuild_customers_list": [ load_csv_to_duckdb_asset( dg.AssetSpec(key=["raw_data", "raw_customers"]), LoadCsvToDuckDbArgs( table_name="raw_customers", csv_path=airflow_dags_path() / "raw_customers.csv", duckdb_path=Path(os.environ["AIRFLOW_HOME"]) / "jaffle_shop.duckdb", names=["id", "first_name", "last_name"], duckdb_schema="raw_data", duckdb_database_name="jaffle_shop", ), ), dbt_project_assets, export_duckdb_to_csv_defs( dg.AssetSpec(key="customers_csv", deps=["customers"]), ExportDuckDbToCsvArgs( table_name="customers", csv_path=Path(os.environ["TUTORIAL_EXAMPLE_DIR"]) / "customers.csv", duckdb_path=Path(os.environ["AIRFLOW_HOME"]) / "jaffle_shop.duckdb", duckdb_database_name="jaffle_shop", ), ), ], },)defs = dg_airlift_core.build_defs_from_airflow_instance( airflow_instance=dg_airlift_core.AirflowInstance( auth_backend=dg_airlift_core.AirflowBasicAuthBackend( webserver_url="http://localhost:8080", username="admin", password="admin", ), name="airflow_instance_one", ), defs=dg.Definitions( assets=mapped_assets, resources={"dbt": dg_dbt.DbtCliResource(project_dir=dbt_project_path())}, ),) Now that all of our assets are fully executable, we can create a simple YAML file to proxy execution for the whole DAG: proxied: True We will similarly use `proxying_to_dagster` at the end of our DAG file. The code is exactly the same here as it is for the per-task migration step: # Dags file can be found at tutorial_example/airflow_dags/dags.pyfrom pathlib import Pathfrom airflow import DAGfrom dagster_airlift.in_airflow import proxying_to_dagsterfrom dagster_airlift.in_airflow.proxied_state import load_proxied_state_from_yamldag = DAG("rebuild_customers_list", ...)...# Set this to True to begin the proxying processPROXYING = Falseif PROXYING: proxying_to_dagster( global_vars=globals(), proxied_state=load_proxied_state_from_yaml(Path(__file__).parent / "proxied_state"), ) Once `proxied` is changed to `True`, we can visit the Airflow UI and see that our tasks have been replaced with a single task: ![Before DAG proxying](https://docs.dagster.io/assets/images/before_dag_override-b2547ceb1d9fab295402a1bf6e9d1129.png) ![After DAG proxying](https://docs.dagster.io/assets/images/after_dag_override-1e5b7aaec6b7988996acc797e952120a.png) When performing DAG-level mapping, we don't preserve task structure in the Airflow DAGs. This single task will materialize all mapped Dagster assets instead of executing the original Airflow task business logic. We can similarly change `proxied` back to `False`, and the original task structure and business logic will return unchanged. --- # Observe multiple Airflow instances from Dagster | Dagster Docs [Skip to main content](https://docs.dagster.io/migration/airflow-to-dagster/airlift-v1/federation/observe#__docusaurus_skipToContent_fallback) On this page note If you are just getting started with Airlift, we recommend using the new [Airlift component](https://docs.dagster.io/migration/airflow-to-dagster/airflow-component-tutorial) . In the [previous step](https://docs.dagster.io/migration/airflow-to-dagster/airlift-v1/federation/setup) , we installed the tutorial example code and started two Airflow instances running locally. In this step, we'll create Dagster asset representations of Airflow DAGs in order to observe the Airflow instances from Dagster. Install the `dagster-airlift` package in your Dagster environment[​](https://docs.dagster.io/migration/airflow-to-dagster/airlift-v1/federation/observe#install-the-dagster-airlift-package-in-your-dagster-environment "Direct link to install-the-dagster-airlift-package-in-your-dagster-environment") ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- First, create a new shell and navigate to the root of the tutorial directory. You will need to install the `dagster-airlift`, `dagster-webserver`, and `dagster` packages in your Dagster environment: source .venv/bin/activateuv pip install 'dagster-airlift[core]' dagster-webserver dagster dagster-airlift API For a full list of `dagster-airlift` classes and methods, see the [API docs](https://docs.dagster.io/api/libraries/dagster-airlift) . Observe the `warehouse` Airflow instance[​](https://docs.dagster.io/migration/airflow-to-dagster/airlift-v1/federation/observe#observe-the-warehouse-airflow-instance "Direct link to observe-the-warehouse-airflow-instance") ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Next, in your `airlift_federation_tutorial/dagster_defs/definitions.py` file, declare a reference to the `warehouse` Airflow instance, which is running at `http://localhost:8081`: import dagster as dgimport dagster_airlift.core as dg_airlift_corewarehouse_airflow_instance = dg_airlift_core.AirflowInstance( auth_backend=dg_airlift_core.AirflowBasicAuthBackend( webserver_url="http://localhost:8081", username="admin", password="admin", ), name="warehouse",) Now you can use the `load_airflow_dag_asset_specs` function to create asset representations ([`AssetSpecs`](https://docs.dagster.io/api/dagster/assets#dagster.AssetSpec) ) of the DAGs in the `warehouse` Airflow instance: assets = dg_airlift_core.load_airflow_dag_asset_specs( airflow_instance=warehouse_airflow_instance,) Add these assets to a [`Definitions`](https://docs.dagster.io/api/dagster/definitions#dagster.Definitions) object: defs = dg.Definitions(assets=assets) Next, set up some environment variables, then point Dagster to the asset created from the Airflow instance: # Set up environment variables to point to the airlift-federation-tutorial directory on your machineexport TUTORIAL_EXAMPLE_DIR=$(pwd)export DAGSTER_HOME="$TUTORIAL_EXAMPLE_DIR/.dagster_home"dagster dev -f airlift_federation_tutorial/dagster_defs/definitions.py If you navigate to the Dagster UI (running at `http://localhost:3000`), you should see the assets created from the `warehouse` Airflow instance: ![Assets from the warehouse Airflow instance in the Dagster UI](https://docs.dagster.io/assets/images/observe_warehouse-d567aab9b05dc0917dd21990fabe5f16.png) There are a lot of DAGs in this instance, but we only want to focus on the `load_customers` DAG. Filter the assets to only include the `load_customers` DAG: load_customers = next( iter( dg_airlift_core.load_airflow_dag_asset_specs( airflow_instance=warehouse_airflow_instance, dag_selector_fn=lambda dag: dag.dag_id == "load_customers", ) )) Add this asset to the [`Definitions`](https://docs.dagster.io/api/dagster/definitions#dagster.Definitions) object: defs = dg.Definitions(assets=[load_customers]) Now, your Dagster environment only includes the `load_customers` DAG from the `warehouse` Airflow instance: ![Assets from the warehouse Airflow instance in the Dagster UI](https://docs.dagster.io/assets/images/only_load_customers-ee603103c6c2798385ae452e8b479fed.png) Finally, create a [sensor](https://docs.dagster.io/guides/automate/sensors) to poll the `warehouse` Airflow instance for new runs. This sensor ensures that whenever there is a successful run of the `load_customers` DAG, there will be a materialization in the Dagster UI: warehouse_sensor = dg_airlift_core.build_airflow_polling_sensor( mapped_assets=[load_customers], airflow_instance=warehouse_airflow_instance,) Next, add this sensor to our [`Definitions`](https://docs.dagster.io/api/dagster/definitions#dagster.Definitions) object: defs = dg.Definitions(assets=[load_customers], sensors=[warehouse_sensor]) You can test this by navigating to the Airflow UI at localhost:8081, and triggering a run of the `load_customers` DAG. When the run completes, you should see a materialization in the Dagster UI: ![Materialization of the load_customers DAG in the Dagster UI](https://docs.dagster.io/assets/images/load_customers_mat-94700d7e867cca3e6de6c40be8809e60.png) Observe the `metrics` Airflow instance[​](https://docs.dagster.io/migration/airflow-to-dagster/airlift-v1/federation/observe#observe-the-metrics-airflow-instance "Direct link to observe-the-metrics-airflow-instance") ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- You can repeat the same process for the `customer_metrics` DAG in the `metrics` Airflow instance, which runs at `http://localhost:8082`. We'll leave this as an exercise to test your understanding. Complete code To see what the code should look like after you have completed all the steps above, check out the [example in GitHub](https://github.com/dagster-io/dagster/blob/master/examples/airlift-federation-tutorial/airlift_federation_tutorial/dagster_defs/stages/observe_complete.py) . Observe the cross-DAG lineage between `load_customer` and `customer_metrics`[​](https://docs.dagster.io/migration/airflow-to-dagster/airlift-v1/federation/observe#observe-the-cross-dag-lineage-between-load_customer-and-customer_metrics "Direct link to observe-the-cross-dag-lineage-between-load_customer-and-customer_metrics") --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Now that you have both DAGs loaded into Dagster, you can observe the cross-DAG lineage between them. To do this, use the `replace_attributes` function to add a dependency from the `load_customers` asset to the `customer_metrics` asset: customer_metrics_dag_asset = customer_metrics_dag_asset.replace_attributes( deps=[load_customers],) Now, after adding the updated `customer_metrics_dag_asset` to our [`Definitions`](https://docs.dagster.io/api/dagster/definitions#dagster.Definitions) object, you should see the lineage between the two DAGs in the Dagster UI: ![Lineage between load_customers and customer_metrics in the Dagster UI](https://docs.dagster.io/assets/images/dag_lineage-229bddbb6c1fa403772e73ef33ba7f5e.png) Next steps[​](https://docs.dagster.io/migration/airflow-to-dagster/airlift-v1/federation/observe#next-steps "Direct link to Next steps") ----------------------------------------------------------------------------------------------------------------------------------------- In the next step, "[Federate execution across Airflow instances](https://docs.dagster.io/migration/airflow-to-dagster/airlift-v1/federation/federate-execution) ", we'll federate the execution of our DAGs across both Airflow instances. * [Install the `dagster-airlift` package in your Dagster environment](https://docs.dagster.io/migration/airflow-to-dagster/airlift-v1/federation/observe#install-the-dagster-airlift-package-in-your-dagster-environment) * [Observe the `warehouse` Airflow instance](https://docs.dagster.io/migration/airflow-to-dagster/airlift-v1/federation/observe#observe-the-warehouse-airflow-instance) * [Observe the `metrics` Airflow instance](https://docs.dagster.io/migration/airflow-to-dagster/airlift-v1/federation/observe#observe-the-metrics-airflow-instance) * [Observe the cross-DAG lineage between `load_customer` and `customer_metrics`](https://docs.dagster.io/migration/airflow-to-dagster/airlift-v1/federation/observe#observe-the-cross-dag-lineage-between-load_customer-and-customer_metrics) * [Next steps](https://docs.dagster.io/migration/airflow-to-dagster/airlift-v1/federation/observe#next-steps) --- # Migrate from Airflow to Dagster | Dagster Docs [Skip to main content](https://docs.dagster.io/migration/airflow-to-dagster/airlift-v1#__docusaurus_skipToContent_fallback) note If you are just getting started with Airlift, we recommend using the new [Airlift component](https://docs.dagster.io/migration/airflow-to-dagster/airflow-component-tutorial) . [Airlift](https://docs.dagster.io/integrations/libraries/airlift) is a toolkit for integrating Airflow into Dagster that you can use to migrate and consolidate existing Airflow DAGs into the Dagster control plane. Airflow allows Dagster to connect to live Airflow instances through Airflow’s REST API to observe Airflow executions as they happen. This makes it easy to transition the operation of Airflow pipelines into Dagster, or use Dagster to act as the control plane across multiple Airflow instances. A complete Airlift migration works through the following steps: * **Peer** - View the Airflow instance within Dagster. * **Observe** - Map the Airflow DAG to a full lineage of assets in Dagster. * **Migrate** - Move execution of specific Airflow tasks or an entire Airflow DAG to Dagster. * **Decommission** - Remove your Airflow code and move execution responsibilities over to Dagster. However, you don't need to complete every step with Airlift, and should tailor the migration process to your organization's needs. You may find immediate value from simply observing Airflow processes in Dagster and building around those workflows. To get started, see the documentation that best fits your situation: * [Federate execution between multiple Airflow instances with Dagster](https://docs.dagster.io/migration/airflow-to-dagster/airlift-v1/federation) * [Migrate from a single Airflow instance to Dagster at the DAG level](https://docs.dagster.io/migration/airflow-to-dagster/airlift-v1/dag-level-migration) * [Migrate from a single Airflow instance to Dagster at the task level](https://docs.dagster.io/migration/airflow-to-dagster/airlift-v1/task-level-migration) --- # 14 docs tagged with "compute" | Dagster Docs [Skip to main content](https://docs.dagster.io/tags/integrations/compute#__docusaurus_skipToContent_fallback) [Dagster & AWS EMR\ -----------------](https://docs.dagster.io/integrations/libraries/aws/emr) The AWS integration provides ways orchestrating data pipelines that leverage AWS services, including AWS EMR (Elastic MapReduce). This integration allows you to run and scale big data workloads using open source tools such as Apache Spark, Hive, Presto, and more. [Dagster & AWS Glue\ ------------------](https://docs.dagster.io/integrations/libraries/aws/glue) The AWS integration library provides the PipesGlueClient resource, enabling you to launch AWS Glue jobs directly from Dagster assets and ops. This integration allows you to pass parameters to Glue code while Dagster receives real-time events, such as logs, asset checks, and asset materializations, from the initiated jobs. With minimal code changes required on the job side, this integration is both efficient and easy to implement. [Dagster & AWS Lambda\ --------------------](https://docs.dagster.io/integrations/libraries/aws/lambda) Using this integration, you can leverage AWS Lambda to execute external code as part of your Dagster pipelines. This is particularly useful for running serverless functions that can scale automatically and handle various workloads without the need for managing infrastructure. The PipesLambdaClient class allows you to invoke AWS Lambda functions and stream logs and structured metadata back to Dagster's UI and tools. [Dagster & Databricks\ --------------------](https://docs.dagster.io/integrations/libraries/databricks) The Databricks integration library provides the \`PipesDatabricksClient\` resource, enabling you to launch Databricks jobs directly from Dagster assets and ops. This integration allows you to pass parameters to Databricks code while Dagster receives real-time events, such as logs, asset checks, and asset materializations, from the initiated jobs. With minimal code changes required on the job side, this integration is both efficient and easy to implement. [Dagster & Docker\ ----------------](https://docs.dagster.io/integrations/libraries/docker) The Docker integration library provides the PipesDockerClient resource, enabling you to launch Docker containers and execute external code directly from Dagster assets and ops. This integration allows you to pass parameters to Docker containers while Dagster receives real-time events, such as logs, asset checks, and asset materializations, from the initiated jobs. With minimal code changes required on the job side, this integration is both efficient and easy to implement. [Dagster & GCP Cloud Run\ -----------------------](https://docs.dagster.io/integrations/libraries/gcp/cloud-run-launcher) The community-supported dagster-contrib-gcp package provides integrations with Google Cloud Platform (GCP) services. [Dagster & GCP Dataproc\ ----------------------](https://docs.dagster.io/integrations/libraries/gcp/dataproc) Using this integration, you can manage and interact with Google Cloud Platform's Dataproc service directly from Dagster. This integration allows you to create, manage, and delete Dataproc clusters, and submit and monitor jobs on these clusters. [Dagster & HashiCorp\ -------------------](https://docs.dagster.io/integrations/libraries/hashicorp-nomad) The community-supported Nomad package provides an integration with HashiCorp Nomad. [Dagster & Hex\ -------------](https://docs.dagster.io/integrations/libraries/hex) The community-supported Hex package provides an integration with Hex. [Dagster & Jupyter Notebooks\ ---------------------------](https://docs.dagster.io/integrations/libraries/jupyter/) Dagstermill eliminates the tedious "productionization" of Jupyter notebooks. [Dagster & Kubernetes\ --------------------](https://docs.dagster.io/integrations/libraries/kubernetes) The Kubernetes integration library provides the PipesK8sClient resource, enabling you to launch Kubernetes pods and execute external code directly from Dagster assets and ops. This integration allows you to pass parameters to Kubernetes pods while Dagster receives real-time events, such as logs, asset checks, and asset materializations, from the initiated jobs. With minimal code changes required on the job side, this integration is both efficient and easy to implement. [Dagster & Modal\ ---------------](https://docs.dagster.io/integrations/libraries/modal) The community-supported Modal package provides an integration with Modal. [Dagster & Perian\ ----------------](https://docs.dagster.io/integrations/libraries/perian) The Perian integration allows you to easily dockerize your codebase and execute it on the PERIAN platform, PERIAN's serverless GPU environment. [Dagster & Spark\ ---------------](https://docs.dagster.io/integrations/libraries/spark) Running Spark code often requires submitting code to a Databricks or EMR cluster. The Pyspark integration provides a Spark class with methods for configuration and constructing the spark-submit command for a Spark job. --- # Peer the Airflow instance with a Dagster code location | Dagster Docs [Skip to main content](https://docs.dagster.io/migration/airflow-to-dagster/airlift-v1/dag-level-migration/peer#__docusaurus_skipToContent_fallback) On this page note If you are just getting started with Airlift, we recommend using the new [Airlift component](https://docs.dagster.io/migration/airflow-to-dagster/airflow-component-tutorial) . In the [setup step](https://docs.dagster.io/migration/airflow-to-dagster/airlift-v1/dag-level-migration/setup) , we created a virtual environment, installed Dagster and the tutorial example code, and set up a local Airflow instance. Now we can start writing Dagster code. We call the first stage of migration from Airflow to Dagster the "peering" stage, since we will "peer" the Airflow instance with a Dagster code location, which will create an asset representation of each Airflow DAG that you can view in Dagster. This step does not require any changes to your Airflow instance. Install `dagster-airlift`[​](https://docs.dagster.io/migration/airflow-to-dagster/airlift-v1/dag-level-migration/peer#install-dagster-airlift "Direct link to install-dagster-airlift") ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- First, you will want a new shell and navigate to the same directory. You will need to set up the `dagster-airlift` package in your Dagster environment: source .venv/bin/activateuv pip install 'dagster-airlift[core]' dagster-webserver dagster Create asset representations of DAGs in Dagster[​](https://docs.dagster.io/migration/airflow-to-dagster/airlift-v1/dag-level-migration/peer#create-asset-representations-of-dags-in-dagster "Direct link to Create asset representations of DAGs in Dagster") -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Next, use the [`build_defs_from_airflow_instance`](https://docs.dagster.io/api/libraries/dagster-airlift#dagster_airlift.core.build_defs_from_airflow_instance) function to create a `Definitions` object. Copy the following code into the empty `tutorial_example/dagster_defs/definitions.py` file: import dagster_airlift.core as dg_airlift_coredefs = dg_airlift_core.build_defs_from_airflow_instance( airflow_instance=dg_airlift_core.AirflowInstance( # other backends available (e.g. MwaaSessionAuthBackend) auth_backend=dg_airlift_core.AirflowBasicAuthBackend( webserver_url="http://localhost:8080", username="admin", password="admin", ), name="airflow_instance_one", )) This function creates: * An external asset representing each Airflow DAG. This asset is marked as materialized whenever a DAG run completes. * A [sensor](https://docs.dagster.io/guides/automate/sensors) that polls the Airflow instance for operational information. This sensor is responsible for creating materializations when a DAG executes and must remain on in order to properly update execution status. Initiate an asset materialization in Dagster from Airflow[​](https://docs.dagster.io/migration/airflow-to-dagster/airlift-v1/dag-level-migration/peer#initiate-an-asset-materialization-in-dagster-from-airflow "Direct link to Initiate an asset materialization in Dagster from Airflow") -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Next, set up some environment variables, then run `dagster dev` to start Dagster pointed at the asset created from the Airflow DAG: # Set up environment variables to point to the airlift-migration-tutorial directory on your machineexport TUTORIAL_EXAMPLE_DIR=$(pwd)export TUTORIAL_DBT_PROJECT_DIR="$TUTORIAL_EXAMPLE_DIR/tutorial_example/shared/dbt"export AIRFLOW_HOME="$TUTORIAL_EXAMPLE_DIR/.airflow_home"dagster dev -f tutorial_example/dagster_defs/definitions.py ![Peered asset in Dagster UI](https://docs.dagster.io/images/integrations/airlift/peer.svg) Initiate a run of the `reubild_customers_list` DAG in Airflow: airflow dags backfill rebuild_customers_list --start-date $(shell date +"%Y-%m-%d") When this run has completed in Airflow, you should be able to navigate to the Dagster UI and see that Dagster has registered an asset materialization corresponding to that run: ![Materialized peer asset in Dagster UI](https://docs.dagster.io/images/integrations/airlift/peer_materialize.svg) Clean the Airflow and Dagster run history[​](https://docs.dagster.io/migration/airflow-to-dagster/airlift-v1/dag-level-migration/peer#clean-the-airflow-and-dagster-run-history "Direct link to Clean the Airflow and Dagster run history") -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Later in this tutorial, you will need to run the `rebuild_customers_list` DAG again, so go ahead and run the following command to clean the Airflow and Dagster run history. This command deletes runs from Airflow and asset materializations from Dagster: make clean note When the code location loads, Dagster will query the Airflow REST API to build a representation of your DAGs. For Dagster to reflect changes to your DAGs, you will need to reload your code location. Validate data quality with asset checks[​](https://docs.dagster.io/migration/airflow-to-dagster/airlift-v1/dag-level-migration/peer#validate-data-quality-with-asset-checks "Direct link to Validate data quality with asset checks") -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Once you have peered your Airflow DAGs in Dagster, you can add [asset checks](https://docs.dagster.io/guides/test/asset-checks) to your Dagster code. In Dagster, asset checks can be used to validate the quality of your data assets, and can provide additional observability and value on top of your Airflow DAG even before you begin migration. Asset checks can act as user acceptance tests to ensure that any migration steps taken are successful, as well as outlive the migration itself. In this example, we're going to add an asset check to ensure that the final `customers` CSV output exists, and has a nonzero number of rows: import osfrom pathlib import Pathimport dagster as dgimport dagster_airlift.core as dg_airlift_core# Attach a check to the DAG representation asset, which will be executed by Dagster# any time the DAG is run in Airflow@dg.asset_check(asset=dg.AssetKey(["airflow_instance_one", "dag", "rebuild_customers_list"]))def validate_exported_csv() -> dg.AssetCheckResult: csv_path = Path(os.environ["TUTORIAL_EXAMPLE_DIR"]) / "customers.csv" if not csv_path.exists(): return dg.AssetCheckResult( passed=False, description=f"Export CSV {csv_path} does not exist" ) rows = len(csv_path.read_text().split("\n")) if rows < 2: return dg.AssetCheckResult( passed=False, description=f"Export CSV {csv_path} is empty", severity=dg.AssetCheckSeverity.WARN, ) return dg.AssetCheckResult( passed=True, description=f"Export CSV {csv_path} exists", metadata={"rows": rows}, )defs = dg_airlift_core.build_defs_from_airflow_instance( airflow_instance=dg_airlift_core.AirflowInstance( # other backends available (e.g. MwaaSessionAuthBackend) auth_backend=dg_airlift_core.AirflowBasicAuthBackend( webserver_url="http://localhost:8080", username="admin", password="admin", ), name="airflow_instance_one", ), # Pass a Definitions object containing the relevant check, so that it is attached to the DAG # asset. defs=dg.Definitions(asset_checks=[validate_exported_csv]),) Once you reload the code location, you should see a `checks` tab indicating the presence of an asset check on the `rebuild_customers_list` asset: ![Asset check on peer DAG](https://docs.dagster.io/assets/images/asset_check_peered_dag-866f74537b923a3ec355343778236568.png) Run the backfill again: airflow dags backfill rebuild_customers_list --start-date $(shell date +"%Y-%m-%d") You should see that the asset check executed successfully in Dagster (indicated by the green check mark): ![Asset check success](https://docs.dagster.io/assets/images/peer_check_success-02d11bbb1d6700ee9642412f494550bc.png) Finally, run `make clean` to delete runs and materializations: make clean Next steps[​](https://docs.dagster.io/migration/airflow-to-dagster/airlift-v1/dag-level-migration/peer#next-steps "Direct link to Next steps") ----------------------------------------------------------------------------------------------------------------------------------------------- In the next step, "[Observe an Airflow DAG](https://docs.dagster.io/migration/airflow-to-dagster/airlift-v1/dag-level-migration/observe) ", we'll create and observe assets that map to the entire example DAG. * [Install `dagster-airlift`](https://docs.dagster.io/migration/airflow-to-dagster/airlift-v1/dag-level-migration/peer#install-dagster-airlift) * [Create asset representations of DAGs in Dagster](https://docs.dagster.io/migration/airflow-to-dagster/airlift-v1/dag-level-migration/peer#create-asset-representations-of-dags-in-dagster) * [Initiate an asset materialization in Dagster from Airflow](https://docs.dagster.io/migration/airflow-to-dagster/airlift-v1/dag-level-migration/peer#initiate-an-asset-materialization-in-dagster-from-airflow) * [Clean the Airflow and Dagster run history](https://docs.dagster.io/migration/airflow-to-dagster/airlift-v1/dag-level-migration/peer#clean-the-airflow-and-dagster-run-history) * [Validate data quality with asset checks](https://docs.dagster.io/migration/airflow-to-dagster/airlift-v1/dag-level-migration/peer#validate-data-quality-with-asset-checks) * [Next steps](https://docs.dagster.io/migration/airflow-to-dagster/airlift-v1/dag-level-migration/peer#next-steps) --- # Setup | Dagster Docs [Skip to main content](https://docs.dagster.io/migration/airflow-to-dagster/airlift-v1/task-level-migration/setup#__docusaurus_skipToContent_fallback) On this page note If you are just getting started with Airlift, we recommend using the new [Airlift component](https://docs.dagster.io/migration/airflow-to-dagster/airflow-component-tutorial) . In order the complete this tutorial, you'll need to: * Create a virtual environment * Install Dagster and the tutorial example code * Set up a local Airflow instance Create a virtual environment[​](https://docs.dagster.io/migration/airflow-to-dagster/airlift-v1/task-level-migration/setup#create-a-virtual-environment "Direct link to Create a virtual environment") ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- First, create a fresh virtual environment using `uv` and activate it: pip install uvuv venvsource .venv/bin/activate Install Dagster and the tutorial example code[​](https://docs.dagster.io/migration/airflow-to-dagster/airlift-v1/task-level-migration/setup#install-dagster-and-the-tutorial-example-code "Direct link to Install Dagster and the tutorial example code") ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Next, install Dagster and verify that the `dagster` CLI is available: uv pip install dagsterdagster --version Finally, install the tutorial example code: dagster project from-example --name airlift-migration-tutorial --example airlift-migration-tutorial ### Project structure[​](https://docs.dagster.io/migration/airflow-to-dagster/airlift-v1/task-level-migration/setup#project-structure "Direct link to Project structure") The tutorial example contains the following files and directories: tutorial_example├── shared: Contains shared Python & SQL code used Airflow and proxied Dagster code│├── dagster_defs: Contains Dagster definitions│ ├── stages: Contains reference implementations of each stage of the migration process│ ├── definitions.py: Empty starter file for following along with the tutorial│├── airflow_dags: Contains the Airflow DAG and associated files│ ├── proxied_state: Contains migration state files for each DAG, see migration step below│ ├── dags.py: The Airflow DAG definition Set up a local Airflow instance[​](https://docs.dagster.io/migration/airflow-to-dagster/airlift-v1/task-level-migration/setup#set-up-a-local-airflow-instance "Direct link to Set up a local Airflow instance") ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- This tutorial involves running a local Airflow instance, which you can do by following commands from the root of the `airlift-migration-tutorial` directory. First, install the required Python packages: make airflow_install Next, scaffold the Airflow instance and initialize the `dbt` project: make airflow_setup Finally, run the Airflow instance with environment variables set: make airflow_run This will run the Airflow Web UI in a shell. You should now be able to access the Airflow UI at `http://localhost:8080`, with the default username and password set to `admin`. You should be able to see the `rebuild_customers_list` DAG in the Airflow UI, made up of three tasks: `load_raw_customers`, `run_dbt_model`, and `export_customers`: ![Rebuild customers list DAG](https://docs.dagster.io/assets/images/rebuild_customers_dag-76748681991c12a410926b6d665902d4.png) Next steps[​](https://docs.dagster.io/migration/airflow-to-dagster/airlift-v1/task-level-migration/setup#next-steps "Direct link to Next steps") ------------------------------------------------------------------------------------------------------------------------------------------------- In the next step, "[Peer your Airflow instance with a Dagster code location](https://docs.dagster.io/migration/airflow-to-dagster/airlift-v1/task-level-migration/peer) ", we'll peer our Dagster installation with our Airflow instance. * [Create a virtual environment](https://docs.dagster.io/migration/airflow-to-dagster/airlift-v1/task-level-migration/setup#create-a-virtual-environment) * [Install Dagster and the tutorial example code](https://docs.dagster.io/migration/airflow-to-dagster/airlift-v1/task-level-migration/setup#install-dagster-and-the-tutorial-example-code) * [Project structure](https://docs.dagster.io/migration/airflow-to-dagster/airlift-v1/task-level-migration/setup#project-structure) * [Set up a local Airflow instance](https://docs.dagster.io/migration/airflow-to-dagster/airlift-v1/task-level-migration/setup#set-up-a-local-airflow-instance) * [Next steps](https://docs.dagster.io/migration/airflow-to-dagster/airlift-v1/task-level-migration/setup#next-steps) --- # Decommission the Airflow DAG | Dagster Docs [Skip to main content](https://docs.dagster.io/migration/airflow-to-dagster/airlift-v1/task-level-migration/decommission#__docusaurus_skipToContent_fallback) note If you are just getting started with Airlift, we recommend using the new [Airlift component](https://docs.dagster.io/migration/airflow-to-dagster/airflow-component-tutorial) . Previously, we completed [migration](https://docs.dagster.io/migration/airflow-to-dagster/airlift-v1/task-level-migration/migrate) of our example Airflow DAG to Dagster assets. Once we are confident in our migrated versions of the tasks, we can decommission the Airflow DAG. First, we can remove the DAG from our Airflow DAG directory. Next, we can remove the task associations from our Dagster definitions. This can be done by removing the [`assets_with_task_mappings`](https://docs.dagster.io/api/libraries/dagster-airlift#dagster_airlift.core.assets_with_task_mappings) call. Finally, we can attach our example assets to a [`ScheduleDefinition`](https://docs.dagster.io/api/dagster/schedules-sensors#dagster.ScheduleDefinition) so Dagster's scheduler can manage their execution. When you have finished the above steps, your code should look like the following: import osfrom pathlib import Pathimport dagster as dgimport dagster_dbt as dg_dbtfrom dagster._time import get_current_datetime_midnight# Code also invoked from Airflowfrom tutorial_example.shared.export_duckdb_to_csv import ExportDuckDbToCsvArgs, export_duckdb_to_csvfrom tutorial_example.shared.load_csv_to_duckdb import LoadCsvToDuckDbArgs, load_csv_to_duckdbPARTITIONS_DEF = dg.DailyPartitionsDefinition(start_date=get_current_datetime_midnight())def dbt_project_path() -> Path: env_val = os.getenv("TUTORIAL_DBT_PROJECT_DIR") assert env_val, "TUTORIAL_DBT_PROJECT_DIR must be set" return Path(env_val)def airflow_dags_path() -> Path: return Path(os.environ["TUTORIAL_EXAMPLE_DIR"]) / "tutorial_example" / "airflow_dags"def load_csv_to_duckdb_asset(spec: dg.AssetSpec, args: LoadCsvToDuckDbArgs) -> dg.AssetsDefinition: @dg.multi_asset(name=f"load_{args.table_name}", specs=[spec]) def _multi_asset() -> None: load_csv_to_duckdb(args) return _multi_assetdef export_duckdb_to_csv_defs( spec: dg.AssetSpec, args: ExportDuckDbToCsvArgs) -> dg.AssetsDefinition: @dg.multi_asset(name=f"export_{args.table_name}", specs=[spec]) def _multi_asset() -> None: export_duckdb_to_csv(args) return _multi_asset@dg_dbt.dbt_assets( manifest=dbt_project_path() / "target" / "manifest.json", project=dg_dbt.DbtProject(dbt_project_path()), partitions_def=PARTITIONS_DEF,)def dbt_project_assets(context: dg.AssetExecutionContext, dbt: dg_dbt.DbtCliResource): yield from dbt.cli(["build"], context=context).stream()assets = [ load_csv_to_duckdb_asset( dg.AssetSpec(key=["raw_data", "raw_customers"], partitions_def=PARTITIONS_DEF), LoadCsvToDuckDbArgs( table_name="raw_customers", csv_path=airflow_dags_path() / "raw_customers.csv", duckdb_path=Path(os.environ["AIRFLOW_HOME"]) / "jaffle_shop.duckdb", names=["id", "first_name", "last_name"], duckdb_schema="raw_data", duckdb_database_name="jaffle_shop", ), ), dbt_project_assets, export_duckdb_to_csv_defs( dg.AssetSpec(key="customers_csv", deps=["customers"], partitions_def=PARTITIONS_DEF), ExportDuckDbToCsvArgs( table_name="customers", csv_path=Path(os.environ["TUTORIAL_EXAMPLE_DIR"]) / "customers.csv", duckdb_path=Path(os.environ["AIRFLOW_HOME"]) / "jaffle_shop.duckdb", duckdb_database_name="jaffle_shop", ), ),]@dg.asset_check(asset=dg.AssetKey(["customers_csv"]))def validate_exported_csv() -> dg.AssetCheckResult: csv_path = Path(os.environ["TUTORIAL_EXAMPLE_DIR"]) / "customers.csv" if not csv_path.exists(): return dg.AssetCheckResult( passed=False, description=f"Export CSV {csv_path} does not exist" ) rows = len(csv_path.read_text().split("\n")) if rows < 2: return dg.AssetCheckResult( passed=False, description=f"Export CSV {csv_path} is empty", severity=dg.AssetCheckSeverity.WARN, ) return dg.AssetCheckResult( passed=True, description=f"Export CSV {csv_path} exists", metadata={"rows": rows}, )# create a schedule to run our assets, which allows us to completely decomission the Airflow dag.rebuild_customer_list_schedule = rebuild_customers_list_schedule = dg.ScheduleDefinition( name="rebuild_customers_list_schedule", target=dg.AssetSelection.assets(*assets), cron_schedule="0 0 * * *",)# we've removed the call to Airlift entirely.defs = dg.Definitions( assets=assets, schedules=[rebuild_customer_list_schedule], asset_checks=[validate_exported_csv], resources={"dbt": dg_dbt.DbtCliResource(project_dir=dbt_project_path())},) --- # 16 docs tagged with "storage" | Dagster Docs [Skip to main content](https://docs.dagster.io/tags/integrations/storage#__docusaurus_skipToContent_fallback) [Dagster & Azure Data Lake Storage Gen 2\ ---------------------------------------](https://docs.dagster.io/integrations/libraries/azure-adls2) Dagster helps you use Azure Storage Accounts as part of your data pipeline. Azure Data Lake Storage Gen 2 (ADLS2) is our primary focus but we also provide utilities for Azure Blob Storage. [Dagster & AWS Athena\ --------------------](https://docs.dagster.io/integrations/libraries/aws/athena) This integration allows you to connect to AWS Athena, a serverless interactive query service that makes it easy to analyze data in Amazon S3 using standard SQL. Using this integration, you can issue queries to Athena, fetch results, and handle query execution states within your Dagster pipelines. [Dagster & AWS Redshift\ ----------------------](https://docs.dagster.io/integrations/libraries/aws/redshift) Using this integration, you can connect to an AWS Redshift cluster and issue queries against it directly from your Dagster assets. This allows you to seamlessly integrate Redshift into your data pipelines, leveraging the power of Redshift's data warehousing capabilities within your Dagster workflows. [Dagster & AWS S3\ ----------------](https://docs.dagster.io/integrations/libraries/aws/s3) The AWS S3 integration allows data engineers to easily read, and write objects to the durable AWS S3 storage enabling engineers to a resilient storage layer when constructing their pipelines. [Dagster & Chroma\ ----------------](https://docs.dagster.io/integrations/libraries/chroma) The Chroma library allows you to easily interact with Chroma's vector database capabilities to build AI-powered data pipelines in Dagster. You can perform vector similarity searches, manage schemas, and handle data operations directly from your Dagster assets. [Dagster & Delta Lake\ --------------------](https://docs.dagster.io/integrations/libraries/deltalake/) Delta Lake is a great storage format for Dagster workflows. With this integration, you can use the Delta Lake I/O Manager to read and write your Dagster assets. [Dagster & DuckDB\ ----------------](https://docs.dagster.io/integrations/libraries/duckdb/) This library provides an integration with the DuckDB database, and allows for an out-of-the-box I/O Manager so that you can make DuckDB your storage of choice. [Dagster & GCP BigQuery\ ----------------------](https://docs.dagster.io/integrations/libraries/gcp/bigquery/) Integrate with GCP BigQuery. [Dagster & GCP GCS\ -----------------](https://docs.dagster.io/integrations/libraries/gcp/gcs) This integration allows you to interact with Google Cloud Storage (GCS) using Dagster. It provides resources, I/O Managers, and utilities to manage and store data in GCS, making it easier to integrate GCS into your data pipelines. [Dagster & Iceberg\ -----------------](https://docs.dagster.io/integrations/libraries/iceberg/) This library provides I/O managers for reading and writing Apache Iceberg tables. It also provides a Dagster resource for accessing Iceberg tables. [Dagster & LakeFS\ ----------------](https://docs.dagster.io/integrations/libraries/lakefs) By integrating with lakeFS, a big data scale version control system, you can leverage the versioning capabilities of lakeFS to track changes to your data. This integration allows you to have a complete lineage of your data, from the initial raw data to the transformed and processed data, making it easier to understand and reproduce data transformations. [Dagster & obstore\ -----------------](https://docs.dagster.io/integrations/libraries/obstore) The community-supported obstore package provides an integration with obstore, providing three lean integrations with object stores, ADLS, GCS & S3. [Dagster & Qdrant\ ----------------](https://docs.dagster.io/integrations/libraries/qdrant) The Qdrant library lets you integrate Qdrant's vector database with Dagster, making it easy to build AI-driven data pipelines. You can run vector searches and manage data directly within Dagster. [Dagster & Snowflake\ -------------------](https://docs.dagster.io/integrations/libraries/snowflake/) This library provides an integration with the Snowflake data warehouse. Connect to Snowflake as a resource, then use the integration-provided functions to construct an op to establish connections and execute Snowflake queries. Read and write natively to Snowflake from Dagster assets. [Dagster & Teradata\ ------------------](https://docs.dagster.io/integrations/libraries/teradata) The community-supported Teradata package provides an integration with Teradata Vantage. [Dagster & Weaviate\ ------------------](https://docs.dagster.io/integrations/libraries/weaviate) The Weaviate library allows you to easily interact with Weaviate's vector database capabilities to build AI-powered data pipelines in Dagster. You can perform vector similarity searches, manage schemas, and handle data operations directly from your Dagster assets. --- # Migrate from Airflow to Dagster at the task level | Dagster Docs [Skip to main content](https://docs.dagster.io/migration/airflow-to-dagster/airlift-v1/task-level-migration#__docusaurus_skipToContent_fallback) On this page note If you are just getting started with Airlift, we recommend using the new [Airlift component](https://docs.dagster.io/migration/airflow-to-dagster/airflow-component-tutorial) . This tutorial demonstrates using [`dagster-airlift`](https://docs.dagster.io/api/libraries/dagster-airlift) to migrate an Airflow DAG to Dagster at the task level. Using `dagster-airlift` you can: * Observe Airflow DAGs and their execution history with no changes to Airflow code * Model and observe assets orchestrated by Airflow with no changes to Airflow code * Enable a migration process that: * Can be done task-by-task in any order with minimal coordination * Has task-by-task rollback to reduce risk * Retains Airflow DAG structure and execution history during the migration Process[​](https://docs.dagster.io/migration/airflow-to-dagster/airlift-v1/task-level-migration#process "Direct link to Process") ---------------------------------------------------------------------------------------------------------------------------------- In this tutorial, you'll take the following steps: * **Peer** - During the peer stage, you'll observe an Airflow instance from within a Dagster Deployment using the Airflow REST API. This loads every Airflow DAG as an asset definition and creates a sensor that polls Airflow for execution history. * **Observe** - In the observe stage, you'll add a mapping that maps the Airflow DAG and task ID to a collection of definitions that you want to observe. (e.g. render the full lineage the dbt models an Airflow task orchestrates). The sensor used for peering also polls for task execution history, and adds materializations to an observed asset when its corresponding task successfully executes. * **Migrate** - Finally, in the migrate stage, you'll selectively move execution of Airflow tasks to Dagster assets. Next steps[​](https://docs.dagster.io/migration/airflow-to-dagster/airlift-v1/task-level-migration#next-steps "Direct link to Next steps") ------------------------------------------------------------------------------------------------------------------------------------------- To get started with this tutorial, follow the [setup steps](https://docs.dagster.io/migration/airflow-to-dagster/airlift-v1/task-level-migration/setup) to install the example code, set up a local environment, and run Airflow locally. Migration best practices When migrating Airflow DAGs to Dagster, we recommend a few best practices: * **Create separate packages for the Airflow and Dagster deployments.** Airflow has complex dependencies and can be difficult to install in the same environment as Dagster. * **Create user acceptance tests in Dagster before migrating.** This will help you catch issues easily during migration. * **Understand the rollback procedure for your migration.** When proxying execution to Dagster from Airflow, you can always roll back by changing a single line of code in the Airflow DAG. * [Process](https://docs.dagster.io/migration/airflow-to-dagster/airlift-v1/task-level-migration#process) * [Next steps](https://docs.dagster.io/migration/airflow-to-dagster/airlift-v1/task-level-migration#next-steps) --- # Migrate from Airflow to Dagster at the DAG level | Dagster Docs [Skip to main content](https://docs.dagster.io/migration/airflow-to-dagster/airlift-v1/dag-level-migration#__docusaurus_skipToContent_fallback) On this page note If you are just getting started with Airlift, we recommend using the new [Airlift component](https://docs.dagster.io/migration/airflow-to-dagster/airflow-component-tutorial) . This tutorial demonstrates mapping assets to a full Airflow DAG using [`dagster-airlift`](https://docs.dagster.io/api/libraries/dagster-airlift) . You might want to map assets to a full Airflow DAG rather than on a per-task basis because: * You're making use of "dynamic tasks" in Airflow, which don't conform neatly to the task mapping protocol in the [task-level migration guide](https://docs.dagster.io/migration/airflow-to-dagster/airlift-v1/task-level-migration) . * You want to refactor the DAG structure in a way that doesn't conform to the existing task structure. Next steps[​](https://docs.dagster.io/migration/airflow-to-dagster/airlift-v1/dag-level-migration#next-steps "Direct link to Next steps") ------------------------------------------------------------------------------------------------------------------------------------------ To get started with this tutorial, follow the [setup steps](https://docs.dagster.io/migration/airflow-to-dagster/airlift-v1/dag-level-migration/setup) to install the example code, set up a local environment, and run Airflow locally. Migration best practices When migrating Airflow DAGs to Dagster, we recommend a few best practices: * **Create separate packages for the Airflow and Dagster deployments.** Airflow has complex dependencies and can be difficult to install in the same environment as Dagster. * **Create user acceptance tests in Dagster before migrating.** This will help you catch issues easily during migration. * **Understand the rollback procedure for your migration.** When proxying execution to Dagster from Airflow, you can always roll back by changing a single line of code in the Airflow DAG. * [Next steps](https://docs.dagster.io/migration/airflow-to-dagster/airlift-v1/dag-level-migration#next-steps) --- # Migrate Airflow tasks | Dagster Docs [Skip to main content](https://docs.dagster.io/migration/airflow-to-dagster/airlift-v1/task-level-migration/migrate#__docusaurus_skipToContent_fallback) On this page note If you are just getting started with Airlift, we recommend using the new [Airlift component](https://docs.dagster.io/migration/airflow-to-dagster/airflow-component-tutorial) . Previously, we completed the ["observe" stage](https://docs.dagster.io/migration/airflow-to-dagster/airlift-v1/task-level-migration/observe) of the Airflow migration process by encoding the assets that are produced by each task. We also introduced partitioning to those assets. So far, we have left the Airflow code alone, but in this step, we will begin the actual migration process, which will require modifying Airflow code. Once you have created corresponding Dagster assets for your Airflow tasks, you can proxy execution to Dagster on a per-task basis while Airflow is still controlling scheduling and orchestration. Once a task has been proxied, Airflow will kick off materializations of corresponding Dagster assets in place of executing the business logic of that task. Create a file to track proxying state[​](https://docs.dagster.io/migration/airflow-to-dagster/airlift-v1/task-level-migration/migrate#create-a-file-to-track-proxying-state "Direct link to Create a file to track proxying state") ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ To begin proxying tasks in an Airflow DAG, you will first need to create a file to track proxying state. For this tutorial, in the example Airflow DAG directory, create a `proxied_state` folder. In that folder, create a YAML file with the same name as the example DAG. The included example at `airflow_dags/proxied_state` is used by `make airflow_run`, and can be used as a template for your own proxied state files. Given our example `rebuild_customers_list` DAG with its three tasks, `load_raw_customers`, `run_dbt_model`, and `export_customers`, your `proxied_state/rebuild_customers_list.yaml` file should look like the following: tasks: - id: load_raw_customers proxied: False - id: build_dbt_models proxied: False - id: export_customers proxied: False Next, you will need to modify your Airflow DAG to make it aware of the proxied state. This is already done in the example DAG: # Dags file can be found at tutorial_example/airflow_dags/dags.pyfrom pathlib import Pathfrom airflow import DAGfrom dagster_airlift.in_airflow import proxying_to_dagsterfrom dagster_airlift.in_airflow.proxied_state import load_proxied_state_from_yamldag = DAG("rebuild_customers_list", ...)...# Set this to True to begin the proxying processPROXYING = Falseif PROXYING: proxying_to_dagster( global_vars=globals(), proxied_state=load_proxied_state_from_yaml(Path(__file__).parent / "proxied_state"), ) Set `PROXYING` to `True` or eliminate the `if` statement. The DAG will now display its proxied state in the Airflow UI. (There is some latency as Airflow evaluates the Python file periodically.) ![Migration state rendering in Airflow UI](https://docs.dagster.io/assets/images/state_in_airflow-d386697cc6e82eb937e96b3cab12eb2e.png) Migrate individual tasks[​](https://docs.dagster.io/migration/airflow-to-dagster/airlift-v1/task-level-migration/migrate#migrate-individual-tasks "Direct link to Migrate individual tasks") --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- In order to proxy a task, you must do two things: 1. Ensure all associated assets are executable in Dagster by providing asset definitions in place of bare asset specs. 2. Set the `proxied: False` status in the `proxied_state` YAML folder to `proxied: True`. Any task marked as proxied will use the [`DefaultProxyTaskToDagsterOperator`](https://docs.dagster.io/api/libraries/dagster-airlift#dagster_airlift.in_airflow.DefaultProxyTaskToDagsterOperator) when executed as part of the DAG. This operator will use the [Dagster GraphQL API](https://docs.dagster.io/guides/operate/graphql) to initiate a Dagster run of the assets corresponding to the task. The proxied file acts as the source of truth for proxied state. The information is attached to the DAG and then accessed by Dagster via the REST API. A task which has been proxied can be easily toggled back to run in Airflow (for example, if a bug in implementation was encountered) simply by editing the file to `proxied: False`. Migrate common operators[​](https://docs.dagster.io/migration/airflow-to-dagster/airlift-v1/task-level-migration/migrate#migrate-common-operators "Direct link to Migrate common operators") --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- For some common operator patterns, like our `dbt` operator, Dagster supplies factories to build software-defined assets for our tasks. In fact, the [`@dagster_dbt.dbt_assets`](https://docs.dagster.io/api/libraries/dagster-dbt#dagster_dbt.dbt_assets) decorator used earlier already backs its assets with definitions, so we can change the proxied state of the `build_dbt_models` task to `proxied: True` in the proxied state file: tasks: - id: load_raw_customers proxied: False - id: build_dbt_models proxied: True - id: export_customers proxied: False info It may take up to 30 seconds for the proxied state in the Airflow UI to reflect this change. You must subsequently reload the definitions in Dagster via the UI or by restarting `dagster dev`. You can now run the `rebuild_customers_list` DAG in Airflow, and the `build_dbt_models` task will be executed in a Dagster run: ![dbt build executing in Dagster](https://docs.dagster.io/assets/images/proxied_dag-92e53643454a5404938c5509113f3120.png) You'll note that we proxied a task in the middle of the Airflow DAG. The Airflow DAG structure and execution history is stable in the Airflow UI, but execution of `build_dbt_models` has moved to Dagster. Migrate the remaining custom operators[​](https://docs.dagster.io/migration/airflow-to-dagster/airlift-v1/task-level-migration/migrate#migrate-the-remaining-custom-operators "Direct link to Migrate the remaining custom operators") --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- For all other operator types, we will need to build our own asset definitions. We recommend creating a factory function whose arguments match the inputs to your Airflow operator. Then, you can use this factory to build definitions for each Airflow task. For example, our `load_raw_customers` task uses a custom `LoadCSVToDuckDB` operator. We'll define a function `load_csv_to_duckdb_defs` factory to build corresponding software-defined assets. Similarly for `export_customers` we'll define a function `export_duckdb_to_csv_defs` to build software-defined assets: import osfrom pathlib import Pathimport dagster as dgimport dagster_airlift.core as dg_airlift_coreimport dagster_dbt as dg_dbtfrom dagster._time import get_current_datetime_midnight# Code also invoked from Airflowfrom tutorial_example.shared.export_duckdb_to_csv import ExportDuckDbToCsvArgs, export_duckdb_to_csvfrom tutorial_example.shared.load_csv_to_duckdb import LoadCsvToDuckDbArgs, load_csv_to_duckdbPARTITIONS_DEF = dg.DailyPartitionsDefinition(start_date=get_current_datetime_midnight())@dg.asset_check(asset=dg.AssetKey(["airflow_instance_one", "dag", "rebuild_customers_list"]))def validate_exported_csv() -> dg.AssetCheckResult: csv_path = Path(os.environ["TUTORIAL_EXAMPLE_DIR"]) / "customers.csv" if not csv_path.exists(): return dg.AssetCheckResult( passed=False, description=f"Export CSV {csv_path} does not exist" ) rows = len(csv_path.read_text().split("\n")) if rows < 2: return dg.AssetCheckResult( passed=False, description=f"Export CSV {csv_path} is empty", severity=dg.AssetCheckSeverity.WARN, ) return dg.AssetCheckResult( passed=True, description=f"Export CSV {csv_path} exists", metadata={"rows": rows}, )def dbt_project_path() -> Path: env_val = os.getenv("TUTORIAL_DBT_PROJECT_DIR") assert env_val, "TUTORIAL_DBT_PROJECT_DIR must be set" return Path(env_val)def airflow_dags_path() -> Path: return Path(os.environ["TUTORIAL_EXAMPLE_DIR"]) / "tutorial_example" / "airflow_dags"# create an executable asset to load the csv file to duckdbdef load_csv_to_duckdb_asset(spec: dg.AssetSpec, args: LoadCsvToDuckDbArgs) -> dg.AssetsDefinition: @dg.multi_asset(name=f"load_{args.table_name}", specs=[spec]) def _multi_asset() -> None: load_csv_to_duckdb(args) return _multi_asset# create an executable asset to export back to csvdef export_duckdb_to_csv_defs( spec: dg.AssetSpec, args: ExportDuckDbToCsvArgs) -> dg.AssetsDefinition: @dg.multi_asset(name=f"export_{args.table_name}", specs=[spec]) def _multi_asset() -> None: export_duckdb_to_csv(args) return _multi_asset@dg_dbt.dbt_assets( manifest=dbt_project_path() / "target" / "manifest.json", project=dg_dbt.DbtProject(dbt_project_path()), partitions_def=PARTITIONS_DEF,)def dbt_project_assets(context: dg.AssetExecutionContext, dbt: dg_dbt.DbtCliResource): yield from dbt.cli(["build"], context=context).stream()mapped_assets = dg_airlift_core.assets_with_task_mappings( dag_id="rebuild_customers_list", task_mappings={ # instead of just loading the asset specs, we're mapping to fully executable assets now. "load_raw_customers": [ load_csv_to_duckdb_asset( dg.AssetSpec(key=["raw_data", "raw_customers"], partitions_def=PARTITIONS_DEF), LoadCsvToDuckDbArgs( table_name="raw_customers", csv_path=airflow_dags_path() / "raw_customers.csv", duckdb_path=Path(os.environ["AIRFLOW_HOME"]) / "jaffle_shop.duckdb", names=["id", "first_name", "last_name"], duckdb_schema="raw_data", duckdb_database_name="jaffle_shop", ), ) ], "build_dbt_models": [dbt_project_assets], # instead of just loading the asset specs, we're mapping to fully executable assets now. "export_customers": [ export_duckdb_to_csv_defs( dg.AssetSpec( key="customers_csv", deps=["customers"], partitions_def=PARTITIONS_DEF ), ExportDuckDbToCsvArgs( table_name="customers", csv_path=Path(os.environ["TUTORIAL_EXAMPLE_DIR"]) / "customers.csv", duckdb_path=Path(os.environ["AIRFLOW_HOME"]) / "jaffle_shop.duckdb", duckdb_database_name="jaffle_shop", ), ) ], },)defs = dg_airlift_core.build_defs_from_airflow_instance( airflow_instance=dg_airlift_core.AirflowInstance( auth_backend=dg_airlift_core.AirflowBasicAuthBackend( webserver_url="http://localhost:8080", username="admin", password="admin", ), name="airflow_instance_one", ), defs=dg.Definitions( assets=mapped_assets, resources={"dbt": dg_dbt.DbtCliResource(project_dir=dbt_project_path())}, asset_checks=[validate_exported_csv], ),) We can then toggle the proxied state of the remaining tasks in the `proxied_state` file: tasks: - id: load_raw_customers proxied: True - id: build_dbt_models proxied: True - id: export_customers proxied: True Next steps[​](https://docs.dagster.io/migration/airflow-to-dagster/airlift-v1/task-level-migration/migrate#next-steps "Direct link to Next steps") --------------------------------------------------------------------------------------------------------------------------------------------------- In the next step, "[Decommission the Airflow DAG](https://docs.dagster.io/migration/airflow-to-dagster/airlift-v1/task-level-migration/decommission) ", we will remove the DAG from the Airflow directory and update the Dagster code to remove task associations and attach the assets to a schedule. * [Create a file to track proxying state](https://docs.dagster.io/migration/airflow-to-dagster/airlift-v1/task-level-migration/migrate#create-a-file-to-track-proxying-state) * [Migrate individual tasks](https://docs.dagster.io/migration/airflow-to-dagster/airlift-v1/task-level-migration/migrate#migrate-individual-tasks) * [Migrate common operators](https://docs.dagster.io/migration/airflow-to-dagster/airlift-v1/task-level-migration/migrate#migrate-common-operators) * [Migrate the remaining custom operators](https://docs.dagster.io/migration/airflow-to-dagster/airlift-v1/task-level-migration/migrate#migrate-the-remaining-custom-operators) * [Next steps](https://docs.dagster.io/migration/airflow-to-dagster/airlift-v1/task-level-migration/migrate#next-steps) --- # Peer the Airflow instance with a Dagster code location | Dagster Docs [Skip to main content](https://docs.dagster.io/migration/airflow-to-dagster/airlift-v1/task-level-migration/peer#__docusaurus_skipToContent_fallback) On this page note If you are just getting started with Airlift, we recommend using the new [Airlift component](https://docs.dagster.io/migration/airflow-to-dagster/airflow-component-tutorial) . In the [setup step](https://docs.dagster.io/migration/airflow-to-dagster/airlift-v1/task-level-migration/setup) , we created a virtual environment, installed Dagster and the tutorial example code, and set up a local Airflow instance. Now we can start writing Dagster code. We call the first stage of migration from Airflow to Dagster the "peering" stage, since we will "peer" the Airflow instance with a Dagster code location, which will create an asset representation of each Airflow DAG that you can view in Dagster. This step does not require any changes to your Airflow instance. Install `dagster-airlift`[​](https://docs.dagster.io/migration/airflow-to-dagster/airlift-v1/task-level-migration/peer#install-dagster-airlift "Direct link to install-dagster-airlift") ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- First, you will want a new shell and navigate to the same directory. You will need to set up the `dagster-airlift` package in your Dagster environment: source .venv/bin/activateuv pip install 'dagster-airlift[core]' dagster-webserver dagster Create asset representations of DAGs in Dagster[​](https://docs.dagster.io/migration/airflow-to-dagster/airlift-v1/task-level-migration/peer#create-asset-representations-of-dags-in-dagster "Direct link to Create asset representations of DAGs in Dagster") --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Next, use the [`build_defs_from_airflow_instance`](https://docs.dagster.io/api/libraries/dagster-airlift#dagster_airlift.core.build_defs_from_airflow_instance) function to create a `Definitions` object. Copy the following code into the empty `tutorial_example/dagster_defs/definitions.py` file: import dagster_airlift.core as dg_airlift_coredefs = dg_airlift_core.build_defs_from_airflow_instance( airflow_instance=dg_airlift_core.AirflowInstance( # other backends available (e.g. MwaaSessionAuthBackend) auth_backend=dg_airlift_core.AirflowBasicAuthBackend( webserver_url="http://localhost:8080", username="admin", password="admin", ), name="airflow_instance_one", )) This function creates: * An external asset representing each Airflow DAG. This asset is marked as materialized whenever a DAG run completes. * A [sensor](https://docs.dagster.io/guides/automate/sensors) that polls the Airflow instance for operational information. This sensor is responsible for creating materializations when a DAG executes and must remain on in order to properly update execution status. Initiate an asset materialization in Dagster from Airflow[​](https://docs.dagster.io/migration/airflow-to-dagster/airlift-v1/task-level-migration/peer#initiate-an-asset-materialization-in-dagster-from-airflow "Direct link to Initiate an asset materialization in Dagster from Airflow") --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Next, set up some environment variables, then run `dagster dev` to start Dagster pointed at the asset created from the Airflow DAG: # Set up environment variables to point to the airlift-migration-tutorial directory on your machineexport TUTORIAL_EXAMPLE_DIR=$(pwd)export TUTORIAL_DBT_PROJECT_DIR="$TUTORIAL_EXAMPLE_DIR/tutorial_example/shared/dbt"export AIRFLOW_HOME="$TUTORIAL_EXAMPLE_DIR/.airflow_home"dagster dev -f tutorial_example/dagster_defs/definitions.py ![Peered asset in Dagster UI](https://docs.dagster.io/images/integrations/airlift/peer.svg) Initiate a run of the `rebuild_customers_list` DAG in Airflow: airflow dags backfill rebuild_customers_list --start-date $(shell date +"%Y-%m-%d") When this run has completed in Airflow, you should be able to navigate to the Dagster UI and see that Dagster has registered an asset materialization corresponding to that run: ![Materialized peer asset in Dagster UI](https://docs.dagster.io/images/integrations/airlift/peer_materialize.svg) Clean the Airflow and Dagster run history[​](https://docs.dagster.io/migration/airflow-to-dagster/airlift-v1/task-level-migration/peer#clean-the-airflow-and-dagster-run-history "Direct link to Clean the Airflow and Dagster run history") --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Later in this tutorial, you will need to run the `rebuild_customers_list` DAG again, so go ahead and run the following command to clean the Airflow and Dagster run history. This command deletes runs from Airflow and asset materializations from Dagster: make clean note When the code location loads, Dagster will query the Airflow REST API to build a representation of your DAGs. For Dagster to reflect changes to your DAGs, you will need to reload your code location. Validate data quality with asset checks[​](https://docs.dagster.io/migration/airflow-to-dagster/airlift-v1/task-level-migration/peer#validate-data-quality-with-asset-checks "Direct link to Validate data quality with asset checks") --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Once you have peered your Airflow DAGs in Dagster, you can add [asset checks](https://docs.dagster.io/guides/test/asset-checks) to your Dagster code. In Dagster, asset checks can be used to validate the quality of your data assets, and can provide additional observability and value on top of your Airflow DAG even before you begin migration. Asset checks can act as user acceptance tests to ensure that any migration steps taken are successful, as well as outlive the migration itself. In this example, we're going to add an asset check to ensure that the final `customers` CSV output exists, and has a nonzero number of rows: import osfrom pathlib import Pathimport dagster as dgimport dagster_airlift.core as dg_airlift_core# Attach a check to the DAG representation asset, which will be executed by Dagster# any time the DAG is run in Airflow@dg.asset_check(asset=dg.AssetKey(["airflow_instance_one", "dag", "rebuild_customers_list"]))def validate_exported_csv() -> dg.AssetCheckResult: csv_path = Path(os.environ["TUTORIAL_EXAMPLE_DIR"]) / "customers.csv" if not csv_path.exists(): return dg.AssetCheckResult( passed=False, description=f"Export CSV {csv_path} does not exist" ) rows = len(csv_path.read_text().split("\n")) if rows < 2: return dg.AssetCheckResult( passed=False, description=f"Export CSV {csv_path} is empty", severity=dg.AssetCheckSeverity.WARN, ) return dg.AssetCheckResult( passed=True, description=f"Export CSV {csv_path} exists", metadata={"rows": rows}, )defs = dg_airlift_core.build_defs_from_airflow_instance( airflow_instance=dg_airlift_core.AirflowInstance( # other backends available (e.g. MwaaSessionAuthBackend) auth_backend=dg_airlift_core.AirflowBasicAuthBackend( webserver_url="http://localhost:8080", username="admin", password="admin", ), name="airflow_instance_one", ), # Pass a Definitions object containing the relevant check, so that it is attached to the DAG # asset. defs=dg.Definitions(asset_checks=[validate_exported_csv]),) Once you reload the code location, you should see a `checks` tab indicating the presence of an asset check on the `rebuild_customers_list` asset: ![Asset check on peer DAG](https://docs.dagster.io/assets/images/asset_check_peered_dag-866f74537b923a3ec355343778236568.png) Run the backfill again: airflow dags backfill rebuild_customers_list --start-date $(shell date +"%Y-%m-%d") You should see that the asset check executed successfully in Dagster (indicated by the green check mark): ![Asset check success](https://docs.dagster.io/assets/images/peer_check_success-02d11bbb1d6700ee9642412f494550bc.png) Finally, run `make clean` to delete runs and materializations: make clean Next steps[​](https://docs.dagster.io/migration/airflow-to-dagster/airlift-v1/task-level-migration/peer#next-steps "Direct link to Next steps") ------------------------------------------------------------------------------------------------------------------------------------------------ In the next step, "[Observe Airflow tasks](https://docs.dagster.io/migration/airflow-to-dagster/airlift-v1/task-level-migration/observe) ", we'll observe asset dependencies within the Airflow DAG. * [Install `dagster-airlift`](https://docs.dagster.io/migration/airflow-to-dagster/airlift-v1/task-level-migration/peer#install-dagster-airlift) * [Create asset representations of DAGs in Dagster](https://docs.dagster.io/migration/airflow-to-dagster/airlift-v1/task-level-migration/peer#create-asset-representations-of-dags-in-dagster) * [Initiate an asset materialization in Dagster from Airflow](https://docs.dagster.io/migration/airflow-to-dagster/airlift-v1/task-level-migration/peer#initiate-an-asset-materialization-in-dagster-from-airflow) * [Clean the Airflow and Dagster run history](https://docs.dagster.io/migration/airflow-to-dagster/airlift-v1/task-level-migration/peer#clean-the-airflow-and-dagster-run-history) * [Validate data quality with asset checks](https://docs.dagster.io/migration/airflow-to-dagster/airlift-v1/task-level-migration/peer#validate-data-quality-with-asset-checks) * [Next steps](https://docs.dagster.io/migration/airflow-to-dagster/airlift-v1/task-level-migration/peer#next-steps) --- # Observe Airflow tasks | Dagster Docs [Skip to main content](https://docs.dagster.io/migration/airflow-to-dagster/airlift-v1/task-level-migration/observe#__docusaurus_skipToContent_fallback) On this page note If you are just getting started with Airlift, we recommend using the new [Airlift component](https://docs.dagster.io/migration/airflow-to-dagster/airflow-component-tutorial) . In the previous step, "[Peer the Airflow instance with a Dagster code location](https://docs.dagster.io/migration/airflow-to-dagster/airlift-v1/task-level-migration/peer) ", we connected the example Airflow instance to a Dagster code location. The next step is to represent the Airflow workflows more richly by observing the data assets that are produced by the Airflow tasks. Similar to the peering step, this step does not require any changes to Airflow code. Create asset specs for Airflow tasks[​](https://docs.dagster.io/migration/airflow-to-dagster/airlift-v1/task-level-migration/observe#create-asset-specs-for-airflow-tasks "Direct link to Create asset specs for Airflow tasks") --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- In order to observe the assets produced by the Airflow tasks in this tutorial, you will need to define the relevant assets in the Dagster code location. In this example, there are three sequential tasks: 1. `load_raw_customers` loads a CSV file of raw customer data into duckdb. 2. `build_dbt_models` builds a series of dbt models (from [jaffle shop](https://github.com/dbt-labs/jaffle_shop_duckdb) ) combining customer, order, and payment data. 3. `export_customers` exports a CSV representation of the final customer file from duckdb to disk. First, you will need to create a set of [`AssetSpecs`](https://docs.dagster.io/api/dagster/assets#dagster.AssetSpec) that correspond to the assets produced by these tasks. Next, you will annotate these asset specs so Dagster can associate them with the Airflow tasks that produce them. The first and third tasks involve a single table each, so we will manually construct asset specs for these two tasks. We will use the [`assets_with_task_mappings`](https://docs.dagster.io/api/libraries/dagster-airlift#dagster_airlift.core.assets_with_task_mappings) function in the `dagster-airlift` package to annotate these asset specs with the tasks that produce them. Assets which are properly annotated will be materialized by the Airlift sensor once the corresponding task completes, and these annotated specs are then provided to the `defs` argument to [`defs_from_airflow_instance`](https://docs.dagster.io/api/libraries/dagster-airlift#dagster_airlift.core.build_defs_from_airflow_instance) . The second task, `build_dbt_models`, will require building a set of `dbt` asset definitions. We will use the [`@dagster_dbt.dbt_assets`](https://docs.dagster.io/api/libraries/dagster-dbt#dagster_dbt.dbt_assets) decorator from the [`dagster-dbt`](https://docs.dagster.io/api/libraries/dagster-dbt) package to generate these definitions using Dagster's dbt integration. First, install the `dbt` extra of `dagster-airlift`: uv pip install 'dagster-airlift[dbt]' Next, construct the assets: import osfrom pathlib import Pathimport dagster as dgimport dagster_airlift.core as dg_airlift_coreimport dagster_dbt as dg_dbt@dg.asset_check(asset=dg.AssetKey(["airflow_instance_one", "dag", "rebuild_customers_list"]))def validate_exported_csv() -> dg.AssetCheckResult: csv_path = Path(os.environ["TUTORIAL_EXAMPLE_DIR"]) / "customers.csv" if not csv_path.exists(): return dg.AssetCheckResult( passed=False, description=f"Export CSV {csv_path} does not exist" ) rows = len(csv_path.read_text().split("\n")) if rows < 2: return dg.AssetCheckResult( passed=False, description=f"Export CSV {csv_path} is empty", severity=dg.AssetCheckSeverity.WARN, ) return dg.AssetCheckResult( passed=True, description=f"Export CSV {csv_path} exists", metadata={"rows": rows}, )# Define the dbt_project_path function to return the path to the dbt project directorydef dbt_project_path() -> Path: env_val = os.getenv("TUTORIAL_DBT_PROJECT_DIR") assert env_val, "TUTORIAL_DBT_PROJECT_DIR must be set" return Path(env_val)# Use the dbt_assets decorator to define assets for models within the dbt project automatically.@dg_dbt.dbt_assets( manifest=dbt_project_path() / "target" / "manifest.json", project=dg_dbt.DbtProject(dbt_project_path()),)def dbt_project_assets(context: dg.AssetExecutionContext, dbt: dg_dbt.DbtCliResource): yield from dbt.cli(["build"], context=context).stream()mapped_assets = dg_airlift_core.assets_with_task_mappings( dag_id="rebuild_customers_list", task_mappings={ # Define an AssetSpec for the csv file created by the load_raw_customers task. "load_raw_customers": [dg.AssetSpec(key=["raw_data", "raw_customers"])], # We map the assets to the build_dbt_models task which creates them Airflow-side. "build_dbt_models": [dbt_project_assets], # Define an AssetSpec for the csv file created by the export_customers task. "export_customers": [dg.AssetSpec(key="customers_csv", deps=["customers"])], },)defs = dg_airlift_core.build_defs_from_airflow_instance( airflow_instance=dg_airlift_core.AirflowInstance( auth_backend=dg_airlift_core.AirflowBasicAuthBackend( webserver_url="http://localhost:8080", username="admin", password="admin", ), name="airflow_instance_one", ), defs=dg.Definitions( assets=mapped_assets, # We need to pass the dbt resource so that it can be utilized by dbt_project_assets. resources={"dbt": dg_dbt.DbtCliResource(project_dir=dbt_project_path())}, asset_checks=[validate_exported_csv], ),) View observed assets[​](https://docs.dagster.io/migration/airflow-to-dagster/airlift-v1/task-level-migration/observe#view-observed-assets "Direct link to View observed assets") --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Once you have created the three assets above, you should be able to navigate to the UI, reload your Dagster definitions, and see a full representation of the `dbt` project and other data assets in your code: ![Observed asset graph in Dagster](https://docs.dagster.io/images/integrations/airlift/observe.svg) After you initiate a run of the DAG in Airflow, you should see the newly created assets materialize in Dagster as each task completes. info There will be a delay between when tasks complete in Airflow and assets materialize in Dagster, managed by the Dagster sensor. This sensor runs every 30 seconds by default, but you can change this interval using the `minimum_interval_seconds` argument to [`sensor`](https://docs.dagster.io/api/dagster/schedules-sensors#dagster.sensor) , down to a minimum of one second. Update the asset check to the `customers_csv` asset[​](https://docs.dagster.io/migration/airflow-to-dagster/airlift-v1/task-level-migration/observe#update-the-asset-check-to-the-customers_csv-asset "Direct link to update-the-asset-check-to-the-customers_csv-asset") -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Now that we've introduced an asset explicitly for the `customers.csv` file output by the DAG, we should update the asset check constructed during the peering step to point to the `customers_csv` asset. To do this, change the `asset` targeted by the `@asset_check` decorator to `AssetKey(["customers_csv"])`. Updating this asset check ensures that even when the DAG is deleted, the asset check will live on: import osfrom pathlib import Pathimport dagster as dgimport dagster_airlift.core as dg_airlift_coreimport dagster_dbt as dg_dbt# The asset check is now directly associated with the customers_csv asset# rather than checking it through the Airflow DAG asset@dg.asset_check(asset=dg.AssetKey(["customers_csv"]))def validate_exported_csv() -> dg.AssetCheckResult: csv_path = Path(os.environ["TUTORIAL_EXAMPLE_DIR"]) / "customers.csv" if not csv_path.exists(): return dg.AssetCheckResult( passed=False, description=f"Export CSV {csv_path} does not exist" ) rows = len(csv_path.read_text().split("\n")) if rows < 2: return dg.AssetCheckResult( passed=False, description=f"Export CSV {csv_path} is empty", severity=dg.AssetCheckSeverity.WARN, ) return dg.AssetCheckResult( passed=True, description=f"Export CSV {csv_path} exists", metadata={"rows": rows}, ) To see what the full code should look like after the asset check, see the [example code in GitHub](https://github.com/dagster-io/dagster/tree/master/examples/airlift-migration-tutorial/tutorial_example/dagster_defs/stages/observe_check_on_asset.py) . Add partitions[​](https://docs.dagster.io/migration/airflow-to-dagster/airlift-v1/task-level-migration/observe#add-partitions "Direct link to Add partitions") --------------------------------------------------------------------------------------------------------------------------------------------------------------- If your Airflow tasks produce time-partitioned assets, Airlift can automatically associate your materializations to the relevant partitions. In this example, in the `rebuild_customers_list` asset, data is partitioned daily in each created table, and the Airflow DAG runs on a `@daily` cron schedule. We can likewise add a `DailyPartitionsDefinition` to each of our assets: import osfrom pathlib import Pathimport dagster as dgimport dagster_airlift.core as dg_airlift_coreimport dagster_dbt as dg_dbtfrom dagster._time import get_current_datetime_midnight# Define a daily partitioning strategy starting from the current date at midnight# This will be used to partition our assets into daily chunksPARTITIONS_DEF = dg.DailyPartitionsDefinition(start_date=get_current_datetime_midnight())@dg.asset_check(asset=dg.AssetKey(["customers_csv"]))def validate_exported_csv() -> dg.AssetCheckResult: csv_path = Path(os.environ["TUTORIAL_EXAMPLE_DIR"]) / "customers.csv" if not csv_path.exists(): return dg.AssetCheckResult( passed=False, description=f"Export CSV {csv_path} does not exist" ) rows = len(csv_path.read_text().split("\n")) if rows < 2: return dg.AssetCheckResult( passed=False, description=f"Export CSV {csv_path} is empty", severity=dg.AssetCheckSeverity.WARN, ) return dg.AssetCheckResult( passed=True, description=f"Export CSV {csv_path} exists", metadata={"rows": rows}, )def dbt_project_path() -> Path: env_val = os.getenv("TUTORIAL_DBT_PROJECT_DIR") assert env_val, "TUTORIAL_DBT_PROJECT_DIR must be set" return Path(env_val)# Add daily partitioning to the dbt assets@dg_dbt.dbt_assets( manifest=dbt_project_path() / "target" / "manifest.json", project=dg_dbt.DbtProject(dbt_project_path()), partitions_def=PARTITIONS_DEF, # Enable daily partitioning for dbt assets)def dbt_project_assets(context: dg.AssetExecutionContext, dbt: dg_dbt.DbtCliResource): yield from dbt.cli(["build"], context=context).stream()mapped_assets = dg_airlift_core.assets_with_task_mappings( dag_id="rebuild_customers_list", task_mappings={ "load_raw_customers": [ dg.AssetSpec(key=["raw_data", "raw_customers"], partitions_def=PARTITIONS_DEF) ], "build_dbt_models": [dbt_project_assets], "export_customers": [ dg.AssetSpec(key="customers_csv", deps=["customers"], partitions_def=PARTITIONS_DEF) ], },)defs = dg_airlift_core.build_defs_from_airflow_instance( airflow_instance=dg_airlift_core.AirflowInstance( auth_backend=dg_airlift_core.AirflowBasicAuthBackend( webserver_url="http://localhost:8080", username="admin", password="admin", ), name="airflow_instance_one", ), defs=dg.Definitions( assets=mapped_assets, resources={"dbt": dg_dbt.DbtCliResource(project_dir=dbt_project_path())}, asset_checks=[validate_exported_csv], ),) Now, every time the sensor triggers a materialization for an asset, it will automatically have a partition associated with it. You can try this out by kicking off an Airflow backfill for today: airflow dags backfill rebuild_customers_list --start-date $(date +"%Y-%m-%d") After this DAG run completes, you should see a partitioned materialization appear in Dagster: ![Partitioned materialization in Dagster](https://docs.dagster.io/assets/images/partitioned_mat-4898356433cdf9962fa3ce67324866b0.png) Finally, run `airflow db clean` to delete Airflow runs so you can initiate this backfill again for testing in the future: airflow db clean note In order for partitioned assets to work with `dagster-airlift`, the following things need to be true: * The asset can only be time-window partitioned. This means static, dynamic, and multi partitioned definitions will require custom functionality. * The partitioning scheme must match up with the [logical\_date/execution\_date](https://airflow.apache.org/docs/apache-airflow/stable/faq.html#what-does-execution-date-mean) of corresponding Airflow runs. That is, each logical_date should correspond \_exactly_ to a partition in Dagster. Next steps[​](https://docs.dagster.io/migration/airflow-to-dagster/airlift-v1/task-level-migration/observe#next-steps "Direct link to Next steps") --------------------------------------------------------------------------------------------------------------------------------------------------- In the next step, "[Migrate Airflow tasks](https://docs.dagster.io/migration/airflow-to-dagster/airlift-v1/task-level-migration/migrate) ", we will migrate Airflow DAG code to Dagster. * [Create asset specs for Airflow tasks](https://docs.dagster.io/migration/airflow-to-dagster/airlift-v1/task-level-migration/observe#create-asset-specs-for-airflow-tasks) * [View observed assets](https://docs.dagster.io/migration/airflow-to-dagster/airlift-v1/task-level-migration/observe#view-observed-assets) * [Update the asset check to the `customers_csv` asset](https://docs.dagster.io/migration/airflow-to-dagster/airlift-v1/task-level-migration/observe#update-the-asset-check-to-the-customers_csv-asset) * [Add partitions](https://docs.dagster.io/migration/airflow-to-dagster/airlift-v1/task-level-migration/observe#add-partitions) * [Next steps](https://docs.dagster.io/migration/airflow-to-dagster/airlift-v1/task-level-migration/observe#next-steps) --- # 15 docs tagged with "ETL" | Dagster Docs [Skip to main content](https://docs.dagster.io/tags/integrations/etl#__docusaurus_skipToContent_fallback) [Dagster & Airbyte\ -----------------](https://docs.dagster.io/integrations/libraries/airbyte/airbyte-oss) Using this integration, you can trigger Airbyte syncs and orchestrate your Airbyte connections from within Dagster, making it easy to chain an Airbyte sync with upstream or downstream steps in your workflow. [Dagster & Airbyte\ -----------------](https://docs.dagster.io/integrations/libraries/airbyte/) Orchestrate Airbyte connections and schedule syncs alongside upstream or downstream dependencies. [Dagster & Census\ ----------------](https://docs.dagster.io/integrations/libraries/census) With the Census integration you can execute a Census sync and poll until that sync completes, raising an error if it's unsuccessful. [Dagster & dbt\ -------------](https://docs.dagster.io/integrations/libraries/dbt/) Orchestrate your dbt transformations directly with Dagster. [Dagster & dbt Cloud\ -------------------](https://docs.dagster.io/integrations/libraries/dbt/dbt-cloud) Dagster allows you to run dbt Cloud jobs alongside other technologies. You can schedule them to run as a step in a larger pipeline and manage them as a data asset. [Dagster & dbt Cloud (Legacy)\ ----------------------------](https://docs.dagster.io/integrations/libraries/dbt/dbt-cloud-legacy) Dagster allows you to run dbt Cloud jobs alongside other technologies. You can schedule them to run as a step in a larger pipeline and manage them as a data asset. [Dagster & dlt\ -------------](https://docs.dagster.io/integrations/libraries/dlt) The dltHub open-source library defines a standardized approach for creating data pipelines that load often messy data sources into well-structured data sets. [Dagster & Embedded ELT\ ----------------------](https://docs.dagster.io/integrations/libraries/embedded-elt) The Embedded ELT package provides a framework for building ELT pipelines with Dagster through helpful asset decorators and resources. It includes the dagster-dlt and dagster-sling packages, which you can also use on their own. [Dagster & Hightouch\ -------------------](https://docs.dagster.io/integrations/libraries/hightouch) With this integration you can trigger Hightouch syncs and monitor them from within Dagster. Fine-tune when Hightouch syncs kick-off, visualize their dependencies, and monitor the steps in your data activation workflow. [Dagster & Meltano\ -----------------](https://docs.dagster.io/integrations/libraries/meltano) The Meltano library allows you to run Meltano using Dagster. Design and configure ingestion jobs using the popular Singer specification. [Dagster & MSSQL Bulk Copy Tool\ ------------------------------](https://docs.dagster.io/integrations/libraries/mssql-bulk-copy-tool) The community-supported MSSQL BCP package is a custom Dagster I/O manager for loading data into SQL Server using the BCP utility. [Dagster & Ray\ -------------](https://docs.dagster.io/integrations/libraries/ray) The community-supported Ray package allows orchestrating distributed Ray compute from Dagster pipelines. [Dagster & Sling\ ---------------](https://docs.dagster.io/integrations/libraries/sling) Sling provides an easy-to-use YAML configuration layer for loading data from files, replicating data between databases, exporting custom SQL queries to cloud storage, and much more. [Using Dagster with Airbyte Cloud\ --------------------------------](https://docs.dagster.io/integrations/libraries/airbyte/airbyte-cloud) Orchestrate Airbyte Cloud connections and schedule syncs alongside upstream or downstream dependencies. [Using Dagster with Fivetran\ ---------------------------](https://docs.dagster.io/integrations/libraries/fivetran) Orchestrate Fivetran connectors syncs with upstream or downstream dependencies. --- # Upgrading Dagster | Dagster Docs [Skip to main content](https://docs.dagster.io/migration/upgrading#__docusaurus_skipToContent_fallback) On this page When new releases include breaking changes or deprecations, this document explains how to upgrade your projects. Upgrading to 1.11.0[​](https://docs.dagster.io/migration/upgrading#upgrading-to-1110 "Direct link to Upgrading to 1.11.0") --------------------------------------------------------------------------------------------------------------------------- ### Breaking changes[​](https://docs.dagster.io/migration/upgrading#breaking-changes "Direct link to Breaking changes") The `FreshnessPolicy` class (which has been marked as deprecated as of Dagster version 1.6), has been renamed to `LegacyFreshnessPolicy`. The deprecated freshness policies will henceforth be referred to in docs and code as "legacy freshness policies". There are no immediate changes in functionality. Existing imports of `FreshnessPolicy` will fail with an `ImportError`: from dagster import FreshnessPolicy You can still import and use the legacy freshness policies from the `deprecated` module: from dagster.deprecated import FreshnessPolicy # imports LegacyFreshnessPolicy Accordingly, the `freshness_policy` parameter has been renamed to `legacy_freshness_policy` in these public APIs: * `AssetsDefinition.from_graph()` * `AssetsDefinition.from_op()` * `@asset` * `@asset_check` * `AssetSpec.replace_attributes()` * `AssetSpec.merge_attributes()` Other relevant parameter renames: * In `AssetsDefinition.from_op()`, parameter `freshness_policies_by_output_name` is renamed to `legacy_freshness_policies_by_output_name` * In `AssetsDefinition.from_graph()`, parameter `freshness_policies_by_output_name` is renamed to `legacy_freshness_policies_by_output_name` Upgrading to 1.10.0[​](https://docs.dagster.io/migration/upgrading#upgrading-to-1100 "Direct link to Upgrading to 1.10.0") --------------------------------------------------------------------------------------------------------------------------- ### Deprecations[​](https://docs.dagster.io/migration/upgrading#deprecations "Direct link to Deprecations") * We've refreshed our integrations with popular ELT tools like [Fivetran](https://docs.dagster.io/integrations/libraries/fivetran) and [Airbyte](https://docs.dagster.io/integrations/libraries/airbyte/airbyte-oss) to leverage Dagster's asset-based patterns better and provide enhanced visibility into your data pipelines. The old integration patterns are still available, so there are no breaking changes, but we encourage users to take advantage of the new capabilities! ### Breaking changes[​](https://docs.dagster.io/migration/upgrading#breaking-changes-1 "Direct link to Breaking changes") * Pool names now only accept letters, numbers, dashes, and underscores. Upgrading to 1.9.0[​](https://docs.dagster.io/migration/upgrading#upgrading-to-190 "Direct link to Upgrading to 1.9.0") ------------------------------------------------------------------------------------------------------------------------ ### Database migration[​](https://docs.dagster.io/migration/upgrading#database-migration "Direct link to Database migration") * This release includes database schema and data migrations to improve the performance of the Runs page. We highly recommend running these migrations to avoid slow page loads of the new Runs page. The migration will add a new column to the `runs` table, a new column to the `bulk_actions` table and a new `backfill_tags` table. A data migration will populate the new columns and table. Run `dagster instance migrate` to run the schema and data migration. ### Notable behavior changes[​](https://docs.dagster.io/migration/upgrading#notable-behavior-changes "Direct link to Notable behavior changes") * Backfills have been moved from their own tab underneath the Overview page to entries within the table on the Runs page. This reflects the fact that backfills and runs are similar entities that share most properties. You can continue to use the legacy Runs page with the “Revert to legacy Runs page” user setting. ([GitHub Discussion](https://github.com/dagster-io/dagster/discussions/24898) ) * By default, `AutomationConditionSensorDefinitions` will now emit backfills to handle cases where more than one partition of an asset is requested on a given tick. This allows that asset's `BackfillPolicy` to be respected. This feature can be disabled by setting `allow_backfills` to `False` on the sensor definition. ### Deprecations[​](https://docs.dagster.io/migration/upgrading#deprecations-1 "Direct link to Deprecations") * The `DataBricksPysparkStepLauncher`, `EmrPySparkStepLauncher`, and any custom subclass of `StepLauncher` have been marked as deprecated, but will not be removed from the codebase until Dagster 2.0 is released, meaning they will continue to function as they currently do for the foreseeable future. Their functionality has been superseded by the interfaces provided by `dagster-pipes`, and so future development work will be focused there. * The experimental `@multi_asset_sensor` has been marked as deprecated, but will not be removed from the codebase until Dagster 2.0 is released, meaning it will continue to function as it currently does for the foreseeable future. Its functionality has been largely superseded by the `AutomationCondition` system. ### Breaking changes[​](https://docs.dagster.io/migration/upgrading#breaking-changes-2 "Direct link to Breaking changes") * `dagster` no longer supports Python 3.8, which hit EOL on 2024-10-07. * `dagster` now requires `pydantic>=2` . * Passing a custom `PartitionsDefinition` subclass into a `Definitions` object now issues an error instead of a deprecation warning. * `AssetExecutionContext` is no longer a subclass of `OpExecutionContext`. At this release, `AssetExecutionContext` and `OpExecutionContext` implement the same methods, but in the future, the methods implemented by each class may diverge. If you have written helper functions with `OpExecutionContext` type annotations, they may need to be updated to include `AssetExecutionContext` depending on your usage. Explicit calls to `isinstance(context, OpExecutionContext)` will now fail if `context` is an `AssetExecutionContext`. * The `dagster/relation_identifier` metadata key has been renamed to `dagster/table_name`. * The `asset_selection` parameter on `AutomationConditionSensorDefinition` has been renamed to `target`, to align with existing sensor APIs. * The experimental `freshness_policy_sensor` has been removed, as it relies on the long-deprecated `FreshnessPolicy` API. * The deprecated `external_assets_from_specs` and `external_asset_from_spec` methods have been removed. Users should use `AssetsDefinition(specs=[...])`, or pass specs directly into the `Definitions` object instead. * `AssetKey` objects can no longer be iterated over or indexed in to. This behavior was never an intended access pattern and in all observed cases was a mistake. * \[dagster-ge\] `dagster-ge` now only supports `great_expectations>=0.17.15`. The `ge_validation_op_factory` API has been replaced with the API previously called `ge_validation_op_factory_v3`. * \[dagster-aws\] Removed deprecated parameters from `dagster_aws.pipes.PipesGlueClient.run`. * \[dagster-embedded-elt\] Removed deprecated parameter `dlt_dagster_translator` from `@dlt_assets`. The `dagster_dlt_translator` parameter should be used instead. Upgrading to 1.8.0[​](https://docs.dagster.io/migration/upgrading#upgrading-to-180 "Direct link to Upgrading to 1.8.0") ------------------------------------------------------------------------------------------------------------------------ ### Notable behavior changes[​](https://docs.dagster.io/migration/upgrading#notable-behavior-changes-1 "Direct link to Notable behavior changes") * The `Definitions` constructor will no longer raise errors when the provided definitions aren’t mutually resolve-able – e.g. when there are conflicting definitions with the same name, unsatisfied resource dependencies, etc. These errors will still be raised at code location load time. The new `Definitions.validate_loadable` static method also allows performing the validation steps that used to occur in constructor. * The “Unsynced” label on an asset is no longer transitive, i.e. it no longer displays purely on account of a parent asset being labeled “Unsynced”. This helps avoid “Unsynced label fatigue”, where huge portions of the graph often have the label because of a distant ancestor. And it also helps the asset graph UI load faster. * The Run Status column on the Backfills page has been removed. This column was only filled out for backfills of jobs. Users should instead click on the backfill to see the status of each run. * The default behavior for evaluating `AutoMaterializePolicy` and `AutomationCondition` objects has changed. Previously, all assets were evaluated in a single process on the `AssetDaemon` , and evaluation history would show up in the UI in a special-purpose tab. Now, a default `AutomationConditionSensorDefinition` with the name `"default_automation_condition_sensor"` will be constructed for each code location, and a history of evaluations can be accessed by navigating to the page of that sensor. These changes are intended to provide a consistent UI/UX for interacting with automation concepts, and the sensor-based APIs allow for greater isolation between separate sets of assets. * The core work of these sensors is still handled by the `AssetDaemon`, so this will need to continue running for your deployment. * If desired, you can retain the current behavior by setting the following in your `dagster.yaml` file: auto_materialize: use_sensors: true * The `datetime` objects that are exposed in Dagster public APIs are now standard Python `datetime.datetime` objects with timezones, instead of [Pendulum](https://pendulum.eustace.io/docs/) `datetime` objects. Technically, this is not a breaking change since Dagster’s public API uses `datetime.datetime` in our APIs, but Pendulum datetimes expose some methods (like `add` and `subtract`) that are not available on standard `datetime.datetime` objects. If your code was using methods that are only available on `Pendulum` datetimes, you can transform your `datetimes` back to Pendulum datetimes before using them. * For example, an asset like this: from dagster import asset, AssetExecutionContext@assetdef my_asset(context: AssetExecutionContext): window_start, window_end = context.partition_time_window in_an_hour = window_start.add(hours=1) # will break since add() is only defined in pendulum * could be changed to this in order to continue using pendulum datetimes: from dagster import asset, AssetExecutionContextimport pendulum@assetdef my_asset(context: AssetExecutionContext): window_start, window_end = context.partition_time_window window_start = pendulum.instance(window_start) # transform to a pendulum time in_an_hour = window_start.add(hours=1) # will continue working ### Breaking changes[​](https://docs.dagster.io/migration/upgrading#breaking-changes-3 "Direct link to Breaking changes") * `AutoMaterializeSensorDefinition` has been renamed to `AutomationConditionSensorDefinition`. All other functionality is identical. * “Op job versioning and memoization”, an experimental and deprecated pre-1.0 feature, has been removed. This feature has been superseded for a long time by `code_version` , data versions, and automation conditions. `MemoizableIOManager`, `VersionStrategy`, `SourceHashVersionStrategy`, `OpVersionContext`, `ResourceVersionContext`, and `MEMOIZED_RUN_TAG` have been removed. * The experimental and deprecated `build_asset_with_blocking_check` has been removed. Use the `blocking` argument on `@asset_check` instead. * \[dagster-dbt\] Support for setting freshness policies through dbt metadata on field `+meta.dagster_freshness_policy` has been removed. Use `+meta.dagster.freshness_policy` instead. * \[dagster-dbt\] `KeyPrefixDagsterDbtTranslator` has been removed. To modify the asset keys for a set of dbt assets, implement`DagsterDbtTranslator.get_asset_key()` instead. * \[dagster-dbt\] Support for setting auto-materialize policies through dbt metadata on field `+meta.dagster_auto_materialize_policy` has been removed. Use `+meta.dagster.auto_materialize_policy` instead. * \[dagster-dbt\] Support for `dbt-core==1.6.*` has been removed because the version is now end-of-life. * \[dagster-dbt\] Support for `load_assets_from_dbt_project`, `load_assets_from_dbt_manifest`, and `dbt_cli_resource` has been removed. Use `@dbt_assets`, `DbtCliResource`, and `DbtProject` instead to define how to load dbt assets from a dbt project and to execute them. * \[dagster-dbt\] Support for rebuilt ops like `dbt_run_op`, `dbt_compile_op`, etc has been removed. Use `@op` and `DbtCliResource` directly to execute dbt commands in an op. ### Deprecations[​](https://docs.dagster.io/migration/upgrading#deprecations-2 "Direct link to Deprecations") * The experimental `external_assets_from_specs` API has been deprecated. Instead, you can directly pass `AssetSpec` objects to the `assets` argument of the `Definitions` constructor. * `AutoMaterializePolicy`, `AutoMaterializeRule`, and the `auto_materialize_policy` arguments to `@asset` and `AssetSpec` have been marked as deprecated, and the new `AutomationCondition` API and `automation_condition` argument should be used instead. These changes are intended to provide a more consistent, composable, and flexible experience for users interested in asset-focused automation. A full migration guide can be found [here](https://github.com/dagster-io/dagster/discussions/23495) , and a more detailed explanation of the thought process behind these changes can be found in the [original RFC](https://github.com/dagster-io/dagster/discussions/22811) . * `AutoMaterializePolicys` and `AutomationConditions` can interoperate without issue, meaning you do not need to migrate all assets at the same time. * The `partitions_def` parameter on `define_asset_job` is now deprecated. The `partitions_def` for an asset job is determined from the `partitions_def` attributes on the assets it targets, so this parameter is redundant. * The `asset_partition_key_for_output`, `asset_partition_keys_for_output`, and `asset_partition_key_range_for_output`, and `asset_partitions_time_window_for_output` methods on `OpExecutionContext` have been deprecated. Instead, use the corresponding property: `partition_key`, `partition_keys`, `partition_key_range`, or `partition_time_window`. * `SourceAsset` is deprecated, in favor of `AssetSpec`. You can now use `AssetSpec`s in any of the places you could previously use `SourceAsset`s, including passing them to the `assets` argument of `Definitions`, passing them to the `assets` argument of `materialize`, and supplying them as inputs in op graphs. `AssetSpec` has all the properties that `SourceAsset` does, except for `io_manager_key`. To set an IO manager key on an `AssetSpec`, you can supply a metadata entry with the `"dagster/io_manager_key"` key: # beforefrom dagster import SourceAssetmy_asset = SourceAsset("my_asset", io_manager_key="abc")# afterfrom dagster import AssetSpecmy_asset = AssetSpec("my_asset", metadata={"dagster/io_manager_key": "abc"}) * \[dagster-shell\] The `dagster-shell` package, which exposes `create_shell_command_op` and `create_shell_script_op`, has been deprecated. Instead, use `PipesSubprocessClient`, from the `dagster` package. * \[dagster-airbyte\] `load_assets_from_airbyte_project` is now deprecated, because the Octavia CLI that it relies on is an experimental feature that is no longer supported. Use `build_airbyte_assets` or `load_assets_from_airbyte_project` instead. Upgrading to 1.7.0[​](https://docs.dagster.io/migration/upgrading#upgrading-to-170 "Direct link to Upgrading to 1.7.0") ------------------------------------------------------------------------------------------------------------------------ ### Breaking Changes[​](https://docs.dagster.io/migration/upgrading#breaking-changes-4 "Direct link to Breaking Changes") * Creating a run with a custom non-UUID `run_id` was previously private and only used for testing. It will now raise an exception. * \[community-contribution\] Previously, calling `get_partition_keys_in_range` on a `MultiPartitionsDefinition` would erroneously return partition keys that were within the one-dimensional range of alphabetically-sorted partition keys for the definition. Now, this method returns the cartesian product of partition keys within each dimension’s range. Thanks, [@mst](https://github.com/mst) ! * Added `AssetCheckExecutionContext` to replace `AssetExecutionContext` as the type of the `context` param passed in to `@asset_check` functions. `@asset_check` was an experimental decorator. * \[experimental\] `@classmethod` decorators have been removed from `[dagster-embedded-slt.sling](http://dagster-embedded-slt.sling)` `DagsterSlingTranslator` * \[dagster-dbt\] `@classmethod` decorators have been removed from `DagsterDbtTranslator`. * \[dagster-k8s\] The default merge behavior when raw kubernetes config is supplied at multiple scopes (for example, at the instance level and for a particluar job) has been changed to be more consistent. Previously, configuration was merged shallowly by default, with fields replacing other fields instead of appending or merging. Now, it is merged deeply by default, with lists appended to each other and dictionaries merged, in order to be more consistent with how kubernetes configuration is combined in all other places. See [the docs](https://docs.dagster.io/deployment/guides/kubernetes/customizing-your-deployment#precedence-rules) for more information, including how to restore the previous default merge behavior. ### Deprecations[​](https://docs.dagster.io/migration/upgrading#deprecations-3 "Direct link to Deprecations") * `AssetSelection.keys()` has been deprecated. Instead, you can now supply asset key arguments to `AssetSelection.assets()` . * Run tag keys with long lengths and certain characters are now deprecated. For consistency with asset tags, run tags keys are expected to only contain alpha-numeric characters, dashes, underscores, and periods. Run tag keys can also contain a prefix section, separated with a slash. The main section and prefix section of a run tag are limited to 63 characters. * `AssetExecutionContext` has been simplified. Op-related methods and methods with existing access paths have been marked deprecated. For a full list of deprecated methods see this [GitHub Discussion](https://github.com/dagster-io/dagster/discussions/20974) . * The `metadata` property on `InputContext` and `OutputContext` has been deprecated and renamed to `definition_metadata` . * `FreshnessPolicy` is now deprecated. For monitoring freshness, use freshness checks instead. If you are using `AutoMaterializePolicy.lazy()`, `FreshnessPolicy` is still recommended, and will continue to be supported until an alternative is provided. Upgrading to 1.6.0[​](https://docs.dagster.io/migration/upgrading#upgrading-to-160 "Direct link to Upgrading to 1.6.0") ------------------------------------------------------------------------------------------------------------------------ ### Breaking changes[​](https://docs.dagster.io/migration/upgrading#breaking-changes-5 "Direct link to Breaking changes") #### Dagster Ingestion-as-Code is being deprecated[​](https://docs.dagster.io/migration/upgrading#dagster-ingestion-as-code-is-being-deprecated "Direct link to Dagster Ingestion-as-Code is being deprecated") With Dagster 1.1.8, we launched experimental “[ingestion-as-code](https://docs.dagster.io/guides/dagster/airbyte-ingestion-as-code) ” functionality for our Airbyte integration, in response to user feedback that users would like to manage their Airbyte connections in code. In the months since, Airbyte has released an official [Terraform provider](https://registry.terraform.io/providers/airbytehq/airbyte/latest/docs) which accomplishes many of the same goals, making ingestion-as-code largely redundant. In light of this, we will no longer be publishing new versions of the `dagster-managed-elements` package. `dagster_airbyte.AirbyteManagedElementReconciler` and objects in `dagster_airbyte.managed.*` will be removed. We suggest that users consider the official Terraform provider if they would like to continue managing their connections in code. #### I/O manager `handle_output` will no longer be called when the output typing type is Nothing[​](https://docs.dagster.io/migration/upgrading#io-manager-handle_output-will-no-longer-be-called-when-the-output-typing-type-is-nothing "Direct link to io-manager-handle_output-will-no-longer-be-called-when-the-output-typing-type-is-nothing") Most Dagster-maintained I/O managers include special logic that does not store outputs typed as `None` or `Nothing` (either via return type annotation or explicitly setting the type in `Out`). In 1.6, the Dagster framework will no longer invoke the `IOManager.handle_output` at all for outputs with these types. This ensures that I/O managers behave consistently and protects against storing unnecessary `None` s in storage. For some I/O managers, e.g. the `InMemoryIOManager` and some user-developed I/O managers, this change may result in input-loading errors when assets downstream try to use the default IO manager to load the upstream output: @assetdef upstream() -> None: # when this asset is materialized, no `None` value will be stored@assetdef downstream(upstream): # if the default IO manager is the InMemoryIOManager, then, when this asset # is executed, it will hit a load_input error because it can't find the # stored value corresponding to "upstream" The best way to avoid these errors is to write the downstream asset in a way that `IOManager.load_input` won’t be invoked: @asset(deps=[upstream])def downstream(): # because `deps` is used instead of a function argument, # IOManager.load_input won't be invoked ### Deprecations[​](https://docs.dagster.io/migration/upgrading#deprecations-4 "Direct link to Deprecations") #### dbt[​](https://docs.dagster.io/migration/upgrading#dbt "Direct link to dbt") * Prebuilt ops for executing common dbt Core operations (e.g. `dbt_build_op`, `dbt_compile_op`, …) have been marked as deprecated. Instead, we recommend creating your op using the `@op` decorator and `DbtCliResource` directly. * `load_assets_from_dbt_manifest` and `load_assets_from_dbt_project` have been marked as deprecated. Instead, we recommend using `@dbt_assets`, `DbtCliResource`, and `DagsterDbtTranslator`. * For examples on how to use `@dbt_assets` and `DbtCliResource` to execute commands like `dbt run` or `dbt build` on your dbt project, see our [API docs](https://docs.dagster.io/_apidocs/libraries/dagster-dbt#dagster_dbt.dbt_assets) . * For examples on how to customize your dbt software-defined assets using `DagsterDbtTranslator`, see the [reference](https://docs.dagster.io/integrations/dbt/reference#understanding-asset-definition-attributes) . * To replicate the behavior of `load_assets_from_dbt_project`, which generates a dbt manifest at run time using `dbt parse`, see the [reference](https://docs.dagster.io/integrations/dbt/reference#loading-dbt-models-from-a-dbt-project) . * To replicate the behavior of `load_assets_from_dbt_manifest`: # Before, using `load_assets_from_dbt_manifest`from dagster_dbt import load_assets_from_dbt_manifestmy_dbt_assets = load_assets_from_dbt_manifest( manifest=manifest, use_build_command=True,)# After, using `@dbt_assets`, `DbtCliResource`, and `DagsterDbtTranslatorfrom dagster import AssetExecutionContextfrom dagster_dbt import dbt_assets, DbtCliResource@dbt_assets(manifest=manifest)def my_dbt_assets(context: AssetExecutionContext, dbt: DbtCliResource): yield from dbt.cli(["build"], context=context).stream() * When using `@dbt_assets`, if a time window partition definition is used without an explicit backfill policy, the backfill policy now defaults to a `BackfillPolicy.single_run()` instead of `BackfillPolicy.multi_run()`. Upgrading to 1.5.0[​](https://docs.dagster.io/migration/upgrading#upgrading-to-150 "Direct link to Upgrading to 1.5.0") ------------------------------------------------------------------------------------------------------------------------ ### Breaking changes[​](https://docs.dagster.io/migration/upgrading#breaking-changes-6 "Direct link to Breaking changes") * The UI dialog for launching a backfill no longer includes a toggle to determine whether the backfill is launched as a single run or multiple runs. This toggle was misleading, because it implied that all backfills could be launched as single-run backfills, when it actually required special handling in the implementations of the assets targeted by the backfill to achieve this behavior. Instead, whether to execute a backfill as a single run is now determined by a setting on the asset definition. To enable single-run backfills, set `backfill_policy=BackfillPolicy.single_run()` on the asset definitions. Refer to the [docs on single-run backfills](https://docs.dagster.io/concepts/partitions-schedules-sensors/backfills#single-run-backfills) for more information. * `AssetExecutionContext` is now a subclass of `OpExecutionContext`, not a type alias. The code def my_helper_function(context: AssetExecutionContext): ...@opdef my_op(context: OpExecutionContext): my_helper_function(context) will cause type checking errors. To migrate, update type hints to respect the new subclassing. * `AssetExecutionContext` cannot be used as the type annotation for `@op`s. To migrate, update the type hint in `@op` to `OpExecutionContext`. `@op`s that are used in `@graph_assets` may still use the `AssetExecutionContext` type hint. # old@opdef my_op(context: AssetExecutionContext): ...# correct@opdef my_op(context: OpExecutionContext): ... * `AssetCheckResult(success=True)` is renamed to `AssetCheckResult(passed=True)` * Asset checks defined with Dagster version 1.4 will no longer work with Dagster Cloud, or with Dagster UI 1.5. Upgrade your `dagster` library to continue using checks. Upgrading to 1.4.0[​](https://docs.dagster.io/migration/upgrading#upgrading-to-140 "Direct link to Upgrading to 1.4.0") ------------------------------------------------------------------------------------------------------------------------ ### Deprecations[​](https://docs.dagster.io/migration/upgrading#deprecations-5 "Direct link to Deprecations") * The `dagit` python package and all references to it are now deprecated. We will continue to publish `dagit` and support APIs that used the term “dagit” until v2.0, but you should transition to newer `dagster-webserver` package. This is a drop-in replacement for `dagit`. Like `dagit`, it exposes an executable of the same name as the package itself, i.e. `dagster-webserver`. * Any Dockerfiles or other Python environment specifications used for running the webserver now use `dagster-webserver` instead, e.g.: # no (deprecated)RUN pip install dagster dagit ......ENTRYPOINT ["dagit", "-h", "0.0.0.0", "-p", "3000"]# yesRUN pip install dagster dagster-webserver...ENTRYPOINT ["dagster-webserver", "-h", "0.0.0.0", "-p", "3000"] * \[Helm Chart\] Three fields that were using the term “dagit” have been deprecated and replaced with “dagsterWebserver” instead: # no (deprecated)dagit: ... # ...ingress: dagit: ... readOnlyDagit: ...# yesdagsterWebserver: ... # ...ingress: dagsterWebserver: ... readOnlyDagsterWebserver: ... * We’ve deprecated the `non_argument_deps` parameter of `@asset` and `@multi_asset` in favor of a new `deps` parameter. To update your code to use `deps`, simply rename any instances of `non_argument_deps` to `deps` and change the type from a set to list. Additionally, you may also want to begin passing the python symbols for assets, rather than their `AssetKey`s to improve in-editor experience with type-aheads and linting. @assetdef my_asset(): ...@asset( non_argument_deps={"my_asset"})def a_downstream_asset(): ...# becomes@assetdef my_asset(): ...@asset( deps=["my_asset"])def a_downstream_asset(): ...# or@assetdef my_asset(): ...@asset( deps=[my_asset])def a_downstream_asset(): ... * \[Dagster Cloud ECS Agent\] We've introduced performance improvements that rely on the [AWS Resource Groups Tagging API](https://docs.aws.amazon.com/resourcegroupstagging/latest/APIReference/overview.html) . To enable, grant your agent's IAM policy permission to `tag:GetResources`. Without this policy, the ECS Agent will log a deprecation warning and fall back to its old behavior (listing all ECS services in the cluster and then listing each service's tags). * \[dagster-dbt\] `DbtCliClientResource`, `dbt_cli_resource` and `DbtCliOutput` are now being deprecated in favor of `DbtCliResource`. `dagster-dbt` Asset APIs like `load_assets_from_dbt_manifest` and `load_assets_from_dbt_project` will continue to work if given either a `DbtCliClientResource` or `DbtCliResource`. # old@opdef my_dbt_op(dbt_resource: DbtCliClientResource): dbt: DbtCliClient = dbt.get_client() dbt.cli("run") dbt.cli("run", full_refresh=True) dbt.cli("test") manifest_json = dbt.get_manifest_json()# newwith Path("my/dbt/manifest").open() as handle: manifest = json.loads(dbt_manifest.read())@opdef my_dbt_op(dbt: DbtCliResource): dbt.cli(["run"], manifest=manifest).stream() dbt.cli(["run", "--full-refresh"], manifest=manifest).stream() dbt_test_invocation = dbt.cli(["test"], manifest_manifest).stream() manifest_json = dbt_test_invocation.get_artifact("manifest.json")# olddbt_assets = load_assets_from_dbt_project(project_dir="my/dbt/project")defs = Definitions( assets=dbt_assets, resources={ "dbt": DbtCliClientResource(project_dir="my/dbt/project") },)# newdbt_assets = load_assets_from_dbt_project(project_dir="my/dbt/project")defs = Definitions( assets=dbt_assets, resources={ "dbt": DbtCliResource(project_dir="my/dbt/project") }) * The following arguments on `load_assets_from_dbt_project` and `load_assets_from_dbt_manifest` are now deprecated in favor of other options. Arguments will continue to work when passed into these functions, but a deprecation warning will be emitted. | Deprecated Arguments | Recommendation | | --- | --- | | `key_prefix` | Instead, provide a custom `DagsterDbtTranslator` that overrides `get_asset_key` | | `source_key_prefix` | Instead, provide a custom `DagsterDbtTranslator` that overrides `get_asset_key` | | `op_name` | Use the `@dbt_assets` decorator if you need to customize your op name. | | `manifest_json` | Use the `manifest` parameter instead. | | `display_raw_sql` | Instead, provide a custom `DagsterDbtTranslator` that overrides `get_description`. | | `selected_unique_ids` | Use the `select` parameter instead. | | `dbt_resource_key` | Use the `@dbt_assets` decorator if you need to customize your resource key. | | `use_build_command` | Use the `@dbt_assets` decorator if you need to customize the underlying dbt commands. | | `partitions_def` | Use the `@dbt_assets` decorator to define partitioned dbt assets. | | `partition_key_to_vars_fn` | Use the `@dbt_assets` decorator to define partitioned dbt assets. | | `runtime_metadata_fn` | Use the `@dbt_assets` decorator if you need to customize runtime metadata. | | `node_info_to_asset_key_fn` | Instead, provide a custom `DagsterDbtTranslator` that overrides `get_asset_key`. | | `node_info_to_group_fn` | Instead, configure dagster groups on a dbt resource's meta field, assign dbt groups, or provide a custom `DagsterDbtTranslator` that overrides `get_group_name`. | | `node_info_to_auto_materialize_policy_fn` | Instead, configure Dagster auto-materialize policies on a dbt resource's meta field. | | `node_info_to_freshness_policy_fn` | Instead, configure Dagster freshness policies on a dbt resource's meta field. | | `node_info_to_definition_metadata_fn` | Instead, provide a custom `DagsterDbtTranslator` that overrides `get_metadata`. | ### Breaking changes[​](https://docs.dagster.io/migration/upgrading#breaking-changes-7 "Direct link to Breaking changes") * From this release forward Dagster will no longer be tested against Python 3.7. Python 3.7 reached end of life on June 27th 2023 meaning it will no longer receive any security fixes. Previously releases will continue to work on 3.7. Details about moving to 3.8 or beyond can be found at [https://docs.python.org/3/whatsnew/3.8.html#porting-to-python-3-8](https://docs.python.org/3/whatsnew/3.8.html#porting-to-python-3-8) . * `build_asset_reconciliation_sensor` (Experimental) has been removed. It was deprecated in 1.3 in favor of `AutoMaterializePolicy`. Docs are [here](https://docs.dagster.io/concepts/assets/asset-auto-execution) . * The `dagster-dbt` integration with `dbt-rpc` has been removed, as [the dbt plugin is being deprecated](https://github.com/dbt-labs/dbt-rpc) . * Previously, `DbtCliResource` was a class alias for `DbtCliClientResource`. Now, `DbtCliResource` is a new resource with a different API. Furthermore, it requires at least `dbt-core>=1.4` to run. * \[Helm Chart\] If upgrading an existing installation to 1.4 and the `dagit.nameOverride` value is set, you will need to either change the value or delete the existing deployment to allow helm to update values that can not be patched for the rename from dagit to dagster-webserver. * \[dagster-dbt\] `load_assets_from_dbt_project` and `load_assets_from_dbt_manifest` now default to `use_build=True`. To switch back to the previous behavior, use `use_build=False`. from dagster_dbt import group_from_dbt_resource_props_fallback_to_directoryload_assets_from_dbt_project( ..., use_build=False,) * \[dagster-dbt\] The default assignment of groups to dbt models loaded from `load_assets_from_dbt_project` and `load_assets_from_dbt_manifest` has changed. Rather than assigning a group name using the model’s subdirectory, a group name will be assigned using the dbt model’s [dbt group](https://docs.getdbt.com/docs/build/groups) . To switch back to the previous behavior, use the following utility function, `group_from_dbt_resource_props_fallback_to_directory`: from dagster_dbt import group_from_dbt_resource_props_fallback_to_directoryload_assets_from_dbt_project( ..., node_info_to_group_fn=group_from_dbt_resource_props_fallback_to_directory,) * \[dagster-dbt\] The argument `node_info_to_definition_metadata_fn` for `load_assets_from_dbt_project` and `load_assets_from_dbt_manifest` now overrides metadata instead of adding to it. To switch back to the previous behavior, use the following utility function: from dagster_dbt import default_metadata_from_dbt_resource_propsdef my_metadata_from_dbt_resource_props(dbt_resource_props): my_metadata = {...} return {**default_metadata_from_dbt_resource_props(dbt_resource_props), **my_metadata}load_assets_from_dbt_manifest( ..., node_info_to_definition_metadata_fn=my_metadata_from_dbt_resource_props) * \[dagster-dbt\] The arguments for `load_assets_from_dbt_project` and `load_assets_from_dbt_manifest` now must be specified using keyword arguments. * \[dagster-dbt\] When using the new `DbtCliResource` with `load_assets_from_dbt_project` and `load_assets_from_dbt_manifest`, stdout logs from the dbt process will now appear in the compute logs instead of the event logs. To view these compute logs, you should ensure that your Dagster instance has [compute log storage configured](https://docs.dagster.io/deployment/dagster-instance#compute-log-storage) . Upgrading to 1.3.0[​](https://docs.dagster.io/migration/upgrading#upgrading-to-130 "Direct link to Upgrading to 1.3.0") ------------------------------------------------------------------------------------------------------------------------ ### Deprecations[​](https://docs.dagster.io/migration/upgrading#deprecations-6 "Direct link to Deprecations") * **\[deprecation, 1.4.0\]** `build_asset_reconciliation_sensor`, which was experimental, is now deprecated, in favor of setting `AutoMaterializePolicy` on assets. Refer to the docs on `AutoMaterializePolicy` for how this works: [https://docs.dagster.io/concepts/assets/asset-auto-execution](https://docs.dagster.io/concepts/assets/asset-auto-execution) . * **\[deprecation, 2.0.0\]** Previously, the recommended pattern for creating a run request for a given partition of a job within a sensor was `yield job_def.run_request_for_partition(partition_key="...")`. This has been deprecated, in favor of `yield RunRequest(partition_key="...")`. ### Breaking Changes[​](https://docs.dagster.io/migration/upgrading#breaking-changes-8 "Direct link to Breaking Changes") * By default, resources defined on `Definitions` are now automatically bound to jobs. This will only result in a change in behavior if you a) have a job with no "io\_manager" defined in its `resource_defs` and b) have supplied an `IOManager` with key "io\_manager" to the `resource_defs` argument of your `Definitions`. Prior to 1.3.0, this would result in the job using the default filesystem-based `IOManager` for the key "io\_manager". In 1.3.0, this will result in the "io\_manager" supplied to your `Definitions` being used instead. The `BindResourcesToJobs` wrapper, introduced in 1.2 to simulate this behavior, no longer has any effect. * **\[experimental\]** The `minutes_late` and `previous_minutes_late` properties on the experimental `FreshnesPolicySensorContext` have been renamed to `minutes_overdue` and `previous_minutes_overdue`, respectively. * **\[previously deprecated, 0.15.0\]** The `metadata_entries` arguments to user-constructed events (`AssetObservation`,  `AssetMaterialization`,  `ExpectationResult`,  `TypeCheck`,  `Failure`,  `Output`,  `DynamicOutput`), as well as the `DagsterType` object have been removed. Instead, a dictionary of metadata should be passed into the `metadata` argument. * **\[dagster-celery-k8s\]** The default kubernetes namespace for run pods when using the Dagster Helm chart with the `CeleryK8sRunLauncher` is now the same namespace as the Helm chart, instead of the `default` namespace. To restore the previous behavior, you can set the `celeryK8sRunLauncher.jobNamespace` field to the string `default`. * **\[dagster-snowflake-pandas\]** Prior to `dagster-snowflake` version `0.19.0` the Snowflake I/O manager converted all timestamp data to strings before loading the data in Snowflake, and did the opposite conversion when fetching a DataFrame from Snowflake. The I/O manager now ensures timestamp data has a timezone attached and stores the data as TIMESTAMP\_NTZ(9) type. If you used the Snowflake I/O manager prior to version `0.19.0` you can set the `store_timestamps_as_strings=True` configuration value for the Snowflake I/O manager to continue storing time data as strings while you do table migrations. To migrate a table created prior to `0.19.0` to one with a TIMESTAMP\_NTZ(9) type, you can run the follow SQL queries in Snowflake. In the example, our table is located at `database.schema.table` and the column we want to migrate is called `time`: // Add a column of type TIMESTAMP_NTZ(9)ALTER TABLE database.schema.tableADD COLUMN time_copy TIMESTAMP_NTZ(9)// copy the data from time and convert to timestamp dataUPDATE database.schema.tableSET time_copy = to_timestamp_ntz(time)// drop the time columnALTER TABLE database.schema.tableDROP COLUMN time// rename the time_copy column to timeALTER TABLER database.schema.tableRENAME COLUMN time_copy TO time Upgrading to 1.2.0[​](https://docs.dagster.io/migration/upgrading#upgrading-to-120 "Direct link to Upgrading to 1.2.0") ------------------------------------------------------------------------------------------------------------------------ ### Database migration[​](https://docs.dagster.io/migration/upgrading#database-migration-1 "Direct link to Database migration") 1.2.0 adds a set of optional database schema migrations, which can be run via `dagster instance migrate`: * Improves Dagit performance by adding a database index which should speed up job run views. * Enables dynamic partitions definitions by creating a database table to store partition keys. This feature is experimental and may require future migrations. * Adds a primary key `id` column to the `kvs`, `daemon_heartbeats` and `instance_info` tables, enforcing that all tables have a primary key. ### Breaking changes[​](https://docs.dagster.io/migration/upgrading#breaking-changes-9 "Direct link to Breaking changes") #### Core changes[​](https://docs.dagster.io/migration/upgrading#core-changes "Direct link to Core changes") * The minimum `grpcio` version supported by Dagster has been increased to 1.44.0 so that Dagster can support both `protobuf` 3 and `protobuf` 4. Similarly, the minimum `protobuf` version supported by Dagster has been increased to 3.20.0. We are working closely with the gRPC team on resolving the upstream issues keeping the upper-bound `grpcio` pin in place in Dagster, and hope to be able to remove it very soon. * Prior to 0.9.19, asset keys were serialized in a legacy format. This release removes support for querying asset events serialized with this legacy format. Contact #dagster-support for tooling to migrate legacy events to the supported version. Users who began using assets after 0.9.19 will not be affected by this change. #### Changes to experimental APIs[​](https://docs.dagster.io/migration/upgrading#changes-to-experimental-apis "Direct link to Changes to experimental APIs") * \[experimental\] `LogicalVersion` has been renamed to `DataVersion` and `LogicalVersionProvenance` has been renamed to `DataProvenance`. * \[experimental\] Methods on the experimental `DynamicPartitionsDefinition` to add, remove, and check for existence of partitions have been removed. Refer to documentation for updated API methods. #### Removal of deprecated APIs[​](https://docs.dagster.io/migration/upgrading#removal-of-deprecated-apis "Direct link to Removal of deprecated APIs") * \[previously deprecated, 0.15.0\] Static constructors on `MetadataEntry` have been removed. * \[previously deprecated, 1.0.0\] `DagsterTypeMaterializer`, `DagsterTypeMaterializerContext`, and `@dagster_type_materializer` have been removed. * \[previously deprecated, 1.0.0\] `PartitionScheduleDefinition` has been removed. * \[previously deprecated, 1.0.0\] `RunRecord.pipeline_run` has been removed (use `RunRecord.dagster_run`). * \[previously deprecated, 1.0.0\] `DependencyDefinition.solid` has been removed (use `DependencyDefinition.node`). * \[previously deprecated, 1.0.0\] The `pipeline_run` argument to `build_resources` has been removed (use `dagster_run`) #### Extension Libraries[​](https://docs.dagster.io/migration/upgrading#extension-libraries "Direct link to Extension Libraries") * \[dagster-snowflake\] The `execute_query`and `execute_queries` methods of the `SnowflakeResource` now have consistent behavior based on the values of the `fetch_results` and `use_pandas_result` parameters. If `fetch_results` is True, the standard Snowflake result will be returned. If `fetch_results` and `use_pandas_result` are True, a pandas DataFrame will be returned. If `fetch_results` is False and `use_pandas_result` is True, an error will be raised. If both are False, no result will be returned. * \[dagster-snowflake\] The `execute_queries` command now returns a list of DataFrames when `use_pandas_result` is True, rather than appending the results of each query to a single DataFrame. * \[dagster-shell\] The default behavior of the `execute` and `execute_shell_command` functions is now to include any environment variables in the calling op. To restore the previous behavior, you can pass in `env={}` to these functions. * \[dagster-k8s\] Several Dagster features that were previously disabled by default in the Dagster Helm chart are now enabled by default. These features are: * The [run queue](https://docs.dagster.io/deployment/run-coordinator#limiting-run-concurrency) (by default, without a limit). Runs will now always be launched from the Daemon. * Run queue parallelism - by default, up to 4 runs can now be pulled off of the queue at a time (as long as the global run limit or tag-based concurrency limits are not exceeded). * [Run retries](https://docs.dagster.io/deployment/run-retries#run-retries) - runs will now retry if they have the `dagster/max_retries` tag set. You can configure a global number of retries in the Helm chart by setting `run_retries.max_retries` to a value greater than the default of 0. * Schedule and sensor parallelism - by default, the daemon will now run up to 4 sensors and up to 4 schedules in parallel. * [Run monitoring](https://docs.dagster.io/deployment/run-monitoring) - Dagster will detect hanging runs and move them into a FAILURE state for you (or start a retry for you if the run is configured to allow retries). By default, runs that have been in STARTING for more than 5 minutes will be assumed to be hanging and will be terminated. Each of these features can be disabled in the Helm chart to restore the previous behavior. * \[dagster-k8s\] The experimental `[k8s_job_op](https://docs.dagster.io/_apidocs/libraries/dagster-k8s#dagster_k8s.k8s_job_op)` op and `[execute_k8s_job](https://docs.dagster.io/_apidocs/libraries/dagster-k8s#dagster_k8s.execute_k8s_job)` functions no longer automatically include configuration from a `dagster-k8s/config` tag on the Dagster job in the launched Kubernetes job. To include raw Kubernetes configuration in a `k8s_job_op`, you can set the `container_config`, `pod_template_spec_metadata`, `pod_spec_config`, or `job_metadata` config fields on the `k8s_job_op` (or arguments to the `execute_k8s_job` function). * \[dagster-databricks\] The integration has now been refactored to support the official Databricks API. * `create_databricks_job_op` is now deprecated. To submit one-off runs of Databricks tasks, you must now use the `create_databricks_submit_run_op`. * The Databricks token that is passed to the `databricks_client` resource must now begin with `https://`. Upgrading to 1.1.1[​](https://docs.dagster.io/migration/upgrading#upgrading-to-111 "Direct link to Upgrading to 1.1.1") ------------------------------------------------------------------------------------------------------------------------ ### Database migration[​](https://docs.dagster.io/migration/upgrading#database-migration-2 "Direct link to Database migration") Two optional database schema migrations, which can be run via `dagster instance migrate`: * Improves Dagit performance by adding database indexes which should speed up the run view as well as a range of asset-based queries. * Enables multi-dimensional asset partitions and asset versioning. ### Breaking changes and deprecations[​](https://docs.dagster.io/migration/upgrading#breaking-changes-and-deprecations "Direct link to Breaking changes and deprecations") * `define_dagstermill_solid`, a legacy API, has been removed from `dagstermill`. Use `define_dagstermill_op` or `define_dagstermill_asset` instead to create an `op` or `asset` from a Jupyter notebook, respectively. * The internal `ComputeLogManager` API is marked as deprecated in favor of an updated interface: `CapturedLogManager`. It will be removed in `1.2.0`. This should only affect dagster instances that have implemented a custom compute log manager. Upgrading to 1.0[​](https://docs.dagster.io/migration/upgrading#upgrading-to-10 "Direct link to Upgrading to 1.0") ------------------------------------------------------------------------------------------------------------------- * Most of the classes and decorators in Dagster have moved to using a bare asterisk argument, enforcing that arguments are provided as keywords. **If using long lists of non-keyword arguments with dagster decorators or classes, you will likely run into errors in 1.0.** This can be fixed by switching to using keyword arguments. * In an upcoming 1.x release, we plan to make a change that renders values supplied to `configured` in Dagit. Up through this point, values provided to `configured` have not been sent anywhere outside the process where they were used. This change will mean that, like other places you can supply configuration, `configured` is not a good place to put secrets: **You should not include any values in configuration that you don't want to be stored in the Dagster database and displayed inside Dagit.** * **All submodules of dagster have been marked private.** We currently provide aliasing to avoid incurring linting errors, but in a future 1.x release, this will be removed, and imports from submodules of dagster may incur errors. * The `dagster.experimental` submodule has been deleted, which previously contained dynamic output APIs, which are available from the top level of the `dagster` module. * As of 1.0, **Dagster no longer guarantees support for python 3.6.** This is in line with [PEP 494](https://peps.python.org/pep-0494/) , which outlines that 3.6 has reached end of life. * Dagster’s integration libraries haven’t yet achieved the same API maturity as Dagster core. For this reason, all integration libraries will remain on a pre-1.0 (0.16.x) versioning track for the time being. However, 0.16.x library releases remain fully compatible with Dagster 1.x. In the coming months, we will graduate integration libraries one-by-one to the 1.x versioning track as they achieve API maturity. If you have installs of the form: pip install dagster=={DAGSTER_VERSION} dagster-somelibrary=={DAGSTER_VERSION} this should be converted to: pip install dagster=={DAGSTER_VERSION} dagster-somelibrary to make sure the correct library version is installed. ### Legacy API Removals[​](https://docs.dagster.io/migration/upgrading#legacy-api-removals "Direct link to Legacy API Removals") * Dagster's legacy APIs, which were marked "legacy" in 0.13.0, have been removed. This includes `@solid`, `SolidDefinition`, `@pipeline`, `PipelineDefinition`, `@composite_solid`, `CompositeSolidDefinition`, `ModeDefinition`, `PresetDefinition`, `PartitionSetDefinition`, `InputDefinition`, `OutputDefinition`, `DynamicOutputDefinition`, `pipeline_failure_sensor`, `@hourly_schedule`, `@daily_schedule`, `@weekly_schedule`, and `@monthly_schedule`. [Here is a guide](https://legacy-versioned-docs.dagster.dagster-docs.io/0.15.6/guides/dagster/graph_job_op) to migrating from the legacy APIs to the stable APIs. * Deprecated arguments to library ops have been switched to reflect stable APIs. This includes `input_defs`/`output_defs` arguments on `define_dagstermill_op`, which have been changed to `ins`/`outs` respectively, and `input_defs` argument on `create_shell_script_op`, which has been changed to `ins`. * The `pipeline_selection` argument has been removed from `run_failure_sensor` and related decorators / functions, and `job_selection` has been deprecated. Instead, use `monitored_jobs`. * `ScheduleExecutionContext` and `SensorExecutionContext` APIs have been removed. In 0.13.0, these were renamed to `ScheduleEvaluationContext` and `SensorEvaluationContext` respectively, and marked deprecated. * Along with the rest of the legacy APIs, `execute_pipeline` has been removed. The functionality previously supplied by `execute_pipeline` has been split between `JobDefinition.execute_in_process` ([docs](https://docs.dagster.io/_apidocs/jobs#dagster.JobDefinition.execute_in_process) ) and `execute_job` ([docs](https://docs.dagster.io/_apidocs/execution#dagster.execute_job) ). If you were previously using `execute_pipeline` for in-process testing, then `JobDefinition.execute_in_process` should replace. If using `execute_pipeline` for out-of-process execution, or non-testing workflows, then `execute_job` is the recommended replacement. * Alongside other removals of pipeline-related APIs, the `dagster pipeline` CLI subgroup has been removed in favor of `dagster job`. * The `dagster new-project` CLI subgroup has been removed in favor of `dagster project`. * `AssetGroup` and `build_assets_job`, which were advertised in an experimental iteration of software-defined assets, have been removed. Instead, check out the docs on [grouping assets](https://docs.dagster.io/concepts/assets/software-defined-assets#assigning-assets-to-groups) , and the docs on [defining asset jobs](https://docs.dagster.io/concepts/ops-jobs-graphs/jobs#from-software-defined-assets) . * The deprecated `partition_mappings` arguments on `@asset` and `@multi_asset` have been removed. Instead, user the `partition_mapping` argument the corresponding `AssetIn`s. * The deprecated `namespace` arguments on `@asset` and `AssetIn` have been removed. Instead, use the `key_prefix` argument. * The `input_defs` and `output_defs` arguments on [OpDefinition](https://docs.dagster.io/_apidocs/ops#dagster.OpDefinition) have been removed, and replaced with `ins` and `outs` arguments. `input_defs`/`output_defs` have been deprecated since 0.13.0. * The `preset_defs` argument on [JobDefinition](https://docs.dagster.io/_apidocs/jobs#dagster.JobDefinition) has been removed. When constructing a `JobDefinition` directly, config can be provided using the `config` argument instead. `preset_defs` has been deprecated since 0.13.0. * `EventMetadata` and `EventMetadataEntryData` APIs have been removed. Instead, metadata should be specified using the [MetadataValue](https://docs.dagster.io/_apidocs/ops#dagster.MetadataValue) APIs. * APIs referencing pipelines/solids in extension libraries have been removed. This includes `define_dagstermill_solid`, `make_dagster_pipeline_from_airflow_dag`, `create_databricks_job_solid`, the various `dbt_cli_*` and `dbt_rpc_*` solids, `bq_solid_for_queries`, `ge_validation_solid_factory`, `end_mlflow_run_on_pipeline_finished`, the various `shell_command_solid` APIs, `make_slack_on_pipeline_failure_sensor`, `snowflake_solid_for_query`, `end_mlflow_run_on_pipeline_finished`, and `create_spark_solid`. * `custom_path_fs_io_manager` has been removed, as its functionality is entirely subsumed by the `fs_io_manager`, where a custom path can be specified via config. ### Removed API List[​](https://docs.dagster.io/migration/upgrading#removed-api-list "Direct link to Removed API List") This serves as an exhaustive list of the removed APIs. From the main Dagster module: * `AssetGroup` * `DagsterPipelineRunMetadataValue` * `CompositeSolidDefinition` * `InputDefinition` * `Materialization` * `ModeDefinition` * `OutputDefinition` * `PipelineDefinition` * `PresetDefinition` * `SolidDefinition` * `SolidInvocation` * `DynamicOutputDefinition` * `composite_solid` * `lambda_solid` * `pipeline` * `solid` * `pipeline_failure_sensor` * `CompositeSolidExecutionResult` * `PipelineExecutionResult` * `SolidExecutionResult` * `SolidExecutionContext` * `build_solid_context` * `PipelineRun` * `PipelineRunStatus` * `default_executors` * `execute_pipeline_iterator` * `execute_pipeline` * `execute_solid_within_pipeline` * `reexecute_pipeline_iterator` * `reexecute_pipeline` * `execute_solid` * `execute_solids_within_pipeline` * `build_assets_job` * `schedule_from_partitions` * `PartitionSetDefinition` * `ScheduleExecutionContext` * `SensorExecutionContext` * `PipelineFailureSensorContext` * `daily_schedule` * `hourly_schedule` * `monthly_schedule` * `weekly_schedule` * `create_offset_partition_selector` * `date_partition_range` * `identity_partition_selector` * `custom_path_fs_io_manager` From libraries (APIs removed in 0.16.0 onwards): * `dagster_airflow.make_dagster_pipeline_from_airflow_dag` * `dagster_databricks.create_databricks_job_solid` * `dagster_dbt.dbt_cli_compile` * `dagster_dbt.dbt_cli_run` * `dagster_dbt.dbt_cli_run_operation` * `dagster_dbt.dbt_cli_snapshot` * `dagster_dbt.dbt_cli_snapshot_freshness` * `dagster_dbt.dbt_cli_test` * `dagster_dbt.create_dbt_rpc_run_sql_solid` * `dagster_dbt.dbt_rpc_run` * `dagster_dbt.dbt_rpc_run_and_wait` * `dagster_dbt.dbt_rpc_run_operation` * `dagster_dbt.dbt_rpc_run_operation_and_wait` * `dagster_dbt.dbt_rpc_snapshot` * `dagster_dbt.dbt_rpc_snapshot_and_wait` * `dagster_dbt.dbt_rpc_snapshot_freshness` * `dagster_dbt.dbt_rpc_snapshot_freshness_and_wait` * `dagster_dbt.dbt_rpc_test` * `dagster_dbt.dbt_rpc_test_and_wait` * `dagster_gcp.bq_solid_for_queries` * `dagster_ge.ge_validation_solid_factory` * `dagster_mlflow.end_mlflow_run_on_pipeline_finishes` * `dagster_shell.create_shell_command_solid` * `dagster_shell.create_shell_script_solid` * `dagster_shell.shell_solid` * `dagster_slack.make_slack_on_pipeline_failure_sensor` * `dagster_msteams.make_teams_on_pipeline_failure_sensor` * `dagster_snowflake.snowflake_solid_for_query` * `dagster_spark.create_spark_solid` Upgrading to 0.15.0[​](https://docs.dagster.io/migration/upgrading#upgrading-to-0150 "Direct link to Upgrading to 0.15.0") --------------------------------------------------------------------------------------------------------------------------- All items below are breaking changes unless marked with _(deprecation)_. ### Software-defined assets[​](https://docs.dagster.io/migration/upgrading#software-defined-assets "Direct link to Software-defined assets") This release marks the official transition of software-defined assets from experimental to stable. We made some final changes to incorporate feedback and make the APIs as consistent as possible: * Support for adding tags to asset materializations, which was previously marked as experimental, has been removed. * Some of the properties of the previously-experimental AssetsDefinition class have been renamed. group\_names is now group\_names\_by\_key, asset\_keys\_by\_input\_name is now keys\_by\_input\_name, and asset\_keys\_by\_output\_name is now keys\_by\_output\_name, asset\_key is now key, and asset\_keys is now keys. * fs\_asset\_io\_manager has been removed in favor of merging its functionality with fs\_io\_manager. fs\_io\_manager is now the default IO manager for asset jobs, and will store asset outputs in a directory named with the asset key. Similarly, removed adls2\_pickle\_asset\_io\_manager, gcs\_pickle\_asset\_io\_manager , and s3\_pickle\_asset\_io\_manager. Instead, adls2\_pickle\_io\_manager, gcs\_pickle\_io\_manager , and s3\_pickle\_io\_manager now support software-defined assets. * _(deprecation)_ The namespace argument on the @asset decorator and AssetIn has been deprecated. Users should use key\_prefix instead. * _(deprecation)_ AssetGroup has been deprecated. Users should instead place assets directly on repositories, optionally attaching resources using with\_resources. Asset jobs should be defined using define\_asset\_job (replacing AssetGroup.build\_job), and arbitrary sets of assets can be materialized using the standalone function materialize (replacing AssetGroup.materialize). * _(deprecation)_ The outs property of the previously-experimental @multi\_asset decorator now prefers a dictionary whose values are AssetOut objects instead of a dictionary whose values are Out objects. The latter still works, but is deprecated. ### Event records[​](https://docs.dagster.io/migration/upgrading#event-records "Direct link to Event records") * The get\_event\_records method on DagsterInstance now requires a non-None argument event\_records\_filter. Passing a None value for the event\_records\_filter argument will now raise an exception where previously it generated a deprecation warning. * Removed methods events\_for\_asset\_key and get\_asset\_events, which have been deprecated since 0.12.0. ### Extension libraries[​](https://docs.dagster.io/migration/upgrading#extension-libraries-1 "Direct link to Extension libraries") * \[dagster-dbt\] (breaks previously-experimental API) When using the load\_assets\_from\_dbt\_project or load\_assets\_from\_dbt\_manifest , the AssetKeys generated for dbt sources are now the union of the source name and the table name, and the AssetKeys generated for models are now the union of the configured schema name for a given model (if any), and the model name. To revert to the old behavior: dbt\_assets = load\_assets\_from\_dbt\_project(..., node\_info\_to\_asset\_key=lambda node\_info: AssetKey(node\_info\["name"\]). * \[dagster-k8s\] In the Dagster Helm chart, user code deployment configuration (like secrets, configmaps, or volumes) is now automatically included in any runs launched from that code. Previously, this behavior was opt-in. In most cases, this will not be a breaking change, but in less common cases where a user code deployment was running in a different kubernetes namespace or using a different service account, this could result in missing secrets or configmaps in a launched run that previously worked. You can return to the previous behavior where config on the user code deployment was not applied to any runs by setting the includeConfigInLaunchedRuns.enabled field to false for the user code deployment. See the Kubernetes Deployment docs ([https://docs.dagster.io/deployment/guides/kubernetes/deploying-with-helm#configure-your-user-deployment](https://docs.dagster.io/deployment/guides/kubernetes/deploying-with-helm#configure-your-user-deployment) ) for more details. * \[dagster-snowflake\] dagster-snowflake has dropped support for python 3.6. The library it is currently built on, snowflake-connector-python, dropped 3.6 support in their recent 2.7.5 release. ### Other[​](https://docs.dagster.io/migration/upgrading#other "Direct link to Other") * The prior\_attempts\_count parameter is now removed from step-launching APIs. This parameter was not being used, as the information it held was stored elsewhere in all cases. It can safely be removed from invocations without changing behavior. * The FileCache class has been removed. * Previously, when schedules/sensors targeted jobs with the same name as other jobs in the repo, the jobs on the sensor/schedule would silently overwrite the other jobs. Now, this will cause an error. Upgrading to 0.14.0[​](https://docs.dagster.io/migration/upgrading#upgrading-to-0140 "Direct link to Upgrading to 0.14.0") --------------------------------------------------------------------------------------------------------------------------- If migrating from below 0.13.17, you can run dagster instance migrate This optional migration makes performance improvements to the runs page in Dagit. ### Breaking Changes[​](https://docs.dagster.io/migration/upgrading#breaking-changes-10 "Direct link to Breaking Changes") * The Dagster Daemon now uses the same workspace.yaml file as Dagit to locate your Dagster code. You should ensure that if you make any changes to your workspace.yaml file, they are included in both Dagit’s copy and the Dagster Daemon’s copy. When you make changes to the workspace.yaml file, you don’t need to restart either Dagit or the Dagster Daemon - in Dagit, you can reload the workspace from the Workspace tab, and the Dagster Daemon will periodically check the workspace.yaml file for changes every 60 seconds. If you are using the Dagster Helm chart, no changes are required to include the workspace in the Dagster Daemon. * In previous releases, it was possible to supply either an AssetKey, or a function that produced an AssetKey from an OutputContext as the asset\_key argument to an Out/OutputDefinition. The latter behavior makes it impossible to gain information about these relationships without running a job, and has been deprecated. However, we still support supplying a static AssetKey as an argument. * We have renamed many of the core APIs that interact with ScheduleStorage, which keeps track of sensor/schedule state and ticks. The old term for the generic schedule/sensor “job” has been replaced by the term “instigator” in order to avoid confusion with the execution API introduced in 0.12.0. If you have implemented your own schedule storage, you may need to change your method signatures appropriately. * Dagit is now powered by Starlette instead of Flask. If you have implemented a custom run coordinator, you may need to make the following change: from flask import has_request_context, requestdef submit_run(self, context: SubmitRunContext) -> PipelineRun: jwt_claims_header = ( request.headers.get("X-Amzn-Oidc-Data", None) if has_request_context() else None ) Should be replaced by: def submit_run(self, context: SubmitRunContext) -> PipelineRun: jwt_claims_header = context.get_request_header("X-Amzn-Oidc-Data") * The Dagster Daemon now requires a workspace.yaml file, much like Dagit. * Ellipsis (“...”) is now an invalid substring of a partition key. This is because Dagit accepts an ellipsis to specify partition ranges. * \[Helm\] The Dagster Helm chart now only supported Kubernetes clusters above version 1.18. ### Deprecation: Metadata API Renames[​](https://docs.dagster.io/migration/upgrading#deprecation-metadata-api-renames "Direct link to Deprecation: Metadata API Renames") Dagster’s metadata API has undergone a signficant overhaul. Changes include: * To reflect the fact that metadata can be specified on definitions in addition to events, the following names are changing. The old names are deprecated, and will function as aliases for the new names until 0.15.0: * `EventMetadata` > `MetadataValue` * `EventMetadataEntry` > `MetadataEntry` * `XMetadataEntryData` > `XMetadataValue` (e.g. `TextMetadataEntryData` > `TextMetadataValue`) * The `metadata_entries` keyword argument to events and Dagster types is deprecated. Instead, users should use the metadata keyword argument, which takes a dictionary mapping string labels to `MetadataValue`s. * Arbitrary metadata on In/InputDefinition and Out/OutputDefinition is deprecated. In 0.15.0, metadata passed for these classes will need to be resolvable to `MetadataValue` (i.e. function like metadata everywhere else in Dagster). * The description attribute of `EventMetadataEntry` is deprecated. * The static API of `EventMetadataEntry` (e.g. `EventMetadataEntry.text`) is deprecated. In 0.15.0, users should avoid constructing `EventMetadataEntry` objects directly, instead utilizing the metadata dictionary keyword argument, which maps string labels to `MetadataValues`. Upgrading to 0.13.0[​](https://docs.dagster.io/migration/upgrading#upgrading-to-0130 "Direct link to Upgrading to 0.13.0") --------------------------------------------------------------------------------------------------------------------------- Jobs, ops, and graphs have replaced pipelines, solids, modes, and presets as the stable core of the system. [Here](https://legacy-versioned-docs.dagster.dagster-docs.io/0.15.7/guides/dagster/graph_job_op) is a guide you can use to update your code using the legacy APIs into using the new Dagster core APIs. 0.13.0 is still compatible with the pipeline, solid, mode, and preset APIs, which means that you don't need to migrate your code to upgrade to 0.13.0. Upgrading to 0.12.0[​](https://docs.dagster.io/migration/upgrading#upgrading-to-0120 "Direct link to Upgrading to 0.12.0") --------------------------------------------------------------------------------------------------------------------------- The new experimental core API experience in Dagit uses some features that require a data migration. Before enabling the experimental core API flag in Dagit, you will first need to run this command: dagster instance migrate If you are not going to enable the experimental core API experience, this data migration is optional. However, you may still want to run the migration anyway, which will enable better performance in viewing the Asset catalog in Dagit. Upgrading to 0.11.0[​](https://docs.dagster.io/migration/upgrading#upgrading-to-0110 "Direct link to Upgrading to 0.11.0") --------------------------------------------------------------------------------------------------------------------------- ### Action Required: Run and event storage schema changes[​](https://docs.dagster.io/migration/upgrading#action-required-run-and-event-storage-schema-changes "Direct link to Action Required: Run and event storage schema changes") Run this after migrating to 0.11.0: dagster instance migrate This release includes several schema changes to the Dagster storages that improve performance, allow support for MySQL, and enable new features like asset tags and reliable backfills. After upgrading to 0.11.0, run the `dagster instance migrate` command to migrate your instance storage to the latest schema. ### Action Required: Schedule timezones[​](https://docs.dagster.io/migration/upgrading#action-required-schedule-timezones "Direct link to Action Required: Schedule timezones") Schedules now run in UTC (instead of the system timezone) if no timezone has been set on the schedule. If you’re using a deprecated scheduler like `SystemCronScheduler` or `K8sScheduler`, we recommend that you switch to the native Dagster scheduler. The deprecated schedulers will be removed in the next Dagster release. ### Action Required: Asset storage[​](https://docs.dagster.io/migration/upgrading#action-required-asset-storage "Direct link to Action Required: Asset storage") If upgrading directly to `0.11.0` from `0.9.22` or lower, you might notice some asset keys missing from the catalog if they have not been materialized using a version `0.9.16` or greater. We removed some back-compatibility for performance reasons. If this is the case, you can either run `dagster instance reindex` or execute the appropriate pipelines to materialize those assets again. In either case, the full history of the asset will still be maintained. ### Removals of Deprecated APIs[​](https://docs.dagster.io/migration/upgrading#removals-of-deprecated-apis "Direct link to Removals of Deprecated APIs") * The `instance` argument to `RunLauncher.launch_run` has been removed. If you have written a custom RunLauncher, you’ll need to update the signature of that method. You can still access the `DagsterInstance` on the `RunLauncher` via the `_instance` parameter. * The `has_config_entry`, `has_configurable_inputs`, and `has_configurable_outputs` properties of `solid` and `composite_solid` have been removed. * The deprecated optionality of the `name` argument to `PipelineDefinition` has been removed, and the argument is now required. * The `execute_run_with_structured_logs` and `execute_step_with_structured_logs` internal CLI entry points have been removed. Use `execute_run` or `execute_step` instead. * The `python_environment` key has been removed from `workspace.yaml`. Instead, to specify that a repository location should use a custom python environment, set the `executable_path` key within a `python_file` or `python_module` key. See [the docs](https://docs.dagster.io/concepts/code-locations/workspace-files) for more information on configuring your `workspace.yaml` file. * \[dagster-dask\] The deprecated schema for reading or materializing dataframes has been removed. Use the `read` or `to` keys accordingly. ### Breaking Changes[​](https://docs.dagster.io/migration/upgrading#breaking-changes-11 "Direct link to Breaking Changes") * Names provided to `alias` on solids now enforce the same naming rules as solids. You may have to update provided names to meet these requirements. * The `retries` method on `Executor` should now return a `RetryMode` instead of a `Retries`. This will only affect custom `Executor` classes. * Submitting partition backfills in Dagit now requires `dagster-daemon` to be running. The instance setting in `dagster.yaml` to optionally enable daemon-based backfills has been removed, because all backfills are now daemon-based backfills. # removed, no longer a valid setting in dagster.yamlbackfill: daemon_enabled: true The corresponding value flag `dagsterDaemon.backfill.enabled` has also been removed from the Dagster helm chart. * The sensor daemon interval settings in `dagster.yaml` has been removed. The sensor daemon now runs in a continuous loop so this customization is no longer useful. # removed, no longer a valid setting in dagster.yamlsensor_settings: interval_seconds: 10 Upgrading to 0.10.0[​](https://docs.dagster.io/migration/upgrading#upgrading-to-0100 "Direct link to Upgrading to 0.10.0") --------------------------------------------------------------------------------------------------------------------------- ### Action Required: Run and event storage schema changes[​](https://docs.dagster.io/migration/upgrading#action-required-run-and-event-storage-schema-changes-1 "Direct link to Action Required: Run and event storage schema changes") # Run after migrating to 0.10.0$ dagster instance migrate This release includes several schema changes to the Dagster storages that improve performance and enable new features like sensors and run queueing. After upgrading to 0.10.0, run the `dagster instance migrate` command to migrate your instance storage to the latest schema. This will turn off any running schedules, so you will need to restart any previously running schedules after migrating the schema. Before turning them back on, you should follow the steps below to migrate to `DagsterDaemonScheduler`. ### New scheduler: DagsterDaemonScheduler[​](https://docs.dagster.io/migration/upgrading#new-scheduler-dagsterdaemonscheduler "Direct link to New scheduler: DagsterDaemonScheduler") This release includes a new `DagsterDaemonScheduler` with improved fault tolerance and full support for timezones. We highly recommend upgrading to the new scheduler during this release. The existing schedulers, `SystemCronScheduler` and `K8sScheduler`, are deprecated and will be removed in a future release. #### Steps to migrate[​](https://docs.dagster.io/migration/upgrading#steps-to-migrate "Direct link to Steps to migrate") Instead of relying on system cron or k8s cron jobs, the `DaemonScheduler` uses the new `dagster-daemon` service to run schedules. This requires running the `dagster-daemon` service as a part of your deployment. Refer to our [deployment documentation](https://docs.dagster.io/deployment) for a guides on how to set up and run the daemon process for local development, Docker, or Kubernetes deployments. **If you are currently using the SystemCronScheduler or K8sScheduler:** 1. Stop any currently running schedules, to prevent any dangling cron jobs from being left behind. You can do this through the Dagit UI, or using the following command: dagster schedule stop --location {repository_location_name} {schedule_name} If you do not stop running schedules before changing schedulers, Dagster will throw an exception on startup due to the misconfigured running schedules. 2. In your `dagster.yaml` file, remove the `scheduler:` entry. If there is no `scheduler:` entry, the `DagsterDaemonScheduler` is automatically used as the default scheduler. 3. Start the `dagster-daemon` process. Guides can be found in our [deployment documentations](https://docs.dagster.io/deployment) . See our [schedules troubleshooting guide](https://docs.dagster.io/concepts/partitions-schedules-sensors/schedules) for help if you experience any problems with the new scheduler. **If you are not using a legacy scheduler:** No migration steps are needed, but make sure you run `dagster instance migrate` as a part of upgrading to 0.10.0. ### Deprecation: Intermediate Storage[​](https://docs.dagster.io/migration/upgrading#deprecation-intermediate-storage "Direct link to Deprecation: Intermediate Storage") We have deprecated the intermediate storage machinery in favor of the new IO manager abstraction, which offers finer-grained control over how inputs and outputs are serialized and persisted. Check out the [IO Managers Overview](https://docs.dagster.io/concepts/io-management/io-managers) for more information. #### Steps to Migrate[​](https://docs.dagster.io/migration/upgrading#steps-to-migrate-1 "Direct link to Steps to Migrate") * We have deprecated the top level `"storage"` and `"intermediate_storage"` fields on `run_config`. If you are currently executing pipelines as follows: @pipelinedef my_pipeline(): ...execute_pipeline( my_pipeline, run_config={ "intermediate_storage": { "filesystem": {"base_dir": ...} } },)execute_pipeline( my_pipeline, run_config={ "storage": { "filesystem": {"base_dir": ...} } },) You should instead use the built-in IO manager `fs_io_manager`, which can be attached to your pipeline as a resource: @pipeline( mode_defs=[ ModeDefinition( resource_defs={"io_manager": fs_io_manager} ) ],)def my_pipeline(): ...execute_pipeline( my_pipeline, run_config={ "resources": { "io_manager": {"config": {"base_dir": ...}} } },) There are corresponding IO managers for other intermediate storages, such as the S3- and ADLS2-based storages * We have deprecated `IntermediateStorageDefinition` and `@intermediate_storage`. If you have written custom intermediate storage, you should migrate to custom IO managers defined using the `@io_manager` API. We have provided a helper method, `io_manager_from_intermediate_storage`, to help migrate your existing custom intermediate storages to IO managers. my_io_manager_def = io_manager_from_intermediate_storage( my_intermediate_storage_def)@pipeline( mode_defs=[ ModeDefinition( resource_defs={ "io_manager": my_io_manager_def } ), ],)def my_pipeline(): ... * We have deprecated the `intermediate_storage_defs` argument to `ModeDefinition`, in favor of the new IO managers, which should be attached using the `resource_defs` argument. ### Removal: input\_hydration\_config and output\_materialization\_config[​](https://docs.dagster.io/migration/upgrading#removal-input_hydration_config-and-output_materialization_config "Direct link to Removal: input_hydration_config and output_materialization_config") Use `dagster_type_loader` instead of `input_hydration_config` and `dagster_type_materializer` instead of `output_materialization_config`. On `DagsterType` and type constructors in `dagster_pandas` use the `loader` argument instead of `input_hydration_config` and the `materializer` argument instead of `dagster_type_materializer` argument. ### Removal: repository key in workspace YAML[​](https://docs.dagster.io/migration/upgrading#removal-repository-key-in-workspace-yaml "Direct link to Removal: repository key in workspace YAML") We have removed the ability to specify a repository in your workspace using the `repository:` key. Use `load_from:` instead when specifying how to load the repositories in your workspace. ### Deprecated: python\_environment key in workspace YAML[​](https://docs.dagster.io/migration/upgrading#deprecated-python_environment-key-in-workspace-yaml "Direct link to Deprecated: python_environment key in workspace YAML") The `python_environment:` key is now deprecated and will be removed in a future release. Previously, when you wanted to load a repository location in your workspace using a different Python environment from Dagit’s Python environment, you needed to use a `python_environment:` key under `load_from:` instead of the `python_file:` or `python_package:` keys. Now, you can simply customize the `executable_path` in your workspace entries without needing to change to the `python_environment:` key. For example, the following workspace entry: - python_environment: executable_path: "/path/to/venvs/dagster-dev-3.7.6/bin/python" target: python_package: package_name: dagster_examples location_name: dagster_examples should now be expressed as: - python_package: executable_path: "/path/to/venvs/dagster-dev-3.7.6/bin/python" package_name: dagster_examples location_name: dagster_examples See our [Workspaces Overview](https://docs.dagster.io/concepts/code-locations/workspace-files) for more information and examples. ### Removal: config\_field property on definition classes[​](https://docs.dagster.io/migration/upgrading#removal-config_field-property-on-definition-classes "Direct link to Removal: config_field property on definition classes") We have removed the property `config_field` on definition classes. Use `config_schema` instead. ### Removal: System Storage[​](https://docs.dagster.io/migration/upgrading#removal-system-storage "Direct link to Removal: System Storage") We have removed the system storage abstractions, i.e. `SystemStorageDefinition` and `@system_storage` ([deprecated in 0.9.0](https://docs.dagster.io/migration/upgrading#deprecation-system_storage_defs) ). Please note that the intermediate storage abstraction is also deprecated and will be removed in 0.11.0. [Use IO managers instead](https://docs.dagster.io/migration/upgrading#deprecation-intermediate-storage) . * We have removed the `system_storage_defs` argument (deprecated in 0.9.0) to `ModeDefinition`, in favor of `intermediate_storage_defs.` * We have removed the built-in system storages, e.g. `default_system_storage_defs` (deprecated in 0.9.0). ### Removal: step\_keys\_to\_execute[​](https://docs.dagster.io/migration/upgrading#removal-step_keys_to_execute "Direct link to Removal: step_keys_to_execute") We have removed the `step_keys_to_execute` argument to `reexecute_pipeline` and `reexecute_pipeline_iterator`, in favor of `step_selection`. This argument accepts the Dagster selection syntax, so, for example, `*solid_a+` represents `solid_a`, all of its upstream steps, and its immediate downstream steps. ### Breaking Change: date\_partition\_range[​](https://docs.dagster.io/migration/upgrading#breaking-change-date_partition_range "Direct link to Breaking Change: date_partition_range") Starting in 0.10.0, Dagster uses the [pendulum](https://pypi.org/project/pendulum/) library to ensure that schedules and partitions behave correctly with respect to timezones. As part of this change, the `delta` parameter to `date_partition_range` (which determined the time different between partitions and was a `datetime.timedelta`) has been replaced by a `delta_range` parameter (which must be a string that's a valid argument to the `pendulum.period` function, such as `"days"`, `"hours"`, or `"months"`). For example, the following partition range for a monthly partition set: date_partition_range( start=datetime.datetime(2018, 1, 1), end=datetime.datetime(2019, 1, 1), delta=datetime.timedelta(months=1)) should now be expressed as: date_partition_range( start=datetime.datetime(2018, 1, 1), end=datetime.datetime(2019, 1, 1), delta_range="months") ### Breaking Change: PartitionSetDefinition.create\_schedule\_definition[​](https://docs.dagster.io/migration/upgrading#breaking-change-partitionsetdefinitioncreate_schedule_definition "Direct link to Breaking Change: PartitionSetDefinition.create_schedule_definition") When you create a schedule from a partition set using `PartitionSetDefinition.create_schedule_definition`, you now must supply a `partition_selector` argument that tells the scheduler which partition to use for a given schedule time. We have added two helper functions, `create_offset_partition_selector` and `identity_partition_selector`, that capture two common partition selectors (schedules that execute at a fixed offset from the partition times, e.g. a schedule that creates the previous day's partition each morning, and schedules that execute at the same time as the partition times). The previous default partition selector was `last_partition`, which didn't always work as expected when using the default scheduler and has been removed in favor of the two helper partition selectors above. For example, a schedule created from a daily partition set that fills in each partition the next day at 10AM would be created as follows: partition_set = PartitionSetDefinition( name='hello_world_partition_set', pipeline_name='hello_world_pipeline', partition_fn= date_partition_range( start=datetime.datetime(2021, 1, 1), delta_range="days", timezone="US/Central", ) run_config_fn_for_partition=my_run_config_fn,)schedule_definition = partition_set.create_schedule_definition( "daily_10am_schedule", "0 10 * * *", partition_selector=create_offset_partition_selector(lambda d: d.subtract(hours=10, days=1)) execution_timezone="US/Central",) ### Renamed: Helm values[​](https://docs.dagster.io/migration/upgrading#renamed-helm-values "Direct link to Renamed: Helm values") Following convention in the [Helm docs](https://helm.sh/docs/chart_best_practices/values/#naming-conventions) , we now camel case all of our Helm values. To migrate to 0.10.0, you'll need to update your `values.yaml` with the following renames: * `pipeline_run` → `pipelineRun` * `dagster_home` → `dagsterHome` * `env_secrets` → `envSecrets` * `env_config_maps` → `envConfigMaps` ### Restructured: scheduler in Helm values[​](https://docs.dagster.io/migration/upgrading#restructured-scheduler-in-helm-values "Direct link to Restructured: scheduler in Helm values") When specifying the Dagster instance scheduler, rather than using a boolean field to switch between the current options of `K8sScheduler` and `DagsterDaemonScheduler`, we now require the scheduler type to be explicitly defined under `scheduler.type`. If the user specified `scheduler.type` has required config, additional fields will need to be specified under `scheduler.config`. `scheduler.type` and corresponding `scheduler.config` values are enforced via [JSON Schema](https://helm.sh/docs/topics/charts/#schema-files) . For example, if your Helm values previously were set like this to enable the `DagsterDaemonScheduler`: ​ scheduler: k8sEnabled: false ​ You should instead have: ​ scheduler: type: DagsterDaemonScheduler ### Restructured: celery and k8sRunLauncher in Helm values[​](https://docs.dagster.io/migration/upgrading#restructured-celery-and-k8srunlauncher-in-helm-values "Direct link to Restructured: celery and k8sRunLauncher in Helm values") `celery` and `k8sRunLauncher` now live under `runLauncher.config.celeryK8sRunLauncher` and `runLauncher.config.k8sRunLauncher` respectively. Now, to enable celery, `runLauncher.type` must equal `CeleryK8sRunLauncher`. To enable the vanilla K8s run launcher, `runLauncher.type` must equal `K8sRunLauncher`. `runLauncher.type` and corresponding `runLauncher.config` values are enforced via [JSON Schema](https://helm.sh/docs/topics/charts/#schema-files) . For example, if your Helm values previously were set like this to enable the `K8sRunLauncher`: ​ celery: enabled: false​k8sRunLauncher: enabled: true jobNamespace: ~ loadInclusterConfig: true kubeconfigFile: ~ envConfigMaps: [] envSecrets: [] ​ You should instead have: ​ runLauncher: type: K8sRunLauncher config: k8sRunLauncher: jobNamespace: ~ loadInclusterConfig: true kubeconfigFile: ~ envConfigMaps: [] envSecrets: [] ### New Helm defaults[​](https://docs.dagster.io/migration/upgrading#new-helm-defaults "Direct link to New Helm defaults") By default, `userDeployments` is enabled and the `runLauncher` is set to the `K8sRunLauncher`. Along with the latter change, all message brokers (e.g. `rabbitmq` and `redis`) are now disabled by default. If you were using the `CeleryK8sRunLauncher`, one of `rabbitmq` or `redis` must now be explicitly enabled in your Helm values. Upgrading to 0.9.0[​](https://docs.dagster.io/migration/upgrading#upgrading-to-090 "Direct link to Upgrading to 0.9.0") ------------------------------------------------------------------------------------------------------------------------ ### Removal: config argument[​](https://docs.dagster.io/migration/upgrading#removal-config-argument "Direct link to Removal: config argument") We have removed the `config` argument to the `ConfigMapping`, `@composite_solid`, `@solid`, `SolidDefinition`, `@executor`, `ExecutorDefinition`, `@logger`, `LoggerDefinition`, `@resource`, and `ResourceDefinition` APIs, which we deprecated in 0.8.0, in favor of `config_schema`. Upgrading to 0.8.8[​](https://docs.dagster.io/migration/upgrading#upgrading-to-088 "Direct link to Upgrading to 0.8.8") ------------------------------------------------------------------------------------------------------------------------ ### Deprecation: Materialization[​](https://docs.dagster.io/migration/upgrading#deprecation-materialization "Direct link to Deprecation: Materialization") We deprecated the `Materialization` event type in favor of the new `AssetMaterialization` event type, which requires the `asset_key` parameter. Solids yielding `Materialization` events will continue to work as before, though the `Materialization` event will be removed in a future release. ### Deprecation: system\_storage\_defs[​](https://docs.dagster.io/migration/upgrading#deprecation-system_storage_defs "Direct link to Deprecation: system_storage_defs") We are starting to deprecate "system storages" - instead of pipelines having a system storage definition which creates an intermediate storage, pipelines now directly have an intermediate storage definition. * We have added an `intermediate_storage_defs` argument to `ModeDefinition`, which accepts a list of `IntermediateStorageDefinition`s, e.g. `s3_plus_default_intermediate_storage_defs`. As before, the default includes an in-memory intermediate and a local filesystem intermediate storage. * We have deprecated `system_storage_defs` argument to `ModeDefinition` in favor of `intermediate_storage_defs`. `system_storage_defs` will be removed in 0.10.0 at the earliest. * We have added an `@intermediate_storage` decorator, which makes it easy to define intermediate storages. * We have added `s3_file_manager` and `local_file_manager` resources to replace the file managers that previously lived inside system storages. The airline demo has been updated to include an example of how to do this: [https://github.com/dagster-io/dagster/blob/0.8.8/examples/airline\_demo/airline\_demo/solids.py#L171](https://github.com/dagster-io/dagster/blob/0.8.8/examples/airline_demo/airline_demo/solids.py#L171) . For example, if your `ModeDefinition` looks like this: from dagster_aws.s3 import s3_plus_default_storage_defsModeDefinition(system_storage_defs=s3_plus_default_storage_defs) it is recommended to make it look like this: from dagster_aws.s3 import s3_plus_default_intermediate_storage_defsModeDefinition(intermediate_storage_defs=s3_plus_default_intermediate_storage_defs) Upgrading to 0.8.7[​](https://docs.dagster.io/migration/upgrading#upgrading-to-087 "Direct link to Upgrading to 0.8.7") ------------------------------------------------------------------------------------------------------------------------ ### Loading python modules from the working directory[​](https://docs.dagster.io/migration/upgrading#loading-python-modules-from-the-working-directory "Direct link to Loading python modules from the working directory") Loading python modules reliant on the working directory being on the PYTHONPATH is no longer supported. The `dagster` and `dagit` CLI commands no longer add the working directory to the PYTHONPATH when resolving modules, which may break some imports. Explicitly installed python packages can be specified in workspaces using the `python_package` workspace yaml config option. The `python_module` config option is deprecated and will be removed in a future release. Upgrading to 0.8.6[​](https://docs.dagster.io/migration/upgrading#upgrading-to-086 "Direct link to Upgrading to 0.8.6") ------------------------------------------------------------------------------------------------------------------------ ### dagster-celery[​](https://docs.dagster.io/migration/upgrading#dagster-celery "Direct link to dagster-celery") The `dagster-celery` module has been broken apart to manage dependencies more coherently. There are now three modules: `dagster-celery`, `dagster-celery-k8s`, and `dagster-celery-docker`. Related to above, the `dagster-celery worker start` command now takes a required `-A` parameter which must point to the `app.py` file within the appropriate module. E.g if you are using the `celery_k8s_job_executor` then you must use the `-A dagster_celery_k8s.app` option when using the `celery` or `dagster-celery` cli tools. Similar for the `celery_docker_executor`: `-A dagster_celery_docker.app` must be used. ### Deprecation: input\_hydration\_config and output\_materialization\_config[​](https://docs.dagster.io/migration/upgrading#deprecation-input_hydration_config-and-output_materialization_config "Direct link to Deprecation: input_hydration_config and output_materialization_config") We renamed the `input_hydration_config` and `output_materialization_config` decorators to `dagster_type_` and `dagster_type_materializer` respectively. We also renamed DagsterType's `input_hydration_config` and `output_materialization_config` arguments to `loader` and `materializer` respectively. For example, if your dagster type definition looks like this: from dagster import DagsterType, input_hydration_config, output_materialization_config@input_hydration_config(config_schema=my_config_schema)def my_loader(_context, config): '''some implementation'''@output_materialization_config(config_schema=my_config_schema)def my_materializer(_context, config): '''some implementation'''MyType = DagsterType( input_hydration_config=my_loader, output_materialization_config=my_materializer, type_check_fn=my_type_check,) it is recommended to make it look like this: from dagster import DagsterType, dagster_type_loader, dagster_type_materializer@dagster_type_loader(config_schema=my_config_schema)def my_loader(_context, config): '''some implementation'''@dagster_type_materializer(config_schema=my_config_schema)def my_materializer(_context, config): '''some implementation'''MyType = DagsterType( loader=my_loader, materializer=my_materializer, type_check_fn=my_type_check,) Upgrading to 0.8.5[​](https://docs.dagster.io/migration/upgrading#upgrading-to-085 "Direct link to Upgrading to 0.8.5") ------------------------------------------------------------------------------------------------------------------------ ### Python 3.5[​](https://docs.dagster.io/migration/upgrading#python-35 "Direct link to Python 3.5") Python 3.5 is no longer under test. ### Engine and ExecutorConfig -> Executor[​](https://docs.dagster.io/migration/upgrading#engine-and-executorconfig---executor "Direct link to Engine and ExecutorConfig -> Executor") `Engine` and `ExecutorConfig` have been deleted in favor of `Executor`. Instead of the `@executor` decorator decorating a function that returns an `ExecutorConfig` it should now decorate a function that returns an `Executor`. Upgrading to 0.8.3[​](https://docs.dagster.io/migration/upgrading#upgrading-to-083 "Direct link to Upgrading to 0.8.3") ------------------------------------------------------------------------------------------------------------------------ ### Change: gcs\_resource[​](https://docs.dagster.io/migration/upgrading#change-gcs_resource "Direct link to Change: gcs_resource") Previously, the `gcs_resource` returned a `GCSResource` wrapper which had a single `client` property that returned a `google.cloud.storage.client.Client`. Now, the `gcs_resource` returns the client directly. To update solids that use the `gcp_resource`, change: context.resources.gcs.client To: context.resources.gcs Upgrading to 0.8.0[​](https://docs.dagster.io/migration/upgrading#upgrading-to-080 "Direct link to Upgrading to 0.8.0") ------------------------------------------------------------------------------------------------------------------------ ### Repository loading[​](https://docs.dagster.io/migration/upgrading#repository-loading "Direct link to Repository loading") Dagit and other tools no longer load a single repository containing user definitions such as pipelines into the same process as the framework code. Instead, they load a "workspace" that can contain multiple repositories sourced from a variety of different external locations (e.g., Python modules and Python virtualenvs, with containers and source control repositories soon to come). The repositories in a workspace are loaded into their own "user" processes distinct from the "host" framework process. Dagit and other tools now communicate with user code over an IPC mechanism. As a consequence, the former `repository.yaml` and the associated `-y`/`--repository-yaml` CLI arguments are deprecated in favor of a new `workspace.yaml` file format and associated `-w`/`--workspace-yaml` arguments. #### Steps to migrate[​](https://docs.dagster.io/migration/upgrading#steps-to-migrate-2 "Direct link to Steps to migrate") You should replace your `repository.yaml` files with `workspace.yaml` files, which can define a number of possible sources from which to load repositories. load_from: - python_module: module_name: dagster_examples attribute: define_internal_dagit_repository - python_module: dagster_examples.intro_tutorial.repos - python_file: repos.py - python_environment: executable_path: "/path/to/venvs/dagster-dev-3.7.6/bin/python" target: python_module: module_name: dagster_examples location_name: dagster_examples attribute: define_internal_dagit_repository ### Repository definition[​](https://docs.dagster.io/migration/upgrading#repository-definition "Direct link to Repository definition") The `@scheduler` and `@repository_partitions` decorators have been removed. In addition, users should prefer the new `@repository` decorator to instantiating `RepositoryDefinition` directly. One consequence of this change is that `PartitionSetDefinition` names, including those defined by a `PartitionScheduleDefinition`, must now be unique within a single repository. #### Steps to migrate[​](https://docs.dagster.io/migration/upgrading#steps-to-migrate-3 "Direct link to Steps to migrate") Previously you might have defined your pipelines, schedules, partition sets, and repositories in a python file such as the following: @pipelinedef test(): ...@daily_schedule( pipeline_name='test', start_date=datetime.datetime(2020, 1, 1),)def daily_test_schedule(_): return {}test_partition_set = PartitionSetDefinition( name="test", pipeline_name="test", partition_fn=lambda: ["test"], environment_dict_fn_for_partition=lambda _: {},)@schedulesdef define_schedules(): return [daily_test_schedule]@repository_partitionsdef define_partitions(): return [test_partition_set]def define_repository(): return RepositoryDefinition('test', pipeline_defs=[test]) With a `repository.yaml` such as: repository: file: repo.py fn: define_repositoryscheduler: file: repo.py fn: define_schedulespartitions: file: repo.py fn: define_partitions In 0.8.0, you'll write Python like: @pipelinedef test_pipeline(): ...@daily_schedule( pipeline_name='test', start_date=datetime.datetime(2020, 1, 1),)def daily_test_schedule(_): return {}test_partition_set = PartitionSetDefinition( name="test", pipeline_name="test", partition_fn=lambda: ["test"], run_config_fn_for_partition=lambda _: {},)@repositorydef test_repository(): return [test_pipeline, daily_test_schedule, test_partition_set] Your `workspace.yaml` will look like: load_from: - python_file: repo.py If you have more than one repository defined in a single Python file, you'll want to instead load the repository using `workspace.yaml` like: load_from: - python_file: relative_path: repo.py attribute: test_repository - python_file: relative_path: repo.py attribute: other_repository Of course, the `workspace.yaml` also supports loading from a `python_module`, or with a specific Python interpreter from a `python_environment`. Note that the `@repository` decorator also supports more sophisticated, lazily-loaded repositories. Consult the documentation for the decorator for more details. ### Reloadable repositories[​](https://docs.dagster.io/migration/upgrading#reloadable-repositories "Direct link to Reloadable repositories") In 0.7.x, dagster attempted to elide the difference between a pipeline that was defined in memory and one that was loaded through machinery that used the `ExecutionTargetHandle` machinery. This resulted in opaque and hard-to-predict errors and unpleasant workarounds, for instance: * Pipeline execution in test using `execute_pipeline` would suddenly fail when a multiprocess executor was used. * Tests of pipelines with dagstermill solids had to resort to workarounds such as handle = handle_for_pipeline_cli_args( {'module_name': 'some_module.repository', 'fn_name': 'some_pipeline'} ) pipeline = handle.build_pipeline_definition() result = execute_pipeline(pipeline, ...) In 0.8.0, we've added the `reconstructable` helper to explicitly convert in-memory pipelines into reconstructable pipelines that can be passed between processes. @pipeline(...)def some_pipeline(): ...execute_pipeline(reconstructable(some_pipeline), {'execution': {'multiprocess': {}}) Pipelines must be defined in module scope in order for `reconstructable` to be used. Note that pipelines defined _interactively_, e.g., in the Python REPL, cannot be passed between processes. ### Renaming environment\_dict and removing RunConfig[​](https://docs.dagster.io/migration/upgrading#renaming-environment_dict-and-removing-runconfig "Direct link to Renaming environment_dict and removing RunConfig") In 0.8.0, we've renamed the common `environment_dict` parameter to many user-facing APIs to `run_config`, and we've dropped the previous `run_config` parameter. This change affects the `execute_pipeline_iterator` and `execute_pipeline` APIs, the `PresetDefinition` and `ScheduleDefinition`, and the `execute_solid` test API. Similarly, the `environment_dict_fn`, `user_defined_environment_dict_fn_for_partition`, and `environment_dict_fn_for_partition` parameters to `ScheduleDefinition`, `PartitionSetDefinition`, and `PartitionScheduleDefinition` have been renamed to `run_config_fn`, `user_defined_run_config_fn_for_partition`, and `run_config_fn_for_partition` respectively. The previous `run_config` parameter has been removed, as has the backing `RunConfig` class. This change affects the `execute_pipeline_iterator` and `execute_pipeline` APIs, and the `execute_solids_within_pipeline` and `execute_solid_within_pipeline` test APIs. Instead, you should set the `mode`, `preset`, `tags`, `solid_selection`, and, in test, \`raise\_on\_error parameters directly. This change is intended to reduce ambiguity around the notion of a pipeline execution's "environment", since the config value passed as `run_config` is scoped to a single execution. ### Deprecation: config argument[​](https://docs.dagster.io/migration/upgrading#deprecation-config-argument "Direct link to Deprecation: config argument") In 0.8.0, we've renamed the common `config` parameter to the user-facing definition APIs to `config_schema`. This is intended to reduce ambiguity between config values (provided at execution time) and their user-specified schemas (provided at definition time). This change affects the `ConfigMapping`, `@composite_solid`, `@solid`, `SolidDefinition`, `@executor`, `ExecutorDefinition`, `@logger`, `LoggerDefinition`, `@resource`, and `ResourceDefinition` APIs. In the CLI, `dagster pipeline execute` and `dagster pipeline launch` now take `-c/--config` instead of `-e/--env`. ### Renaming solid\_subset and enabling support for solid selection DSL in Python API[​](https://docs.dagster.io/migration/upgrading#renaming-solid_subset-and-enabling-support-for-solid-selection-dsl-in-python-api "Direct link to Renaming solid_subset and enabling support for solid selection DSL in Python API") In 0.8.0, we've renamed the `solid_subset`/`--solid-subset` argument to `solid_selection`/`--solid-selection` throughout the Python API and CLI. This affects the `dagster pipeline execute`, `dagster pipeline launch`, and `dagster pipeline backfill` CLI commands, and the `@schedule`, `@monthly_schedule`, `@weekly_schedule`, `@daily_schedule`, `@hourly_schedule`, `ScheduleDefinition`, `PresetDefinition`, `PartitionSetDefinition`, `PartitionScheduleDefinition`, `execute_pipeline`, `execute_pipeline_iterator`, `DagsterInstance.create_run_for_pipeline`, `DagsterInstance.create_run` APIs. In addition to the names of individual solids, the new `solid_selection` argument supports selection queries like `*solid_name++` (i.e., `solid_name`, all of its ancestors, its immediate descendants, and their immediate descendants), previously supported only in Dagit views. ### Removal of deprectated properties, methods, and arguments[​](https://docs.dagster.io/migration/upgrading#removal-of-deprectated-properties-methods-and-arguments "Direct link to Removal of deprectated properties, methods, and arguments") * The deprecated `runtime_type` property on `InputDefinition` and `OutputDefinition` has been removed. Use `dagster_type` instead. * The deprecated `has_runtime_type`, `runtime_type_named`, and `all_runtime_types` methods on `PipelineDefinition` have been removed. Use `has_dagster_type`, `dagster_type_named`, and `all_dagster_types` instead. * The deprecated `all_runtime_types` method on `SolidDefinition` and `CompositeSolidDefinition` has been removed. Use `all_dagster_types` instead. * The deprecated `metadata` argument to `SolidDefinition` and `@solid` has been removed. Use `tags` instead. * The use of `is_optional` throughout the codebase was deprecated in 0.7.x and has been removed. Use `is_required` instead. ### Removal of Path config type[​](https://docs.dagster.io/migration/upgrading#removal-of-path-config-type "Direct link to Removal of Path config type") The built-in config type `Path` has been removed. Use `String`. ### dagster-bash[​](https://docs.dagster.io/migration/upgrading#dagster-bash "Direct link to dagster-bash") This package has been renamed to dagster-shell. The`bash_command_solid` and `bash_script_solid` solid factory functions have been renamed to `create_shell_command_solid` and `create_shell_script_solid`. ### Dask config[​](https://docs.dagster.io/migration/upgrading#dask-config "Direct link to Dask config") The config schema for the `dagster_dask.dask_executor` has changed. The previous config should now be nested under the key `local`. ### Spark solids[​](https://docs.dagster.io/migration/upgrading#spark-solids "Direct link to Spark solids") `dagster_spark.SparkSolidDefinition` has been removed - use `create_spark_solid` instead. Upgrading to 0.7.0[​](https://docs.dagster.io/migration/upgrading#upgrading-to-070 "Direct link to Upgrading to 0.7.0") ------------------------------------------------------------------------------------------------------------------------ The 0.7.0 release contains a number of breaking API changes. While listed in the changelog, this document goes into more detail about how to resolve the change easily. Most of the eliminated or changed APIs can be adjusted to with relatively straightforward changes. The easiest way to use this guide is to search for associated error text. ### Dagster Types[​](https://docs.dagster.io/migration/upgrading#dagster-types "Direct link to Dagster Types") There have been substantial changes to the core dagster type APIs. Error: `ImportError: cannot import name 'dagster_type' from 'dagster'` Fix: Use `usable_as_dagster_type` instead. If dynamically generating types, construct using `DagsterType` instead. Error: `ImportError: cannot import name 'as_dagster_type' from 'dagster'` Fix: Use `make_python_type_usable_as_dagster_type` instead. Error: `dagster.core.errors.DagsterInvalidDefinitionError: type_check_fn argument type "BadType" must take 2 arguments, received 1` Fix: Add a context argument (named `_`, `_context`, `context`, or `context_`) as the first argument of the `type_check_fn`. The second argument is the value being type-checked. Further Information: We have eliminated the `@dagster_type` and `as_dagster_type` APIs, which previously were promoted as our primary type creation API. This API automatically created a mapping between a Python type and a Dagster Type. While convenient, this ended up causing unpredictable behavior based on import order, as well as being wholly incompatible with dynamically created Dagster types. Our core type creation API is now the `DagsterType` class. It creates a Dagster type (which is just an instance of `DagsterType`) that can be passed to `InputDefinition` and `OutputDefinition`. The functionality of `@dagster_type` is preserved, but under a different name: `usable_as_dagster_type`. This decorator signifies that the author wants a bare Python type to be usable in contexts that expect dagster types, such as an `InputDefinition` or `OutputDefinition`. Any user that had been programmatically creating dagster types and was forced to decorate classes in local scope using `@dagster_type` and return that class should instead just create a `DagsterType` directly. `as_dagster_type` has replaced by `make_python_type_usable_as_dagster_type`. The semantics of `as_dagster_type` did not indicate what is was actually doing very well. This function is meant to take an _existing_ type -- often from a library that one doesn't control -- and make that type usable as a dagster type, the second argument. The `type_check_fn` argument has been renamed from `type_check` and now takes two arguments instead of one. The first argument is a instance of `TypeCheckContext`; the second argument is the value being checked. This allows the type check to have access to resources. ### Config System[​](https://docs.dagster.io/migration/upgrading#config-system "Direct link to Config System") The config APIs have been renamed to have no collisions with names in neither python's `typing` API nor the dagster type system. Here are some example errors: Error: `dagster.core.errors.DagsterInvariantViolationError: Cannot resolve Dagster Type Optional.Int to a config type. Repr of type: ` Fix: Use `Noneable` of `Optional`. Error: `TypeError: 'DagsterDictApi' object is not callable` Fix: Pass a raw python dictionary instead of Dict. `config=Dict({'foo': str})` becomes `config={'foo': str}` Error: `ImportError: cannot import name 'PermissiveDict' from 'dagster'` Fix: Use `Permissive` instead. Error: `dagster.core.errors.DagsterInvariantViolationError: Cannot use List in the context of config. Please use a python list (e.g. [int]) or dagster.Array (e.g. Array(int)) instead.` Fix: This happens when a properly constructed List is used within config. Use Array instead. Error: `dagster.core.errors.DagsterInvalidDefinitionError: Invalid type: dagster_type must be DagsterType, a python scalar, or a python type that has been marked usable as a dagster type via @usable_dagster_type or make_python_type_usable_as_dagster_type: got .` Fix: This happens when a List takes an invalid argument and is never constructed. The error could be much better. This is what happens a config type (in this case `Noneable`) is passed to a `List`. The fix is to use either `Array` or to use a bare list with a single element, which is a config type. ### Required Resources[​](https://docs.dagster.io/migration/upgrading#required-resources "Direct link to Required Resources") Any solid, type, or configuration function that accesses a resource off of a context object must declare that resource key with a `required_resource_key` argument. Error: `DagsterUnknownResourceError: Unknown resource . Specify as a required resource on the compute / config function that accessed it.` Fix: Find any references to `context.resources.`, and ensure that the enclosing solid definition, type definition, or config function has the resource key specified in its `required_resource_key` argument. Further information: When only a subset of solids are being executed in a given process, we only need to initialize resources that will be used by that subset of solids. In order to improve the performance of pipeline execution, we need each solid and type to explicitly declare its required resources. As a result, we should see improved performance for pipeline subset execution, multiprocess execution, and retry execution. ### RunConfig Removed[​](https://docs.dagster.io/migration/upgrading#runconfig-removed "Direct link to RunConfig Removed") Error: `AttributeError: 'ComputeExecutionContext' object has no attribute 'run_config'` Fix: Replace all references to `context.run_config` with `context.pipeline_run`. The `run_config` field on the pipeline execution context has been removed and replaced with `pipeline_run`, a `PipelineRun` instance. Along with the fields previously on `RunConfig`, this also includes the pipeline run status. ### Scheduler[​](https://docs.dagster.io/migration/upgrading#scheduler "Direct link to Scheduler") Scheduler configuration has been moved to the `dagster.yaml`. After upgrading, the previous schedule history is no longer compatible with the new storage. Make sure you delete your existing `$DAGSTER_HOME/schedules` directory, then run: dagster schedule wipe && dagster schedule up Error: `TypeError: schedules() got an unexpected keyword argument 'scheduler'` Fix: The `@schedules` decorator no longer takes a `scheduler` argument. Remove the argument and instead configure the scheduler on the instance. Instead of: @schedules(scheduler=SystemCronScheduler)def define_schedules(): ... Remove the `scheduler` argument: @schedulesdef define_schedules(): ... Configure the scheduler on your instance by adding the following to `$DAGSTER_HOME/dagster.yaml`: scheduler: module: dagster_cron.cron_scheduler class: SystemCronScheduler Error: `TypeError: () takes 0 positional arguments but 1 was given"` Stack Trace: File ".../dagster/python_modules/dagster/dagster/core/definitions/schedule.py", line 171, in should_execute return self._should_execute(context) Fix: The `should_execute` and `environment_dict_fn` argument to `ScheduleDefinition` now has a required first argument `context`, representing the `ScheduleExecutionContext`. * [Upgrading to 1.11.0](https://docs.dagster.io/migration/upgrading#upgrading-to-1110) * [Breaking changes](https://docs.dagster.io/migration/upgrading#breaking-changes) * [Upgrading to 1.10.0](https://docs.dagster.io/migration/upgrading#upgrading-to-1100) * [Deprecations](https://docs.dagster.io/migration/upgrading#deprecations) * [Breaking changes](https://docs.dagster.io/migration/upgrading#breaking-changes-1) * [Upgrading to 1.9.0](https://docs.dagster.io/migration/upgrading#upgrading-to-190) * [Database migration](https://docs.dagster.io/migration/upgrading#database-migration) * [Notable behavior changes](https://docs.dagster.io/migration/upgrading#notable-behavior-changes) * [Deprecations](https://docs.dagster.io/migration/upgrading#deprecations-1) * [Breaking changes](https://docs.dagster.io/migration/upgrading#breaking-changes-2) * [Upgrading to 1.8.0](https://docs.dagster.io/migration/upgrading#upgrading-to-180) * [Notable behavior changes](https://docs.dagster.io/migration/upgrading#notable-behavior-changes-1) * [Breaking changes](https://docs.dagster.io/migration/upgrading#breaking-changes-3) * [Deprecations](https://docs.dagster.io/migration/upgrading#deprecations-2) * [Upgrading to 1.7.0](https://docs.dagster.io/migration/upgrading#upgrading-to-170) * [Breaking Changes](https://docs.dagster.io/migration/upgrading#breaking-changes-4) * [Deprecations](https://docs.dagster.io/migration/upgrading#deprecations-3) * [Upgrading to 1.6.0](https://docs.dagster.io/migration/upgrading#upgrading-to-160) * [Breaking changes](https://docs.dagster.io/migration/upgrading#breaking-changes-5) * [Dagster Ingestion-as-Code is being deprecated](https://docs.dagster.io/migration/upgrading#dagster-ingestion-as-code-is-being-deprecated) * [I/O manager `handle_output` will no longer be called when the output typing type is Nothing](https://docs.dagster.io/migration/upgrading#io-manager-handle_output-will-no-longer-be-called-when-the-output-typing-type-is-nothing) * [Deprecations](https://docs.dagster.io/migration/upgrading#deprecations-4) * [dbt](https://docs.dagster.io/migration/upgrading#dbt) * [Upgrading to 1.5.0](https://docs.dagster.io/migration/upgrading#upgrading-to-150) * [Breaking changes](https://docs.dagster.io/migration/upgrading#breaking-changes-6) * [Upgrading to 1.4.0](https://docs.dagster.io/migration/upgrading#upgrading-to-140) * [Deprecations](https://docs.dagster.io/migration/upgrading#deprecations-5) * [Breaking changes](https://docs.dagster.io/migration/upgrading#breaking-changes-7) * [Upgrading to 1.3.0](https://docs.dagster.io/migration/upgrading#upgrading-to-130) * [Deprecations](https://docs.dagster.io/migration/upgrading#deprecations-6) * [Breaking Changes](https://docs.dagster.io/migration/upgrading#breaking-changes-8) * [Upgrading to 1.2.0](https://docs.dagster.io/migration/upgrading#upgrading-to-120) * [Database migration](https://docs.dagster.io/migration/upgrading#database-migration-1) * [Breaking changes](https://docs.dagster.io/migration/upgrading#breaking-changes-9) * [Core changes](https://docs.dagster.io/migration/upgrading#core-changes) * [Changes to experimental APIs](https://docs.dagster.io/migration/upgrading#changes-to-experimental-apis) * [Removal of deprecated APIs](https://docs.dagster.io/migration/upgrading#removal-of-deprecated-apis) * [Extension Libraries](https://docs.dagster.io/migration/upgrading#extension-libraries) * [Upgrading to 1.1.1](https://docs.dagster.io/migration/upgrading#upgrading-to-111) * [Database migration](https://docs.dagster.io/migration/upgrading#database-migration-2) * [Breaking changes and deprecations](https://docs.dagster.io/migration/upgrading#breaking-changes-and-deprecations) * [Upgrading to 1.0](https://docs.dagster.io/migration/upgrading#upgrading-to-10) * [Legacy API Removals](https://docs.dagster.io/migration/upgrading#legacy-api-removals) * [Removed API List](https://docs.dagster.io/migration/upgrading#removed-api-list) * [Upgrading to 0.15.0](https://docs.dagster.io/migration/upgrading#upgrading-to-0150) * [Software-defined assets](https://docs.dagster.io/migration/upgrading#software-defined-assets) * [Event records](https://docs.dagster.io/migration/upgrading#event-records) * [Extension libraries](https://docs.dagster.io/migration/upgrading#extension-libraries-1) * [Other](https://docs.dagster.io/migration/upgrading#other) * [Upgrading to 0.14.0](https://docs.dagster.io/migration/upgrading#upgrading-to-0140) * [Breaking Changes](https://docs.dagster.io/migration/upgrading#breaking-changes-10) * [Deprecation: Metadata API Renames](https://docs.dagster.io/migration/upgrading#deprecation-metadata-api-renames) * [Upgrading to 0.13.0](https://docs.dagster.io/migration/upgrading#upgrading-to-0130) * [Upgrading to 0.12.0](https://docs.dagster.io/migration/upgrading#upgrading-to-0120) * [Upgrading to 0.11.0](https://docs.dagster.io/migration/upgrading#upgrading-to-0110) * [Action Required: Run and event storage schema changes](https://docs.dagster.io/migration/upgrading#action-required-run-and-event-storage-schema-changes) * [Action Required: Schedule timezones](https://docs.dagster.io/migration/upgrading#action-required-schedule-timezones) * [Action Required: Asset storage](https://docs.dagster.io/migration/upgrading#action-required-asset-storage) * [Removals of Deprecated APIs](https://docs.dagster.io/migration/upgrading#removals-of-deprecated-apis) * [Breaking Changes](https://docs.dagster.io/migration/upgrading#breaking-changes-11) * [Upgrading to 0.10.0](https://docs.dagster.io/migration/upgrading#upgrading-to-0100) * [Action Required: Run and event storage schema changes](https://docs.dagster.io/migration/upgrading#action-required-run-and-event-storage-schema-changes-1) * [New scheduler: DagsterDaemonScheduler](https://docs.dagster.io/migration/upgrading#new-scheduler-dagsterdaemonscheduler) * [Steps to migrate](https://docs.dagster.io/migration/upgrading#steps-to-migrate) * [Deprecation: Intermediate Storage](https://docs.dagster.io/migration/upgrading#deprecation-intermediate-storage) * [Steps to Migrate](https://docs.dagster.io/migration/upgrading#steps-to-migrate-1) * [Removal: input\_hydration\_config and output\_materialization\_config](https://docs.dagster.io/migration/upgrading#removal-input_hydration_config-and-output_materialization_config) * [Removal: repository key in workspace YAML](https://docs.dagster.io/migration/upgrading#removal-repository-key-in-workspace-yaml) * [Deprecated: python\_environment key in workspace YAML](https://docs.dagster.io/migration/upgrading#deprecated-python_environment-key-in-workspace-yaml) * [Removal: config\_field property on definition classes](https://docs.dagster.io/migration/upgrading#removal-config_field-property-on-definition-classes) * [Removal: System Storage](https://docs.dagster.io/migration/upgrading#removal-system-storage) * [Removal: step\_keys\_to\_execute](https://docs.dagster.io/migration/upgrading#removal-step_keys_to_execute) * [Breaking Change: date\_partition\_range](https://docs.dagster.io/migration/upgrading#breaking-change-date_partition_range) * [Breaking Change: PartitionSetDefinition.create\_schedule\_definition](https://docs.dagster.io/migration/upgrading#breaking-change-partitionsetdefinitioncreate_schedule_definition) * [Renamed: Helm values](https://docs.dagster.io/migration/upgrading#renamed-helm-values) * [Restructured: scheduler in Helm values](https://docs.dagster.io/migration/upgrading#restructured-scheduler-in-helm-values) * [Restructured: celery and k8sRunLauncher in Helm values](https://docs.dagster.io/migration/upgrading#restructured-celery-and-k8srunlauncher-in-helm-values) * [New Helm defaults](https://docs.dagster.io/migration/upgrading#new-helm-defaults) * [Upgrading to 0.9.0](https://docs.dagster.io/migration/upgrading#upgrading-to-090) * [Removal: config argument](https://docs.dagster.io/migration/upgrading#removal-config-argument) * [Upgrading to 0.8.8](https://docs.dagster.io/migration/upgrading#upgrading-to-088) * [Deprecation: Materialization](https://docs.dagster.io/migration/upgrading#deprecation-materialization) * [Deprecation: system\_storage\_defs](https://docs.dagster.io/migration/upgrading#deprecation-system_storage_defs) * [Upgrading to 0.8.7](https://docs.dagster.io/migration/upgrading#upgrading-to-087) * [Loading python modules from the working directory](https://docs.dagster.io/migration/upgrading#loading-python-modules-from-the-working-directory) * [Upgrading to 0.8.6](https://docs.dagster.io/migration/upgrading#upgrading-to-086) * [dagster-celery](https://docs.dagster.io/migration/upgrading#dagster-celery) * [Deprecation: input\_hydration\_config and output\_materialization\_config](https://docs.dagster.io/migration/upgrading#deprecation-input_hydration_config-and-output_materialization_config) * [Upgrading to 0.8.5](https://docs.dagster.io/migration/upgrading#upgrading-to-085) * [Python 3.5](https://docs.dagster.io/migration/upgrading#python-35) * [Engine and ExecutorConfig -> Executor](https://docs.dagster.io/migration/upgrading#engine-and-executorconfig---executor) * [Upgrading to 0.8.3](https://docs.dagster.io/migration/upgrading#upgrading-to-083) * [Change: gcs\_resource](https://docs.dagster.io/migration/upgrading#change-gcs_resource) * [Upgrading to 0.8.0](https://docs.dagster.io/migration/upgrading#upgrading-to-080) * [Repository loading](https://docs.dagster.io/migration/upgrading#repository-loading) * [Steps to migrate](https://docs.dagster.io/migration/upgrading#steps-to-migrate-2) * [Repository definition](https://docs.dagster.io/migration/upgrading#repository-definition) * [Steps to migrate](https://docs.dagster.io/migration/upgrading#steps-to-migrate-3) * [Reloadable repositories](https://docs.dagster.io/migration/upgrading#reloadable-repositories) * [Renaming environment\_dict and removing RunConfig](https://docs.dagster.io/migration/upgrading#renaming-environment_dict-and-removing-runconfig) * [Deprecation: config argument](https://docs.dagster.io/migration/upgrading#deprecation-config-argument) * [Renaming solid\_subset and enabling support for solid selection DSL in Python API](https://docs.dagster.io/migration/upgrading#renaming-solid_subset-and-enabling-support-for-solid-selection-dsl-in-python-api) * [Removal of deprectated properties, methods, and arguments](https://docs.dagster.io/migration/upgrading#removal-of-deprectated-properties-methods-and-arguments) * [Removal of Path config type](https://docs.dagster.io/migration/upgrading#removal-of-path-config-type) * [dagster-bash](https://docs.dagster.io/migration/upgrading#dagster-bash) * [Dask config](https://docs.dagster.io/migration/upgrading#dask-config) * [Spark solids](https://docs.dagster.io/migration/upgrading#spark-solids) * [Upgrading to 0.7.0](https://docs.dagster.io/migration/upgrading#upgrading-to-070) * [Dagster Types](https://docs.dagster.io/migration/upgrading#dagster-types) * [Config System](https://docs.dagster.io/migration/upgrading#config-system) * [Required Resources](https://docs.dagster.io/migration/upgrading#required-resources) * [RunConfig Removed](https://docs.dagster.io/migration/upgrading#runconfig-removed) * [Scheduler](https://docs.dagster.io/migration/upgrading#scheduler) --- # Dagster & dbt Cloud | Dagster Docs [Skip to main content](https://docs.dagster.io/integrations/libraries/dbt/dbt-cloud#__docusaurus_skipToContent_fallback) On this page info This feature is considered in a beta stage. It is still being tested and may change. For more information, see the [API lifecycle stages documentation](https://docs.dagster.io/api/api-lifecycle/api-lifecycle-stages) . Dagster allows you to run dbt Cloud jobs alongside other technologies. You can schedule them to run as a step in a larger pipeline and manage them as a data asset. Our updated dbt Cloud integration offers two capabilities: * **Observability** - You can view your dbt Cloud assets in the Dagster Asset Graph and double click into run/materialization history. * **Orchestration** - You can use Dagster to schedule runs/materializations of your dbt Cloud assets, either on a cron schedule, or based on upstream dependencies. Installation[​](https://docs.dagster.io/integrations/libraries/dbt/dbt-cloud#installation "Direct link to Installation") ------------------------------------------------------------------------------------------------------------------------- * uv * pip uv add dagster-dbt pip install dagster-dbt Observability example[​](https://docs.dagster.io/integrations/libraries/dbt/dbt-cloud#observability-example "Direct link to Observability example") ---------------------------------------------------------------------------------------------------------------------------------------------------- To make use of the observability capability, you will need to add code to your Dagster project that does the following: 1. Defines your dbt Cloud credentials and workspace. 2. Uses the integration to create asset specs for models in the workspace. 3. Builds a sensor which will poll dbt Cloud for updates on runs/materialization history and dbt Cloud Assets. defs/dbt\_cloud\_observability.py import osimport dagster as dgfrom dagster_dbt.cloud_v2.resources import ( DbtCloudCredentials, DbtCloudWorkspace, load_dbt_cloud_asset_specs,)from dagster_dbt.cloud_v2.sensor_builder import build_dbt_cloud_polling_sensor# Define credentialscreds = DbtCloudCredentials( account_id=os.getenv("DBT_CLOUD_ACCOUNT_ID"), access_url=os.getenv("DBT_CLOUD_ACCESS_URL"), token=os.getenv("DBT_CLOUD_TOKEN"),)# Define the workspaceworkspace = DbtCloudWorkspace( credentials=creds, project_id=os.getenv("DBT_CLOUD_PROJECT_ID"), environment_id=os.getenv("DBT_CLOUD_ENVIRONMENT_ID"),)# Use the integration to create asset specs for models in the workspacedbt_cloud_asset_specs = load_dbt_cloud_asset_specs(workspace=workspace)# Build a sensor which will poll dbt Cloud for updates on runs/materialization history# and dbt Cloud Assetsdbt_cloud_polling_sensor = build_dbt_cloud_polling_sensor(workspace=workspace) Orchestration example[​](https://docs.dagster.io/integrations/libraries/dbt/dbt-cloud#orchestration-example "Direct link to Orchestration example") ---------------------------------------------------------------------------------------------------------------------------------------------------- To make use of the orchestration capability, you will need to add code to your Dagster project that does the following: 1. Defines your dbt Cloud credentials and workspace. 2. Builds your asset graph in a materializable way. 3. Adds these assets to the Declarative Automation Sensor. 4. Builds a sensor to poll dbt Cloud for updates on runs/materialization history and dbt Cloud Assets. defs/dbt\_cloud\_orchestration.py import osimport dagster as dgfrom dagster_dbt.cloud_v2.asset_decorator import dbt_cloud_assetsfrom dagster_dbt.cloud_v2.resources import DbtCloudCredentials, DbtCloudWorkspacefrom dagster_dbt.cloud_v2.sensor_builder import build_dbt_cloud_polling_sensor# Define credentialscreds = DbtCloudCredentials( account_id=os.getenv("DBT_CLOUD_ACCOUNT_ID"), access_url=os.getenv("DBT_CLOUD_ACCESS_URL"), token=os.getenv("DBT_CLOUD_TOKEN"),)# Define the worskpaceworkspace = DbtCloudWorkspace( credentials=creds, project_id=os.getenv("DBT_CLOUD_PROJECT_ID"), environment_id=os.getenv("DBT_CLOUD_ENVIRONMENT_ID"),)# Builds your asset graph in a materializable way@dbt_cloud_assets(workspace=workspace)def my_dbt_cloud_assets( context: dg.AssetExecutionContext, dbt_cloud: DbtCloudWorkspace): yield from dbt_cloud.cli(args=["build"], context=context).wait()# Automates your assets using Declarative Automation# https://docs.dagster.io/guides/automate/declarative-automationmy_dbt_cloud_assets = my_dbt_cloud_assets.map_asset_specs( lambda spec: spec.replace_attributes( automation_condition=dg.AutomationCondition.eager() ))# Adds these assets to the Declarative Automation Sensorautomation_sensor = dg.AutomationConditionSensorDefinition( name="automation_sensor", target="*", default_status=dg.DefaultSensorStatus.RUNNING, minimum_interval_seconds=1,)# Build a sensor which will poll dbt Cloud for updates on runs/materialization history# and dbt Cloud Assetsdbt_cloud_polling_sensor = build_dbt_cloud_polling_sensor(workspace=workspace) About dbt Cloud[​](https://docs.dagster.io/integrations/libraries/dbt/dbt-cloud#about-dbt-cloud "Direct link to About dbt Cloud") ---------------------------------------------------------------------------------------------------------------------------------- **dbt Cloud** is a hosted service for running dbt jobs. It helps data analysts and engineers productionize dbt deployments. Beyond dbt open source, dbt Cloud provides scheduling , CI/CD, serving documentation, and monitoring & alerting. If you're currently using dbt Cloud™, you can also use Dagster to run `dbt-core` in its place. You can read more about [how to do that here](https://dagster.io/blog/migrate-off-dbt-cloud) . * [Installation](https://docs.dagster.io/integrations/libraries/dbt/dbt-cloud#installation) * [Observability example](https://docs.dagster.io/integrations/libraries/dbt/dbt-cloud#observability-example) * [Orchestration example](https://docs.dagster.io/integrations/libraries/dbt/dbt-cloud#orchestration-example) * [About dbt Cloud](https://docs.dagster.io/integrations/libraries/dbt/dbt-cloud#about-dbt-cloud) --- # Quickstart | Dagster Docs [Skip to main content](https://docs.dagster.io/integrations/libraries/dbt/quickstart#__docusaurus_skipToContent_fallback) On this page Dagster orchestrates dbt alongside other technologies, so you can schedule dbt with Spark, Python, etc. in a single data pipeline. Dagster's asset-oriented approach allows Dagster to understand dbt at the level of individual dbt models. Prerequisites To follow the steps in this guide, you'll need: * A basic understanding of dbt, DuckDB, and Dagster concepts such as [assets](https://docs.dagster.io/guides/build/assets) and [resources](https://docs.dagster.io/guides/build/external-resources) * To install the [dbt](https://docs.getdbt.com/docs/core/installation-overview) and [DuckDB CLIs](https://duckdb.org/docs/api/cli/overview.html) * To install the following packages: * uv * pip uv add dagster duckdb plotly pandas dagster-dbt dbt-duckdb pip install dagster duckdb plotly pandas dagster-dbt dbt-duckdb Setting up a basic dbt project[​](https://docs.dagster.io/integrations/libraries/dbt/quickstart#setting-up-a-basic-dbt-project "Direct link to Setting up a basic dbt project") -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Start by downloading this basic dbt project, which includes a few models and a DuckDB backend: git clone https://github.com/dagster-io/basic-dbt-project The project structure should look like this: ├── README.md├── dbt_project.yml├── profiles.yml├── models│ └── example│ ├── my_first_dbt_model.sql│ ├── my_second_dbt_model.sql│ └── schema.yml First, you need to point Dagster at the dbt project and ensure Dagster has what it needs to build an asset graph. Create a `definitions.py` in the same directory as the dbt project: definitions.py from pathlib import Pathfrom dagster_dbt import DbtCliResource, DbtProject, dbt_assetsimport dagster as dg# Points to the dbt project pathdbt_project_directory = Path(__file__).absolute().parent / "basic-dbt-project"dbt_project = DbtProject(project_dir=dbt_project_directory)# References the dbt project objectdbt_resource = DbtCliResource(project_dir=dbt_project)# Compiles the dbt project & allow Dagster to build an asset graphdbt_project.prepare_if_dev()# Yields Dagster events streamed from the dbt CLI@dbt_assets(manifest=dbt_project.manifest_path)def dbt_models(context: dg.AssetExecutionContext, dbt: DbtCliResource): yield from dbt.cli(["build"], context=context).stream()# Dagster object that contains the dbt assets and resourcedefs = dg.Definitions(assets=[dbt_models], resources={"dbt": dbt_resource}) Adding upstream dependencies[​](https://docs.dagster.io/integrations/libraries/dbt/quickstart#adding-upstream-dependencies "Direct link to Adding upstream dependencies") -------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Often, you'll want Dagster to generate data that will be used by downstream dbt models. To do this, add an upstream asset that the dbt project will use as a source: definitions.py import osfrom pathlib import Pathimport duckdbimport pandas as pdfrom dagster_dbt import DbtCliResource, DbtProject, dbt_assetsimport dagster as dgduckdb_database_path = "basic-dbt-project/dev.duckdb"@dg.asset(compute_kind="python")def raw_customers(context: dg.AssetExecutionContext) -> None: # Pull customer data from a CSV data = pd.read_csv("https://docs.dagster.io/assets/customers.csv") connection = duckdb.connect(os.fspath(duckdb_database_path)) # Create a schema named raw connection.execute("create schema if not exists raw") # Create/replace table named raw_customers connection.execute( "create or replace table raw.raw_customers as select * from data" ) # Log some metadata about the new table. It will show up in the UI. context.add_output_metadata({"num_rows": data.shape[0]})dbt_project_directory = Path(__file__).absolute().parent / "basic-dbt-project"dbt_project = DbtProject(project_dir=dbt_project_directory)dbt_resource = DbtCliResource(project_dir=dbt_project)dbt_project.prepare_if_dev()@dbt_assets(manifest=dbt_project.manifest_path)def dbt_models(context: dg.AssetExecutionContext, dbt: DbtCliResource): yield from dbt.cli(["build"], context=context).stream()# Add Dagster definitions to Definitions objectdefs = dg.Definitions( assets=[raw_customers, dbt_models], resources={"dbt": dbt_resource},) Next, add a dbt model that will source the `raw_customers` asset and define the dependency for Dagster. Create the dbt model: customers.sql select id as customer_id, first_name, last_namefrom {{ source('raw', 'raw_customers') }} # Define the raw_customers asset as a source Next, create a `_source.yml` file that points dbt to the upstream `raw_customers` asset: \_source.yml\_ version: 2sources: - name: raw tables: - name: raw_customers meta: # This metadata: dagster: # Tells dbt where this model's source data is, and asset_key: ["raw_customers"] # Tells Dagster which asset represents the source data ![Screenshot of dbt lineage](https://docs.dagster.io/assets/images/dbt-lineage-b96f38a3b75748c96b9578aa02ebcbfd.png) Adding downstream dependencies[​](https://docs.dagster.io/integrations/libraries/dbt/quickstart#adding-downstream-dependencies "Direct link to Adding downstream dependencies") -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- You may also have assets that depend on the output of dbt models. Next, create an asset that depends on the result of the new `customers` model. This asset will create a histogram of the first names of the customers: definitions.py import osfrom pathlib import Pathimport duckdbimport pandas as pdimport plotly.express as pxfrom dagster_dbt import DbtCliResource, DbtProject, dbt_assets, get_asset_key_for_modelimport dagster as dgduckdb_database_path = "basic-dbt-project/dev.duckdb"@dg.asset(compute_kind="python")def raw_customers(context: dg.AssetExecutionContext) -> None: # Pull customer data from a CSV data = pd.read_csv("https://docs.dagster.io/assets/customers.csv") connection = duckdb.connect(os.fspath(duckdb_database_path)) # Create a schema named raw connection.execute("create schema if not exists raw") # Create/replace table named raw_customers connection.execute( "create or replace table raw.raw_customers as select * from data" ) # Log some metadata about the new table. It will show up in the UI. context.add_output_metadata({"num_rows": data.shape[0]})dbt_project_directory = Path(__file__).absolute().parent / "basic-dbt-project"dbt_project = DbtProject(project_dir=dbt_project_directory)dbt_resource = DbtCliResource(project_dir=dbt_project)dbt_project.prepare_if_dev()@dbt_assets(manifest=dbt_project.manifest_path)def dbt_models(context: dg.AssetExecutionContext, dbt: DbtCliResource): yield from dbt.cli(["build"], context=context).stream()@dg.asset( compute_kind="python", # Defines the dependency on the customers model, # which is represented as an asset in Dagster deps=[get_asset_key_for_model([dbt_models], "customers")],)def customer_histogram(context: dg.AssetExecutionContext): # Read the contents of the customers table into a Pandas DataFrame connection = duckdb.connect(os.fspath(duckdb_database_path)) customers = connection.sql("select * from customers").df() # Create a customer histogram and write it out to an HTML file fig = px.histogram(customers, x="FIRST_NAME") fig.update_layout(bargap=0.2) fig.update_xaxes(categoryorder="total ascending") save_chart_path = Path(duckdb_database_path).parent.joinpath( "order_count_chart.html" ) fig.write_html(save_chart_path, auto_open=True) # Tell Dagster about the location of the HTML file, # so it's easy to access from the Dagster UI context.add_output_metadata( {"plot_url": dg.MetadataValue.url("file://" + os.fspath(save_chart_path))} )# Add Dagster definitions to Definitions objectdefs = dg.Definitions( assets=[raw_customers, dbt_models, customer_histogram], resources={"dbt": dbt_resource},) Scheduling dbt models[​](https://docs.dagster.io/integrations/libraries/dbt/quickstart#scheduling-dbt-models "Direct link to Scheduling dbt models") ----------------------------------------------------------------------------------------------------------------------------------------------------- You can schedule your dbt models by using the `dagster-dbt`'s `build_schedule_from_dbt_selection` function: Scheduling our dbt models import osfrom pathlib import Pathimport duckdbimport pandas as pdimport plotly.express as pxfrom dagster_dbt import ( DbtCliResource, DbtProject, build_schedule_from_dbt_selection, dbt_assets, get_asset_key_for_model,)import dagster as dgduckdb_database_path = "basic-dbt-project/dev.duckdb"@dg.asset(compute_kind="python")def raw_customers(context: dg.AssetExecutionContext) -> None: # Pull customer data from a CSV data = pd.read_csv("https://docs.dagster.io/assets/customers.csv") connection = duckdb.connect(os.fspath(duckdb_database_path)) # Create a schema named raw connection.execute("create schema if not exists raw") # Create/replace table named raw_customers connection.execute( "create or replace table raw.raw_customers as select * from data" ) # Log some metadata about the new table. It will show up in the UI. context.add_output_metadata({"num_rows": data.shape[0]})dbt_project_directory = Path(__file__).absolute().parent / "basic-dbt-project"dbt_project = DbtProject(project_dir=dbt_project_directory)dbt_resource = DbtCliResource(project_dir=dbt_project)dbt_project.prepare_if_dev()@dbt_assets(manifest=dbt_project.manifest_path)def dbt_models(context: dg.AssetExecutionContext, dbt: DbtCliResource): yield from dbt.cli(["build"], context=context).stream()@dg.asset( compute_kind="python", # Defines the dependency on the customers model, # which is represented as an asset in Dagster deps=[get_asset_key_for_model([dbt_models], "customers")],)def customer_histogram(context: dg.AssetExecutionContext): # Read the contents of the customers table into a Pandas DataFrame connection = duckdb.connect(os.fspath(duckdb_database_path)) customers = connection.sql("select * from customers").df() # Create a customer histogram and write it out to an HTML file fig = px.histogram(customers, x="FIRST_NAME") fig.update_layout(bargap=0.2) fig.update_xaxes(categoryorder="total ascending") save_chart_path = Path(duckdb_database_path).parent.joinpath( "order_count_chart.html" ) fig.write_html(save_chart_path, auto_open=True) # Tell Dagster about the location of the HTML file, # so it's easy to access from the Dagster UI context.add_output_metadata( {"plot_url": dg.MetadataValue.url("file://" + os.fspath(save_chart_path))} )# Build a schedule for the job that materializes a selection of dbt assetsdbt_schedule = build_schedule_from_dbt_selection( [dbt_models], job_name="materialize_dbt_models", cron_schedule="0 0 * * *", dbt_select="fqn:*",)# Add Dagster definitions to Definitions objectdefs = dg.Definitions( assets=[raw_customers, dbt_models, customer_histogram], resources={"dbt": dbt_resource}, schedules=[dbt_schedule],) * [Setting up a basic dbt project](https://docs.dagster.io/integrations/libraries/dbt/quickstart#setting-up-a-basic-dbt-project) * [Adding upstream dependencies](https://docs.dagster.io/integrations/libraries/dbt/quickstart#adding-upstream-dependencies) * [Adding downstream dependencies](https://docs.dagster.io/integrations/libraries/dbt/quickstart#adding-downstream-dependencies) * [Scheduling dbt models](https://docs.dagster.io/integrations/libraries/dbt/quickstart#scheduling-dbt-models) --- # 32 docs tagged with "Community supported" | Dagster Docs [Skip to main content](https://docs.dagster.io/tags/integrations/community-supported#__docusaurus_skipToContent_fallback) [Dagster & Anthropic\ -------------------](https://docs.dagster.io/integrations/libraries/anthropic) The Anthropic integration allows you to easily interact with the Anthropic REST API using the Anthropic Python API to build AI steps into your Dagster pipelines. You can also log Anthropic API usage metadata in Dagster Insights, giving you detailed observability on API call credit consumption. [Dagster & Census\ ----------------](https://docs.dagster.io/integrations/libraries/census) With the Census integration you can execute a Census sync and poll until that sync completes, raising an error if it's unsuccessful. [Dagster & Chroma\ ----------------](https://docs.dagster.io/integrations/libraries/chroma) The Chroma library allows you to easily interact with Chroma's vector database capabilities to build AI-powered data pipelines in Dagster. You can perform vector similarity searches, manage schemas, and handle data operations directly from your Dagster assets. [Dagster & Cube\ --------------](https://docs.dagster.io/integrations/libraries/cube) With the Cube integration you can setup Cube and Dagster to work together so that Dagster can push changes from upstream data sources to Cube using its integration API. [Dagster & DingTalk\ ------------------](https://docs.dagster.io/integrations/libraries/dingtalk) The community-supported DingTalk package provides an integration with DingTalk. [Dagster & Evidence\ ------------------](https://docs.dagster.io/integrations/libraries/evidence) The Evidence library offers a component to easily generate dashboards from your Evidence project. [Dagster & GCP Cloud Run\ -----------------------](https://docs.dagster.io/integrations/libraries/gcp/cloud-run-launcher) The community-supported dagster-contrib-gcp package provides integrations with Google Cloud Platform (GCP) services. [Dagster & Gemini\ ----------------](https://docs.dagster.io/integrations/libraries/gemini) The Gemini library allows you to easily interact with the Gemini REST API using the Gemini Python API to build AI steps into your Dagster pipelines. You can also log Gemini API usage metadata in Dagster Insights, giving you detailed observability on API call credit consumption. [Dagster & HashiCorp\ -------------------](https://docs.dagster.io/integrations/libraries/hashicorp-nomad) The community-supported Nomad package provides an integration with HashiCorp Nomad. [Dagster & HashiCorp Vault\ -------------------------](https://docs.dagster.io/integrations/libraries/hashicorp) A package for integrating HashiCorp Vault into Dagster so that you can securely manage tokens and passwords. [Dagster & Hex\ -------------](https://docs.dagster.io/integrations/libraries/hex) The community-supported Hex package provides an integration with Hex. [Dagster & Hightouch\ -------------------](https://docs.dagster.io/integrations/libraries/hightouch) With this integration you can trigger Hightouch syncs and monitor them from within Dagster. Fine-tune when Hightouch syncs kick-off, visualize their dependencies, and monitor the steps in your data activation workflow. [Dagster & Iceberg\ -----------------](https://docs.dagster.io/integrations/libraries/iceberg/) This library provides I/O managers for reading and writing Apache Iceberg tables. It also provides a Dagster resource for accessing Iceberg tables. [Dagster & Java\ --------------](https://docs.dagster.io/integrations/libraries/java) The Java Pipes client provides a Java implementation of the Dagster Pipes protocol that can be used to orchestrate data processing pipelines written in Java from Dagster, while receiving logs and metadata from the Java application. [Dagster & LakeFS\ ----------------](https://docs.dagster.io/integrations/libraries/lakefs) By integrating with lakeFS, a big data scale version control system, you can leverage the versioning capabilities of lakeFS to track changes to your data. This integration allows you to have a complete lineage of your data, from the initial raw data to the transformed and processed data, making it easier to understand and reproduce data transformations. [Dagster & Meltano\ -----------------](https://docs.dagster.io/integrations/libraries/meltano) The Meltano library allows you to run Meltano using Dagster. Design and configure ingestion jobs using the popular Singer specification. [Dagster & Microsoft Teams\ -------------------------](https://docs.dagster.io/integrations/libraries/microsoft-teams) An integration with Microsoft Teams to post messages to MS Teams from any Dagster op or asset. [Dagster & Modal\ ---------------](https://docs.dagster.io/integrations/libraries/modal) The community-supported Modal package provides an integration with Modal. [Dagster & MSSQL Bulk Copy Tool\ ------------------------------](https://docs.dagster.io/integrations/libraries/mssql-bulk-copy-tool) The community-supported MSSQL BCP package is a custom Dagster I/O manager for loading data into SQL Server using the BCP utility. [Dagster & Not Diamond\ ---------------------](https://docs.dagster.io/integrations/libraries/notdiamond) Leverage the Not Diamond resource to easily determine which LLM provider is most appropriate for your use case. [Dagster & obstore\ -----------------](https://docs.dagster.io/integrations/libraries/obstore) The community-supported obstore package provides an integration with obstore, providing three lean integrations with object stores, ADLS, GCS & S3. [Dagster & Open Metadata\ -----------------------](https://docs.dagster.io/integrations/libraries/open-metadata) With this integration you can create a Open Metadata service to ingest metadata produced by the Dagster application. View the Ingestion Pipeline running from the Open Metadata Service Page. [Dagster & Patito\ ----------------](https://docs.dagster.io/integrations/libraries/patito) Patito is a data validation framework for Polars, based on Pydantic. [Dagster & Perian\ ----------------](https://docs.dagster.io/integrations/libraries/perian) The Perian integration allows you to easily dockerize your codebase and execute it on the PERIAN platform, PERIAN's serverless GPU environment. [Dagster & Polars\ ----------------](https://docs.dagster.io/integrations/libraries/polars) The Polars integration allows using Polars eager or lazy DataFrames as inputs and outputs with Dagster’s assets and ops. Type annotations are used to control whether to load an eager or lazy DataFrame. Lazy DataFrames can be sinked as output. Multiple serialization formats (Parquet, Delta Lake, BigQuery) and filesystems (local, S3, GCS, …) are supported. [Dagster & Qdrant\ ----------------](https://docs.dagster.io/integrations/libraries/qdrant) The Qdrant library lets you integrate Qdrant's vector database with Dagster, making it easy to build AI-driven data pipelines. You can run vector searches and manage data directly within Dagster. [Dagster & Ray\ -------------](https://docs.dagster.io/integrations/libraries/ray) The community-supported Ray package allows orchestrating distributed Ray compute from Dagster pipelines. [Dagster & Rust\ --------------](https://docs.dagster.io/integrations/libraries/rust) The Rust Pipes client allows full observability into your Rust workloads when orchestrating through Dagster. [Dagster & Secoda\ ----------------](https://docs.dagster.io/integrations/libraries/secoda) Connect Dagster to Secoda and see metadata related to your Dagster assets, asset groups and jobs right in Secoda. Simplify your team's access, and remove the need to switch between tools. [Dagster & Teradata\ ------------------](https://docs.dagster.io/integrations/libraries/teradata) The community-supported Teradata package provides an integration with Teradata Vantage. [Dagster & Weaviate\ ------------------](https://docs.dagster.io/integrations/libraries/weaviate) The Weaviate library allows you to easily interact with Weaviate's vector database capabilities to build AI-powered data pipelines in Dagster. You can perform vector similarity searches, manage schemas, and handle data operations directly from your Dagster assets. [Dagster & Weights & Biases\ --------------------------](https://docs.dagster.io/integrations/libraries/wandb) Use Dagster and Weights & Biases (W&B) to orchestrate your MLOps pipelines and maintain ML assets. --- # Dagster & dbt Cloud (Legacy) | Dagster Docs [Skip to main content](https://docs.dagster.io/integrations/libraries/dbt/dbt-cloud-legacy#__docusaurus_skipToContent_fallback) On this page warning This feature is considered superseded. While it is still available, it is no longer the best practice. For more information, see the [API lifecycle stages documentation](https://docs.dagster.io/api/api-lifecycle/api-lifecycle-stages) . Dagster allows you to run dbt Cloud jobs alongside other technologies. You can schedule them to run as a step in a larger pipeline and manage them as a data asset. Installation[​](https://docs.dagster.io/integrations/libraries/dbt/dbt-cloud-legacy#installation "Direct link to Installation") -------------------------------------------------------------------------------------------------------------------------------- * uv * pip uv add dagster-dbt pip install dagster-dbt Example[​](https://docs.dagster.io/integrations/libraries/dbt/dbt-cloud-legacy#example "Direct link to Example") ----------------------------------------------------------------------------------------------------------------- import osfrom dagster_dbt import dbt_cloud_resource, load_assets_from_dbt_cloud_jobimport dagster as dgdef scope_define_instance(): # start_define_dbt_cloud_instance from dagster_dbt import DbtCloudClientResource from dagster import EnvVar dbt_cloud_instance = DbtCloudClientResource( auth_token=EnvVar("DBT_CLOUD_API_TOKEN"), account_id=EnvVar.int("DBT_CLOUD_ACCOUNT_ID"), ) # end_define_dbt_cloud_instance return dbt_cloud_instancedef scope_load_assets_from_dbt_cloud_job(): from dagster_dbt import DbtCloudClientResource from dagster import EnvVar dbt_cloud_instance = DbtCloudClientResource( auth_token=EnvVar("DBT_CLOUD_API_TOKEN"), account_id=EnvVar.int("DBT_CLOUD_ACCOUNT_ID"), ) # start_load_assets_from_dbt_cloud_job from dagster_dbt import load_assets_from_dbt_cloud_job # Use the dbt_cloud_instance resource we defined in Step 1, and the job_id from Prerequisites dbt_cloud_assets = load_assets_from_dbt_cloud_job( dbt_cloud=dbt_cloud_instance, job_id=33333, ) # end_load_assets_from_dbt_cloud_jobdef scope_schedule_dbt_cloud_assets(dbt_cloud_assets): # start_schedule_dbt_cloud_assets import dagster as dg # Materialize all assets run_everything_job = dg.define_asset_job( "run_everything_job", dg.AssetSelection.all() ) defs = dg.Definitions( # Use the dbt_cloud_assets defined in Step 2 assets=[dbt_cloud_assets], schedules=[ dg.ScheduleDefinition( job=run_everything_job, cron_schedule="@daily", ), ], ) # end_schedule_dbt_cloud_assets About dbt Cloud[​](https://docs.dagster.io/integrations/libraries/dbt/dbt-cloud-legacy#about-dbt-cloud "Direct link to About dbt Cloud") ----------------------------------------------------------------------------------------------------------------------------------------- **dbt Cloud** is a hosted service for running dbt jobs. It helps data analysts and engineers productionize dbt deployments. Beyond dbt open source, dbt Cloud provides scheduling , CI/CD, serving documentation, and monitoring & alerting. If you're currently using dbt Cloud™, you can also use Dagster to run `dbt-core` in its place. You can read more about [how to do that here](https://dagster.io/blog/migrate-off-dbt-cloud) . * [Installation](https://docs.dagster.io/integrations/libraries/dbt/dbt-cloud-legacy#installation) * [Example](https://docs.dagster.io/integrations/libraries/dbt/dbt-cloud-legacy#example) * [About dbt Cloud](https://docs.dagster.io/integrations/libraries/dbt/dbt-cloud-legacy#about-dbt-cloud) --- # dagster-deltalake integration reference | Dagster Docs [Skip to main content](https://docs.dagster.io/integrations/libraries/deltalake/reference#__docusaurus_skipToContent_fallback) On this page This reference page provides information for working with [`dagster-deltalake`](https://docs.dagster.io/api/libraries/dagster-deltalake) features that are not covered as part of the [Using Delta Lake with Dagster tutorial](https://docs.dagster.io/integrations/libraries/deltalake/using-deltalake-with-dagster) . * [Selecting specific columns in a downstream asset](https://docs.dagster.io/integrations/libraries/deltalake/reference#selecting-specific-columns-in-a-downstream-asset) * [Storing partitioned assets](https://docs.dagster.io/integrations/libraries/deltalake/reference#storing-partitioned-assets) * [Storing tables in multiple schemas](https://docs.dagster.io/integrations/libraries/deltalake/reference#storing-tables-in-multiple-schemas) * [Using the Delta Lake I/O manager with other I/O managers](https://docs.dagster.io/integrations/libraries/deltalake/reference#using-the-delta-lake-io-manager-with-other-io-managers) * [Storing and loading PyArrow Tables or Polars DataFrames in Delta Lake](https://docs.dagster.io/integrations/libraries/deltalake/reference#storing-and-loading-pyarrow-tables-or-polars-dataframes-in-delta-lake) * [Configuring storage backends](https://docs.dagster.io/integrations/libraries/deltalake/reference#configuring-storage-backends) Selecting specific columns in a downstream asset[​](https://docs.dagster.io/integrations/libraries/deltalake/reference#selecting-specific-columns-in-a-downstream-asset "Direct link to Selecting specific columns in a downstream asset") ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Sometimes you may not want to fetch an entire table as the input to a downstream asset. With the Delta Lake I/O manager, you can select specific columns to load by supplying metadata on the downstream asset. import pandas as pdfrom dagster import AssetIn, asset# this example uses the iris_dataset asset from Step 2 of the Using Dagster with Delta Lake tutorial@asset( ins={ "iris_sepal": AssetIn( key="iris_dataset", metadata={"columns": ["sepal_length_cm", "sepal_width_cm"]}, ) })def sepal_data(iris_sepal: pd.DataFrame) -> pd.DataFrame: iris_sepal["sepal_area_cm2"] = ( iris_sepal["sepal_length_cm"] * iris_sepal["sepal_width_cm"] ) return iris_sepal In this example, we only use the columns containing sepal data from the `iris_dataset` table created in [Step 2](https://docs.dagster.io/integrations/libraries/deltalake/using-deltalake-with-dagster#step-2-create-delta-lake-tables) of the [Using Dagster with Delta Lake tutorial](https://docs.dagster.io/integrations/libraries/deltalake/using-deltalake-with-dagster) . To select specific columns, we can add metadata to the input asset. We do this in the `metadata` parameter of the `AssetIn` that loads the `iris_dataset` asset in the `ins` parameter. We supply the key `columns` with a list of names of the columns we want to fetch. When Dagster materializes `sepal_data` and loads the `iris_dataset` asset using the Delta Lake I/O manager, it will only fetch the `sepal_length_cm` and `sepal_width_cm` columns of the `iris/iris_dataset` table and pass them to `sepal_data` as a Pandas DataFrame. Storing partitioned assets[​](https://docs.dagster.io/integrations/libraries/deltalake/reference#storing-partitioned-assets "Direct link to Storing partitioned assets") ------------------------------------------------------------------------------------------------------------------------------------------------------------------------- The Delta Lake I/O manager supports storing and loading partitioned data. To correctly store and load data from the Delta table, the Delta Lake I/O manager needs to know which column contains the data defining the partition bounds. The Delta Lake I/O manager uses this information to construct the correct queries to select or replace the data. In the following sections, we describe how the I/O manager constructs these queries for different types of partitions. ::: For partitioning to work, the partition dimension needs to be one of the partition columns defined on the Delta table. Tables created via the I/O manager will be configured accordingly. ::: * Static partitioned assets * Time-partitioned assets * Multi-partitioned assets * Dynamic-partitioned assets **Storing static partitioned assets** To store static partitioned assets in your Delta Lake, specify `partition_expr` metadata on the asset to tell the Delta Lake I/O manager which column contains the partition data: import pandas as pdfrom dagster import StaticPartitionsDefinition, asset@asset( partitions_def=StaticPartitionsDefinition( ["Iris-setosa", "Iris-virginica", "Iris-versicolor"] ), metadata={"partition_expr": "species"},)def iris_dataset_partitioned(context) -> pd.DataFrame: species = context.partition_key full_df = pd.read_csv( "https://docs.dagster.io/assets/iris.csv", names=[ "sepal_length_cm", "sepal_width_cm", "petal_length_cm", "petal_width_cm", "species", ], ) return full_df[full_df["species"] == species]@assetdef iris_cleaned(iris_dataset_partitioned: pd.DataFrame): return iris_dataset_partitioned.dropna().drop_duplicates() Dagster uses the `partition_expr` metadata to generate appropriate function parameters when loading the partition in the downstream asset. When loading a static partition this roughly corresponds to the following SQL statement: SELECT * WHERE [partition_expr] in ([selected partitions]) A partition must be selected when materializing the above assets, as described in the [Materializing partitioned assets](https://docs.dagster.io/guides/build/partitions-and-backfills/partitioning-assets) documentation. In this example, the query used when materializing the `Iris-setosa` partition of the above assets would be: SELECT * WHERE species = 'Iris-setosa' **Storing time-partitioned assets** Like static partitioned assets, you can specify `partition_expr` metadata on the asset to tell the Delta Lake I/O manager which column contains the partition data: import pandas as pdfrom dagster import DailyPartitionsDefinition, asset@asset( partitions_def=DailyPartitionsDefinition(start_date="2023-01-01"), metadata={"partition_expr": "time"},)def iris_data_per_day(context) -> pd.DataFrame: partition = context.partition_key # get_iris_data_for_date fetches all of the iris data for a given date, # the returned dataframe contains a column named 'time' with that stores # the time of the row as an integer of seconds since epoch return get_iris_data_for_date(partition)@assetdef iris_cleaned(iris_data_per_day: pd.DataFrame): return iris_data_per_day.dropna().drop_duplicates() Dagster uses the `partition_expr` metadata to craft the `SELECT` statement when loading the correct partition in the downstream asset. When loading a dynamic partition, the following statement is used: SELECT * WHERE [partition_expr] = [partition_start] A partition must be selected when materializing the above assets, as described in the [Materializing partitioned assets](https://docs.dagster.io/guides/build/partitions-and-backfills/partitioning-assets) documentation. The `[partition_start]` and `[partition_end]` bounds are of the form `YYYY-MM-DD HH:MM:SS`. In this example, the query when materializing the `2023-01-02` partition of the above assets would be: SELECT * WHERE time = '2023-01-02 00:00:00' **Storing multi-partitioned assets** The Delta Lake I/O manager can also store data partitioned on multiple dimensions. To do this, specify the column for each partition as a dictionary of `partition_expr` metadata: import pandas as pdimport dagster as dg@dg.asset( partitions_def=dg.MultiPartitionsDefinition( { "date": dg.DailyPartitionsDefinition(start_date="2023-01-01"), "species": dg.StaticPartitionDefinition( ["Iris-setosa", "Iris-virginica", "Iris-versicolor"] ), } ), metadata={"partition_expr": {"date": "time", "species": "species"}},)def iris_dataset_partitioned(context) -> pd.DataFrame: partition = context.partition_key.keys_by_dimension species = partition["species"] date = partition["date"] # get_iris_data_for_date fetches all of the iris data for a given date, # the returned dataframe contains a column named 'time' with that stores # the time of the row as an integer of seconds since epoch full_df = get_iris_data_for_date(date) return full_df[full_df["species"] == species]@dg.assetdef iris_cleaned(iris_dataset_partitioned: pd.DataFrame): return iris_dataset_partitioned.dropna().drop_duplicates() Dagster uses the `partition_expr` metadata to craft the `SELECT` statement when loading the correct partition in a downstream asset. For multi-partitions, Dagster concatenates the `WHERE` statements described in the above sections to craft the correct `SELECT` statement. A partition must be selected when materializing the above assets, as described in the [Partitioning assets](https://docs.dagster.io/guides/build/partitions-and-backfills/partitioning-assets#materializing-partitioned-assets) documentation. For example, when materializing the `2023-01-02|Iris-setosa` partition of the above assets, the following query will be used: SELECT * WHERE species = 'Iris-setosa' AND time = '2023-01-02 00:00:00' Storing tables in multiple schemas[​](https://docs.dagster.io/integrations/libraries/deltalake/reference#storing-tables-in-multiple-schemas "Direct link to Storing tables in multiple schemas") ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- You may want to have different assets stored in different Delta Lake schemas. The Delta Lake I/O manager allows you to specify the schema in several ways. If you want all of your assets to be stored in the same schema, you can specify the schema as configuration to the I/O manager, as we did in [Step 1](https://docs.dagster.io/integrations/libraries/deltalake/using-deltalake-with-dagster#step-1-configure-the-delta-lake-io-manager) of the [Using Dagster with Delta Lake tutorial](https://docs.dagster.io/integrations/libraries/deltalake/using-deltalake-with-dagster) . If you want to store assets in different schemas, you can specify the schema as part of the asset's key: import pandas as pdfrom dagster import AssetSpec, assetdaffodil_dataset = AssetSpec(key=["daffodil", "daffodil_dataset"])@asset(key_prefix=["iris"])def iris_dataset() -> pd.DataFrame: return pd.read_csv( "https://docs.dagster.io/assets/iris.csv", names=[ "sepal_length_cm", "sepal_width_cm", "petal_length_cm", "petal_width_cm", "species", ], ) In this example, the `iris_dataset` asset will be stored in the `IRIS` schema, and the `daffodil_dataset` asset will be found in the `DAFFODIL` schema. ::: The two options for specifying schema are mutually exclusive. If you provide `schema` configuration to the I/O manager, you cannot also provide it via the asset key and vice versa. If no `schema` is provided, either from configuration or asset keys, the default schema `public` will be used. ::: Using the Delta Lake I/O manager with other I/O managers[​](https://docs.dagster.io/integrations/libraries/deltalake/reference#using-the-delta-lake-io-manager-with-other-io-managers "Direct link to Using the Delta Lake I/O manager with other I/O managers") ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- You may have assets that you don't want to store in Delta Lake. You can provide an I/O manager to each asset using the `io_manager_key` parameter in the [`@dg.asset`](https://docs.dagster.io/api/dagster/assets#dagster.asset) decorator: import pandas as pdfrom dagster_aws.s3.io_manager import s3_pickle_io_managerfrom dagster_deltalake import LocalConfigfrom dagster_deltalake_pandas import DeltaLakePandasIOManagerfrom dagster import Definitions, asset@asset(io_manager_key="warehouse_io_manager")def iris_dataset() -> pd.DataFrame: return pd.read_csv( "https://docs.dagster.io/assets/iris.csv", names=[ "sepal_length_cm", "sepal_width_cm", "petal_length_cm", "petal_width_cm", "species", ], )@asset(io_manager_key="blob_io_manager")def iris_plots(iris_dataset): # plot_data is a function we've defined somewhere else # that plots the data in a DataFrame return plot_data(iris_dataset)defs = Definitions( assets=[iris_dataset, iris_plots], resources={ "warehouse_io_manager": DeltaLakePandasIOManager( root_uri="path/to/deltalalke", storage_options=LocalConfig(), schema="iris", ), "blob_io_manager": s3_pickle_io_manager, },) In this example: * The `iris_dataset` asset uses the I/O manager bound to the key `warehouse_io_manager` and `iris_plots` uses the I/O manager bound to the key `blob_io_manager` * In the [`Definitions`](https://docs.dagster.io/api/dagster/definitions#dagster.Definitions) object, we supply the I/O managers for those keys * When the assets are materialized, the `iris_dataset` will be stored in Delta Lake, and `iris_plots` will be saved in Amazon S3 Storing and loading PyArrow tables or Polars DataFrames in Delta Lake[​](https://docs.dagster.io/integrations/libraries/deltalake/reference#storing-and-loading-pyarrow-tables-or-polars-dataframes-in-delta-lake "Direct link to Storing and loading PyArrow tables or Polars DataFrames in Delta Lake") ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- The Delta Lake I/O manager also supports storing and loading PyArrow and Polars DataFrames. * PyArrow Tables **Storing and loading PyArrow Tables with Delta Lake** The `deltalake` package relies heavily on Apache Arrow for efficient data transfer, so PyArrow is natively supported. You can use the `DeltaLakePyArrowIOManager` in a [`Definitions`](https://docs.dagster.io/api/dagster/definitions#dagster.Definitions) object as in [Step 1](https://docs.dagster.io/integrations/libraries/deltalake/using-deltalake-with-dagster#step-1-configure-the-delta-lake-io-manager) of the [Using Dagster with Delta Lake tutorial](https://docs.dagster.io/integrations/libraries/deltalake/using-deltalake-with-dagster) . from dagster_deltalake import DeltaLakePyarrowIOManager, LocalConfigfrom dagster import Definitionsdefs = Definitions( assets=[iris_dataset], resources={ "io_manager": DeltaLakePyarrowIOManager( root_uri="path/to/deltalake", # required storage_options=LocalConfig(), # required schema="iris", # optional, defaults to PUBLIC ) },) Configuring storage backends[​](https://docs.dagster.io/integrations/libraries/deltalake/reference#configuring-storage-backends "Direct link to Configuring storage backends") ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- The deltalake library comes with support for many storage backends out of the box. Which exact storage is to be used, is derived from the URL of a storage location. ### S3 compatible storages[​](https://docs.dagster.io/integrations/libraries/deltalake/reference#s3-compatible-storages "Direct link to S3 compatible storages") The S3 APIs are implemented by a number of providers and it is possible to interact with many of them. However, most S3 implementations do not offer support for atomic operations, which is a requirement for multi writer support. As such some additional setup and configuration is required. * Unsafe rename * Set-up a locking client * Cloudflare R2 storage In case there will always be only a single writer to a table - this includes no concurrent dagster jobs writing to the same table - you can allow unsafe writes to the table. from dagster_deltalake import S3Configconfig = S3Config(allow_unsafe_rename=True) To use DynamoDB, set the `AWS_S3_LOCKING_PROVIDER` variable to `dynamodb` and create a table named delta\_rs\_lock\_table in Dynamo. An example DynamoDB table creation snippet using the aws CLI follows, and should be customized for your environment’s needs (e.g. read/write capacity modes): aws dynamodb create-table --table-name delta_rs_lock_table \ --attribute-definitions \ AttributeName=key,AttributeType=S \ --key-schema \ AttributeName=key,KeyType=HASH \ --provisioned-throughput \ ReadCapacityUnits=10,WriteCapacityUnits=10 ::: The delta-rs community is actively working on extending the available options for locking backends. This includes locking backends compatible with Databricks to allow concurrent writes from Databricks and external environments. ::: Cloudflare R2 storage has built-in support for atomic copy operations. This can be leveraged by sending additional headers with the copy requests. from dagster_deltalake import S3Configconfig = S3Config(copy_if_not_exists="header: cf-copy-destination-if-none-match: *") In cases where non-AWS S3 implementations are used, the endpoint URL or the S3 service needs to be provided. config = S3Config(endpoint="https://") ### Working with locally running storage (emulators)[​](https://docs.dagster.io/integrations/libraries/deltalake/reference#working-with-locally-running-storage-emulators "Direct link to Working with locally running storage (emulators)") A common pattern for e.g. integration tests is to run a storage emulator like Azurite, Localstack, o.a. If not configured to use TLS, we need to configure the http client, to allow for http traffic. config = AzureConfig(use_emulator=True, client=ClientConfig(allow_http=True)) * [Selecting specific columns in a downstream asset](https://docs.dagster.io/integrations/libraries/deltalake/reference#selecting-specific-columns-in-a-downstream-asset) * [Storing partitioned assets](https://docs.dagster.io/integrations/libraries/deltalake/reference#storing-partitioned-assets) * [Storing tables in multiple schemas](https://docs.dagster.io/integrations/libraries/deltalake/reference#storing-tables-in-multiple-schemas) * [Using the Delta Lake I/O manager with other I/O managers](https://docs.dagster.io/integrations/libraries/deltalake/reference#using-the-delta-lake-io-manager-with-other-io-managers) * [Storing and loading PyArrow tables or Polars DataFrames in Delta Lake](https://docs.dagster.io/integrations/libraries/deltalake/reference#storing-and-loading-pyarrow-tables-or-polars-dataframes-in-delta-lake) * [Configuring storage backends](https://docs.dagster.io/integrations/libraries/deltalake/reference#configuring-storage-backends) * [S3 compatible storages](https://docs.dagster.io/integrations/libraries/deltalake/reference#s3-compatible-storages) * [Working with locally running storage (emulators)](https://docs.dagster.io/integrations/libraries/deltalake/reference#working-with-locally-running-storage-emulators) --- # 48 docs tagged with "Dagster supported" | Dagster Docs [Skip to main content](https://docs.dagster.io/tags/integrations/dagster-supported#__docusaurus_skipToContent_fallback) [Dagster & Azure Data Lake Storage Gen 2\ ---------------------------------------](https://docs.dagster.io/integrations/libraries/azure-adls2) Dagster helps you use Azure Storage Accounts as part of your data pipeline. Azure Data Lake Storage Gen 2 (ADLS2) is our primary focus but we also provide utilities for Azure Blob Storage. [Dagster & Airbyte\ -----------------](https://docs.dagster.io/integrations/libraries/airbyte/airbyte-oss) Using this integration, you can trigger Airbyte syncs and orchestrate your Airbyte connections from within Dagster, making it easy to chain an Airbyte sync with upstream or downstream steps in your workflow. [Dagster & Airbyte\ -----------------](https://docs.dagster.io/integrations/libraries/airbyte/) Orchestrate Airbyte connections and schedule syncs alongside upstream or downstream dependencies. [Dagster & Airlift\ -----------------](https://docs.dagster.io/integrations/libraries/airlift) Airlift is a toolkit for integrating Dagster and Airflow. [Dagster & AWS Athena\ --------------------](https://docs.dagster.io/integrations/libraries/aws/athena) This integration allows you to connect to AWS Athena, a serverless interactive query service that makes it easy to analyze data in Amazon S3 using standard SQL. Using this integration, you can issue queries to Athena, fetch results, and handle query execution states within your Dagster pipelines. [Dagster & AWS CloudWatch\ ------------------------](https://docs.dagster.io/integrations/libraries/aws/cloudwatch) This integration allows you to send Dagster logs to AWS CloudWatch, enabling centralized logging and monitoring of your Dagster jobs. By using AWS CloudWatch, you can take advantage of its powerful log management features, such as real-time log monitoring, log retention policies, and alerting capabilities. [Dagster & AWS ECR\ -----------------](https://docs.dagster.io/integrations/libraries/aws/ecr) This integration allows you to connect to AWS Elastic Container Registry (ECR). It provides resources to interact with AWS ECR, enabling you to manage your container images. [Dagster & AWS EMR\ -----------------](https://docs.dagster.io/integrations/libraries/aws/emr) The AWS integration provides ways orchestrating data pipelines that leverage AWS services, including AWS EMR (Elastic MapReduce). This integration allows you to run and scale big data workloads using open source tools such as Apache Spark, Hive, Presto, and more. [Dagster & AWS Glue\ ------------------](https://docs.dagster.io/integrations/libraries/aws/glue) The AWS integration library provides the PipesGlueClient resource, enabling you to launch AWS Glue jobs directly from Dagster assets and ops. This integration allows you to pass parameters to Glue code while Dagster receives real-time events, such as logs, asset checks, and asset materializations, from the initiated jobs. With minimal code changes required on the job side, this integration is both efficient and easy to implement. [Dagster & AWS Lambda\ --------------------](https://docs.dagster.io/integrations/libraries/aws/lambda) Using this integration, you can leverage AWS Lambda to execute external code as part of your Dagster pipelines. This is particularly useful for running serverless functions that can scale automatically and handle various workloads without the need for managing infrastructure. The PipesLambdaClient class allows you to invoke AWS Lambda functions and stream logs and structured metadata back to Dagster's UI and tools. [Dagster & AWS Redshift\ ----------------------](https://docs.dagster.io/integrations/libraries/aws/redshift) Using this integration, you can connect to an AWS Redshift cluster and issue queries against it directly from your Dagster assets. This allows you to seamlessly integrate Redshift into your data pipelines, leveraging the power of Redshift's data warehousing capabilities within your Dagster workflows. [Dagster & AWS S3\ ----------------](https://docs.dagster.io/integrations/libraries/aws/s3) The AWS S3 integration allows data engineers to easily read, and write objects to the durable AWS S3 storage enabling engineers to a resilient storage layer when constructing their pipelines. [Dagster & AWS Secrets Manager\ -----------------------------](https://docs.dagster.io/integrations/libraries/aws/secretsmanager) This integration allows you to manage, retrieve, and rotate credentials, API keys, and other secrets using AWS Secrets Manager. [Dagster & AWS Systems Parameter Store\ -------------------------------------](https://docs.dagster.io/integrations/libraries/aws/ssm) The Dagster AWS Systems Manager (SSM) Parameter Store integration allows you to manage and retrieve parameters stored in AWS SSM Parameter Store directly within your Dagster pipelines. This integration provides resources to fetch parameters by name, tags, or paths, and optionally set them as environment variables for your operations. [Dagster & Databricks\ --------------------](https://docs.dagster.io/integrations/libraries/databricks) The Databricks integration library provides the \`PipesDatabricksClient\` resource, enabling you to launch Databricks jobs directly from Dagster assets and ops. This integration allows you to pass parameters to Databricks code while Dagster receives real-time events, such as logs, asset checks, and asset materializations, from the initiated jobs. With minimal code changes required on the job side, this integration is both efficient and easy to implement. [Dagster & Datadog\ -----------------](https://docs.dagster.io/integrations/libraries/datadog) While Dagster provides comprehensive monitoring and observability of the pipelines it orchestrates, many teams look to centralize all their monitoring across apps, processes and infrastructure using Datadog's 'Cloud Monitoring as a Service'. The Datadog integration allows you to publish metrics to Datadog from within Dagster ops. [Dagster & dbt\ -------------](https://docs.dagster.io/integrations/libraries/dbt/) Orchestrate your dbt transformations directly with Dagster. [Dagster & dbt Cloud\ -------------------](https://docs.dagster.io/integrations/libraries/dbt/dbt-cloud) Dagster allows you to run dbt Cloud jobs alongside other technologies. You can schedule them to run as a step in a larger pipeline and manage them as a data asset. [Dagster & dbt Cloud (Legacy)\ ----------------------------](https://docs.dagster.io/integrations/libraries/dbt/dbt-cloud-legacy) Dagster allows you to run dbt Cloud jobs alongside other technologies. You can schedule them to run as a step in a larger pipeline and manage them as a data asset. [Dagster & Delta Lake\ --------------------](https://docs.dagster.io/integrations/libraries/deltalake/) Delta Lake is a great storage format for Dagster workflows. With this integration, you can use the Delta Lake I/O Manager to read and write your Dagster assets. [Dagster & dlt\ -------------](https://docs.dagster.io/integrations/libraries/dlt) The dltHub open-source library defines a standardized approach for creating data pipelines that load often messy data sources into well-structured data sets. [Dagster & Docker\ ----------------](https://docs.dagster.io/integrations/libraries/docker) The Docker integration library provides the PipesDockerClient resource, enabling you to launch Docker containers and execute external code directly from Dagster assets and ops. This integration allows you to pass parameters to Docker containers while Dagster receives real-time events, such as logs, asset checks, and asset materializations, from the initiated jobs. With minimal code changes required on the job side, this integration is both efficient and easy to implement. [Dagster & DuckDB\ ----------------](https://docs.dagster.io/integrations/libraries/duckdb/) This library provides an integration with the DuckDB database, and allows for an out-of-the-box I/O Manager so that you can make DuckDB your storage of choice. [Dagster & Embedded ELT\ ----------------------](https://docs.dagster.io/integrations/libraries/embedded-elt) The Embedded ELT package provides a framework for building ELT pipelines with Dagster through helpful asset decorators and resources. It includes the dagster-dlt and dagster-sling packages, which you can also use on their own. [Dagster & GCP BigQuery\ ----------------------](https://docs.dagster.io/integrations/libraries/gcp/bigquery/) Integrate with GCP BigQuery. [Dagster & GCP Dataproc\ ----------------------](https://docs.dagster.io/integrations/libraries/gcp/dataproc) Using this integration, you can manage and interact with Google Cloud Platform's Dataproc service directly from Dagster. This integration allows you to create, manage, and delete Dataproc clusters, and submit and monitor jobs on these clusters. [Dagster & GCP GCS\ -----------------](https://docs.dagster.io/integrations/libraries/gcp/gcs) This integration allows you to interact with Google Cloud Storage (GCS) using Dagster. It provides resources, I/O Managers, and utilities to manage and store data in GCS, making it easier to integrate GCS into your data pipelines. [Dagster & GitHub\ ----------------](https://docs.dagster.io/integrations/libraries/github) This library provides an integration with GitHub Apps by providing a thin wrapper on the GitHub v4 GraphQL API. This allows for automating operations within your GitHub repositories and with the tighter permissions scopes that GitHub Apps allow for vs using a personal token. [Dagster & Jupyter Notebooks\ ---------------------------](https://docs.dagster.io/integrations/libraries/jupyter/) Dagstermill eliminates the tedious "productionization" of Jupyter notebooks. [Dagster & Kubernetes\ --------------------](https://docs.dagster.io/integrations/libraries/kubernetes) The Kubernetes integration library provides the PipesK8sClient resource, enabling you to launch Kubernetes pods and execute external code directly from Dagster assets and ops. This integration allows you to pass parameters to Kubernetes pods while Dagster receives real-time events, such as logs, asset checks, and asset materializations, from the initiated jobs. With minimal code changes required on the job side, this integration is both efficient and easy to implement. [Dagster & Looker\ ----------------](https://docs.dagster.io/integrations/libraries/looker) The Looker integration allows you to monitor your Looker project as assets in Dagster, along with other data assets. [Dagster & OpenAI\ ----------------](https://docs.dagster.io/integrations/libraries/openai) The OpenAI library allows you to easily interact with the OpenAI REST API using the OpenAI Python API to build AI steps into your Dagster pipelines. You can also log OpenAI API usage metadata in Dagster Insights, giving you detailed observability on API call credit consumption. [Dagster & PagerDuty\ -------------------](https://docs.dagster.io/integrations/libraries/pagerduty) This library provides an integration between Dagster and PagerDuty to support creating alerts from your Dagster code. [Dagster & Pandas\ ----------------](https://docs.dagster.io/integrations/libraries/pandas) Implement validation on pandas DataFrames. [Dagster & Pandera\ -----------------](https://docs.dagster.io/integrations/libraries/pandera) The Pandera integration library provides an API for generating Dagster Types from Pandera dataframe schemas. Like all Dagster types, Pandera-generated types can be used to annotate op inputs and outputs. [Dagster & Power BI\ ------------------](https://docs.dagster.io/integrations/libraries/powerbi) Your Power BI assets, such as semantic models, data sources, reports, and dashboards, can be represented in the Dagster asset graph, allowing you to track lineage and dependencies between Power BI assets and upstream data assets you are already modeling in Dagster. You can also use Dagster to orchestrate Power BI semantic models, allowing you to trigger refreshes of these models on a cadence or based on upstream data changes. [Dagster & Prometheus\ --------------------](https://docs.dagster.io/integrations/libraries/prometheus) This integration allows you to push metrics to the Prometheus gateway from within a Dagster pipeline. [Dagster & Sigma\ ---------------](https://docs.dagster.io/integrations/libraries/sigma) Your Sigma assets, including datasets and workbooks, can be represented in the Dagster asset graph, allowing you to track lineage and dependencies between Sigma assets and upstream data assets you are already modeling in Dagster. [Dagster & Slack\ ---------------](https://docs.dagster.io/integrations/libraries/slack) This library provides an integration with Slack to support posting messages in your company's Slack workspace. [Dagster & Sling\ ---------------](https://docs.dagster.io/integrations/libraries/sling) Sling provides an easy-to-use YAML configuration layer for loading data from files, replicating data between databases, exporting custom SQL queries to cloud storage, and much more. [Dagster & Snowflake\ -------------------](https://docs.dagster.io/integrations/libraries/snowflake/) This library provides an integration with the Snowflake data warehouse. Connect to Snowflake as a resource, then use the integration-provided functions to construct an op to establish connections and execute Snowflake queries. Read and write natively to Snowflake from Dagster assets. [Dagster & Spark\ ---------------](https://docs.dagster.io/integrations/libraries/spark) Running Spark code often requires submitting code to a Databricks or EMR cluster. The Pyspark integration provides a Spark class with methods for configuration and constructing the spark-submit command for a Spark job. [Dagster & SSH/SFTP\ ------------------](https://docs.dagster.io/integrations/libraries/ssh-sftp) This integration provides a resource for SSH remote execution using Paramiko. It allows you to establish secure connections to networked resources and execute commands remotely. The integration also provides an SFTP client for secure file transfers between the local and remote systems. [Dagster & Tableau\ -----------------](https://docs.dagster.io/integrations/libraries/tableau) Your Tableau assets, such as data sources, sheets, and dashboards, can be represented in the Dagster asset graph, allowing you to track lineage and dependencies between Tableau assets and upstream data assets you are already modeling in Dagster. [Dagster & Twilio\ ----------------](https://docs.dagster.io/integrations/libraries/twilio) Use your Twilio Account SID and Auth Token to build Twilio tasks right into your Dagster pipeline. [Dagster & TypeScript\ --------------------](https://docs.dagster.io/integrations/libraries/typescript) The dagster-pipes-typescript npm package is a Dagster Pipes implementation for the TypeScript programming language that allows integration between any TypeScript process and the Dagster orchestrator. [Using Dagster with Airbyte Cloud\ --------------------------------](https://docs.dagster.io/integrations/libraries/airbyte/airbyte-cloud) Orchestrate Airbyte Cloud connections and schedule syncs alongside upstream or downstream dependencies. [Using Dagster with Fivetran\ ---------------------------](https://docs.dagster.io/integrations/libraries/fivetran) Orchestrate Fivetran connectors syncs with upstream or downstream dependencies. --- # Dagster & DingTalk | Dagster Docs [Skip to main content](https://docs.dagster.io/integrations/libraries/dingtalk#__docusaurus_skipToContent_fallback) On this page The community-supported DingTalk package provides an integration with DingTalk. For more information, see the [dagster-dingtalk GitHub repository](https://github.com/sqkkyzx/dagster-dingtalk) . Installation[​](https://docs.dagster.io/integrations/libraries/dingtalk#installation "Direct link to Installation") -------------------------------------------------------------------------------------------------------------------- * uv * pip uv add dagster-dingtalk pip install dagster-dingtalk * [Installation](https://docs.dagster.io/integrations/libraries/dingtalk#installation) --- # Dagster & Delta Lake | Dagster Docs [Skip to main content](https://docs.dagster.io/integrations/libraries/deltalake#__docusaurus_skipToContent_fallback) On this page Delta Lake is a great storage format for Dagster workflows. With this integration, you can use the Delta Lake I/O Manager to read and write your Dagster assets. Here are some of the benefits that Delta Lake provides Dagster users: * Native PyArrow integration for lazy computation of large datasets * More efficient querying with file skipping with Z Ordering and liquid clustering * Built-in vacuuming to remove unnecessary files and versions * ACID transactions for reliable writes * Smooth versioning integration (versions can be use to trigger downstream updates). * Surfacing table stats based on the file statistics Installation[​](https://docs.dagster.io/integrations/libraries/deltalake#installation "Direct link to Installation") --------------------------------------------------------------------------------------------------------------------- * uv * pip uv add dagster-deltalake dagster-deltalake-pandas dagster-deltalake-polars pip install dagster-deltalake dagster-deltalake-pandas dagster-deltalake-polars About Delta Lake[​](https://docs.dagster.io/integrations/libraries/deltalake#about-delta-lake "Direct link to About Delta Lake") --------------------------------------------------------------------------------------------------------------------------------- Delta Lake is an open source storage framework that enables building a Lakehouse architecture with compute engines including Spark, PrestoDB, Flink, Trino, and Hive and APIs for Scala, Java, Rust, and Python. * [Installation](https://docs.dagster.io/integrations/libraries/deltalake#installation) * [About Delta Lake](https://docs.dagster.io/integrations/libraries/deltalake#about-delta-lake) --- # Dagster & Embedded ELT | Dagster Docs [Skip to main content](https://docs.dagster.io/integrations/libraries/embedded-elt#__docusaurus_skipToContent_fallback) On this page The Embedded ELT package provides a framework for building ELT pipelines with Dagster through helpful asset decorators and resources. It includes the dagster-dlt and dagster-sling packages, which you can also use on their own. This package includes two integrations: * [Sling](https://slingdata.io/) provides a simple way to sync data between databases and file systems. * [data Load Tool (dlt)](https://dlthub.com/) easily loads data from external systems and APIs. Installation[​](https://docs.dagster.io/integrations/libraries/embedded-elt#installation "Direct link to Installation") ------------------------------------------------------------------------------------------------------------------------ * uv * pip uv add dagster-embedded-elt pip install dagster-embedded-elt Sling[​](https://docs.dagster.io/integrations/libraries/embedded-elt#sling "Direct link to Sling") --------------------------------------------------------------------------------------------------- Sling provides an easy-to-use YAML configuration layer for loading data from files, replicating data between databases, exporting custom SQL queries to cloud storage, and much more. The Dagster integration allows you to derive Dagster assets from a replication configuration file. For more information, see the [Sling integration docs](https://docs.dagster.io/integrations/libraries/sling) . dlt[​](https://docs.dagster.io/integrations/libraries/embedded-elt#dlt "Direct link to dlt") --------------------------------------------------------------------------------------------- With the ability to leverage pre-made [verified sources](https://dlthub.com/docs/dlt-ecosystem/verified-sources) like [Hubspot](https://dlthub.com/docs/dlt-ecosystem/verified-sources/hubspot) and [Notion](https://dlthub.com/docs/dlt-ecosystem/verified-sources/notion) , and [destinations](https://dlthub.com/docs/dlt-ecosystem/destinations) like [Databricks](https://dlthub.com/docs/dlt-ecosystem/destinations/databricks) and [Snowflake](https://dlthub.com/docs/dlt-ecosystem/destinations/snowflake) , integrating dlt into your Dagster project enables you to load a data in an easy and structured way. For more information, see the [dlt integration docs](https://docs.dagster.io/integrations/libraries/dlt) . * [Installation](https://docs.dagster.io/integrations/libraries/embedded-elt#installation) * [Sling](https://docs.dagster.io/integrations/libraries/embedded-elt#sling) * [dlt](https://docs.dagster.io/integrations/libraries/embedded-elt#dlt) --- # Using Delta Lake with Dagster | Dagster Docs [Skip to main content](https://docs.dagster.io/integrations/libraries/deltalake/using-deltalake-with-dagster#__docusaurus_skipToContent_fallback) On this page This tutorial focuses on how to store and load Dagster [asset definitions](https://docs.dagster.io/guides/build/assets/defining-assets) in a Delta Lake. By the end of the tutorial, you will: * Configure a Delta Lake I/O manager * Create a table in Delta Lake using a Dagster asset * Make a Delta Lake table available in Dagster * Load Delta tables in downstream assets While this guide focuses on storing and loading Pandas DataFrames in Delta Lakes, Dagster also supports using PyArrow Tables and Polars DataFrames. Learn more about setting up and using the Delta Lake I/O manager with PyArrow Tables and Polars DataFrames in the [Delta Lake reference](https://docs.dagster.io/integrations/libraries/deltalake/reference) . Prerequisites[​](https://docs.dagster.io/integrations/libraries/deltalake/using-deltalake-with-dagster#prerequisites "Direct link to Prerequisites") ----------------------------------------------------------------------------------------------------------------------------------------------------- To complete this tutorial, you'll need to install the `dagster-deltalake` and `dagster-deltalake-pandas` libraries: * uv * pip uv add dagster-deltalake dagster-deltalake-pandas pip install dagster-deltalake dagster-deltalake-pandas Step 1: Configure the Delta Lake I/O manager[​](https://docs.dagster.io/integrations/libraries/deltalake/using-deltalake-with-dagster#step-1-configure-the-delta-lake-io-manager "Direct link to Step 1: Configure the Delta Lake I/O manager") ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ The Delta Lake I/O manager requires some configuration to set up your Delta Lake. You must provide a root path where your Delta tables will be created. Additionally, you can specify a `schema` where the Delta Lake I/O manager will create tables. from dagster_deltalake import LocalConfigfrom dagster_deltalake_pandas import DeltaLakePandasIOManagerfrom dagster import Definitionsdefs = Definitions( assets=[iris_dataset], resources={ "io_manager": DeltaLakePandasIOManager( root_uri="path/to/deltalake", # required storage_options=LocalConfig(), # required schema="iris", # optional, defaults to "public" ) },) With this configuration, if you materialized an asset called `iris_dataset`, the Delta Lake I/O manager would store the data within a folder `iris/iris_dataset` under the provided root directory `path/to/deltalake`. Finally, in the [`Definitions`](https://docs.dagster.io/api/dagster/definitions#dagster.Definitions) object, we assign the [`DeltaLakePandasIOManager`](https://docs.dagster.io/api/libraries/dagster-deltalake-pandas#dagster_deltalake_pandas.DeltaLakePandasIOManager) to the `io_manager` key. `io_manager` is a reserved key to set the default I/O manager for your assets. Step 2: Create Delta Lake tables[​](https://docs.dagster.io/integrations/libraries/deltalake/using-deltalake-with-dagster#step-2-create-delta-lake-tables "Direct link to Step 2: Create Delta Lake tables") ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- The Delta Lake I/O manager can create and update tables for your Dagster-defined assets, but you can also make existing Delta Lake tables available to Dagster. * Create Delta tables from Dagster assets * Make existing tables available in Dagster **Store a Dagster asset as a table in Delta Lake** To store data in Delta Lake using the Delta Lake I/O manager, the definitions of your assets don't need to change. You can tell Dagster to use the Delta Lake I/O manager, like in [Step 1](https://docs.dagster.io/integrations/libraries/deltalake/using-deltalake-with-dagster#step-1-configure-the-delta-lake-io-manager) , and Dagster will handle storing and loading your assets in Delta Lake. import pandas as pdfrom dagster import asset@assetdef iris_dataset() -> pd.DataFrame: return pd.read_csv( "https://docs.dagster.io/assets/iris.csv", names=[ "sepal_length_cm", "sepal_width_cm", "petal_length_cm", "petal_width_cm", "species", ], ) In this example, we first define an [asset](https://docs.dagster.io/guides/build/assets/defining-assets) . Here, we fetch the Iris dataset as a Pandas DataFrame and rename the columns. The type signature of the function tells the I/O manager what data type it is working with, so it's important to include the return type `pd.DataFrame`. When Dagster materializes the `iris_dataset` asset using the configuration from [Step 1](https://docs.dagster.io/integrations/libraries/deltalake/using-deltalake-with-dagster#step-1-configure-the-delta-lake-io-manager) , the Delta Lake I/O manager will create the table `iris/iris_dataset` if it doesn't exist and replace the contents of the table with the value returned from the `iris_dataset` asset. ### Make an existing table available in Dagster[​](https://docs.dagster.io/integrations/libraries/deltalake/using-deltalake-with-dagster#make-an-existing-table-available-in-dagster "Direct link to Make an existing table available in Dagster") If you already have tables in your Delta Lake, you may want to make them available to other Dagster assets. You can accomplish this by defining [external assets](https://docs.dagster.io/guides/build/assets/external-assets) for these tables. By creating an external asset for the existing table, you tell Dagster how to find the table so it can be fetched for downstream assets. from dagster import AssetSpeciris_harvest_data = AssetSpec(key="iris_harvest_data") In this example, we create a [`AssetSpec`](https://docs.dagster.io/api/dagster/assets#dagster.AssetSpec) for an existing table containing iris harvest data. To make the data available to other Dagster assets, we need to tell the Delta Lake I/O manager how to find the data. Because we already supplied the database and schema in the I/O manager configuration in [Step 1](https://docs.dagster.io/integrations/libraries/deltalake/using-deltalake-with-dagster#step-1-configure-the-delta-lake-io-manager) , we only need to provide the table name. We do this with the `key` parameter in `AssetSpec`. When the I/O manager needs to load the `iris_harvest_data` in a downstream asset, it will select the data in the `iris/iris_harvest_data` folder as a Pandas DataFrame and provide it to the downstream asset. Step 3: Load Delta Lake tables in downstream assets[​](https://docs.dagster.io/integrations/libraries/deltalake/using-deltalake-with-dagster#step-3-load-delta-lake-tables-in-downstream-assets "Direct link to Step 3: Load Delta Lake tables in downstream assets") ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Once you've created an asset that represents a table in your Delta Lake, you will likely want to create additional assets that work with the data. Dagster and the Delta Lake I/O manager allow you to load the data stored in Delta tables into downstream assets. import pandas as pdfrom dagster import asset# this example uses the iris_dataset asset from Step 2@assetdef iris_cleaned(iris_dataset: pd.DataFrame) -> pd.DataFrame: return iris_dataset.dropna().drop_duplicates() In this example, we want to provide the `iris_dataset` asset to the `iris_cleaned` asset. Refer to the Store a Dagster asset as a table in Delta Lake example in [step 2](https://docs.dagster.io/integrations/libraries/deltalake/using-deltalake-with-dagster#step-2-create-delta-lake-tables) for a look at the `iris_dataset` asset. In `iris_cleaned`, the `iris_dataset` parameter tells Dagster that the value for the `iris_dataset` asset should be provided as input to `iris_cleaned`. If this feels too magical for you, refer to the docs for explicitly specifying dependencies. When materializing these assets, Dagster will use the `DeltaLakePandasIOManager` to fetch the `iris/iris_dataset` as a Pandas DataFrame and pass the DataFrame as the `iris_dataset` parameter to `iris_cleaned`. When `iris_cleaned` returns a Pandas DataFrame, Dagster will use the `DeltaLakePandasIOManager` to store the DataFrame as the `iris/iris_cleaned` table in your Delta Lake. Completed code example[​](https://docs.dagster.io/integrations/libraries/deltalake/using-deltalake-with-dagster#completed-code-example "Direct link to Completed code example") -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- When finished, your code should look like the following: import pandas as pdfrom dagster_deltalake import LocalConfigfrom dagster_deltalake_pandas import DeltaLakePandasIOManagerfrom dagster import AssetSpec, Definitions, assetiris_harvest_data = AssetSpec(key="iris_harvest_data")@assetdef iris_dataset() -> pd.DataFrame: return pd.read_csv( "https://docs.dagster.io/assets/iris.csv", names=[ "sepal_length_cm", "sepal_width_cm", "petal_length_cm", "petal_width_cm", "species", ], )@assetdef iris_cleaned(iris_dataset: pd.DataFrame) -> pd.DataFrame: return iris_dataset.dropna().drop_duplicates()defs = Definitions( assets=[iris_dataset, iris_harvest_data, iris_cleaned], resources={ "io_manager": DeltaLakePandasIOManager( root_uri="path/to/deltalake", storage_options=LocalConfig(), schema="IRIS", ) },) Related[​](https://docs.dagster.io/integrations/libraries/deltalake/using-deltalake-with-dagster#related "Direct link to Related") ----------------------------------------------------------------------------------------------------------------------------------- For more Delta Lake features, refer to the [Delta Lake reference](https://docs.dagster.io/integrations/libraries/deltalake/reference) . For more information on asset definitions, see the [Assets documentation](https://docs.dagster.io/guides/build/assets/defining-assets) . For more information on I/O managers, refer to the [I/O manager documentation](https://docs.dagster.io/guides/build/io-managers) . * [Prerequisites](https://docs.dagster.io/integrations/libraries/deltalake/using-deltalake-with-dagster#prerequisites) * [Step 1: Configure the Delta Lake I/O manager](https://docs.dagster.io/integrations/libraries/deltalake/using-deltalake-with-dagster#step-1-configure-the-delta-lake-io-manager) * [Step 2: Create Delta Lake tables](https://docs.dagster.io/integrations/libraries/deltalake/using-deltalake-with-dagster#step-2-create-delta-lake-tables) * [Make an existing table available in Dagster](https://docs.dagster.io/integrations/libraries/deltalake/using-deltalake-with-dagster#make-an-existing-table-available-in-dagster) * [Step 3: Load Delta Lake tables in downstream assets](https://docs.dagster.io/integrations/libraries/deltalake/using-deltalake-with-dagster#step-3-load-delta-lake-tables-in-downstream-assets) * [Completed code example](https://docs.dagster.io/integrations/libraries/deltalake/using-deltalake-with-dagster#completed-code-example) * [Related](https://docs.dagster.io/integrations/libraries/deltalake/using-deltalake-with-dagster#related) --- # Making a dbt project accessible to Dagster+ Hybrid | Dagster Docs [Skip to main content](https://docs.dagster.io/integrations/libraries/dbt/using-dbt-with-dagster-plus/hybrid#__docusaurus_skipToContent_fallback) On this page If you have a Hybrid deployment, you must make the dbt project accessible to the Dagster code executed by your agent. * When using Amazon Elastic Container Service (ECS), Kubernetes, or Docker agent, you must include the dbt project in the Docker Image containing your Dagster code. * When using a local agent, you must make your dbt project accessible to your Dagster code on the same machine as your agent. For the dbt project to be used by Dagster, it must contain an up-to-date [manifest file](https://docs.getdbt.com/reference/artifacts/manifest-json) and [project dependencies](https://docs.getdbt.com/docs/collaborate/govern/project-dependencies) . In this guide, we'll demonstrate how to prepare your dbt project for use in your Hybrid deployment in Dagster+. Prerequisites[​](https://docs.dagster.io/integrations/libraries/dbt/using-dbt-with-dagster-plus/hybrid#prerequisites "Direct link to Prerequisites") ----------------------------------------------------------------------------------------------------------------------------------------------------- To follow the steps in this guide, you'll need **an existing dbt project** that contains the following files in the repository root: * [`dbt_project.yml`](https://docs.getdbt.com/reference/dbt_project.yml) * [`profiles.yml`](https://docs.getdbt.com/docs/core/connect-data-platform/profiles.yml) Using an Amazon Elastic Container Service (ECS), Kubernetes, or Docker agent[​](https://docs.dagster.io/integrations/libraries/dbt/using-dbt-with-dagster-plus/hybrid#using-an-amazon-elastic-container-service-ecs-kubernetes-or-docker-agent "Direct link to Using an Amazon Elastic Container Service (ECS), Kubernetes, or Docker agent") ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- If you are using an Amazon Elastic Container Service (ECS), Kubernetes, or Docker agent for your Hybrid deployments in Dagster+, your Dagster code must be packaged in a Docker image and pushed to a registry your agent can access. In this scenario, to use a dbt project with Dagster, you'll need to include it with your code in the Docker image. Before including the dbt project in the Docker image, you'll need to make sure it contains an up-to-date [manifest file](https://docs.getdbt.com/reference/artifacts/manifest-json) and [project dependencies](https://docs.getdbt.com/docs/collaborate/govern/project-dependencies) . This can be done by running the [`dagster-dbt project prepare-and-package`](https://docs.dagster.io/api/libraries/dagster-dbt#prepare-and-package) command. In the workflow building and pushing your Docker image, make sure this command runs before building your Docker image to ensure all required dbt files are included. Note that this command runs `dbt deps` and `dbt parse` to create your manifest file. ### Using CI/CD files[​](https://docs.dagster.io/integrations/libraries/dbt/using-dbt-with-dagster-plus/hybrid#using-cicd-files "Direct link to Using CI/CD files") If you are using [CI/CD files](https://docs.dagster.io/deployment/dagster-plus/ci-cd/ci-cd-file-reference) in a Git repository to build and push your Docker image, you'll need to add a few steps to allow the dbt project to deploy successfully. Our example updates the CI/CD files of a project from a GitHub repository, but this could be achieved in other platform like GitLab. 1. In your Dagster project, locate the `.github/workflows` directory. 2. Open the `deploy.yml` file. 3. Locate the step in which which you build and push your docker image. 4. Before this step, add the following: - name: Prepare DBT project for deployment run: | python -m pip install pip --upgrade pip install . --upgrade --upgrade-strategy eager ## Install the Python dependencies from the setup.py file, ex: dbt-core and dbt-duckdb dagster-dbt project prepare-and-package --file /project.py ## Replace with the project.py location in the Dagster project folder shell: bash When you add this step, you'll need to: * **Add any [adapters](https://docs.getdbt.com/docs/connect-adapters) and libraries used by dbt to your `setup.py` file**. * **Add the location of your file defining your DbtProject** to the `dagster-dbt project prepare-and-package` command. If you are using [Components](https://docs.dagster.io/guides/build/components) , you can use the `--components` flag with a path to your project root. 5. Save the changes. 6. Open the `branch_deployments.yml` file and repeat steps 3 - 5. 7. Commit the changes to the repository. Once the new step is pushed to the remote, your workflow will be updated to prepare your dbt project before building and pushing your docker image. Using a local agent[​](https://docs.dagster.io/integrations/libraries/dbt/using-dbt-with-dagster-plus/hybrid#using-a-local-agent "Direct link to Using a local agent") ----------------------------------------------------------------------------------------------------------------------------------------------------------------------- When using a local agent for your Hybrid deployments in Dagster+, your Dagster code and dbt project must be in a Python environment that can be accessed on the same machine as your agent. When updating the dbt project, it is important to refresh the [manifest file](https://docs.getdbt.com/reference/artifacts/manifest-json) and [project dependencies](https://docs.getdbt.com/docs/collaborate/govern/project-dependencies) to ensure that they are up-to-date when used with your Dagster code. This can be done by running the [`dagster-dbt project prepare-and-package`](https://docs.dagster.io/api/libraries/dagster-dbt#prepare-and-package) command. Note that this command runs `dbt deps` and `dbt parse` to refresh your manifest file. * [Prerequisites](https://docs.dagster.io/integrations/libraries/dbt/using-dbt-with-dagster-plus/hybrid#prerequisites) * [Using an Amazon Elastic Container Service (ECS), Kubernetes, or Docker agent](https://docs.dagster.io/integrations/libraries/dbt/using-dbt-with-dagster-plus/hybrid#using-an-amazon-elastic-container-service-ecs-kubernetes-or-docker-agent) * [Using CI/CD files](https://docs.dagster.io/integrations/libraries/dbt/using-dbt-with-dagster-plus/hybrid#using-cicd-files) * [Using a local agent](https://docs.dagster.io/integrations/libraries/dbt/using-dbt-with-dagster-plus/hybrid#using-a-local-agent) --- # Dagster & DuckDB | Dagster Docs [Skip to main content](https://docs.dagster.io/integrations/libraries/duckdb#__docusaurus_skipToContent_fallback) On this page This library provides an integration with the DuckDB database, and allows for an out-of-the-box I/O Manager so that you can make DuckDB your storage of choice. Installation[​](https://docs.dagster.io/integrations/libraries/duckdb#installation "Direct link to Installation") ------------------------------------------------------------------------------------------------------------------ * uv * pip uv add dagster-duckdb pip install dagster-duckdb Example[​](https://docs.dagster.io/integrations/libraries/duckdb#example "Direct link to Example") --------------------------------------------------------------------------------------------------- import pandas as pdfrom dagster_duckdb_pandas import DuckDBPandasIOManagerimport dagster as dg@dg.asset( key_prefix=["my_schema"] # will be used as the schema in duckdb)def my_table() -> pd.DataFrame: # the name of the asset will be the table name return pd.DataFrame()defs = dg.Definitions( assets=[my_table], resources={"io_manager": DuckDBPandasIOManager(database="my_db.duckdb")},) About DuckDB[​](https://docs.dagster.io/integrations/libraries/duckdb#about-duckdb "Direct link to About DuckDB") ------------------------------------------------------------------------------------------------------------------ **DuckDB** is a column-oriented in-process OLAP database. A typical OLTP relational database like SQLite is row-oriented. In row-oriented database, data is organised physically as consecutive tuples. * [Installation](https://docs.dagster.io/integrations/libraries/duckdb#installation) * [Example](https://docs.dagster.io/integrations/libraries/duckdb#example) * [About DuckDB](https://docs.dagster.io/integrations/libraries/duckdb#about-duckdb) --- # Using dbt with Dagster+ | Dagster Docs [Skip to main content](https://docs.dagster.io/integrations/libraries/dbt/using-dbt-with-dagster-plus#__docusaurus_skipToContent_fallback) On this page Using a dbt project in Dagster+ allows you to automatically load your dbt models as Dagster assets. This can be be done with both deployment options in Dagster+: Serverless and Hybrid. [Learn more about deployment options in Dagster+](https://docs.dagster.io/deployment/dagster-plus) . Serverless deployments[​](https://docs.dagster.io/integrations/libraries/dbt/using-dbt-with-dagster-plus#serverless-deployments "Direct link to Serverless deployments") ------------------------------------------------------------------------------------------------------------------------------------------------------------------------- If you have a Serverless deployment, you can directly import an existing dbt project in Dagster+ when adding a new code location. This can be done with: * An existing dbt project that is not already using Dagster, or * A Dagster project in which your dbt project is included For more information, see "[Importing a dbt project to Dagster+ Serverless](https://docs.dagster.io/integrations/libraries/dbt/using-dbt-with-dagster-plus/serverless) ". Hybrid deployments[​](https://docs.dagster.io/integrations/libraries/dbt/using-dbt-with-dagster-plus#hybrid-deployments "Direct link to Hybrid deployments") ------------------------------------------------------------------------------------------------------------------------------------------------------------- If you have a Hybrid deployment, you must make the dbt project accessible to the Dagster code executed by your agent. * When using Amazon Elastic Container Service (ECS), Kubernetes, or Docker agent, you must include the dbt project in the Docker Image containing your Dagster code. * When using a local agent, you must make your dbt project accessible to your Dagster code on the same machine as your agent. For more information, see "[Using dbt with Hybrid deployments in Dagster+](https://docs.dagster.io/integrations/libraries/dbt/using-dbt-with-dagster-plus/hybrid) ". * [Serverless deployments](https://docs.dagster.io/integrations/libraries/dbt/using-dbt-with-dagster-plus#serverless-deployments) * [Hybrid deployments](https://docs.dagster.io/integrations/libraries/dbt/using-dbt-with-dagster-plus#hybrid-deployments) --- # Using Dagster with Fivetran | Dagster Docs [Skip to main content](https://docs.dagster.io/integrations/libraries/fivetran#__docusaurus_skipToContent_fallback) On this page note If you are just getting started with the Fivetran integration, we recommend using the new [Fivetran component](https://docs.dagster.io/guides/build/components/integrations/fivetran-component-tutorial) . This guide provides instructions for using Dagster with Fivetran using the `dagster-fivetran` library. Your Fivetran connector tables can be represented as assets in the Dagster asset graph, allowing you to track lineage and dependencies between Fivetran assets and data assets you are already modeling in Dagster. You can also use Dagster to orchestrate Fivetran connectors, allowing you to trigger syncs for these on a cadence or based on upstream data changes. note Your Fivetran connectors must have been synced at least once to be represented in Dagster. What you'll learn[​](https://docs.dagster.io/integrations/libraries/fivetran#what-youll-learn "Direct link to What you'll learn") ---------------------------------------------------------------------------------------------------------------------------------- * How to represent Fivetran assets in the Dagster asset graph, including lineage to other Dagster assets. * How to customize asset definition metadata for these Fivetran assets. * How to materialize Fivetran connector tables from Dagster. * How to customize how Fivetran connector tables are materialized. Prerequisites * The `dagster` and `dagster-fivetran` libraries installed in your environment * Familiarity with asset definitions and the Dagster asset graph * Familiarity with Dagster resources * Familiarity with Fivetran concepts, like connectors and connector tables * A Fivetran workspace * A Fivetran API key and API secret. For more information, see [Getting Started](https://fivetran.com/docs/rest-api/getting-started) in the Fivetran REST API documentation. Set up your environment[​](https://docs.dagster.io/integrations/libraries/fivetran#set-up-your-environment "Direct link to Set up your environment") ----------------------------------------------------------------------------------------------------------------------------------------------------- To get started, you'll need to install the `dagster` and `dagster-fivetran` Python packages: * uv * pip uv add dagster-fivetran pip install dagster-fivetran Represent Fivetran assets in the asset graph[​](https://docs.dagster.io/integrations/libraries/fivetran#represent-fivetran-assets-in-the-asset-graph "Direct link to Represent Fivetran assets in the asset graph") -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- To load Fivetran assets into the Dagster asset graph, you must first construct a [`FivetranWorkspace`](https://docs.dagster.io/api/libraries/dagster-fivetran#dagster_fivetran.FivetranWorkspace) resource, which allows Dagster to communicate with your Fivetran workspace. You'll need to supply your account ID, API key and API secret. See [Getting Started](https://fivetran.com/docs/rest-api/getting-started) in the Fivetran REST API documentation for more information on how to create your API key and API secret. Dagster can automatically load all connector tables from your Fivetran workspace as asset specs. Call the [`load_fivetran_asset_specs`](https://docs.dagster.io/api/libraries/dagster-fivetran#dagster_fivetran.load_fivetran_asset_specs) function, which returns list of [`AssetSpec`](https://docs.dagster.io/api/dagster/assets#dagster.AssetSpec) s representing your Fivetran assets. You can then include these asset specs in your [`Definitions`](https://docs.dagster.io/api/dagster/definitions#dagster.Definitions) object: from dagster_fivetran import FivetranWorkspace, load_fivetran_asset_specsimport dagster as dgfivetran_workspace = FivetranWorkspace( account_id=dg.EnvVar("FIVETRAN_ACCOUNT_ID"), api_key=dg.EnvVar("FIVETRAN_API_KEY"), api_secret=dg.EnvVar("FIVETRAN_API_SECRET"),)fivetran_specs = load_fivetran_asset_specs(fivetran_workspace)defs = dg.Definitions(assets=fivetran_specs, resources={"fivetran": fivetran_workspace}) ### Sync and materialize Fivetran assets[​](https://docs.dagster.io/integrations/libraries/fivetran#sync-and-materialize-fivetran-assets "Direct link to Sync and materialize Fivetran assets") You can use Dagster to sync Fivetran connectors and materialize Fivetran connector tables. You can use the [`build_fivetran_assets_definitions`](https://docs.dagster.io/api/libraries/dagster-fivetran#dagster_fivetran.build_fivetran_assets_definitions) factory to create all assets definitions for your Fivetran workspace. note When syncing a Fivetran connector via Dagster, all Fivetran assets for this connector are materialized in Dagster. from dagster_fivetran import FivetranWorkspace, build_fivetran_assets_definitionsimport dagster as dgfivetran_workspace = FivetranWorkspace( account_id=dg.EnvVar("FIVETRAN_ACCOUNT_ID"), api_key=dg.EnvVar("FIVETRAN_API_KEY"), api_secret=dg.EnvVar("FIVETRAN_API_SECRET"),)all_fivetran_assets = build_fivetran_assets_definitions(workspace=fivetran_workspace)defs = dg.Definitions( assets=all_fivetran_assets, resources={"fivetran": fivetran_workspace},) ### Customize the materialization of Fivetran assets[​](https://docs.dagster.io/integrations/libraries/fivetran#customize-the-materialization-of-fivetran-assets "Direct link to Customize the materialization of Fivetran assets") If you want to customize the sync of your connectors, you can use the [`fivetran_assets`](https://docs.dagster.io/api/libraries/dagster-fivetran#dagster_fivetran.fivetran_assets) decorator to do so. This allows you to execute custom code before and after the call to the Fivetran sync. from dagster_fivetran import FivetranWorkspace, fivetran_assetsimport dagster as dgfivetran_workspace = FivetranWorkspace( account_id=dg.EnvVar("FIVETRAN_ACCOUNT_ID"), api_key=dg.EnvVar("FIVETRAN_API_KEY"), api_secret=dg.EnvVar("FIVETRAN_API_SECRET"),)@fivetran_assets( connector_id="fivetran_connector_id", # Replace with your connector ID name="fivetran_connector_name", # Replace with your connection name group_name="fivetran_connector_name", workspace=fivetran_workspace,)def fivetran_connector_assets( context: dg.AssetExecutionContext, fivetran: FivetranWorkspace): # Do something before the materialization... yield from fivetran.sync_and_poll(context=context) # Do something after the materialization...defs = dg.Definitions( assets=[fivetran_connector_assets], resources={"fivetran": fivetran_workspace},) ### Customize asset definition metadata for Fivetran assets[​](https://docs.dagster.io/integrations/libraries/fivetran#customize-asset-definition-metadata-for-fivetran-assets "Direct link to Customize asset definition metadata for Fivetran assets") By default, Dagster will generate asset specs for each Fivetran asset and populate default metadata. You can further customize asset properties by passing an instance of the custom [`DagsterFivetranTranslator`](https://docs.dagster.io/api/libraries/dagster-fivetran#dagster_fivetran.DagsterFivetranTranslator) to the [`load_fivetran_asset_specs`](https://docs.dagster.io/api/libraries/dagster-fivetran#dagster_fivetran.load_fivetran_asset_specs) function. from dagster_fivetran import ( DagsterFivetranTranslator, FivetranConnectorTableProps, FivetranWorkspace, load_fivetran_asset_specs,)import dagster as dgfivetran_workspace = FivetranWorkspace( account_id=dg.EnvVar("FIVETRAN_ACCOUNT_ID"), api_key=dg.EnvVar("FIVETRAN_API_KEY"), api_secret=dg.EnvVar("FIVETRAN_API_SECRET"),)# A translator class lets us customize properties of the built# Fivetran assets, such as the owners or asset keyclass MyCustomFivetranTranslator(DagsterFivetranTranslator): def get_asset_spec(self, props: FivetranConnectorTableProps) -> dg.AssetSpec: # We create the default asset spec using super() default_spec = super().get_asset_spec(props) # We customize the metadata and asset key prefix for all assets return default_spec.replace_attributes( key=default_spec.key.with_prefix("prefix"), ).merge_attributes(metadata={"custom": "metadata"})fivetran_specs = load_fivetran_asset_specs( fivetran_workspace, dagster_fivetran_translator=MyCustomFivetranTranslator())defs = dg.Definitions(assets=fivetran_specs, resources={"fivetran": fivetran_workspace}) Note that `super()` is called in each of the overridden methods to generate the default asset spec. It is best practice to generate the default asset spec before customizing it. You can pass an instance of the custom [`DagsterFivetranTranslator`](https://docs.dagster.io/api/libraries/dagster-fivetran#dagster_fivetran.DagsterFivetranTranslator) to the [`fivetran_assets`](https://docs.dagster.io/api/libraries/dagster-fivetran#dagster_fivetran.fivetran_assets) decorator or the [`build_fivetran_assets_definitions`](https://docs.dagster.io/api/libraries/dagster-fivetran#dagster_fivetran.build_fivetran_assets_definitions) factory. ### Fetching column-level metadata for Fivetran assets[​](https://docs.dagster.io/integrations/libraries/fivetran#fetching-column-level-metadata-for-fivetran-assets "Direct link to Fetching column-level metadata for Fivetran assets") Dagster allows you to emit column-level metadata, like [column schema](https://docs.dagster.io/guides/build/assets/metadata-and-tags/#standard-metadata-types) and [column lineage](https://docs.dagster.io/guides/build/assets/metadata-and-tags/#column-lineage) , as [materialization metadata](https://docs.dagster.io/guides/build/assets/metadata-and-tags/#runtime-metadata) . With this metadata, you can view documentation in Dagster for all columns in your Fivetran connector tables. To enable this feature, call [`fetch_column_metadata()`](https://docs.dagster.io/api/libraries/dagster-fivetran#dagster_fivetran.fivetran_event_iterator.FivetranEventIterator.fetch_column_metadata) on the [`fivetran_event_iterator.FivetranEventIterator`](https://docs.dagster.io/api/libraries/dagster-fivetran#dagster_fivetran.fivetran_event_iterator.FivetranEventIterator) returned by the `sync_and_poll()` call on the [`FivetranWorkspace`](https://docs.dagster.io/api/libraries/dagster-fivetran#dagster_fivetran.FivetranWorkspace) resource. from dagster_fivetran import FivetranWorkspace, fivetran_assetsimport dagster as dgfivetran_workspace = FivetranWorkspace( account_id=dg.EnvVar("FIVETRAN_ACCOUNT_ID"), api_key=dg.EnvVar("FIVETRAN_API_KEY"), api_secret=dg.EnvVar("FIVETRAN_API_SECRET"),)@fivetran_assets( # Replace with your connector ID connector_id="fivetran_connector_id", workspace=fivetran_workspace,)def fivetran_connector_assets( context: dg.AssetExecutionContext, fivetran: FivetranWorkspace): yield from fivetran.sync_and_poll(context=context).fetch_column_metadata()defs = dg.Definitions( assets=[fivetran_connector_assets], resources={"fivetran": fivetran_workspace},) ### Load Fivetran assets for selected connectors[​](https://docs.dagster.io/integrations/libraries/fivetran#load-fivetran-assets-for-selected-connectors "Direct link to Load Fivetran assets for selected connectors") To select a subset of Fivetran connectors for which your Fivetran assets will be loaded, you can use the [`ConnectorSelectorFn`](https://docs.dagster.io/api/libraries/dagster-fivetran#dagster_fivetran.ConnectorSelectorFn) callback and define your selection conditions. from dagster_fivetran import FivetranWorkspace, build_fivetran_assets_definitionsimport dagster as dgfivetran_workspace = FivetranWorkspace( account_id=dg.EnvVar("FIVETRAN_ACCOUNT_ID"), api_key=dg.EnvVar("FIVETRAN_API_KEY"), api_secret=dg.EnvVar("FIVETRAN_API_SECRET"),)all_fivetran_assets = build_fivetran_assets_definitions( workspace=fivetran_workspace, connector_selector_fn=( lambda connector: connector.id in {"some_connector_id", "another_connector_id"} ),)defs = dg.Definitions( assets=all_fivetran_assets, resources={"fivetran": fivetran_workspace},) ### Load Fivetran assets using a snapshot[​](https://docs.dagster.io/integrations/libraries/fivetran#load-fivetran-assets-using-a-snapshot "Direct link to Load Fivetran assets using a snapshot") Fivetran assets can be loaded using the snapshot of a Fivetran workspace, which allows organizations with large amounts of Fivetran data to speed up their deployment process. from dagster_fivetran import FivetranWorkspace, load_fivetran_asset_specsimport dagster as dgfivetran_workspace = FivetranWorkspace( account_id=dg.EnvVar("FIVETRAN_ACCOUNT_ID"), api_key=dg.EnvVar("FIVETRAN_API_KEY"), api_secret=dg.EnvVar("FIVETRAN_API_SECRET"), snapshot_path=dg.EnvVar("FIVETRAN_SNAPSHOT_PATH"),)fivetran_specs = load_fivetran_asset_specs(workspace=fivetran_workspace)defs = dg.Definitions(assets=fivetran_specs) To capture the snapshot, the `dagster-fivetran snapshot` CLI can be used. dagster-fivetran snapshot --python-module my_dagster_package --output-path snapshot.snap ### Load Fivetran assets from multiple workspaces[​](https://docs.dagster.io/integrations/libraries/fivetran#load-fivetran-assets-from-multiple-workspaces "Direct link to Load Fivetran assets from multiple workspaces") Definitions from multiple Fivetran workspaces can be combined by instantiating multiple [`FivetranWorkspace`](https://docs.dagster.io/api/libraries/dagster-fivetran#dagster_fivetran.FivetranWorkspace) resources and merging their specs. This lets you view all your Fivetran assets in a single asset graph: from dagster_fivetran import FivetranWorkspace, load_fivetran_asset_specsimport dagster as dgsales_fivetran_workspace = FivetranWorkspace( account_id=dg.EnvVar("FIVETRAN_SALES_ACCOUNT_ID"), api_key=dg.EnvVar("FIVETRAN_SALES_API_KEY"), api_secret=dg.EnvVar("FIVETRAN_SALES_API_SECRET"),)marketing_fivetran_workspace = FivetranWorkspace( account_id=dg.EnvVar("FIVETRAN_MARKETING_ACCOUNT_ID"), api_key=dg.EnvVar("FIVETRAN_MARKETING_API_KEY"), api_secret=dg.EnvVar("FIVETRAN_MARKETING_API_SECRET"),)sales_fivetran_specs = load_fivetran_asset_specs(sales_fivetran_workspace)marketing_fivetran_specs = load_fivetran_asset_specs(marketing_fivetran_workspace)# Merge the specs into a single set of definitionsdefs = dg.Definitions( assets=[*sales_fivetran_specs, *marketing_fivetran_specs], resources={ "marketing_fivetran": marketing_fivetran_workspace, "sales_fivetran": sales_fivetran_workspace, },) ### Define upstream dependencies[​](https://docs.dagster.io/integrations/libraries/fivetran#define-upstream-dependencies "Direct link to Define upstream dependencies") By default, Dagster does not set upstream dependencies when generating asset specs for your Fivetran assets. You can set upstream dependencies on your Fivetran assets by passing an instance of the custom [`DagsterFivetranTranslator`](https://docs.dagster.io/api/libraries/dagster-fivetran#dagster_fivetran.DagsterFivetranTranslator) to the [`load_fivetran_asset_specs`](https://docs.dagster.io/api/libraries/dagster-fivetran#dagster_fivetran.load_fivetran_asset_specs) function. class MyCustomFivetranTranslator(DagsterFivetranTranslator): def get_asset_spec(self, props: FivetranConnectorTableProps) -> dg.AssetSpec: # We create the default asset spec using super() default_spec = super().get_asset_spec(props) # We set an upstream dependency for our assets return default_spec.replace_attributes(deps=["my_upstream_asset_key"])fivetran_specs = load_fivetran_asset_specs( fivetran_workspace, dagster_fivetran_translator=MyCustomFivetranTranslator()) Note that `super()` is called in each of the overridden methods to generate the default asset spec. It is best practice to generate the default asset spec before customizing it. You can pass an instance of the custom [`DagsterFivetranTranslator`](https://docs.dagster.io/api/libraries/dagster-fivetran#dagster_fivetran.DagsterFivetranTranslator) to the [`fivetran_assets`](https://docs.dagster.io/api/libraries/dagster-fivetran#dagster_fivetran.fivetran_assets) decorator or the [`build_fivetran_assets_definitions`](https://docs.dagster.io/api/libraries/dagster-fivetran#dagster_fivetran.build_fivetran_assets_definitions) factory. ### Define downstream dependencies[​](https://docs.dagster.io/integrations/libraries/fivetran#define-downstream-dependencies "Direct link to Define downstream dependencies") Dagster allows you to define assets that are downstream of specific Fivetran tables using their asset keys. The asset key for a Fivetran table can be retrieved using the asset definitions created using the [`fivetran_assets`](https://docs.dagster.io/api/libraries/dagster-fivetran#dagster_fivetran.fivetran_assets) decorator. The below example defines `my_downstream_asset` as a downstream dependency of `my_fivetran_table`: @fivetran_assets( # Replace with your connector ID connector_id="fivetran_connector_id", workspace=fivetran_workspace,)def fivetran_connector_assets( context: dg.AssetExecutionContext, fivetran: FivetranWorkspace): ...my_fivetran_table_asset_key = next( iter( [ spec.key for spec in fivetran_connector_assets.specs if spec.metadata.get("dagster/table_name") == "my_database.my_schema.my_fivetran_table" ] ))@dg.asset(deps=[my_fivetran_table_asset_key])def my_downstream_asset(): ... In the downstream asset, you may want direct access to the contents of the Fivetran table. To do so, you can customize the code within your `@asset`\-decorated function to load upstream data. ### About Fivetran[​](https://docs.dagster.io/integrations/libraries/fivetran#about-fivetran "Direct link to About Fivetran") **Fivetran** ingests data from SaaS applications, databases, and servers. The data is stored and typically used for analytics. * [What you'll learn](https://docs.dagster.io/integrations/libraries/fivetran#what-youll-learn) * [Set up your environment](https://docs.dagster.io/integrations/libraries/fivetran#set-up-your-environment) * [Represent Fivetran assets in the asset graph](https://docs.dagster.io/integrations/libraries/fivetran#represent-fivetran-assets-in-the-asset-graph) * [Sync and materialize Fivetran assets](https://docs.dagster.io/integrations/libraries/fivetran#sync-and-materialize-fivetran-assets) * [Customize the materialization of Fivetran assets](https://docs.dagster.io/integrations/libraries/fivetran#customize-the-materialization-of-fivetran-assets) * [Customize asset definition metadata for Fivetran assets](https://docs.dagster.io/integrations/libraries/fivetran#customize-asset-definition-metadata-for-fivetran-assets) * [Fetching column-level metadata for Fivetran assets](https://docs.dagster.io/integrations/libraries/fivetran#fetching-column-level-metadata-for-fivetran-assets) * [Load Fivetran assets for selected connectors](https://docs.dagster.io/integrations/libraries/fivetran#load-fivetran-assets-for-selected-connectors) * [Load Fivetran assets using a snapshot](https://docs.dagster.io/integrations/libraries/fivetran#load-fivetran-assets-using-a-snapshot) * [Load Fivetran assets from multiple workspaces](https://docs.dagster.io/integrations/libraries/fivetran#load-fivetran-assets-from-multiple-workspaces) * [Define upstream dependencies](https://docs.dagster.io/integrations/libraries/fivetran#define-upstream-dependencies) * [Define downstream dependencies](https://docs.dagster.io/integrations/libraries/fivetran#define-downstream-dependencies) * [About Fivetran](https://docs.dagster.io/integrations/libraries/fivetran#about-fivetran) --- # dagster-duckdb integration reference | Dagster Docs [Skip to main content](https://docs.dagster.io/integrations/libraries/duckdb/reference#__docusaurus_skipToContent_fallback) On this page This reference page provides information for working with [`dagster-duckdb`](https://docs.dagster.io/api/libraries/dagster-duckdb) features that are not covered as part of the [Using Dagster with DuckDB tutorial](https://docs.dagster.io/integrations/libraries/duckdb/using-duckdb-with-dagster) . DuckDB resource: * [Executing custom SQL queries](https://docs.dagster.io/integrations/libraries/duckdb/reference#executing-custom-sql-queries) DuckDB I/O manager: * [Selecting specific columns in a downstream asset](https://docs.dagster.io/integrations/libraries/duckdb/reference#selecting-specific-columns-in-a-downstream-asset) * [Storing partitioned assets](https://docs.dagster.io/integrations/libraries/duckdb/reference#storing-partitioned-assets) * [Storing tables in multiple schemas](https://docs.dagster.io/integrations/libraries/duckdb/reference#storing-tables-in-multiple-schemas) * [Using the DuckDB I/O manager with other I/O managers](https://docs.dagster.io/integrations/libraries/duckdb/reference#using-the-duckdb-io-manager-with-other-io-managers) * [Storing and loading PySpark or Polars DataFrames in DuckDB](https://docs.dagster.io/integrations/libraries/duckdb/reference#storing-and-loading-pyspark-or-polars-dataframes-in-duckdb) * [Storing multiple DataFrame types in DuckDB](https://docs.dagster.io/integrations/libraries/duckdb/reference#storing-multiple-dataframe-types-in-duckdb) DuckDB resource[​](https://docs.dagster.io/integrations/libraries/duckdb/reference#duckdb-resource "Direct link to DuckDB resource") ------------------------------------------------------------------------------------------------------------------------------------- The DuckDB resource provides access to a [`duckdb.DuckDBPyConnection`](https://duckdb.org/docs/api/python/reference/#duckdb.DuckDBPyConnection) object. This allows you full control over how your data is stored and retrieved in your database. For further information on the DuckDB resource, see the [DuckDB resource API docs](https://docs.dagster.io/api/libraries/dagster-duckdb#dagster_duckdb.DuckDBResource) . ### Executing custom SQL queries[​](https://docs.dagster.io/integrations/libraries/duckdb/reference#executing-custom-sql-queries "Direct link to Executing custom SQL queries") from dagster_duckdb import DuckDBResourcefrom dagster import asset# this example executes a query against the iris_dataset table created in Step 2 of the# Using Dagster with DuckDB tutorial@asset(deps=[iris_dataset])def small_petals(duckdb: DuckDBResource) -> None: with duckdb.get_connection() as conn: # conn is a DuckDBPyConnection conn.execute( "CREATE TABLE iris.small_petals AS SELECT * FROM iris.iris_dataset WHERE" " 'petal_length_cm' < 1 AND 'petal_width_cm' < 1" ) In this example, we attach the DuckDB resource to the `small_petals` asset. In the body of the asset function, we use the `get_connection` context manager on the resource to get a [`duckdb.DuckDBPyConnection`](https://duckdb.org/docs/api/python/reference/#duckdb.DuckDBPyConnection) . We can use this connection to execute a custom SQL query against the `iris_dataset` table created in [Step 2: Create tables in DuckDB](https://docs.dagster.io/integrations/libraries/duckdb/using-duckdb-with-dagster#option-1-step-2) of the [Using Dagster with DuckDB tutorial](https://docs.dagster.io/integrations/libraries/duckdb/using-duckdb-with-dagster) . When the `duckdb.get_connection` context is exited, the DuckDB connection will be closed. DuckDB I/O manager[​](https://docs.dagster.io/integrations/libraries/duckdb/reference#duckdb-io-manager "Direct link to DuckDB I/O manager") --------------------------------------------------------------------------------------------------------------------------------------------- The DuckDB I/O manager provides several ways to customize how your data is stored and loaded in DuckDB. However, if you find that these options do not provide enough customization for your use case, we recommend using the DuckDB resource to save and load your data. By using the resource, you will have more fine-grained control over how your data is handled, since you have full control over the SQL queries that are executed. ### Selecting specific columns in a downstream asset[​](https://docs.dagster.io/integrations/libraries/duckdb/reference#selecting-specific-columns-in-a-downstream-asset "Direct link to Selecting specific columns in a downstream asset") Sometimes you may not want to fetch an entire table as the input to a downstream asset. With the DuckDB I/O manager, you can select specific columns to load by supplying metadata on the downstream asset. import pandas as pdfrom dagster import AssetIn, asset# this example uses the iris_dataset asset from Step 2 of the Using Dagster with DuckDB tutorial@asset( ins={ "iris_sepal": AssetIn( key="iris_dataset", metadata={"columns": ["sepal_length_cm", "sepal_width_cm"]}, ) })def sepal_data(iris_sepal: pd.DataFrame) -> pd.DataFrame: iris_sepal["sepal_area_cm2"] = ( iris_sepal["sepal_length_cm"] * iris_sepal["sepal_width_cm"] ) return iris_sepal In this example, we only use the columns containing sepal data from the `IRIS_DATASET` table created in [Step 2: Create tables in DuckDB](https://docs.dagster.io/integrations/libraries/duckdb/using-duckdb-with-dagster#option-2-step-2) of the [Using Dagster with DuckDB tutorial](https://docs.dagster.io/integrations/libraries/duckdb/using-duckdb-with-dagster) . To select specific columns, we can add metadata to the input asset. We do this in the `metadata` parameter of the `AssetIn` that loads the `iris_dataset` asset in the `ins` parameter. We supply the key `columns` with a list of names of the columns we want to fetch. When Dagster materializes `sepal_data` and loads the `iris_dataset` asset using the DuckDB I/O manager, it will only fetch the `sepal_length_cm` and `sepal_width_cm` columns of the `IRIS.IRIS_DATASET` table and pass them to `sepal_data` as a Pandas DataFrame. ### Storing partitioned assets[​](https://docs.dagster.io/integrations/libraries/duckdb/reference#storing-partitioned-assets "Direct link to Storing partitioned assets") The DuckDB I/O manager supports storing and loading partitioned data. To correctly store and load data from the DuckDB table, the DuckDB I/O manager needs to know which column contains the data defining the partition bounds. The DuckDB I/O manager uses this information to construct the correct queries to select or replace the data. In the following sections, we describe how the I/O manager constructs these queries for different types of partitions. * Storing static partitioned assets * Storing time-partitioned assets * Storing multi-partitioned assets To store static partitioned assets in DuckDB, specify `partition_expr` metadata on the asset to tell the DuckDB I/O manager which column contains the partition data: import pandas as pdfrom dagster import AssetExecutionContext, StaticPartitionsDefinition, asset@asset( partitions_def=StaticPartitionsDefinition( ["Iris-setosa", "Iris-virginica", "Iris-versicolor"] ), metadata={"partition_expr": "SPECIES"},)def iris_dataset_partitioned(context: AssetExecutionContext) -> pd.DataFrame: species = context.partition_key full_df = pd.read_csv( "https://docs.dagster.io/assets/iris.csv", names=[ "sepal_length_cm", "sepal_width_cm", "petal_length_cm", "petal_width_cm", "species", ], ) return full_df[full_df["Species"] == species]@assetdef iris_cleaned(iris_dataset_partitioned: pd.DataFrame): return iris_dataset_partitioned.dropna().drop_duplicates() Dagster uses the `partition_expr` metadata to craft the `SELECT` statement when loading the partition in the downstream asset. When loading a static partition (or multiple static partitions), the following statement is used: SELECT * WHERE [partition_expr] in ([selected partitions]) When the `partition_expr` value is injected into this statement, the resulting SQL query must follow DuckDB's SQL syntax. Refer to the [DuckDB documentation](https://duckdb.org/docs/sql/query_syntax/select) for more information. A partition must be selected when materializing the above assets. In this example, the query used when materializing the `Iris-setosa` partition of the above assets would be: SELECT * WHERE SPECIES in ('Iris-setosa') Like static partitioned assets, you can specify `partition_expr` metadata on the asset to tell the DuckDB I/O manager which column contains the partition data: import pandas as pdfrom dagster import AssetExecutionContext, DailyPartitionsDefinition, asset@asset( partitions_def=DailyPartitionsDefinition(start_date="2023-01-01"), metadata={"partition_expr": "TO_TIMESTAMP(TIME)"},)def iris_data_per_day(context: AssetExecutionContext) -> pd.DataFrame: partition = context.partition_key # get_iris_data_for_date fetches all of the iris data for a given date, # the returned dataframe contains a column named 'time' with that stores # the time of the row as an integer of seconds since epoch return get_iris_data_for_date(partition)@assetdef iris_cleaned(iris_data_per_day: pd.DataFrame): return iris_data_per_day.dropna().drop_duplicates() Dagster uses the `partition_expr` metadata to craft the `SELECT` statement when loading the correct partition in the downstream asset. When loading a dynamic partition, the following statement is used: SELECT * WHERE [partition_expr] >= [partition_start] AND [partition_expr] < [partition_end] When the `partition_expr` value is injected into this statement, the resulting SQL query must follow DuckDB's SQL syntax. Refer to the [DuckDB documentation](https://duckdb.org/docs/sql/query_syntax/select) for more information. A partition must be selected when materializing assets. The `[partition_start]` and `[partition_end]` bounds are of the form `YYYY-MM-DD HH:MM:SS`. In this example, the query when materializing the `2023-01-02` partition of the above assets would be: SELECT * WHERE TO_TIMESTAMP(TIME) >= '2023-01-02 00:00:00' AND TO_TIMESTAMP(TIME) < '2023-01-03 00:00:00' In this example, the data in the `TIME` column are integers, so the `partition_expr` metadata includes a SQL statement to convert integers to timestamps. A full list of DuckDB functions can be found [here](https://duckdb.org/docs/sql/functions/overview) . The DuckDB I/O manager can also store data partitioned on multiple dimensions. To do this, specify the column for each partition as a dictionary of `partition_expr` metadata: import pandas as pdimport dagster as dg@dg.asset( partitions_def=dg.MultiPartitionsDefinition( { "date": dg.DailyPartitionsDefinition(start_date="2023-01-01"), "species": dg.StaticPartitionsDefinition( ["Iris-setosa", "Iris-virginica", "Iris-versicolor"] ), } ), metadata={"partition_expr": {"date": "TO_TIMESTAMP(TIME)", "species": "SPECIES"}},)def iris_dataset_partitioned(context: dg.AssetExecutionContext) -> pd.DataFrame: partition = context.partition_key.keys_by_dimension species = partition["species"] date = partition["date"] # get_iris_data_for_date fetches all of the iris data for a given date, # the returned dataframe contains a column named 'time' with that stores # the time of the row as an integer of seconds since epoch full_df = get_iris_data_for_date(date) return full_df[full_df["species"] == species]@dg.assetdef iris_cleaned(iris_dataset_partitioned: pd.DataFrame): return iris_dataset_partitioned.dropna().drop_duplicates() Dagster uses the `partition_expr` metadata to craft the `SELECT` statement when loading the correct partition in a downstream asset. For multi-partitions, Dagster concatenates the `WHERE` statements described in the above sections to craft the correct `SELECT` statement. A partition must be selected when materializing assets. For example, when materializing the `2023-01-02|Iris-setosa` partition of the above assets, the following query will be used: SELECT * WHERE SPECIES in ('Iris-setosa') AND TO_TIMESTAMP(TIME) >= '2023-01-02 00:00:00' AND TO_TIMESTAMP(TIME) < '2023-01-03 00:00:00' In this example, the data in the `TIME` column are integers, so the `partition_expr` metadata includes a SQL statement to convert integers to timestamps. A full list of DuckDB functions can be found [here](https://duckdb.org/docs/sql/functions/overview) . ### Storing tables in multiple schemas[​](https://docs.dagster.io/integrations/libraries/duckdb/reference#storing-tables-in-multiple-schemas "Direct link to Storing tables in multiple schemas") You may want to have different assets stored in different DuckDB schemas. The DuckDB I/O manager allows you to specify the schema in several ways. You can specify the default schema where data will be stored as configuration to the I/O manager, as we did in [Step 1: Configure the DuckDB I/O manager](https://docs.dagster.io/integrations/libraries/duckdb/using-duckdb-with-dagster#step-1-configure-the-duckdb-io-manager) of the [Using Dagster with DuckDB tutorial](https://docs.dagster.io/integrations/libraries/duckdb/using-duckdb-with-dagster) . If you want to store assets in different schemas, you can specify the schema as metadata: daffodil_dataset = AssetSpec( key=["daffodil_dataset"], metadata={"schema": "daffodil"} ) @asset(metadata={"schema": "iris"}) def iris_dataset() -> pd.DataFrame: return pd.read_csv( "https://docs.dagster.io/assets/iris.csv", names=[ "sepal_length_cm", "sepal_width_cm", "petal_length_cm", "petal_width_cm", "species", ], ) You can also specify the schema as part of the asset's key: daffodil_dataset = AssetSpec(key=["daffodil", "daffodil_dataset"]) @asset(key_prefix=["iris"]) def iris_dataset() -> pd.DataFrame: return pd.read_csv( "https://docs.dagster.io/assets/iris.csv", names=[ "sepal_length_cm", "sepal_width_cm", "petal_length_cm", "petal_width_cm", "species", ], ) In this example, the `iris_dataset` asset will be stored in the `IRIS` schema, and the `daffodil_dataset` asset will be found in the `DAFFODIL` schema. note The schema is determined in this order: 1. If the schema is set via metadata, that schema will be used 2. Otherwise, the schema set as configuration on the I/O manager will be used 3. Otherwise, if there is a `key_prefix`, that schema will be used 4. If none of the above are provided, the default schema will be `PUBLIC` ### Using the DuckDB I/O manager with other I/O managers[​](https://docs.dagster.io/integrations/libraries/duckdb/reference#using-the-duckdb-io-manager-with-other-io-managers "Direct link to Using the DuckDB I/O manager with other I/O managers") You may have assets that you don't want to store in DuckDB. You can provide an I/O manager to each asset using the `io_manager_key` parameter in the [`@dg.asset`](https://docs.dagster.io/api/dagster/assets#dagster.asset) decorator: import pandas as pdfrom dagster_aws.s3.io_manager import s3_pickle_io_managerfrom dagster_duckdb_pandas import DuckDBPandasIOManagerfrom dagster import Definitions, asset@asset(io_manager_key="warehouse_io_manager")def iris_dataset() -> pd.DataFrame: return pd.read_csv( "https://docs.dagster.io/assets/iris.csv", names=[ "sepal_length_cm", "sepal_width_cm", "petal_length_cm", "petal_width_cm", "species", ], )@asset(io_manager_key="blob_io_manager")def iris_plots(iris_dataset): # plot_data is a function we've defined somewhere else # that plots the data in a DataFrame return plot_data(iris_dataset)defs = Definitions( assets=[iris_dataset, iris_plots], resources={ "warehouse_io_manager": DuckDBPandasIOManager( database="path/to/my_duckdb_database.duckdb", schema="IRIS", ), "blob_io_manager": s3_pickle_io_manager, },) In this example: * The `iris_dataset` asset uses the I/O manager bound to the key `warehouse_io_manager` and `iris_plots` uses the I/O manager bound to the key `blob_io_manager` * In the [`Definitions`](https://docs.dagster.io/api/dagster/definitions#dagster.Definitions) object, we supply the I/O managers for those keys * When the assets are materialized, the `iris_dataset` will be stored in DuckDB, and `iris_plots` will be saved in Amazon S3 ### Storing and loading PySpark or Polars DataFrames in DuckDB[​](https://docs.dagster.io/integrations/libraries/duckdb/reference#storing-and-loading-pyspark-or-polars-dataframes-in-duckdb "Direct link to Storing and loading PySpark or Polars DataFrames in DuckDB") The DuckDB I/O manager also supports storing and loading PySpark and Polars DataFrames. * Storing and loading PySpark DataFrames in DuckDB * Storing and loading Polars DataFrames in DuckDB To use the [`DuckDBPySparkIOManager`](https://docs.dagster.io/api/libraries/dagster-duckdb-pyspark#dagster_duckdb_pyspark.DuckDBPySparkIOManager) , first install the package: * uv * pip uv add dagster-duckdb-pyspark pip install dagster-duckdb-pyspark Then you can use the `DuckDBPySparkIOManager` in your [`Definitions`](https://docs.dagster.io/api/dagster/definitions#dagster.Definitions) as in [Step 1: Configure the DuckDB I/O manager](https://docs.dagster.io/integrations/libraries/duckdb/using-duckdb-with-dagster#step-1-configure-the-duckdb-io-manager) of the [Using Dagster with DuckDB tutorial](https://docs.dagster.io/integrations/libraries/duckdb/using-duckdb-with-dagster) . from dagster_duckdb_pyspark import DuckDBPySparkIOManagerfrom dagster import Definitionsdefs = Definitions( assets=[iris_dataset], resources={ "io_manager": DuckDBPySparkIOManager( database="path/to/my_duckdb_database.duckdb", # required schema="IRIS", # optional, defaults to PUBLIC ) },) The `DuckDBPySparkIOManager` requires an active `SparkSession`. You can either create your own `SparkSession` or use the [`spark_resource`](https://docs.dagster.io/api/libraries/dagster-spark#dagster_spark.spark_resource) . * With the spark\_resource * With your own SparkSession from dagster_duckdb_pyspark import DuckDBPySparkIOManagerfrom dagster_pyspark import pyspark_resourcefrom pyspark import SparkFilesfrom pyspark.sql import DataFramefrom pyspark.sql.types import DoubleType, StringType, StructField, StructTypefrom dagster import AssetExecutionContext, Definitions, asset@asset(required_resource_keys={"pyspark"})def iris_dataset(context: AssetExecutionContext) -> DataFrame: spark = context.resources.pyspark.spark_session schema = StructType( [ StructField("sepal_length_cm", DoubleType()), StructField("sepal_width_cm", DoubleType()), StructField("petal_length_cm", DoubleType()), StructField("petal_width_cm", DoubleType()), StructField("species", StringType()), ] ) url = "https://docs.dagster.io/assets/iris.csv" spark.sparkContext.addFile(url) return spark.read.schema(schema).csv("file://" + SparkFiles.get("iris.csv"))defs = Definitions( assets=[iris_dataset], resources={ "io_manager": DuckDBPySparkIOManager( database="path/to/my_duckdb_database.duckdb", schema="IRIS", ), "pyspark": pyspark_resource, },) from dagster_duckdb_pyspark import DuckDBPySparkIOManagerfrom pyspark import SparkFilesfrom pyspark.sql import DataFrame, SparkSessionfrom pyspark.sql.types import DoubleType, StringType, StructField, StructTypefrom dagster import Definitions, asset@assetdef iris_dataset() -> DataFrame: spark = SparkSession.builder.getOrCreate() schema = StructType( [ StructField("sepal_length_cm", DoubleType()), StructField("sepal_width_cm", DoubleType()), StructField("petal_length_cm", DoubleType()), StructField("petal_width_cm", DoubleType()), StructField("species", StringType()), ] ) url = "https://docs.dagster.io/assets/iris.csv" spark.sparkContext.addFile(url) return spark.read.schema(schema).csv("file://" + SparkFiles.get("iris.csv"))defs = Definitions( assets=[iris_dataset], resources={ "io_manager": DuckDBPySparkIOManager( database="path/to/my_duckdb_database.duckdb", schema="IRIS", ) },) To use the [`DuckDBPolarsIOManager`](https://docs.dagster.io/api/libraries/dagster-duckdb-polars#dagster_duckdb_polars.DuckDBPolarsIOManager) , first install the package: * uv * pip uv add dagster-duckdb-polars pip install dagster-duckdb-polars Then you can use the `DuckDBPolarsIOManager` in your [`Definitions`](https://docs.dagster.io/api/dagster/definitions#dagster.Definitions) as in [Step 1: Configure the DuckDB I/O manager](https://docs.dagster.io/integrations/libraries/duckdb/using-duckdb-with-dagster#step-1-configure-the-duckdb-io-manager) of the [Using Dagster with DuckDB tutorial](https://docs.dagster.io/integrations/libraries/duckdb/using-duckdb-with-dagster) . from dagster_duckdb_polars import DuckDBPolarsIOManagerfrom dagster import Definitionsdefs = Definitions( assets=[iris_dataset], resources={ "io_manager": DuckDBPolarsIOManager( database="path/to/my_duckdb_database.duckdb", # required schema="IRIS", # optional, defaults to PUBLIC ) },) ### Storing multiple DataFrame types in DuckDB[​](https://docs.dagster.io/integrations/libraries/duckdb/reference#storing-multiple-dataframe-types-in-duckdb "Direct link to Storing multiple DataFrame types in DuckDB") If you work with several DataFrame libraries and want a single I/O manager to handle storing and loading these DataFrames in DuckDB, you can write a new I/O manager that handles the DataFrame types. To do this, inherit from the [`DuckDBIOManager`](https://docs.dagster.io/api/libraries/dagster-duckdb#dagster_duckdb.DuckDBIOManager) base class and implement the `type_handlers` and `default_load_type` methods. The resulting I/O manager will inherit the configuration fields of the base `DuckDBIOManager`. from typing import Optionalimport pandas as pdfrom dagster_duckdb import DuckDBIOManagerfrom dagster_duckdb_pandas import DuckDBPandasTypeHandlerfrom dagster_duckdb_polars import DuckDBPolarsTypeHandlerfrom dagster_duckdb_pyspark import DuckDBPySparkTypeHandlerfrom dagster import Definitionsclass DuckDBPandasPySparkPolarsIOManager(DuckDBIOManager): @staticmethod def type_handlers(): """type_handlers should return a list of the TypeHandlers that the I/O manager can use. Here we return the DuckDBPandasTypeHandler, DuckDBPySparkTypeHandler, and DuckDBPolarsTypeHandler so that the I/O manager can store Pandas DataFrames, PySpark DataFrames, and Polars DataFrames. """ return [ DuckDBPandasTypeHandler(), DuckDBPySparkTypeHandler(), DuckDBPolarsTypeHandler(), ] @staticmethod def default_load_type() -> Optional[type]: """If an asset is not annotated with an return type, default_load_type will be used to determine which TypeHandler to use to store and load the output. In this case, unannotated assets will be stored and loaded as Pandas DataFrames. """ return pd.DataFramedefs = Definitions( assets=[iris_dataset, rose_dataset], resources={ "io_manager": DuckDBPandasPySparkPolarsIOManager( database="path/to/my_duckdb_database.duckdb", schema="IRIS", ) },) * [DuckDB resource](https://docs.dagster.io/integrations/libraries/duckdb/reference#duckdb-resource) * [Executing custom SQL queries](https://docs.dagster.io/integrations/libraries/duckdb/reference#executing-custom-sql-queries) * [DuckDB I/O manager](https://docs.dagster.io/integrations/libraries/duckdb/reference#duckdb-io-manager) * [Selecting specific columns in a downstream asset](https://docs.dagster.io/integrations/libraries/duckdb/reference#selecting-specific-columns-in-a-downstream-asset) * [Storing partitioned assets](https://docs.dagster.io/integrations/libraries/duckdb/reference#storing-partitioned-assets) * [Storing tables in multiple schemas](https://docs.dagster.io/integrations/libraries/duckdb/reference#storing-tables-in-multiple-schemas) * [Using the DuckDB I/O manager with other I/O managers](https://docs.dagster.io/integrations/libraries/duckdb/reference#using-the-duckdb-io-manager-with-other-io-managers) * [Storing and loading PySpark or Polars DataFrames in DuckDB](https://docs.dagster.io/integrations/libraries/duckdb/reference#storing-and-loading-pyspark-or-polars-dataframes-in-duckdb) * [Storing multiple DataFrame types in DuckDB](https://docs.dagster.io/integrations/libraries/duckdb/reference#storing-multiple-dataframe-types-in-duckdb) --- # Dagster & HashiCorp Vault | Dagster Docs [Skip to main content](https://docs.dagster.io/integrations/libraries/hashicorp#__docusaurus_skipToContent_fallback) On this page A package for integrating HashiCorp Vault into Dagster so that you can securely manage tokens and passwords. Installation[​](https://docs.dagster.io/integrations/libraries/hashicorp#installation "Direct link to Installation") --------------------------------------------------------------------------------------------------------------------- * uv * pip uv add dagster-hashicorp pip install dagster-hashicorp Example[​](https://docs.dagster.io/integrations/libraries/hashicorp#example "Direct link to Example") ------------------------------------------------------------------------------------------------------ # See the Resources docs to learn more: https://docs.dagster.io/concepts/resourcesimport osfrom dagster_hashicorp.vault import vault_resourceimport dagster as dg@dg.asset(required_resource_keys={"vault"})def example_asset(context): secret_data = context.resources.vault.read_secret(secret_path="secret/data/foo/bar") context.log.debug(f"Secret: {secret_data}")defs = dg.Definitions( assets=[example_asset], resources={ "vault": vault_resource.configured( { "url": "vault-host:8200", "auth_type": {"token": {"token": dg.EnvVar("VAULT_AUTH_TOKEN")}}, } ) },) About HashiCorp Vault[​](https://docs.dagster.io/integrations/libraries/hashicorp#about-hashicorp-vault "Direct link to About HashiCorp Vault") ------------------------------------------------------------------------------------------------------------------------------------------------ **HashiCorp** provides open source tools and commercial products that enable developers, operators and security professionals to provision, secure, run and connect cloud-computing infrastructure. **HashiCorp Vault** secures, stores, and tightly controls access to tokens, passwords, certificates, API keys, and other secrets in modern computing. * [Installation](https://docs.dagster.io/integrations/libraries/hashicorp#installation) * [Example](https://docs.dagster.io/integrations/libraries/hashicorp#example) * [About HashiCorp Vault](https://docs.dagster.io/integrations/libraries/hashicorp#about-hashicorp-vault) --- # Dagster & Evidence | Dagster Docs [Skip to main content](https://docs.dagster.io/integrations/libraries/evidence#__docusaurus_skipToContent_fallback) On this page The Evidence library offers a component to easily generate dashboards from your Evidence project. Installation[​](https://docs.dagster.io/integrations/libraries/evidence#installation "Direct link to Installation") -------------------------------------------------------------------------------------------------------------------- * uv * pip uv add dagster-evidence pip install dagster-evidence Example[​](https://docs.dagster.io/integrations/libraries/evidence#example "Direct link to Example") ----------------------------------------------------------------------------------------------------- Create an instance of an `EvidenceProject` component in your `defs.yaml` file and define the asset(s) generated by your Evidence project. type: dagster_evidence.EvidenceProjectattributes: project_path: ../jaffle_dashboard asset: key: jaffle_dashboard deps: - target/main/orders - target/main/customers deploy_command: 'echo "Dashboard built at $EVIDENCE_BUILD_PATH"' About Evidence[​](https://docs.dagster.io/integrations/libraries/evidence#about-evidence "Direct link to About Evidence") -------------------------------------------------------------------------------------------------------------------------- **Evidence** is a lightweight framework for building data apps. It's open source and free to get started. * [Installation](https://docs.dagster.io/integrations/libraries/evidence#installation) * [Example](https://docs.dagster.io/integrations/libraries/evidence#example) * [About Evidence](https://docs.dagster.io/integrations/libraries/evidence#about-evidence) --- # Dagster & GitHub | Dagster Docs [Skip to main content](https://docs.dagster.io/integrations/libraries/github#__docusaurus_skipToContent_fallback) On this page warning This feature is considered deprecated. It is still available, but will be removed in the future, and we recommend avoiding new usage. For more information, see the [API lifecycle stages documentation](https://docs.dagster.io/api/api-lifecycle/api-lifecycle-stages) . This library provides an integration with GitHub Apps by providing a thin wrapper on the GitHub v4 GraphQL API. This allows for automating operations within your GitHub repositories and with the tighter permissions scopes that GitHub Apps allow for vs using a personal token. Installation[​](https://docs.dagster.io/integrations/libraries/github#installation "Direct link to Installation") ------------------------------------------------------------------------------------------------------------------ * uv * pip uv add dagster-github pip install dagster-github Example[​](https://docs.dagster.io/integrations/libraries/github#example "Direct link to Example") --------------------------------------------------------------------------------------------------- from dagster_github import GithubResourceimport dagster as dg@dg.assetdef github_asset(github: GithubResource): github.get_client().create_issue( repo_name="dagster", repo_owner="dagster-io", title="Dagster's first github issue", body="this open source thing seems like a pretty good idea", )defs = dg.Definitions( assets=[github_asset], resources={ "github": GithubResource( github_app_id=dg.EnvVar.int("GITHUB_APP_ID"), github_app_private_rsa_key=dg.EnvVar("GITHUB_PRIVATE_KEY"), github_installation_id=dg.EnvVar.int("GITHUB_INSTALLATION_ID"), ) },) About GitHub[​](https://docs.dagster.io/integrations/libraries/github#about-github "Direct link to About GitHub") ------------------------------------------------------------------------------------------------------------------ **GitHub** provides a highly available git repo, access control, bug tracking, software feature requests, task management, continuous integration, and wikis for open source and commercial projects. * [Installation](https://docs.dagster.io/integrations/libraries/github#installation) * [Example](https://docs.dagster.io/integrations/libraries/github#example) * [About GitHub](https://docs.dagster.io/integrations/libraries/github#about-github) --- # Dagster & dlt | Dagster Docs [Skip to main content](https://docs.dagster.io/integrations/libraries/dlt#__docusaurus_skipToContent_fallback) On this page note If you are just getting started with the dlt integration, we recommend using the new [dlt component](https://docs.dagster.io/guides/build/components/integrations/dlt-component-tutorial) . The dltHub open-source library defines a standardized approach for creating data pipelines that load often messy data sources into well-structured data sets. It offers many advanced features, such as: * Handling connection secrets * Converting data into the structure required for a destination * Incremental updates and merges dlt also provides a large collection of [pre-built, verified sources](https://dlthub.com/docs/dlt-ecosystem/verified-sources) and [destinations](https://dlthub.com/docs/dlt-ecosystem/destinations) , allowing you to write less code (if any!) by leveraging the work of the dlt community. In this guide, we'll explain how the dlt integration works, how to set up a Dagster project for dlt, and how to use a pre-defined dlt source. How it works[​](https://docs.dagster.io/integrations/libraries/dlt#how-it-works "Direct link to How it works") --------------------------------------------------------------------------------------------------------------- The Dagster dlt integration uses [multi-assets](https://docs.dagster.io/guides/build/assets/defining-assets#multi-asset) , a single definition that results in multiple assets. These assets are derived from the `DltSource`. The following is an example of a dlt source definition where a source is made up of two resources: @dlt.sourcedef example(api_key: str = dlt.secrets.value): @dlt.resource(primary_key="id", write_disposition="merge") def courses(): response = requests.get(url=BASE_URL + "courses") response.raise_for_status() yield response.json().get("items") @dlt.resource(primary_key="id", write_disposition="merge") def users(): for page in _paginate(BASE_URL + "users"): yield page return courses, users Each resource queries an API endpoint and yields the data that we wish to load into our data warehouse. The two resources defined on the source will map to Dagster assets. Next, we defined a dlt pipeline that specifies how we want the data to be loaded: pipeline = dlt.pipeline( pipeline_name="example_pipeline", destination="snowflake", dataset_name="example_data", progress="log",) A dlt source and pipeline are the two components required to load data using dlt. These will be the parameters of our multi-asset, which will integrate dlt and Dagster. Prerequisites[​](https://docs.dagster.io/integrations/libraries/dlt#prerequisites "Direct link to Prerequisites") ------------------------------------------------------------------------------------------------------------------ To follow the steps in this guide, you'll need: * **To read the [dlt introduction](https://dlthub.com/docs/intro) **, if you've never worked with dlt before. * **[To install](https://docs.dagster.io/getting-started/installation) the following libraries**: * uv * pip uv add dagster-dlt pip install dagster-dlt Installing `dagster-dlt` will also install the `dlt` package. Step 1: Configure your Dagster project to support dlt[​](https://docs.dagster.io/integrations/libraries/dlt#step-1-configure-your-dagster-project-to-support-dlt "Direct link to Step 1: Configure your Dagster project to support dlt") ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- The first step is to define a location for the `dlt` code used for ingesting data. We recommend creating a `dlt_sources` directory at the root of your Dagster project, but this code can reside anywhere within your Python project. Run the following to create the `dlt_sources` directory: cd $DAGSTER_HOME && mkdir dlt_sources Step 2: Initialize dlt ingestion code[​](https://docs.dagster.io/integrations/libraries/dlt#step-2-initialize-dlt-ingestion-code "Direct link to Step 2: Initialize dlt ingestion code") ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- In the `dlt_sources` directory, you can write ingestion code following the [dlt tutorial](https://dlthub.com/docs/tutorial/load-data-from-an-api) or you can use a verified source. In this example, we'll use the [GitHub source](https://dlthub.com/docs/dlt-ecosystem/verified-sources/github) provided by dlt. 1. Run the following to create a location for the dlt source code and initialize the GitHub source: cd dlt_sourcesdlt init github snowflake At which point you'll see the following in the command line: Looking up the init scripts in https://github.com/dlt-hub/verified-sources.git...Cloning and configuring a verified source github (Source that load github issues, pull requests and reactions for a specific repository via customizable graphql query. Loads events incrementally.) 2. When prompted to proceed, enter `y`. You should see the following confirming that the GitHub source was added to the project: Verified source github was added to your project!* See the usage examples and code snippets to copy from github_pipeline.py* Add credentials for snowflake and other secrets in ./.dlt/secrets.toml* requirements.txt was created. Install it with:pip3 install -r requirements.txt* Read https://dlthub.com/docs/walkthroughs/create-a-pipeline for more information This downloaded the code required to collect data from the GitHub API. It also created a `requirements.txt` and a `.dlt/` configuration directory. These files can be removed, as we will configure our pipelines through Dagster, however, you may still find it informative to reference. $ tree -a.├── .dlt # can be removed│   ├── .sources│   ├── config.toml│   └── secrets.toml├── .gitignore├── github│   ├── README.md│   ├── __init__.py│   ├── helpers.py│   ├── queries.py│   └── settings.py├── github_pipeline.py└── requirements.txt # can be removed Step 3: Define dlt environment variables[​](https://docs.dagster.io/integrations/libraries/dlt#step-3-define-dlt-environment-variables "Direct link to Step 3: Define dlt environment variables") -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- This integration manages connections and secrets using environment variables as `dlt`. The `dlt` library can infer required environment variables used by its sources and resources. Refer to [dlt's Secrets and Configs](https://dlthub.com/docs/general-usage/credentials/configuration) documentation for more information. In the example we've been using: * The `github_reactions` source requires a GitHub access token * The Snowflake destination requires database connection details This results in the following required environment variables: SOURCES__GITHUB__ACCESS_TOKEN=""DESTINATION__SNOWFLAKE__CREDENTIALS__DATABASE=""DESTINATION__SNOWFLAKE__CREDENTIALS__PASSWORD=""DESTINATION__SNOWFLAKE__CREDENTIALS__USERNAME=""DESTINATION__SNOWFLAKE__CREDENTIALS__HOST=""DESTINATION__SNOWFLAKE__CREDENTIALS__WAREHOUSE=""DESTINATION__SNOWFLAKE__CREDENTIALS__ROLE="" Ensure that these variables are defined in your environment, either in your `.env` file when running locally or in the [Dagster deployment's environment variables](https://docs.dagster.io/guides/operate/configuration/using-environment-variables-and-secrets) . Step 4: Define a DagsterDltResource[​](https://docs.dagster.io/integrations/libraries/dlt#step-4-define-a-dagsterdltresource "Direct link to Step 4: Define a DagsterDltResource") ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Next, we'll define a [`DagsterDltResource`](https://docs.dagster.io/api/libraries/dagster-dlt#dagster_dlt.DagsterDltResource) , which provides a wrapper of a dlt pipeline runner. Use the following to define the resource, which can be shared across all dlt pipelines: from dagster_dlt import DagsterDltResourcedlt_resource = DagsterDltResource() We'll add the resource to our [`Definitions`](https://docs.dagster.io/api/dagster/definitions#dagster.Definitions) in a later step. Step 5: Create a dlt\_assets definition for GitHub[​](https://docs.dagster.io/integrations/libraries/dlt#step-5-create-a-dlt_assets-definition-for-github "Direct link to Step 5: Create a dlt_assets definition for GitHub") ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ The [`@dagster_dlt.dlt_assets`](https://docs.dagster.io/api/libraries/dagster-dlt#dagster_dlt.dlt_assets) decorator takes a `dlt_source` and `dlt_pipeline` parameter. In this example, we used the `github_reactions` source and created a `dlt_pipeline` to ingest data from Github to Snowflake. In the same file containing your Dagster assets, you can create an instance of your [`@dagster_dlt.dlt_assets`](https://docs.dagster.io/api/libraries/dagster-dlt#dagster_dlt.dlt_assets) by doing something like the following: ::: If you are using the [sql\_database](https://dlthub.com/docs/api_reference/sources/sql_database/__init__#sql_database) source, consider setting `defer_table_reflect=True` to reduce database reads. By default, the Dagster daemon will refresh definitions roughly every minute, which will query the database for resource definitions. ::: from dagster import AssetExecutionContext, Definitionsfrom dagster_dlt import DagsterDltResource, dlt_assetsfrom dlt import pipelinefrom dlt_sources.github import github_reactions@dlt_assets( dlt_source=github_reactions( "dagster-io", "dagster", max_items=250 ), dlt_pipeline=pipeline( pipeline_name="github_issues", dataset_name="github", destination="snowflake", progress="log", ), name="github", group_name="github",)def dagster_github_assets(context: AssetExecutionContext, dlt: DagsterDltResource): yield from dlt.run(context=context) Step 6: Create the Definitions object[​](https://docs.dagster.io/integrations/libraries/dlt#step-6-create-the-definitions-object "Direct link to Step 6: Create the Definitions object") ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- The last step is to include the assets and resource in a [`Definitions`](https://docs.dagster.io/api/dagster/definitions#dagster.Definitions) object. This enables Dagster tools to load everything we've defined: defs = Definitions( assets=[ dagster_github_assets, ], resources={ "dlt": dlt_resource, },) And that's it! You should now have two assets that load data to corresponding Snowflake tables: one for issues and the other for pull requests. Advanced usage[​](https://docs.dagster.io/integrations/libraries/dlt#advanced-usage "Direct link to Advanced usage") --------------------------------------------------------------------------------------------------------------------- ### Overriding the translator to customize dlt assets[​](https://docs.dagster.io/integrations/libraries/dlt#overriding-the-translator-to-customize-dlt-assets "Direct link to Overriding the translator to customize dlt assets") The [`DagsterDltTranslator`](https://docs.dagster.io/api/libraries/dagster-dlt#dagster_dlt.DagsterDltTranslator) object can be used to customize how dlt properties map to Dagster concepts. For example, to change how the name of the asset is derived, or if you would like to change the key of the upstream source asset, you can override the [`DagsterDltTranslator`](https://docs.dagster.io/api/libraries/dagster-dlt#dagster_dlt.DagsterDltTranslator) method. import dltfrom dagster_dlt import DagsterDltResource, DagsterDltTranslator, dlt_assetsfrom dagster_dlt.translator import DltResourceTranslatorDatafrom dagster import AssetExecutionContext, AssetKey, AssetSpec@dlt.sourcedef example_dlt_source(): def example_resource(): ... return example_resourceclass CustomDagsterDltTranslator(DagsterDltTranslator): def get_asset_spec(self, data: DltResourceTranslatorData) -> AssetSpec: """Overrides asset spec to override asset key to be the dlt resource name.""" default_spec = super().get_asset_spec(data) return default_spec.replace_attributes( key=AssetKey(f"{data.resource.name}"), )@dlt_assets( name="example_dlt_assets", dlt_source=example_dlt_source(), dlt_pipeline=dlt.pipeline( pipeline_name="example_pipeline_name", dataset_name="example_dataset_name", destination="snowflake", progress="log", ), dagster_dlt_translator=CustomDagsterDltTranslator(),)def dlt_example_assets(context: AssetExecutionContext, dlt: DagsterDltResource): yield from dlt.run(context=context) In this example, we customized the translator to change how the dlt assets' names are defined. ### Assigning metadata to upstream external assets[​](https://docs.dagster.io/integrations/libraries/dlt#assigning-metadata-to-upstream-external-assets "Direct link to Assigning metadata to upstream external assets") A common question is how to define metadata on the external assets upstream of the dlt assets. This can be accomplished by defining a [`AssetSpec`](https://docs.dagster.io/api/dagster/assets#dagster.AssetSpec) with a key that matches the one defined in the [`DagsterDltTranslator`](https://docs.dagster.io/api/libraries/dagster-dlt#dagster_dlt.DagsterDltTranslator) method. For example, let's say we have defined a set of dlt assets named `thinkific_assets`, we can iterate over those assets and derive a [`AssetSpec`](https://docs.dagster.io/api/dagster/assets#dagster.AssetSpec) with attributes like `group_name`. import dltfrom dagster_dlt import DagsterDltResource, dlt_assetsfrom dagster import AssetExecutionContext, AssetSpec@dlt.sourcedef example_dlt_source(): def example_resource(): ... return example_resource@dlt_assets( dlt_source=example_dlt_source(), dlt_pipeline=dlt.pipeline( pipeline_name="example_pipeline_name", dataset_name="example_dataset_name", destination="snowflake", progress="log", ),)def example_dlt_assets(context: AssetExecutionContext, dlt: DagsterDltResource): yield from dlt.run(context=context)thinkific_source_assets = [ AssetSpec(key, group_name="thinkific") for key in example_dlt_assets.dependency_keys] ### Customize upstream dependencies[​](https://docs.dagster.io/integrations/libraries/dlt#customize-upstream-dependencies "Direct link to Customize upstream dependencies") By default, Dagster sets upstream dependencies when generating asset specs for your dlt assets. To do so, Dagster parses information about assets that are upstream of specific dlt assets from the dlt resource itself. You can customize how upstream dependencies are set on your dlt assets by passing an instance of the custom [`DagsterDltTranslator`](https://docs.dagster.io/api/libraries/dagster-dlt#dagster_dlt.DagsterDltTranslator) to the [`build_dlt_asset_specs`](https://docs.dagster.io/api/libraries/dagster-dlt#dagster_dlt.build_dlt_asset_specs) function. @dlt.sourcedef example_dlt_source(): def example_resource(): ... return example_resourceclass CustomDagsterDltTranslator(DagsterDltTranslator): def get_asset_spec(self, data: DltResourceTranslatorData) -> AssetSpec: """Overrides asset spec to override upstream asset key to be a single source asset.""" # We create the default asset spec using super() default_spec = super().get_asset_spec(data) # We set an upstream dependency for our assets return default_spec.replace_attributes( deps=[AssetKey("common_upstream_dlt_dependency")], )dlt_specs = build_dlt_asset_specs( dlt_source=example_dlt_source(), dlt_pipeline=dlt.pipeline( pipeline_name="example_pipeline_name", dataset_name="example_dataset_name", destination="snowflake", progress="log", ), dagster_dlt_translator=CustomDagsterDltTranslator(),) Note that `super()` is called in each of the overridden methods to generate the default asset spec. It is best practice to generate the default asset spec before customizing it. You can also pass an instance of the custom [`DagsterDltTranslator`](https://docs.dagster.io/api/libraries/dagster-dlt#dagster_dlt.DagsterDltTranslator) to the [`dlt_assets`](https://docs.dagster.io/api/libraries/dagster-dlt#dagster_dlt.dlt_assets) decorator. ### Define downstream dependencies[​](https://docs.dagster.io/integrations/libraries/dlt#define-downstream-dependencies "Direct link to Define downstream dependencies") Dagster allows you to define assets that are downstream of specific dlt resources using their asset keys. The asset key for a dlt resource can be retrieved using the [`DagsterDltTranslator`](https://docs.dagster.io/api/libraries/dagster-dlt#dagster_dlt.DagsterDltTranslator) . The below example defines `example_downstream_asset` as a downstream dependency of `example_dlt_resource`: import dltfrom dagster_dlt import DagsterDltResource, DagsterDltTranslator, dlt_assetsfrom dagster_dlt.translator import DltResourceTranslatorDatafrom dagster import AssetExecutionContext, asset@dlt.sourcedef example_dlt_source(): def example_resource(): ... return example_resourceexample_dlt_pipeline = dlt.pipeline( pipeline_name="example_pipeline_name", dataset_name="example_dataset_name", destination="snowflake", progress="log",)@dlt_assets( dlt_source=example_dlt_source(), dlt_pipeline=example_dlt_pipeline,)def example_dlt_assets(context: AssetExecutionContext, dlt: DagsterDltResource): yield from dlt.run(context=context)example_dlt_resource_asset_key = next( iter( [ DagsterDltTranslator().get_asset_spec( data=DltResourceTranslatorData( resource=dlt_source_resource, pipeline=example_dlt_pipeline, ) ) for dlt_source_resource in example_dlt_source().selected_resources.values() if dlt_source_resource.name == "example_resource" ] ))@asset(deps=[example_dlt_resource_asset_key])def example_downstream_asset(): ... In the downstream asset, you may want direct access to the contents of the dlt resource. To do so, you can customize the code within your `@asset`\-decorated function to load upstream data. ### Using partitions in your dlt assets[​](https://docs.dagster.io/integrations/libraries/dlt#using-partitions-in-your-dlt-assets "Direct link to Using partitions in your dlt assets") It is possible to use partitions within your dlt assets. However, it should be noted that this may result in concurrency related issues as state is managed by dlt. For this reason, it is recommended to set concurrency limits for your partitioned dlt assets. See the [Limiting concurrency in data pipelines](https://docs.dagster.io/guides/operate/managing-concurrency) guide for more details. That said, here is an example of using static named partitions from a dlt source. from typing import Optionalimport dltfrom dagster_dlt import DagsterDltResource, dlt_assetsfrom dagster import AssetExecutionContext, StaticPartitionsDefinitioncolor_partitions = StaticPartitionsDefinition(["red", "green", "blue"])@dlt.sourcedef example_dlt_source(color: Optional[str] = None): def load_colors(): if color: # partition-specific processing ... else: # non-partitioned processing ...@dlt_assets( dlt_source=example_dlt_source(), name="example_dlt_assets", dlt_pipeline=dlt.pipeline( pipeline_name="example_pipeline_name", dataset_name="example_dataset_name", destination="snowflake", ), partitions_def=color_partitions,)def compute(context: AssetExecutionContext, dlt: DagsterDltResource): color = context.partition_key yield from dlt.run(context=context, dlt_source=example_dlt_source(color=color)) What's next?[​](https://docs.dagster.io/integrations/libraries/dlt#whats-next "Direct link to What's next?") ------------------------------------------------------------------------------------------------------------- Want to see real-world examples of dlt in production? Check out how we use it internally at Dagster in the [Dagster Open Platform](https://github.com/dagster-io/dagster-open-platform) project. ### About dlt[​](https://docs.dagster.io/integrations/libraries/dlt#about-dlt "Direct link to About dlt") [Data Load Tool (dlt)](https://dlthub.com/) is an open source library for creating efficient data pipelines. It offers features like secret management, data structure conversion, incremental updates, and pre-built sources and destinations, simplifying the process of loading messy data into well-structured datasets. * [How it works](https://docs.dagster.io/integrations/libraries/dlt#how-it-works) * [Prerequisites](https://docs.dagster.io/integrations/libraries/dlt#prerequisites) * [Step 1: Configure your Dagster project to support dlt](https://docs.dagster.io/integrations/libraries/dlt#step-1-configure-your-dagster-project-to-support-dlt) * [Step 2: Initialize dlt ingestion code](https://docs.dagster.io/integrations/libraries/dlt#step-2-initialize-dlt-ingestion-code) * [Step 3: Define dlt environment variables](https://docs.dagster.io/integrations/libraries/dlt#step-3-define-dlt-environment-variables) * [Step 4: Define a DagsterDltResource](https://docs.dagster.io/integrations/libraries/dlt#step-4-define-a-dagsterdltresource) * [Step 5: Create a dlt\_assets definition for GitHub](https://docs.dagster.io/integrations/libraries/dlt#step-5-create-a-dlt_assets-definition-for-github) * [Step 6: Create the Definitions object](https://docs.dagster.io/integrations/libraries/dlt#step-6-create-the-definitions-object) * [Advanced usage](https://docs.dagster.io/integrations/libraries/dlt#advanced-usage) * [Overriding the translator to customize dlt assets](https://docs.dagster.io/integrations/libraries/dlt#overriding-the-translator-to-customize-dlt-assets) * [Assigning metadata to upstream external assets](https://docs.dagster.io/integrations/libraries/dlt#assigning-metadata-to-upstream-external-assets) * [Customize upstream dependencies](https://docs.dagster.io/integrations/libraries/dlt#customize-upstream-dependencies) * [Define downstream dependencies](https://docs.dagster.io/integrations/libraries/dlt#define-downstream-dependencies) * [Using partitions in your dlt assets](https://docs.dagster.io/integrations/libraries/dlt#using-partitions-in-your-dlt-assets) * [What's next?](https://docs.dagster.io/integrations/libraries/dlt#whats-next) * [About dlt](https://docs.dagster.io/integrations/libraries/dlt#about-dlt) --- # GCP | Dagster Docs [Skip to main content](https://docs.dagster.io/integrations/libraries/gcp#__docusaurus_skipToContent_fallback) --- # Dagster & HashiCorp | Dagster Docs [Skip to main content](https://docs.dagster.io/integrations/libraries/hashicorp-nomad#__docusaurus_skipToContent_fallback) On this page The community-supported Nomad package provides an integration with HashiCorp Nomad. For more information, see the [dagster-nomad GitHub repository](https://github.com/PayLead/dagster-nomad) . Installation[​](https://docs.dagster.io/integrations/libraries/hashicorp-nomad#installation "Direct link to Installation") --------------------------------------------------------------------------------------------------------------------------- * uv * pip uv add dagster-nomad pip install dagster-nomad * [Installation](https://docs.dagster.io/integrations/libraries/hashicorp-nomad#installation) --- # Dagster & Java | Dagster Docs [Skip to main content](https://docs.dagster.io/integrations/libraries/java#__docusaurus_skipToContent_fallback) On this page The Java Pipes client provides a Java implementation of the Dagster Pipes protocol that can be used to orchestrate data processing pipelines written in Java from Dagster, while receiving logs and metadata from the Java application. Installation and examples[​](https://docs.dagster.io/integrations/libraries/java#installation-and-examples "Direct link to Installation and examples") ------------------------------------------------------------------------------------------------------------------------------------------------------- For installation information and examples, see the [community integrations GitHub repository](https://github.com/dagster-io/community-integrations/blob/main/libraries/pipes/implementations/java/README.md) . About Java[​](https://docs.dagster.io/integrations/libraries/java#about-java "Direct link to About Java") ---------------------------------------------------------------------------------------------------------- [Java](https://www.java.com/en) is a programming language and computing platform first released by Sun Microsystems in 1995. It has evolved from humble beginnings to power a large share of today’s digital world, by providing the reliable platform upon which many services and applications are built. New, innovative products and digital services designed for the future continue to rely on Java, as well. To get started with Java, see the [Java developer resources](https://dev.java/) . * [Installation and examples](https://docs.dagster.io/integrations/libraries/java#installation-and-examples) * [About Java](https://docs.dagster.io/integrations/libraries/java#about-java) --- # Using DuckDB with Dagster | Dagster Docs [Skip to main content](https://docs.dagster.io/integrations/libraries/duckdb/using-duckdb-with-dagster#__docusaurus_skipToContent_fallback) On this page This tutorial focuses on creating and interacting with DuckDB tables using Dagster's [asset definitions](https://docs.dagster.io/guides/build/assets/defining-assets) . The `dagster-duckdb` library provides two ways to interact with DuckDB tables: * [Resource](https://docs.dagster.io/guides/build/external-resources) : The resource allows you to directly run SQL queries against tables within an asset's compute function. Available resources: [`DuckDBResource`](https://docs.dagster.io/api/libraries/dagster-duckdb#dagster_duckdb.DuckDBResource) . * [I/O manager](https://docs.dagster.io/guides/build/io-managers) : The I/O manager transfers the responsibility of storing and loading DataFrames as DuckdB tables to Dagster. Available I/O managers: [`DuckDBPandasIOManager`](https://docs.dagster.io/api/libraries/dagster-duckdb-pandas#dagster_duckdb_pandas.DuckDBPandasIOManager) , [`DuckDBPySparkIOManager`](https://docs.dagster.io/api/libraries/dagster-duckdb-pyspark#dagster_duckdb_pyspark.DuckDBPySparkIOManager) , [`DuckDBPolarsIOManager`](https://docs.dagster.io/api/libraries/dagster-duckdb-polars#dagster_duckdb_polars.DuckDBPolarsIOManager) . This tutorial is divided into two sections to demonstrate the differences between the DuckDB resource and the DuckDB I/O manager. Each section will create the same assets, but the first section will use the DuckDB resource to store data in DuckDB, whereas the second section will use the DuckDB I/O manager. When writing your own assets, you may choose one or the other (or both) approaches depending on your storage requirements. In [Option 1](https://docs.dagster.io/integrations/libraries/duckdb/using-duckdb-with-dagster#option-1-using-the-duckdb-resource) you will: * Set up and configure the DuckDB resource. * Use the DuckDB resource to execute a SQL query to create a table. * Use the DuckDB resource to execute a SQL query to interact with the table. In [Option 2](https://docs.dagster.io/integrations/libraries/duckdb/using-duckdb-with-dagster#option-2-using-the-duckdb-io-manager) you will: * Set up and configure the DuckDB I/O manager. * Use Pandas to create a DataFrame, then delegate responsibility creating a table to the DuckDB I/O manager. * Use the DuckDB I/O manager to load the table into memory so that you can interact with it using the Pandas library. When writing your own assets, you may choose one or the other (or both) approaches depending on your storage requirements. By the end of the tutorial, you will: * Understand how to interact with a DuckDB database using the DuckDB resource. * Understand how to use the DuckDB I/O manager to store and load DataFrames as DuckDB tables. * Understand how to define dependencies between assets corresponding to tables in a DuckDB database. Prerequisites[​](https://docs.dagster.io/integrations/libraries/duckdb/using-duckdb-with-dagster#prerequisites "Direct link to Prerequisites") ----------------------------------------------------------------------------------------------------------------------------------------------- To complete this tutorial, you'll need: * **To install the `dagster-duckdb` and `dagster-duckdb-pandas` libraries**: * uv * pip uv add dagster-duckdb dagster-duckdb-pandas pip install dagster-duckdb dagster-duckdb-pandas Option 1: Using the DuckDB resource[​](https://docs.dagster.io/integrations/libraries/duckdb/using-duckdb-with-dagster#option-1-using-the-duckdb-resource "Direct link to Option 1: Using the DuckDB resource") ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- ### Step 1: Configure the DuckDB resource[​](https://docs.dagster.io/integrations/libraries/duckdb/using-duckdb-with-dagster#step-1-configure-the-duckdb-resource "Direct link to Step 1: Configure the DuckDB resource") To use the DuckDB resource, you'll need to add it to your `Definitions` object. The DuckDB resource requires some configuration. You must set a path to a DuckDB database as the `database` configuration value. If the database does not already exist, it will be created for you: from dagster_duckdb import DuckDBResourcefrom dagster import Definitionsdefs = Definitions( assets=[iris_dataset], resources={ "duckdb": DuckDBResource( database="path/to/my_duckdb_database.duckdb", # required ) },) ### Step 2: Create tables in DuckDB[​](https://docs.dagster.io/integrations/libraries/duckdb/using-duckdb-with-dagster#option-1-step-2 "Direct link to Step 2: Create tables in DuckDB") * Create DuckDB tables in Dagster * Making Dagster aware of existing tables **Create DuckDB tables in Dagster** Using the DuckDB resource, you can create DuckDB tables using the DuckDB Python API: import pandas as pdfrom dagster_duckdb import DuckDBResourcefrom dagster import asset@assetdef iris_dataset(duckdb: DuckDBResource) -> None: iris_df = pd.read_csv( "https://docs.dagster.io/assets/iris.csv", names=[ "sepal_length_cm", "sepal_width_cm", "petal_length_cm", "petal_width_cm", "species", ], ) with duckdb.get_connection() as conn: conn.execute("CREATE TABLE iris.iris_dataset AS SELECT * FROM iris_df") In this example, you're defining an asset that fetches the Iris dataset as a Pandas DataFrame and renames the columns. Then, using the DuckDB resource, the DataFrame is stored in DuckDB as the `iris.iris_dataset` table. **Making Dagster aware of existing tables** If you already have existing tables in DuckDB and other assets defined in Dagster depend on those tables, you may want Dagster to be aware of those upstream dependencies. Making Dagster aware of these tables will allow you to track the full data lineage in Dagster. You can accomplish this by defining [external assets](https://docs.dagster.io/guides/build/assets/external-assets) for these tables. from dagster import AssetSpeciris_harvest_data = AssetSpec(key="iris_harvest_data") In this example, you're creating a [`AssetSpec`](https://docs.dagster.io/api/dagster/assets#dagster.AssetSpec) for a pre-existing table called `iris_harvest_data`. Now you can run `dagster dev` and materialize the `iris_dataset` asset from the Dagster UI. ### Step 3: Define downstream assets[​](https://docs.dagster.io/integrations/libraries/duckdb/using-duckdb-with-dagster#step-3-define-downstream-assets "Direct link to Step 3: Define downstream assets") Once you have created an asset that represents a table in DuckDB, you will likely want to create additional assets that work with the data. from dagster import asset# this example uses the iris_dataset asset from Step 1@asset(deps=[iris_dataset])def iris_setosa(duckdb: DuckDBResource) -> None: with duckdb.get_connection() as conn: conn.execute( "CREATE TABLE iris.iris_setosa AS SELECT * FROM iris.iris_dataset WHERE" " species = 'Iris-setosa'" ) In this asset, you're creating second table that only contains the data for the _Iris Setosa_ species. This asset has a dependency on the `iris_dataset` asset. To define this dependency, you provide the `iris_dataset` asset as the `deps` parameter to the `iris_setosa` asset. You can then run the SQL query to create the table of _Iris Setosa_ data. ### Completed code example[​](https://docs.dagster.io/integrations/libraries/duckdb/using-duckdb-with-dagster#completed-code-example "Direct link to Completed code example") When finished, your code should look like the following: import pandas as pdfrom dagster_duckdb import DuckDBResourcefrom dagster import AssetSpec, Definitions, assetiris_harvest_data = AssetSpec(key="iris_harvest_data")@assetdef iris_dataset(duckdb: DuckDBResource) -> None: iris_df = pd.read_csv( "https://docs.dagster.io/assets/iris.csv", names=[ "sepal_length_cm", "sepal_width_cm", "petal_length_cm", "petal_width_cm", "species", ], ) with duckdb.get_connection() as conn: conn.execute("CREATE TABLE iris.iris_dataset AS SELECT * FROM iris_df")@asset(deps=[iris_dataset])def iris_setosa(duckdb: DuckDBResource) -> None: with duckdb.get_connection() as conn: conn.execute( "CREATE TABLE iris.iris_setosa AS SELECT * FROM iris.iris_dataset WHERE" " species = 'Iris-setosa'" )defs = Definitions( assets=[iris_dataset], resources={ "duckdb": DuckDBResource( database="path/to/my_duckdb_database.duckdb", ) },) Option 2: Using the DuckDB I/O manager[​](https://docs.dagster.io/integrations/libraries/duckdb/using-duckdb-with-dagster#option-2-using-the-duckdb-io-manager "Direct link to Option 2: Using the DuckDB I/O manager") ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ You may want to use an I/O manager to handle storing DataFrames as tables in DuckDB and loading DuckDB tables as DataFrames in downstream assets. You may want to use an I/O manager if: * You want your data to be loaded in memory so that you can interact with it using Python. * You'd like to have Dagster manage how you store the data and load it as an input in downstream assets. This section of the guide focuses on storing and loading Pandas DataFrames in DuckDB, but Dagster also supports using PySpark and Polars DataFrames with DuckDB. The concepts from this guide apply to working with PySpark and Polars DataFrames, and you can learn more about setting up and using the DuckDB I/O manager with PySpark and Polars DataFrames in the [reference guide](https://docs.dagster.io/integrations/libraries/duckdb/reference) . ### Step 1: Configure the DuckDB I/O manager[​](https://docs.dagster.io/integrations/libraries/duckdb/using-duckdb-with-dagster#step-1-configure-the-duckdb-io-manager "Direct link to Step 1: Configure the DuckDB I/O manager") To use the DuckDB I/O, you'll need to add it to your `Definitions` object. The DuckDB I/O manager requires some configuration to connect to your database. You must provide a path where a DuckDB database will be created. Additionally, you can specify a `schema` where the DuckDB I/O manager will create tables. from dagster_duckdb_pandas import DuckDBPandasIOManagerfrom dagster import Definitionsdefs = Definitions( assets=[iris_dataset], resources={ "io_manager": DuckDBPandasIOManager( database="path/to/my_duckdb_database.duckdb", # required schema="IRIS", # optional, defaults to PUBLIC ) },) ### Step 2: Create tables in DuckDB[​](https://docs.dagster.io/integrations/libraries/duckdb/using-duckdb-with-dagster#option-2-step-2 "Direct link to Step 2: Create tables in DuckDB") The DuckDB I/O manager can create and update tables for your Dagster-defined assets, but you can also make existing DuckDB tables available to Dagster. * Create tables in DuckDB from Dagster assets * Make existing tables available in Dagster #### Store a Dagster asset as a table in DuckDB[​](https://docs.dagster.io/integrations/libraries/duckdb/using-duckdb-with-dagster#store-a-dagster-asset-as-a-table-in-duckdb "Direct link to Store a Dagster asset as a table in DuckDB") To store data in DuckDB using the DuckDB I/O manager, you can simply return a Pandas DataFrame from your asset. Dagster will handle storing and loading your assets in DuckDB. import pandas as pdfrom dagster import asset@assetdef iris_dataset() -> pd.DataFrame: return pd.read_csv( "https://docs.dagster.io/assets/iris.csv", names=[ "sepal_length_cm", "sepal_width_cm", "petal_length_cm", "petal_width_cm", "species", ], ) In this example, you're defining an asset that fetches the Iris dataset as a Pandas DataFrame, renames the columns, then returns the DataFrame. The type signature of the function tells the I/O manager what data type it is working with, so it is important to include the return type `pd.DataFrame`. When Dagster materializes the `iris_dataset` asset using the configuration from [Step 1: Configure the DuckDB I/O manager](https://docs.dagster.io/integrations/libraries/duckdb/using-duckdb-with-dagster#step-1-configure-the-duckdb-io-manager) , the DuckDB I/O manager will create the table `IRIS.IRIS_DATASET` if it does not exist and replace the contents of the table with the value returned from the `iris_dataset` asset. **Make an existing table available in Dagster** If you already have existing tables in DuckDB and other assets defined in Dagster depend on those tables, you may want Dagster to be aware of those upstream dependencies. Making Dagster aware of these tables will allow you to track the full data lineage in Dagster. You can accomplish this by defining [external assets](https://docs.dagster.io/guides/build/assets/external-assets) for these tables. from dagster import AssetSpeciris_harvest_data = AssetSpec(key="iris_harvest_data") In this example, you're creating a [`AssetSpec`](https://docs.dagster.io/api/dagster/assets#dagster.AssetSpec) for a pre-existing table containing iris harvests data. To make the data available to other Dagster assets, you need to tell the DuckDB I/O manager how to find the data. Because you already supplied the database and schema in the I/O manager configuration in [Step 1: Configure the DuckDB I/O manager](https://docs.dagster.io/integrations/libraries/duckdb/using-duckdb-with-dagster#step-1-configure-the-duckdb-io-manager) , you only need to provide the table name. This is done with the `key` parameter in `AssetSpec`. When the I/O manager needs to load the `iris_harvest_data` in a downstream asset, it will select the data in the `IRIS.IRIS_HARVEST_DATA` table as a Pandas DataFrame and provide it to the downstream asset. ### Step 3: Load DuckDB tables in downstream assets[​](https://docs.dagster.io/integrations/libraries/duckdb/using-duckdb-with-dagster#step-3-load-duckdb-tables-in-downstream-assets "Direct link to Step 3: Load DuckDB tables in downstream assets") Once you have created an asset that represents a table in DuckDB, you will likely want to create additional assets that work with the data. Dagster and the DuckDB I/O manager allow you to load the data stored in DuckDB tables into downstream assets. import pandas as pdfrom dagster import asset# this example uses the iris_dataset asset from Step 2@assetdef iris_setosa(iris_dataset: pd.DataFrame) -> pd.DataFrame: return iris_dataset[iris_dataset["species"] == "Iris-setosa"] In this asset, you're providing the `iris_dataset` asset as a dependency to `iris_setosa`. By supplying `iris_dataset` as a parameter to `iris_setosa`, Dagster knows to use the `DuckDBPandasIOManager` to load this asset into memory as a Pandas DataFrame and pass it as an argument to `iris_setosa`. Next, a DataFrame that only contains the data for the _Iris Setosa_ species is created and returned. Then the `DuckDBPandasIOManager` will store the DataFrame as the `IRIS.IRIS_SETOSA` table in DuckDB. ### Completed code example[​](https://docs.dagster.io/integrations/libraries/duckdb/using-duckdb-with-dagster#completed-code-example-1 "Direct link to Completed code example") When finished, your code should look like the following: import pandas as pdfrom dagster_duckdb_pandas import DuckDBPandasIOManagerfrom dagster import AssetSpec, Definitions, assetiris_harvest_data = AssetSpec(key="iris_harvest_data")@assetdef iris_dataset() -> pd.DataFrame: return pd.read_csv( "https://docs.dagster.io/assets/iris.csv", names=[ "sepal_length_cm", "sepal_width_cm", "petal_length_cm", "petal_width_cm", "species", ], )@assetdef iris_setosa(iris_dataset: pd.DataFrame) -> pd.DataFrame: return iris_dataset[iris_dataset["species"] == "Iris-setosa"]defs = Definitions( assets=[iris_dataset, iris_harvest_data, iris_setosa], resources={ "io_manager": DuckDBPandasIOManager( database="path/to/my_duckdb_database.duckdb", schema="IRIS", ) },) Related[​](https://docs.dagster.io/integrations/libraries/duckdb/using-duckdb-with-dagster#related "Direct link to Related") ----------------------------------------------------------------------------------------------------------------------------- For more DuckDB features, refer to the [DuckDB reference](https://docs.dagster.io/integrations/libraries/duckdb/reference) . For more information on asset definitions, see the [Assets documentation](https://docs.dagster.io/guides/build/assets) . For more information on I/O managers, see the [I/O manager documentation](https://docs.dagster.io/guides/build/io-managers) . * [Prerequisites](https://docs.dagster.io/integrations/libraries/duckdb/using-duckdb-with-dagster#prerequisites) * [Option 1: Using the DuckDB resource](https://docs.dagster.io/integrations/libraries/duckdb/using-duckdb-with-dagster#option-1-using-the-duckdb-resource) * [Step 1: Configure the DuckDB resource](https://docs.dagster.io/integrations/libraries/duckdb/using-duckdb-with-dagster#step-1-configure-the-duckdb-resource) * [Step 2: Create tables in DuckDB](https://docs.dagster.io/integrations/libraries/duckdb/using-duckdb-with-dagster#option-1-step-2) * [Step 3: Define downstream assets](https://docs.dagster.io/integrations/libraries/duckdb/using-duckdb-with-dagster#step-3-define-downstream-assets) * [Completed code example](https://docs.dagster.io/integrations/libraries/duckdb/using-duckdb-with-dagster#completed-code-example) * [Option 2: Using the DuckDB I/O manager](https://docs.dagster.io/integrations/libraries/duckdb/using-duckdb-with-dagster#option-2-using-the-duckdb-io-manager) * [Step 1: Configure the DuckDB I/O manager](https://docs.dagster.io/integrations/libraries/duckdb/using-duckdb-with-dagster#step-1-configure-the-duckdb-io-manager) * [Step 2: Create tables in DuckDB](https://docs.dagster.io/integrations/libraries/duckdb/using-duckdb-with-dagster#option-2-step-2) * [Store a Dagster asset as a table in DuckDB](https://docs.dagster.io/integrations/libraries/duckdb/using-duckdb-with-dagster#store-a-dagster-asset-as-a-table-in-duckdb) * [Step 3: Load DuckDB tables in downstream assets](https://docs.dagster.io/integrations/libraries/duckdb/using-duckdb-with-dagster#step-3-load-duckdb-tables-in-downstream-assets) * [Completed code example](https://docs.dagster.io/integrations/libraries/duckdb/using-duckdb-with-dagster#completed-code-example-1) * [Related](https://docs.dagster.io/integrations/libraries/duckdb/using-duckdb-with-dagster#related) --- # Dagster & Hex | Dagster Docs [Skip to main content](https://docs.dagster.io/integrations/libraries/hex#__docusaurus_skipToContent_fallback) On this page The community-supported Hex package provides an integration with Hex. For more information, see the [Dagster Community Integrations GitHub repository](https://github.com/dagster-io/community-integrations/tree/main/libraries/dagster-hex) . Installation[​](https://docs.dagster.io/integrations/libraries/hex#installation "Direct link to Installation") --------------------------------------------------------------------------------------------------------------- * uv * pip uv add dagster-hex pip install dagster-hex * [Installation](https://docs.dagster.io/integrations/libraries/hex#installation) --- # Dagster & Hightouch | Dagster Docs [Skip to main content](https://docs.dagster.io/integrations/libraries/hightouch#__docusaurus_skipToContent_fallback) On this page With this integration you can trigger Hightouch syncs and monitor them from within Dagster. Fine-tune when Hightouch syncs kick-off, visualize their dependencies, and monitor the steps in your data activation workflow. This native integration helps your team more effectively orchestrate the last mile of data analytics—bringing that data from the warehouse back into the SaaS tools your business teams live in. With the `dagster-hightouch` integration, Hightouch users have more granular and sophisticated control over when data gets activated. Installation[​](https://docs.dagster.io/integrations/libraries/hightouch#installation "Direct link to Installation") --------------------------------------------------------------------------------------------------------------------- * uv * pip uv add dagster-hightouch pip install dagster-hightouch Example[​](https://docs.dagster.io/integrations/libraries/hightouch#example "Direct link to Example") ------------------------------------------------------------------------------------------------------ import osfrom dagster_hightouch.ops import hightouch_sync_opfrom dagster_hightouch.resources import ht_resourceimport dagster as dgHT_ORG = "39619"run_ht_sync_orgs = hightouch_sync_op.configured( {"sync_id": HT_ORG}, name="hightouch_sfdc_organizations")@dg.jobdef ht_sfdc_job(): ht_orgs = run_ht_sync_orgs()defs = dg.Definitions( jobs=[ht_sfdc_job], resources={ "hightouch": ht_resource.configured( {"api_key": dg.EnvVar("HIGHTOUCH_API_KEY")}, ), },) About Hightouch[​](https://docs.dagster.io/integrations/libraries/hightouch#about-hightouch "Direct link to About Hightouch") ------------------------------------------------------------------------------------------------------------------------------ **Hightouch** syncs data from any data warehouse into popular SaaS tools that businesses run on. Hightouch uses the power of Reverse ETL to transform core business applications from isolated data islands into powerful integrated solutions. * [Installation](https://docs.dagster.io/integrations/libraries/hightouch#installation) * [Example](https://docs.dagster.io/integrations/libraries/hightouch#example) * [About Hightouch](https://docs.dagster.io/integrations/libraries/hightouch#about-hightouch) --- # Importing a dbt project to Dagster+ Serverless | Dagster Docs [Skip to main content](https://docs.dagster.io/integrations/libraries/dbt/using-dbt-with-dagster-plus/serverless#__docusaurus_skipToContent_fallback) Importing an existing dbt project in Dagster+ allows you to automatically load your dbt models as Dagster assets. This can be be done with: * An existing dbt project that is not already using Dagster, or * A Dagster project in which your dbt project is included tip To create a Dagster project with an integrated dbt project, see "[Creating a dbt project in a Dagster project](https://docs.dagster.io/integrations/libraries/dbt/creating-a-dbt-project-in-dagster) ". --- # Dagster & Meltano | Dagster Docs [Skip to main content](https://docs.dagster.io/integrations/libraries/meltano#__docusaurus_skipToContent_fallback) On this page The Meltano library allows you to run Meltano using Dagster. Design and configure ingestion jobs using the popular Singer specification. **Note** that this integration can also be [managed from the Meltano platform](https://hub.meltano.com/utilities/dagster) using `meltano add utility dagster` and configured using `meltano config dagster set --interactive`. Installation[​](https://docs.dagster.io/integrations/libraries/meltano#installation "Direct link to Installation") ------------------------------------------------------------------------------------------------------------------- * uv * pip uv add dagster-meltano pip install dagster-meltano Example[​](https://docs.dagster.io/integrations/libraries/meltano#example "Direct link to Example") ---------------------------------------------------------------------------------------------------- from dagster_meltano import meltano_resource, meltano_run_opimport dagster as dg@dg.job(resource_defs={"meltano": meltano_resource})def meltano_run_job(): tap_done = meltano_run_op("tap-1 target-1")() meltano_run_op("tap-2 target-2")(tap_done)defs = dg.Definitions(jobs=[meltano_run_job]) About Meltano[​](https://docs.dagster.io/integrations/libraries/meltano#about-meltano "Direct link to About Meltano") ---------------------------------------------------------------------------------------------------------------------- [Meltano](https://meltano.com/) provides data engineers with a set of tools for easily creating and managing pipelines as code by providing a wide array of composable connectors. Meltano's 'CLI for ELT+' lets you test your changes before they go live. * [Installation](https://docs.dagster.io/integrations/libraries/meltano#installation) * [Example](https://docs.dagster.io/integrations/libraries/meltano#example) * [About Meltano](https://docs.dagster.io/integrations/libraries/meltano#about-meltano) --- # Dagster & LakeFS | Dagster Docs [Skip to main content](https://docs.dagster.io/integrations/libraries/lakefs#__docusaurus_skipToContent_fallback) On this page By integrating with lakeFS, a big data scale version control system, you can leverage the versioning capabilities of lakeFS to track changes to your data. This integration allows you to have a complete lineage of your data, from the initial raw data to the transformed and processed data, making it easier to understand and reproduce data transformations. With lakeFS and Dagster integration, you can ensure that data flowing through your Dagster jobs is easily reproducible. lakeFS provides a consistent view of your data across different versions, allowing you to troubleshoot pipeline runs and ensure consistent results. Furthermore, with lakeFS branching capabilities, Dagster jobs can run on separate branches without additional storage costs, creating isolation and allowing promotion of only high-quality data to production leveraging a CI/CD pipeline for your data. Installation[​](https://docs.dagster.io/integrations/libraries/lakefs#installation "Direct link to Installation") ------------------------------------------------------------------------------------------------------------------ * uv * pip uv add lakefs-client pip install lakefs-client Example[​](https://docs.dagster.io/integrations/libraries/lakefs#example "Direct link to Example") --------------------------------------------------------------------------------------------------- import lakefs_clientfrom lakefs_client import modelsfrom lakefs_client.client import LakeFSClientimport dagster as dglogger = dg.get_dagster_logger()configuration = lakefs_client.Configuration()configuration.username = "AAAA"configuration.password = "BBBBB"configuration.host = "https://my-org.us-east-1.lakefscloud.io"@dg.assetdef create_branch(client: dg.ResourceParam[LakeFSClient]): branch_id = client.branches.create_branch( repository="test-repo", branch_creation=models.BranchCreation(name="experiment", source="main"), ) logger.info(branch_id)@dg.asset(deps=[create_branch])def list_branches(client: dg.ResourceParam[LakeFSClient]): list_branches = client.branches.list_branches(repository="test-repo") logger.info(list_branches)defs = dg.Definitions( assets=[create_branch, list_branches], resources={"client": LakeFSClient(configuration)},) About lakeFS[​](https://docs.dagster.io/integrations/libraries/lakefs#about-lakefs "Direct link to About lakeFS") ------------------------------------------------------------------------------------------------------------------ **lakeFS** is on a mission to simplify the lives of data engineers, data scientists and analysts providing a data version control platform at scale. * [Installation](https://docs.dagster.io/integrations/libraries/lakefs#installation) * [Example](https://docs.dagster.io/integrations/libraries/lakefs#example) * [About lakeFS](https://docs.dagster.io/integrations/libraries/lakefs#about-lakefs) --- # Dagster & GCP Cloud Run | Dagster Docs [Skip to main content](https://docs.dagster.io/integrations/libraries/gcp/cloud-run-launcher#__docusaurus_skipToContent_fallback) The community-supported dagster-contrib-gcp package provides integrations with Google Cloud Platform (GCP) services. It currently includes the following integrations: * Google Cloud Run For more information, see the [Dagster Community Integrations GitHub repository](https://github.com/dagster-io/community-integrations/tree/main/libraries/dagster-contrib-gcp) . --- # Features | Dagster Docs [Skip to main content](https://docs.dagster.io/integrations/libraries/iceberg/features#__docusaurus_skipToContent_fallback) On this page Support for Iceberg features depends upon the execution engine you choose. Using Spark[​](https://docs.dagster.io/integrations/libraries/iceberg/features#using-spark "Direct link to Using Spark") ------------------------------------------------------------------------------------------------------------------------- Spark is currently the most feature-rich compute engine for Iceberg operations. ### Configuration[​](https://docs.dagster.io/integrations/libraries/iceberg/features#configuration "Direct link to Configuration") Spark configuration can be set directly on the [`io_manager.spark.SparkIcebergIOManager`](https://docs.dagster.io/api/libraries/dagster-iceberg#dagster_iceberg.io_manager.spark.SparkIcebergIOManager) or in the `spark-defaults.conf` file. Properties set directly on the I/O manager take precedence over those set in the `spark-defaults.conf` file. To set properties directly, pass a dictionary of configurations to set in the `spark_config` argument of the I/O manager: from dagster_iceberg.io_manager.spark import SparkIcebergIOManagerSPARK_CONFIG = { "spark.sql.extensions": "org.apache.iceberg.spark.extensions.IcebergSparkSessionExtensions", "spark.sql.catalog.postgres": "org.apache.iceberg.spark.SparkCatalog", "spark.sql.catalog.postgres.type": "jdbc", "spark.sql.catalog.postgres.uri": "jdbc:postgresql://postgres:5432/test", "spark.sql.catalog.postgres.jdbc.user": "test", "spark.sql.catalog.postgres.jdbc.password": "test", "spark.sql.catalog.postgres.warehouse": "/home/iceberg/warehouse", "spark.sql.defaultCatalog": "postgres", "spark.eventLog.enabled": "true", "spark.eventLog.dir": "/home/iceberg/spark-events", "spark.history.fs.logDirectory": "/home/iceberg/spark-events", "spark.sql.catalogImplementation": "in-memory", "spark.sql.execution.arrow.pyspark.enabled": "true",}io_manager = SparkIcebergIOManager( catalog_name="test", namespace="dagster", spark_config=SPARK_CONFIG, remote_url="sc://localhost",) Using PyIceberg[​](https://docs.dagster.io/integrations/libraries/iceberg/features#using-pyiceberg "Direct link to Using PyIceberg") ------------------------------------------------------------------------------------------------------------------------------------- The following engines are implemented using PyIceberg: * [Apache Arrow](https://arrow.apache.org/docs/python/index.html) * [Daft](https://www.getdaft.io/) * [pandas](https://pandas.pydata.org/) * [Polars](https://pola.rs/) ### Supported catalogs[​](https://docs.dagster.io/integrations/libraries/iceberg/features#supported-catalogs "Direct link to Supported catalogs") `dagster-iceberg` supports [all catalogs available through PyIceberg](https://py.iceberg.apache.org/configuration/#catalogs) . ### Configuration[​](https://docs.dagster.io/integrations/libraries/iceberg/features#configuration-1 "Direct link to Configuration") `dagster-iceberg` supports setting configuration values using a `.pyiceberg.yaml` configuration file and environment variables. For more information, see the [PyIceberg documentation](https://py.iceberg.apache.org/configuration/#setting-configuration-values) . You can also pass your catalog configuration using [`config.IcebergCatalogConfig`](https://docs.dagster.io/api/libraries/dagster-iceberg#dagster_iceberg.config.IcebergCatalogConfig) : from dagster_iceberg.config import IcebergCatalogConfigfrom dagster_iceberg.io_manager.arrow import PyArrowIcebergIOManagerio_manager = PyArrowIcebergIOManager( name="test", config=IcebergCatalogConfig( properties={ "uri": "postgresql+psycopg2://test:test@localhost:5432/test", "warehouse": "file:///path/to/warehouse", } ), namespace="dagster",) ### PyIceberg features[​](https://docs.dagster.io/integrations/libraries/iceberg/features#pyiceberg-features "Direct link to PyIceberg features") The table below indicates PyIceberg features are currently available in `dagster-iceberg`: | Feature | Supported | Link | Comment | | --- | --- | --- | --- | | Add existing files | ❌ | [https://py.iceberg.apache.org/api/#add-files](https://py.iceberg.apache.org/api/#add-files) | Useful for existing partitions that users don't want to re-materialize/re-compute. | | Schema evolution | ✅ | [https://py.iceberg.apache.org/api/#schema-evolution](https://py.iceberg.apache.org/api/#schema-evolution) | More complicated than e.g. delta lake since updates require diffing input table with existing Iceberg table. This is implemented by checking the schema of incoming data, dropping any columns that no longer exist in the data schema, and then using the `union_by_name()` method to merge the current schema with the table schema. Current implementation has a chance of creating a race condition when e.g. partition A tries to write to a table that has not yet processed a schema update. Should be covered by retrying when writing. | | Sort order | ❌ | [https://shorturl.at/TycZN](https://shorturl.at/TycZN) | Currently limited support in PyIceberg. Sort ordering is supported when creating a table from an Iceberg schema (one must pass the source\_id which can be inferred from a PyArrow schema but this is shaky). However, we cannot simply update a sort ordering like a partition or schema spec. | | PyIceberg commit retries | ✅ | [https://github.com/apache/iceberg-python/pull/330](https://github.com/apache/iceberg-python/pull/330)
[https://github.com/apache/iceberg-python/issues/269](https://github.com/apache/iceberg-python/issues/269) | PR to add this to PyIceberg is open. Will probably be merged for an upcoming release. Added a custom retry function using Tenacity for the time being. | | Partition evolution | ✅ | [https://py.iceberg.apache.org/api/#partition-evolution](https://py.iceberg.apache.org/api/#partition-evolution) | Create, Update, Delete partitions by updating the Dagster partitions definition. | | Table properties | ✅ | [https://py.iceberg.apache.org/api/#table-properties](https://py.iceberg.apache.org/api/#table-properties) | Added as metadata on an asset. NB: config options are not checked explicitly because users can add any key-value pair to a table. Available properties [here](https://py.iceberg.apache.org/configuration/#tables)
. | | Snapshot properties | ✅ | [https://py.iceberg.apache.org/api/#snapshot-properties](https://py.iceberg.apache.org/api/#snapshot-properties) | Useful for correlating Dagster runs to snapshots by adding tags to snapshot. Not configurable by end-user. | * [Using Spark](https://docs.dagster.io/integrations/libraries/iceberg/features#using-spark) * [Configuration](https://docs.dagster.io/integrations/libraries/iceberg/features#configuration) * [Using PyIceberg](https://docs.dagster.io/integrations/libraries/iceberg/features#using-pyiceberg) * [Supported catalogs](https://docs.dagster.io/integrations/libraries/iceberg/features#supported-catalogs) * [Configuration](https://docs.dagster.io/integrations/libraries/iceberg/features#configuration-1) * [PyIceberg features](https://docs.dagster.io/integrations/libraries/iceberg/features#pyiceberg-features) --- # Quickstart | Dagster Docs [Skip to main content](https://docs.dagster.io/integrations/libraries/iceberg/quickstart#__docusaurus_skipToContent_fallback) On this page Dagster supports saving and loading Iceberg tables as assets using I/O managers. Prerequisites To follow the steps in this guide, you'll need to: * [Install `dagster-iceberg`](https://docs.dagster.io/integrations/libraries/iceberg#installation) . * [Create a temporary location for Iceberg and set up the catalog](https://py.iceberg.apache.org/#connecting-to-a-catalog) . * Create the `default` namespace: catalog.create_namespace("default") Defining the I/O manager[​](https://docs.dagster.io/integrations/libraries/iceberg/quickstart#defining-the-io-manager "Direct link to Defining the I/O manager") ----------------------------------------------------------------------------------------------------------------------------------------------------------------- To use an Iceberg I/O manager, add it to your [`Definitions`](https://docs.dagster.io/api/dagster/definitions#dagster.Definitions) : from dagster_iceberg.config import IcebergCatalogConfigfrom dagster_iceberg.io_manager.arrow import PyArrowIcebergIOManagerfrom dagster import Definitionswarehouse_path = "/tmp/warehouse"resources = { "io_manager": PyArrowIcebergIOManager( name="default", config=IcebergCatalogConfig( properties={ "type": "sql", "uri": f"sqlite:///{warehouse_path}/pyiceberg_catalog.db", "warehouse": f"file://{warehouse_path}", } ), namespace="default", )}defs = Definitions(assets=[my_table, my_table_with_year], resources=resources) Storing a Dagster asset as an Iceberg table[​](https://docs.dagster.io/integrations/libraries/iceberg/quickstart#storing-a-dagster-asset-as-an-iceberg-table "Direct link to Storing a Dagster asset as an Iceberg table") --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- The I/O manager will automatically persist the returned data to your warehouse: import pyarrow as pafrom dagster import asset@assetdef my_table() -> pa.Table: n_legs = pa.array([2, 4, 5, 100]) animals = pa.array(["Flamingo", "Horse", "Brittle stars", "Centipede"]) names = ["n_legs", "animals"] return pa.Table.from_arrays([n_legs, animals], names=names) Loading Iceberg tables in downstream assets[​](https://docs.dagster.io/integrations/libraries/iceberg/quickstart#loading-iceberg-tables-in-downstream-assets "Direct link to Loading Iceberg tables in downstream assets") --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- The I/O manager will also load the data stored in your warehouse when referenced in a dependent asset: import pyarrow as pafrom dagster import asset@assetdef my_table_with_year(my_table: pa.Table) -> pa.Table: year = [2021, 2022, 2019, 2021] return my_table.append_column("year", [year]) * [Defining the I/O manager](https://docs.dagster.io/integrations/libraries/iceberg/quickstart#defining-the-io-manager) * [Storing a Dagster asset as an Iceberg table](https://docs.dagster.io/integrations/libraries/iceberg/quickstart#storing-a-dagster-asset-as-an-iceberg-table) * [Loading Iceberg tables in downstream assets](https://docs.dagster.io/integrations/libraries/iceberg/quickstart#loading-iceberg-tables-in-downstream-assets) --- # Dagster & Modal | Dagster Docs [Skip to main content](https://docs.dagster.io/integrations/libraries/modal#__docusaurus_skipToContent_fallback) On this page The community-supported Modal package provides an integration with Modal. For more information, see the [Dagster Community Integrations GitHub repository](https://github.com/dagster-io/community-integrations/tree/main/libraries/dagster-modal) . Installation[​](https://docs.dagster.io/integrations/libraries/modal#installation "Direct link to Installation") ----------------------------------------------------------------------------------------------------------------- * uv * pip uv add dagster-modal pip install dagster-modal * [Installation](https://docs.dagster.io/integrations/libraries/modal#installation) --- # Dagster & MSSQL Bulk Copy Tool | Dagster Docs [Skip to main content](https://docs.dagster.io/integrations/libraries/mssql-bulk-copy-tool#__docusaurus_skipToContent_fallback) On this page The community-supported MSSQL BCP package is a custom Dagster I/O manager for loading data into SQL Server using the BCP utility. For more information, see the [dagster-mssql-bcp GitHub repository](https://github.com/cody-scott/dagster-mssql-bcp) . Installation[​](https://docs.dagster.io/integrations/libraries/mssql-bulk-copy-tool#installation "Direct link to Installation") -------------------------------------------------------------------------------------------------------------------------------- * uv * pip uv add dagster-mssql-bcp pip install dagster-mssql-bcp * [Installation](https://docs.dagster.io/integrations/libraries/mssql-bulk-copy-tool#installation) --- # Dagster & PagerDuty | Dagster Docs [Skip to main content](https://docs.dagster.io/integrations/libraries/pagerduty#__docusaurus_skipToContent_fallback) On this page This library provides an integration between Dagster and PagerDuty to support creating alerts from your Dagster code. Installation[​](https://docs.dagster.io/integrations/libraries/pagerduty#installation "Direct link to Installation") --------------------------------------------------------------------------------------------------------------------- * uv * pip uv add dagster-pagerduty pip install dagster-pagerduty Example[​](https://docs.dagster.io/integrations/libraries/pagerduty#example "Direct link to Example") ------------------------------------------------------------------------------------------------------ from dagster_pagerduty import PagerDutyServiceimport dagster as dg@dg.assetdef pagerduty_alert(pagerduty: PagerDutyService): pagerduty.EventV2_create( summary="alert from dagster", source="localhost", severity="error", event_action="trigger", )defs = dg.Definitions( assets=[pagerduty_alert], resources={ "pagerduty": PagerDutyService(routing_key="0123456789abcdef0123456789abcdef") },) About PagerDuty[​](https://docs.dagster.io/integrations/libraries/pagerduty#about-pagerduty "Direct link to About PagerDuty") ------------------------------------------------------------------------------------------------------------------------------ **PagerDuty** is a popular SaaS incident response platform. It integrates machine data & human intelligence to improve visibility & agility for Real-Time Operations. * [Installation](https://docs.dagster.io/integrations/libraries/pagerduty#installation) * [Example](https://docs.dagster.io/integrations/libraries/pagerduty#example) * [About PagerDuty](https://docs.dagster.io/integrations/libraries/pagerduty#about-pagerduty) --- # Dagster & Not Diamond | Dagster Docs [Skip to main content](https://docs.dagster.io/integrations/libraries/notdiamond#__docusaurus_skipToContent_fallback) On this page Leverage the Not Diamond resource to easily determine which LLM provider is most appropriate for your use case. Installation[​](https://docs.dagster.io/integrations/libraries/notdiamond#installation "Direct link to Installation") ---------------------------------------------------------------------------------------------------------------------- * uv * pip uv add dagster-notdiamond pip install dagster-notdiamond Example[​](https://docs.dagster.io/integrations/libraries/notdiamond#example "Direct link to Example") ------------------------------------------------------------------------------------------------------- import timeimport dagster as dgimport dagster_notdiamond as ndimport dagster_openai as oai@dg.asset(kinds={"python"})def book_review_data(context: dg.AssetExecutionContext) -> dict: data = { "title": "Cat's Cradle", "author": "Kurt Vonnegut", "genre": "Science Fiction", "publicationYear": 1963, "reviews": [ { "reviewer": "John Doe", "rating": 4.5, "content": "A thought-provoking satire on science and religion. Vonnegut's wit shines through.", }, { "reviewer": "Jane Smith", "rating": 5, "content": "An imaginative and darkly humorous exploration of humanity's follies. A must-read!", }, { "reviewer": "Alice Johnson", "rating": 3.5, "content": "Intriguing premise but felt a bit disjointed at times. Still enjoyable.", }, ], } context.add_output_metadata(metadata={"num_reviews": len(data.get("reviews", []))}) return data@dg.asset( kinds={"openai", "notdiamond"}, automation_condition=dg.AutomationCondition.eager())def book_reviews_summary( context: dg.AssetExecutionContext, notdiamond: nd.NotDiamondResource, openai: oai.OpenAIResource, book_review_data: dict,) -> dg.MaterializeResult: prompt = f""" Given the book reviews for {book_review_data["title"]}, provide a detailed summary: {'|'.join([r['content'] for r in book_review_data["reviews"]])} """ with notdiamond.get_client(context) as client: start = time.time() session_id, best_llm = client.model_select( model=["openai/gpt-4o", "openai/gpt-4o-mini"], tradeoff="cost", messages=[ {"role": "system", "content": "You are an expert in literature"}, {"role": "user", "content": prompt}, ], ) duration = time.time() - start with openai.get_client(context) as client: chat_completion = client.chat.completions.create( model=best_llm.model, messages=[{"role": "user", "content": prompt}], ) summary = chat_completion.choices[0].message.content or "" return dg.MaterializeResult( metadata={ "nd_session_id": session_id, "nd_best_llm_model": best_llm.model, "nd_best_llm_provider": best_llm.provider, "nd_routing_latency": duration, "summary": dg.MetadataValue.md(summary), } )defs = dg.Definitions( assets=[book_review_data, book_reviews_summary], resources={ "notdiamond": nd.NotDiamondResource(api_key=dg.EnvVar("NOTDIAMOND_API_KEY")), "openai": oai.OpenAIResource(api_key=dg.EnvVar("OPENAI_API_KEY")), },) About Not Diamond[​](https://docs.dagster.io/integrations/libraries/notdiamond#about-not-diamond "Direct link to About Not Diamond") ------------------------------------------------------------------------------------------------------------------------------------- [Not Diamond](https://www.notdiamond.ai/) is a service that recommends the best model for every query, improving accuracy and reducing costs. It can train your own router with your evaluation data and support joint prompt optimization. * [Installation](https://docs.dagster.io/integrations/libraries/notdiamond#installation) * [Example](https://docs.dagster.io/integrations/libraries/notdiamond#example) * [About Not Diamond](https://docs.dagster.io/integrations/libraries/notdiamond#about-not-diamond) --- # Dagster & Rust | Dagster Docs [Skip to main content](https://docs.dagster.io/integrations/libraries/rust#__docusaurus_skipToContent_fallback) On this page The Rust Pipes client allows full observability into your Rust workloads when orchestrating through Dagster. For more information, see the [community integrations GitHub repository](https://github.com/dagster-io/community-integrations/blob/main/libraries/pipes/implementations/rust/README.md) . Installation[​](https://docs.dagster.io/integrations/libraries/rust#installation "Direct link to Installation") ---------------------------------------------------------------------------------------------------------------- `cargo add dagster_pipes_rust` Example[​](https://docs.dagster.io/integrations/libraries/rust#example "Direct link to Example") ------------------------------------------------------------------------------------------------- For a usage example, see the [community integrations GitHub repository](https://github.com/dagster-io/community-integrations/blob/main/libraries/pipes/implementations/rust/README.md#example) . About Rust[​](https://docs.dagster.io/integrations/libraries/rust#about-rust "Direct link to About Rust") ---------------------------------------------------------------------------------------------------------- [Rust](https://www.rust-lang.org/) is a language empowering everyone to build reliable and efficient software. To get started with Rust, see the [Rust docs](https://www.rust-lang.org/learn) . * [Installation](https://docs.dagster.io/integrations/libraries/rust#installation) * [Example](https://docs.dagster.io/integrations/libraries/rust#example) * [About Rust](https://docs.dagster.io/integrations/libraries/rust#about-rust) --- # Dagster & GCP GCS | Dagster Docs [Skip to main content](https://docs.dagster.io/integrations/libraries/gcp/gcs#__docusaurus_skipToContent_fallback) On this page This integration allows you to interact with Google Cloud Storage (GCS) using Dagster. It provides resources, I/O Managers, and utilities to manage and store data in GCS, making it easier to integrate GCS into your data pipelines. Installation[​](https://docs.dagster.io/integrations/libraries/gcp/gcs#installation "Direct link to Installation") ------------------------------------------------------------------------------------------------------------------- * uv * pip uv add dagster-gcp pip install dagster-gcp Examples[​](https://docs.dagster.io/integrations/libraries/gcp/gcs#examples "Direct link to Examples") ------------------------------------------------------------------------------------------------------- import pandas as pdfrom dagster_gcp.gcs import GCSResourceimport dagster as dg@dg.assetdef my_gcs_asset(gcs: GCSResource): df = pd.DataFrame({"column1": [1, 2, 3], "column2": ["A", "B", "C"]}) csv_data = df.to_csv(index=False) gcs_client = gcs.get_client() bucket = gcs_client.bucket("my-cool-bucket") blob = bucket.blob("path/to/my_dataframe.csv") blob.upload_from_string(csv_data)defs = dg.Definitions( assets=[my_gcs_asset], resources={"gcs": GCSResource(project="my-gcp-project")},) About Google Cloud Platform GCS[​](https://docs.dagster.io/integrations/libraries/gcp/gcs#about-google-cloud-platform-gcs "Direct link to About Google Cloud Platform GCS") ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- **Google Cloud Storage (GCS)**, is a scalable and secure object storage service. GCS is designed for storing and accessing any amount of data at any time, making it ideal for data science, AI infrastructure, and frameworks for ML like AutoML. With this integration, you can leverage GCS for efficient data storage and retrieval within your Dagster pipelines. * [Installation](https://docs.dagster.io/integrations/libraries/gcp/gcs#installation) * [Examples](https://docs.dagster.io/integrations/libraries/gcp/gcs#examples) * [About Google Cloud Platform GCS](https://docs.dagster.io/integrations/libraries/gcp/gcs#about-google-cloud-platform-gcs) --- # Dagster & Perian | Dagster Docs [Skip to main content](https://docs.dagster.io/integrations/libraries/perian#__docusaurus_skipToContent_fallback) On this page The Perian integration allows you to easily dockerize your codebase and execute it on the PERIAN platform, PERIAN's serverless GPU environment. For more information, please visit the [dagster-perian GitHub repository](https://github.com/Perian-io/dagster-perian) and the [PERIAN documentation](https://perian.io/docs) . Installation[​](https://docs.dagster.io/integrations/libraries/perian#installation "Direct link to Installation") ------------------------------------------------------------------------------------------------------------------ * uv * pip uv add dagster-perian pip install dagster-perian * [Installation](https://docs.dagster.io/integrations/libraries/perian#installation) --- # Dagster & obstore | Dagster Docs [Skip to main content](https://docs.dagster.io/integrations/libraries/obstore#__docusaurus_skipToContent_fallback) On this page The community-supported obstore package provides an integration with obstore, providing three lean integrations with object stores, ADLS, GCS & S3. S3ComputeLogManager[​](https://docs.dagster.io/integrations/libraries/obstore#s3computelogmanager "Direct link to S3ComputeLogManager") ---------------------------------------------------------------------------------------------------------------------------------------- The `S3ComputeLogManager` writes `stdout` and `stderr` to any S3 compatible endpoint. # there are multiples ways to configure the S3ComputeLogManager# Explicitly set S3 secretscompute_logs: module: dagster_obstore.s3.compute_log_manager class: S3ComputeLogManager config: bucket: "dagster-logs" access_key_id: env: ACCESS_KEY_ID secret_access_key: env: SECRET_KEY local_dir: "/tmp/dagster-logs" allow_http: false allow_invalid_certificates: false timeout: "60s" # Timeout for obstore requests region: "us-west-1"# Use S3 with custom endpoint (Minio, Cloudflare R2 etc.)compute_logs: module: dagster_obstore.s3.compute_log_manager class: S3ComputeLogManager config: bucket: "dagster-logs" access_key_id: "access-key-id" secret_access_key: "my-key" local_dir: "/tmp/dagster-logs" endpoint: "http://alternate-s3-host.io" region: "us-west-1"# Don't set secrets through config, but let obstore pick it up from ENV VARScompute_logs: module: dagster_obstore.s3.compute_log_manager class: S3ComputeLogManager config: bucket: "dagster-logs" local_dir: "/tmp/dagster-logs" ADLSComputeLogManager[​](https://docs.dagster.io/integrations/libraries/obstore#adlscomputelogmanager "Direct link to ADLSComputeLogManager") ---------------------------------------------------------------------------------------------------------------------------------------------- The `ADLSComputeLogManager` writes `stdout` and `stderr` to Azure Datalake/Blob storage. # there are multiples ways to configure the ADLSComputeLogManager# Authenticate with access keycompute_logs: module: dagster_obstore.azure.compute_log_manager class: ADLSComputeLogManager config: storage_account: "my-az-account" container: "dagster-logs" access_key: env: ACCESS_KEY local_dir: "/tmp/dagster-logs" allow_http: false allow_invalid_certificates: false timeout: "60s" # Timeout for obstore requests# Authenticate with service principalcompute_logs: module: dagster_obstore.azure.compute_log_manager class: ADLSComputeLogManager config: storage_account: "my-az-account" container: "dagster-logs" client_id: "access-key-id" client_secret: "my-key" tenant_id: "tenant-id" local_dir: "/tmp/dagster-logs"# Authenticate with use_azure_clicompute_logs: module: dagster_obstore.azure.compute_log_manager class: ADLSComputeLogManager config: storage_account: "my-az-account" container: "dagster-logs" use_azure_cli: true local_dir: "/tmp/dagster-logs"# Don't set secrets through config, but let obstore pick it up from ENV VARScompute_logs: module: dagster_obstore.azure.compute_log_manager class: ADLSComputeLogManager config: storage_account: "my-az-account" container: "dagster-logs" local_dir: "/tmp/dagster-logs" GCSComputeLogManager[​](https://docs.dagster.io/integrations/libraries/obstore#gcscomputelogmanager "Direct link to GCSComputeLogManager") ------------------------------------------------------------------------------------------------------------------------------------------- The `GCSComputeLogManager` writes `stdout` and `stderr` to Google Cloud Storage. # there are multiples ways to configure the GCSComputeLogManager# Authenticate with service accountcompute_logs: module: dagster_obstore.gcs.compute_log_manager class: GCSComputeLogManager config: bucket: "dagster-logs" service_account: "access-key-id" service_account_key: "my-key" local_dir: "/tmp/dagster-logs"# Don't set secrets through config, but let obstore pick it up from ENV VARScompute_logs: module: dagster_obstore.gcs.compute_log_manager class: GCSComputeLogManager config: bucket: "dagster-logs" local_dir: "/tmp/dagster-logs" * * * For more information, see the [dagster-obstore GitHub repository](https://github.com/dagster-io/community-integrations/tree/main/libraries/dagster-obstore) . * [S3ComputeLogManager](https://docs.dagster.io/integrations/libraries/obstore#s3computelogmanager) * [ADLSComputeLogManager](https://docs.dagster.io/integrations/libraries/obstore#adlscomputelogmanager) * [GCSComputeLogManager](https://docs.dagster.io/integrations/libraries/obstore#gcscomputelogmanager) --- # dagster-dbt integration reference | Dagster Docs [Skip to main content](https://docs.dagster.io/integrations/libraries/dbt/reference#__docusaurus_skipToContent_fallback) On this page note Using dbt Cloud? Check out the [dbt Cloud with Dagster guide](https://docs.dagster.io/integrations/libraries/dbt/dbt-cloud) . This reference provides a high-level look at working with dbt models through Dagster's [software-defined assets](https://docs.dagster.io/guides/build/assets) framework using the [`dagster-dbt` integration library](https://docs.dagster.io/api/libraries/dagster-dbt) . For a step-by-step implementation walkthrough, refer to the [Using dbt with Dagster asset definitions tutorial](https://docs.dagster.io/integrations/libraries/dbt/creating-a-dbt-project-in-dagster) . Relevant APIs[​](https://docs.dagster.io/integrations/libraries/dbt/reference#relevant-apis "Direct link to Relevant APIs") ---------------------------------------------------------------------------------------------------------------------------- | Name | Description | | --- | --- | | [`dagster-dbt project scaffold`](https://docs.dagster.io/api/libraries/dagster-dbt#scaffold) | A CLI command to initialize a new Dagster project for an existing dbt project. | | [`@dagster_dbt.dbt_assets`](https://docs.dagster.io/api/libraries/dagster-dbt#dagster_dbt.dbt_assets) | A decorator used to define Dagster assets for dbt models defined in a dbt manifest. | | [`DbtCliResource`](https://docs.dagster.io/api/libraries/dagster-dbt#dagster_dbt.DbtCliResource) | A class that defines a Dagster resource used to execute dbt CLI commands. | | [`DbtCliInvocation`](https://docs.dagster.io/api/libraries/dagster-dbt#dagster_dbt.DbtCliInvocation) | A class that defines the representation of an invoked dbt command. | | [`DbtProject`](https://docs.dagster.io/api/libraries/dagster-dbt#dagster_dbt.DbtProject) | A class that defines the representation of a dbt project and related settings that assist with managing dependencies and `manifest.json` preparation. | | [`DagsterDbtTranslator`](https://docs.dagster.io/api/libraries/dagster-dbt#dagster_dbt.DagsterDbtTranslator) | A class that can be overridden to customize how Dagster asset metadata is derived from a dbt manifest. | | [`DagsterDbtTranslatorSettings`](https://docs.dagster.io/api/libraries/dagster-dbt#dagster_dbt.DagsterDbtTranslatorSettings) | A class with settings to enable Dagster features for a dbt project. | | [`DbtManifestAssetSelection`](https://docs.dagster.io/api/libraries/dagster-dbt#dagster_dbt.DbtManifestAssetSelection) | A class that defines a selection of assets from a dbt manifest and a dbt selection string. | | [`build_dbt_asset_selection`](https://docs.dagster.io/api/libraries/dagster-dbt#dagster_dbt.build_dbt_asset_selection) | A helper method that builds a [`DbtManifestAssetSelection`](https://docs.dagster.io/api/libraries/dagster-dbt#dagster_dbt.DbtManifestAssetSelection)
from a dbt manifest and dbt selection string. | | [`build_schedule_from_dbt_selection`](https://docs.dagster.io/api/libraries/dagster-dbt#dagster_dbt.build_schedule_from_dbt_selection) | A helper method that builds a [`ScheduleDefinition`](https://docs.dagster.io/api/dagster/schedules-sensors#dagster.ScheduleDefinition)
from a dbt manifest, dbt selection string, and cron string. | | [`get_asset_key_for_model`](https://docs.dagster.io/api/libraries/dagster-dbt#dagster_dbt.get_asset_key_for_model) | A helper method that retrieves the [`AssetKey`](https://docs.dagster.io/api/dagster/assets#dagster.AssetKey)
for a dbt model. | | [`get_asset_key_for_source`](https://docs.dagster.io/api/libraries/dagster-dbt#dagster_dbt.get_asset_key_for_source) | A helper method that retrieves the [`AssetKey`](https://docs.dagster.io/api/dagster/assets#dagster.AssetKey)
for a dbt source with a singular table. | | [`get_asset_keys_by_output_name_for_source`](https://docs.dagster.io/api/libraries/dagster-dbt#dagster_dbt.get_asset_keys_by_output_name_for_source) | A helper method that retrieves the [`AssetKeys`](https://docs.dagster.io/api/dagster/assets#dagster.AssetKey)
for a dbt source with multiple tables. | dbt models and Dagster asset definitions[​](https://docs.dagster.io/integrations/libraries/dbt/reference#dbt-models-and-dagster-asset-definitions "Direct link to dbt models and Dagster asset definitions") ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Dagster’s [asset definitions](https://docs.dagster.io/guides/build/assets) bear several similarities to dbt models. An asset definition contains an asset key, a set of upstream asset keys, and an operation that is responsible for computing the asset from its upstream dependencies. Models defined in a dbt project can be interpreted as Dagster asset definitions: * The asset key for a dbt model is (by default) the name of the model. * The upstream dependencies of a dbt model are defined with `ref` or `source` calls within the model's definition. * The computation required to compute the asset from its upstream dependencies is the SQL within the model's definition. These similarities make it natural to interact with dbt models as asset definitions. Let’s take a look at a dbt model and an asset definition, in code: ![Comparison of a dbt model and Dagster asset in code](https://docs.dagster.io/assets/images/asset-dbt-model-comparison-7b2d780c5a491cf5377d44961aa9d025.png) Here's what's happening in this example: * The first code block is a **dbt model** * As dbt models are named using file names, this model is named `orders` * The data for this model comes from a dependency named `raw_orders` * The second code block is a **Dagster asset** * The asset key corresponds to the name of the dbt model, `orders` * `raw_orders` is provided as an argument to the asset, defining it as a dependency Scaffolding a Dagster project from a dbt project[​](https://docs.dagster.io/integrations/libraries/dbt/reference#scaffolding-a-dagster-project-from-a-dbt-project "Direct link to Scaffolding a Dagster project from a dbt project") ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- note Check out [part two of the dbt & Dagster tutorial](https://docs.dagster.io/integrations/libraries/dbt/creating-a-dbt-project-in-dagster/load-dbt-models) to see this concept in context. You can create a Dagster project that wraps your dbt project by using the [`dagster-dbt project scaffold`](https://docs.dagster.io/api/libraries/dagster-dbt#scaffold) command line interface. dagster-dbt project scaffold --project-name project_dagster --dbt-project-dir path/to/dbt/project This creates a directory called `project_dagster/` inside the current directory. The `project_dagster/` directory contains a set of files that define a Dagster project that loads the dbt project at the path defined by `--dbt-project-dir`. The path to the dbt project must contain a `dbt_project.yml`. Loading dbt models from a dbt project[​](https://docs.dagster.io/integrations/libraries/dbt/reference#loading-dbt-models-from-a-dbt-project "Direct link to Loading dbt models from a dbt project") ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- note Check out [part two of the dbt & Dagster tutorial](https://docs.dagster.io/integrations/libraries/dbt/creating-a-dbt-project-in-dagster/load-dbt-models) to see this concept in context. The `dagster-dbt` library offers [`@dagster_dbt.dbt_assets`](https://docs.dagster.io/api/libraries/dagster-dbt#dagster_dbt.dbt_assets) to define Dagster assets for dbt models. It requires a [dbt manifest](https://docs.getdbt.com/reference/artifacts/manifest-json) , or `manifest.json`, to be created from your dbt project to parse your dbt project's representation. The manifest can be created in two ways: 1. **At run time**: A dbt manifest is generated when your Dagster definitions are loaded, or 2. **At build time**: A dbt manifest is generated before loading your Dagster definitions and is included as part of your Python package. When deploying your Dagster project to production, **we recommend generating the manifest at build time** to avoid the overhead of recompiling your dbt project every time your Dagster code is executed. A `manifest.json` should be precompiled and included in the Python package for your Dagster code. The easiest way to handle the creation of your manifest file is to use [`DbtProject`](https://docs.dagster.io/api/libraries/dagster-dbt#dagster_dbt.DbtProject) . In the Dagster project created by the [`dagster-dbt project scaffold`](https://docs.dagster.io/api/libraries/dagster-dbt#scaffold) command, the creation of your manifest is handled at run time during development: from pathlib import Path from dagster_dbt import DbtProject my_dbt_project = DbtProject( project_dir=Path(__file__).joinpath("..", "..", "..").resolve(), packaged_project_dir=Path(__file__) .joinpath("..", "..", "dbt-project") .resolve(), ) my_dbt_project.prepare_if_dev() The manifest path can then be accessed with `my_dbt_project.manifest_path`. When developing locally, you can run the following command to generate the manifest at run time for your dbt and Dagster project: dagster dev In production, a precompiled manifest should be used. Using [`DbtProject`](https://docs.dagster.io/api/libraries/dagster-dbt#dagster_dbt.DbtProject) , the manifest can be created at build time by running the [`dagster-dbt project prepare-and-package`](https://docs.dagster.io/api/libraries/dagster-dbt#prepare-and-package) command in your CI/CD workflow. For more information, see the [Deploying a Dagster project with a dbt project](https://docs.dagster.io/integrations/libraries/dbt/reference#deploying-a-dagster-project-with-a-dbt-project) section. Selecting a profiles directory, profile and target for your dbt project[​](https://docs.dagster.io/integrations/libraries/dbt/reference#selecting-a-profiles-directory-profile-and-target-for-your-dbt-project "Direct link to Selecting a profiles directory, profile and target for your dbt project") --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- You can specify which connection information dbt should use when parsing and executing your models. This can be done by passing the profiles directory, profile and target when creating your [`DbtProject`](https://docs.dagster.io/api/libraries/dagster-dbt#dagster_dbt.DbtProject) object. These fields are optional - the default values defined in your dbt project will be used for each parameter that is not passed. from pathlib import Path from dagster_dbt import DbtProject my_dbt_project = DbtProject( project_dir=Path(__file__).joinpath("..", "..", "..").resolve(), profiles_dir=Path(__file__) .joinpath("..", "..", "..", "my_profiles_dir") .resolve(), profile="my_profile", target="my_target", ) For more information, see dbt's guide about [connection profiles](https://docs.getdbt.com/docs/core/connect-data-platform/connection-profiles) . Deploying a Dagster project with a dbt project[​](https://docs.dagster.io/integrations/libraries/dbt/reference#deploying-a-dagster-project-with-a-dbt-project "Direct link to Deploying a Dagster project with a dbt project") ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- note **Got questions about our recommendations or something to add?** Join our [GitHub discussion](https://github.com/dagster-io/dagster/discussions/13767) to share how you deploy your Dagster code with your dbt project. When deploying your Dagster project to production, your dbt project must be present alongside your Dagster project so that dbt commands can be executed. As a result, we recommend that you set up your continuous integration and continuous deployment (CI/CD) workflows to package the dbt project with your Dagster project. ### Deploying a dbt project from a separate git repository[​](https://docs.dagster.io/integrations/libraries/dbt/reference#deploying-a-dbt-project-from-a-separate-git-repository "Direct link to Deploying a dbt project from a separate git repository") If you are managing your Dagster project in a separate git repository from your dbt project, you should include the following steps in your CI/CD workflows. In your CI/CD workflows for your Dagster project: 1. Include any secrets that are required by your dbt project in your CI/CD environment. 2. Clone the dbt project repository as a subdirectory of your Dagster project. 3. Run `dagster-dbt project prepare-and-package --file path/to/project.py` to * Build your dbt project's dependencies, * Create a dbt manifest for your Dagster project, and * Package your dbt project note If you are using [Components](https://docs.dagster.io/guides/build/components) , you can prepare your `DbtProjectComponent` using `dagster-dbt project prepare-and-package --components path/to/project-root` In the CI/CD workflows for your dbt project, set up a dispatch action to trigger a deployment of your Dagster project when your dbt project changes. ### Deploying a dbt project from a monorepo[​](https://docs.dagster.io/integrations/libraries/dbt/reference#deploying-a-dbt-project-from-a-monorepo "Direct link to Deploying a dbt project from a monorepo") note With [Dagster+](https://dagster.io/cloud) , we streamline this option. As part of our Dagster+ onboarding for dbt users, we can automatically create a Dagster project in an existing dbt project repository. If you are managing your Dagster project in the same git repository as your dbt project, you should include the following steps in your CI/CD workflows. In your CI/CD workflows for your Dagster and dbt project: 1. Include any secrets that are required by your dbt project in your CI/CD environment. 2. Run `dagster-dbt project prepare-and-package --file path/to/project.py` to * Build your dbt project's dependencies, * Create a dbt manifest for your Dagster project, and * Package your dbt project note If you are using [Components](https://docs.dagster.io/guides/build/components) , you can prepare your `DbtProjectComponent` using `dagster-dbt project prepare-and-package --components path/to/project-root` Leveraging dbt defer with branch deployments[​](https://docs.dagster.io/integrations/libraries/dbt/reference#leveraging-dbt-defer-with-branch-deployments "Direct link to Leveraging dbt defer with branch deployments") ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- note This feature requires the `DAGSTER_BUILD_STATEDIR` environment variable to be set in your CI/CD. Learn more about required environment variables in CI/CD for Dagster+ [here](https://docs.dagster.io/deployment/dagster-plus/ci-cd/ci-cd-in-hybrid) . You will also need to run the `dagster-cloud ci dagster-dbt project manage-state` command in your prod deployment before it can be run in branch deployments. This will create the baseline for comparison in the branch deployments. It is possible to leverage [dbt defer](https://docs.getdbt.com/reference/node-selection/defer) by passing a `state_path` to [`DbtProject`](https://docs.dagster.io/api/libraries/dagster-dbt#dagster_dbt.DbtProject) . This is useful for testing recent changes made in development against the state of the dbt project in production. Using `dbt defer`, you can run a subset of models or tests, those that have been changed between development and production, without having to build their upstream parents first. In practice, this is most useful when combined with branch deployments in Dagster+, to test changes made in your branches. This can be done by updating your CI/CD files and your Dagster code. First, let's take a look at your CI/CD files. You might have one or two CI/CD files to manage your production and branch deployments. In these files, find the steps related to your dbt project - refer to the [Deploying a Dagster project with a dbt project](https://docs.dagster.io/integrations/libraries/dbt/reference#deploying-a-dagster-project-with-a-dbt-project) section for more information. Once your dbt steps are located, add the following step to manage the state of your dbt project. dagster-cloud ci dagster-dbt project manage-state --file path/to/project.py The `dagster-cloud ci dagster-dbt project manage-state` CLI command fetches the `manifest.json` file from your production branch and saves it to a state directory, in order to power the `dbt defer` command. In practice, this command fetches the `manifest.json` file from your production branch and add it to the state directory set to the `state_path` of the DbtProject found in `path/to/project.py`. The production `manifest.json` file can then be used as the deferred dbt artifacts. Now that your CI/CD files are updated to manage the state of your dbt project using the dagster-cloud CLI, we need to update the Dagster code to pass a state directory to the DbtProject. Update your Dagster code to pass a `state_path` to your `DbtProject` object. Note that value passed to `state_path` must be a path, relative to the dbt project directory, to a state directory of dbt artifacts. In the code below, we set the `state_path` to 'state/'. If this directory does not exist in your project structure, it will be created by Dagster. Also, update the dbt command in your `@dbt_assets` definition to pass the defer args using `get_defer_args`. from pathlib import Path from dagster import AssetExecutionContext from dagster_dbt import DbtCliResource, DbtProject, dbt_assets my_dbt_project = DbtProject( project_dir=Path(__file__).joinpath("..", "..", "..").resolve(), packaged_project_dir=Path(__file__) .joinpath("..", "..", "dbt-project") .resolve(), state_path=Path("state"), ) my_dbt_project.prepare_if_dev() @dbt_assets(manifest=my_dbt_project.manifest_path) def my_dbt_assets( context: AssetExecutionContext, dbt: DbtCliResource, ): yield from dbt.cli(["build", *dbt.get_defer_args()], context=context).stream() Using config with `@dbt_assets`[​](https://docs.dagster.io/integrations/libraries/dbt/reference#using-config-with-dbt_assets "Direct link to using-config-with-dbt_assets") ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Similar to Dagster software-defined assets, `@dbt_assets` can use a config system to enable [run configuration](https://docs.dagster.io/guides/operate/configuration/run-configuration) . This allows to provide parameters to jobs at the time they're executed. In the context of dbt, this can be useful if you want to run commands or flags for specific use cases. For instance, you may want to add [the --full-refresh flag](https://docs.getdbt.com/reference/resource-configs/full_refresh) to your dbt commands in some cases. Using a config system, the `@dbt_assets` object can be easily modified to support this use case. from pathlib import Path from dagster import AssetExecutionContext, Config from dagster_dbt import DbtCliResource, DbtProject, dbt_assets my_dbt_project = DbtProject(project_dir=Path("path/to/dbt_project")) class MyDbtConfig(Config): full_refresh: bool @dbt_assets(manifest=my_dbt_project.manifest_path) def my_dbt_assets( context: AssetExecutionContext, dbt: DbtCliResource, config: MyDbtConfig ): dbt_build_args = ["build"] if config.full_refresh: dbt_build_args += ["--full-refresh"] yield from dbt.cli(dbt_build_args, context=context).stream() Now that the `@dbt_assets` object is updated, the run configuration can be passed to a job. from dagster import RunConfig, define_asset_job from dagster_dbt import build_dbt_asset_selection my_job = define_asset_job( name="all_dbt_assets", selection=build_dbt_asset_selection( [my_dbt_assets], ), config=RunConfig( ops={"my_dbt_assets": MyDbtConfig(full_refresh=True, seed=True)} ), ) In the example above, the job is configured to use the `--full-refresh` flag with the dbt build command when materializing the assets. Scheduling dbt jobs[​](https://docs.dagster.io/integrations/libraries/dbt/reference#scheduling-dbt-jobs "Direct link to Scheduling dbt jobs") ---------------------------------------------------------------------------------------------------------------------------------------------- Once you have your dbt assets, you can define a job to materialize a selection of these assets on a schedule. ### Scheduling jobs that contain only dbt assets[​](https://docs.dagster.io/integrations/libraries/dbt/reference#scheduling-jobs-that-contain-only-dbt-assets "Direct link to Scheduling jobs that contain only dbt assets") In this example, we use the [`build_schedule_from_dbt_selection`](https://docs.dagster.io/api/libraries/dagster-dbt#dagster_dbt.build_schedule_from_dbt_selection) function to create a job, `daily_dbt_models`, as well as a schedule which will execute this job once a day. We define the set of models we'd like to execute using [dbt's selection syntax](https://docs.getdbt.com/reference/node-selection/syntax#how-does-selection-work) , in this case selecting only the models with the tag `daily`. from dagster_dbt import build_schedule_from_dbt_selection, dbt_assets @dbt_assets(manifest=manifest) def my_dbt_assets(): ... daily_dbt_assets_schedule = build_schedule_from_dbt_selection( [my_dbt_assets], job_name="daily_dbt_models", cron_schedule="@daily", dbt_select="tag:daily", # If your definition of `@dbt_assets` has Dagster Configuration, you can specify it here. # config=RunConfig(ops={"my_dbt_assets": MyDbtConfig(full_refresh=True)}), ) ### Scheduling jobs that contain dbt assets and non-dbt assets[​](https://docs.dagster.io/integrations/libraries/dbt/reference#scheduling-jobs-that-contain-dbt-assets-and-non-dbt-assets "Direct link to Scheduling jobs that contain dbt assets and non-dbt assets") In many cases, it's useful to be able to schedule dbt assets alongside non-dbt assets. In this example, we build an [`AssetSelection`](https://docs.dagster.io/api/dagster/assets#dagster.AssetSelection) of dbt assets using [`build_dbt_asset_selection`](https://docs.dagster.io/api/libraries/dagster-dbt#dagster_dbt.build_dbt_asset_selection) , then select all assets (dbt-related or not) which are downstream of these dbt models. From there, we create a job that targets that selection of assets and schedule it to run daily. from dagster import define_asset_job, ScheduleDefinition from dagster_dbt import build_dbt_asset_selection, dbt_assets @dbt_assets(manifest=manifest) def my_dbt_assets(): ... # selects all models tagged with "daily", and all their downstream asset dependencies daily_selection = build_dbt_asset_selection( [my_dbt_assets], dbt_select="tag:daily" ).downstream() daily_dbt_assets_and_downstream_schedule = ScheduleDefinition( job=define_asset_job("daily_assets", selection=daily_selection), cron_schedule="@daily", ) Refer to the [Schedule documentation](https://docs.dagster.io/guides/automate/schedules) for more info on running jobs on a schedule. Understanding asset definition attributes[​](https://docs.dagster.io/integrations/libraries/dbt/reference#understanding-asset-definition-attributes "Direct link to Understanding asset definition attributes") ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- In Dagster, each asset definition has attributes. Dagster automatically generates these attributes for each asset definition loaded from the dbt project. These attributes can optionally be overridden by the user. * [Customizing asset keys](https://docs.dagster.io/integrations/libraries/dbt/reference#customizing-asset-keys) * [Customizing group names](https://docs.dagster.io/integrations/libraries/dbt/reference#customizing-group-names) * [Customizing owners](https://docs.dagster.io/integrations/libraries/dbt/reference#customizing-owners) * [Customizing descriptions](https://docs.dagster.io/integrations/libraries/dbt/reference#customizing-descriptions) * [Customizing metadata](https://docs.dagster.io/integrations/libraries/dbt/reference#customizing-metadata) * [Customizing tags](https://docs.dagster.io/integrations/libraries/dbt/reference#customizing-tags) * [Customizing automation conditions](https://docs.dagster.io/integrations/libraries/dbt/reference#customizing-automation-conditions) ### Customizing asset keys[​](https://docs.dagster.io/integrations/libraries/dbt/reference#customizing-asset-keys "Direct link to Customizing asset keys") For dbt models, seeds, and snapshots, the default asset key will be the configured schema for that node, concatenated with the name of the node. | dbt node type | Schema | Model name | Resulting asset key | | --- | --- | --- | --- | | model, seed, snapshot | `None` | `MODEL_NAME` | `MODEL_NAME` | | | `SCHEMA` | `MODEL_NAME` | `SCHEMA/MODEL_NAME` | | | `None` | my\_model | some\_model | | | marketing | my\_model | marketing/my\_model | For dbt sources, the default asset key will be the name of the source concatenated with the name of the source table. | dbt node type | Source name | Table name | Resulting asset key | | --- | --- | --- | --- | | source | `SOURCE_NAME` | `TABLE_NAME` | `SOURCE_NAME/TABLE_NAME` | | | jaffle\_shop | orders | jaffle\_shop/orders | There are two ways to customize the asset keys generated by Dagster for dbt assets: 1. Defining [meta config](https://docs.getdbt.com/reference/resource-configs/meta) on your dbt node, or 2. Overriding Dagster's asset key generation by implementing a custom [`DagsterDbtTranslator`](https://docs.dagster.io/api/libraries/dagster-dbt#dagster_dbt.DagsterDbtTranslator) . To override an asset key generated by Dagster for a dbt node, you can define a `meta` key on your dbt node's `.yml` file. The following example overrides the asset key for a source and table as `snowflake/jaffle_shop/orders`: sources: - name: jaffle_shop tables: - name: orders meta: dagster: asset_key: ['snowflake', 'jaffle_shop', 'orders'] Alternatively, to override the asset key generation for all dbt nodes in your dbt project, you can create a custom [`DagsterDbtTranslator`](https://docs.dagster.io/api/libraries/dagster-dbt#dagster_dbt.DagsterDbtTranslator) and implement [`DagsterDbtTranslator.get_asset_key`](https://docs.dagster.io/api/libraries/dagster-dbt#dagster_dbt.DagsterDbtTranslator.get_asset_key) . The following example adds a `snowflake` prefix to the default generated asset key: from pathlib import Path from dagster import AssetKey, AssetExecutionContext from dagster_dbt import DagsterDbtTranslator, DbtCliResource, DbtProject, dbt_assets from typing import Any from collections.abc import Mapping my_dbt_project = DbtProject(project_dir=Path("path/to/dbt_project")) class CustomDagsterDbtTranslator(DagsterDbtTranslator): def get_asset_key(self, dbt_resource_props: Mapping[str, Any]) -> AssetKey: return super().get_asset_key(dbt_resource_props).with_prefix("snowflake") @dbt_assets( manifest=my_dbt_project.manifest_path, dagster_dbt_translator=CustomDagsterDbtTranslator(), ) def my_dbt_assets(context: AssetExecutionContext, dbt: DbtCliResource): yield from dbt.cli(["build"], context=context).stream() ### Customizing group names[​](https://docs.dagster.io/integrations/libraries/dbt/reference#customizing-group-names "Direct link to Customizing group names") For dbt models, seeds, and snapshots, the default Dagster group name will be the [dbt group](https://docs.getdbt.com/docs/build/groups) defined for that node. | dbt node type | dbt group name | Resulting Dagster group name | | --- | --- | --- | | model, seed, snapshot | `GROUP_NAME` | `GROUP_NAME` | | | `None` | `None` | | | finance | finance | There are three ways to customize the group names generated by Dagster for dbt assets: 1. Defining [meta config](https://docs.getdbt.com/reference/resource-configs/meta) on your dbt node, or 2. Overriding Dagster's group name generation by implementing a custom [`DagsterDbtTranslator`](https://docs.dagster.io/api/libraries/dagster-dbt#dagster_dbt.DagsterDbtTranslator) 3. Overriding Dagster's group name generation using [`map_asset_specs`](https://docs.dagster.io/api/dagster/definitions#dagster.Definitions.map_asset_specs) To override the group name generated by Dagster for a dbt node, you can define a `meta` key in your dbt project file, on your dbt node's property file, or on the node's in-file config block. The following example overrides the Dagster group name for the following model as `marketing`: models: - name: customers config: meta: dagster: group: marketing Alternatively, to override the Dagster group name generation for all dbt nodes in your dbt project, you can create a custom [`DagsterDbtTranslator`](https://docs.dagster.io/api/libraries/dagster-dbt#dagster_dbt.DagsterDbtTranslator) and implement [`DagsterDbtTranslator.get_group_name`](https://docs.dagster.io/api/libraries/dagster-dbt#dagster_dbt.DagsterDbtTranslator.get_group_name) . The following example defines `snowflake` as the group name for all dbt nodes: from pathlib import Path from dagster import AssetExecutionContext from dagster_dbt import DagsterDbtTranslator, DbtCliResource, DbtProject, dbt_assets from typing import Any, Optional from collections.abc import Mapping my_dbt_project = DbtProject(project_dir=Path("path/to/dbt_project")) class CustomDagsterDbtTranslator(DagsterDbtTranslator): def get_group_name( self, dbt_resource_props: Mapping[str, Any] ) -> Optional[str]: return "snowflake" @dbt_assets( manifest=my_dbt_project.manifest_path, dagster_dbt_translator=CustomDagsterDbtTranslator(), ) def my_dbt_assets(context: AssetExecutionContext, dbt: DbtCliResource): yield from dbt.cli(["build"], context=context).stream() Similarly, to override the Dagster group name generation for all dbt nodes in your dbt project, you can also use [`map_asset_specs`](https://docs.dagster.io/api/dagster/definitions#dagster.Definitions.map_asset_specs) . The following example defines `snowflake` as the group name for all dbt nodes: from pathlib import Path from dagster import AssetExecutionContext from dagster_dbt import DbtCliResource, DbtProject, dbt_assets my_dbt_project = DbtProject(project_dir=Path("path/to/dbt_project")) @dbt_assets( manifest=my_dbt_project.manifest_path, ) def my_dbt_assets(context: AssetExecutionContext, dbt: DbtCliResource): yield from dbt.cli(["build"], context=context).stream() my_dbt_assets = my_dbt_assets.map_asset_specs( lambda spec: spec.replace_attributes(group_name="snowflake") ) ### Customizing owners[​](https://docs.dagster.io/integrations/libraries/dbt/reference#customizing-owners "Direct link to Customizing owners") For dbt models, seeds, and snapshots, the default Dagster owner will be the email associated with the [dbt group](https://docs.getdbt.com/docs/build/groups) defined for that node. | dbt node type | dbt group name | dbt group's email | Resulting Dagster owner | | --- | --- | --- | --- | | model, seed, snapshot | `GROUP_NAME` | `OWNER@DOMAIN.COM` | `OWNER@DOMAIN.COM` | | | `GROUP_NAME` | `None` | `None` | | | `None` | `None` | `None` | | | finance | `owner@company.com` | `owner@company.com` | | | finance | `None` | `None` | There are three ways to customize the owners generated by Dagster for dbt assets: 1. Defining [meta config](https://docs.getdbt.com/reference/resource-configs/meta) on your dbt node, or 2. Overriding Dagster's generation of owners by implementing a custom [`DagsterDbtTranslator`](https://docs.dagster.io/api/libraries/dagster-dbt#dagster_dbt.DagsterDbtTranslator) 3. Overriding Dagster's owners generation using [`map_asset_specs`](https://docs.dagster.io/api/dagster/definitions#dagster.Definitions.map_asset_specs) To override the owners generated by Dagster for a dbt node, you can define a `meta` key in your dbt project file, on your dbt node's property file, or on the node's in-file config block. The following example overrides the Dagster owners for the following model as `owner@company.com` and `team:data@company.com`: models: - name: customers config: meta: dagster: owners: ['owner@company.com', 'team:data@company.com'] Alternatively, to override the Dagster generation of owners for all dbt nodes in your dbt project, you can create a custom [`DagsterDbtTranslator`](https://docs.dagster.io/api/libraries/dagster-dbt#dagster_dbt.DagsterDbtTranslator) and implement [`DagsterDbtTranslator.get_group_name`](https://docs.dagster.io/api/libraries/dagster-dbt#dagster_dbt.DagsterDbtTranslator.get_group_name) . The following example defines `owner@company.com` and `team:data@company.com` as the owners for all dbt nodes: from pathlib import Path from dagster import AssetExecutionContext from dagster_dbt import DagsterDbtTranslator, DbtCliResource, DbtProject, dbt_assets from typing import Any, Optional from collections.abc import Mapping, Sequence my_dbt_project = DbtProject(project_dir=Path("path/to/dbt_project")) class CustomDagsterDbtTranslator(DagsterDbtTranslator): def get_owners( self, dbt_resource_props: Mapping[str, Any] ) -> Optional[Sequence[str]]: return ["owner@company.com", "team:data@company.com"] @dbt_assets( manifest=my_dbt_project.manifest_path, dagster_dbt_translator=CustomDagsterDbtTranslator(), ) def my_dbt_assets(context: AssetExecutionContext, dbt: DbtCliResource): yield from dbt.cli(["build"], context=context).stream() Similarly, to override the Dagster owners generation for all dbt nodes in your dbt project, you can also use [`map_asset_specs`](https://docs.dagster.io/api/dagster/definitions#dagster.Definitions.map_asset_specs) . The following example defines `owner@company.com` and `team:data@company.com` as the owners for all dbt nodes: from pathlib import Path from dagster import AssetExecutionContext from dagster_dbt import DbtCliResource, DbtProject, dbt_assets my_dbt_project = DbtProject(project_dir=Path("path/to/dbt_project")) @dbt_assets( manifest=my_dbt_project.manifest_path, ) def my_dbt_assets(context: AssetExecutionContext, dbt: DbtCliResource): yield from dbt.cli(["build"], context=context).stream() my_dbt_assets = my_dbt_assets.map_asset_specs( lambda spec: spec.replace_attributes( owners=["owner@company.com", "team:data@company.com"] ) ) ### Customizing descriptions[​](https://docs.dagster.io/integrations/libraries/dbt/reference#customizing-descriptions "Direct link to Customizing descriptions") For dbt models, seeds, and snapshots, the default Dagster description will be the dbt node's description. To override the Dagster description for all dbt nodes in your dbt project, you can create a custom [`DagsterDbtTranslator`](https://docs.dagster.io/api/libraries/dagster-dbt#dagster_dbt.DagsterDbtTranslator) and implement [`DagsterDbtTranslator.get_description`](https://docs.dagster.io/api/libraries/dagster-dbt#dagster_dbt.DagsterDbtTranslator.get_description) . The following example defines the raw SQL of the dbt node as the Dagster description: import textwrap from pathlib import Path from dagster import AssetExecutionContext from dagster_dbt import DagsterDbtTranslator, DbtCliResource, DbtProject, dbt_assets from typing import Any from collections.abc import Mapping my_dbt_project = DbtProject(project_dir=Path("path/to/dbt_project")) class CustomDagsterDbtTranslator(DagsterDbtTranslator): def get_description(self, dbt_resource_props: Mapping[str, Any]) -> str: return textwrap.indent(dbt_resource_props.get("raw_sql", ""), "\t") @dbt_assets( manifest=my_dbt_project.manifest_path, dagster_dbt_translator=CustomDagsterDbtTranslator(), ) def my_dbt_assets(context: AssetExecutionContext, dbt: DbtCliResource): yield from dbt.cli(["build"], context=context).stream() ### Customizing metadata[​](https://docs.dagster.io/integrations/libraries/dbt/reference#customizing-metadata "Direct link to Customizing metadata") For dbt models, seeds, and snapshots, the default Dagster definition metadata will be the dbt node's declared column schema. To override the Dagster definition metadata for all dbt nodes in your dbt project, you can create a custom [`DagsterDbtTranslator`](https://docs.dagster.io/api/libraries/dagster-dbt#dagster_dbt.DagsterDbtTranslator) and implement [`DagsterDbtTranslator.get_metadata`](https://docs.dagster.io/api/libraries/dagster-dbt#dagster_dbt.DagsterDbtTranslator.get_metadata) . The following example defines the metadata of the dbt node as the Dagster metadata, using [`MetadataValue`](https://docs.dagster.io/api/dagster/metadata#dagster.MetadataValue) : from pathlib import Path from dagster import MetadataValue, AssetExecutionContext from dagster_dbt import DagsterDbtTranslator, DbtCliResource, DbtProject, dbt_assets from typing import Any from collections.abc import Mapping my_dbt_project = DbtProject(project_dir=Path("path/to/dbt_project")) class CustomDagsterDbtTranslator(DagsterDbtTranslator): def get_metadata( self, dbt_resource_props: Mapping[str, Any] ) -> Mapping[str, Any]: return { "dbt_metadata": MetadataValue.json(dbt_resource_props.get("meta", {})) } @dbt_assets( manifest=my_dbt_project.manifest_path, dagster_dbt_translator=CustomDagsterDbtTranslator(), ) def my_dbt_assets(context: AssetExecutionContext, dbt: DbtCliResource): yield from dbt.cli(["build"], context=context).stream() Dagster also supports fetching additional metadata at dbt execution time to attach to asset materializations. For more information, see the [Customizing asset materialization metadata](https://docs.dagster.io/integrations/libraries/dbt/reference#customizing-asset-materialization-metadata) section. #### Attaching code reference metadata[​](https://docs.dagster.io/integrations/libraries/dbt/reference#attaching-code-reference-metadata "Direct link to Attaching code reference metadata") Dagster's dbt integration can automatically attach [code reference](https://docs.dagster.io/guides/build/assets/metadata-and-tags/#source-code) metadata to the SQL files backing your dbt assets. To enable this feature, set the `enable_code_references` parameter to `True` in the [`DagsterDbtTranslatorSettings`](https://docs.dagster.io/api/libraries/dagster-dbt#dagster_dbt.DagsterDbtTranslatorSettings) passed to your [`DagsterDbtTranslator`](https://docs.dagster.io/api/libraries/dagster-dbt#dagster_dbt.DagsterDbtTranslator) : from pathlib import Pathfrom dagster_dbt import ( DagsterDbtTranslator, DagsterDbtTranslatorSettings, DbtCliResource, DbtProject, dbt_assets,)from dagster import AssetExecutionContext, Definitions, with_source_code_referencesmy_project = DbtProject(project_dir=Path("path/to/dbt_project"))# links to dbt model source code from assetsdagster_dbt_translator = DagsterDbtTranslator( settings=DagsterDbtTranslatorSettings(enable_code_references=True))@dbt_assets( manifest=my_project.manifest_path, dagster_dbt_translator=dagster_dbt_translator, project=my_project,)def my_dbt_assets(context: AssetExecutionContext, dbt: DbtCliResource): yield from dbt.cli(["build"], context=context).stream()defs = Definitions(assets=with_source_code_references([my_dbt_assets])) ### Customizing tags[​](https://docs.dagster.io/integrations/libraries/dbt/reference#customizing-tags "Direct link to Customizing tags") note In Dagster, tags are key-value pairs. However, in dbt, tags are strings. To bridge this divide, the dbt tag string is used as the Dagster tag key, and the Dagster tag value is set to the empty string, `""`. Any dbt tags that don't match Dagster's supported tag key format (e.g. they contain unsupported characters) will be ignored by default. For dbt models, seeds, and snapshots, the default Dagster tags will be the dbt node's configured tags. Any dbt tags that don't match Dagster's supported tag key format (e.g. they contain unsupported characters) will be ignored. To override the Dagster tags for all dbt nodes in your dbt project, you can create a custom [`DagsterDbtTranslator`](https://docs.dagster.io/api/libraries/dagster-dbt#dagster_dbt.DagsterDbtTranslator) and implement [`DagsterDbtTranslator.get_tags`](https://docs.dagster.io/api/libraries/dagster-dbt#dagster_dbt.DagsterDbtTranslator.get_tags) . The following converts dbt tags of the form `foo=bar` to key/value pairs: from pathlib import Path from dagster import AssetExecutionContext from dagster_dbt import DagsterDbtTranslator, DbtCliResource, DbtProject, dbt_assets from typing import Any from collections.abc import Mapping my_dbt_project = DbtProject(project_dir=Path("path/to/dbt_project")) class CustomDagsterDbtTranslator(DagsterDbtTranslator): def get_tags(self, dbt_resource_props: Mapping[str, Any]) -> Mapping[str, str]: dbt_tags = dbt_resource_props.get("tags", []) dagster_tags = {} for tag in dbt_tags: key, _, value = tag.partition("=") dagster_tags[key] = value if value else "" return dagster_tags @dbt_assets( manifest=my_dbt_project.manifest_path, dagster_dbt_translator=CustomDagsterDbtTranslator(), ) def my_dbt_assets(context: AssetExecutionContext, dbt: DbtCliResource): yield from dbt.cli(["build"], context=context).stream() ### Customizing automation conditions[​](https://docs.dagster.io/integrations/libraries/dbt/reference#customizing-automation-conditions "Direct link to Customizing automation conditions") To override the [`AutomationCondition`](https://docs.dagster.io/api/dagster/assets#dagster.AutomationCondition) generated for each dbt node in your dbt project, you can create a custom [`DagsterDbtTranslator`](https://docs.dagster.io/api/libraries/dagster-dbt#dagster_dbt.DagsterDbtTranslator) and implement [`DagsterDbtTranslator.get_automation_condition`](https://docs.dagster.io/api/libraries/dagster-dbt#dagster_dbt.DagsterDbtTranslator.get_automation_condition) . The following example defines [`AutomationCondition.eager`](https://docs.dagster.io/api/dagster/assets#dagster.AutomationCondition.eager) as the condition for all dbt nodes: from pathlib import Path from dagster import AssetExecutionContext, AutomationCondition from dagster_dbt import DagsterDbtTranslator, DbtCliResource, DbtProject, dbt_assets from typing import Any, Optional from collections.abc import Mapping my_dbt_project = DbtProject(project_dir=Path("path/to/dbt_project")) class CustomDagsterDbtTranslator(DagsterDbtTranslator): def get_automation_condition( self, dbt_resource_props: Mapping[str, Any] ) -> Optional[AutomationCondition]: return AutomationCondition.eager() @dbt_assets( manifest=my_dbt_project.manifest_path, dagster_dbt_translator=CustomDagsterDbtTranslator(), ) def my_dbt_assets(context: AssetExecutionContext, dbt: DbtCliResource): yield from dbt.cli(["build"], context=context).stream() note Ensure that the [`default_automation_condition_sensor` is enabled](https://docs.dagster.io/guides/automate/declarative-automation/automation-condition-sensors) for automation conditions to be evaluated. dbt models, code versions, and "Unsynced"[​](https://docs.dagster.io/integrations/libraries/dbt/reference#dbt-models-code-versions-and-unsynced "Direct link to dbt models, code versions, and "Unsynced"") ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ Note that Dagster allows the optional specification of a [`code_version`](https://docs.dagster.io/guides/build/assets/defining-assets#asset-code-versions) for each asset definition, which is used to track changes. The `code_version` for an asset arising from a dbt model is defined automatically as the hash of the SQL defining the DBT model. This allows the asset graph in the UI to use the "Unsynced" status to indicate which dbt models have new SQL since they were last materialized. Loading dbt tests as asset checks[​](https://docs.dagster.io/integrations/libraries/dbt/reference#loading-dbt-tests-as-asset-checks "Direct link to Loading dbt tests as asset checks") ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- note **Asset checks for dbt have been enabled by default, starting in `dagster-dbt` 0.23.0.** `dbt-core` 1.6 or later is required for full functionality. Dagster automatically loads your dbt tests on _models_ as [asset checks](https://docs.dagster.io/guides/test/asset-checks) . To load dbt tests on sources as asset checks as well, see [Loading dbt source tests as asset checks](https://docs.dagster.io/integrations/libraries/dbt/reference#loading-dbt-source-tests-as-asset-checks) section. ### Indirect selection[​](https://docs.dagster.io/integrations/libraries/dbt/reference#indirect-selection "Direct link to Indirect selection") Dagster uses [dbt indirect selection](https://docs.getdbt.com/reference/global-configs/indirect-selection) to select dbt tests. By default, Dagster won't set `DBT_INDIRECT_SELECTION` so that the set of tests selected by Dagster is the same as the selected by dbt. When required, Dagster will override `DBT_INDIRECT_SELECTION` to `empty` in order to explicitly select dbt tests. For example: * Materializing dbt assets and excluding their asset checks * Executing dbt asset checks without materializing their assets ### Singular tests[​](https://docs.dagster.io/integrations/libraries/dbt/reference#singular-tests "Direct link to Singular tests") Dagster will load both generic and singular tests as asset checks. In the event that your singular test depends on multiple dbt models, you can use dbt metadata to specify which Dagster asset it should target. These fields can be filled in as they would be for the dbt [ref function](https://docs.getdbt.com/reference/dbt-jinja-functions/ref) . The configuration can be supplied in a [config block](https://docs.getdbt.com/reference/data-test-configs) for the singular test. {{ config( meta={ 'dagster': { 'ref': { 'name': 'customers', 'package': 'my_dbt_assets', 'version': 1, }, } } )}} `dbt-core` version 1.6 or later is required for Dagster to read this metadata. If this metadata isn't provided, Dagster won't ingest the test as an asset check. It will still run the test and emit a [`AssetObservation`](https://docs.dagster.io/api/dagster/assets#dagster.AssetObservation) event with the test results. ### Disabling asset checks[​](https://docs.dagster.io/integrations/libraries/dbt/reference#disabling-asset-checks "Direct link to Disabling asset checks") You can disable modeling your dbt tests as asset checks. The tests will still run and will be emitted as [`AssetObservation`](https://docs.dagster.io/api/dagster/assets#dagster.AssetObservation) events. To do so you'll need to define a [`DagsterDbtTranslator`](https://docs.dagster.io/api/libraries/dagster-dbt#dagster_dbt.DagsterDbtTranslator) with [`DagsterDbtTranslatorSettings`](https://docs.dagster.io/api/libraries/dagster-dbt#dagster_dbt.DagsterDbtTranslatorSettings) that have asset checks disabled. The following example disables asset checks when using [`@dagster_dbt.dbt_assets`](https://docs.dagster.io/api/libraries/dagster-dbt#dagster_dbt.dbt_assets) : from pathlib import Path from dagster import AssetExecutionContext from dagster_dbt import ( DagsterDbtTranslator, DagsterDbtTranslatorSettings, DbtCliResource, DbtProject, dbt_assets, ) my_dbt_project = DbtProject(project_dir=Path("path/to/dbt_project")) dagster_dbt_translator = DagsterDbtTranslator( settings=DagsterDbtTranslatorSettings(enable_asset_checks=False) ) @dbt_assets( manifest=my_dbt_project.manifest_path, dagster_dbt_translator=dagster_dbt_translator, ) def my_dbt_assets(context: AssetExecutionContext, dbt: DbtCliResource): yield from dbt.cli(["build"], context=context).stream() ### Loading dbt source tests as asset checks[​](https://docs.dagster.io/integrations/libraries/dbt/reference#loading-dbt-source-tests-as-asset-checks "Direct link to Loading dbt source tests as asset checks") It's common to have the body of your dbt assets execute a `dbt build` command. In addition to executing all of your dbt models and their tests, this will also execute any dbt tests on sources that are upstream of your dbt models. By default, Dagster does not load dbt source tests as asset checks. To enable this feature, you can define a [`DagsterDbtTranslator`](https://docs.dagster.io/api/libraries/dagster-dbt#dagster_dbt.DagsterDbtTranslator) with [`DagsterDbtTranslatorSettings`](https://docs.dagster.io/api/libraries/dagster-dbt#dagster_dbt.DagsterDbtTranslatorSettings) that have source tests enabled. The following example enables loading dbt source tests as asset checks: from pathlib import Path from dagster import AssetExecutionContext from dagster_dbt import ( DagsterDbtTranslator, DagsterDbtTranslatorSettings, DbtCliResource, DbtProject, dbt_assets, ) my_dbt_project = DbtProject(project_dir=Path("path/to/dbt_project")) dagster_dbt_translator = DagsterDbtTranslator( settings=DagsterDbtTranslatorSettings(enable_source_tests_as_checks=True) ) @dbt_assets( manifest=my_dbt_project.manifest_path, dagster_dbt_translator=dagster_dbt_translator, ) def my_dbt_assets(context: AssetExecutionContext, dbt: DbtCliResource): yield from dbt.cli(["build"], context=context).stream() Customizing asset materialization metadata[​](https://docs.dagster.io/integrations/libraries/dbt/reference#customizing-asset-materialization-metadata "Direct link to Customizing asset materialization metadata") ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Dagster supports fetching additional metadata at dbt execution time to attach as [materialization metadata](https://docs.dagster.io/guides/build/assets/metadata-and-tags) , which is recorded each time your models are rebuilt and displayed in the Dagster UI. ### Fetching row count data[​](https://docs.dagster.io/integrations/libraries/dbt/reference#fetching-row-count-data "Direct link to Fetching row count data") note To use this feature, you'll need to be on at least `dagster>=0.17.6` and `dagster-dbt>=0.23.6`. Dagster can automatically fetch [row counts](https://docs.dagster.io/guides/build/assets/metadata-and-tags) for dbt-generated tables and emit them as [materialization metadata](https://docs.dagster.io/guides/build/assets/metadata-and-tags) to be displayed in the Dagster UI. Row counts are fetched in parallel to the execution of your dbt models. To enable this feature, call [`fetch_row_counts()`](https://docs.dagster.io/api/libraries/dagster-dbt#dagster_dbt.core.dbt_cli_invocation.DbtEventIterator.fetch_row_counts) on the [`core.dbt_cli_invocation.DbtEventIterator`](https://docs.dagster.io/api/libraries/dagster-dbt#dagster_dbt.core.dbt_cli_invocation.DbtEventIterator) returned by the `stream()` dbt CLI call: from pathlib import Path from dagster import AssetExecutionContext from dagster_dbt import DbtProject, DbtCliResource, dbt_assets my_dbt_project = DbtProject(project_dir=Path("path/to/dbt_project")) @dbt_assets( manifest=my_dbt_project.manifest_path, ) def my_dbt_assets(context: AssetExecutionContext, dbt: DbtCliResource): yield from dbt.cli(["build"], context=context).stream().fetch_row_counts() Once your dbt models have been materialized, you can view the row count data in the metadata table. ### Fetching column-level metadata[​](https://docs.dagster.io/integrations/libraries/dbt/reference#fetching-column-level-metadata "Direct link to Fetching column-level metadata") note To use this feature, you'll need to be on at least `dagster>=1.8.0` and `dagster-dbt>=0.24.0`. Dagster allows you to emit column-level metadata, like [column schema](https://docs.dagster.io/guides/build/assets/metadata-and-tags) and [column lineage](https://docs.dagster.io/guides/build/assets/metadata-and-tags) , as [materialization metadata](https://docs.dagster.io/guides/build/assets/metadata-and-tags) . With this metadata, you can view documentation in Dagster for all columns, not just columns described in your dbt project. Column-level metadata is fetched in parallel to the execution of your dbt models. To enable this feature, call [`fetch_column_metadata()`](https://docs.dagster.io/api/libraries/dagster-dbt#dagster_dbt.core.dbt_cli_invocation.DbtEventIterator.fetch_column_metadata) on the [`core.dbt_cli_invocation.DbtEventIterator`](https://docs.dagster.io/api/libraries/dagster-dbt#dagster_dbt.core.dbt_cli_invocation.DbtEventIterator) returned by the `stream()` dbt CLI call: from pathlib import Path from dagster import AssetExecutionContext from dagster_dbt import DbtProject, DbtCliResource, dbt_assets my_dbt_project = DbtProject(project_dir=Path("path/to/dbt_project")) @dbt_assets( manifest=my_dbt_project.manifest_path, ) def my_dbt_assets(context: AssetExecutionContext, dbt: DbtCliResource): yield from ( dbt.cli(["build"], context=context).stream().fetch_column_metadata() ) ### Composing metadata fetching methods[​](https://docs.dagster.io/integrations/libraries/dbt/reference#composing-metadata-fetching-methods "Direct link to Composing metadata fetching methods") Metadata fetching methods such as [`fetch_column_metadata()`](https://docs.dagster.io/api/libraries/dagster-dbt#dagster_dbt.core.dbt_cli_invocation.DbtEventIterator.fetch_column_metadata) can be chained with other metadata fetching methods like [`fetch_row_counts()`](https://docs.dagster.io/api/libraries/dagster-dbt#dagster_dbt.core.dbt_cli_invocation.DbtEventIterator.fetch_row_counts) : from pathlib import Path from dagster import AssetExecutionContext from dagster_dbt import DbtProject, DbtCliResource, dbt_assets my_dbt_project = DbtProject(project_dir=Path("path/to/dbt_project")) @dbt_assets( manifest=my_dbt_project.manifest_path, ) def my_dbt_assets(context: AssetExecutionContext, dbt: DbtCliResource): yield from ( dbt.cli(["build"], context=context) .stream() .fetch_row_counts() .fetch_column_metadata() ) Defining dependencies[​](https://docs.dagster.io/integrations/libraries/dbt/reference#defining-dependencies "Direct link to Defining dependencies") ---------------------------------------------------------------------------------------------------------------------------------------------------- * [Upstream dependencies](https://docs.dagster.io/integrations/libraries/dbt/reference#upstream-dependencies) * [Downstream dependencies](https://docs.dagster.io/integrations/libraries/dbt/reference#downstream-dependencies) ### Upstream dependencies[​](https://docs.dagster.io/integrations/libraries/dbt/reference#upstream-dependencies "Direct link to Upstream dependencies") #### Defining a dbt source as a Dagster asset[​](https://docs.dagster.io/integrations/libraries/dbt/reference#defining-a-dbt-source-as-a-dagster-asset "Direct link to Defining a dbt source as a Dagster asset") Dagster parses information about assets that are upstream of specific dbt models from the dbt project itself. Whenever a model is downstream of a [dbt source](https://docs.getdbt.com/docs/building-a-dbt-project/using-sources) , that upstream source will be parsed as an upstream asset. For example, if you defined a source in your `sources.yml` file like this: sources: - name: jaffle_shop tables: - name: orders and use it in a model: select * from {{ source("jaffle_shop", "orders") }} where foo=1 Then this model has an upstream source with the `jaffle_shop/orders` asset key. In order to manage this upstream asset with Dagster, you can define it by passing the key into an asset definition via [`get_asset_key_for_source`](https://docs.dagster.io/api/libraries/dagster-dbt#dagster_dbt.get_asset_key_for_source) : from dagster import asset, AssetExecutionContext from dagster_dbt import DbtCliResource, get_asset_key_for_source, dbt_assets @dbt_assets(manifest=MANIFEST_PATH) def my_dbt_assets(context: AssetExecutionContext, dbt: DbtCliResource): ... @asset(key=get_asset_key_for_source([my_dbt_assets], "jaffle_shop")) def orders(): return ... This allows you to change asset keys within your dbt project without having to update the corresponding Dagster definitions. The [`get_asset_key_for_source`](https://docs.dagster.io/api/libraries/dagster-dbt#dagster_dbt.get_asset_key_for_source) method is used when a source has only one table. However, if a source contains multiple tables, like this example: sources: - name: clients_data tables: - name: names - name: history You can use define a [`@dg.multi_asset`](https://docs.dagster.io/api/dagster/assets#dagster.multi_asset) with keys from [`get_asset_keys_by_output_name_for_source`](https://docs.dagster.io/api/libraries/dagster-dbt#dagster_dbt.get_asset_keys_by_output_name_for_source) instead: from dagster import multi_asset, AssetOut, Output from dagster_dbt import get_asset_keys_by_output_name_for_source @multi_asset( outs={ name: AssetOut(key=asset_key) for name, asset_key in get_asset_keys_by_output_name_for_source( [my_dbt_assets], "jaffle_shop" ).items() } ) def jaffle_shop(context: AssetExecutionContext): output_names = list(context.op_execution_context.selected_output_names) yield Output(value=..., output_name=output_names[0]) yield Output(value=..., output_name=output_names[1]) #### Defining an asset as an upstream data dependency of a dbt model[​](https://docs.dagster.io/integrations/libraries/dbt/reference#defining-an-asset-as-an-upstream-data-dependency-of-a-dbt-model "Direct link to Defining an asset as an upstream data dependency of a dbt model") Dagster allows you to define existing assets as upstream data dependencies of dbt models, meaning that an upstream Dagster asset creates data for the dbt model to read. For example, say you have the following asset with asset key `upstream`: from dagster import asset @asset def upstream(): ... You can define that asset as a source in your `sources.yml` file: sources: - name: dagster tables: - name: upstream Then, in the downstream model, you can select from this source data. This defines a data dependency relationship between your upstream asset and dbt model: select * from {{ source("dagster", "upstream") }} where foo=1 #### Defining an asset as an upstream temporal dependency of a dbt model[​](https://docs.dagster.io/integrations/libraries/dbt/reference#defining-an-asset-as-an-upstream-temporal-dependency-of-a-dbt-model "Direct link to Defining an asset as an upstream temporal dependency of a dbt model") Dagster allows you to define existing assets as upstream temporal dependencies of dbt models, meaning that Dagster needs to schedule the dbt model after a Dagster asset has materialized, but that the model does not need to read data from the asset. For example, say you have the following asset with asset key `upstream`: from dagster import asset @asset def upstream(): ... First, define that asset as a source in your `sources.yml` file: sources: - name: dagster tables: - name: upstream Then, in the downstream model, you can specify that the downstream model depends on the upstream Dagster asset. This defines a temporal dependency relationship between your upstream asset and dbt model: -- depends_on: {{ source('dagster','upstream') }}SELECT ... ### Downstream dependencies[​](https://docs.dagster.io/integrations/libraries/dbt/reference#downstream-dependencies "Direct link to Downstream dependencies") Dagster allows you to define assets that are downstream of specific dbt models via [`get_asset_key_for_model`](https://docs.dagster.io/api/libraries/dagster-dbt#dagster_dbt.get_asset_key_for_model) . The below example defines `my_downstream_asset` as a downstream dependency of `my_dbt_model`: from dagster_dbt import get_asset_key_for_model from dagster import asset @asset(deps=[get_asset_key_for_model([my_dbt_assets], "my_dbt_model")]) def my_downstream_asset(): ... In the downstream asset, you may want direct access to the contents of the dbt model. To do so, you can customize the code within your `@asset`\-decorated function to load upstream data. Dagster alternatively allows you to delegate loading data to an I/O manager. For example, if you wanted to consume a dbt model with the asset key `my_dbt_model` as a Pandas dataframe, that would look something like the following: from dagster_dbt import get_asset_key_for_model from dagster import AssetIn, asset @asset( ins={ "my_dbt_model": AssetIn( input_manager_key="pandas_df_manager", key=get_asset_key_for_model([my_dbt_assets], "my_dbt_model"), ) }, ) def my_downstream_asset(my_dbt_model): # my_dbt_model is a Pandas dataframe return my_dbt_model.where(foo="bar") Building incremental models using partitions[​](https://docs.dagster.io/integrations/libraries/dbt/reference#building-incremental-models-using-partitions "Direct link to Building incremental models using partitions") ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- You can define a Dagster [`PartitionsDefinition`](https://docs.dagster.io/api/dagster/partitions#dagster.PartitionsDefinition) alongside dbt in order to build incremental models. Partitioned assets will be able to access the [`TimeWindow`](https://docs.dagster.io/api/dagster/partitions#dagster.TimeWindow) 's start and end dates, and these can be passed to dbt's CLI as variables which can be used to filter incremental models. When a partition definition to passed to the [`@dagster_dbt.dbt_assets`](https://docs.dagster.io/api/libraries/dagster-dbt#dagster_dbt.dbt_assets) decorator, all assets are defined to operate on the same partitions. With this in mind, we can retrieve any time window from [`AssetExecutionContext.partition_time_window`](https://docs.dagster.io/api/dagster/execution#dagster.AssetExecutionContext.partition_time_window) property in order to get the current start and end partitions. import json from pathlib import Path from dagster import DailyPartitionsDefinition, OpExecutionContext from dagster_dbt import DbtCliResource, DbtProject, dbt_assets my_dbt_project = DbtProject(project_dir=Path("path/to/dbt_project")) @dbt_assets( manifest=my_dbt_project.manifest_path, partitions_def=DailyPartitionsDefinition(start_date="2023-01-01"), ) def partitionshop_dbt_assets(context: OpExecutionContext, dbt: DbtCliResource): start, end = context.partition_time_window dbt_vars = {"min_date": start.isoformat(), "max_date": end.isoformat()} dbt_build_args = ["build", "--vars", json.dumps(dbt_vars)] yield from dbt.cli(dbt_build_args, context=context).stream() With the variables defined, we can now reference `min_date` and `max_date` in our SQL and configure the dbt model as incremental. Here, we define an incremental run to operate on rows with `order_date` that is between our `min_date` and `max_date`. -- Configure the model as incremental, use a unique_key and the delete+insert strategy to ensure the pipeline is idempotent.{{ config(materialized='incremental', unique_key='order_date', incremental_strategy="delete+insert") }}select * from {{ ref('my_model') }}-- Use the Dagster partition variables to filter rows on an incremental run{% if is_incremental() %}where order_date >= '{{ var('min_date') }}' and order_date <= '{{ var('max_date') }}'{% endif %} * [Relevant APIs](https://docs.dagster.io/integrations/libraries/dbt/reference#relevant-apis) * [dbt models and Dagster asset definitions](https://docs.dagster.io/integrations/libraries/dbt/reference#dbt-models-and-dagster-asset-definitions) * [Scaffolding a Dagster project from a dbt project](https://docs.dagster.io/integrations/libraries/dbt/reference#scaffolding-a-dagster-project-from-a-dbt-project) * [Loading dbt models from a dbt project](https://docs.dagster.io/integrations/libraries/dbt/reference#loading-dbt-models-from-a-dbt-project) * [Selecting a profiles directory, profile and target for your dbt project](https://docs.dagster.io/integrations/libraries/dbt/reference#selecting-a-profiles-directory-profile-and-target-for-your-dbt-project) * [Deploying a Dagster project with a dbt project](https://docs.dagster.io/integrations/libraries/dbt/reference#deploying-a-dagster-project-with-a-dbt-project) * [Deploying a dbt project from a separate git repository](https://docs.dagster.io/integrations/libraries/dbt/reference#deploying-a-dbt-project-from-a-separate-git-repository) * [Deploying a dbt project from a monorepo](https://docs.dagster.io/integrations/libraries/dbt/reference#deploying-a-dbt-project-from-a-monorepo) * [Leveraging dbt defer with branch deployments](https://docs.dagster.io/integrations/libraries/dbt/reference#leveraging-dbt-defer-with-branch-deployments) * [Using config with `@dbt_assets`](https://docs.dagster.io/integrations/libraries/dbt/reference#using-config-with-dbt_assets) * [Scheduling dbt jobs](https://docs.dagster.io/integrations/libraries/dbt/reference#scheduling-dbt-jobs) * [Scheduling jobs that contain only dbt assets](https://docs.dagster.io/integrations/libraries/dbt/reference#scheduling-jobs-that-contain-only-dbt-assets) * [Scheduling jobs that contain dbt assets and non-dbt assets](https://docs.dagster.io/integrations/libraries/dbt/reference#scheduling-jobs-that-contain-dbt-assets-and-non-dbt-assets) * [Understanding asset definition attributes](https://docs.dagster.io/integrations/libraries/dbt/reference#understanding-asset-definition-attributes) * [Customizing asset keys](https://docs.dagster.io/integrations/libraries/dbt/reference#customizing-asset-keys) * [Customizing group names](https://docs.dagster.io/integrations/libraries/dbt/reference#customizing-group-names) * [Customizing owners](https://docs.dagster.io/integrations/libraries/dbt/reference#customizing-owners) * [Customizing descriptions](https://docs.dagster.io/integrations/libraries/dbt/reference#customizing-descriptions) * [Customizing metadata](https://docs.dagster.io/integrations/libraries/dbt/reference#customizing-metadata) * [Attaching code reference metadata](https://docs.dagster.io/integrations/libraries/dbt/reference#attaching-code-reference-metadata) * [Customizing tags](https://docs.dagster.io/integrations/libraries/dbt/reference#customizing-tags) * [Customizing automation conditions](https://docs.dagster.io/integrations/libraries/dbt/reference#customizing-automation-conditions) * [dbt models, code versions, and "Unsynced"](https://docs.dagster.io/integrations/libraries/dbt/reference#dbt-models-code-versions-and-unsynced) * [Loading dbt tests as asset checks](https://docs.dagster.io/integrations/libraries/dbt/reference#loading-dbt-tests-as-asset-checks) * [Indirect selection](https://docs.dagster.io/integrations/libraries/dbt/reference#indirect-selection) * [Singular tests](https://docs.dagster.io/integrations/libraries/dbt/reference#singular-tests) * [Disabling asset checks](https://docs.dagster.io/integrations/libraries/dbt/reference#disabling-asset-checks) * [Loading dbt source tests as asset checks](https://docs.dagster.io/integrations/libraries/dbt/reference#loading-dbt-source-tests-as-asset-checks) * [Customizing asset materialization metadata](https://docs.dagster.io/integrations/libraries/dbt/reference#customizing-asset-materialization-metadata) * [Fetching row count data](https://docs.dagster.io/integrations/libraries/dbt/reference#fetching-row-count-data) * [Fetching column-level metadata](https://docs.dagster.io/integrations/libraries/dbt/reference#fetching-column-level-metadata) * [Composing metadata fetching methods](https://docs.dagster.io/integrations/libraries/dbt/reference#composing-metadata-fetching-methods) * [Defining dependencies](https://docs.dagster.io/integrations/libraries/dbt/reference#defining-dependencies) * [Upstream dependencies](https://docs.dagster.io/integrations/libraries/dbt/reference#upstream-dependencies) * [Defining a dbt source as a Dagster asset](https://docs.dagster.io/integrations/libraries/dbt/reference#defining-a-dbt-source-as-a-dagster-asset) * [Defining an asset as an upstream data dependency of a dbt model](https://docs.dagster.io/integrations/libraries/dbt/reference#defining-an-asset-as-an-upstream-data-dependency-of-a-dbt-model) * [Defining an asset as an upstream temporal dependency of a dbt model](https://docs.dagster.io/integrations/libraries/dbt/reference#defining-an-asset-as-an-upstream-temporal-dependency-of-a-dbt-model) * [Downstream dependencies](https://docs.dagster.io/integrations/libraries/dbt/reference#downstream-dependencies) * [Building incremental models using partitions](https://docs.dagster.io/integrations/libraries/dbt/reference#building-incremental-models-using-partitions) --- # Dagster & Ray | Dagster Docs [Skip to main content](https://docs.dagster.io/integrations/libraries/ray#__docusaurus_skipToContent_fallback) On this page The community-supported Ray package allows orchestrating distributed Ray compute from Dagster pipelines. For more information, see the [dagster-ray GitHub repository](https://github.com/danielgafni/dagster-ray) . Installation[​](https://docs.dagster.io/integrations/libraries/ray#installation "Direct link to Installation") --------------------------------------------------------------------------------------------------------------- * uv * pip uv add dagster-ray pip install dagster-ray * [Installation](https://docs.dagster.io/integrations/libraries/ray#installation) --- # Dagster & Patito | Dagster Docs [Skip to main content](https://docs.dagster.io/integrations/libraries/patito#__docusaurus_skipToContent_fallback) On this page Patito is a data validation framework for Polars, based on Pydantic. For more information on how to use Dagster with Polars, see [dagster-polars documentation](https://docs.dagster.io/integrations/libraries/polars) . Installation[​](https://docs.dagster.io/integrations/libraries/patito#installation "Direct link to Installation") ------------------------------------------------------------------------------------------------------------------ * uv * pip uv add dagster-polars[patito] pip install dagster-polars[patito] Usage[​](https://docs.dagster.io/integrations/libraries/patito#usage "Direct link to Usage") --------------------------------------------------------------------------------------------- `dagster-polars` automatically recognizes Patito type annotations, performs data validation using the specified model, and infers table metadata such as column constraints, data types, and descriptions. import dagster as dgimport patito as ptimport polars as plclass User(pt.Model): uid: str = pt.Field(unique=True, description="User ID") age: int | None = pt.Field(gt=18, description="User age")@dg.asset(io_manager_key="polars_parquet_io_manager")def my_asset() -> User.DataFrame: my_data = ... return User.DataFrame(my_data) The specified `User` model will be used to validate the data returned by the `my_asset` asset. If the data does not conform to the model, an error will be raised. Upstream assets can also be annotated with Patito models, ensuring that the input data is always validated. Alternatively, the standalone `dagster_polars.patito.patito_model_to_dagster_type` function can be used to build a dagster type for a given Patito model, which can then be used with any IOManager. In this case, the type annotation can be a normal `pl.DataFrame`. from dagster_polars.patito import patito_model_to_dagster_typeuser_type = patito_model_to_dagster_type(User)@dg.asset(io_manager_key="polars_parquet_io_manager", dagster_type=user_type)def my_asset() -> pl.DataFrame: my_data = ... return pl.DataFrame(my_data) # <- you better behave, mr. data! The same dagster type can be used when loading data into downstream assets, ensuring that the data is always validated. * [Installation](https://docs.dagster.io/integrations/libraries/patito#installation) * [Usage](https://docs.dagster.io/integrations/libraries/patito#usage) --- # Dagster & Polars | Dagster Docs [Skip to main content](https://docs.dagster.io/integrations/libraries/polars#__docusaurus_skipToContent_fallback) On this page The Polars integration allows using Polars eager or lazy DataFrames as inputs and outputs with Dagster’s assets and ops. Type annotations are used to control whether to load an eager or lazy DataFrame. Lazy DataFrames can be sinked as output. Multiple serialization formats (Parquet, Delta Lake, BigQuery) and filesystems (local, S3, GCS, …) are supported. Installation[​](https://docs.dagster.io/integrations/libraries/polars#installation "Direct link to Installation") ------------------------------------------------------------------------------------------------------------------ * uv * pip uv add dagster-polars pip install dagster-polars Example[​](https://docs.dagster.io/integrations/libraries/polars#example "Direct link to Example") --------------------------------------------------------------------------------------------------- import polars as plfrom dagster import asset, Definitionsfrom dagster_polars import PolarsParquetIOManager@asset(io_manager_key="polars_parquet_io_manager")def upstream(): return DataFrame({"foo": [1, 2, 3]})@asset(io_manager_key="polars_parquet_io_manager")def downstream(upstream) -> pl.LazyFrame: assert isinstance(upstream, pl.DataFrame) return upstream.lazy() # LazyFrame will be sinkeddefinitions = Definitions(assets=[upstream, downstream], resources={"polars_parquet_io_manager": PolarsParquetIOManager(...)}) Lazy pl.LazyFrame can be scanned by annotating the input with pl.LazyFrame, and returning a pl.LazyFrame will sink it: @asset(io_manager_key="polars_parquet_io_manager")def downstream(upstream: pl.LazyFrame) -> pl.LazyFrame: assert isinstance(upstream, pl.LazyFrame) return upstream Supplementary[​](https://docs.dagster.io/integrations/libraries/polars#supplementary "Direct link to Supplementary") --------------------------------------------------------------------------------------------------------------------- * [Patito integration](https://docs.dagster.io/integrations/libraries/patito) * [Installation](https://docs.dagster.io/integrations/libraries/polars#installation) * [Example](https://docs.dagster.io/integrations/libraries/polars#example) * [Supplementary](https://docs.dagster.io/integrations/libraries/polars#supplementary) --- # Dagster & Secoda | Dagster Docs [Skip to main content](https://docs.dagster.io/integrations/libraries/secoda#__docusaurus_skipToContent_fallback) On this page Connect Dagster to Secoda and see metadata related to your Dagster assets, asset groups and jobs right in Secoda. Simplify your team's access, and remove the need to switch between tools. See the [Secoda documentation](https://www.secoda.co/integrations/dagster) for more information. When you connect Dagster to Secoda, you can use Secoda's tools to add further context to your Dagster assets and jobs. Help your team understand metadata from Dagster by adding context in Secoda, like creating Documents, defining Metrics, and adding Tags. About Secoda[​](https://docs.dagster.io/integrations/libraries/secoda#about-secoda "Direct link to About Secoda") ------------------------------------------------------------------------------------------------------------------ Secoda is a AI-powered data search, cataloging, lineage, and documentation platform that empowers data teams to manage data sprawl, scale infrastructure, and overcome common issues such as lack of observability, governance, and lengthy setup and integration periods. * [About Secoda](https://docs.dagster.io/integrations/libraries/secoda#about-secoda) --- # Dagster & Slack | Dagster Docs [Skip to main content](https://docs.dagster.io/integrations/libraries/slack#__docusaurus_skipToContent_fallback) On this page This library provides an integration with Slack to support posting messages in your company's Slack workspace. Installation[​](https://docs.dagster.io/integrations/libraries/slack#installation "Direct link to Installation") ----------------------------------------------------------------------------------------------------------------- * uv * pip uv add dagster-slack pip install dagster-slack Example[​](https://docs.dagster.io/integrations/libraries/slack#example "Direct link to Example") -------------------------------------------------------------------------------------------------- # Read the docs on Resources to learn more: https://docs.dagster.io/deployment/resourcesfrom dagster_slack import SlackResourceimport dagster as dg@dg.assetdef slack_message(slack: SlackResource): slack.get_client().chat_postMessage(channel="#noise", text=":wave: hey there!")defs = dg.Definitions( assets=[slack_message], resources={"slack": SlackResource(token=dg.EnvVar("SLACK_TOKEN"))},) About Slack[​](https://docs.dagster.io/integrations/libraries/slack#about-slack "Direct link to About Slack") -------------------------------------------------------------------------------------------------------------- The **Slack** messaging app provides chat, video and voice communication tools and is used extensively across companies and communities. The Dagster slack community can be found at [dagster.io/slack](https://dagster.io/slack) . * [Installation](https://docs.dagster.io/integrations/libraries/slack#installation) * [Example](https://docs.dagster.io/integrations/libraries/slack#example) * [About Slack](https://docs.dagster.io/integrations/libraries/slack#about-slack) --- # Dagster & Qdrant | Dagster Docs [Skip to main content](https://docs.dagster.io/integrations/libraries/qdrant#__docusaurus_skipToContent_fallback) On this page The Qdrant library lets you integrate Qdrant's vector database with Dagster, making it easy to build AI-driven data pipelines. You can run vector searches and manage data directly within Dagster. Installation[​](https://docs.dagster.io/integrations/libraries/qdrant#installation "Direct link to Installation") ------------------------------------------------------------------------------------------------------------------ * uv * pip uv add dagster-qdrant pip install dagster-qdrant Example[​](https://docs.dagster.io/integrations/libraries/qdrant#example "Direct link to Example") --------------------------------------------------------------------------------------------------- from dagster_qdrant import QdrantConfig, QdrantResourceimport dagster as dg@dg.assetdef my_table(qdrant_resource: QdrantResource): with qdrant_resource.get_client() as qdrant: qdrant.add( collection_name="test_collection", documents=[ "This is a document about oranges", "This is a document about pineapples", "This is a document about strawberries", "This is a document about cucumbers", ], ) results = qdrant.query( collection_name="test_collection", query_text="hawaii", limit=3 )defs = dg.Definitions( assets=[my_table], resources={ "qdrant_resource": QdrantResource( config=QdrantConfig( host="xyz-example.eu-central.aws.cloud.qdrant.io", api_key="", ) ) },) About Qdrant[​](https://docs.dagster.io/integrations/libraries/qdrant#about-qdrant "Direct link to About Qdrant") ------------------------------------------------------------------------------------------------------------------ Qdrant (read: quadrant) is a vector similarity search engine. It provides a production-ready service with a convenient API to store, search, and manage vectors with additional payload and extended filtering support. It makes it useful for all sorts of neural network or semantic-based matching, faceted search, and other applications. Learn more from the [Qdrant documentation](https://qdrant.tech/) . * [Installation](https://docs.dagster.io/integrations/libraries/qdrant#installation) * [Example](https://docs.dagster.io/integrations/libraries/qdrant#example) * [About Qdrant](https://docs.dagster.io/integrations/libraries/qdrant#about-qdrant) --- # Dagster & Gemini | Dagster Docs [Skip to main content](https://docs.dagster.io/integrations/libraries/gemini#__docusaurus_skipToContent_fallback) On this page The Gemini library allows you to easily interact with the Gemini REST API using the Gemini Python API to build AI steps into your Dagster pipelines. You can also log Gemini API usage metadata in Dagster Insights, giving you detailed observability on API call credit consumption. When paired with Dagster assets, the resource automatically logs Gemini usage metadata in asset metadata. Installation[​](https://docs.dagster.io/integrations/libraries/gemini#installation "Direct link to Installation") ------------------------------------------------------------------------------------------------------------------ * uv * pip uv add dagster-gemini pip install dagster-gemini Example[​](https://docs.dagster.io/integrations/libraries/gemini#example "Direct link to Example") --------------------------------------------------------------------------------------------------- from dagster_gemini import GeminiResourceimport dagster as dg@dg.asset(compute_kind="gemini")def gemini_asset(context: dg.AssetExecutionContext, gemini: GeminiResource): with gemini.get_model(context) as model: response = model.generate_content("Generate a short sentence on tests")defs = dg.Definitions( assets=[gemini_asset], resources={ "gemini": GeminiResource( api_key=dg.EnvVar("GEMINI_API_KEY"), generative_model_name="gemini-1.5-flash", ), },) About Gemini[​](https://docs.dagster.io/integrations/libraries/gemini#about-gemini "Direct link to About Gemini") ------------------------------------------------------------------------------------------------------------------ Gemini is Google's most capable AI model family, designed to be multimodal from the ground up. It can understand and combine different types of information like text, code, audio, images, and video. Gemini comes in different sizes optimized for different use cases, from the lightweight Gemini Nano for on-device tasks to the powerful Gemini Ultra for complex reasoning. The model demonstrates strong performance across language understanding, coding, reasoning, and creative tasks. * [Installation](https://docs.dagster.io/integrations/libraries/gemini#installation) * [Example](https://docs.dagster.io/integrations/libraries/gemini#example) * [About Gemini](https://docs.dagster.io/integrations/libraries/gemini#about-gemini) --- # Dagster & GCP BigQuery | Dagster Docs [Skip to main content](https://docs.dagster.io/integrations/libraries/gcp/bigquery#__docusaurus_skipToContent_fallback) On this page info This feature is considered in a beta stage. It is still being tested and may change. For more information, see the [API lifecycle stages documentation](https://docs.dagster.io/api/api-lifecycle/api-lifecycle-stages) . The Google Cloud Platform BigQuery integration allows data engineers to easily query and store data in the BigQuery data warehouse through the use of the `BigQueryResource`. ### Installation[​](https://docs.dagster.io/integrations/libraries/gcp/bigquery#installation "Direct link to Installation") * uv * pip uv add dagster-gcp pip install dagster-gcp ### Examples[​](https://docs.dagster.io/integrations/libraries/gcp/bigquery#examples "Direct link to Examples") from dagster_gcp import BigQueryResourceimport dagster as dg@dg.assetdef my_table(bigquery: BigQueryResource): with bigquery.get_client() as client: client.query("SELECT * FROM my_dataset.my_table")defs = dg.Definitions( assets=[my_table], resources={"bigquery": BigQueryResource(project="my-project")}) ### About Google Cloud Platform BigQuery[​](https://docs.dagster.io/integrations/libraries/gcp/bigquery#about-google-cloud-platform-bigquery "Direct link to About Google Cloud Platform BigQuery") The Google Cloud Platform BigQuery service, offers a fully managed enterprise data warehouse that enables fast SQL queries using the processing power of Google's infrastructure. * [Installation](https://docs.dagster.io/integrations/libraries/gcp/bigquery#installation) * [Examples](https://docs.dagster.io/integrations/libraries/gcp/bigquery#examples) * [About Google Cloud Platform BigQuery](https://docs.dagster.io/integrations/libraries/gcp/bigquery#about-google-cloud-platform-bigquery) --- # Dagster & Docker | Dagster Docs [Skip to main content](https://docs.dagster.io/integrations/libraries/docker#__docusaurus_skipToContent_fallback) On this page The Docker integration library provides the PipesDockerClient resource, enabling you to launch Docker containers and execute external code directly from Dagster assets and ops. This integration allows you to pass parameters to Docker containers while Dagster receives real-time events, such as logs, asset checks, and asset materializations, from the initiated jobs. With minimal code changes required on the job side, this integration is both efficient and easy to implement. Installation[​](https://docs.dagster.io/integrations/libraries/docker#installation "Direct link to Installation") ------------------------------------------------------------------------------------------------------------------ * uv * pip uv add dagster-docker pip install dagster-docker Example[​](https://docs.dagster.io/integrations/libraries/docker#example "Direct link to Example") --------------------------------------------------------------------------------------------------- from dagster_docker import PipesDockerClientimport dagster as dg@dg.assetdef docker_pipes_asset( context: dg.AssetExecutionContext, docker_pipes_client: PipesDockerClient): docker_image = "python:3.9-slim" return docker_pipes_client.run( image=docker_image, command=[ "python", "-m", "my_module", ], context=context, ).get_results()defs = dg.Definitions( assets=[docker_pipes_asset], resources={ "docker_pipes_client": PipesDockerClient(), },) Deploying to Docker?[​](https://docs.dagster.io/integrations/libraries/docker#deploying-to-docker "Direct link to Deploying to Docker?") ----------------------------------------------------------------------------------------------------------------------------------------- * Deploying to Dagster+: Use with a Dagster+ Hybrid deployment, the Docker agent executes Dagster jobs on a Docker cluster. Checkout the [Dagster+ Docker Agent](https://docs.dagster.io/dagster-plus/deployment/deployment-types/hybrid/docker) guide for more information. * Deploying to Open Source: Visit the [Deploying Dagster to Docker](https://docs.dagster.io/guides/deploy/deployment-options/docker) guide for more information. About Docker[​](https://docs.dagster.io/integrations/libraries/docker#about-docker "Direct link to About Docker") ------------------------------------------------------------------------------------------------------------------ **Docker** is a set of platform-as-a-service products that use OS-level virtualization to deliver software in packages called containers. The service has both free and premium tiers. The software that hosts the containers is called Docker Engine. * [Installation](https://docs.dagster.io/integrations/libraries/docker#installation) * [Example](https://docs.dagster.io/integrations/libraries/docker#example) * [Deploying to Docker?](https://docs.dagster.io/integrations/libraries/docker#deploying-to-docker) * [About Docker](https://docs.dagster.io/integrations/libraries/docker#about-docker) --- # Dagster & Prometheus | Dagster Docs [Skip to main content](https://docs.dagster.io/integrations/libraries/prometheus#__docusaurus_skipToContent_fallback) On this page info This feature is considered in a beta stage. It is still being tested and may change. For more information, see the [API lifecycle stages documentation](https://docs.dagster.io/api/api-lifecycle/api-lifecycle-stages) . This integration allows you to push metrics to the Prometheus gateway from within a Dagster pipeline. Installation[​](https://docs.dagster.io/integrations/libraries/prometheus#installation "Direct link to Installation") ---------------------------------------------------------------------------------------------------------------------- * uv * pip uv add dagster-prometheus pip install dagster-prometheus Example[​](https://docs.dagster.io/integrations/libraries/prometheus#example "Direct link to Example") ------------------------------------------------------------------------------------------------------- from dagster_prometheus import PrometheusResourceimport dagster as dg@dg.assetdef prometheus_metric(prometheus: PrometheusResource): prometheus.push_to_gateway(job="my_job_label")defs = dg.Definitions( assets=[prometheus_metric], resources={ "prometheus": PrometheusResource(gateway="http://pushgateway.example.org:9091") },) About Prometheus[​](https://docs.dagster.io/integrations/libraries/prometheus#about-prometheus "Direct link to About Prometheus") ---------------------------------------------------------------------------------------------------------------------------------- **Prometheus** is an open source systems monitoring and alerting toolkit. Originally built at SoundCloud, Prometheus joined the Cloud Native Computing Foundation in 2016 as the second hosted project, after Kubernetes. Prometheus collects and stores metrics as time series data along with the timestamp at which it was recorded, alongside optional key-value pairs called labels. * [Installation](https://docs.dagster.io/integrations/libraries/prometheus#installation) * [Example](https://docs.dagster.io/integrations/libraries/prometheus#example) * [About Prometheus](https://docs.dagster.io/integrations/libraries/prometheus#about-prometheus) --- # Dagster & TypeScript | Dagster Docs [Skip to main content](https://docs.dagster.io/integrations/libraries/typescript#__docusaurus_skipToContent_fallback) On this page The dagster-pipes-typescript npm package is a Dagster Pipes implementation for the TypeScript programming language that allows integration between any TypeScript process and the Dagster orchestrator. For more information, see the [community integrations GitHub repository](https://github.com/dagster-io/community-integrations/blob/main/libraries/pipes/implementations/typescript/README.md) . Prerequisites[​](https://docs.dagster.io/integrations/libraries/typescript#prerequisites "Direct link to Prerequisites") ------------------------------------------------------------------------------------------------------------------------- * [Install node and npm](https://nodejs.org/en/download) * Install the typescript compiler (`npm install -g typescript`) Installation[​](https://docs.dagster.io/integrations/libraries/typescript#installation "Direct link to Installation") ---------------------------------------------------------------------------------------------------------------------- `@dagster-io/dagster-pipes` is available as an npm package: npm install @dagster-io/dagster-pipes Example[​](https://docs.dagster.io/integrations/libraries/typescript#example "Direct link to Example") ------------------------------------------------------------------------------------------------------- For a usage example, see the [README](https://github.com/dagster-io/community-integrations/blob/main/libraries/pipes/implementations/typescript/README.md) in the community integrations repository. About Typescript[​](https://docs.dagster.io/integrations/libraries/typescript#about-typescript "Direct link to About Typescript") ---------------------------------------------------------------------------------------------------------------------------------- [TypeScript](https://www.typescriptlang.org/) is a strongly typed programming language that builds on JavaScript, giving you better tooling at any scale. To get started with TypeScript, see the [TypeScript docs](https://www.typescriptlang.org/docs) . * [Prerequisites](https://docs.dagster.io/integrations/libraries/typescript#prerequisites) * [Installation](https://docs.dagster.io/integrations/libraries/typescript#installation) * [Example](https://docs.dagster.io/integrations/libraries/typescript#example) * [About Typescript](https://docs.dagster.io/integrations/libraries/typescript#about-typescript) --- # Dagster & Weights & Biases | Dagster Docs [Skip to main content](https://docs.dagster.io/integrations/libraries/wandb#__docusaurus_skipToContent_fallback) On this page Use Dagster and Weights & Biases (W&B) to orchestrate your MLOps pipelines and maintain ML assets. The integration with W&B makes it easy within Dagster to: * use and create W&B Artifacts * use and create Registered Models in W&B Model Registry * run training jobs on dedicated compute using W&B Launch * use the Weights & Biases client in ops and assets The W&B Dagster integration provides a W&B-specific Dagster resource and I/O Manager: * `wandb_resource`: a Dagster resource used to authenticate and communicate to the W&B API. * `wandb_artifacts_io_manager`: a Dagster I/O Manager used to consume W&B Artifacts. Installation[​](https://docs.dagster.io/integrations/libraries/wandb#installation "Direct link to Installation") ----------------------------------------------------------------------------------------------------------------- To use this integration you will need a Weights and Biases account. Then you will need a W&B API Key, a W&B entity (user or team), and a W&B project. Full installation details can be found on [the Weights and Biases website here](https://docs.wandb.ai/guides/integrations/other/dagster) . **Note** that Weights & Biases do offer a free cloud account for personal (non-corporate) use. Check out their [pricing page](https://wandb.ai/site/pricing) for details. Example[​](https://docs.dagster.io/integrations/libraries/wandb#example "Direct link to Example") -------------------------------------------------------------------------------------------------- A complete tutorial can be found on [the Weights and Biases website here](https://docs.wandb.ai/guides/integrations/other/dagster) . About Weights & Biases[​](https://docs.dagster.io/integrations/libraries/wandb#about-weights--biases "Direct link to About Weights & Biases") ---------------------------------------------------------------------------------------------------------------------------------------------- [Weights & Biases](https://wandb.ai/site) makes it easy to track your experiments, manage & version your data, and collaborate with your team so you can focus on building the best machine learning models. * [Installation](https://docs.dagster.io/integrations/libraries/wandb#installation) * [Example](https://docs.dagster.io/integrations/libraries/wandb#example) * [About Weights & Biases](https://docs.dagster.io/integrations/libraries/wandb#about-weights--biases) --- # Dagster & Jupyter Notebooks | Dagster Docs [Skip to main content](https://docs.dagster.io/integrations/libraries/jupyter#__docusaurus_skipToContent_fallback) On this page info This feature is considered in a beta stage. It is still being tested and may change. For more information, see the [API lifecycle stages documentation](https://docs.dagster.io/api/api-lifecycle/api-lifecycle-stages) . Dagstermill eliminates the tedious "productionization" of Jupyter notebooks. Using the Dagstermill library enables you to: * View notebooks directly in the Dagster UI without needing to set up a Jupyter kernel * Define data dependencies to flow inputs and outputs from assets/ops to notebooks, between notebooks, and from notebooks to other assets/ops * Use Dagster resources and the Dagster config system inside notebooks * Aggregate notebook logs with logs from other Dagster assets and ops * Yield custom materializations and other Dagster events from your notebook code About Jupyter[​](https://docs.dagster.io/integrations/libraries/jupyter#about-jupyter "Direct link to About Jupyter") ---------------------------------------------------------------------------------------------------------------------- Fast iteration, the literate combination of arbitrary code with markdown blocks, and inline plotting make notebooks an indispensable tool for data science. The **Dagstermill** package makes it easy to run notebooks using the Dagster tools and to integrate them into data jobs with heterogeneous ops: for instance, Spark jobs, SQL statements run against a data warehouse, or arbitrary Python code. * [About Jupyter](https://docs.dagster.io/integrations/libraries/jupyter#about-jupyter) --- # Dagster & SSH/SFTP | Dagster Docs [Skip to main content](https://docs.dagster.io/integrations/libraries/ssh-sftp#__docusaurus_skipToContent_fallback) On this page info This feature is considered in a beta stage. It is still being tested and may change. For more information, see the [API lifecycle stages documentation](https://docs.dagster.io/api/api-lifecycle/api-lifecycle-stages) . This integration provides a resource for SSH remote execution using Paramiko. It allows you to establish secure connections to networked resources and execute commands remotely. The integration also provides an SFTP client for secure file transfers between the local and remote systems. Installation[​](https://docs.dagster.io/integrations/libraries/ssh-sftp#installation "Direct link to Installation") -------------------------------------------------------------------------------------------------------------------- * uv * pip uv add dagster-ssh pip install dagster-ssh Example[​](https://docs.dagster.io/integrations/libraries/ssh-sftp#example "Direct link to Example") ----------------------------------------------------------------------------------------------------- from dagster_ssh import SSHResourceimport dagster as dg@dg.assetdef ssh_asset(ssh: SSHResource): ssh.sftp_get("/path/to/remote.csv", "path/to/local.csv")defs = dg.Definitions( assets=[ssh_asset], resources={"ssh": SSHResource(remote_host="foo.com", key_file="path/to/id_rsa")},) About SSH SFTP[​](https://docs.dagster.io/integrations/libraries/ssh-sftp#about-ssh-sftp "Direct link to About SSH SFTP") -------------------------------------------------------------------------------------------------------------------------- The **SSH protocol** allows for secure remote login with strong authentication to networked resources. It protects network connections with strong encryption. The Dagster library provides direct SSH and SFTP calls from within the execution of your pipelines. * [Installation](https://docs.dagster.io/integrations/libraries/ssh-sftp#installation) * [Example](https://docs.dagster.io/integrations/libraries/ssh-sftp#example) * [About SSH SFTP](https://docs.dagster.io/integrations/libraries/ssh-sftp#about-ssh-sftp) --- # Dagster & Iceberg | Dagster Docs [Skip to main content](https://docs.dagster.io/integrations/libraries/iceberg#__docusaurus_skipToContent_fallback) On this page info This feature is considered in a preview stage, and is under active development, and not considered ready for production use. You may encounter feature gaps, and the APIs may change. For more information, see the [API lifecycle stages documentation](https://docs.dagster.io/api/api-lifecycle/api-lifecycle-stages) . This library provides I/O managers for reading and writing Apache Iceberg tables. It also provides a Dagster resource for accessing Iceberg tables. Installation[​](https://docs.dagster.io/integrations/libraries/iceberg#installation "Direct link to Installation") ------------------------------------------------------------------------------------------------------------------- * uv * pip uv add dagster-iceberg pip install dagster-iceberg `dagster-iceberg` defines the following extras for interoperability with various DataFrame libraries: * `daft` for interoperability with Daft DataFrames * `pandas` for interoperability with pandas DataFrames * `polars` for interoperability with Polars DataFrames * `spark` for interoperability with PySpark DataFrames (specifically, via Spark Connect) `pyarrow` is a core package dependency, so the [`io_manager.arrow.PyArrowIcebergIOManager`](https://docs.dagster.io/api/libraries/dagster-iceberg#dagster_iceberg.io_manager.arrow.PyArrowIcebergIOManager) is always available. Example[​](https://docs.dagster.io/integrations/libraries/iceberg#example "Direct link to Example") ---------------------------------------------------------------------------------------------------- import pyarrow as pafrom dagster_iceberg.config import IcebergCatalogConfigfrom dagster_iceberg.io_manager.arrow import PyArrowIcebergIOManagerimport dagster as dg@dg.assetdef my_table() -> pa.Table: n_legs = pa.array([2, 4, 5, 100]) animals = pa.array(["Flamingo", "Horse", "Brittle stars", "Centipede"]) names = ["n_legs", "animals"] return pa.Table.from_arrays([n_legs, animals], names=names)warehouse_path = "/tmp/warehouse"defs = dg.Definitions( assets=[my_table], resources={ "io_manager": PyArrowIcebergIOManager( name="default", config=IcebergCatalogConfig( properties={ "type": "sql", "uri": f"sqlite:///{warehouse_path}/pyiceberg_catalog.db", "warehouse": f"file://{warehouse_path}", } ), namespace="default", ) },) About Apache Iceberg[​](https://docs.dagster.io/integrations/libraries/iceberg#about-apache-iceberg "Direct link to About Apache Iceberg") ------------------------------------------------------------------------------------------------------------------------------------------- **Iceberg** is a high-performance format for huge analytic tables. It brings the reliability and simplicity of SQL tables to big data, while making it possible for multiple engines to safely work with the same tables, at the same time. * [Installation](https://docs.dagster.io/integrations/libraries/iceberg#installation) * [Example](https://docs.dagster.io/integrations/libraries/iceberg#example) * [About Apache Iceberg](https://docs.dagster.io/integrations/libraries/iceberg#about-apache-iceberg) --- # Dagster & Twilio | Dagster Docs [Skip to main content](https://docs.dagster.io/integrations/libraries/twilio#__docusaurus_skipToContent_fallback) On this page Use your Twilio Account SID and Auth Token to build Twilio tasks right into your Dagster pipeline. Installation[​](https://docs.dagster.io/integrations/libraries/twilio#installation "Direct link to Installation") ------------------------------------------------------------------------------------------------------------------ * uv * pip uv add dagster-twilio pip install dagster-twilio Example[​](https://docs.dagster.io/integrations/libraries/twilio#example "Direct link to Example") --------------------------------------------------------------------------------------------------- # Read the docs on Resources to learn more: https://docs.dagster.io/deployment/resourcesfrom dagster_twilio import TwilioResourceimport dagster as dg@dg.assetdef twilio_message(twilio: TwilioResource): twilio.get_client().messages.create( to="+15551234567", from_="+15558901234", body="Hello world!" )defs = dg.Definitions( assets=[twilio_message], resources={ "twilio": TwilioResource( account_sid=dg.EnvVar("TWILIO_ACCOUNT_SID"), auth_token=dg.EnvVar("TWILIO_AUTH_TOKEN"), ) },) About Twilio[​](https://docs.dagster.io/integrations/libraries/twilio#about-twilio "Direct link to About Twilio") ------------------------------------------------------------------------------------------------------------------ **Twilio** provides communication APIs for phone calls, text messages, and other communication functions. * [Installation](https://docs.dagster.io/integrations/libraries/twilio#installation) * [Example](https://docs.dagster.io/integrations/libraries/twilio#example) * [About Twilio](https://docs.dagster.io/integrations/libraries/twilio#about-twilio) --- # Dagster & GCP Dataproc | Dagster Docs [Skip to main content](https://docs.dagster.io/integrations/libraries/gcp/dataproc#__docusaurus_skipToContent_fallback) On this page info This feature is considered in a beta stage. It is still being tested and may change. For more information, see the [API lifecycle stages documentation](https://docs.dagster.io/api/api-lifecycle/api-lifecycle-stages) . Using this integration, you can manage and interact with Google Cloud Platform's Dataproc service directly from Dagster. This integration allows you to create, manage, and delete Dataproc clusters, and submit and monitor jobs on these clusters. Installation[​](https://docs.dagster.io/integrations/libraries/gcp/dataproc#installation "Direct link to Installation") ------------------------------------------------------------------------------------------------------------------------ * uv * pip uv add dagster-gcp pip install dagster-gcp Examples[​](https://docs.dagster.io/integrations/libraries/gcp/dataproc#examples "Direct link to Examples") ------------------------------------------------------------------------------------------------------------ from dagster_gcp import DataprocResourceimport dagster as dgdataproc_resource = DataprocResource( project_id="your-gcp-project-id", region="your-gcp-region", cluster_name="your-cluster-name", cluster_config_yaml_path="path/to/your/cluster/config.yaml",)@dg.assetdef my_dataproc_asset(dataproc: DataprocResource): client = dataproc.get_client() job_details = { "job": { "placement": {"clusterName": dataproc.cluster_name}, } } client.submit_job(job_details)defs = dg.Definitions( assets=[my_dataproc_asset], resources={"dataproc": dataproc_resource}) About Google Cloud Platform Dataproc[​](https://docs.dagster.io/integrations/libraries/gcp/dataproc#about-google-cloud-platform-dataproc "Direct link to About Google Cloud Platform Dataproc") ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ Google Cloud Platform's **Dataproc** is a fully managed and highly scalable service for running Apache Spark, Apache Hadoop, and other open source data processing frameworks. Dataproc simplifies the process of setting up and managing clusters, allowing you to focus on your data processing tasks without worrying about the underlying infrastructure. With Dataproc, you can quickly create clusters, submit jobs, and monitor their progress, all while benefiting from the scalability and reliability of Google Cloud Platform. * [Installation](https://docs.dagster.io/integrations/libraries/gcp/dataproc#installation) * [Examples](https://docs.dagster.io/integrations/libraries/gcp/dataproc#examples) * [About Google Cloud Platform Dataproc](https://docs.dagster.io/integrations/libraries/gcp/dataproc#about-google-cloud-platform-dataproc) --- # Dagster & Pandera | Dagster Docs [Skip to main content](https://docs.dagster.io/integrations/libraries/pandera#__docusaurus_skipToContent_fallback) On this page info This feature is considered in a beta stage. It is still being tested and may change. For more information, see the [API lifecycle stages documentation](https://docs.dagster.io/api/api-lifecycle/api-lifecycle-stages) . The Pandera integration library provides an API for generating Dagster Types from Pandera dataframe schemas. Like all Dagster types, Pandera-generated types can be used to annotate op inputs and outputs. Using Pandera with Dagster allows you to: * Visualize the shape of the data by displaying dataframe structure information in the Dagster UI * Implement runtime type-checking with rich error reporting Limitations[​](https://docs.dagster.io/integrations/libraries/pandera#limitations "Direct link to Limitations") ---------------------------------------------------------------------------------------------------------------- Currently, `dagster-pandera` only supports pandas and Polars dataframes, despite Pandera supporting validation on other dataframe backends. Prerequisites[​](https://docs.dagster.io/integrations/libraries/pandera#prerequisites "Direct link to Prerequisites") ---------------------------------------------------------------------------------------------------------------------- To get started, you'll need: * [To install](https://docs.dagster.io/getting-started/installation) the `dagster` and `dagster-pandera` Python packages: * uv * pip uv add dagster-pandera pip install dagster-pandera * Familiarity with \[Dagster Types\](/api/dagster/types Usage[​](https://docs.dagster.io/integrations/libraries/pandera#usage "Direct link to Usage") ---------------------------------------------------------------------------------------------- The `dagster-pandera` library exposes only a single public function, `pandera_schema_to_dagster_type`, which generates Dagster types from Pandera schemas. The Dagster type wraps the Pandera schema and invokes the schema's `validate()` method inside its type check function. import randomimport pandas as pdimport pandera as pafrom dagster_pandera import pandera_schema_to_dagster_typefrom pandera.typing import Seriesfrom dagster import Out, job, opAPPLE_STOCK_PRICES = { "name": ["AAPL", "AAPL", "AAPL", "AAPL", "AAPL"], "date": ["2018-01-22", "2018-01-23", "2018-01-24", "2018-01-25", "2018-01-26"], "open": [177.3, 177.3, 177.25, 174.50, 172.0], "close": [177.0, 177.04, 174.22, 171.11, 171.51],}class StockPrices(pa.DataFrameModel): """Open/close prices for one or more stocks by day.""" name: Series[str] = pa.Field(description="Ticker symbol of stock") date: Series[str] = pa.Field(description="Date of prices") open: Series[float] = pa.Field(ge=0, description="Price at market open") close: Series[float] = pa.Field(ge=0, description="Price at market close")@op(out=Out(dagster_type=pandera_schema_to_dagster_type(StockPrices)))def apple_stock_prices_dirty(): prices = pd.DataFrame(APPLE_STOCK_PRICES) i = random.choice(prices.index) prices.loc[i, "open"] = pd.NA prices.loc[i, "close"] = pd.NA return prices@jobdef stocks_job(): apple_stock_prices_dirty() In the above example, we defined a toy job (`stocks_job`) with a single asset, `apple_stock_prices_dirty`. This asset returns a pandas `DataFrame` containing the opening and closing prices of Apple stock (AAPL) for a random week. The `_dirty` suffix is included because we've corrupted the data with a few random nulls. Let's look at this job in the UI: ![Pandera job in the Dagster UI](https://docs.dagster.io/assets/images/schema-6ce9b793f24b5e7663df10eb5299a827.png) Notice that information from the `StockPrices` Pandera schema is rendered in the asset detail area of the right sidebar. This is possible because `pandera_schema_to_dagster_type` extracts this information from the Pandera schema and attaches it to the returned Dagster type. If we try to run `stocks_job`, our run will fail. This is expected, as our (dirty) data contains nulls and Pandera columns are non-nullable by default. The [Dagster Typ](https://docs.dagster.io/api/dagster/types) returned by `pandera_schema_to_dagster_type` contains a type check function that calls `StockPrices.validate()`. This is invoked automatically on the return value of `apple_stock_prices_dirty`, leading to a type check failure. Notice the `STEP_OUTPUT` event in the following screenshot to see Pandera's full output: ![Error report for a Pandera job in the Dagster UI](https://docs.dagster.io/assets/images/error-report-20d13db09715960041884f590aa13fcd.png) About Pandera[​](https://docs.dagster.io/integrations/libraries/pandera#about-pandera "Direct link to About Pandera") ---------------------------------------------------------------------------------------------------------------------- **Pandera** is a statistical data testing toolkit, and a data validation library for scientists, engineers, and analysts seeking correctness. * [Limitations](https://docs.dagster.io/integrations/libraries/pandera#limitations) * [Prerequisites](https://docs.dagster.io/integrations/libraries/pandera#prerequisites) * [Usage](https://docs.dagster.io/integrations/libraries/pandera#usage) * [About Pandera](https://docs.dagster.io/integrations/libraries/pandera#about-pandera) --- # Dagster & Weaviate | Dagster Docs [Skip to main content](https://docs.dagster.io/integrations/libraries/weaviate#__docusaurus_skipToContent_fallback) On this page The Weaviate library allows you to easily interact with Weaviate's vector database capabilities to build AI-powered data pipelines in Dagster. You can perform vector similarity searches, manage schemas, and handle data operations directly from your Dagster assets. Installation[​](https://docs.dagster.io/integrations/libraries/weaviate#installation "Direct link to Installation") -------------------------------------------------------------------------------------------------------------------- * uv * pip uv add dagster-weaviate pip install dagster-weaviate Examples[​](https://docs.dagster.io/integrations/libraries/weaviate#examples "Direct link to Examples") -------------------------------------------------------------------------------------------------------- from dagster_weaviate import CloudConfig, WeaviateResourceimport dagster as dg@dg.assetdef my_table(weaviate: WeaviateResource): with weaviate.get_client() as weaviate_client: questions = weaviate_client.collections.get("Question") questions.query.near_text(query="biology", limit=2)defs = dg.Definitions( assets=[my_table], resources={ "weaviate": WeaviateResource( connection_config=CloudConfig(cluster_url=dg.EnvVar("WCD_URL")), auth_credentials={"api_key": dg.EnvVar("WCD_API_KEY")}, headers={ "X-Cohere-Api-Key": dg.EnvVar("COHERE_API_KEY"), }, ), },) About Weaviate[​](https://docs.dagster.io/integrations/libraries/weaviate#about-weaviate "Direct link to About Weaviate") -------------------------------------------------------------------------------------------------------------------------- **Weaviate** is an open-source vector database that enables you to store and manage vector embeddings at scale. You can start with a small dataset and scale up as your needs grow. This enables you to build powerful AI applications with semantic search and similarity matching capabilities. Weaviate offers fast query performance using vector-based search and GraphQL APIs, making it a powerful tool for AI-powered applications and machine learning workflows. * [Installation](https://docs.dagster.io/integrations/libraries/weaviate#installation) * [Examples](https://docs.dagster.io/integrations/libraries/weaviate#examples) * [About Weaviate](https://docs.dagster.io/integrations/libraries/weaviate#about-weaviate) --- # dagstermill integration reference | Dagster Docs [Skip to main content](https://docs.dagster.io/integrations/libraries/jupyter/reference#__docusaurus_skipToContent_fallback) On this page This reference provides a high-level look at working with Jupyter notebooks using the [`dagstermill` integration library](https://docs.dagster.io/api/libraries/dagstermill) . For a step-by-step implementation walkthrough, refer to the [Using notebooks with Dagster tutorial](https://docs.dagster.io/integrations/libraries/jupyter/using-notebooks-with-dagster) . Notebooks as assets[​](https://docs.dagster.io/integrations/libraries/jupyter/reference#notebooks-as-assets "Direct link to Notebooks as assets") -------------------------------------------------------------------------------------------------------------------------------------------------- To load a Jupyter notebook as a Dagster [asset](https://docs.dagster.io/guides/build/assets/defining-assets) , use [`define_dagstermill_asset`](https://docs.dagster.io/api/libraries/dagstermill#dagstermill.define_dagstermill_asset) : from dagstermill import define_dagstermill_assetfrom dagster import file_relative_pathiris_kmeans_notebook = define_dagstermill_asset( name="iris_kmeans", notebook_path=file_relative_path(__file__, "../notebooks/iris-kmeans.ipynb"),) In this code block, we use `define_dagstermill_asset` to create a Dagster asset. We provide the name for the asset with the `name` parameter and the path to our `.ipynb` file with the `notebook_path` parameter. The resulting asset will execute our notebook and store the resulting `.ipynb` file in a persistent location. Notebooks as ops[​](https://docs.dagster.io/integrations/libraries/jupyter/reference#notebooks-as-ops "Direct link to Notebooks as ops") ----------------------------------------------------------------------------------------------------------------------------------------- Dagstermill also supports running Jupyter notebooks as [ops](https://docs.dagster.io/guides/build/ops) . We can use [`define_dagstermill_op`](https://docs.dagster.io/api/libraries/dagstermill#dagstermill.define_dagstermill_op) to turn a notebook into an op. from dagstermill import ConfigurableLocalOutputNotebookIOManager, define_dagstermill_opfrom dagster import file_relative_path, jobk_means_iris = define_dagstermill_op( name="k_means_iris", notebook_path=file_relative_path(__file__, "./notebooks/iris-kmeans.ipynb"), output_notebook_name="iris_kmeans_output",)@job( resource_defs={ "output_notebook_io_manager": ConfigurableLocalOutputNotebookIOManager(), })def iris_classify(): k_means_iris() In this code block, we use `define_dagstermill_op` to create an op that will execute the Jupyter notebook. We give the op the name `k_means_iris`, and provide the path to the notebook file. We also specify `output_notebook_name=iris_kmeans_output`. This means that the executed notebook will be returned in a buffered file object as one of the outputs of the op, and that output will have the name `iris_kmeans_output`. We then include the `k_means_iris` op in the `iris_classify` [job](https://docs.dagster.io/guides/build/jobs) and specify the `ConfigurableLocalOutputNotebookIOManager` as the `output_notebook_io_manager` to store the executed notebook file. Notebook context[​](https://docs.dagster.io/integrations/libraries/jupyter/reference#notebook-context "Direct link to Notebook context") ----------------------------------------------------------------------------------------------------------------------------------------- If you look at one of the notebooks executed by Dagster, you'll notice that the `injected-parameters` cell in your output notebooks defines a variable called `context`. This context object mirrors the execution context object that's available in the body of any other asset or op's compute function. As with the parameters that `dagstermill` injects, you can also construct a context object for interactive exploration and development by using the `dagstermill.get_context` API in the tagged `parameters` cell of your input notebook. When Dagster executes your notebook, this development context will be replaced with the injected runtime context. You can use the development context to access asset and op config and resources, to log messages, and to yield results and other Dagster events just as you would in production. When the runtime context is injected by Dagster, none of your other code needs to change. For instance, suppose we want to make the number of clusters (the _k_ in k-means) configurable. We'll change our asset definition to include a config field: from dagstermill import define_dagstermill_assetfrom dagster import AssetIn, Field, Int, file_relative_pathiris_kmeans_jupyter_notebook = define_dagstermill_asset( name="iris_kmeans_jupyter", notebook_path=file_relative_path(__file__, "./notebooks/iris-kmeans.ipynb"), group_name="template_tutorial", ins={"iris": AssetIn("iris_dataset")}, config_schema=Field( Int, default_value=3, is_required=False, description="The number of clusters to find", ),) You can also provide `config_schema` to `define_dagstermill_op` in the same way demonstrated in this code snippet. In our notebook, we'll stub the context as follows (in the `parameters` cell): import dagstermillcontext = dagstermill.get_context(op_config=3) Now we can use our config value in our estimator. In production, this will be replaced by the config value provided to the job: estimator = sklearn.cluster.KMeans(n_clusters=context.op_config) Results and custom materializations[​](https://docs.dagster.io/integrations/libraries/jupyter/reference#results-and-custom-materializations "Direct link to Results and custom materializations") -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- note The functionality described in this section only works for notebooks run with `define_dagstermill_op`. If you'd like adding this feature to `define_dagstermill_asset` to be prioritized, give this [GitHub issue](https://github.com/dagster-io/dagster/issues/10557) a thumbs up. If you are using `define_dagstermill_op` and you'd like to yield a result to be consumed downstream of a notebook, you can call [`yield_result`](https://docs.dagster.io/api/libraries/dagstermill#dagstermill.yield_result) with the value of the result and its name. In interactive execution, this is a no-op, so you don't need to change anything when moving from interactive exploration and development to production. # my_notebook.ipynbimport dagstermilldagstermill.yield_result(3, output_name="my_output") And then: from dagstermill import ConfigurableLocalOutputNotebookIOManager, define_dagstermill_opfrom dagster import Out, file_relative_path, job, opmy_notebook_op = define_dagstermill_op( name="my_notebook", notebook_path=file_relative_path(__file__, "./notebooks/my_notebook.ipynb"), output_notebook_name="output_notebook", outs={"my_output": Out(int)},)@opdef add_two(x): return x + 2@job( resource_defs={ "output_notebook_io_manager": ConfigurableLocalOutputNotebookIOManager(), })def my_job(): three, _ = my_notebook_op() add_two(three) Dagster events[​](https://docs.dagster.io/integrations/libraries/jupyter/reference#dagster-events "Direct link to Dagster events") ----------------------------------------------------------------------------------------------------------------------------------- You can also yield Dagster events from your notebook using [`yield_event`](https://docs.dagster.io/api/libraries/dagstermill#dagstermill.yield_event) . For example, if you'd like to yield a custom [`AssetMaterialization`](https://docs.dagster.io/api/dagster/ops#dagster.AssetMaterialization) object (for instance, to tell the Dagster UI where you've saved a plot), you can do the following: import dagstermillfrom dagster import AssetMaterializationdagstermill.yield_event(AssetMaterialization(asset_key="marketing_data_plotted")) * [Notebooks as assets](https://docs.dagster.io/integrations/libraries/jupyter/reference#notebooks-as-assets) * [Notebooks as ops](https://docs.dagster.io/integrations/libraries/jupyter/reference#notebooks-as-ops) * [Notebook context](https://docs.dagster.io/integrations/libraries/jupyter/reference#notebook-context) * [Results and custom materializations](https://docs.dagster.io/integrations/libraries/jupyter/reference#results-and-custom-materializations) * [Dagster events](https://docs.dagster.io/integrations/libraries/jupyter/reference#dagster-events) --- # Usage | Dagster Docs [Skip to main content](https://docs.dagster.io/integrations/libraries/iceberg/usage#__docusaurus_skipToContent_fallback) On this page This guide walks through common scenarios for using Iceberg with Dagster. Selecting specific columns in a downstream asset[​](https://docs.dagster.io/integrations/libraries/iceberg/usage#selecting-specific-columns-in-a-downstream-asset "Direct link to Selecting specific columns in a downstream asset") ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- At times, you might prefer not to retrieve an entire table for a downstream asset. The Iceberg I/O manager allows you to load specific columns by providing metadata related to the downstream asset: import pandas as pdfrom dagster_iceberg.config import IcebergCatalogConfigfrom dagster_iceberg.io_manager.pandas import PandasIcebergIOManagerfrom dagster import AssetIn, Definitions, assetCATALOG_URI = "sqlite:////home/vscode/workspace/.tmp/examples/select_columns/catalog.db"CATALOG_WAREHOUSE = ( "file:///home/vscode/workspace/.tmp/examples/select_columns/warehouse")resources = { "io_manager": PandasIcebergIOManager( name="test", config=IcebergCatalogConfig( properties={"uri": CATALOG_URI, "warehouse": CATALOG_WAREHOUSE} ), namespace="dagster", )}@assetdef iris_dataset() -> pd.DataFrame: return pd.read_csv( "https://docs.dagster.io/assets/iris.csv", names=[ "sepal_length_cm", "sepal_width_cm", "petal_length_cm", "petal_width_cm", "species", ], )@asset( ins={ "iris_sepal": AssetIn( key="iris_dataset", metadata={"columns": ["sepal_length_cm", "sepal_width_cm"]}, ) })def sepal_data(iris_sepal: pd.DataFrame) -> pd.DataFrame: iris_sepal["sepal_area_cm2"] = ( iris_sepal["sepal_length_cm"] * iris_sepal["sepal_width_cm"] ) return iris_sepaldefs = Definitions(assets=[iris_dataset, sepal_data], resources=resources) In this example, we focus exclusively on the columns containing sepal data from the `iris_dataset` table. To select specific columns, we can include metadata in the input asset. This is done using the `metadata` parameter of the [`AssetIn`](https://docs.dagster.io/api/dagster/assets#dagster.AssetIn) that loads the `iris_dataset` asset within the `ins` parameter. We provide the key `columns` along with a list of the desired column names. When Dagster materializes `sepal_data` and retrieves the `iris_dataset` asset via the Iceberg I/O manager, it will only extract the `sepal_length_cm` and `sepal_width_cm` columns from the `iris/iris_dataset` table and make them available in `sepal_data` as a pandas DataFrame. Storing partitioned assets[​](https://docs.dagster.io/integrations/libraries/iceberg/usage#storing-partitioned-assets "Direct link to Storing partitioned assets") ------------------------------------------------------------------------------------------------------------------------------------------------------------------- The Iceberg I/O manager facilitates the storage and retrieval of partitioned data. To effectively manage data in the Iceberg table, it is essential for the Iceberg I/O manager to identify the column that specifies the partition boundaries. This information allows the I/O manager to formulate the appropriate queries for selecting or replacing data. Below, we outline how the I/O manager generates these queries for various partition types. Configuring partition dimensions For partitioning to function correctly, the partition dimension must correspond to one of the partition columns defined in the Iceberg table. Tables created through the I/O manager will be configured accordingly. * Static partitions * Time-based partitions * Multi-dimensional partitions To save static-partitioned assets in your Iceberg table, you need to set the `partition_expr` metadata on the asset. This informs the Iceberg I/O manager which column holds the partition data: import pandas as pdfrom dagster_iceberg.config import IcebergCatalogConfigfrom dagster_iceberg.io_manager.pandas import PandasIcebergIOManagerfrom dagster import Definitions, StaticPartitionsDefinition, assetCATALOG_URI = "sqlite:////home/vscode/workspace/.tmp/examples/catalog.db"CATALOG_WAREHOUSE = "file:///home/vscode/workspace/.tmp/examples/warehouse"resources = { "io_manager": PandasIcebergIOManager( name="test", config=IcebergCatalogConfig( properties={"uri": CATALOG_URI, "warehouse": CATALOG_WAREHOUSE} ), namespace="dagster", )}@asset( partitions_def=StaticPartitionsDefinition( ["Iris-setosa", "Iris-virginica", "Iris-versicolor"] ), metadata={"partition_expr": "species"},)def iris_dataset_partitioned(context) -> pd.DataFrame: species = context.partition_key full_df = pd.read_csv( "https://docs.dagster.io/assets/iris.csv", names=[ "sepal_length_cm", "sepal_width_cm", "petal_length_cm", "petal_width_cm", "species", ], ) return full_df[full_df["species"] == species]@assetdef iris_cleaned(iris_dataset_partitioned: pd.DataFrame): return iris_dataset_partitioned.dropna().drop_duplicates()defs = Definitions(assets=[iris_dataset_partitioned, iris_cleaned], resources=resources) Dagster uses the `partition_expr` metadata to create the necessary function parameters when retrieving the partition in the downstream asset. For static partitions, this is roughly equivalent to the following SQL query: SELECT *WHERE [partition_expr] IN ([selected partitions]) A partition must be specified when materializing the above assets, as explained in the [Materializing partitioned assets](https://docs.dagster.io/guides/build/partitions-and-backfills/partitioning-assets#materializing-partitioned-assets) documentation. For instance, the query used to materialize the `Iris-setosa` partition of the assets would be: SELECT *WHERE species = 'Iris-setosa' Like static-partitioned assets, you can specify `partition_expr` metadata on the asset to tell the Iceberg I/O manager which column contains the partition data: import datetime as dtimport randomimport pandas as pdfrom dagster_iceberg.config import IcebergCatalogConfigfrom dagster_iceberg.io_manager.pandas import PandasIcebergIOManagerfrom dagster import DailyPartitionsDefinition, Definitions, assetCATALOG_URI = "sqlite:////home/vscode/workspace/.tmp/examples/catalog.db"CATALOG_WAREHOUSE = "file:///home/vscode/workspace/.tmp/examples/warehouse"resources = { "io_manager": PandasIcebergIOManager( name="test", config=IcebergCatalogConfig( properties={"uri": CATALOG_URI, "warehouse": CATALOG_WAREHOUSE} ), namespace="dagster", )}def get_iris_data_for_date(partition: str) -> pd.DataFrame: random.seed(876) N = 1440 d = { "timestamp": [dt.date.fromisoformat(partition)], "species": [ random.choice(["Iris-setosa", "Iris-virginica", "Iris-versicolor"]) for _ in range(N) ], "sepal_length_cm": [random.uniform(0, 1) for _ in range(N)], "sepal_width_cm": [random.uniform(0, 1) for _ in range(N)], "petal_length_cm": [random.uniform(0, 1) for _ in range(N)], "petal_width_cm": [random.uniform(0, 1) for _ in range(N)], } return pd.DataFrame.from_dict(d)@asset( partitions_def=DailyPartitionsDefinition(start_date="2023-01-01"), metadata={"partition_expr": "time"},)def iris_data_per_day(context) -> pd.DataFrame: partition = context.partition_key # get_iris_data_for_date fetches all of the iris data for a given date, # the returned dataframe contains a column named 'time' with that stores # the time of the row as an integer of seconds since epoch return get_iris_data_for_date(partition)@assetdef iris_cleaned(iris_data_per_day: pd.DataFrame): return iris_data_per_day.dropna().drop_duplicates()defs = Definitions(assets=[iris_data_per_day, iris_cleaned], resources=resources) Dagster uses the `partition_expr` metadata to craft the `SELECT` statement when loading the correct partition in the downstream asset. When loading a dynamic partition, the following statement is used: SELECT *WHERE [partition_expr] = [partition_start] A partition must be selected when materializing the above assets, as described in the [Materializing partitioned assets](https://docs.dagster.io/guides/build/partitions-and-backfills/partitioning-assets#materializing-partitioned-assets) documentation. The `[partition_start]` and `[partition_end]` bounds are of the form `YYYY-MM-DD HH:MM:SS`. In this example, the query when materializing the `2023-01-02` partition of the above assets would be: SELECT *WHERE time = '2023-01-02 00:00:00' The Iceberg I/O manager can also store data partitioned on multiple dimensions. To do this, specify the column for each partition as a dictionary of `partition_expr` metadata: import datetime as dtimport randomimport pandas as pdfrom dagster_iceberg.config import IcebergCatalogConfigfrom dagster_iceberg.io_manager.pandas import PandasIcebergIOManagerfrom dagster import ( DailyPartitionsDefinition, Definitions, MultiPartitionsDefinition, StaticPartitionsDefinition, asset,)CATALOG_URI = "sqlite:////home/vscode/workspace/.tmp/examples/catalog.db"CATALOG_WAREHOUSE = "file:///home/vscode/workspace/.tmp/examples/warehouse"resources = { "io_manager": PandasIcebergIOManager( name="test", config=IcebergCatalogConfig( properties={"uri": CATALOG_URI, "warehouse": CATALOG_WAREHOUSE} ), namespace="dagster", )}def get_iris_data_for_date(partition: str) -> pd.DataFrame: random.seed(876) N = 1440 d = { "timestamp": [dt.date.fromisoformat(partition)], "species": [ random.choice(["Iris-setosa", "Iris-virginica", "Iris-versicolor"]) for _ in range(N) ], "sepal_length_cm": [random.uniform(0, 1) for _ in range(N)], "sepal_width_cm": [random.uniform(0, 1) for _ in range(N)], "petal_length_cm": [random.uniform(0, 1) for _ in range(N)], "petal_width_cm": [random.uniform(0, 1) for _ in range(N)], } return pd.DataFrame.from_dict(d)@asset( partitions_def=MultiPartitionsDefinition( { "date": DailyPartitionsDefinition(start_date="2023-01-01"), "species": StaticPartitionsDefinition( ["Iris-setosa", "Iris-virginica", "Iris-versicolor"] ), } ), metadata={"partition_expr": {"date": "time", "species": "species"}},)def iris_dataset_partitioned(context) -> pd.DataFrame: partition = context.partition_key.keys_by_dimension species = partition["species"] date = partition["date"] # get_iris_data_for_date fetches all of the iris data for a given date, # the returned dataframe contains a column named 'time' with that stores # the time of the row as an integer of seconds since epoch full_df = get_iris_data_for_date(date) return full_df[full_df["species"] == species]@assetdef iris_cleaned(iris_dataset_partitioned: pd.DataFrame): return iris_dataset_partitioned.dropna().drop_duplicates()defs = Definitions(assets=[iris_dataset_partitioned, iris_cleaned], resources=resources) Dagster uses the `partition_expr` metadata to craft the `SELECT` statement when loading the correct partition in a downstream asset. For multi-dimensional partitions, Dagster concatenates the `WHERE` statements described in the static and time-based cases to craft the correct `SELECT` statement. A partition must be selected when materializing the above assets, as described in the [Materializing partitioned assets](https://docs.dagster.io/guides/build/partitions-and-backfills/partitioning-assets#materializing-partitioned-assets) documentation. For example, when materializing the `2023-01-02|Iris-setosa` partition of the above assets, the following query will be used: SELECT *WHERE species = 'Iris-setosa' AND time = '2023-01-02 00:00:00' Storing tables in multiple schemas[​](https://docs.dagster.io/integrations/libraries/iceberg/usage#storing-tables-in-multiple-schemas "Direct link to Storing tables in multiple schemas") ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- You may want to have different assets stored in different Iceberg schemas. The Iceberg I/O manager allows you to specify the schema in several ways. If you want all of your assets to be stored in the same schema, you can specify the schema as configuration to the I/O manager. If you want to store assets in different schemas, you can specify the schema as part of the asset key: import pandas as pdfrom dagster_iceberg.config import IcebergCatalogConfigfrom dagster_iceberg.io_manager.pandas import PandasIcebergIOManagerfrom dagster import Definitions, assetCATALOG_URI = "sqlite:////home/vscode/workspace/.tmp/examples/catalog.db"CATALOG_WAREHOUSE = "file:///home/vscode/workspace/.tmp/examples/warehouse"resources = { "io_manager": PandasIcebergIOManager( name="test", config=IcebergCatalogConfig( properties={"uri": CATALOG_URI, "warehouse": CATALOG_WAREHOUSE} ), namespace="dagster", )}@asset(key_prefix=["iris"]) # will be stored in "iris" schemadef iris_dataset() -> pd.DataFrame: return pd.read_csv( "https://docs.dagster.io/assets/iris.csv", names=[ "sepal_length_cm", "sepal_width_cm", "petal_length_cm", "petal_width_cm", "species", ], )@asset(key_prefix=["wine"]) # will be stored in "wine" schemadef wine_dataset() -> pd.DataFrame: return pd.read_csv( "https://gist.githubusercontent.com/tijptjik/9408623/raw/b237fa5848349a14a14e5d4107dc7897c21951f5/wine.csv", names=[ "fixed_acidity", "volatile_acidity", "citric_acid", "residual_sugar", "chlorides", "free_sulfur_dioxide", "total_sulfur_dioxide", "density", "ph", "sulphates", "alcohol", "quality", ], )defs = Definitions(assets=[iris_dataset, wine_dataset], resources=resources) In this example, the `iris_dataset` asset will be stored in the `iris` schema, and the `daffodil_dataset` asset will be found in the `daffodil` schema. Specifying a schema The two options for specifying schema are mutually exclusive. If you provide `schema` configuration to the I/O manager, you cannot also provide it via the asset key, and vice versa. If no `schema` is provided, either from configuration or asset keys, the default `public` schema will be used. Using the Iceberg I/O manager with other I/O managers[​](https://docs.dagster.io/integrations/libraries/iceberg/usage#using-the-iceberg-io-manager-with-other-io-managers "Direct link to Using the Iceberg I/O manager with other I/O managers") -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- You may have assets that you don't want to store in Iceberg. You can provide an I/O manager to each asset using the `io_manager_key` parameter in the [`@dg.asset`](https://docs.dagster.io/api/dagster/assets#dagster.asset) decorator: import pandas as pdfrom dagster_iceberg.config import IcebergCatalogConfigfrom dagster_iceberg.io_manager.pandas import PandasIcebergIOManagerfrom dagster import Definitions, FilesystemIOManager, assetCATALOG_URI = "sqlite:////home/vscode/workspace/.tmp/examples/catalog.db"CATALOG_WAREHOUSE = "file:///home/vscode/workspace/.tmp/examples/warehouse"FS_BASE_DIR = "/home/vscode/workspace/.tmp/examples/images"resources = { "dwh_io_manager": PandasIcebergIOManager( name="test", config=IcebergCatalogConfig( properties={"uri": CATALOG_URI, "warehouse": CATALOG_WAREHOUSE} ), namespace="dagster", ), "blob_io_manager": FilesystemIOManager(base_dir=FS_BASE_DIR),}@asset(io_manager_key="dwh_io_manager")def iris_dataset() -> pd.DataFrame: return pd.read_csv( "https://docs.dagster.io/assets/iris.csv", names=[ "sepal_length_cm", "sepal_width_cm", "petal_length_cm", "petal_width_cm", "species", ], )@asset(io_manager_key="blob_io_manager")def iris_plots(iris_dataset: pd.DataFrame): # plot_data is a function we've defined somewhere else # that plots the data in a DataFrame return iris_dataset["sepal_length_cm"].plot.hist()defs = Definitions(assets=[iris_dataset, iris_plots], resources=resources) In the above example: * The `iris_dataset` asset uses the I/O manager bound to the key `warehouse_io_manager`, and `iris_plots` uses the I/O manager bound to the key `blob_io_manager`. * We define the I/O managers for those keys in the [`Definitions`](https://docs.dagster.io/api/dagster/definitions#dagster.Definitions) object. * When the assets are materialized, the `iris_dataset` will be stored in Iceberg, and `iris_plots` will be saved in Amazon S3. Using different compute engines to read from and write to Iceberg[​](https://docs.dagster.io/integrations/libraries/iceberg/usage#using-different-compute-engines-to-read-from-and-write-to-iceberg "Direct link to Using different compute engines to read from and write to Iceberg") ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- `dagster-iceberg` supports several compute engines out-of-the-box. You can [find examples of how to use each engine in the API docs](https://docs.dagster.io/api/libraries/dagster-iceberg#io-managers) . Executing custom SQL commands[​](https://docs.dagster.io/integrations/libraries/iceberg/usage#executing-custom-sql-commands "Direct link to Executing custom SQL commands") ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- In addition to the Iceberg I/O manager, Dagster also provides an [`resource.IcebergTableResource`](https://docs.dagster.io/api/libraries/dagster-iceberg#dagster_iceberg.resource.IcebergTableResource) for executing custom SQL queries. import pandas as pdfrom dagster_iceberg.config import IcebergCatalogConfigfrom dagster_iceberg.resource import IcebergTableResourcefrom dagster import Definitions, assetCATALOG_URI = "sqlite:////home/vscode/workspace/.tmp/examples/catalog.db"CATALOG_WAREHOUSE = "file:///home/vscode/workspace/.tmp/examples/warehouse"@assetdef small_petals(iceberg: IcebergTableResource) -> pd.DataFrame: return iceberg.load().scan().to_pandas()defs = Definitions( assets=[small_petals], resources={ "iceberg": IcebergTableResource( name="test", config=IcebergCatalogConfig( properties={"uri": CATALOG_URI, "warehouse": CATALOG_WAREHOUSE} ), namespace="dagster", table="ingested_data", # assuming that `ingested_data` Iceberg table exists ) },) In this example, we attach the resource to the `small_petals` asset. In the body of the asset function, we use the `load()` method to retrieve the Iceberg table object, which can then be used for further processing. Configuring table behavior using table properties[​](https://docs.dagster.io/integrations/libraries/iceberg/usage#configuring-table-behavior-using-table-properties "Direct link to Configuring table behavior using table properties") ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- PyIceberg tables support table properties to configure table behavior. You can find a [full list of properties in the PyIceberg documentation](https://py.iceberg.apache.org/configuration) . Use asset metadata to set table properties: import pandas as pdfrom dagster_iceberg.config import IcebergCatalogConfigfrom dagster_iceberg.io_manager.pandas import PandasIcebergIOManagerfrom dagster import Definitions, assetCATALOG_URI = "sqlite:////home/vscode/workspace/.tmp/examples/select_columns/catalog.db"CATALOG_WAREHOUSE = ( "file:///home/vscode/workspace/.tmp/examples/select_columns/warehouse")resources = { "io_manager": PandasIcebergIOManager( name="test", config=IcebergCatalogConfig( properties={"uri": CATALOG_URI, "warehouse": CATALOG_WAREHOUSE} ), namespace="dagster", )}@assetdef iris_dataset() -> pd.DataFrame: return pd.read_csv( "https://docs.dagster.io/assets/iris.csv", names=[ "sepal_length_cm", "sepal_width_cm", "petal_length_cm", "petal_width_cm", "species", ], )@asset( metadata={ "table_properties": { "write.parquet.page-size-bytes": "2097152", # 2MB "write.parquet.page-row-limit": "10000", } })def sepal_data(iris_sepal: pd.DataFrame) -> pd.DataFrame: iris_sepal["sepal_area_cm2"] = ( iris_sepal["sepal_length_cm"] * iris_sepal["sepal_width_cm"] ) return iris_sepaldefs = Definitions(assets=[iris_dataset, sepal_data], resources=resources) Allowing updates to schema and partitions[​](https://docs.dagster.io/integrations/libraries/iceberg/usage#allowing-updates-to-schema-and-partitions "Direct link to Allowing updates to schema and partitions") ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- By default, assets will error when you change the partition spec (e.g. if you change a partition from hourly to daily) or the schema (e.g. when you add a column). You can allow updates to an asset's partition spec and/or schema by setting `partition_spec_update_mode` and/or `schema_update_mode`, respectively, on the asset metadata: @asset( partitions_def=MultiPartitionsDefinition( { "date": DailyPartitionsDefinition(start_date="2023-01-01"), "species": StaticPartitionsDefinition( ["Iris-setosa", "Iris-virginica", "Iris-versicolor"] ), } ), metadata={ "partition_expr": {"date": "time", "species": "species"}, "partition_spec_update_mode": "update", "schema_update_mode": "update", },)def iris_dataset_partitioned(context) -> pd.DataFrame: ... Using the custom I/O manager[​](https://docs.dagster.io/integrations/libraries/iceberg/usage#using-the-custom-io-manager "Direct link to Using the custom I/O manager") ------------------------------------------------------------------------------------------------------------------------------------------------------------------------ The `dagster-iceberg` library leans heavily on Dagster's `DbIOManager` implementation. However, this I/O manager comes with some limitations, such as the lack of support for various [partition mappings](https://docs.dagster.io/_apidocs/partitions#partition-mapping) . A custom (experimental) `DbIOManager` implementation is available that supports partition mappings as long as any time-based partition is _consecutive_ and static partitions are of string type. You can enable it as follows: resources = { "io_manager": PyArrowIcebergIOManager( name="my_catalog", config=IcebergCatalogConfig( properties={ "type": "sql", "uri": f"sqlite:///{warehouse_path}/pyiceberg_catalog.db", "warehouse": f"file://{warehouse_path}", } ), namespace="my_schema", )} For example, a [`MultiToSingleDimensionPartitionMapping`](https://docs.dagster.io/api/dagster/partitions#dagster.MultiToSingleDimensionPartitionMapping) is supported: @asset( key_prefix=["my_schema"], partitions_def=DailyPartitionsDefinition(start_date="2022-01-01"), ins={ "multi_partitioned_asset": AssetIn( ["my_schema", "multi_partitioned_asset_1"], partition_mapping=MultiToSingleDimensionPartitionMapping( partition_dimension_name="date" ), ) }, metadata={ "partition_expr": "date_column", },)def single_partitioned_asset_date(multi_partitioned_asset: pa.Table) -> pa.Table: ... However, a [`SpecificPartitionsPartitionMapping`](https://docs.dagster.io/api/dagster/partitions#dagster.SpecificPartitionsPartitionMapping) is not, because these dates are not consecutive: @asset( partitions_def=MultiPartitionsDefinition( partitions_defs={ "date": DailyPartitionsDefinition( start_date="2022-01-01", end_date="2022-01-10", ), "letter": StaticPartitionsDefinition(["a", "b", "c"]), }, ), key_prefix=["my_schema"], metadata={"partition_expr": {"time": "time", "letter": "letter"}}, ins={ "multi_partitioned_asset": AssetIn( ["my_schema", "multi_partitioned_asset_1"], partition_mapping=MultiPartitionMapping( { "color": DimensionPartitionMapping( dimension_name="letter", partition_mapping=StaticPartitionMapping( {"blue": "a", "red": "b", "yellow": "c"} ), ), "date": DimensionPartitionMapping( dimension_name="date", partition_mapping=SpecificPartitionsPartitionMapping( ["2022-01-01", "2024-01-01"] ), ), } ), ) },)def mapped_multi_partition( context: AssetExecutionContext, multi_partitioned_asset: pa.Table) -> pa.Table: ... * [Selecting specific columns in a downstream asset](https://docs.dagster.io/integrations/libraries/iceberg/usage#selecting-specific-columns-in-a-downstream-asset) * [Storing partitioned assets](https://docs.dagster.io/integrations/libraries/iceberg/usage#storing-partitioned-assets) * [Storing tables in multiple schemas](https://docs.dagster.io/integrations/libraries/iceberg/usage#storing-tables-in-multiple-schemas) * [Using the Iceberg I/O manager with other I/O managers](https://docs.dagster.io/integrations/libraries/iceberg/usage#using-the-iceberg-io-manager-with-other-io-managers) * [Using different compute engines to read from and write to Iceberg](https://docs.dagster.io/integrations/libraries/iceberg/usage#using-different-compute-engines-to-read-from-and-write-to-iceberg) * [Executing custom SQL commands](https://docs.dagster.io/integrations/libraries/iceberg/usage#executing-custom-sql-commands) * [Configuring table behavior using table properties](https://docs.dagster.io/integrations/libraries/iceberg/usage#configuring-table-behavior-using-table-properties) * [Allowing updates to schema and partitions](https://docs.dagster.io/integrations/libraries/iceberg/usage#allowing-updates-to-schema-and-partitions) * [Using the custom I/O manager](https://docs.dagster.io/integrations/libraries/iceberg/usage#using-the-custom-io-manager) --- # Dagster & Microsoft Teams | Dagster Docs [Skip to main content](https://docs.dagster.io/integrations/libraries/microsoft-teams#__docusaurus_skipToContent_fallback) On this page An integration with Microsoft Teams to post messages to MS Teams from any Dagster op or asset. Installation[​](https://docs.dagster.io/integrations/libraries/microsoft-teams#installation "Direct link to Installation") --------------------------------------------------------------------------------------------------------------------------- * uv * pip uv add dagster-msteams pip install dagster-msteams Example[​](https://docs.dagster.io/integrations/libraries/microsoft-teams#example "Direct link to Example") ------------------------------------------------------------------------------------------------------------ # Read the docs on Resources to learn more: https://docs.dagster.io/deployment/resourcesfrom dagster_msteams import Card, MSTeamsResourceimport dagster as dg@dg.assetdef microsoft_teams_message(msteams: MSTeamsResource): card = Card() card.add_attachment(text_message="Hello there!") msteams.get_client().post_message(payload=card.payload)defs = dg.Definitions( assets=[microsoft_teams_message], resources={"msteams": MSTeamsResource(hook_url=dg.EnvVar("TEAMS_WEBHOOK_URL"))},) About Microsoft Teams[​](https://docs.dagster.io/integrations/libraries/microsoft-teams#about-microsoft-teams "Direct link to About Microsoft Teams") ------------------------------------------------------------------------------------------------------------------------------------------------------ **Microsoft Teams** is a business communication platform. Teams offers workspace chat and videoconferencing, file storage, and application integration. * [Installation](https://docs.dagster.io/integrations/libraries/microsoft-teams#installation) * [Example](https://docs.dagster.io/integrations/libraries/microsoft-teams#example) * [About Microsoft Teams](https://docs.dagster.io/integrations/libraries/microsoft-teams#about-microsoft-teams) --- # Dagster & Open Metadata | Dagster Docs [Skip to main content](https://docs.dagster.io/integrations/libraries/open-metadata#__docusaurus_skipToContent_fallback) On this page With this integration you can create a Open Metadata service to ingest metadata produced by the Dagster application. View the Ingestion Pipeline running from the Open Metadata Service Page. About Open Metadata[​](https://docs.dagster.io/integrations/libraries/open-metadata#about-open-metadata "Direct link to About Open Metadata") ---------------------------------------------------------------------------------------------------------------------------------------------- Poorly organized metadata is preventing organizations from realizing the full potential of data. Most metadata is incorrect, inconsistent, stale, missing, and fragmented in silos across various disconnected tools obscuring a holistic picture of data. **Open Metadata** is an all-in-one platform for data discovery, data lineage, data quality, observability, governance, and team collaboration. It's one of the fastest growing open source projects with a vibrant community and adoption by a diverse set of companies in a variety of industry verticals. Powered by a centralized metadata store based on Open Metadata Standards/APIs, supporting connectors to a wide range of data services, OpenMetadata enables end-to-end metadata management, giving you the freedom to unlock the value of your data assets. * [About Open Metadata](https://docs.dagster.io/integrations/libraries/open-metadata#about-open-metadata) --- # Dagster & Power BI | Dagster Docs [Skip to main content](https://docs.dagster.io/integrations/libraries/powerbi#__docusaurus_skipToContent_fallback) On this page info This feature is considered in a beta stage. It is still being tested and may change. For more information, see the [API lifecycle stages documentation](https://docs.dagster.io/api/api-lifecycle/api-lifecycle-stages) . Your Power BI assets, such as semantic models, data sources, reports, and dashboards, can be represented in the Dagster asset graph, allowing you to track lineage and dependencies between Power BI assets and upstream data assets you are already modeling in Dagster. You can also use Dagster to orchestrate Power BI semantic models, allowing you to trigger refreshes of these models on a cadence or based on upstream data changes. What you'll learn[​](https://docs.dagster.io/integrations/libraries/powerbi#what-youll-learn "Direct link to What you'll learn") --------------------------------------------------------------------------------------------------------------------------------- * How to represent Power BI assets in the Dagster asset graph, including lineage to other Dagster assets. * How to customize asset definition metadata for these Power BI assets. * How to materialize Power BI semantic models from Dagster. * How to customize how Power BI semantic models are materialized. Prerequisites * The `dagster` and `dagster-powerbi` libraries installed in your environment * Familiarity with asset definitions and the Dagster asset graph * Familiarity with Dagster resources * Familiarity with Power BI concepts, like semantic models, data sources, reports, and dashboards * A Power BI workspace * A service principal configured to access Power BI, or an API access token. For more information, see [Embed Power BI content with service principal and an application secret](https://learn.microsoft.com/en-us/power-bi/developer/embedded/embed-service-principal) in the Power BI documentation. Set up your environment[​](https://docs.dagster.io/integrations/libraries/powerbi#set-up-your-environment "Direct link to Set up your environment") ---------------------------------------------------------------------------------------------------------------------------------------------------- To get started, you'll need to install the `dagster` and `dagster-powerbi` Python packages: * uv * pip uv add dagster-powerbi pip install dagster-powerbi Represent Power BI assets in the asset graph[​](https://docs.dagster.io/integrations/libraries/powerbi#represent-power-bi-assets-in-the-asset-graph "Direct link to Represent Power BI assets in the asset graph") ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- To load Power BI assets into the Dagster asset graph, you must first construct a [`PowerBIWorkspace`](https://docs.dagster.io/api/libraries/dagster-powerbi#dagster_powerbi.PowerBIWorkspace) resource, which allows Dagster to communicate with your Power BI workspace. You'll need to supply your workspace ID and credentials. You may configure a service principal or use an API access token, which can be passed directly or accessed from the environment using [`EnvVar`](https://docs.dagster.io/api/dagster/resources#dagster.EnvVar) . Dagster can automatically load all semantic models, data sources, reports, and dashboards from your Power BI workspace as asset specs. Call the [`load_powerbi_asset_specs`](https://docs.dagster.io/api/libraries/dagster-powerbi#dagster_powerbi.load_powerbi_asset_specs) function, which returns a list of [`AssetSpec`](https://docs.dagster.io/api/dagster/assets#dagster.AssetSpec) s representing your Power BI assets. You can then include these asset specs in your [`Definitions`](https://docs.dagster.io/api/dagster/definitions#dagster.Definitions) object: from dagster_powerbi import ( PowerBIServicePrincipal, PowerBIToken, PowerBIWorkspace, load_powerbi_asset_specs,)import dagster as dg# Connect using a service principalpower_bi_workspace = PowerBIWorkspace( credentials=PowerBIServicePrincipal( client_id=dg.EnvVar("POWER_BI_CLIENT_ID"), client_secret=dg.EnvVar("POWER_BI_CLIENT_SECRET"), tenant_id=dg.EnvVar("POWER_BI_TENANT_ID"), ), workspace_id=dg.EnvVar("POWER_BI_WORKSPACE_ID"),)# Alternatively, connect directly using an API access tokenpower_bi_workspace = PowerBIWorkspace( credentials=PowerBIToken(api_token=dg.EnvVar("POWER_BI_API_TOKEN")), workspace_id=dg.EnvVar("POWER_BI_WORKSPACE_ID"),)power_bi_specs = load_powerbi_asset_specs(power_bi_workspace)defs = dg.Definitions( assets=[*power_bi_specs], resources={"power_bi": power_bi_workspace}) By default, Dagster will attempt to snapshot your entire workspace using Power BI's [metadata scanner APIs](https://learn.microsoft.com/en-us/fabric/governance/metadata-scanning-overview) , which are able to retrieve more detailed information about your Power BI assets, but rely on the workspace being configured to allow this access. If you encounter issues with the scanner APIs, you may disable them using `load_powerbi_asset_specs(power_bi_workspace, use_workspace_scan=False)`. Customize asset definition metadata for Power BI assets[​](https://docs.dagster.io/integrations/libraries/powerbi#customize-asset-definition-metadata-for-power-bi-assets "Direct link to Customize asset definition metadata for Power BI assets") ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- By default, Dagster will generate asset specs for each Power BI asset based on its type, and populate default metadata. You can further customize asset properties by passing a custom [`DagsterPowerBITranslator`](https://docs.dagster.io/api/libraries/dagster-powerbi#dagster_powerbi.DagsterPowerBITranslator) subclass to the [`load_powerbi_asset_specs`](https://docs.dagster.io/api/libraries/dagster-powerbi#dagster_powerbi.load_powerbi_asset_specs) function. This subclass can implement methods to customize the asset specs for each Power BI asset type. from dagster_powerbi import ( DagsterPowerBITranslator, PowerBIServicePrincipal, PowerBIWorkspace, load_powerbi_asset_specs,)from dagster_powerbi.translator import PowerBIContentType, PowerBITranslatorDataimport dagster as dgpower_bi_workspace = PowerBIWorkspace( credentials=PowerBIServicePrincipal( client_id=dg.EnvVar("POWER_BI_CLIENT_ID"), client_secret=dg.EnvVar("POWER_BI_CLIENT_SECRET"), tenant_id=dg.EnvVar("POWER_BI_TENANT_ID"), ), workspace_id=dg.EnvVar("POWER_BI_WORKSPACE_ID"),)# A translator class lets us customize properties of the built# Power BI assets, such as the owners or asset keyclass MyCustomPowerBITranslator(DagsterPowerBITranslator): def get_asset_spec(self, data: PowerBITranslatorData) -> dg.AssetSpec: # We create the default asset spec using super() default_spec = super().get_asset_spec(data) # We customize the team owner tag for all assets, # and we customize the asset key prefix only for dashboards. return default_spec.replace_attributes( key=( default_spec.key.with_prefix("prefix") if data.content_type == PowerBIContentType.DASHBOARD else default_spec.key ), owners=["team:my_team"], )power_bi_specs = load_powerbi_asset_specs( power_bi_workspace, dagster_powerbi_translator=MyCustomPowerBITranslator())defs = dg.Definitions( assets=[*power_bi_specs], resources={"power_bi": power_bi_workspace}) Note that `super()` is called in each of the overridden methods to generate the default asset spec. It is best practice to generate the default asset spec before customizing it. Load Power BI assets from multiple workspaces[​](https://docs.dagster.io/integrations/libraries/powerbi#load-power-bi-assets-from-multiple-workspaces "Direct link to Load Power BI assets from multiple workspaces") ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Definitions from multiple Power BI workspaces can be combined by instantiating multiple [`PowerBIWorkspace`](https://docs.dagster.io/api/libraries/dagster-powerbi#dagster_powerbi.PowerBIWorkspace) resources and merging their specs. This lets you view all your Power BI assets in a single asset graph: from dagster_powerbi import ( PowerBIServicePrincipal, PowerBIWorkspace, load_powerbi_asset_specs,)import dagster as dgcredentials = PowerBIServicePrincipal( client_id=dg.EnvVar("POWER_BI_CLIENT_ID"), client_secret=dg.EnvVar("POWER_BI_CLIENT_SECRET"), tenant_id=dg.EnvVar("POWER_BI_TENANT_ID"),)sales_team_workspace = PowerBIWorkspace( credentials=credentials, workspace_id="726c94ff-c408-4f43-8edf-61fbfa1753c7",)marketing_team_workspace = PowerBIWorkspace( credentials=credentials, workspace_id="8b7f815d-4e64-40dd-993c-cfa4fb12edee",)sales_team_specs = load_powerbi_asset_specs(sales_team_workspace)marketing_team_specs = load_powerbi_asset_specs(marketing_team_workspace)# Merge the specs into a single set of definitionsdefs = dg.Definitions( assets=[*sales_team_specs, *marketing_team_specs], resources={ "marketing_power_bi": marketing_team_workspace, "sales_power_bi": sales_team_workspace, },) Materialize Power BI semantic models from Dagster[​](https://docs.dagster.io/integrations/libraries/powerbi#materialize-power-bi-semantic-models-from-dagster "Direct link to Materialize Power BI semantic models from Dagster") ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Dagster's default behavior is to pull in representations of Power BI semantic models as external assets, which appear in the asset graph but can't be materialized. However, you can build executable asset definitions that trigger the refresh of Power BI semantic models. The [`build_semantic_model_refresh_asset_definition`](https://docs.dagster.io/api/libraries/dagster-powerbi#dagster_powerbi.build_semantic_model_refresh_asset_definition) utility will construct an asset definition that triggers a refresh of a semantic model when materialized. from dagster_powerbi import ( PowerBIServicePrincipal, PowerBIWorkspace, build_semantic_model_refresh_asset_definition, load_powerbi_asset_specs,)import dagster as dgpower_bi_workspace = PowerBIWorkspace( credentials=PowerBIServicePrincipal( client_id=dg.EnvVar("POWER_BI_CLIENT_ID"), client_secret=dg.EnvVar("POWER_BI_CLIENT_SECRET"), tenant_id=dg.EnvVar("POWER_BI_TENANT_ID"), ), workspace_id=dg.EnvVar("POWER_BI_WORKSPACE_ID"),)# Load Power BI asset specs, and use the asset definition builder to# construct a semantic model refresh definition for each semantic modelpower_bi_assets = [ build_semantic_model_refresh_asset_definition(resource_key="power_bi", spec=spec) if spec.tags.get("dagster-powerbi/asset_type") == "semantic_model" else spec for spec in load_powerbi_asset_specs(power_bi_workspace)]defs = dg.Definitions( assets=[*power_bi_assets], resources={"power_bi": power_bi_workspace}) You can then add these semantic models to jobs or as targets of Dagster sensors or schedules to trigger refreshes of the models on a cadence or based on other conditions. Customizing how Power BI semantic models are materialized[​](https://docs.dagster.io/integrations/libraries/powerbi#customizing-how-power-bi-semantic-models-are-materialized "Direct link to Customizing how Power BI semantic models are materialized") ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Instead of using the out-of-the-box [`build_semantic_model_refresh_asset_definition`](https://docs.dagster.io/api/libraries/dagster-powerbi#dagster_powerbi.build_semantic_model_refresh_asset_definition) utility, you can build your own asset definitions that trigger the refresh of Power BI semantic models. This allows you to customize how the refresh is triggered or to run custom code before or after the refresh. from dagster_powerbi import ( PowerBIServicePrincipal, PowerBIWorkspace, load_powerbi_asset_specs,)import dagster as dgpower_bi_workspace = PowerBIWorkspace( credentials=PowerBIServicePrincipal( client_id=dg.EnvVar("POWER_BI_CLIENT_ID"), client_secret=dg.EnvVar("POWER_BI_CLIENT_SECRET"), tenant_id=dg.EnvVar("POWER_BI_TENANT_ID"), ), workspace_id=dg.EnvVar("POWER_BI_WORKSPACE_ID"),)# Asset definition factory which triggers a semantic model refresh and sends a notification# once completedef build_semantic_model_refresh_and_notify_asset_def( spec: dg.AssetSpec,) -> dg.AssetsDefinition: dataset_id = spec.metadata["dagster-powerbi/id"] @dg.multi_asset(specs=[spec], name=spec.key.to_python_identifier()) def rebuild_semantic_model( context: dg.AssetExecutionContext, power_bi: PowerBIWorkspace ) -> None: power_bi.trigger_and_poll_refresh(dataset_id) # Do some custom work after refreshing here, such as sending an email notification return rebuild_semantic_model# Load Power BI asset specs, and use our custom asset definition builder to# construct a definition for each semantic modelpower_bi_assets = [ build_semantic_model_refresh_and_notify_asset_def(spec=spec) if spec.tags.get("dagster-powerbi/asset_type") == "semantic_model" else spec for spec in load_powerbi_asset_specs(power_bi_workspace)]defs = dg.Definitions( assets=[*power_bi_assets], resources={"power_bi": power_bi_workspace}) Customize upstream dependencies[​](https://docs.dagster.io/integrations/libraries/powerbi#customize-upstream-dependencies "Direct link to Customize upstream dependencies") ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- By default, Dagster sets upstream dependencies when generating asset specs for your PowerBI assets. To do so, Dagster parses information about assets that are upstream of specific PowerBI assets from the PowerBI workspace itself. You can customize how upstream dependencies are set on your PowerBI assets by passing an instance of the custom [`DagsterPowerBITranslator`](https://docs.dagster.io/api/libraries/dagster-powerbi#dagster_powerbi.DagsterPowerBITranslator) to the [`load_powerbi_asset_specs`](https://docs.dagster.io/api/libraries/dagster-powerbi#dagster_powerbi.load_powerbi_asset_specs) function. The below example defines `my_upstream_asset` as an upstream dependency of `my_powerbi_semantic_model`: class MyCustomPowerBITranslator(DagsterPowerBITranslator): def get_asset_spec(self, data: PowerBITranslatorData) -> dg.AssetSpec: # We create the default asset spec using super() default_spec = super().get_asset_spec(data) # We customize upstream dependencies for the PowerBI semantic model named `my_powerbi_semantic_model` return default_spec.replace_attributes( deps=["my_upstream_asset"] if data.content_type == PowerBIContentType.SEMANTIC_MODEL and data.properties.get("name") == "my_powerbi_semantic_model" else ... )power_bi_specs = load_powerbi_asset_specs( power_bi_workspace, dagster_powerbi_translator=MyCustomPowerBITranslator()) Note that `super()` is called in each of the overridden methods to generate the default asset spec. It is best practice to generate the default asset spec before customizing it. * [What you'll learn](https://docs.dagster.io/integrations/libraries/powerbi#what-youll-learn) * [Set up your environment](https://docs.dagster.io/integrations/libraries/powerbi#set-up-your-environment) * [Represent Power BI assets in the asset graph](https://docs.dagster.io/integrations/libraries/powerbi#represent-power-bi-assets-in-the-asset-graph) * [Customize asset definition metadata for Power BI assets](https://docs.dagster.io/integrations/libraries/powerbi#customize-asset-definition-metadata-for-power-bi-assets) * [Load Power BI assets from multiple workspaces](https://docs.dagster.io/integrations/libraries/powerbi#load-power-bi-assets-from-multiple-workspaces) * [Materialize Power BI semantic models from Dagster](https://docs.dagster.io/integrations/libraries/powerbi#materialize-power-bi-semantic-models-from-dagster) * [Customizing how Power BI semantic models are materialized](https://docs.dagster.io/integrations/libraries/powerbi#customizing-how-power-bi-semantic-models-are-materialized) * [Customize upstream dependencies](https://docs.dagster.io/integrations/libraries/powerbi#customize-upstream-dependencies) --- # Using Jupyter notebooks with Papermill and Dagster | Dagster Docs [Skip to main content](https://docs.dagster.io/integrations/libraries/jupyter/using-notebooks-with-dagster#__docusaurus_skipToContent_fallback) On this page Title You can find the code for this example on [GitHub](https://github.com/dagster-io/dagster/tree/master/examples/tutorial_notebook_assets/) . In this tutorial, we'll walk you through integrating a Jupyter notebook with Dagster using an example project. Before we get started, let's cover some common approaches to writing and integrating Jupyter notebooks with Dagster: * **Doing standalone development in a Jupyter notebook**. You could then create two Dagster assets: one for the notebook itself and another for data-fetching logic. This approach, which we'll use to start the tutorial, allows you to configure existing notebooks to work with Dagster. * **Using existing Dagster assets as input to notebooks**. If the data you want to analyze is already a Dagster asset, you can directly load the asset's value into the notebook. When the notebook is complete, you can create a Dagster asset for the notebook and factor any data-fetching logic into a second asset, if applicable. This approach allows you to develop new notebooks that work with assets that are already a part of your Dagster project. By the end of this tutorial, you will: * Explore a Jupyter notebook that fetches and explores a dataset * Create a Dagster asset from the notebook * Create a second Dagster asset that only fetches the dataset * Load existing Dagster assets into a new Jupyter notebook Dagster concepts[​](https://docs.dagster.io/integrations/libraries/jupyter/using-notebooks-with-dagster#dagster-concepts "Direct link to Dagster concepts") ------------------------------------------------------------------------------------------------------------------------------------------------------------ In this guide, we'll use the following Dagster concepts: * [Assets](https://docs.dagster.io/guides/build/assets/defining-assets) - An asset is a software object that models a data asset. The prototypical example is a table in a database or a file in cloud storage. An executed Jupyter notebook file can also be an asset! That's what we'll be creating in this guide. * [Definitions](https://docs.dagster.io/api/dagster/definitions) - A Dagster `Definitions` object is a collection of Dagster objects, including assets. * [I/O managers](https://docs.dagster.io/guides/build/io-managers) - An I/O manager handles storing and loading assets. In this guide, we'll be using a special I/O manager to store executed Jupyter notebook files. Prerequisites[​](https://docs.dagster.io/integrations/libraries/jupyter/using-notebooks-with-dagster#prerequisites "Direct link to Prerequisites") --------------------------------------------------------------------------------------------------------------------------------------------------- To complete this tutorial, you'll need: * **To install Dagster and Jupyter**. Run the following to install using pip: * uv * pip uv add dagster notebook pip install dagster notebook Refer to the [Dagster](https://docs.dagster.io/getting-started/installation) installation docs for more info. * **To download the [`tutorial_notebook_assets`](https://github.com/dagster-io/dagster/tree/master/examples/tutorial_notebook_assets) Dagster example and install its dependencies:** dagster project from-example --name tutorial_notebook_assets --example tutorial_notebook_assetscd tutorial_notebook_assetspip install -e ".[dev]" This example includes: * **A finished version of the tutorial project**, which you can use to check out the finished project. This is the `tutorial_finished` subfolder. * **A template version of the tutorial project**, which you can use to follow along with the tutorial. This is the `tutorial_template` subfolder. In this folder, you'll also find: * `assets`, a subfolder containing Dagster assets. We'll use `/assets.py` to write these. * `notebooks`, a subfolder containing Jupyter notebooks. We'll use `/notebooks/iris-kmeans.ipynb` to write a Jupyter notebook. Step 1: Explore the Jupyter notebook[​](https://docs.dagster.io/integrations/libraries/jupyter/using-notebooks-with-dagster#step-1-explore-the-jupyter-notebook "Direct link to Step 1: Explore the Jupyter notebook") ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- In this tutorial, we'll analyze the Iris dataset, collected in 1936 by the American botanist Edgar Anderson and made famous by statistician Ronald Fisher. The Iris dataset is a basic example of machine learning because it contains three classes of observation: one class is straightforwardly linearly separable from the other two, which can only be distinguished by more sophisticated methods. The `/tutorial_template/notebooks/iris-kmeans.ipynb` Jupyter notebook, which is already completed for you, does some analysis on the Iris dataset. In the Jupyter notebook, we first fetch the Iris dataset: # /tutorial_template/notebooks/iris-kmeans.ipynbiris = pd.read_csv( "https://archive.ics.uci.edu/ml/machine-learning-databases/iris/iris.data", names=[ "Sepal length (cm)", "Sepal width (cm)", "Petal length (cm)", "Petal width (cm)", "Species", ],) Next, we'll perform some descriptive analysis to explore the dataset. If you execute these cells, several plots of the Iris dataset will be created: ![Iris dataset plots](https://docs.dagster.io/assets/images/descriptive-plots-8a3e551f951a7e7a53d00345cab23819.png) Next, we conduct our K-means analysis: estimator = sklearn.cluster.KMeans(n_clusters=3)estimator.fit( iris[["Sepal length (cm)", "Sepal width (cm)", "Petal length (cm)", "Petal width (cm)"]]) Lastly, we plot the results of the K-means analysis. From the plots, we can see that one species of Iris is separable from the other two, but a more sophisticated model will be required to distinguish the other two species: ![kmeans plots](https://docs.dagster.io/assets/images/kmeans-plots-63e0495ad923cf477953265eb934459f.png) Like many notebooks, this example does some fairly sophisticated work, including producing diagnostic plots and a statistical model. For now, this work is locked away in the `.ipynb` format, only reproducible using a complex Jupyter setup, and only programmatically accessible within the notebook context. We'll address this in the remainder of the tutorial. Step 2: Create a Dagster asset from the Jupyter Notebook[​](https://docs.dagster.io/integrations/libraries/jupyter/using-notebooks-with-dagster#step-2-create-a-dagster-asset-from-the-jupyter-notebook "Direct link to Step 2: Create a Dagster asset from the Jupyter Notebook") ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- By creating a Dagster asset from our notebook, we can integrate the notebook as part of our data platform. This enables us to make its contents more accessible to developers, stakeholders, and other assets in Dagster. To create a Dagster asset from a Jupyter notebook, we can use the [`define_dagstermill_asset`](https://docs.dagster.io/api/libraries/dagstermill#dagstermill.define_dagstermill_asset) function. In `/tutorial_template/assets.py` add the following code snippet: # /tutorial_template/assets.pyfrom dagstermill import define_dagstermill_assetfrom dagster import file_relative_pathiris_kmeans_jupyter_notebook = define_dagstermill_asset( name="iris_kmeans_jupyter", notebook_path=file_relative_path(__file__, "notebooks/iris-kmeans.ipynb"), group_name="template_tutorial",) If you are following along in the template code, uncomment the code block under the `TODO 1` comment. Using `define_dagstermill_asset`, we've created and returned a Dagster asset. Let's take a look at the arguments we provided: * `name` - This argument names the asset, in this case `iris_kmeans_jupyter` * `notebook_path` - This argument tells Dagster where to find the notebook the asset should use as a source. In this case, that's our `/notebooks/iris-kmeans.ipynb` file. * `group_name` - This optional argument places the asset into a group named `template_tutorial`, which is helpful for organizating your assets in the UI. When materialized, the `iris_kmeans_jupyter` asset will execute the notebook (`/notebooks/iris-kmeans.ipynb`) and store the resulting `.ipynb` file in a persistent location. Step 3: Add a Dagster Definitions object and supply an I/O manager[​](https://docs.dagster.io/integrations/libraries/jupyter/using-notebooks-with-dagster#step-3-add-a-dagster-definitions-object-and-supply-an-io-manager "Direct link to Step 3: Add a Dagster Definitions object and supply an I/O manager") ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- We want to execute our Dagster asset and save the resulting notebook to a persistent location. This is called materializing the asset and to do this, we need to add the asset to a Dagster [`Definitions`](https://docs.dagster.io/api/dagster/definitions#dagster.Definitions) object. Additionally, we need to provide a [resource](https://docs.dagster.io/guides/build/external-resources) to the notebook to tell Dagster how to store the resulting `.ipynb` file. We'll use an [I/O manager](https://docs.dagster.io/guides/build/io-managers) to accomplish this. Open the `/tutorial_template/definitions.py` file and add the following code: # tutorial_template/definitions.pyfrom dagster import load_assets_from_modules, Definitionsfrom dagstermill import ConfigurableLocalOutputNotebookIOManagerfrom . import assetsdefs = Definitions( assets=load_assets_from_modules([assets]), resources={ "output_notebook_io_manager": ConfigurableLocalOutputNotebookIOManager() }) Let's take a look at what's happening here: * Using [`load_assets_from_modules`](https://docs.dagster.io/api/dagster/assets#dagster.load_assets_from_modules) , we've imported all assets in the `assets` module. This approach allows any new assets we create to be automatically added to the `Definitions` object instead of needing to manually add them one by one. * We provided a dictionary of resources to the `resources` parameter. In this example, that's the [`ConfigurableLocalOutputNotebookIOManager`](https://docs.dagster.io/api/libraries/dagstermill#dagstermill.ConfigurableLocalOutputNotebookIOManager) resource. This I/O manager, bound to the `output_notebook_io_manager` key, is responsible for handling the storage of the notebook asset's resulting `.ipynb` file. Step 4: Materialize the notebook asset[​](https://docs.dagster.io/integrations/libraries/jupyter/using-notebooks-with-dagster#step-4-materialize-the-notebook-asset "Direct link to Step 4: Materialize the notebook asset") ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Now that you've created an asset, a resource, and a `Definitions` object, it's time to materialize the notebook asset! Materializing an asset runs the op it contains and saves the results to persistent storage. 1. To start the Dagster UI, run the following in `/tutorial_template`: dagster dev Which will result in output similar to: Serving dagster-webserver on http://127.0.0.1:3000 in process 70635 2. In your browser, navigate to [http://127.0.0.1:3000](http://127.0.0.1:3000/) . The page will display the notebook asset in the **Asset Graph**. If you click the notebook asset, a sidebar containing info about the asset will slide out from the right side of the page. In the **Description** section of the panel is a **View Source Notebook** button: ![Notebook asset in UI](https://docs.dagster.io/assets/images/ui-one-a3decbaa4e40b217a4c3ed3287e43e4e.png) This button allows you to view the notebook directly in the UI. When clicked, Dagster will render the notebook - referenced in the `notebook_path` parameter - that'll be executed when the `iris_kmeans_jupyter` asset is materialized: ![View Source Notebook display in the Dagster UI](https://docs.dagster.io/assets/images/view-source-notebook-dd9ebcb4923924e8748c2f21faa43689.png) 3. Click the **Materialize** button. To view the execution as it happens, click the **View** button in the alert that displays. After the run completes successfully, you can view the executed notebook in the UI. Click the asset again and locate the **View Notebook** button in the **Materialization in Last Run** section of the sidebar: ![View notebook button in materialization in last run area](https://docs.dagster.io/assets/images/ui-two-aa664401048ad4c13e8023cb1d313fa3.png) Click the button to display the executed notebook - specifically, the notebook that was executed and written to a persistent location: ![Executed notebook display in the Dagster UI](https://docs.dagster.io/assets/images/view-executed-notebook-eb9b0d9667abc562fad6c93ab300e1c9.png) Step 5: Add an upstream asset[​](https://docs.dagster.io/integrations/libraries/jupyter/using-notebooks-with-dagster#step-5-add-an-upstream-asset "Direct link to Step 5: Add an upstream asset") -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- While our `iris-kmeans` notebook asset now materializes successfully, there are still some improvements we can make. The beginning of the notebook fetches the Iris dataset, which means that every time the notebook is materialized, the data is re-fetched. To address this, we can factor the Iris dataset into its own asset. This will allow us to: * **Use the asset as input to additional notebooks.** This means all notebooks analyzing the Iris dataset will use the same source data, which we only have to fetch once. * **Materialize notebooks without fetching data for each materialization.** Instead of making potentially expensive API calls, Dagster can fetch the data from the previous materialization of the Iris dataset and provide that data as input to the notebook. In this step, you'll: * [Create the Iris dataset asset](https://docs.dagster.io/integrations/libraries/jupyter/using-notebooks-with-dagster#step-51-create-the-iris-dataset-asset) * [Provide the Iris dataset as input to the notebook](https://docs.dagster.io/integrations/libraries/jupyter/using-notebooks-with-dagster#step-52-provide-the-iris_dataset-asset-to-the-notebook-asset) * [Modify the notebook](https://docs.dagster.io/integrations/libraries/jupyter/using-notebooks-with-dagster#step-53-modify-the-notebook) ### Step 5.1: Create the Iris dataset asset[​](https://docs.dagster.io/integrations/libraries/jupyter/using-notebooks-with-dagster#step-51-create-the-iris-dataset-asset "Direct link to Step 5.1: Create the Iris dataset asset") To create an asset for the Iris dataset, add the following code to `/tutorial_template/assets.py`: # /tutorial_template/assets.pyfrom dagstermill import define_dagstermill_assetfrom dagster import asset, file_relative_pathimport pandas as pd@asset( group_name="template_tutorial")def iris_dataset(): return pd.read_csv( "https://archive.ics.uci.edu/ml/machine-learning-databases/iris/iris.data", names=[ "Sepal length (cm)", "Sepal width (cm)", "Petal length (cm)", "Petal width (cm)", "Species", ], ) If you're following along in the template tutorial, uncomment the code block under the `TODO 2` comment. Let's go over what's happening in this code block: * Using [`@dg.asset`](https://docs.dagster.io/api/dagster/assets#dagster.asset) , we create a standard Dagster asset. The name of the Python function (`iris_dataset`) is the name of the asset. * As with the `iris_kmeans_jupyter` asset, we set the `group_name` parameter to organize our assets in the UI. * The body of the Python function fetches the Iris dataset, renames the columns, and outputs a Pandas DataFrame. ### Step 5.2: Provide the iris\_dataset asset to the notebook asset[​](https://docs.dagster.io/integrations/libraries/jupyter/using-notebooks-with-dagster#step-52-provide-the-iris_dataset-asset-to-the-notebook-asset "Direct link to Step 5.2: Provide the iris_dataset asset to the notebook asset") Next, we need to tell Dagster that the `iris_dataset` asset is input data for the `iris-kmeans` notebook. To do this, add the `ins` parameter to the notebook asset: # tutorial_template/assets.pyfrom dagstermill import define_dagstermill_assetfrom dagster import asset, file_relative_path, AssetInimport pandas as pd# iris_dataset asset removed for clarityiris_kmeans_jupyter_notebook = define_dagstermill_asset( name="iris_kmeans_jupyter", notebook_path=file_relative_path(__file__, "notebooks/iris-kmeans.ipynb"), group_name="template_tutorial", ins={"iris": AssetIn("iris_dataset")}, # this is the new parameter!) If you are following along with the template tutorial, uncomment the line with the `TODO 3` comment. The `ins` parameter tells Dagster that the `iris_dataset` asset should be mapped to a variable named `iris` in our notebook. Recall that in our `iris-kmeans` notebook, the Iris dataset is assigned to a variable named `iris`. ### Step 5.3: Modify the notebook[​](https://docs.dagster.io/integrations/libraries/jupyter/using-notebooks-with-dagster#step-53-modify-the-notebook "Direct link to Step 5.3: Modify the notebook") We need to make a small change in our Jupyter notebook to allow Dagster to supply the `iris_dataset` asset as input. Behind the scenes, Dagster uses `papermill` to inject parameters into notebooks. `papermill` works by replacing a notebook cell with the `parameters` tag with a custom cell that can fetch the desired data. To accomplish this, we need to tag the cell in the `iris-kmeans` notebook that fetches the Iris dataset. This allows us to replace the cell with the data-fetching logic that loads the `iris_dataset` asset and retain the ability to run the Jupyter notebook in a standalone context. We'll cover this in more detail later in the tutorial. To add the `parameters` tag, you may need to turn on the display of cell tags in Jupyter: 1. In Jupyter, navigate to **View > Cell Toolbar > Tags**: ![Jupyer turn on display of cell tags](https://docs.dagster.io/assets/images/jupyter-view-menu-d3fb4f1fd93e4cd51803cba0120d47ec.png) 2. Click **Add Tag** to add a `parameters` tag: ![Jupyer add tag button](https://docs.dagster.io/assets/images/jupyter-tags-5d541ecbfcf58982e98a670d6927fdb4.png) Step 6: Materialize the assets[​](https://docs.dagster.io/integrations/libraries/jupyter/using-notebooks-with-dagster#step-6-materialize-the-assets "Direct link to Step 6: Materialize the assets") ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Next, we'll materialize our `iris_dataset` and notebook assets. 1. In the UI, open the **Asset Graph** page. 2. Click the **Reload definitions** button. This ensures that the UI picks up the changes you made in the previous steps. At this point, the `iris_dataset` asset should display above the `iris_kmeans_jupyter` asset as an upstream dependency: ![Upstream Iris dataset asset](https://docs.dagster.io/assets/images/ui-three-7ed769fb7bf01eadec872a7066839c5c.png) 3. Click the **Materialize all** button near the top right corner of the page, which will launch a run to materialize the assets. That's it! You now have working Jupyter and Dagster assets! Extra credit: Fetch a Dagster asset in a Jupyter notebook[​](https://docs.dagster.io/integrations/libraries/jupyter/using-notebooks-with-dagster#extra-credit-fetch-a-dagster-asset-in-a-jupyter-notebook "Direct link to Extra credit: Fetch a Dagster asset in a Jupyter notebook") -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- What if you want to do additional analysis of the Iris dataset and create a new notebook? How can you accomplish this without duplicating code or re-fetching data? The answer is simple: use the `iris_dataset` Dagster asset! In the Jupyter notebook, import the Dagster `Definitions` object and use the [`Definitions.load_asset_value`](https://docs.dagster.io/api/dagster/definitions#dagster.Definitions.load_asset_value) function to load the data for the `iris_dataset` asset we created in [Step 5.1: Create the Iris dataset asset](https://docs.dagster.io/integrations/libraries/jupyter/using-notebooks-with-dagster#step-51-create-the-iris-dataset-asset) : from tutorial_template import template_tutorialiris = template_tutorial.load_asset_value("iris_dataset") Then, whenever you run the notebook using Jupyter, you'll be able to work with the `iris_dataset` asset: jupyter notebook /path/to/new/notebook.ipynb Behind the scenes, when `load_asset_value` is called, Dagster fetches the value of `iris_dataset` that was most recently materialized and stored by an I/O manager. To integrate the new notebook, follow the steps from [Step 5.3](https://docs.dagster.io/integrations/libraries/jupyter/using-notebooks-with-dagster#step-53-modify-the-notebook) to add the `parameters` tag to the cell that fetches the `iris_dataset` asset via `load_asset_value`. Conclusion[​](https://docs.dagster.io/integrations/libraries/jupyter/using-notebooks-with-dagster#conclusion "Direct link to Conclusion") ------------------------------------------------------------------------------------------------------------------------------------------ Now we have successfully created an asset from a Jupyter notebook and integrated it with our Dagster project! To learn about additional `dagstermill` features, refer to the [Dagstermill integration reference](https://docs.dagster.io/integrations/libraries/jupyter/reference) . * [Dagster concepts](https://docs.dagster.io/integrations/libraries/jupyter/using-notebooks-with-dagster#dagster-concepts) * [Prerequisites](https://docs.dagster.io/integrations/libraries/jupyter/using-notebooks-with-dagster#prerequisites) * [Step 1: Explore the Jupyter notebook](https://docs.dagster.io/integrations/libraries/jupyter/using-notebooks-with-dagster#step-1-explore-the-jupyter-notebook) * [Step 2: Create a Dagster asset from the Jupyter Notebook](https://docs.dagster.io/integrations/libraries/jupyter/using-notebooks-with-dagster#step-2-create-a-dagster-asset-from-the-jupyter-notebook) * [Step 3: Add a Dagster Definitions object and supply an I/O manager](https://docs.dagster.io/integrations/libraries/jupyter/using-notebooks-with-dagster#step-3-add-a-dagster-definitions-object-and-supply-an-io-manager) * [Step 4: Materialize the notebook asset](https://docs.dagster.io/integrations/libraries/jupyter/using-notebooks-with-dagster#step-4-materialize-the-notebook-asset) * [Step 5: Add an upstream asset](https://docs.dagster.io/integrations/libraries/jupyter/using-notebooks-with-dagster#step-5-add-an-upstream-asset) * [Step 5.1: Create the Iris dataset asset](https://docs.dagster.io/integrations/libraries/jupyter/using-notebooks-with-dagster#step-51-create-the-iris-dataset-asset) * [Step 5.2: Provide the iris\_dataset asset to the notebook asset](https://docs.dagster.io/integrations/libraries/jupyter/using-notebooks-with-dagster#step-52-provide-the-iris_dataset-asset-to-the-notebook-asset) * [Step 5.3: Modify the notebook](https://docs.dagster.io/integrations/libraries/jupyter/using-notebooks-with-dagster#step-53-modify-the-notebook) * [Step 6: Materialize the assets](https://docs.dagster.io/integrations/libraries/jupyter/using-notebooks-with-dagster#step-6-materialize-the-assets) * [Extra credit: Fetch a Dagster asset in a Jupyter notebook](https://docs.dagster.io/integrations/libraries/jupyter/using-notebooks-with-dagster#extra-credit-fetch-a-dagster-asset-in-a-jupyter-notebook) * [Conclusion](https://docs.dagster.io/integrations/libraries/jupyter/using-notebooks-with-dagster#conclusion) --- # Dagster & Looker | Dagster Docs [Skip to main content](https://docs.dagster.io/integrations/libraries/looker#__docusaurus_skipToContent_fallback) On this page info This feature is considered in a beta stage. It is still being tested and may change. For more information, see the [API lifecycle stages documentation](https://docs.dagster.io/api/api-lifecycle/api-lifecycle-stages) . This guide provides instructions for using Dagster with Looker using the `dagster-looker` library. Your Looker assets, such as views, explores, and dashboards, can be represented in the Dagster asset graph, allowing you to track lineage and dependencies between Looker assets. You can also use Dagster to orchestrate Looker PDTs, allowing you to trigger refreshes of these materialized tables on a cadence or based on upstream data changes. What you'll learn[​](https://docs.dagster.io/integrations/libraries/looker#what-youll-learn "Direct link to What you'll learn") -------------------------------------------------------------------------------------------------------------------------------- * How to represent Looker assets in the Dagster asset graph. * How to customize asset definition metadata for these Looker assets. * How to materialize Looker PDTs from Dagster. Prerequisites * The `dagster-looker` library installed in your environment * Familiarity with asset definitions and the Dagster asset graph * Familiarity with Dagster resources * Familiarity with Looker concepts, like views, explores, and dashboards * A Looker instance * Looker API credentials to access your Looker instance. For more information, see [Looker API authentication](https://cloud.google.com/looker/docs/api-auth) in the Looker documentation. Set up your environment[​](https://docs.dagster.io/integrations/libraries/looker#set-up-your-environment "Direct link to Set up your environment") --------------------------------------------------------------------------------------------------------------------------------------------------- To get started, you'll need to install the `dagster` and `dagster-looker` Python packages: * uv * pip uv add dagster-looker pip install dagster-looker Represent Looker assets in the asset graph[​](https://docs.dagster.io/integrations/libraries/looker#represent-looker-assets-in-the-asset-graph "Direct link to Represent Looker assets in the asset graph") ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ To load Looker assets into the Dagster asset graph, you must first construct a [`LookerResource`](https://docs.dagster.io/api/libraries/dagster-looker#dagster_looker.LookerResource) , which allows Dagster to communicate with your Looker instance. You'll need to supply your Looker instance URL and API credentials, which can be passed directly or accessed from the environment using [`EnvVar`](https://docs.dagster.io/api/dagster/resources#dagster.EnvVar) . Dagster can automatically load all views, explores, and dashboards from your Looker instance as asset specs. Call the [`load_looker_asset_specs`](https://docs.dagster.io/api/libraries/dagster-looker#dagster_looker.load_looker_asset_specs) function, which returns a list of [`AssetSpecs`](https://docs.dagster.io/api/dagster/assets#dagster.AssetSpec) representing your Looker assets. You can then include these asset specs in your [`Definitions`](https://docs.dagster.io/api/dagster/definitions#dagster.Definitions) object: from dagster_looker import LookerResource, load_looker_asset_specsimport dagster as dglooker_resource = LookerResource( client_id=dg.EnvVar("LOOKERSDK_CLIENT_ID"), client_secret=dg.EnvVar("LOOKERSDK_CLIENT_SECRET"), base_url=dg.EnvVar("LOOKERSDK_HOST_URL"),)looker_specs = load_looker_asset_specs(looker_resource=looker_resource)defs = dg.Definitions(assets=[*looker_specs], resources={"looker": looker_resource}) Load Looker assets from filtered dashboards and explores[​](https://docs.dagster.io/integrations/libraries/looker#load-looker-assets-from-filtered-dashboards-and-explores "Direct link to Load Looker assets from filtered dashboards and explores") ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ It is possible to load a subset of your Looker assets by providing a [`LookerFilter`](https://docs.dagster.io/api/libraries/dagster-looker#dagster_looker.LookerFilter) to the [`load_looker_asset_specs`](https://docs.dagster.io/api/libraries/dagster-looker#dagster_looker.load_looker_asset_specs) function. All dashboards contained in the folders provided to your [`LookerFilter`](https://docs.dagster.io/api/libraries/dagster-looker#dagster_looker.LookerFilter) will be fetched. Additionally, only the explores used in these dashboards will be fetched by passing `only_fetch_explores_used_in_dashboards=True` to your [`LookerFilter`](https://docs.dagster.io/api/libraries/dagster-looker#dagster_looker.LookerFilter) . Note that the content and size of Looker instance may affect the performance of your Dagster deployments. Filtering the dashboards and explores selection from which your Looker assets will be loaded is particularly useful for improving loading times. from dagster_looker import LookerFilter, LookerResource, load_looker_asset_specsimport dagster as dglooker_resource = LookerResource( client_id=dg.EnvVar("LOOKERSDK_CLIENT_ID"), client_secret=dg.EnvVar("LOOKERSDK_CLIENT_SECRET"), base_url=dg.EnvVar("LOOKERSDK_HOST_URL"),)looker_specs = load_looker_asset_specs( looker_resource=looker_resource, looker_filter=LookerFilter( dashboard_folders=[ ["my_folder", "my_subfolder"], ["my_folder", "my_other_subfolder"], ], only_fetch_explores_used_in_dashboards=True, ),)defs = dg.Definitions(assets=[*looker_specs], resources={"looker": looker_resource}) Customize asset definition metadata for Looker assets[​](https://docs.dagster.io/integrations/libraries/looker#customize-asset-definition-metadata-for-looker-assets "Direct link to Customize asset definition metadata for Looker assets") --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- By default, Dagster will generate asset specs for each Looker asset based on its type, and populate default metadata. You can further customize asset properties by passing a custom [`DagsterLookerApiTranslator`](https://docs.dagster.io/api/libraries/dagster-looker#dagster_looker.DagsterLookerApiTranslator) subclass to the [`load_looker_asset_specs`](https://docs.dagster.io/api/libraries/dagster-looker#dagster_looker.load_looker_asset_specs) function. This subclass can implement methods to customize the asset specs for each Looker asset type. from dagster_looker import ( DagsterLookerApiTranslator, LookerApiTranslatorStructureData, LookerResource, LookerStructureType, load_looker_asset_specs,)import dagster as dglooker_resource = LookerResource( client_id=dg.EnvVar("LOOKERSDK_CLIENT_ID"), client_secret=dg.EnvVar("LOOKERSDK_CLIENT_SECRET"), base_url=dg.EnvVar("LOOKERSDK_HOST_URL"),)class CustomDagsterLookerApiTranslator(DagsterLookerApiTranslator): def get_asset_spec( self, looker_structure: LookerApiTranslatorStructureData ) -> dg.AssetSpec: # We create the default asset spec using super() default_spec = super().get_asset_spec(looker_structure) # We customize the team owner tag for all assets, # and we customize the asset key prefix only for dashboards. return default_spec.replace_attributes( key=( default_spec.key.with_prefix("looker") if looker_structure.structure_type == LookerStructureType.DASHBOARD else default_spec.key ), owners=["team:my_team"], )looker_specs = load_looker_asset_specs( looker_resource, dagster_looker_translator=CustomDagsterLookerApiTranslator())defs = dg.Definitions(assets=[*looker_specs], resources={"looker": looker_resource}) Note that `super()` is called in each of the overridden methods to generate the default asset spec. It is best practice to generate the default asset spec before customizing it. Materialize Looker PDTs from Dagster[​](https://docs.dagster.io/integrations/libraries/looker#materialize-looker-pdts-from-dagster "Direct link to Materialize Looker PDTs from Dagster") ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ You can use Dagster to orchestrate the materialization of Looker PDTs. To model PDTs as assets, build their asset definitions by passing a list of [`RequestStartPdtBuild`](https://docs.dagster.io/api/libraries/dagster-looker#dagster_looker.RequestStartPdtBuild) to [`build_looker_pdt_assets_definitions`](https://docs.dagster.io/api/libraries/dagster-looker#dagster_looker.build_looker_pdt_assets_definitions) function. from dagster_looker import ( LookerResource, RequestStartPdtBuild, build_looker_pdt_assets_definitions, load_looker_asset_specs,)import dagster as dglooker_resource = LookerResource( client_id=dg.EnvVar("LOOKERSDK_CLIENT_ID"), client_secret=dg.EnvVar("LOOKERSDK_CLIENT_SECRET"), base_url=dg.EnvVar("LOOKERSDK_HOST_URL"),)looker_specs = load_looker_asset_specs(looker_resource=looker_resource)pdts = build_looker_pdt_assets_definitions( resource_key="looker", request_start_pdt_builds=[ RequestStartPdtBuild(model_name="my_model", view_name="my_view") ],)defs = dg.Definitions( assets=[*pdts, *looker_specs], resources={"looker": looker_resource},) Customize upstream dependencies[​](https://docs.dagster.io/integrations/libraries/looker#customize-upstream-dependencies "Direct link to Customize upstream dependencies") --------------------------------------------------------------------------------------------------------------------------------------------------------------------------- By default, Dagster sets upstream dependencies when generating asset specs for your Looker assets. To do so, Dagster parses information about assets that are upstream of specific Looker assets from the Looker instance itself. You can customize how upstream dependencies are set on your Looker assets by passing an instance of the custom [`DagsterLookerApiTranslator`](https://docs.dagster.io/api/libraries/dagster-looker#dagster_looker.DagsterLookerApiTranslator) to the [`load_looker_asset_specs`](https://docs.dagster.io/api/libraries/dagster-looker#dagster_looker.load_looker_asset_specs) function. The below example defines `my_upstream_asset` as an upstream dependency of `my_looker_view`: class CustomDagsterLookerApiTranslator(DagsterLookerApiTranslator): def get_asset_spec( self, looker_structure: LookerApiTranslatorStructureData ) -> dg.AssetSpec: # We create the default asset spec using super() default_spec = super().get_asset_spec(looker_structure) # We customize upstream dependencies for the Looker view named `my_looker_view` return default_spec.replace_attributes( deps=["my_upstream_asset"] if looker_structure.structure_type == LookerStructureType.VIEW and looker_structure.data.view_name == "my_looker_view" else ... )looker_specs = load_looker_asset_specs( looker_resource, dagster_looker_translator=CustomDagsterLookerApiTranslator()) Note that `super()` is called in each of the overridden methods to generate the default asset spec. It is best practice to generate the default asset spec before customizing it. About Looker[​](https://docs.dagster.io/integrations/libraries/looker#about-looker "Direct link to About Looker") ------------------------------------------------------------------------------------------------------------------ **Looker** is a modern platform for data analytics and visualization. It provides a unified interface for data exploration, modeling, and visualization, making it easier to understand and analyze data. Looker integrates with various data sources and can be used to create interactive reports, dashboards, and visualizations. Related[​](https://docs.dagster.io/integrations/libraries/looker#related "Direct link to Related") --------------------------------------------------------------------------------------------------- * [`dagster-looker` API reference](https://docs.dagster.io/api/libraries/dagster-looker) * [Asset definitions](https://docs.dagster.io/guides/build/assets/defining-assets) * [Resources](https://docs.dagster.io/guides/build/external-resources) * [Using environment variables and secrets](https://docs.dagster.io/guides/operate/configuration/using-environment-variables-and-secrets) * [What you'll learn](https://docs.dagster.io/integrations/libraries/looker#what-youll-learn) * [Set up your environment](https://docs.dagster.io/integrations/libraries/looker#set-up-your-environment) * [Represent Looker assets in the asset graph](https://docs.dagster.io/integrations/libraries/looker#represent-looker-assets-in-the-asset-graph) * [Load Looker assets from filtered dashboards and explores](https://docs.dagster.io/integrations/libraries/looker#load-looker-assets-from-filtered-dashboards-and-explores) * [Customize asset definition metadata for Looker assets](https://docs.dagster.io/integrations/libraries/looker#customize-asset-definition-metadata-for-looker-assets) * [Materialize Looker PDTs from Dagster](https://docs.dagster.io/integrations/libraries/looker#materialize-looker-pdts-from-dagster) * [Customize upstream dependencies](https://docs.dagster.io/integrations/libraries/looker#customize-upstream-dependencies) * [About Looker](https://docs.dagster.io/integrations/libraries/looker#about-looker) * [Related](https://docs.dagster.io/integrations/libraries/looker#related) --- # Dagster & Pandas | Dagster Docs [Skip to main content](https://docs.dagster.io/integrations/libraries/pandas#__docusaurus_skipToContent_fallback) On this page info This feature is considered in a beta stage. It is still being tested and may change. For more information, see the [API lifecycle stages documentation](https://docs.dagster.io/api/api-lifecycle/api-lifecycle-stages) . note This page describes the `dagster-pandas` library, which is used for performing data validation. To simply use pandas with Dagster, start with the [Dagster Quickstart](https://docs.dagster.io/getting-started/quickstart) . Dagster makes it easy to use pandas code to manipulate data and then store that data in other systems such as [files on Amazon S3](https://docs.dagster.io/api/libraries/dagster-aws#dagster_aws.s3.s3_pickle_io_manager) or [tables in Snowflake](https://docs.dagster.io/integrations/libraries/snowflake/using-snowflake-with-dagster) * [Creating Dagster DataFrame Types](https://docs.dagster.io/integrations/libraries/pandas#creating-dagster-dataframe-types) * [Dagster DataFrame Level Validation](https://docs.dagster.io/integrations/libraries/pandas#dagster-dataframe-level-validation) * [Dagster DataFrame Summary Statistics](https://docs.dagster.io/integrations/libraries/pandas#dagster-dataframe-summary-statistics) The `dagster_pandas` library provides the ability to perform data validation, emit summary statistics, and enable reliable dataframe serialization/deserialization. On top of this, the Dagster type system generates documentation of your dataframe constraints and makes it accessible in the Dagster UI. Creating Dagster DataFrame Types[​](https://docs.dagster.io/integrations/libraries/pandas#creating-dagster-dataframe-types "Direct link to Creating Dagster DataFrame Types") ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ To create a custom `dagster_pandas` type, use `create_dagster_pandas_dataframe_type` and provide a list of `PandasColumn` objects which specify column-level schema and constraints. For example, we can construct a custom dataframe type to represent a set of e-bike trips in the following way: TripDataFrame = create_dagster_pandas_dataframe_type( name="TripDataFrame", columns=[ PandasColumn.integer_column("bike_id", min_value=0), PandasColumn.categorical_column("color", categories={"red", "green", "blue"}), PandasColumn.datetime_column( "start_time", min_datetime=Timestamp(year=2020, month=2, day=10) ), PandasColumn.datetime_column( "end_time", min_datetime=Timestamp(year=2020, month=2, day=10) ), PandasColumn.string_column("station"), PandasColumn.exists("amount_paid"), PandasColumn.boolean_column("was_member"), ],) Once our custom data type is defined, we can use it as the type declaration for the inputs / outputs of our ops: @op(out=Out(TripDataFrame))def load_trip_dataframe() -> DataFrame: return read_csv( file_relative_path(__file__, "./ebike_trips.csv"), parse_dates=["start_time", "end_time"], date_parser=lambda x: datetime.strptime(x, "%Y-%m-%d %H:%M:%S.%f"), dtype={"color": "category"}, ) By passing in these `PandasColumn` objects, we are expressing the schema and constraints we expect our dataframes to follow when Dagster performs type checks for our ops. Moreover, if we go to the op viewer, we can follow our schema documented in the UI: ![tutorial2](https://docs.dagster.io/assets/images/tutorial2-80148af9446de9eec07a26c343be5d33.png) Dagster DataFrame Level Validation[​](https://docs.dagster.io/integrations/libraries/pandas#dagster-dataframe-level-validation "Direct link to Dagster DataFrame Level Validation") ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ Now that we have a custom dataframe type that performs schema validation during a run, we can express dataframe level constraints (e.g number of rows, or columns). To do this, we provide a list of dataframe constraints to `create_dagster_pandas_dataframe_type`; for example, using `RowCountConstraint`. More information on the available constraints can be found in the `dagster_pandas` [API docs](https://docs.dagster.io/api/libraries/dagster-pandas) . This looks like: ShapeConstrainedTripDataFrame = create_dagster_pandas_dataframe_type( name="ShapeConstrainedTripDataFrame", dataframe_constraints=[RowCountConstraint(4)]) If we rerun the above example with this dataframe, nothing should change. However, if we pass in 100 to the row count constraint, we can watch our job fail that type check. Dagster DataFrame Summary Statistics[​](https://docs.dagster.io/integrations/libraries/pandas#dagster-dataframe-summary-statistics "Direct link to Dagster DataFrame Summary Statistics") ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ Aside from constraint validation, `create_dagster_pandas_dataframe_type` also takes in a summary statistics function that emits metadata dictionaries which are surfaced during runs. Since data systems seldom control the quality of the data they receive, it becomes important to monitor data as it flows through your systems. In complex jobs, this can help debug and monitor data drift over time. Let's illustrate how this works in our example: def compute_trip_dataframe_summary_statistics(dataframe): return { "min_start_time": min(dataframe["start_time"]).strftime("%Y-%m-%d"), "max_end_time": max(dataframe["end_time"]).strftime("%Y-%m-%d"), "num_unique_bikes": str(dataframe["bike_id"].nunique()), "n_rows": len(dataframe), "columns": str(dataframe.columns), }SummaryStatsTripDataFrame = create_dagster_pandas_dataframe_type( name="SummaryStatsTripDataFrame", metadata_fn=compute_trip_dataframe_summary_statistics,) Now if we run this job in the UI launchpad, we can see that the `SummaryStatsTripDataFrame` type is displayed in the logs along with the emitted metadata. ![tutorial1.png](https://docs.dagster.io/assets/images/tutorial1-9c165e4704f7bfd16f85a60a19c07a4e.png) Dagster DataFrame Custom Validation[​](https://docs.dagster.io/integrations/libraries/pandas#dagster-dataframe-custom-validation "Direct link to Dagster DataFrame Custom Validation") --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- `PandasColumn` is user-pluggable with custom constraints. They can be constructed directly and passed a list of `ColumnConstraint` objects. To tie this back to our example, let's say that we want to validate that the amount paid for a e-bike must be in 5 dollar increments because that is the price per mile rounded up. As a result, let's implement a `DivisibleByFiveConstraint`. To do this, all it needs is a `markdown_description` for the UI which accepts and renders markdown syntax, an `error_description` for error logs, and a validation method which throws a `ColumnConstraintViolationException` if a row fails validation. This would look like the following: class DivisibleByFiveConstraint(ColumnConstraint): def __init__(self): message = "Value must be divisible by 5" super().__init__(error_description=message, markdown_description=message) def validate(self, dataframe, column_name): rows_with_unexpected_buckets = dataframe[ dataframe[column_name].apply(lambda x: x % 5 != 0) ] if not rows_with_unexpected_buckets.empty: raise ColumnConstraintViolationException( constraint_name=self.name, constraint_description=self.error_description, column_name=column_name, offending_rows=rows_with_unexpected_buckets, )CustomTripDataFrame = create_dagster_pandas_dataframe_type( name="CustomTripDataFrame", columns=[ PandasColumn( "amount_paid", constraints=[ ColumnDTypeInSetConstraint({"int64"}), DivisibleByFiveConstraint(), ], ) ],) About Pandas[​](https://docs.dagster.io/integrations/libraries/pandas#about-pandas "Direct link to About Pandas") ------------------------------------------------------------------------------------------------------------------ **Pandas** is a popular Python package that provides data structures designed to make working with "relational" or "labeled" data both easy and intuitive. Pandas aims to be the fundamental high-level building block for doing practical, real-world data analysis in Python. * [Creating Dagster DataFrame Types](https://docs.dagster.io/integrations/libraries/pandas#creating-dagster-dataframe-types) * [Dagster DataFrame Level Validation](https://docs.dagster.io/integrations/libraries/pandas#dagster-dataframe-level-validation) * [Dagster DataFrame Summary Statistics](https://docs.dagster.io/integrations/libraries/pandas#dagster-dataframe-summary-statistics) * [Dagster DataFrame Custom Validation](https://docs.dagster.io/integrations/libraries/pandas#dagster-dataframe-custom-validation) * [About Pandas](https://docs.dagster.io/integrations/libraries/pandas#about-pandas) --- # BigQuery integration reference | Dagster Docs [Skip to main content](https://docs.dagster.io/integrations/libraries/gcp/bigquery/reference#__docusaurus_skipToContent_fallback) On this page This reference page provides information for working with features that are not covered as part of the [Using Dagster with BigQuery tutorial](https://docs.dagster.io/integrations/libraries/gcp/bigquery/using-bigquery-with-dagster) . * [Providing credentials as configuration](https://docs.dagster.io/integrations/libraries/gcp/bigquery/reference#providing-credentials-as-configuration) * [Selecting specific columns in a downstream asset](https://docs.dagster.io/integrations/libraries/gcp/bigquery/reference#selecting-specific-columns-in-a-downstream-asset) * [Storing partitioned assets](https://docs.dagster.io/integrations/libraries/gcp/bigquery/reference#storing-partitioned-assets) * [Storing tables in multiple datasets](https://docs.dagster.io/integrations/libraries/gcp/bigquery/reference#storing-tables-in-multiple-datasets) * [Using the BigQuery I/O manager with other I/O managers](https://docs.dagster.io/integrations/libraries/gcp/bigquery/reference#using-the-bigquery-io-manager-with-other-io-managers) * [Storing and loading PySpark DataFrames in BigQuery](https://docs.dagster.io/integrations/libraries/gcp/bigquery/reference#storing-and-loading-pyspark-dataframes-in-bigquery) * [Using Pandas and PySpark DataFrames with BigQuery](https://docs.dagster.io/integrations/libraries/gcp/bigquery/reference#using-pandas-and-pyspark-dataframes-with-bigquery) * [Executing custom SQL commands with the BigQuery resource](https://docs.dagster.io/integrations/libraries/gcp/bigquery/reference#executing-custom-sql-commands-with-the-bigquery-resource) Providing credentials as configuration[​](https://docs.dagster.io/integrations/libraries/gcp/bigquery/reference#providing-credentials-as-configuration "Direct link to Providing credentials as configuration") ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- In most cases, you will authenticate with Google Cloud Project (GCP) using one of the methods outlined in the [GCP documentation](https://cloud.google.com/docs/authentication/provide-credentials-adc) . However, in some cases you may find that you need to provide authentication credentials directly to the BigQuery I/O manager. For example, if you are using [Dagster+ Serverless](https://docs.dagster.io/deployment/dagster-plus/serverless) you cannot upload a credential file, so must provide your credentials as an environment variable. You can provide credentials directly to the BigQuery I/O manager by using the `gcp_credentials` configuration value. The BigQuery I/O manager will create a temporary file to store the credential and will set `GOOGLE_APPLICATION_CREDENTIALS` to point to this file. When the Dagster run is completed, the temporary file is deleted and `GOOGLE_APPLICATION_CREDENTIALS` is unset. To avoid issues with newline characters in the GCP credential key, you must base64 encode the key. For example, if your GCP key is stored at `~/.gcp/key.json` you can base64 encode the key by using the following shell command: cat ~/.gcp/key.json | base64 Then you can [set an environment variable](https://docs.dagster.io/guides/operate/configuration/using-environment-variables-and-secrets) in your Dagster deployment (for example `GCP_CREDS`) to the encoded key and provide it to the BigQuery I/O manager: from dagster_gcp_pandas import BigQueryPandasIOManagerfrom dagster import Definitions, EnvVardefs = Definitions( assets=[iris_data], resources={ "io_manager": BigQueryPandasIOManager( project="my-gcp-project", location="us-east5", dataset="IRIS", timeout=15.0, gcp_credentials=EnvVar("GCP_CREDS"), ) },) Selecting specific columns in a downstream asset[​](https://docs.dagster.io/integrations/libraries/gcp/bigquery/reference#selecting-specific-columns-in-a-downstream-asset "Direct link to Selecting specific columns in a downstream asset") ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Sometimes you may not want to fetch an entire table as the input to a downstream asset. With the BigQuery I/O manager, you can select specific columns to load by supplying metadata on the downstream asset. import pandas as pdfrom dagster import AssetIn, asset# this example uses the iris_data asset from Step 2 of the Using Dagster with BigQuery tutorial@asset( ins={ "iris_sepal": AssetIn( key="iris_data", metadata={"columns": ["sepal_length_cm", "sepal_width_cm"]}, ) })def sepal_data(iris_sepal: pd.DataFrame) -> pd.DataFrame: iris_sepal["sepal_area_cm2"] = ( iris_sepal["sepal_length_cm"] * iris_sepal["sepal_width_cm"] ) return iris_sepal In this example, we only use the columns containing sepal data from the `IRIS_DATA` table created in [Step 2: Create tables in BigQuery](https://docs.dagster.io/integrations/libraries/gcp/bigquery/using-bigquery-with-dagster#step-2-create-tables-in-bigquery) of the [Using Dagster with BigQuery tutorial](https://docs.dagster.io/integrations/libraries/gcp/bigquery/using-bigquery-with-dagster) . Fetching the entire table would be unnecessarily costly, so to select specific columns, we can add metadata to the input asset. We do this in the `metadata` parameter of the `AssetIn` that loads the `iris_data` asset in the `ins` parameter. We supply the key `columns` with a list of names of the columns we want to fetch. When Dagster materializes `sepal_data` and loads the `iris_data` asset using the BigQuery I/O manager, it will only fetch the `sepal_length_cm` and `sepal_width_cm` columns of the `IRIS.IRIS_DATA` table and pass them to `sepal_data` as a Pandas DataFrame. Storing partitioned assets[​](https://docs.dagster.io/integrations/libraries/gcp/bigquery/reference#storing-partitioned-assets "Direct link to Storing partitioned assets") ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- The BigQuery I/O manager supports storing and loading partitioned data. In order to correctly store and load data from the BigQuery table, the BigQuery I/O manager needs to know which column contains the data defining the partition bounds. The BigQuery I/O manager uses this information to construct the correct queries to select or replace the data. In the following sections, we describe how the I/O manager constructs these queries for different types of partitions. * Static partitioned assets * Time-partitioned assets * Multi-partitioned assets **Storing static partitioned assets** In order to store static partitioned assets in BigQuery, you must specify `partition_expr` metadata on the asset to tell the BigQuery I/O manager which column contains the partition data: import pandas as pdfrom dagster import AssetExecutionContext, StaticPartitionsDefinition, asset@asset( partitions_def=StaticPartitionsDefinition( ["Iris-setosa", "Iris-virginica", "Iris-versicolor"] ), metadata={"partition_expr": "SPECIES"},)def iris_data_partitioned(context: AssetExecutionContext) -> pd.DataFrame: species = context.partition_key full_df = pd.read_csv( "https://docs.dagster.io/assets/iris.csv", names=[ "sepal_length_cm", "sepal_width_cm", "petal_length_cm", "petal_width_cm", "species", ], ) return full_df[full_df["species"] == species]@assetdef iris_cleaned(iris_data_partitioned: pd.DataFrame): return iris_data_partitioned.dropna().drop_duplicates() Dagster uses the `partition_expr` metadata to craft the `SELECT` statement when loading the partition in the downstream asset. When loading a static partition, the following statement is used: SELECT * WHERE [partition_expr] = ([selected partitions]) When the `partition_expr` value is injected into this statement, the resulting SQL query must follow BigQuery's SQL syntax. Refer to the [BigQuery documentation](https://cloud.google.com/bigquery/docs/reference/standard-sql/query-syntax) for more information. When materializing the above assets, a partition must be selected, as described in the [Partitioning assets](https://docs.dagster.io/guides/build/partitions-and-backfills/partitioning-assets#materializing-partitioned-assets) documentation. In this example, the query used when materializing the `Iris-setosa` partition of the above assets would be: SELECT * WHERE SPECIES in ('Iris-setosa') **Storing time partitioned assets** Like static partitioned assets, you can specify `partition_expr` metadata on the asset to tell the BigQuery I/O manager which column contains the partition data: import pandas as pdfrom dagster import AssetExecutionContext, DailyPartitionsDefinition, asset@asset( partitions_def=DailyPartitionsDefinition(start_date="2023-01-01"), metadata={"partition_expr": "TIMESTAMP_SECONDS(TIME)"},)def iris_data_per_day(context: AssetExecutionContext) -> pd.DataFrame: partition = context.partition_key # get_iris_data_for_date fetches all of the iris data for a given date, # the returned dataframe contains a column named 'TIME' with that stores # the time of the row as an integer of seconds since epoch return get_iris_data_for_date(partition)@assetdef iris_cleaned(iris_data_per_day: pd.DataFrame): return iris_data_per_day.dropna().drop_duplicates() Dagster uses the `partition_expr` metadata to craft the `SELECT` statement when loading the correct partition in the downstream asset. When loading a dynamic partition, the following statement is used: SELECT * WHERE [partition_expr] >= [partition_start] AND [partition_expr] < [partition_end] When the `partition_expr` value is injected into this statement, the resulting SQL query must follow BigQuery's SQL syntax. Refer to the [BigQuery documentation](https://cloud.google.com/bigquery/docs/reference/standard-sql/query-syntax) for more information. When materializing the above assets, a partition must be selected, as described in [Materializing partitioned assets](https://docs.dagster.io/guides/build/partitions-and-backfills/partitioning-assets) . The `[partition_start]` and `[partition_end]` bounds are of the form `YYYY-MM-DD HH:MM:SS`. In this example, the query when materializing the `2023-01-02` partition of the above assets would be: SELECT * WHERE TIMESTAMP_SECONDS(TIME) >= '2023-01-02 00:00:00' AND TIMESTAMP_SECONDS(TIME) < '2023-01-03 00:00:00' In this example, the data in the `TIME` column are integers, so the `partition_expr` metadata includes a SQL statement to convert integers to timestamps. A full list of BigQuery functions can be found [here](https://cloud.google.com/bigquery/docs/reference/standard-sql/functions-and-operators) . **Storing multi-partitioned assets** The BigQuery I/O manager can also store data partitioned on multiple dimensions. To do this, you must specify the column for each partition as a dictionary of `partition_expr` metadata: import pandas as pdimport dagster as dg@dg.asset( partitions_def=dg.MultiPartitionsDefinition( { "date": dg.DailyPartitionsDefinition(start_date="2023-01-01"), "species": dg.StaticPartitionsDefinition( ["Iris-setosa", "Iris-virginica", "Iris-versicolor"] ), } ), metadata={ "partition_expr": {"date": "TIMESTAMP_SECONDS(TIME)", "species": "SPECIES"} },)def iris_data_partitioned(context: dg.AssetExecutionContext) -> pd.DataFrame: partition = context.partition_key.keys_by_dimension species = partition["species"] date = partition["date"] # get_iris_data_for_date fetches all of the iris data for a given date, # the returned dataframe contains a column named 'TIME' with that stores # the time of the row as an integer of seconds since epoch full_df = get_iris_data_for_date(date) return full_df[full_df["species"] == species]@dg.assetdef iris_cleaned(iris_data_partitioned: pd.DataFrame): return iris_data_partitioned.dropna().drop_duplicates() Dagster uses the `partition_expr` metadata to craft the `SELECT` statement when loading the correct partition in a downstream asset. For multi-partitions, Dagster concatenates the `WHERE` statements described in the static partition and time-window partition sections to craft the correct `SELECT` statement. When materializing the above assets, a partition must be selected, as described in [Materializing partitioned assets](https://docs.dagster.io/guides/build/partitions-and-backfills/partitioning-assets) . For example, when materializing the `2023-01-02|Iris-setosa` partition of the above assets, the following query will be used: SELECT * WHERE SPECIES in ('Iris-setosa') AND TIMESTAMP_SECONDS(TIME) >= '2023-01-02 00:00:00' AND TIMESTAMP_SECONDS(TIME) < '2023-01-03 00:00:00'` Storing tables in multiple datasets[​](https://docs.dagster.io/integrations/libraries/gcp/bigquery/reference#storing-tables-in-multiple-datasets "Direct link to Storing tables in multiple datasets") ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- You may want to have different assets stored in different BigQuery datasets. The BigQuery I/O manager allows you to specify the dataset in several ways. You can specify the default dataset where data will be stored as configuration to the I/O manager, like we did in [Step 1: Configure the BigQuery I/O manager](https://docs.dagster.io/integrations/libraries/gcp/bigquery/using-bigquery-with-dagster#step-1-configure-the-bigquery-io-manager) of the [Using Dagster with BigQuery tutorial](https://docs.dagster.io/integrations/libraries/gcp/bigquery/using-bigquery-with-dagster) . If you want to store assets in different datasets, you can specify the dataset as metadata: daffodil_data = AssetSpec(key=["daffodil_data"], metadata={"schema": "daffodil"}) @asset(metadata={"schema": "iris"}) def iris_data() -> pd.DataFrame: return pd.read_csv( "https://docs.dagster.io/assets/iris.csv", names=[ "sepal_length_cm", "sepal_width_cm", "petal_length_cm", "petal_width_cm", "species", ], ) You can also specify the dataset as part of the asset's asset key: daffodil_data = AssetSpec(key=["gcp", "bigquery", "daffodil", "daffodil_data"]) @asset(key_prefix=["gcp", "bigquery", "iris"]) def iris_data() -> pd.DataFrame: return pd.read_csv( "https://docs.dagster.io/assets/iris.csv", names=[ "sepal_length_cm", "sepal_width_cm", "petal_length_cm", "petal_width_cm", "species", ], ) The dataset will be the last prefix before the asset's name. In this example, the `iris_data` asset will be stored in the `IRIS` dataset, and the `daffodil_data` asset will be found in the `DAFFODIL` dataset. note The dataset is determined in this order: 1. If the dataset is set via metadata, that dataset will be used 2. Otherwise, the dataset set as configuration on the I/O manager will be used 3. Otherwise, if there is a `key_prefix`, that dataset will be used 4. If none of the above are provided, the default dataset will be `PUBLIC` Using the BigQuery I/O manager with other I/O managers[​](https://docs.dagster.io/integrations/libraries/gcp/bigquery/reference#using-the-bigquery-io-manager-with-other-io-managers "Direct link to Using the BigQuery I/O manager with other I/O managers") -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- You may have assets that you don't want to store in BigQuery. You can provide an I/O manager to each asset using the `io_manager_key` parameter in the `asset` decorator: import pandas as pdfrom dagster_aws.s3.io_manager import s3_pickle_io_managerfrom dagster_gcp_pandas import BigQueryPandasIOManagerfrom dagster import Definitions, asset@asset(io_manager_key="warehouse_io_manager")def iris_data() -> pd.DataFrame: return pd.read_csv( "https://docs.dagster.io/assets/iris.csv", names=[ "sepal_length_cm", "sepal_width_cm", "petal_length_cm", "petal_width_cm", "species", ], )@asset(io_manager_key="blob_io_manager")def iris_plots(iris_data): # plot_data is a function we've defined somewhere else # that plots the data in a DataFrame return plot_data(iris_data)defs = Definitions( assets=[iris_data, iris_plots], resources={ "warehouse_io_manager": BigQueryPandasIOManager( project="my-gcp-project", dataset="IRIS", ), "blob_io_manager": s3_pickle_io_manager, },) In this example, the `iris_data` asset uses the I/O manager bound to the key `warehouse_io_manager` and `iris_plots` will use the I/O manager bound to the key `blob_io_manager`. In the [`Definitions`](https://docs.dagster.io/api/dagster/definitions#dagster.Definitions) object, we supply the I/O managers for those keys. When the assets are materialized, the `iris_data` will be stored in BigQuery, and `iris_plots` will be saved in Amazon S3. Storing and loading PySpark DataFrames in BigQuery[​](https://docs.dagster.io/integrations/libraries/gcp/bigquery/reference#storing-and-loading-pyspark-dataframes-in-bigquery "Direct link to Storing and loading PySpark DataFrames in BigQuery") ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- The BigQuery I/O manager also supports storing and loading PySpark DataFrames. To use the [`BigQueryPySparkIOManager`](https://docs.dagster.io/api/libraries/dagster-gcp-pyspark#dagster_gcp_pyspark.BigQueryPySparkIOManager) , first install the package: * uv * pip uv add dagster-gcp-pyspark pip install dagster-gcp-pyspark Then you can use the `gcp_pyspark_io_manager` in your `Definitions` as in [Step 1: Configure the BigQuery I/O manager](https://docs.dagster.io/integrations/libraries/gcp/bigquery/using-bigquery-with-dagster#step-1-configure-the-bigquery-io-manager) of the [Using Dagster with BigQuery tutorial](https://docs.dagster.io/integrations/libraries/gcp/bigquery/using-bigquery-with-dagster) . from dagster_gcp_pyspark import BigQueryPySparkIOManagerfrom dagster import Definitionsdefs = Definitions( assets=[iris_data], resources={ "io_manager": BigQueryPySparkIOManager( project="my-gcp-project", # required location="us-east5", # optional, defaults to the default location for the project - see https://cloud.google.com/bigquery/docs/locations for a list of locations dataset="IRIS", # optional, defaults to PUBLIC temporary_gcs_bucket="my-gcs-bucket", # optional, defaults to None, which will result in a direct write to BigQuery ) },) note When using the `BigQueryPySparkIOManager` you may provide the `temporary_gcs_bucket` configuration. This will store the data is a temporary GCS bucket, then all of the data into BigQuery in one operation. If not provided, data will be directly written to BigQuery. If you choose to use a temporary GCS bucket, you must include the [GCS Hadoop connector](https://github.com/GoogleCloudDataproc/hadoop-connectors/tree/master/gcs) in your Spark Session, in addition to the BigQuery connector (described below). The `BigQueryPySparkIOManager` requires that a `SparkSession` be active and configured with the [BigQuery connector for Spark](https://cloud.google.com/dataproc/docs/tutorials/bigquery-connector-spark-example) . You can either create your own `SparkSession` or use the [`spark_resource`](https://docs.dagster.io/api/libraries/dagster-spark#dagster_spark.spark_resource) . * With the spark\_resource * With your own SparkSession from dagster_gcp_pyspark import BigQueryPySparkIOManagerfrom dagster_pyspark import pyspark_resourcefrom pyspark import SparkFilesfrom pyspark.sql import DataFramefrom pyspark.sql.types import DoubleType, StringType, StructField, StructTypefrom dagster import AssetExecutionContext, Definitions, assetBIGQUERY_JARS = "com.google.cloud.spark:spark-bigquery-with-dependencies_2.12:0.28.0"@asset(required_resource_keys={"pyspark"})def iris_data(context: AssetExecutionContext) -> DataFrame: spark = context.resources.pyspark.spark_session schema = StructType( [ StructField("sepal_length_cm", DoubleType()), StructField("sepal_width_cm", DoubleType()), StructField("petal_length_cm", DoubleType()), StructField("petal_width_cm", DoubleType()), StructField("species", StringType()), ] ) url = "https://docs.dagster.io/assets/iris.csv" spark.sparkContext.addFile(url) return spark.read.schema(schema).csv("file://" + SparkFiles.get("iris.csv"))defs = Definitions( assets=[iris_data], resources={ "io_manager": BigQueryPySparkIOManager( project="my-gcp-project", location="us-east5", ), "pyspark": pyspark_resource.configured( {"spark_conf": {"spark.jars.packages": BIGQUERY_JARS}} ), },) from dagster_gcp_pyspark import BigQueryPySparkIOManagerfrom pyspark import SparkFilesfrom pyspark.sql import DataFrame, SparkSessionfrom pyspark.sql.types import DoubleType, StringType, StructField, StructTypefrom dagster import Definitions, assetBIGQUERY_JARS = "com.google.cloud.spark:spark-bigquery-with-dependencies_2.12:0.28.0"@assetdef iris_data() -> DataFrame: spark = SparkSession.builder.config( key="spark.jars.packages", value=BIGQUERY_JARS, ).getOrCreate() schema = StructType( [ StructField("sepal_length_cm", DoubleType()), StructField("sepal_width_cm", DoubleType()), StructField("petal_length_cm", DoubleType()), StructField("petal_width_cm", DoubleType()), StructField("species", StringType()), ] ) url = "https://docs.dagster.io/assets/iris.csv" spark.sparkContext.addFile(url) return spark.read.schema(schema).csv("file://" + SparkFiles.get("iris.csv"))defs = Definitions( assets=[iris_data], resources={ "io_manager": BigQueryPySparkIOManager( project="my-gcp-project", location="us-east5", ), },) note In order to load data from BigQuery as a PySpark DataFrame, the BigQuery PySpark connector will create a view containing the data. This will result in the creation of a temporary table in your BigQuery dataset. For more details, see the [BigQuery PySpark connector documentation](https://github.com/GoogleCloudDataproc/spark-bigquery-connector#reading-data-from-a-bigquery-query) . Using Pandas and PySpark DataFrames with BigQuery[​](https://docs.dagster.io/integrations/libraries/gcp/bigquery/reference#using-pandas-and-pyspark-dataframes-with-bigquery "Direct link to Using Pandas and PySpark DataFrames with BigQuery") ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- If you work with both Pandas and PySpark DataFrames and want a single I/O manager to handle storing and loading these DataFrames in BigQuery, you can write a new I/O manager that handles both types. To do this, inherit from the [`BigQueryIOManager`](https://docs.dagster.io/api/libraries/dagster-gcp#dagster_gcp.BigQueryIOManager) base class and implement the `type_handlers` and `default_load_type` methods. The resulting I/O manager will inherit the configuration fields of the base `BigQueryIOManager`. from collections.abc import Sequencefrom typing import Optionalimport pandas as pdfrom dagster_gcp import BigQueryIOManagerfrom dagster_gcp_pandas import BigQueryPandasTypeHandlerfrom dagster_gcp_pyspark import BigQueryPySparkTypeHandlerfrom dagster import Definitionsfrom dagster._core.storage.db_io_manager import DbTypeHandlerclass MyBigQueryIOManager(BigQueryIOManager): @staticmethod def type_handlers() -> Sequence[DbTypeHandler]: """type_handlers should return a list of the TypeHandlers that the I/O manager can use. Here we return the BigQueryPandasTypeHandler and BigQueryPySparkTypeHandler so that the I/O manager can store Pandas DataFrames and PySpark DataFrames. """ return [BigQueryPandasTypeHandler(), BigQueryPySparkTypeHandler()] @staticmethod def default_load_type() -> Optional[type]: """If an asset is not annotated with an return type, default_load_type will be used to determine which TypeHandler to use to store and load the output. In this case, unannotated assets will be stored and loaded as Pandas DataFrames. """ return pd.DataFramedefs = Definitions( assets=[iris_data, rose_data], resources={ "io_manager": MyBigQueryIOManager(project="my-gcp-project", dataset="FLOWERS") },) Executing custom SQL commands with the BigQuery resource[​](https://docs.dagster.io/integrations/libraries/gcp/bigquery/reference#executing-custom-sql-commands-with-the-bigquery-resource "Direct link to Executing custom SQL commands with the BigQuery resource") ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- In addition to the BigQuery I/O manager, Dagster also provides a BigQuery [resource](https://docs.dagster.io/guides/build/external-resources) for executing custom SQL queries. from dagster_gcp import BigQueryResourcefrom dagster import Definitions, asset# this example executes a query against the IRIS.IRIS_DATA table created in Step 2 of the# Using Dagster with BigQuery tutorial@assetdef small_petals(bigquery: BigQueryResource): with bigquery.get_client() as client: return client.query( 'SELECT * FROM IRIS.IRIS_DATA WHERE "petal_length_cm" < 1 AND' ' "petal_width_cm" < 1', ).result()defs = Definitions( assets=[small_petals], resources={ "bigquery": BigQueryResource( project="my-gcp-project", location="us-east5", ) },) In this example, we attach the BigQuery resource to the `small_petals` asset. In the body of the asset function, we use the `get_client` context manager method of the resource to get a [`bigquery.client.Client`](https://cloud.google.com/python/docs/reference/bigquery/latest/google.cloud.bigquery.client.Client) . We can use the client to execute a custom SQL query against the `IRIS_DATA` table created in [Step 2: Create tables in BigQuery](https://docs.dagster.io/integrations/libraries/gcp/bigquery/using-bigquery-with-dagster#step-2-create-tables-in-bigquery) of the [Using Dagster with BigQuery tutorial](https://docs.dagster.io/integrations/libraries/gcp/bigquery/using-bigquery-with-dagster) . * [Providing credentials as configuration](https://docs.dagster.io/integrations/libraries/gcp/bigquery/reference#providing-credentials-as-configuration) * [Selecting specific columns in a downstream asset](https://docs.dagster.io/integrations/libraries/gcp/bigquery/reference#selecting-specific-columns-in-a-downstream-asset) * [Storing partitioned assets](https://docs.dagster.io/integrations/libraries/gcp/bigquery/reference#storing-partitioned-assets) * [Storing tables in multiple datasets](https://docs.dagster.io/integrations/libraries/gcp/bigquery/reference#storing-tables-in-multiple-datasets) * [Using the BigQuery I/O manager with other I/O managers](https://docs.dagster.io/integrations/libraries/gcp/bigquery/reference#using-the-bigquery-io-manager-with-other-io-managers) * [Storing and loading PySpark DataFrames in BigQuery](https://docs.dagster.io/integrations/libraries/gcp/bigquery/reference#storing-and-loading-pyspark-dataframes-in-bigquery) * [Using Pandas and PySpark DataFrames with BigQuery](https://docs.dagster.io/integrations/libraries/gcp/bigquery/reference#using-pandas-and-pyspark-dataframes-with-bigquery) * [Executing custom SQL commands with the BigQuery resource](https://docs.dagster.io/integrations/libraries/gcp/bigquery/reference#executing-custom-sql-commands-with-the-bigquery-resource) --- # Dagster & Kubernetes | Dagster Docs [Skip to main content](https://docs.dagster.io/integrations/libraries/kubernetes#__docusaurus_skipToContent_fallback) On this page The Kubernetes integration library provides the PipesK8sClient resource, enabling you to launch Kubernetes pods and execute external code directly from Dagster assets and ops. This integration allows you to pass parameters to Kubernetes pods while Dagster receives real-time events, such as logs, asset checks, and asset materializations, from the initiated jobs. With minimal code changes required on the job side, this integration is both efficient and easy to implement. Installation[​](https://docs.dagster.io/integrations/libraries/kubernetes#installation "Direct link to Installation") ---------------------------------------------------------------------------------------------------------------------- * uv * pip uv add dagster-k8s pip install dagster-k8s Example[​](https://docs.dagster.io/integrations/libraries/kubernetes#example "Direct link to Example") ------------------------------------------------------------------------------------------------------- from dagster_k8s import PipesK8sClientimport dagster as dg@dg.assetdef k8s_pipes_asset( context: dg.AssetExecutionContext, k8s_pipes_client: PipesK8sClient): return k8s_pipes_client.run( context=context, image="pipes-example:v1", ).get_materialize_result()defs = dg.Definitions( assets=[k8s_pipes_asset], resources={ "k8s_pipes_client": PipesK8sClient(), },) Deploying to Kubernetes?[​](https://docs.dagster.io/integrations/libraries/kubernetes#deploying-to-kubernetes "Direct link to Deploying to Kubernetes?") --------------------------------------------------------------------------------------------------------------------------------------------------------- * Deploying to Dagster+: Use with a Dagster+ Hybrid deployment, the Kubernetes agent executes Dagster jobs on a Kubernetes cluster. Checkout the [Dagster+ Kubernetes Agent](https://docs.dagster.io/dagster-plus/deployment/deployment-types/hybrid/kubernetes) guide for more information. * Deploying to Open Source: Visit the [Deploying Dagster to Kubernetes](https://docs.dagster.io/guides/deploy/deployment-options/kubernetes) guide for more information. About Kubernetes[​](https://docs.dagster.io/integrations/libraries/kubernetes#about-kubernetes "Direct link to About Kubernetes") ---------------------------------------------------------------------------------------------------------------------------------- **Kubernetes** is an open source container orchestration system for automating software deployment, scaling, and management. Google originally designed Kubernetes, but the Cloud Native Computing Foundation now maintains the project. * [Installation](https://docs.dagster.io/integrations/libraries/kubernetes#installation) * [Example](https://docs.dagster.io/integrations/libraries/kubernetes#example) * [Deploying to Kubernetes?](https://docs.dagster.io/integrations/libraries/kubernetes#deploying-to-kubernetes) * [About Kubernetes](https://docs.dagster.io/integrations/libraries/kubernetes#about-kubernetes) --- # Dagster & Snowflake | Dagster Docs [Skip to main content](https://docs.dagster.io/integrations/libraries/snowflake#__docusaurus_skipToContent_fallback) On this page This library provides an integration with the Snowflake data warehouse. Connect to Snowflake as a resource, then use the integration-provided functions to construct an op to establish connections and execute Snowflake queries. Read and write natively to Snowflake from Dagster assets. Installation[​](https://docs.dagster.io/integrations/libraries/snowflake#installation "Direct link to Installation") --------------------------------------------------------------------------------------------------------------------- * uv * pip uv add dagster-snowflake pip install dagster-snowflake Example[​](https://docs.dagster.io/integrations/libraries/snowflake#example "Direct link to Example") ------------------------------------------------------------------------------------------------------ from dagster_snowflake import SnowflakeResourceimport dagster as dg@dg.assetdef my_table(snowflake: SnowflakeResource): with snowflake.get_connection() as conn: return conn.cursor().execute_query("SELECT * FROM foo")defs = dg.Definitions( assets=[my_table], resources={ "snowflake": SnowflakeResource( account="snowflake account", user="snowflake user", password=dg.EnvVar("SNOWFLAKE_PASSWORD"), database="snowflake database", schema="snowflake schema", warehouse="snowflake warehouse", ) },) About Snowflake[​](https://docs.dagster.io/integrations/libraries/snowflake#about-snowflake "Direct link to About Snowflake") ------------------------------------------------------------------------------------------------------------------------------ A cloud-based data storage and analytics service, generally termed "data-as-a-service". **Snowflake**'s data warehouse is one of the most widely adopted cloud warehouses for analytics. * [Installation](https://docs.dagster.io/integrations/libraries/snowflake#installation) * [Example](https://docs.dagster.io/integrations/libraries/snowflake#example) * [About Snowflake](https://docs.dagster.io/integrations/libraries/snowflake#about-snowflake) --- # Using Google BigQuery with Dagster | Dagster Docs [Skip to main content](https://docs.dagster.io/integrations/libraries/gcp/bigquery/using-bigquery-with-dagster#__docusaurus_skipToContent_fallback) On this page info This feature is considered in a beta stage. It is still being tested and may change. For more information, see the [API lifecycle stages documentation](https://docs.dagster.io/api/api-lifecycle/api-lifecycle-stages) . This tutorial focuses on creating and interacting with BigQuery tables using Dagster's [asset definitions](https://docs.dagster.io/guides/build/assets/defining-assets) . The `dagster-gcp` library provides two ways to interact with BigQuery tables: * [Resource](https://docs.dagster.io/guides/build/external-resources) : The resource allows you to directly run SQL queries against tables within an asset's compute function. Available resources: [`BigQueryResource`](https://docs.dagster.io/api/libraries/dagster-gcp#dagster_gcp.BigQueryResource) * [I/O manager](https://docs.dagster.io/guides/build/io-managers) : The I/O manager transfers the responsibility of storing and loading DataFrames as BigQuery tables to Dagster. Available I/O managers: [`BigQueryPandasIOManager`](https://docs.dagster.io/api/libraries/dagster-gcp-pandas#dagster_gcp_pandas.BigQueryPandasIOManager) , [`BigQueryPySparkIOManager`](https://docs.dagster.io/api/libraries/dagster-gcp-pyspark#dagster_gcp_pyspark.BigQueryPySparkIOManager) This tutorial is divided into two sections to demonstrate the differences between the BigQuery resource and the BigQuery I/O manager. Each section will create the same assets, but the first section will use the BigQuery resource to store data in BigQuery, whereas the second section will use the BigQuery I/O manager. When writing your own assets, you may choose one or the other (or both) approaches depending on your storage requirements. In [Option 1](https://docs.dagster.io/integrations/libraries/gcp/bigquery/using-bigquery-with-dagster#option-1-using-the-bigquery-resource) you will: * Set up and configure the BigQuery resource. * Use the BigQuery resource to execute a SQL query to create a table. * Use the BigQuery resource to execute a SQL query to interact with the table. In [Option 2](https://docs.dagster.io/integrations/libraries/gcp/bigquery/using-bigquery-with-dagster#option-2-using-the-bigquery-io-manager) you will: * Set up and configure the BigQuery I/O manager. * Use Pandas to create a DataFrame, then delegate responsibility creating a table to the BigQuery I/O manager. * Use the BigQuery I/O manager to load the table into memory so that you can interact with it using the Pandas library. By the end of the tutorial, you will: * Understand how to interact with a BigQuery database using the BigQuery resource. * Understand how to use the BigQuery I/O manager to store and load DataFrames as BigQuery tables. * Understand how to define dependencies between assets corresponding to tables in a BigQuery database. Prerequisites[​](https://docs.dagster.io/integrations/libraries/gcp/bigquery/using-bigquery-with-dagster#prerequisites "Direct link to Prerequisites") ------------------------------------------------------------------------------------------------------------------------------------------------------- To complete this tutorial, you'll need: * **To install the `dagster-gcp` and `dagster-gcp-pandas` libraries**: * uv * pip uv add dagster-gcp dagster-gcp-pandas pip install dagster-gcp dagster-gcp-pandas * **To gather the following information**: * **Google Cloud Project (GCP) project name**: You can find this by logging into GCP and choosing one of the project names listed in the dropdown in the top left corner. * **GCP credentials**: You can authenticate with GCP two ways: by following GCP authentication instructions [here](https://cloud.google.com/docs/authentication/provide-credentials-adc) , or by providing credentials directly to the BigQuery I/O manager. In this guide, we assume that you have run one of the `gcloud auth` commands or have set `GOOGLE_APPLICATION_CREDENTIALS` as specified in the linked instructions. For more information on providing credentials directly to the BigQuery resource and I/O manager, see [Providing credentials as configuration](https://docs.dagster.io/integrations/libraries/gcp/bigquery/reference#providing-credentials-as-configuration) in the BigQuery reference guide. Option 1: Using the BigQuery resource[​](https://docs.dagster.io/integrations/libraries/gcp/bigquery/using-bigquery-with-dagster#option-1-using-the-bigquery-resource "Direct link to Option 1: Using the BigQuery resource") ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ ### Step 1: Configure the BigQuery resource[​](https://docs.dagster.io/integrations/libraries/gcp/bigquery/using-bigquery-with-dagster#step-1-configure-the-bigquery-resource "Direct link to Step 1: Configure the BigQuery resource") To use the BigQuery resource, you'll need to add it to your `Definitions` object. The BigQuery resource requires some configuration: * A `project` * One method of authentication. You can follow the GCP authentication instructions [here](https://cloud.google.com/docs/authentication/provide-credentials-adc) , or see [Providing credentials as configuration](https://docs.dagster.io/integrations/libraries/gcp/bigquery/reference#providing-credentials-as-configuration) in the BigQuery reference guide. You can also specify a `location` where computation should take place. from dagster_gcp import BigQueryResourcefrom dagster import Definitionsdefs = Definitions( assets=[iris_data], resources={ "bigquery": BigQueryResource( project="my-gcp-project", # required location="us-east5", # optional, defaults to the default location for the project - see https://cloud.google.com/bigquery/docs/locations for a list of locations ) },) ### Step 2: Create tables in BigQuery[​](https://docs.dagster.io/integrations/libraries/gcp/bigquery/using-bigquery-with-dagster#step-2-create-tables-in-bigquery "Direct link to Step 2: Create tables in BigQuery") * Create BigQuery tables in Dagster * Making Dagster aware of existing tables **Create BigQuery tables in Dagster** Using the BigQuery resource, you can create BigQuery tables using the BigQuery Python API: import pandas as pdfrom dagster_gcp import BigQueryResourcefrom dagster import asset@assetdef iris_data(bigquery: BigQueryResource) -> None: iris_df = pd.read_csv( "https://docs.dagster.io/assets/iris.csv", names=[ "sepal_length_cm", "sepal_width_cm", "petal_length_cm", "petal_width_cm", "species", ], ) with bigquery.get_client() as client: job = client.load_table_from_dataframe( dataframe=iris_df, destination="iris.iris_data", ) job.result() In this example, you're defining an asset that fetches the Iris dataset as a Pandas DataFrame and renames the columns. Then, using the BigQuery resource, the DataFrame is stored in BigQuery as the `iris.iris_data` table. Now you can run `dagster dev` and materialize the `iris_data` asset from the Dagster UI. **Making Dagster aware of existing tables** If you already have existing tables in BigQuery and other assets defined in Dagster depend on those tables, you may want Dagster to be aware of those upstream dependencies. Making Dagster aware of these tables will allow you to track the full data lineage in Dagster. You can accomplish this by defining [external assets](https://docs.dagster.io/guides/build/assets/external-assets) for these tables. from dagster import AssetSpeciris_harvest_data = AssetSpec(key="iris_harvest_data") In this example, you're creating an [`AssetSpec`](https://docs.dagster.io/api/dagster/assets#dagster.AssetSpec) for a pre-existing table called `iris_harvest_data`. ### Step 3: Define downstream assets[​](https://docs.dagster.io/integrations/libraries/gcp/bigquery/using-bigquery-with-dagster#step-3-define-downstream-assets "Direct link to Step 3: Define downstream assets") Once you have created an asset that represents a table in BigQuery, you will likely want to create additional assets that work with the data. from dagster import assetfrom .create_table import iris_data# this example uses the iris_dataset asset from Step 2@asset(deps=[iris_data])def iris_setosa(bigquery: BigQueryResource) -> None: job_config = bq.QueryJobConfig(destination="iris.iris_setosa") sql = "SELECT * FROM iris.iris_data WHERE species = 'Iris-setosa'" with bigquery.get_client() as client: job = client.query(sql, job_config=job_config) job.result() In this asset, you're creating second table that only contains the data for the _Iris Setosa_ species. This asset has a dependency on the `iris_data` asset. To define this dependency, you provide the `iris_data` asset as the `deps` parameter to the `iris_setosa` asset. You can then run the SQL query to create the table of _Iris Setosa_ data. ### Completed code example[​](https://docs.dagster.io/integrations/libraries/gcp/bigquery/using-bigquery-with-dagster#completed-code-example "Direct link to Completed code example") When finished, your code should look like the following: import pandas as pdfrom dagster_gcp import BigQueryResourcefrom google.cloud import bigquery as bqfrom dagster import AssetSpec, Definitions, assetiris_harvest_data = AssetSpec(key="iris_harvest_data")@assetdef iris_data(bigquery: BigQueryResource) -> None: iris_df = pd.read_csv( "https://docs.dagster.io/assets/iris.csv", names=[ "sepal_length_cm", "sepal_width_cm", "petal_length_cm", "petal_width_cm", "species", ], ) with bigquery.get_client() as client: job = client.load_table_from_dataframe( dataframe=iris_df, destination="iris.iris_data", ) job.result()@asset(deps=[iris_data])def iris_setosa(bigquery: BigQueryResource) -> None: job_config = bq.QueryJobConfig(destination="iris.iris_setosa") sql = "SELECT * FROM iris.iris_data WHERE species = 'Iris-setosa'" with bigquery.get_client() as client: job = client.query(sql, job_config=job_config) job.result()defs = Definitions( assets=[iris_data, iris_setosa, iris_harvest_data], resources={ "bigquery": BigQueryResource( project="my-gcp-project", location="us-east5", ) },) Option 2: Using the BigQuery I/O manager[​](https://docs.dagster.io/integrations/libraries/gcp/bigquery/using-bigquery-with-dagster#option-2-using-the-bigquery-io-manager "Direct link to Option 2: Using the BigQuery I/O manager") -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- While using an I/O manager is not required, you may want to use an I/O manager to handle storing DataFrames as tables in BigQuery and loading BigQuery tables as DataFrames in downstream assets. You may want to use an I/O manager if: * You want your data to be loaded in memory so that you can interact with it using Python. * You'd like to have Dagster manage how you store the data and load it as an input in downstream assets. This section of the guide focuses on storing and loading Pandas DataFrames in BigQuery, but Dagster also supports using PySpark DataFrames with BigQuery. The concepts from this guide apply to working with PySpark DataFrames, and you can learn more about setting up and using the BigQuery I/O manager with PySpark DataFrames in the [reference guide](https://docs.dagster.io/integrations/libraries/gcp/bigquery/reference) . ### Step 1: Configure the BigQuery I/O manager[​](https://docs.dagster.io/integrations/libraries/gcp/bigquery/using-bigquery-with-dagster#step-1-configure-the-bigquery-io-manager "Direct link to Step 1: Configure the BigQuery I/O manager") To use the BigQuery I/O manager, you'll need to add it to your `Definitions` object. The BigQuery I/O manager requires some configuration to connect to your Bigquery instance: * A `project` * One method of authentication. You can follow the GCP authentication instructions [here](https://cloud.google.com/docs/authentication/provide-credentials-adc) , or see [Providing credentials as configuration](https://docs.dagster.io/integrations/libraries/gcp/bigquery/reference#providing-credentials-as-configuration) in the BigQuery reference guide. You can also specify a `location` where data should be stored and processed and `dataset` that should hold the created tables. You can also set a `timeout` when working with Pandas DataFrames. from dagster_gcp_pandas import BigQueryPandasIOManagerfrom dagster import Definitionsdefs = Definitions( assets=[iris_data], resources={ "io_manager": BigQueryPandasIOManager( project="my-gcp-project", # required location="us-east5", # optional, defaults to the default location for the project - see https://cloud.google.com/bigquery/docs/locations for a list of locations dataset="IRIS", # optional, defaults to PUBLIC timeout=15.0, # optional, defaults to None ) },) With this configuration, if you materialized an asset called `iris_data`, the BigQuery I/O manager would store the data in the `IRIS.IRIS_DATA` table in the `my-gcp-project` project. The BigQuery instance would be located in `us-east5`. Finally, in the [`Definitions`](https://docs.dagster.io/api/dagster/definitions#dagster.Definitions) object, we assign the [`BigQueryPandasIOManager`](https://docs.dagster.io/api/libraries/dagster-gcp-pandas#dagster_gcp_pandas.BigQueryPandasIOManager) to the `io_manager` key. `io_manager` is a reserved key to set the default I/O manager for your assets. For more info about each of the configuration values, refer to the [`BigQueryPandasIOManager`](https://docs.dagster.io/api/libraries/dagster-gcp-pandas#dagster_gcp_pandas.BigQueryPandasIOManager) API documentation. ### Step 2: Create tables in BigQuery[​](https://docs.dagster.io/integrations/libraries/gcp/bigquery/using-bigquery-with-dagster#option-2-step-2 "Direct link to Step 2: Create tables in BigQuery") The BigQuery I/O manager can create and update tables for your Dagster defined assets, but you can also make existing BigQuery tables available to Dagster. * Create tables in BigQuery from Dagster assets * Making Dagster aware of existing tables **Store a Dagster asset as a table in BigQuery** To store data in BigQuery using the BigQuery I/O manager, you can simply return a Pandas DataFrame from your asset. Dagster will handle storing and loading your assets in BigQuery. import pandas as pdfrom dagster import asset@assetdef iris_data() -> pd.DataFrame: return pd.read_csv( "https://docs.dagster.io/assets/iris.csv", names=[ "sepal_length_cm", "sepal_width_cm", "petal_length_cm", "petal_width_cm", "species", ], ) In this example, you're defining an [asset](https://docs.dagster.io/guides/build/assets/defining-assets) that fetches the Iris dataset as a Pandas DataFrame, renames the columns, then returns the DataFrame. The type signature of the function tells the I/O manager what data type it is working with, so it is important to include the return type `pd.DataFrame`. When Dagster materializes the `iris_data` asset using the configuration from [Step 1: Configure the BigQuery I/O manager](https://docs.dagster.io/integrations/libraries/gcp/bigquery/using-bigquery-with-dagster#step-1-configure-the-bigquery-io-manager) , the BigQuery I/O manager will create the table `IRIS.IRIS_DATA` if it does not exist and replace the contents of the table with the value returned from the `iris_data` asset. **Making Dagster aware of existing tables** If you already have existing tables in BigQuery and other assets defined in Dagster depend on those tables, you may want Dagster to be aware of those upstream dependencies. Making Dagster aware of these tables will allow you to track the full data lineage in Dagster. You can define [external assets](https://docs.dagster.io/guides/build/assets/external-assets) for these tables. When using an I/O manager, defining an external asset for an existing table also allows you to tell Dagster how to find the table so it can be fetched for downstream assets. from dagster import AssetSpeciris_harvest_data = AssetSpec(key="iris_harvest_data") In this example, you're creating a [`AssetSpec`](https://docs.dagster.io/api/dagster/assets#dagster.AssetSpec) for a pre-existing table - perhaps created by an external data ingestion tool - that contains data about iris harvests. To make the data available to other Dagster assets, you need to tell the BigQuery I/O manager how to find the data, so that the I/O manager can load the data into memory. Because you already supplied the project and dataset in the I/O manager configuration in [Step 1: Configure the BigQuery I/O manager](https://docs.dagster.io/integrations/libraries/gcp/bigquery/using-bigquery-with-dagster#step-1-configure-the-bigquery-io-manager) , you only need to provide the table name. This is done with the `key` parameter in `AssetSpec`. When the I/O manager needs to load the `iris_harvest_data` in a downstream asset, it will select the data in the `IRIS.IRIS_HARVEST_DATA` table as a Pandas DataFrame and provide it to the downstream asset. ### Step 3: Load BigQuery tables in downstream assets[​](https://docs.dagster.io/integrations/libraries/gcp/bigquery/using-bigquery-with-dagster#step-3-load-bigquery-tables-in-downstream-assets "Direct link to Step 3: Load BigQuery tables in downstream assets") Once you have created an asset that represents a table in BigQuery, you will likely want to create additional assets that work with the data. Dagster and the BigQuery I/O manager allow you to load the data stored in BigQuery tables into downstream assets. import pandas as pdfrom dagster import asset# this example uses the iris_data asset from Step 2@assetdef iris_setosa(iris_data: pd.DataFrame) -> pd.DataFrame: return iris_data[iris_data["species"] == "Iris-setosa"] In this asset, you're providing the `iris_data` asset from the [Store a Dagster asset as a table in BigQuery](https://docs.dagster.io/integrations/libraries/gcp/bigquery/using-bigquery-with-dagster#option-2-step-2) example to the `iris_setosa` asset. In this asset, you're providing the `iris_data` asset as a dependency to `iris_setosa`. By supplying `iris_data` as a parameter to `iris_setosa`, Dagster knows to use the `BigQueryPandasIOManager` to load this asset into memory as a Pandas DataFrame and pass it as an argument to `iris_setosa`. Next, a DataFrame that only contains the data for the _Iris Setosa_ species is created and returned. Then the `BigQueryPandasIOManager` will store the DataFrame as the `IRIS.IRIS_SETOSA` table in BigQuery. ### Completed code example[​](https://docs.dagster.io/integrations/libraries/gcp/bigquery/using-bigquery-with-dagster#completed-code-example-1 "Direct link to Completed code example") When finished, your code should look like the following: import pandas as pdfrom dagster_gcp_pandas import BigQueryPandasIOManagerfrom dagster import AssetSpec, Definitions, assetiris_harvest_data = AssetSpec(key="iris_harvest_data")@assetdef iris_data() -> pd.DataFrame: return pd.read_csv( "https://docs.dagster.io/assets/iris.csv", names=[ "sepal_length_cm", "sepal_width_cm", "petal_length_cm", "petal_width_cm", "species", ], )@assetdef iris_setosa(iris_data: pd.DataFrame) -> pd.DataFrame: return iris_data[iris_data["species"] == "Iris-setosa"]defs = Definitions( assets=[iris_data, iris_harvest_data, iris_setosa], resources={ "io_manager": BigQueryPandasIOManager( project="my-gcp-project", location="us-east5", dataset="IRIS", timeout=15.0, ) },) Related[​](https://docs.dagster.io/integrations/libraries/gcp/bigquery/using-bigquery-with-dagster#related "Direct link to Related") ------------------------------------------------------------------------------------------------------------------------------------- For more BigQuery features, refer to the [BigQuery reference](https://docs.dagster.io/integrations/libraries/gcp/bigquery/reference) . For more information on asset definitions, see the [Assets documentation](https://docs.dagster.io/guides/build/assets) . For more information on I/O managers, see the [I/O manager documentation](https://docs.dagster.io/guides/build/io-managers) . * [Prerequisites](https://docs.dagster.io/integrations/libraries/gcp/bigquery/using-bigquery-with-dagster#prerequisites) * [Option 1: Using the BigQuery resource](https://docs.dagster.io/integrations/libraries/gcp/bigquery/using-bigquery-with-dagster#option-1-using-the-bigquery-resource) * [Step 1: Configure the BigQuery resource](https://docs.dagster.io/integrations/libraries/gcp/bigquery/using-bigquery-with-dagster#step-1-configure-the-bigquery-resource) * [Step 2: Create tables in BigQuery](https://docs.dagster.io/integrations/libraries/gcp/bigquery/using-bigquery-with-dagster#step-2-create-tables-in-bigquery) * [Step 3: Define downstream assets](https://docs.dagster.io/integrations/libraries/gcp/bigquery/using-bigquery-with-dagster#step-3-define-downstream-assets) * [Completed code example](https://docs.dagster.io/integrations/libraries/gcp/bigquery/using-bigquery-with-dagster#completed-code-example) * [Option 2: Using the BigQuery I/O manager](https://docs.dagster.io/integrations/libraries/gcp/bigquery/using-bigquery-with-dagster#option-2-using-the-bigquery-io-manager) * [Step 1: Configure the BigQuery I/O manager](https://docs.dagster.io/integrations/libraries/gcp/bigquery/using-bigquery-with-dagster#step-1-configure-the-bigquery-io-manager) * [Step 2: Create tables in BigQuery](https://docs.dagster.io/integrations/libraries/gcp/bigquery/using-bigquery-with-dagster#option-2-step-2) * [Step 3: Load BigQuery tables in downstream assets](https://docs.dagster.io/integrations/libraries/gcp/bigquery/using-bigquery-with-dagster#step-3-load-bigquery-tables-in-downstream-assets) * [Completed code example](https://docs.dagster.io/integrations/libraries/gcp/bigquery/using-bigquery-with-dagster#completed-code-example-1) * [Related](https://docs.dagster.io/integrations/libraries/gcp/bigquery/using-bigquery-with-dagster#related) --- # Dagster & OpenAI | Dagster Docs [Skip to main content](https://docs.dagster.io/integrations/libraries/openai#__docusaurus_skipToContent_fallback) On this page The OpenAI library allows you to easily interact with the OpenAI REST API using the OpenAI Python API to build AI steps into your Dagster pipelines. You can also log OpenAI API usage metadata in Dagster Insights, giving you detailed observability on API call credit consumption. Using this library's [`OpenAIResource`](https://docs.dagster.io/api/libraries/dagster-openai#dagster_openai.OpenAIResource) , you can easily interact with the [OpenAI REST API](https://platform.openai.com/docs/introduction) via the [OpenAI Python API](https://github.com/openai/openai-python) . When used with Dagster's [asset definitions](https://docs.dagster.io/guides/build/assets/defining-assets) , the resource automatically logs OpenAI usage metadata in [asset metadata](https://docs.dagster.io/guides/build/assets/metadata-and-tags) . Getting started[​](https://docs.dagster.io/integrations/libraries/openai#getting-started "Direct link to Getting started") --------------------------------------------------------------------------------------------------------------------------- Before you get started with the `dagster-openai` library, we recommend familiarizing yourself with the [OpenAI Python API library](https://github.com/openai/openai-python) , which this integration uses to interact with the [OpenAI REST API](https://platform.openai.com/docs/introduction) . Prerequisites[​](https://docs.dagster.io/integrations/libraries/openai#prerequisites "Direct link to Prerequisites") --------------------------------------------------------------------------------------------------------------------- To get started, install the `dagster` and `dagster-openai` Python packages: * uv * pip uv add dagster-openai pip install dagster-openai Note that you will need an OpenAI [API key](https://platform.openai.com/api-keys) to use the resource, which can be generated in your OpenAI account. Connecting to OpenAI[​](https://docs.dagster.io/integrations/libraries/openai#connecting-to-openai "Direct link to Connecting to OpenAI") ------------------------------------------------------------------------------------------------------------------------------------------ The first step in using OpenAI with Dagster is to tell Dagster how to connect to an OpenAI client using an OpenAI [resource](https://docs.dagster.io/guides/build/external-resources) . This resource contains the credentials needed to interact with OpenAI API. We will supply our credentials as environment variables by adding them to a `.env` file. For more information on setting environment variables in a production setting, see [Using environment variables and secrets](https://docs.dagster.io/guides/operate/configuration/using-environment-variables-and-secrets) . # .envOPENAI_API_KEY=... Then, we can instruct Dagster to authorize the OpenAI resource using the environment variables: from dagster_openai import OpenAIResourcefrom dagster import EnvVar# Pull API key from environment variablesopenai = OpenAIResource( api_key=EnvVar("OPENAI_API_KEY"),) Using the OpenAI resource with assets[​](https://docs.dagster.io/integrations/libraries/openai#using-the-openai-resource-with-assets "Direct link to Using the OpenAI resource with assets") --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- The OpenAI resource can be used in assets in order to interact with the OpenAI API. Note that in this example, we supply our credentials as environment variables directly when instantiating the [`Definitions`](https://docs.dagster.io/api/dagster/definitions#dagster.Definitions) object. from dagster_openai import OpenAIResourcefrom dagster import AssetExecutionContext, Definitions, EnvVar, asset, define_asset_job@asset(compute_kind="OpenAI")def openai_asset(context: AssetExecutionContext, openai: OpenAIResource): with openai.get_client(context) as client: client.chat.completions.create( model="gpt-3.5-turbo", messages=[{"role": "user", "content": "Say this is a test."}], )openai_asset_job = define_asset_job(name="openai_asset_job", selection="openai_asset")defs = Definitions( assets=[openai_asset], jobs=[openai_asset_job], resources={ "openai": OpenAIResource(api_key=EnvVar("OPENAI_API_KEY")), },) After materializing your asset, your OpenAI API usage metadata will be available in the **Events** and **Plots** tabs of your asset in the Dagster UI. If you are using [Dagster+](https://docs.dagster.io/deployment/dagster-plus) , your usage metadata will also be available in [Dagster Insights](https://docs.dagster.io/guides/monitor/insights) . Using the OpenAI resource with ops[​](https://docs.dagster.io/integrations/libraries/openai#using-the-openai-resource-with-ops "Direct link to Using the OpenAI resource with ops") ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ The OpenAI resource can also be used in [ops](https://docs.dagster.io/guides/build/ops) . note Currently, the OpenAI resource doesn't (out-of-the-box) log OpenAI usage metadata when used in ops. from dagster_openai import OpenAIResourcefrom dagster import Definitions, EnvVar, GraphDefinition, OpExecutionContext, op@opdef openai_op(context: OpExecutionContext, openai: OpenAIResource): with openai.get_client(context) as client: client.chat.completions.create( model="gpt-3.5-turbo", messages=[{"role": "user", "content": "Say this is a test"}], )openai_op_job = GraphDefinition(name="openai_op_job", node_defs=[openai_op]).to_job()defs = Definitions( jobs=[openai_op_job], resources={ "openai": OpenAIResource(api_key=EnvVar("OPENAI_API_KEY")), },) About OpenAI[​](https://docs.dagster.io/integrations/libraries/openai#about-openai "Direct link to About OpenAI") ------------------------------------------------------------------------------------------------------------------ OpenAI is a U.S. based artificial intelligence (AI) research organization with the goal of developing "safe and beneficial" artificial general intelligence, which it defines as "highly autonomous systems that outperform humans at most economically valuable work". * [Getting started](https://docs.dagster.io/integrations/libraries/openai#getting-started) * [Prerequisites](https://docs.dagster.io/integrations/libraries/openai#prerequisites) * [Connecting to OpenAI](https://docs.dagster.io/integrations/libraries/openai#connecting-to-openai) * [Using the OpenAI resource with assets](https://docs.dagster.io/integrations/libraries/openai#using-the-openai-resource-with-assets) * [Using the OpenAI resource with ops](https://docs.dagster.io/integrations/libraries/openai#using-the-openai-resource-with-ops) * [About OpenAI](https://docs.dagster.io/integrations/libraries/openai#about-openai) --- # Dagster & Sling | Dagster Docs [Skip to main content](https://docs.dagster.io/integrations/libraries/sling#__docusaurus_skipToContent_fallback) On this page note If you are just getting started with the Sling integration, we recommend using the new [Sling component](https://docs.dagster.io/guides/build/components/integrations/sling-component-tutorial) . Sling provides an easy-to-use YAML configuration layer for loading data from files, replicating data between databases, exporting custom SQL queries to cloud storage, and much more. How it works[​](https://docs.dagster.io/integrations/libraries/sling#how-it-works "Direct link to How it works") ----------------------------------------------------------------------------------------------------------------- The Dagster integration allows you to derive Dagster assets from a replication configuration file. The typical pattern for building an ELT pipeline with Sling has three steps: 1. Define a Sling [`replication.yaml`](https://docs.slingdata.io/sling-cli/run/configuration/replication) file that specifies the source and target connections, as well as which streams to sync from. 2. Create a [`SlingResource`](https://docs.dagster.io/api/libraries/dagster-sling#dagster_sling.SlingResource) and pass a list of [`SlingConnectionResource`](https://docs.dagster.io/api/libraries/dagster-sling#dagster_sling.SlingConnectionResource) for each connection to the `connection` parameter, ensuring the resource uses the same name given to the connection in the Sling configuration. 3. Use the [`@dagster_sling.sling_assets`](https://docs.dagster.io/api/libraries/dagster-sling#dagster_sling.sling_assets) decorator to define an asset that runs the Sling replication job and yields from the [`SlingResource`](https://docs.dagster.io/api/libraries/dagster-sling#dagster_sling.SlingResource) method to run the sync. We'll walk you through each of these steps in this guide. Prerequisites[​](https://docs.dagster.io/integrations/libraries/sling#prerequisites "Direct link to Prerequisites") -------------------------------------------------------------------------------------------------------------------- To follow the steps in this guide: * **Familiarize yourself with [Sling's replication configuration](https://docs.slingdata.io/sling-cli/run/configuration/replication) **, if you've never worked with Sling before. The replication configuration is a YAML file that specifies the source and target connections, as well as which streams to sync from. The `dagster-sling` integration uses this configuration to build assets for both sources and destinations. * **To install the following libraries**: * uv * pip uv add dagster-sling pip install dagster-sling Refer to the [Dagster installation](https://docs.dagster.io/getting-started/installation) guide for more info. Step 1: Set up a Sling replication configuration[​](https://docs.dagster.io/integrations/libraries/sling#step-1-set-up-a-sling-replication-configuration "Direct link to Step 1: Set up a Sling replication configuration") ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Dagster's Sling integration is built around Sling's replication configuration. You may provide either a path to an existing `replication.yaml` file or construct a dictionary that represents the configuration in Python. This configuration is passed to the Sling CLI to run the replication job. * replication.yaml * Python ### replication.yaml[​](https://docs.dagster.io/integrations/libraries/sling#replicationyaml "Direct link to replication.yaml") This example creates a replication configuration in a `replication.yaml` file: # replication.yamlsource: MY_POSTGREStarget: MY_SNOWFLAKEdefaults: mode: full-refresh object: '{stream_schema}_{stream_table}'streams: public.accounts: public.users: public.finance_departments: object: 'departments' ### Python[​](https://docs.dagster.io/integrations/libraries/sling#python "Direct link to Python") This example creates a replication configuration using Python: replication_config = { "source": "MY_POSTGRES", "target": "MY_DUCKDB", "defaults": {"mode": "full-refresh", "object": "{stream_schema}_{stream_table}"}, "streams": { "public.accounts": None, "public.users": None, "public.finance_departments": {"object": "departments"}, },} Step 2: Create a Sling resource[​](https://docs.dagster.io/integrations/libraries/sling#step-2-create-a-sling-resource "Direct link to Step 2: Create a Sling resource") ------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Next, you'll create a [`SlingResource`](https://docs.dagster.io/api/libraries/dagster-sling#dagster_sling.SlingResource) object that contains references to the connections specified in the replication configuration: # pyright: reportCallIssue=nonefrom dagster_sling import SlingConnectionResource, SlingResourcefrom dagster import EnvVarsling_resource = SlingResource( connections=[ # Using a connection string from an environment variable SlingConnectionResource( name="MY_POSTGRES", type="postgres", connection_string=EnvVar("POSTGRES_CONNECTION_STRING"), ), # Using a hard-coded connection string SlingConnectionResource( name="MY_DUCKDB", type="duckdb", connection_string="duckdb:///var/tmp/duckdb.db", ), # Using a keyword-argument constructor SlingConnectionResource( name="MY_SNOWFLAKE", type="snowflake", host=EnvVar("SNOWFLAKE_HOST"), user=EnvVar("SNOWFLAKE_USER"), role="REPORTING", ), ]) A [`SlingResource`](https://docs.dagster.io/api/libraries/dagster-sling#dagster_sling.SlingResource) takes a `connections` parameter, where each [`SlingConnectionResource`](https://docs.dagster.io/api/libraries/dagster-sling#dagster_sling.SlingConnectionResource) represents a connection to a source or target database. You may provide as many connections to the `SlingResource` as needed. The `name` parameter in the [`SlingConnectionResource`](https://docs.dagster.io/api/libraries/dagster-sling#dagster_sling.SlingConnectionResource) should match the `source` and `target` keys in the replication configuration. You can pass a connection string or arbitrary keyword arguments to the [`SlingConnectionResource`](https://docs.dagster.io/api/libraries/dagster-sling#dagster_sling.SlingConnectionResource) to specify the connection details. Refer to [Sling's connections reference](https://docs.slingdata.io/connections/database-connections) for the specific connection types and parameters. Step 3: Define the Sling assets[​](https://docs.dagster.io/integrations/libraries/sling#step-3-define-the-sling-assets "Direct link to Step 3: Define the Sling assets") ------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Next, define a Sling asset using the [`@dagster_sling.sling_assets`](https://docs.dagster.io/api/libraries/dagster-sling#dagster_sling.sling_assets) decorator. Dagster will read the replication configuration to produce assets. Each stream will render two assets, one for the source stream and one for the target destination. You can override how assets are named by passing in a custom [`DagsterSlingTranslator`](https://docs.dagster.io/api/libraries/dagster-sling#dagster_sling.DagsterSlingTranslator) object. from dagster_sling import SlingResource, sling_assetsfrom dagster import Definitions, file_relative_pathreplication_config = file_relative_path(__file__, "../sling_replication.yaml")sling_resource = SlingResource(connections=[...]) # Add connections here@sling_assets(replication_config=replication_config)def my_assets(context, sling: SlingResource): yield from sling.replicate(context=context) for row in sling.stream_raw_logs(): context.log.info(row) Step 4: Create the Definitions object[​](https://docs.dagster.io/integrations/libraries/sling#step-4-create-the-definitions-object "Direct link to Step 4: Create the Definitions object") ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- The last step is to include the Sling assets and resource in a [`Definitions`](https://docs.dagster.io/api/dagster/definitions#dagster.Definitions) object. This enables Dagster tools to load everything we've defined: defs = Definitions( assets=[ my_assets, ], resources={ "sling": sling_resource, },) That's it! You should now be able to view your assets in the [Dagster UI](https://docs.dagster.io/guides/operate/webserver) and run the replication job. Examples[​](https://docs.dagster.io/integrations/libraries/sling#examples "Direct link to Examples") ----------------------------------------------------------------------------------------------------- ### Example 1: Database to database[​](https://docs.dagster.io/integrations/libraries/sling#example-1-database-to-database "Direct link to Example 1: Database to database") To set up a Sling sync between two databases, such as Postgres and Snowflake, you could do something like the following: # pyright: reportCallIssue=none# pyright: reportOptionalMemberAccess=nonefrom dagster_sling import SlingConnectionResource, SlingResource, sling_assetsfrom dagster import EnvVarsource = SlingConnectionResource( name="MY_PG", type="postgres", host="localhost", port=5432, database="my_database", user="my_user", password=EnvVar("PG_PASS"),)target = SlingConnectionResource( name="MY_SF", type="snowflake", host="hostname.snowflake", user="username", database="database", password=EnvVar("SF_PASSWORD"), role="role",)sling = SlingResource( connections=[ source, target, ])replication_config = { "SOURCE": "MY_PG", "TARGET": "MY_SF", "defaults": {"mode": "full-refresh", "object": "{stream_schema}_{stream_table}"}, "streams": { "public.accounts": None, "public.users": None, "public.finance_departments": {"object": "departments"}, },}@sling_assets(replication_config=replication_config)def my_assets(context, sling: SlingResource): yield from sling.replicate(context=context) ### Example 2: File to database[​](https://docs.dagster.io/integrations/libraries/sling#example-2-file-to-database "Direct link to Example 2: File to database") To set up a Sling sync between a file in an object store and a database, such as from Amazon S3 to Snowflake, you could do something like the following: from dagster_sling import SlingConnectionResource, SlingResource, sling_assetsfrom dagster import EnvVartarget = SlingConnectionResource( name="MY_SF", type="snowflake", host="hostname.snowflake", user="username", database="database", password=EnvVar("SF_PASSWORD"), role="role",)source = SlingConnectionResource( name="MY_S3", type="s3", bucket="sling-bucket", access_key_id=EnvVar("AWS_ACCESS_KEY_ID"), secret_access_key=EnvVar("AWS_SECRET_ACCESS_KEY"),)sling = SlingResource(connections=[source, target])replication_config = { "SOURCE": "MY_S3", "TARGET": "MY_SF", "defaults": {"mode": "full-refresh", "object": "{stream_schema}_{stream_table}"}, "streams": { "s3://my-bucket/my_file.parquet": { "object": "marts.my_table", "primary_key": "id", }, },}@sling_assets(replication_config=replication_config)def my_assets(context, sling: SlingResource): yield from sling.replicate(context=context) Advanced usage[​](https://docs.dagster.io/integrations/libraries/sling#advanced-usage "Direct link to Advanced usage") ----------------------------------------------------------------------------------------------------------------------- ### Customize upstream dependencies[​](https://docs.dagster.io/integrations/libraries/sling#customize-upstream-dependencies "Direct link to Customize upstream dependencies") By default, Dagster sets upstream dependencies when generating asset specs for your Sling assets. To do so, Dagster parses information about assets that are upstream of specific Sling assets from the Sling replication configuration itself. You can customize how upstream dependencies are set on your Sling assets by passing an instance of the custom [`DagsterSlingTranslator`](https://docs.dagster.io/api/libraries/dagster-sling#dagster_sling.DagsterSlingTranslator) to the [`sling_assets`](https://docs.dagster.io/api/libraries/dagster-sling#dagster_sling.sling_assets) decorator. class CustomDagsterSlingTranslator(DagsterSlingTranslator): def get_asset_spec(self, stream_definition: Mapping[str, Any]) -> dg.AssetSpec: """Overrides asset spec to override upstream asset key to be a single source asset.""" # We create the default asset spec using super() default_spec = super().get_asset_spec(stream_definition) # We set an upstream dependency for our assets return default_spec.replace_attributes( deps=[dg.AssetKey("common_upstream_sling_dependency")], )@sling_assets( replication_config=replication_config, dagster_sling_translator=CustomDagsterSlingTranslator(),)def my_sling_assets(context, sling: SlingResource): yield from sling.replicate(context=context) Note that `super()` is called in each of the overridden methods to generate the default asset spec. It is best practice to generate the default asset spec before customizing it. ### Define downstream dependencies[​](https://docs.dagster.io/integrations/libraries/sling#define-downstream-dependencies "Direct link to Define downstream dependencies") Dagster allows you to define assets that are downstream of specific Sling streams using their asset keys. The asset key for a Sling stream can be retrieved using the [`DagsterSlingTranslator`](https://docs.dagster.io/api/libraries/dagster-sling#dagster_sling.DagsterSlingTranslator) . The below example defines `my_downstream_asset` as a downstream dependency of `my_sling_stream`: from dagster_sling.asset_decorator import get_streams_from_replication@sling_assets( replication_config=replication_config,)def my_sling_assets(context, sling: SlingResource): yield from sling.replicate(context=context)my_sling_stream_asset_key = next( iter( [ DagsterSlingTranslator().get_asset_spec(stream_definition=stream) for stream in get_streams_from_replication(replication_config) if stream["name"] == "my_sling_stream" ] ))@dg.asset(deps=[my_sling_stream_asset_key])def my_downstream_asset(): ... In the downstream asset, you may want direct access to the contents of the Sling asset. To do so, you can customize the code within your `@asset`\-decorated function to load upstream data. APIs in this guide[​](https://docs.dagster.io/integrations/libraries/sling#apis-in-this-guide "Direct link to APIs in this guide") ----------------------------------------------------------------------------------------------------------------------------------- | Name | Description | | --- | --- | | [`@dagster_sling.sling_assets`](https://docs.dagster.io/api/libraries/dagster-sling#dagster_sling.sling_assets) | The core Sling asset factory for building syncs | | [`SlingResource`](https://docs.dagster.io/api/libraries/dagster-sling#dagster_sling.SlingResource) | The Sling resource used for handing credentials to databases and object stores | | [`DagsterSlingTranslator`](https://docs.dagster.io/api/libraries/dagster-sling#dagster_sling.DagsterSlingTranslator) | A translator for specifying how to map between Sling and Dagster types | | [`SlingConnectionResource`](https://docs.dagster.io/api/libraries/dagster-sling#dagster_sling.SlingConnectionResource) | A Sling connection resource for specifying database and storage connection credentials | ### About Sling[​](https://docs.dagster.io/integrations/libraries/sling#about-sling "Direct link to About Sling") Sling provides an easy-to-use YAML configuration layer for loading data from files, replicating data between databases, exporting custom SQL queries to cloud storage, and much more. #### Key Features[​](https://docs.dagster.io/integrations/libraries/sling#key-features "Direct link to Key Features") * **Data Movement**: Transfer data between different storage systems and databases efficiently * **Flexible Connectivity**: Support for numerous databases, data warehouses, and file storage systems * **Transformation Capabilities**: Built-in data transformation features during transfer * **Multiple Operation Modes**: Support for various replication modes including full-refresh, incremental, and snapshot * **Production-Ready**: Deployable with monitoring, scheduling, and error handling * [How it works](https://docs.dagster.io/integrations/libraries/sling#how-it-works) * [Prerequisites](https://docs.dagster.io/integrations/libraries/sling#prerequisites) * [Step 1: Set up a Sling replication configuration](https://docs.dagster.io/integrations/libraries/sling#step-1-set-up-a-sling-replication-configuration) * [replication.yaml](https://docs.dagster.io/integrations/libraries/sling#replicationyaml) * [Python](https://docs.dagster.io/integrations/libraries/sling#python) * [Step 2: Create a Sling resource](https://docs.dagster.io/integrations/libraries/sling#step-2-create-a-sling-resource) * [Step 3: Define the Sling assets](https://docs.dagster.io/integrations/libraries/sling#step-3-define-the-sling-assets) * [Step 4: Create the Definitions object](https://docs.dagster.io/integrations/libraries/sling#step-4-create-the-definitions-object) * [Examples](https://docs.dagster.io/integrations/libraries/sling#examples) * [Example 1: Database to database](https://docs.dagster.io/integrations/libraries/sling#example-1-database-to-database) * [Example 2: File to database](https://docs.dagster.io/integrations/libraries/sling#example-2-file-to-database) * [Advanced usage](https://docs.dagster.io/integrations/libraries/sling#advanced-usage) * [Customize upstream dependencies](https://docs.dagster.io/integrations/libraries/sling#customize-upstream-dependencies) * [Define downstream dependencies](https://docs.dagster.io/integrations/libraries/sling#define-downstream-dependencies) * [APIs in this guide](https://docs.dagster.io/integrations/libraries/sling#apis-in-this-guide) * [About Sling](https://docs.dagster.io/integrations/libraries/sling#about-sling) * [Key Features](https://docs.dagster.io/integrations/libraries/sling#key-features) --- # Dagster & Teradata | Dagster Docs [Skip to main content](https://docs.dagster.io/integrations/libraries/teradata#__docusaurus_skipToContent_fallback) On this page The community-supported Teradata package provides an integration with Teradata Vantage. For more information, see the [dagster-teradata GitHub repository](https://github.com/dagster-io/community-integrations/tree/main/libraries/dagster-teradata) . To begin integrating Dagster with Teradata Vantage for building and managing ETL pipelines, this guide provides step-by-step instructions on installing and configuring the required packages, setting up a Dagster project, and implementing a pipeline that interacts with Teradata Vantage. Prerequisites[​](https://docs.dagster.io/integrations/libraries/teradata#prerequisites "Direct link to Prerequisites") ----------------------------------------------------------------------------------------------------------------------- * Access to a Teradata Vantage instance. note If you need a test instance of Vantage, you can provision one for free at [https://clearscape.teradata.com](https://clearscape.teradata.com/sign-in?utm_source=dev_portal&utm_medium=quickstart_tutorial&utm_campaign=quickstarts) * Python **3.9** or higher, Python **3.13** is recommended. * pip Install dagster-teradata[​](https://docs.dagster.io/integrations/libraries/teradata#install-dagster-teradata "Direct link to Install dagster-teradata") -------------------------------------------------------------------------------------------------------------------------------------------------------- With your virtual environment active, the next step is to install dagster and the Teradata provider package (dagster-teradata) to interact with Teradata Vantage. 1. Install the Required Packages: * uv * pip uv add dagster-teradata pip install dagster-teradata 2. Note about Optional Dependencies: a) `dagster-teradata` relies on dagster-aws for ingesting data from an S3 bucket into Teradata Vantage. Since `dagster-aws` is an optional dependency, users can install it by running: * uv * pip uv add dagster-teradata[aws] pip install dagster-teradata[aws] b) `dagster-teradata` also relies on `dagster-azure` for ingesting data from an Azure Blob Storage container into Teradata Vantage. To install this dependency, run: * uv * pip uv add dagster-teradata[azure] pip install dagster-teradata[azure] 3. Verify the Installation: To confirm that Dagster is correctly installed, run: dagster –version If installed correctly, it should show the version of Dagster. Initialize a Dagster Project[​](https://docs.dagster.io/integrations/libraries/teradata#initialize-a-dagster-project "Direct link to Initialize a Dagster Project") -------------------------------------------------------------------------------------------------------------------------------------------------------------------- Now that you have the necessary packages installed, the next step is to create a new Dagster project. ### Scaffold a New Dagster Project[​](https://docs.dagster.io/integrations/libraries/teradata#scaffold-a-new-dagster-project "Direct link to Scaffold a New Dagster Project") Run the following command: dagster project scaffold --name dagster-quickstart This command will create a new project named dagster-quickstart. It will automatically generate the following directory structure: dagster-quickstart│ pyproject.toml│ README.md│ setup.cfg│ setup.py│├───dagster_quickstart│ assets.py│ definitions.py│ __init__.py│└───dagster_quickstart_tests test_assets.py __init__.py Refer [here](https://docs.dagster.io/guides/build/projects/dagster-project-file-reference) to know more above this directory structure Create Sample Data[​](https://docs.dagster.io/integrations/libraries/teradata#create-sample-data "Direct link to Create Sample Data") -------------------------------------------------------------------------------------------------------------------------------------- To simulate an ETL pipeline, create a CSV file with sample data that your pipeline will process. **Create the CSV File:** Inside the dagster\_quickstart/data/ directory, create a file named sample\_data.csv with the following content: id,name,age,city1,Alice,28,New York2,Bob,35,San Francisco3,Charlie,42,Chicago4,Diana,31,Los Angeles This file represents sample data that will be used as input for your ETL pipeline. Define Assets for the ETL Pipeline[​](https://docs.dagster.io/integrations/libraries/teradata#define-assets-for-the-etl-pipeline "Direct link to Define Assets for the ETL Pipeline") -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Now, we'll define a series of assets for the ETL pipeline inside the assets.py file. Edit the assets.py File: Open the dagster\_quickstart/assets.py file and add the following code to define the pipeline: import pandas as pdfrom dagster import asset@asset(required_resource_keys={"teradata"})def read_csv_file(context): df = pd.read_csv("dagster_quickstart/data/sample_data.csv") context.log.info(df) return df@asset(required_resource_keys={"teradata"})def drop_table(context): result = context.resources.teradata.drop_table(["tmp_table"]) context.log.info(result)@asset(required_resource_keys={"teradata"})def create_table(context, drop_table): result = context.resources.teradata.execute_query('''CREATE TABLE tmp_table ( id INTEGER, name VARCHAR(50), age INTEGER, city VARCHAR(50));''') context.log.info(result)@asset(required_resource_keys={"teradata"}, deps=[read_csv_file])def insert_rows(context, create_table, read_csv_file): data_tuples = [tuple(row) for row in read_csv_file.to_numpy()] for row in data_tuples: result = context.resources.teradata.execute_query( f"INSERT INTO tmp_table (id, name, age, city) VALUES ({row[0]}, '{row[1]}', {row[2]}, '{row[3]}');" ) context.log.info(result)@asset(required_resource_keys={"teradata"})def read_table(context, insert_rows): result = context.resources.teradata.execute_query("select * from tmp_table;", True) context.log.info(result) This Dagster pipeline defines a series of assets that interact with Teradata. It starts by reading data from a CSV file, then drops and recreates a table in Teradata. After that, it inserts rows from the CSV into the table and finally retrieves the data from the table. Define the Pipeline Definitions[​](https://docs.dagster.io/integrations/libraries/teradata#define-the-pipeline-definitions "Direct link to Define the Pipeline Definitions") ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------- The next step is to configure the pipeline by defining the necessary resources and jobs. **Edit the definitions.py File**: Open dagster\_quickstart/definitions.py and define your Dagster pipeline as follows: from dagster import EnvVar, Definitionsfrom dagster_teradata import TeradataResourcefrom .assets import read_csv_file, read_table, create_table, drop_table, insert_rows# Define the pipeline and resourcesdefs = Definitions( assets=[read_csv_file, read_table, create_table, drop_table, insert_rows], resources={ "teradata": TeradataResource( host=EnvVar("TERADATA_HOST"), user=EnvVar("TERADATA_USER"), password=EnvVar("TERADATA_PASSWORD"), database=EnvVar("TERADATA_DATABASE"), ) }) This code sets up a Dagster project that interacts with Teradata by defining assets and resources 1. It imports necessary modules, including pandas, Dagster, and dagster-teradata. 2. It imports asset functions (read\_csv\_file, read\_table, create\_table, drop\_table, insert\_rows) from the assets.py module. 3. It registers these assets with Dagster using Definitions, allowing Dagster to track and execute them. 4. It defines a Teradata resource (TeradataResource) that reads database connection details from environment variables (TERADATA\_HOST, TERADATA\_USER, TERADATA\_PASSWORD, TERADATA\_DATABASE). Running the Pipeline[​](https://docs.dagster.io/integrations/libraries/teradata#running-the-pipeline "Direct link to Running the Pipeline") -------------------------------------------------------------------------------------------------------------------------------------------- After setting up the project, you can now run your Dagster pipeline: 1. **Start the Dagster Dev Server:** In your terminal, navigate to the root directory of your project and run: dagster dev After executing the command dagster dev, the Dagster logs will be displayed directly in the terminal. Any errors encountered during startup will also be logged here. Once you see a message similar to: 2025-02-04 09:15:46 +0530 - dagster-webserver - INFO - Serving dagster-webserver on http://127.0.0.1:3000 in process 32564, It indicates that the Dagster webserver is running successfully. At this point, you can proceed to the next step. 2. **Access the Dagster UI:** Open a web browser and navigate to [http://127.0.0.1:3000](http://127.0.0.1:3000/) . This will open the Dagster UI where you can manage and monitor your pipelines. 3. **Run the Pipeline:** * In the top navigation of the Dagster UI, click Assets > View global asset lineage. * Click Materialize to execute the pipeline. * In the popup window, click View to see the details of the pipeline run. 4. **Monitor the Run:** The Dagster UI allows you to visualize the pipeline's progress, view logs, and inspect the status of each step. You can switch between different views to see the execution logs and metadata for each asset. Below are some of the operations provided by the TeradataResource:[​](https://docs.dagster.io/integrations/libraries/teradata#below-are-some-of-the-operations-provided-by-the-teradataresource "Direct link to Below are some of the operations provided by the TeradataResource:") ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- ### 1\. Execute a Query (`execute_query`)[​](https://docs.dagster.io/integrations/libraries/teradata#1-execute-a-query-execute_query "Direct link to 1-execute-a-query-execute_query") This operation executes a SQL query within Teradata Vantage. **Args:** * `sql` (str) – The query to be executed. * `fetch_results` (bool, optional) – If True, fetch the query results. Defaults to False. * `single_result_row` (bool, optional) – If True, return only the first row of the result set. Effective only if `fetch_results` is True. Defaults to False. ### 2\. Execute Multiple Queries (`execute_queries`)[​](https://docs.dagster.io/integrations/libraries/teradata#2-execute-multiple-queries-execute_queries "Direct link to 2-execute-multiple-queries-execute_queries") This operation executes a series of SQL queries within Teradata Vantage. **Args:** * `sql_queries` (Sequence\[str\]) – List of queries to be executed in series. * `fetch_results` (bool, optional) – If True, fetch the query results. Defaults to False. * `single_result_row` (bool, optional) – If True, return only the first row of the result set. Effective only if `fetch_results` is True. Defaults to False. ### 3\. Drop a Database (`drop_database`)[​](https://docs.dagster.io/integrations/libraries/teradata#3-drop-a-database-drop_database "Direct link to 3-drop-a-database-drop_database") This operation drops one or more databases from Teradata Vantage. **Args:** * `databases` (Union\[str, Sequence\[str\]\]) – Database name or list of database names to drop. ### 4\. Drop a Table (`drop_table`)[​](https://docs.dagster.io/integrations/libraries/teradata#4-drop-a-table-drop_table "Direct link to 4-drop-a-table-drop_table") This operation drops one or more tables from Teradata Vantage. **Args:** * `tables` (Union\[str, Sequence\[str\]\]) – Table name or list of table names to drop. * * * Data Transfer from AWS S3 to Teradata Vantage Using dagster-teradata:[​](https://docs.dagster.io/integrations/libraries/teradata#data-transfer-from-aws-s3-to-teradata-vantage-using-dagster-teradata "Direct link to Data Transfer from AWS S3 to Teradata Vantage Using dagster-teradata:") ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- import osfrom dagster import job, op, Definitions, EnvVar, DagsterErrorfrom dagster_aws.s3 import S3Resource, s3_resourcefrom dagster_teradata import TeradataResource, teradata_resources3_resource = S3Resource( aws_access_key_id=os.getenv("AWS_ACCESS_KEY_ID"), aws_secret_access_key=os.getenv("AWS_SECRET_ACCESS_KEY"), aws_session_token=os.getenv("AWS_SESSION_TOKEN"),)td_resource = TeradataResource( host=os.getenv("TERADATA_HOST"), user=os.getenv("TERADATA_USER"), password=os.getenv("TERADATA_PASSWORD"), database=os.getenv("TERADATA_DATABASE"),)@op(required_resource_keys={"teradata"})def drop_existing_table(context): context.resources.teradata.drop_table("people") return "Tables Dropped"@op(required_resource_keys={"teradata", "s3"})def ingest_s3_to_teradata(context, status): if status == "Tables Dropped": context.resources.teradata.s3_to_teradata(s3_resource, os.getenv("AWS_S3_LOCATION"), "people") else: raise DagsterError("Tables not dropped")@job(resource_defs={"teradata": td_resource, "s3": s3_resource})def example_job(): ingest_s3_to_teradata(drop_existing_table())defs = Definitions( jobs=[example_job]) The `s3_to_teradata` method is used to load data from an S3 bucket into a Teradata table. It leverages Teradata Vantage Native Object Store (NOS), which allows direct querying and loading of external object store data (like AWS S3) into Teradata tables. ### Arguments Supported by `s3_blob_to_teradata`[​](https://docs.dagster.io/integrations/libraries/teradata#arguments-supported-by-s3_blob_to_teradata "Direct link to arguments-supported-by-s3_blob_to_teradata") * **s3 (S3Resource)**: The `S3Resource` object used to interact with the S3 bucket. * **s3\_source\_key (str)**: The URI specifying the location of the S3 bucket. The URI format is: `/s3/YOUR-BUCKET.s3.amazonaws.com/YOUR-BUCKET-NAME` For more details, refer to: [Teradata Documentation - Native Object Store](https://docs.teradata.com/search/documents?query=native+object+store&sort=last_update&virtual-field=title_only&content-lang=en-US) * **teradata\_table (str)**: The name of the Teradata table to which the data will be loaded. * **public\_bucket (bool)**: Indicates whether the provided S3 bucket is public. If `True`, the objects within the bucket can be accessed via a URL without authentication. If `False`, the bucket is considered private, and authentication must be provided. Defaults to `False`. * **teradata\_authorization\_name (str)**: The name of the Teradata Authorization Database Object, which controls access to the S3 object store. For more details, refer to: [Teradata Vantage Native Object Store - Setting Up Access](https://docs.teradata.com/r/Enterprise_IntelliFlex_VMware/Teradata-VantageTM-Native-Object-Store-Getting-Started-Guide-17.20/Setting-Up-Access/Controlling-Foreign-Table-Access-with-an-AUTHORIZATION-Object) * * * Data Transfer from Azure Blob to Teradata Vantage Using dagster-teradata:[​](https://docs.dagster.io/integrations/libraries/teradata#data-transfer-from-azure-blob-to-teradata-vantage-using-dagster-teradata "Direct link to Data Transfer from Azure Blob to Teradata Vantage Using dagster-teradata:") ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- import osfrom dagster import job, op, Definitions, EnvVar, DagsterErrorfrom dagster_azure.adls2 import ADLS2Resource, ADLS2SASTokenfrom dagster_teradata import TeradataResource, teradata_resourceazure_resource = ADLS2Resource( storage_account="", credential=ADLS2SASToken(token=""),)td_resource = TeradataResource( host=os.getenv("TERADATA_HOST"), user=os.getenv("TERADATA_USER"), password=os.getenv("TERADATA_PASSWORD"), database=os.getenv("TERADATA_DATABASE"),)@op(required_resource_keys={"teradata"})def drop_existing_table(context): context.resources.teradata.drop_table("people") return "Tables Dropped"@op(required_resource_keys={"teradata", "azure"})def ingest_azure_to_teradata(context, status): if status == "Tables Dropped": context.resources.teradata.azure_blob_to_teradata(azure_resource, "/az/akiaxox5jikeotfww4ul.blob.core.windows.net/td-usgs/CSVDATA/09380000/2018/06/", "people", True) else: raise DagsterError("Tables not dropped")@job(resource_defs={"teradata": td_resource, "azure": azure_resource})def example_job(): ingest_azure_to_teradata(drop_existing_table())defs = Definitions( jobs=[example_job]) The `azure_blob_to_teradata` method is used to load data from Azure Data Lake Storage (ADLS) into a Teradata table. This method leverages Teradata Vantage Native Object Store (NOS) to directly query and load external object store data (such as Azure Blob Storage) into Teradata. ### Arguments Supported by `azure_blob_to_teradata`[​](https://docs.dagster.io/integrations/libraries/teradata#arguments-supported-by-azure_blob_to_teradata "Direct link to arguments-supported-by-azure_blob_to_teradata") * **azure (ADLS2Resource)**: The `ADLS2Resource` object used to interact with the Azure Blob Storage. * **blob\_source\_key (str)**: The URI specifying the location of the Azure Blob object. The format is: `/az/YOUR-STORAGE-ACCOUNT.blob.core.windows.net/YOUR-CONTAINER/YOUR-BLOB-LOCATION` For more details, refer to the Teradata documentation: [Teradata Documentation - Native Object Store](https://docs.teradata.com/search/documents?query=native+object+store&sort=last_update&virtual-field=title_only&content-lang=en-US) * **teradata\_table (str)**: The name of the Teradata table where the data will be loaded. * **public\_bucket (bool, optional)**: Indicates whether the Azure Blob container is public. If `True`, the objects in the container can be accessed without authentication. Defaults to `False`. * **teradata\_authorization\_name (str, optional)**: The name of the Teradata Authorization Database Object used to control access to the Azure Blob object store. This is required for secure access to private containers. Defaults to an empty string. For more details, refer to the documentation: [Teradata Vantage Native Object Store - Setting Up Access](https://docs.teradata.com/r/Enterprise_IntelliFlex_VMware/Teradata-VantageTM-Native-Object-Store-Getting-Started-Guide-17.20/Setting-Up-Access/Controlling-Foreign-Table-Access-with-an-AUTHORIZATION-Object) ### Transfer data from Private Blob Storage Container to Teradata instance[​](https://docs.dagster.io/integrations/libraries/teradata#transfer-data-from-private-blob-storage-container-to-teradata-instance "Direct link to Transfer data from Private Blob Storage Container to Teradata instance") To successfully transfer data from a Private Blob Storage Container to a Teradata instance, the following prerequisites are necessary. * An Azure account. You can start with a [free account](https://azure.microsoft.com/free) . * Create an [Azure storage account](https://docs.microsoft.com/en-us/azure/storage/common/storage-quickstart-create-account?tabs=azure-portal) * Create a [blob container](https://learn.microsoft.com/en-us/azure/storage/blobs/blob-containers-portal) under Azure storage account * [Upload](https://learn.microsoft.com/en-us/azure/storage/blobs/storage-quickstart-blobs-portal) CSV/JSON/Parquest format files to blob container * Create a Teradata Authorization object with the Azure Blob Storage Account and the Account Secret Key CREATE AUTHORIZATION azure_authorization USER 'azuretestquickstart' PASSWORD 'AZURE_BLOB_ACCOUNT_SECRET_KEY' note Replace `AZURE_BLOB_ACCOUNT_SECRET_KEY` with Azure storage account `azuretestquickstart` [access key](https://learn.microsoft.com/en-us/azure/storage/common/storage-account-keys-manage?toc=%2Fazure%2Fstorage%2Fblobs%2Ftoc.json&bc=%2Fazure%2Fstorage%2Fblobs%2Fbreadcrumb%2Ftoc.json&tabs=azure-portal) * * * Manage VantageCloud Lake Compute Clusters with dagster-teradata:[​](https://docs.dagster.io/integrations/libraries/teradata#manage-vantagecloud-lake-compute-clusters-with-dagster-teradata "Direct link to Manage VantageCloud Lake Compute Clusters with dagster-teradata:") ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- from dagster import Definitions, DagsterError, op, materialize, jobfrom dagster_dbt import DbtCliResourcefrom dagster_teradata import teradata_resource, TeradataResourcefrom .assets import jaffle_shop_dbt_assetsfrom .project import jaffle_shop_projectfrom .schedules import schedules@op(required_resource_keys={"teradata"})def create_compute_cluster(context): context.resources.teradata.create_teradata_compute_cluster( "ShippingCG01", "Shipping", "STANDARD", "TD_COMPUTE_MEDIUM", "MIN_COMPUTE_COUNT(1) MAX_COMPUTE_COUNT(1) INITIALLY_SUSPENDED('FALSE')", ) return "Compute Cluster Created"@op(required_resource_keys={"teradata", "dbt"})def run_dbt(context, status): if status == "Compute Cluster Created": materialize( [jaffle_shop_dbt_assets], resources={ "dbt": DbtCliResource(project_dir=jaffle_shop_project) } ) return "DBT Run Completed" else: raise DagsterError("DBT Run Failed")@op(required_resource_keys={"teradata"})def drop_compute_cluster(context, status): if status == "DBT Run Completed": context.resources.teradata.drop_teradata_compute_cluster("ShippingCG01", "Shipping", True) else: raise DagsterError("DBT Run Failed")@job(resource_defs={"teradata": teradata_resource, "dbt": DbtCliResource})def example_job(): drop_compute_cluster(run_dbt(create_compute_cluster()))defs = Definitions( assets=[jaffle_shop_dbt_assets], jobs=[example_job], schedules=schedules, resources={ "dbt": DbtCliResource(project_dir=jaffle_shop_project), "teradata": TeradataResource(), },) Teradata VantageCloud Lake provides robust compute cluster management capabilities, enabling users to dynamically allocate, suspend, resume, and delete compute resources. These operations are fully supported through **`dagster-teradata`**, allowing users to manage compute clusters directly within their Dagster pipelines. This integration ensures optimal performance, scalability, and cost efficiency. The following operations facilitate seamless compute cluster management within Dagster: ### 1\. Create a Compute Cluster (`create_teradata_compute_cluster`)[​](https://docs.dagster.io/integrations/libraries/teradata#1-create-a-compute-cluster-create_teradata_compute_cluster "Direct link to 1-create-a-compute-cluster-create_teradata_compute_cluster") This operation provisions a new compute cluster within Teradata VantageCloud Lake using `dagster-teradata`. It enables users to define the cluster's configuration, including compute profiles, resource allocation, and query execution strategies, directly within a Dagster job. **Args:** * `compute_profile_name` (str) – Specifies the name of the compute profile. * `compute_group_name` (str) – Identifies the compute group to which the profile belongs. * `query_strategy` (str, optional, default="STANDARD") – Defines the method used by the Teradata Optimizer to execute SQL queries efficiently. Acceptable values: * `STANDARD` – The default strategy at the database level, optimized for general query execution. * `ANALYTIC` – Optimized for complex analytical workloads. * `compute_map` (Optional\[str\], default=None) – Maps compute resources to specific nodes within the cluster. * `compute_attribute` (Optional\[str\], default=None) – Specifies additional configuration attributes for the compute profile, such as: * `MIN_COMPUTE_COUNT(1) MAX_COMPUTE_COUNT(5) INITIALLY_SUSPENDED('FALSE')` * `timeout` (int, optional, default=constants.CC\_OPR\_TIME\_OUT) – The maximum duration (in seconds) to wait for the cluster creation process to complete. Default: 20 minutes. ### 2\. Suspend a Compute Cluster (`suspend_teradata_compute_cluster`)[​](https://docs.dagster.io/integrations/libraries/teradata#2-suspend-a-compute-cluster-suspend_teradata_compute_cluster "Direct link to 2-suspend-a-compute-cluster-suspend_teradata_compute_cluster") This operation temporarily suspends a compute cluster within Teradata VantageCloud Lake using **`dagster-teradata`**, reducing resource consumption while retaining the compute profile for future use. **Args:** * `compute_profile_name` (str) – Specifies the name of the compute profile. * `compute_group_name` (str) – Identifies the compute group associated with the profile. * `timeout` (int, optional, default=constants.CC\_OPR\_TIME\_OUT) – The maximum wait time for the suspension process to complete. Default: 20 minutes. ### 3\. Resume a Compute Cluster (`resume_teradata_compute_cluster`)[​](https://docs.dagster.io/integrations/libraries/teradata#3-resume-a-compute-cluster-resume_teradata_compute_cluster "Direct link to 3-resume-a-compute-cluster-resume_teradata_compute_cluster") This operation restores a previously suspended compute cluster using **`dagster-teradata`**, allowing workloads to resume execution within a Dagster pipeline. **Args:** * `compute_profile_name` (str) – Specifies the name of the compute profile. * `compute_group_name` (str) – Identifies the compute group associated with the profile. * `timeout` (int, optional, default=constants.CC\_OPR\_TIME\_OUT) – The maximum wait time for the resumption process to complete. Default: 20 minutes. ### 4\. Delete a Compute Cluster (`drop_teradata_compute_cluster`)[​](https://docs.dagster.io/integrations/libraries/teradata#4-delete-a-compute-cluster-drop_teradata_compute_cluster "Direct link to 4-delete-a-compute-cluster-drop_teradata_compute_cluster") This operation removes a compute cluster from Teradata VantageCloud Lake using **`dagster-teradata`**, with an option to delete the associated compute group. You can run this operation directly from your Dagster workflow. **Args:** * `compute_profile_name` (str) – Specifies the name of the compute profile. * `compute_group_name` (str) – Identifies the compute group associated with the profile. * `delete_compute_group` (bool, optional, default=False) – Determines whether the compute group should be deleted: * `True` – Deletes the compute group. * `False` – Retains the compute group without modifications. These operations are designed to be fully integrated into **`dagster-teradata`** for managing compute clusters in Teradata VantageCloud Lake. By utilizing these operations within Dagster jobs, users can optimize resource allocation, perform complex transformations, and automate compute cluster management to align with workload demands. * * * Further reading[​](https://docs.dagster.io/integrations/libraries/teradata#further-reading "Direct link to Further reading") ----------------------------------------------------------------------------------------------------------------------------- * [dagster-teradata with Teradata Vantage](https://developers.teradata.com/quickstarts/manage-data/use-dagster-with-teradata-vantage) * [Data Transfer from AWS S3 to Teradata Vantage Using dagster-teradata](https://developers.teradata.com/quickstarts/manage-data/dagster-teradata-s3-to-teradata-transfer) * [Data Transfer from Azure Blob to Teradata Vantage Using dagster-teradata](https://developers.teradata.com/quickstarts/manage-data/dagster-teradata-azure-to-teradata-transfer) * [Manage VantageCloud Lake Compute Clusters with dagster-teradata](https://developers.teradata.com/quickstarts/vantagecloud-lake/vantagecloud-lake-compute-cluster-dagster) * [Teradata Authorization](https://docs.teradata.com/r/Enterprise_IntelliFlex_VMware/SQL-Data-Definition-Language-Syntax-and-Examples/Authorization-Statements-for-External-Routines/CREATE-AUTHORIZATION-and-REPLACE-AUTHORIZATION) * [Teradata VantageCloud Lake Compute Clusters](https://docs.teradata.com/r/Teradata-VantageCloud-Lake/Managing-Compute-Resources/Compute-Clusters) * [Prerequisites](https://docs.dagster.io/integrations/libraries/teradata#prerequisites) * [Install dagster-teradata](https://docs.dagster.io/integrations/libraries/teradata#install-dagster-teradata) * [Initialize a Dagster Project](https://docs.dagster.io/integrations/libraries/teradata#initialize-a-dagster-project) * [Scaffold a New Dagster Project](https://docs.dagster.io/integrations/libraries/teradata#scaffold-a-new-dagster-project) * [Create Sample Data](https://docs.dagster.io/integrations/libraries/teradata#create-sample-data) * [Define Assets for the ETL Pipeline](https://docs.dagster.io/integrations/libraries/teradata#define-assets-for-the-etl-pipeline) * [Define the Pipeline Definitions](https://docs.dagster.io/integrations/libraries/teradata#define-the-pipeline-definitions) * [Running the Pipeline](https://docs.dagster.io/integrations/libraries/teradata#running-the-pipeline) * [Below are some of the operations provided by the TeradataResource:](https://docs.dagster.io/integrations/libraries/teradata#below-are-some-of-the-operations-provided-by-the-teradataresource) * [1\. Execute a Query (`execute_query`)](https://docs.dagster.io/integrations/libraries/teradata#1-execute-a-query-execute_query) * [2\. Execute Multiple Queries (`execute_queries`)](https://docs.dagster.io/integrations/libraries/teradata#2-execute-multiple-queries-execute_queries) * [3\. Drop a Database (`drop_database`)](https://docs.dagster.io/integrations/libraries/teradata#3-drop-a-database-drop_database) * [4\. Drop a Table (`drop_table`)](https://docs.dagster.io/integrations/libraries/teradata#4-drop-a-table-drop_table) * [Data Transfer from AWS S3 to Teradata Vantage Using dagster-teradata:](https://docs.dagster.io/integrations/libraries/teradata#data-transfer-from-aws-s3-to-teradata-vantage-using-dagster-teradata) * [Arguments Supported by `s3_blob_to_teradata`](https://docs.dagster.io/integrations/libraries/teradata#arguments-supported-by-s3_blob_to_teradata) * [Data Transfer from Azure Blob to Teradata Vantage Using dagster-teradata:](https://docs.dagster.io/integrations/libraries/teradata#data-transfer-from-azure-blob-to-teradata-vantage-using-dagster-teradata) * [Arguments Supported by `azure_blob_to_teradata`](https://docs.dagster.io/integrations/libraries/teradata#arguments-supported-by-azure_blob_to_teradata) * [Transfer data from Private Blob Storage Container to Teradata instance](https://docs.dagster.io/integrations/libraries/teradata#transfer-data-from-private-blob-storage-container-to-teradata-instance) * [Manage VantageCloud Lake Compute Clusters with dagster-teradata:](https://docs.dagster.io/integrations/libraries/teradata#manage-vantagecloud-lake-compute-clusters-with-dagster-teradata) * [1\. Create a Compute Cluster (`create_teradata_compute_cluster`)](https://docs.dagster.io/integrations/libraries/teradata#1-create-a-compute-cluster-create_teradata_compute_cluster) * [2\. Suspend a Compute Cluster (`suspend_teradata_compute_cluster`)](https://docs.dagster.io/integrations/libraries/teradata#2-suspend-a-compute-cluster-suspend_teradata_compute_cluster) * [3\. Resume a Compute Cluster (`resume_teradata_compute_cluster`)](https://docs.dagster.io/integrations/libraries/teradata#3-resume-a-compute-cluster-resume_teradata_compute_cluster) * [4\. Delete a Compute Cluster (`drop_teradata_compute_cluster`)](https://docs.dagster.io/integrations/libraries/teradata#4-delete-a-compute-cluster-drop_teradata_compute_cluster) * [Further reading](https://docs.dagster.io/integrations/libraries/teradata#further-reading) --- # Dagster & Sigma | Dagster Docs [Skip to main content](https://docs.dagster.io/integrations/libraries/sigma#__docusaurus_skipToContent_fallback) On this page info This feature is considered in a beta stage. It is still being tested and may change. For more information, see the [API lifecycle stages documentation](https://docs.dagster.io/api/api-lifecycle/api-lifecycle-stages) . Your Sigma assets, including datasets and workbooks, can be represented in the Dagster asset graph, allowing you to track lineage and dependencies between Sigma assets and upstream data assets you are already modeling in Dagster. What you'll learn[​](https://docs.dagster.io/integrations/libraries/sigma#what-youll-learn "Direct link to What you'll learn") ------------------------------------------------------------------------------------------------------------------------------- * How to represent Sigma assets in the Dagster asset graph, including lineage to other Dagster assets. * How to customize asset definition metadata for these Sigma assets. Prerequisites * The `dagster-sigma` library installed in your environment * Familiarity with asset definitions and the Dagster asset graph * Familiarity with Dagster resources * Familiarity with Sigma concepts, like datasets and workbooks * A Sigma organization * A Sigma client ID and client secret. For more information, see [Generate API client credentials](https://help.sigmacomputing.com/reference/generate-client-credentials#generate-api-client-credentials) in the Sigma documentation. Set up your environment[​](https://docs.dagster.io/integrations/libraries/sigma#set-up-your-environment "Direct link to Set up your environment") -------------------------------------------------------------------------------------------------------------------------------------------------- To get started, you'll need to install the `dagster` and `dagster-sigma` Python packages: * uv * pip uv add dagster-sigma pip install dagster-sigma Represent Sigma assets in the asset graph[​](https://docs.dagster.io/integrations/libraries/sigma#represent-sigma-assets-in-the-asset-graph "Direct link to Represent Sigma assets in the asset graph") -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- To load Sigma assets into the Dagster asset graph, you must first construct a [`SigmaOrganization`](https://docs.dagster.io/api/libraries/dagster-sigma#dagster_sigma.SigmaOrganization) resource, which allows Dagster to communicate with your Sigma organization. You'll need to supply your client ID and client secret alongside the base URL. See [Identify your API request URL](https://help.sigmacomputing.com/reference/get-started-sigma-api#identify-your-api-request-url) in the Sigma documentation for more information on how to find your base URL. Dagster can automatically load all datasets and workbooks from your Sigma organization as asset specs. Call the [`load_sigma_asset_specs`](https://docs.dagster.io/api/libraries/dagster-sigma#dagster_sigma.load_sigma_asset_specs) function, which returns list of [`AssetSpec`](https://docs.dagster.io/api/dagster/assets#dagster.AssetSpec) s representing your Sigma assets. You can then include these asset specs in your [`Definitions`](https://docs.dagster.io/api/dagster/definitions#dagster.Definitions) object: from dagster_sigma import SigmaBaseUrl, SigmaOrganization, load_sigma_asset_specsimport dagster as dgsigma_organization = SigmaOrganization( base_url=SigmaBaseUrl.AWS_US, client_id=dg.EnvVar("SIGMA_CLIENT_ID"), client_secret=dg.EnvVar("SIGMA_CLIENT_SECRET"),)sigma_specs = load_sigma_asset_specs(sigma_organization)defs = dg.Definitions(assets=[*sigma_specs], resources={"sigma": sigma_organization}) Load Sigma assets from filtered workbooks[​](https://docs.dagster.io/integrations/libraries/sigma#load-sigma-assets-from-filtered-workbooks "Direct link to Load Sigma assets from filtered workbooks") -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- It is possible to load a subset of your Sigma assets by providing a [`SigmaFilter`](https://docs.dagster.io/api/libraries/dagster-sigma#dagster_sigma.SigmaFilter) to the [`load_sigma_asset_specs`](https://docs.dagster.io/api/libraries/dagster-sigma#dagster_sigma.load_sigma_asset_specs) function. This `SigmaFilter` object allows you to specify the folders from which you want to load Sigma workbooks, and also will allow you to configure which datasets are represented as assets. Note that the content and size of Sigma organization may affect the performance of your Dagster deployments. Filtering the workbooks selection from which your Sigma assets will be loaded is particularly useful for improving loading times. from dagster_sigma import ( SigmaBaseUrl, SigmaFilter, SigmaOrganization, load_sigma_asset_specs,)import dagster as dgsigma_organization = SigmaOrganization( base_url=SigmaBaseUrl.AWS_US, client_id=dg.EnvVar("SIGMA_CLIENT_ID"), client_secret=dg.EnvVar("SIGMA_CLIENT_SECRET"),)sigma_specs = load_sigma_asset_specs( organization=sigma_organization, sigma_filter=SigmaFilter( # Filter down to only the workbooks in these folders workbook_folders=[ ("my_folder", "my_subfolder"), ("my_folder", "my_other_subfolder"), ], # Specify whether to include datasets that are not used in any workbooks # default is True include_unused_datasets=False, ),)defs = dg.Definitions(assets=[*sigma_specs], resources={"sigma": sigma_organization}) ### Load Sigma assets using a snapshot[​](https://docs.dagster.io/integrations/libraries/sigma#load-sigma-assets-using-a-snapshot "Direct link to Load Sigma assets using a snapshot") Sigma assets can be loaded using the snapshot of a Sigma organization, which allows organizations with large amounts of Sigma data to speed up their deployment process. from dagster_sigma import SigmaBaseUrl, SigmaOrganization, load_sigma_asset_specsimport dagster as dgsigma_organization = SigmaOrganization( base_url=SigmaBaseUrl.AWS_US, client_id=dg.EnvVar("SIGMA_CLIENT_ID"), client_secret=dg.EnvVar("SIGMA_CLIENT_SECRET"),)sigma_specs = load_sigma_asset_specs( organization=sigma_organization, snapshot_path=dg.EnvVar("SIGMA_SNAPSHOT_PATH"))defs = dg.Definitions(assets=[*sigma_specs], resources={"sigma": sigma_organization}) To capture the snapshot, the `dagster-sigma snapshot` CLI can be used. dagster-sigma snapshot --python-module my_dagster_package --output-path snapshot.snap Customize asset definition metadata for Sigma assets[​](https://docs.dagster.io/integrations/libraries/sigma#customize-asset-definition-metadata-for-sigma-assets "Direct link to Customize asset definition metadata for Sigma assets") ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- By default, Dagster will generate asset specs for each Sigma asset based on its type, and populate default metadata. You can further customize asset properties by passing a custom [`DagsterSigmaTranslator`](https://docs.dagster.io/api/libraries/dagster-sigma#dagster_sigma.DagsterSigmaTranslator) subclass to the [`load_sigma_asset_specs`](https://docs.dagster.io/api/libraries/dagster-sigma#dagster_sigma.load_sigma_asset_specs) function. This subclass can implement methods to customize the asset specs for each Sigma asset type. from dagster_sigma import ( DagsterSigmaTranslator, SigmaBaseUrl, SigmaOrganization, SigmaWorkbookTranslatorData, load_sigma_asset_specs,)import dagster as dgsigma_organization = SigmaOrganization( base_url=SigmaBaseUrl.AWS_US, client_id=dg.EnvVar("SIGMA_CLIENT_ID"), client_secret=dg.EnvVar("SIGMA_CLIENT_SECRET"),)# A translator class lets us customize properties of the built Sigma assets, such as the owners or asset keyclass MyCustomSigmaTranslator(DagsterSigmaTranslator): def get_asset_spec(self, data: SigmaWorkbookTranslatorData) -> dg.AssetSpec: # pyright: ignore[reportIncompatibleMethodOverride] # We create the default asset spec using super() default_spec = super().get_asset_spec(data) # we customize the team owner tag for all Sigma assets return default_spec.replace_attributes(owners=["team:my_team"])sigma_specs = load_sigma_asset_specs( sigma_organization, dagster_sigma_translator=MyCustomSigmaTranslator())defs = dg.Definitions(assets=[*sigma_specs], resources={"sigma": sigma_organization}) Note that `super()` is called in each of the overridden methods to generate the default asset spec. It is best practice to generate the default asset spec before customizing it. Load Sigma assets from multiple organizations[​](https://docs.dagster.io/integrations/libraries/sigma#load-sigma-assets-from-multiple-organizations "Direct link to Load Sigma assets from multiple organizations") -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Definitions from multiple Sigma organizations can be combined by instantiating multiple [`SigmaOrganization`](https://docs.dagster.io/api/libraries/dagster-sigma#dagster_sigma.SigmaOrganization) resources and merging their specs. This lets you view all your Sigma assets in a single asset graph: from dagster_sigma import SigmaBaseUrl, SigmaOrganization, load_sigma_asset_specsimport dagster as dgsales_team_organization = SigmaOrganization( base_url=SigmaBaseUrl.AWS_US, client_id=dg.EnvVar("SALES_SIGMA_CLIENT_ID"), client_secret=dg.EnvVar("SALES_SIGMA_CLIENT_SECRET"),)marketing_team_organization = SigmaOrganization( base_url=SigmaBaseUrl.AWS_US, client_id=dg.EnvVar("MARKETING_SIGMA_CLIENT_ID"), client_secret=dg.EnvVar("MARKETING_SIGMA_CLIENT_SECRET"),)sales_team_specs = load_sigma_asset_specs(sales_team_organization)marketing_team_specs = load_sigma_asset_specs(marketing_team_organization)# Merge the specs into a single set of definitionsdefs = dg.Definitions( assets=[*sales_team_specs, *marketing_team_specs], resources={ "marketing_sigma": marketing_team_organization, "sales_sigma": sales_team_organization, },) Customize upstream dependencies[​](https://docs.dagster.io/integrations/libraries/sigma#customize-upstream-dependencies "Direct link to Customize upstream dependencies") -------------------------------------------------------------------------------------------------------------------------------------------------------------------------- By default, Dagster sets upstream dependencies when generating asset specs for your Sigma assets. To do so, Dagster parses information about assets that are upstream of specific Sigma assets from the Sigma organization itself. You can customize how upstream dependencies are set on your Sigma assets by passing an instance of the custom [`DagsterSigmaTranslator`](https://docs.dagster.io/api/libraries/dagster-sigma#dagster_sigma.DagsterSigmaTranslator) to the [`load_sigma_asset_specs`](https://docs.dagster.io/api/libraries/dagster-sigma#dagster_sigma.load_sigma_asset_specs) function. The below example defines `my_upstream_asset` as an upstream dependency of `my_sigma_workbook`: class MyCustomSigmaTranslator(DagsterSigmaTranslator): def get_asset_spec( self, data: Union[SigmaDatasetTranslatorData, SigmaWorkbookTranslatorData] ) -> dg.AssetSpec: # We create the default asset spec using super() default_spec = super().get_asset_spec(data) # We customize upstream dependencies for the Sigma workbook named `my_sigma_workbook` return default_spec.replace_attributes( deps=["my_upstream_asset"] if data.properties["name"] == "my_sigma_workbook" else ... )sigma_specs = load_sigma_asset_specs( sigma_organization, dagster_sigma_translator=MyCustomSigmaTranslator()) Note that `super()` is called in each of the overridden methods to generate the default asset spec. It is best practice to generate the default asset spec before customizing it. Related[​](https://docs.dagster.io/integrations/libraries/sigma#related "Direct link to Related") -------------------------------------------------------------------------------------------------- * [`dagster-sigma` API reference](https://docs.dagster.io/api/libraries/dagster-sigma) * [Asset definitions](https://docs.dagster.io/guides/build/assets/defining-assets) * [Resources](https://docs.dagster.io/guides/build/external-resources) * [Using environment variables and secrets](https://docs.dagster.io/guides/operate/configuration/using-environment-variables-and-secrets) * [What you'll learn](https://docs.dagster.io/integrations/libraries/sigma#what-youll-learn) * [Set up your environment](https://docs.dagster.io/integrations/libraries/sigma#set-up-your-environment) * [Represent Sigma assets in the asset graph](https://docs.dagster.io/integrations/libraries/sigma#represent-sigma-assets-in-the-asset-graph) * [Load Sigma assets from filtered workbooks](https://docs.dagster.io/integrations/libraries/sigma#load-sigma-assets-from-filtered-workbooks) * [Load Sigma assets using a snapshot](https://docs.dagster.io/integrations/libraries/sigma#load-sigma-assets-using-a-snapshot) * [Customize asset definition metadata for Sigma assets](https://docs.dagster.io/integrations/libraries/sigma#customize-asset-definition-metadata-for-sigma-assets) * [Load Sigma assets from multiple organizations](https://docs.dagster.io/integrations/libraries/sigma#load-sigma-assets-from-multiple-organizations) * [Customize upstream dependencies](https://docs.dagster.io/integrations/libraries/sigma#customize-upstream-dependencies) * [Related](https://docs.dagster.io/integrations/libraries/sigma#related) --- # dagster-snowflake integration reference | Dagster Docs [Skip to main content](https://docs.dagster.io/integrations/libraries/snowflake/reference#__docusaurus_skipToContent_fallback) On this page This reference page provides information for working with [`dagster-snowflake`](https://docs.dagster.io/api/libraries/dagster-snowflake) features that are not covered as part of the Snowflake & Dagster tutorials ([resources](https://docs.dagster.io/integrations/libraries/snowflake/using-snowflake-with-dagster) , [I/O managers](https://docs.dagster.io/integrations/libraries/snowflake/using-snowflake-with-dagster-io-managers) ). Authenticating using a private key[​](https://docs.dagster.io/integrations/libraries/snowflake/reference#authenticating-using-a-private-key "Direct link to Authenticating using a private key") ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- In addition to password-based authentication, you can authenticate with Snowflake using a key pair. To set up private key authentication for your Snowflake account, see the instructions in the [Snowflake docs](https://docs.snowflake.com/en/user-guide/key-pair-auth.html#configuring-key-pair-authentication) . Currently, the Dagster's Snowflake integration only supports encrypted private keys. You can provide the private key directly to the Snowflake resource or I/O manager, or via a file containing the private key. * Resources * I/O managers **Directly to the resource** from dagster_snowflake import SnowflakeResourcefrom dagster import Definitions, EnvVardefs = Definitions( assets=[iris_dataset], resources={ "snowflake": SnowflakeResource( account="abc1234.us-east-1", user=EnvVar("SNOWFLAKE_USER"), private_key=EnvVar("SNOWFLAKE_PK"), private_key_password=EnvVar("SNOWFLAKE_PK_PASSWORD"), database="FLOWERS", ) },) **Via a file** from dagster_snowflake import SnowflakeResourcefrom dagster import Definitions, EnvVardefs = Definitions( assets=[iris_dataset], resources={ "snowflake": SnowflakeResource( account="abc1234.us-east-1", user=EnvVar("SNOWFLAKE_USER"), private_key_path="/path/to/private/key/file.p8", private_key_password=EnvVar("SNOWFLAKE_PK_PASSWORD"), database="FLOWERS", ) },) **Directly to the I/O manager** from dagster_snowflake_pandas import SnowflakePandasIOManagerfrom dagster import Definitions, EnvVardefs = Definitions( assets=[iris_dataset], resources={ "io_manager": SnowflakePandasIOManager( account="abc1234.us-east-1", user=EnvVar("SNOWFLAKE_USER"), private_key=EnvVar("SNOWFLAKE_PK"), private_key_password=EnvVar("SNOWFLAKE_PK_PASSWORD"), database="FLOWERS", ) },) **Via a file** from dagster_snowflake_pandas import SnowflakePandasIOManagerfrom dagster import Definitions, EnvVardefs = Definitions( assets=[iris_dataset], resources={ "io_manager": SnowflakePandasIOManager( account="abc1234.us-east-1", user=EnvVar("SNOWFLAKE_USER"), private_key_path="/path/to/private/key/file.p8", private_key_password=EnvVar("SNOWFLAKE_PK_PASSWORD"), database="FLOWERS", ) },) Using the Snowflake resource[​](https://docs.dagster.io/integrations/libraries/snowflake/reference#using-the-snowflake-resource "Direct link to Using the Snowflake resource") ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- ### Executing custom SQL commands[​](https://docs.dagster.io/integrations/libraries/snowflake/reference#executing-custom-sql-commands "Direct link to Executing custom SQL commands") Using a [Snowflake resource](https://docs.dagster.io/api/libraries/dagster-snowflake#dagster_snowflake.SnowflakeResource) , you can execute custom SQL queries on a Snowflake database: from dagster_snowflake import SnowflakeResourcefrom dagster import Definitions, EnvVar, asset# this example executes a query against the IRIS_DATASET table created in Step 2 of the# Using Dagster with Snowflake tutorial@assetdef small_petals(snowflake: SnowflakeResource): query = """ create or replace table iris.small_petals as ( SELECT * FROM iris.iris_dataset WHERE species = 'petal_length_cm' < 1 AND 'petal_width_cm' < 1 ); """ with snowflake.get_connection() as conn: conn.cursor.execute(query) # pyright: ignore[reportFunctionMemberAccess]defs = Definitions( assets=[small_petals], resources={ "snowflake": SnowflakeResource( account="abc1234.us-east-1", user=EnvVar("SNOWFLAKE_USER"), password=EnvVar("SNOWFLAKE_PASSWORD"), database="FLOWERS", schema="IRIS", ) },) Let's review what's happening in this example: * Attached the `SnowflakeResource` to the `small_petals` asset * Used the `get_connection` context manager method of the Snowflake resource to get a [`snowflake.connector.Connection`](https://docs.snowflake.com/en/developer-guide/python-connector/python-connector-api#object-connection) object * Used the connection to execute a custom SQL query against the `IRIS_DATASET` table created in [Step 2](https://docs.dagster.io/integrations/libraries/snowflake/using-snowflake-with-dagster#step-2-create-tables-in-snowflake) of the [Snowflake resource tutorial](https://docs.dagster.io/integrations/libraries/snowflake/using-snowflake-with-dagster) For more information on the Snowflake resource, including additional configuration settings, see the [`SnowflakeResource`](https://docs.dagster.io/api/libraries/dagster-snowflake#dagster_snowflake.SnowflakeResource) API docs. Using the Snowflake I/O manager[​](https://docs.dagster.io/integrations/libraries/snowflake/reference#using-the-snowflake-io-manager "Direct link to Using the Snowflake I/O manager") --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- ### Selecting specific columns in a downstream asset[​](https://docs.dagster.io/integrations/libraries/snowflake/reference#selecting-specific-columns-in-a-downstream-asset "Direct link to Selecting specific columns in a downstream asset") Sometimes you may not want to fetch an entire table as the input to a downstream asset. With the Snowflake I/O manager, you can select specific columns to load by supplying metadata on the downstream asset. import pandas as pdfrom dagster import AssetIn, asset# this example uses the iris_dataset asset from Step 2 of the Using Dagster with Snowflake tutorial@asset( ins={ "iris_sepal": AssetIn( key="iris_dataset", metadata={"columns": ["sepal_length_cm", "sepal_width_cm"]}, ) })def sepal_data(iris_sepal: pd.DataFrame) -> pd.DataFrame: iris_sepal["sepal_area_cm2"] = ( iris_sepal["sepal_length_cm"] * iris_sepal["sepal_width_cm"] ) return iris_sepal In this example, we only use the columns containing sepal data from the `IRIS_DATASET` table created in [Step 2](https://docs.dagster.io/integrations/libraries/snowflake/using-snowflake-with-dagster-io-managers#store-a-dagster-asset-as-a-table-in-snowflake) of the [Snowflake I/O manager tutorial](https://docs.dagster.io/integrations/libraries/snowflake/using-snowflake-with-dagster-io-managers) . Fetching the entire table would be unnecessarily costly, so to select specific columns, we can add metadata to the input asset. We do this in the `metadata` parameter of the `AssetIn` that loads the `iris_dataset` asset in the `ins` parameter. We supply the key `columns` with a list of names of the columns we want to fetch. When Dagster materializes `sepal_data` and loads the `iris_dataset` asset using the Snowflake I/O manager, it will only fetch the `sepal_length_cm` and `sepal_width_cm` columns of the `FLOWERS.IRIS.IRIS_DATASET` table and pass them to `sepal_data` as a Pandas DataFrame. ### Storing partitioned assets[​](https://docs.dagster.io/integrations/libraries/snowflake/reference#storing-partitioned-assets "Direct link to Storing partitioned assets") The Snowflake I/O manager supports storing and loading partitioned data. In order to correctly store and load data from the Snowflake table, the Snowflake I/O manager needs to know which column contains the data defining the partition bounds. The Snowflake I/O manager uses this information to construct the correct queries to select or replace the data. In the following sections, we describe how the I/O manager constructs these queries for different types of partitions. * Statically-partitioned assets * Time-partitioned assets * Multi-partitioned assets To store statically-partitioned assets in Snowflake, specify `partition_expr` metadata on the asset to tell the Snowflake I/O manager which column contains the partition data: import pandas as pdfrom dagster import AssetExecutionContext, StaticPartitionsDefinition, asset@asset( partitions_def=StaticPartitionsDefinition( ["Iris-setosa", "Iris-virginica", "Iris-versicolor"] ), metadata={"partition_expr": "SPECIES"},)def iris_dataset_partitioned(context: AssetExecutionContext) -> pd.DataFrame: species = context.partition_key full_df = pd.read_csv( "https://docs.dagster.io/assets/iris.csv", names=[ "sepal_length_cm", "sepal_width_cm", "petal_length_cm", "petal_width_cm", "species", ], ) return full_df[full_df["Species"] == species]@assetdef iris_cleaned(iris_dataset_partitioned: pd.DataFrame): return iris_dataset_partitioned.dropna().drop_duplicates() Dagster uses the `partition_expr` metadata to craft the `SELECT` statement when loading the partition in the downstream asset. When loading a static partition (or multiple static partitions), the following statement is used: SELECT * WHERE [partition_expr] in ([selected partitions]) When the `partition_expr` value is injected into this statement, the resulting SQL query must follow Snowflake's SQL syntax. Refer to the [Snowflake documentation](https://docs.snowflake.com/en/sql-reference/constructs) for more information. When materializing the above assets, a partition must be selected. In this example, the query used when materializing the `Iris-setosa` partition of the above assets would be: SELECT * WHERE SPECIES in ('Iris-setosa') Like statically-partitioned assets, you can specify `partition_expr` metadata on the asset to tell the Snowflake I/O manager which column contains the partition data: import pandas as pdfrom dagster import AssetExecutionContext, DailyPartitionsDefinition, asset@asset( partitions_def=DailyPartitionsDefinition(start_date="2023-01-01"), metadata={"partition_expr": "TO_TIMESTAMP(TIME::INT)"},)def iris_data_per_day(context: AssetExecutionContext) -> pd.DataFrame: partition = context.partition_key # get_iris_data_for_date fetches all of the iris data for a given date, # the returned dataframe contains a column named 'time' with that stores # the time of the row as an integer of seconds since epoch return get_iris_data_for_date(partition)@assetdef iris_cleaned(iris_data_per_day: pd.DataFrame): return iris_data_per_day.dropna().drop_duplicates() Dagster uses the `partition_expr` metadata to craft the `SELECT` statement when loading the correct partition in the downstream asset. When loading a dynamic partition, the following statement is used: SELECT * WHERE [partition_expr] >= [partition_start] AND [partition_expr] < [partition_end] When the `partition_expr` value is injected into this statement, the resulting SQL query must follow Snowflake's SQL syntax. Refer to the [Snowflake documentation](https://docs.snowflake.com/en/sql-reference/constructs) for more information. When materializing the above assets, a partition must be selected. The `[partition_start]` and `[partition_end]` bounds are of the form `YYYY-MM-DD HH:MM:SS`. In this example, the query when materializing the `2023-01-02` partition of the above assets would be: SELECT * WHERE TO_TIMESTAMP(TIME::INT) >= '2023-01-02 00:00:00' AND TO_TIMESTAMP(TIME::INT) < '2023-01-03 00:00:00' In this example, the data in the `TIME` column are integers, so the `partition_expr` metadata includes a SQL statement to convert integers to timestamps. A full list of Snowflake functions can be found [here](https://docs.snowflake.com/en/sql-reference/functions-all) . The Snowflake I/O manager can also store data partitioned on multiple dimensions. To do this, you must specify the column for each partition as a dictionary of `partition_expr` metadata: import pandas as pdimport dagster as dg@dg.asset( partitions_def=dg.MultiPartitionsDefinition( { "date": dg.DailyPartitionsDefinition(start_date="2023-01-01"), "species": dg.StaticPartitionsDefinition( ["Iris-setosa", "Iris-virginica", "Iris-versicolor"] ), } ), metadata={ "partition_expr": {"date": "TO_TIMESTAMP(TIME::INT)", "species": "SPECIES"} },)def iris_dataset_partitioned(context: dg.AssetExecutionContext) -> pd.DataFrame: partition = context.partition_key.keys_by_dimension species = partition["species"] date = partition["date"] # get_iris_data_for_date fetches all of the iris data for a given date, # the returned dataframe contains a column named 'time' with that stores # the time of the row as an integer of seconds since epoch full_df = get_iris_data_for_date(date) return full_df[full_df["species"] == species]@dg.assetdef iris_cleaned(iris_dataset_partitioned: pd.DataFrame): return iris_dataset_partitioned.dropna().drop_duplicates() Dagster uses the `partition_expr` metadata to craft the `SELECT` statement when loading the correct partition in a downstream asset. For multi-partitions, Dagster concatenates the `WHERE` statements described in the above sections to craft the correct `SELECT` statement. When materializing the above assets, a partition must be selected. For example, when materializing the `2023-01-02|Iris-setosa` partition of the above assets, the following query will be used: SELECT * WHERE SPECIES in ('Iris-setosa') AND TO_TIMESTAMP(TIME::INT) >= '2023-01-02 00:00:00' AND TO_TIMESTAMP(TIME::INT) < '2023-01-03 00:00:00' ### Storing tables in multiple schemas[​](https://docs.dagster.io/integrations/libraries/snowflake/reference#storing-tables-in-multiple-schemas "Direct link to Storing tables in multiple schemas") If you want to have different assets stored in different Snowflake schemas, the Snowflake I/O manager allows you to specify the schema in a few ways. You can specify the default schema where data will be stored as configuration to the I/O manager, like we did in [Step 1](https://docs.dagster.io/integrations/libraries/snowflake/using-snowflake-with-dagster-io-managers#step-1-configure-the-snowflake-io-manager) of the [Snowflake I/O manager tutorial](https://docs.dagster.io/integrations/libraries/snowflake/using-snowflake-with-dagster-io-managers) . To store assets in different schemas, specify the schema as metadata: daffodil_dataset = AssetSpec( key=["daffodil_dataset"], metadata={"schema": "daffodil"} ) @asset(metadata={"schema": "iris"}) def iris_dataset() -> pd.DataFrame: return pd.read_csv( "https://docs.dagster.io/assets/iris.csv", names=[ "sepal_length_cm", "sepal_width_cm", "petal_length_cm", "petal_width_cm", "species", ], ) You can also specify the schema as part of the asset's asset key: daffodil_dataset = AssetSpec(key=["daffodil", "daffodil_dataset"]) @asset(key_prefix=["iris"]) def iris_dataset() -> pd.DataFrame: return pd.read_csv( "https://docs.dagster.io/assets/iris.csv", names=[ "sepal_length_cm", "sepal_width_cm", "petal_length_cm", "petal_width_cm", "species", ], ) In this example, the `iris_dataset` asset will be stored in the `IRIS` schema, and the `daffodil_dataset` asset will be found in the `DAFFODIL` schema. note The schema is determined in this order: 1. If the schema is set via metadata, that schema will be used 2. Otherwise, the schema set as configuration on the I/O manager will be used 3. Otherwise, if there is a `key_prefix`, that schema will be used 4. If none of the above are provided, the default schema will be `PUBLIC` ### Storing timestamp data in Pandas DataFrames[​](https://docs.dagster.io/integrations/libraries/snowflake/reference#storing-timestamp-data-in-pandas-dataframes "Direct link to Storing timestamp data in Pandas DataFrames") When storing a Pandas DataFrame with the Snowflake I/O manager, the I/O manager will check if timestamp data has a timezone attached, and if not, **it will assign the UTC timezone**. In Snowflake, you will see the timestamp data stored as the `TIMESTAMP_NTZ(9)` type, as this is the type assigned by the Snowflake Pandas connector. note Prior to `dagster-snowflake` version `0.19.0` the Snowflake I/O manager converted all timestamp data to strings before loading the data in Snowflake, and did the opposite conversion when fetching a DataFrame from Snowflake. If you have used a version of `dagster-snowflake` prior to version `0.19.0`, see the [Dagster version upgrade guide](https://docs.dagster.io/migration/upgrading#extension-libraries) for information about migrating database tables. ### Using the Snowflake I/O manager with other I/O managers[​](https://docs.dagster.io/integrations/libraries/snowflake/reference#using-the-snowflake-io-manager-with-other-io-managers "Direct link to Using the Snowflake I/O manager with other I/O managers") You may have assets that you don't want to store in Snowflake. You can provide an I/O manager to each asset using the `io_manager_key` parameter in the `asset` decorator: import pandas as pdfrom dagster_aws.s3.io_manager import s3_pickle_io_managerfrom dagster_snowflake_pandas import SnowflakePandasIOManagerfrom dagster import Definitions, EnvVar, asset@asset(io_manager_key="warehouse_io_manager")def iris_dataset() -> pd.DataFrame: return pd.read_csv( "https://docs.dagster.io/assets/iris.csv", names=[ "sepal_length_cm", "sepal_width_cm", "petal_length_cm", "petal_width_cm", "species", ], )@asset(io_manager_key="blob_io_manager")def iris_plots(iris_dataset): # plot_data is a function we've defined somewhere else # that plots the data in a DataFrame return plot_data(iris_dataset)defs = Definitions( assets=[iris_dataset, iris_plots], resources={ "warehouse_io_manager": SnowflakePandasIOManager( database="FLOWERS", schema="IRIS", account="abc1234.us-east-1", user=EnvVar("SNOWFLAKE_USER"), password=EnvVar("SNOWFLAKE_PASSWORD"), ), "blob_io_manager": s3_pickle_io_manager, },) In this example, the `iris_dataset` asset uses the I/O manager bound to the key `warehouse_io_manager` and `iris_plots` will use the I/O manager bound to the key `blob_io_manager`. In the [`Definitions`](https://docs.dagster.io/api/dagster/definitions#dagster.Definitions) object, we supply the I/O managers for those keys. When the assets are materialized, the `iris_dataset` will be stored in Snowflake, and `iris_plots` will be saved in Amazon S3. ### Storing and loading PySpark DataFrames in Snowflake[​](https://docs.dagster.io/integrations/libraries/snowflake/reference#storing-and-loading-pyspark-dataframes-in-snowflake "Direct link to Storing and loading PySpark DataFrames in Snowflake") The Snowflake I/O manager also supports storing and loading PySpark DataFrames. To use the [`SnowflakePySparkIOManager`](https://docs.dagster.io/api/libraries/dagster-snowflake-pyspark#dagster_snowflake_pyspark.SnowflakePySparkIOManager) , first install the package: * uv * pip uv add dagster-snowflake-pyspark pip install dagster-snowflake-pyspark Then you can use the `SnowflakePySparkIOManager` in your `Definitions` as in [Step 1](https://docs.dagster.io/integrations/libraries/snowflake/using-snowflake-with-dagster-io-managers#step-1-configure-the-snowflake-io-manager) of the [Snowflake I/O manager tutorial](https://docs.dagster.io/integrations/libraries/snowflake/using-snowflake-with-dagster-io-managers) . from dagster_snowflake_pyspark import SnowflakePySparkIOManagerfrom dagster import Definitions, EnvVardefs = Definitions( assets=[iris_dataset], resources={ "io_manager": SnowflakePySparkIOManager( account="abc1234.us-east-1", # required user=EnvVar("SNOWFLAKE_USER"), # required password=EnvVar("SNOWFLAKE_PASSWORD"), # password or private key required database="FLOWERS", # required warehouse="PLANTS", # required for PySpark role="writer", # optional, defaults to the default role for the account schema="IRIS", # optional, defaults to PUBLIC ) },) note When using the `snowflake_pyspark_io_manager` the `warehouse` configuration is required. The `SnowflakePySparkIOManager` requires that a `SparkSession` be active and configured with the [Snowflake connector for Spark](https://docs.snowflake.com/en/user-guide/spark-connector.html) . You can either create your own `SparkSession` or use the [`spark_resource`](https://docs.dagster.io/api/libraries/dagster-spark#dagster_spark.spark_resource) . * With the spark\_resource * With your own SparkSession from dagster_pyspark import pyspark_resourcefrom dagster_snowflake_pyspark import SnowflakePySparkIOManagerfrom pyspark import SparkFilesfrom pyspark.sql import DataFramefrom pyspark.sql.types import DoubleType, StringType, StructField, StructTypefrom dagster import AssetExecutionContext, Definitions, EnvVar, assetSNOWFLAKE_JARS = "net.snowflake:snowflake-jdbc:3.8.0,net.snowflake:spark-snowflake_2.12:2.8.2-spark_3.0"@asset(required_resource_keys={"pyspark"})def iris_dataset(context: AssetExecutionContext) -> DataFrame: spark = context.resources.pyspark.spark_session schema = StructType( [ StructField("sepal_length_cm", DoubleType()), StructField("sepal_width_cm", DoubleType()), StructField("petal_length_cm", DoubleType()), StructField("petal_width_cm", DoubleType()), StructField("species", StringType()), ] ) url = "https://docs.dagster.io/assets/iris.csv" spark.sparkContext.addFile(url) return spark.read.schema(schema).csv("file://" + SparkFiles.get("iris.csv"))defs = Definitions( assets=[iris_dataset], resources={ "io_manager": SnowflakePySparkIOManager( account="abc1234.us-east-1", user=EnvVar("SNOWFLAKE_USER"), password=EnvVar("SNOWFLAKE_PASSWORD"), database="FLOWERS", warehouse="PLANTS", schema="IRIS", ), "pyspark": pyspark_resource.configured( {"spark_conf": {"spark.jars.packages": SNOWFLAKE_JARS}} ), },) from dagster_snowflake_pyspark import SnowflakePySparkIOManagerfrom pyspark import SparkFilesfrom pyspark.sql import DataFrame, SparkSessionfrom pyspark.sql.types import DoubleType, StringType, StructField, StructTypefrom dagster import Definitions, EnvVar, assetSNOWFLAKE_JARS = "net.snowflake:snowflake-jdbc:3.8.0,net.snowflake:spark-snowflake_2.12:2.8.2-spark_3.0"@assetdef iris_dataset() -> DataFrame: spark = SparkSession.builder.config( key="spark.jars.packages", value=SNOWFLAKE_JARS, ).getOrCreate() schema = StructType( [ StructField("sepal_length_cm", DoubleType()), StructField("sepal_width_cm", DoubleType()), StructField("petal_length_cm", DoubleType()), StructField("petal_width_cm", DoubleType()), StructField("species", StringType()), ] ) url = ("https://docs.dagster.io/assets/iris.csv",) spark.sparkContext.addFile(url) return spark.read.schema(schema).csv("file://" + SparkFiles.get("iris.csv"))defs = Definitions( assets=[iris_dataset], resources={ "io_manager": SnowflakePySparkIOManager( account="abc1234.us-east-1", user=EnvVar("SNOWFLAKE_USER"), password=EnvVar("SNOWFLAKE_PASSWORD"), database="FLOWERS", warehouse="PLANTS", schema="IRIS", ), },) ### Using Pandas and PySpark DataFrames with Snowflake[​](https://docs.dagster.io/integrations/libraries/snowflake/reference#using-pandas-and-pyspark-dataframes-with-snowflake "Direct link to Using Pandas and PySpark DataFrames with Snowflake") If you work with both Pandas and PySpark DataFrames and want a single I/O manager to handle storing and loading these DataFrames in Snowflake, you can write a new I/O manager that handles both types. To do this, inherit from the [`SnowflakeIOManager`](https://docs.dagster.io/api/libraries/dagster-snowflake#dagster_snowflake.SnowflakeIOManager) base class and implement the `type_handlers` and `default_load_type` methods. The resulting I/O manager will inherit the configuration fields of the base `SnowflakeIOManager`. from typing import Optionalimport pandas as pdfrom dagster_snowflake import SnowflakeIOManagerfrom dagster_snowflake_pandas import SnowflakePandasTypeHandlerfrom dagster_snowflake_pyspark import SnowflakePySparkTypeHandlerfrom dagster import Definitions, EnvVarclass SnowflakePandasPySparkIOManager(SnowflakeIOManager): @staticmethod def type_handlers(): """type_handlers should return a list of the TypeHandlers that the I/O manager can use. Here we return the SnowflakePandasTypeHandler and SnowflakePySparkTypeHandler so that the I/O manager can store Pandas DataFrames and PySpark DataFrames. """ return [SnowflakePandasTypeHandler(), SnowflakePySparkTypeHandler()] @staticmethod def default_load_type() -> Optional[type]: """If an asset is not annotated with an return type, default_load_type will be used to determine which TypeHandler to use to store and load the output. In this case, unannotated assets will be stored and loaded as Pandas DataFrames. """ return pd.DataFramedefs = Definitions( assets=[iris_dataset, rose_dataset], resources={ "io_manager": SnowflakePandasPySparkIOManager( account="abc1234.us-east-1", user=EnvVar("SNOWFLAKE_USER"), password=EnvVar("SNOWFLAKE_PASSWORD"), database="FLOWERS", role="writer", warehouse="PLANTS", schema="IRIS", ) },) * [Authenticating using a private key](https://docs.dagster.io/integrations/libraries/snowflake/reference#authenticating-using-a-private-key) * [Using the Snowflake resource](https://docs.dagster.io/integrations/libraries/snowflake/reference#using-the-snowflake-resource) * [Executing custom SQL commands](https://docs.dagster.io/integrations/libraries/snowflake/reference#executing-custom-sql-commands) * [Using the Snowflake I/O manager](https://docs.dagster.io/integrations/libraries/snowflake/reference#using-the-snowflake-io-manager) * [Selecting specific columns in a downstream asset](https://docs.dagster.io/integrations/libraries/snowflake/reference#selecting-specific-columns-in-a-downstream-asset) * [Storing partitioned assets](https://docs.dagster.io/integrations/libraries/snowflake/reference#storing-partitioned-assets) * [Storing tables in multiple schemas](https://docs.dagster.io/integrations/libraries/snowflake/reference#storing-tables-in-multiple-schemas) * [Storing timestamp data in Pandas DataFrames](https://docs.dagster.io/integrations/libraries/snowflake/reference#storing-timestamp-data-in-pandas-dataframes) * [Using the Snowflake I/O manager with other I/O managers](https://docs.dagster.io/integrations/libraries/snowflake/reference#using-the-snowflake-io-manager-with-other-io-managers) * [Storing and loading PySpark DataFrames in Snowflake](https://docs.dagster.io/integrations/libraries/snowflake/reference#storing-and-loading-pyspark-dataframes-in-snowflake) * [Using Pandas and PySpark DataFrames with Snowflake](https://docs.dagster.io/integrations/libraries/snowflake/reference#using-pandas-and-pyspark-dataframes-with-snowflake) --- # Using Snowflake with with Dagster I/O managers | Dagster Docs [Skip to main content](https://docs.dagster.io/integrations/libraries/snowflake/using-snowflake-with-dagster-io-managers#__docusaurus_skipToContent_fallback) On this page This tutorial focuses on how to store and load Dagster's [asset definitions](https://docs.dagster.io/guides/build/assets/defining-assets) in Snowflake by using a Snowflake I/O manager. An [**I/O manager**](https://docs.dagster.io/guides/build/io-managers) transfers the responsibility of storing and loading DataFrames as Snowflake tables to Dagster. By the end of the tutorial, you will: * Configure a Snowflake I/O manager * Create a table in Snowflake using a Dagster asset * Make a Snowflake table available in Dagster * Load Snowflake tables in downstream assets This guide focuses on storing and loading Pandas DataFrames in Snowflake, but Dagster also supports using PySpark DataFrames with Snowflake. The concepts from this guide apply to working with PySpark DataFrames, and you can learn more about setting up and using the Snowflake I/O manager with PySpark DataFrames in the [Snowflake reference](https://docs.dagster.io/integrations/libraries/snowflake/reference) . **Prefer to use resources instead?** Unlike an I/O manager, resources allow you to run SQL queries directly against tables within an asset's compute function. For details, see "[Using Snowlake with Dagster resources](https://docs.dagster.io/integrations/libraries/snowflake/using-snowflake-with-dagster) ". Prerequisites[​](https://docs.dagster.io/integrations/libraries/snowflake/using-snowflake-with-dagster-io-managers#prerequisites "Direct link to Prerequisites") ----------------------------------------------------------------------------------------------------------------------------------------------------------------- To complete this tutorial, you'll need: * **To install the `dagster-snowflake` and `dagster-snowflake-pandas` libraries**: * uv * pip uv add dagster-snowflake dagster-snowflake-pandas pip install dagster-snowflake dagster-snowflake-pandas * **To gather the following information**, which is required to use the Snowflake I/O manager: * **Snowflake account name**: You can find this by logging into Snowflake and getting the account name from the URL: ![Snowflake account name from URL](https://docs.dagster.io/assets/images/snowflake-account-1fd2b596a574c69b6e74356530e9372a.png) * **Snowflake credentials**: You can authenticate with Snowflake two ways: with a username and password, or with a username and private key. The Snowflake I/O manager can read all of these authentication values from environment variables. In this guide, we use password authentication and store the username and password as `SNOWFLAKE_USER` and `SNOWFLAKE_PASSWORD`, respectively. export SNOWFLAKE_USER=export SNOWFLAKE_PASSWORD= Refer to the [Using environment variables and secrets guide](https://docs.dagster.io/guides/operate/configuration/using-environment-variables-and-secrets) for more info. For more information on authenticating with a private key, see [Authenticating with a private key](https://docs.dagster.io/integrations/libraries/snowflake/reference#authenticating-using-a-private-key) in the Snowflake reference guide. Step 1: Configure the Snowflake I/O manager[​](https://docs.dagster.io/integrations/libraries/snowflake/using-snowflake-with-dagster-io-managers#step-1-configure-the-snowflake-io-manager "Direct link to Step 1: Configure the Snowflake I/O manager") --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- The Snowflake I/O manager requires some configuration to connect to your Snowflake instance. The `account`, `user` are required to connect with Snowflake. One method of authentication is required. You can use a password or a private key. Additionally, you need to specify a `database` to where all the tables should be stored. You can also provide some optional configuration to further customize the Snowflake I/O manager. You can specify a `warehouse` and `schema` where data should be stored, and a `role` for the I/O manager. from dagster_snowflake_pandas import SnowflakePandasIOManagerfrom dagster import Definitions, EnvVardefs = Definitions( assets=[iris_dataset], resources={ "io_manager": SnowflakePandasIOManager( account="abc1234.us-east-1", # required user=EnvVar("SNOWFLAKE_USER"), # required password=EnvVar("SNOWFLAKE_PASSWORD"), # password or private key required database="FLOWERS", # required role="writer", # optional, defaults to the default role for the account warehouse="PLANTS", # optional, defaults to default warehouse for the account schema="IRIS", # optional, defaults to PUBLIC ) },) With this configuration, if you materialized an asset called `iris_dataset`, the Snowflake I/O manager would be permissioned with the role `writer` and would store the data in the `FLOWERS.IRIS.IRIS_DATASET` table in the `PLANTS` warehouse. Finally, in the [`Definitions`](https://docs.dagster.io/api/dagster/definitions#dagster.Definitions) object, we assign the [`SnowflakePandasIOManager`](https://docs.dagster.io/api/libraries/dagster-snowflake-pandas#dagster_snowflake_pandas.SnowflakePandasIOManager) to the `io_manager` key. `io_manager` is a reserved key to set the default I/O manager for your assets. For more info about each of the configuration values, refer to the [`SnowflakePandasIOManager`](https://docs.dagster.io/api/libraries/dagster-snowflake-pandas#dagster_snowflake_pandas.SnowflakePandasIOManager) API documentation. Step 2: Create tables in Snowflake[​](https://docs.dagster.io/integrations/libraries/snowflake/using-snowflake-with-dagster-io-managers#step-2-create-tables-in-snowflake "Direct link to Step 2: Create tables in Snowflake") ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- The Snowflake I/O manager can create and update tables for your Dagster defined assets, but you can also make existing Snowflake tables available to Dagster. * Create tables in Snowflake from Dagster assets * Make an existing table available in Dagster ### Store a Dagster asset as a table in Snowflake[​](https://docs.dagster.io/integrations/libraries/snowflake/using-snowflake-with-dagster-io-managers#store-a-dagster-asset-as-a-table-in-snowflake "Direct link to Store a Dagster asset as a table in Snowflake") To store data in Snowflake using the Snowflake I/O manager, the definitions of your assets don't need to change. You can tell Dagster to use the Snowflake I/O manager, like in [Step 1: Configure the Snowflake I/O manager](https://docs.dagster.io/integrations/libraries/snowflake/using-snowflake-with-dagster-io-managers#step-1-configure-the-snowflake-io-manager) , and Dagster will handle storing and loading your assets in Snowflake. import pandas as pdfrom dagster import asset@assetdef iris_dataset() -> pd.DataFrame: return pd.read_csv( "https://docs.dagster.io/assets/iris.csv", names=[ "sepal_length_cm", "sepal_width_cm", "petal_length_cm", "petal_width_cm", "species", ], ) In this example, we first define our [asset](https://docs.dagster.io/guides/build/assets/defining-assets) . Here, we are fetching the Iris dataset as a Pandas DataFrame and renaming the columns. The type signature of the function tells the I/O manager what data type it is working with, so it is important to include the return type `pd.DataFrame`. When Dagster materializes the `iris_dataset` asset using the configuration from [Step 1: Configure the Snowflake I/O manager](https://docs.dagster.io/integrations/libraries/snowflake/using-snowflake-with-dagster-io-managers#step-1-configure-the-snowflake-io-manager) , the Snowflake I/O manager will create the table `FLOWERS.IRIS.IRIS_DATASET` if it does not exist and replace the contents of the table with the value returned from the `iris_dataset` asset. You may already have tables in Snowflake that you want to make available to other Dagster assets. You can define [external assets](https://docs.dagster.io/guides/build/assets/external-assets) for these tables. By defining an external asset for the existing table, you tell Dagster how to find the table so it can be fetched for downstream assets. from dagster import AssetSpeciris_harvest_data = AssetSpec(key="iris_harvest_data") In this example, we create a [`AssetSpec`](https://docs.dagster.io/api/dagster/assets#dagster.AssetSpec) for a pre-existing table - perhaps created by an external data ingestion tool - that contains data about iris harvests. To make the data available to other Dagster assets, we need to tell the Snowflake I/O manager how to find the data. Since we supply the database and the schema in the I/O manager configuration in [Step 1: Configure the Snowflake I/O manager](https://docs.dagster.io/integrations/libraries/snowflake/using-snowflake-with-dagster-io-managers#step-1-configure-the-snowflake-io-manager) , we only need to provide the table name. We do this with the `key` parameter in `AssetSpec`. When the I/O manager needs to load the `iris_harvest_data` in a downstream asset, it will select the data in the `FLOWERS.IRIS.IRIS_HARVEST_DATA` table as a Pandas DataFrame and provide it to the downstream asset. Step 3: Load Snowflake tables in downstream assets[​](https://docs.dagster.io/integrations/libraries/snowflake/using-snowflake-with-dagster-io-managers#step-3-load-snowflake-tables-in-downstream-assets "Direct link to Step 3: Load Snowflake tables in downstream assets") ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Once you have created an asset that represents a table in Snowflake, you will likely want to create additional assets that work with the data. Dagster and the Snowflake I/O manager allow you to load the data stored in Snowflake tables into downstream assets. import pandas as pdfrom dagster import asset# this example uses the iris_dataset asset from Step 2@assetdef iris_cleaned(iris_dataset: pd.DataFrame) -> pd.DataFrame: return iris_dataset.dropna().drop_duplicates() In this example, we want to provide the `iris_dataset` asset from the [Store a Dagster asset as a table in Snowflake](https://docs.dagster.io/integrations/libraries/snowflake/using-snowflake-with-dagster-io-managers#store-a-dagster-asset-as-a-table-in-snowflake) example to the `iris_cleaned` asset. In `iris_cleaned`, the `iris_dataset` parameter tells Dagster that the value for the `iris_dataset` asset should be provided as input to `iris_cleaned`. When materializing these assets, Dagster will use the `SnowflakePandasIOManager` to fetch the `FLOWERS.IRIS.IRIS_DATASET` as a Pandas DataFrame and pass this DataFrame as the `iris_dataset` parameter to `iris_cleaned`. When `iris_cleaned` returns a Pandas DataFrame, Dagster will use the `SnowflakePandasIOManager` to store the DataFrame as the `FLOWERS.IRIS.IRIS_CLEANED` table in Snowflake. Completed code example[​](https://docs.dagster.io/integrations/libraries/snowflake/using-snowflake-with-dagster-io-managers#completed-code-example "Direct link to Completed code example") -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- When finished, your code should look like the following: import pandas as pdfrom dagster_snowflake_pandas import SnowflakePandasIOManagerfrom dagster import AssetSpec, Definitions, EnvVar, assetiris_harvest_data = AssetSpec(key="iris_harvest_data")@assetdef iris_dataset() -> pd.DataFrame: return pd.read_csv( "https://docs.dagster.io/assets/iris.csv", names=[ "sepal_length_cm", "sepal_width_cm", "petal_length_cm", "petal_width_cm", "species", ], )@assetdef iris_cleaned(iris_dataset: pd.DataFrame) -> pd.DataFrame: return iris_dataset.dropna().drop_duplicates()defs = Definitions( assets=[iris_dataset, iris_harvest_data, iris_cleaned], resources={ "io_manager": SnowflakePandasIOManager( account="abc1234.us-east-1", user=EnvVar("SNOWFLAKE_USER"), password=EnvVar("SNOWFLAKE_PASSWORD"), database="FLOWERS", role="writer", warehouse="PLANTS", schema="IRIS", ) },) * [Prerequisites](https://docs.dagster.io/integrations/libraries/snowflake/using-snowflake-with-dagster-io-managers#prerequisites) * [Step 1: Configure the Snowflake I/O manager](https://docs.dagster.io/integrations/libraries/snowflake/using-snowflake-with-dagster-io-managers#step-1-configure-the-snowflake-io-manager) * [Step 2: Create tables in Snowflake](https://docs.dagster.io/integrations/libraries/snowflake/using-snowflake-with-dagster-io-managers#step-2-create-tables-in-snowflake) * [Store a Dagster asset as a table in Snowflake](https://docs.dagster.io/integrations/libraries/snowflake/using-snowflake-with-dagster-io-managers#store-a-dagster-asset-as-a-table-in-snowflake) * [Step 3: Load Snowflake tables in downstream assets](https://docs.dagster.io/integrations/libraries/snowflake/using-snowflake-with-dagster-io-managers#step-3-load-snowflake-tables-in-downstream-assets) * [Completed code example](https://docs.dagster.io/integrations/libraries/snowflake/using-snowflake-with-dagster-io-managers#completed-code-example) --- # Using Snowflake with Dagster resources | Dagster Docs [Skip to main content](https://docs.dagster.io/integrations/libraries/snowflake/using-snowflake-with-dagster#__docusaurus_skipToContent_fallback) On this page This tutorial focuses on how to store and load Dagster's [asset definitions](https://docs.dagster.io/guides/build/assets/defining-assets) in Snowflake by using Dagster's [`SnowflakeResource`](https://docs.dagster.io/api/libraries/dagster-snowflake#dagster_snowflake.SnowflakeResource) . A [**resource**](https://docs.dagster.io/guides/build/external-resources) allows you to directly run SQL queries against tables within an asset's compute function. By the end of the tutorial, you will: * Configure a Snowflake resource * Use the Snowflake resource to execute a SQL query that creates a table * Load Snowflake tables in downstream assets * Add the assets and Snowflake resource to a `Definitions` object **Prefer to use an I/O manager?** Unlike resources, an [I/O manager](https://docs.dagster.io/guides/build/io-managers) transfers the responsibility of storing and loading DataFrames as Snowflake tables to Dagster. Refer to the [Snowlake I/O manager guide](https://docs.dagster.io/integrations/libraries/snowflake/using-snowflake-with-dagster-io-managers) for more info. Prerequisites[​](https://docs.dagster.io/integrations/libraries/snowflake/using-snowflake-with-dagster#prerequisites "Direct link to Prerequisites") ----------------------------------------------------------------------------------------------------------------------------------------------------- To complete this tutorial, you'll need: * **To install the following libraries**: * uv * pip uv add dagster-snowflake pandas pip install dagster-snowflake pandas * **To gather the following information**, which is required to use the Snowflake resource: * **Snowflake account name**: You can find this by logging into Snowflake and getting the account name from the URL: ![Snowflake account name in URL](https://docs.dagster.io/assets/images/snowflake-account-1fd2b596a574c69b6e74356530e9372a.png) * **Snowflake credentials**: You can authenticate with Snowflake two ways: with a username and password or with a username and private key. The Snowflake resource can read these authentication values from environment variables. In this guide, we use password authentication and store the username and password as `SNOWFLAKE_USER` and `SNOWFLAKE_PASSWORD`, respectively: export SNOWFLAKE_USER=export SNOWFLAKE_PASSWORD= Refer to the [Using environment variables and secrets guide](https://docs.dagster.io/guides/operate/configuration/using-environment-variables-and-secrets) for more info. For more information on authenticating with a private key, see [Authenticating with a private key](https://docs.dagster.io/integrations/libraries/snowflake/reference#authenticating-using-a-private-key) in the Snowflake reference guide. Step 1: Configure the Snowflake resource[​](https://docs.dagster.io/integrations/libraries/snowflake/using-snowflake-with-dagster#step-1-configure-the-snowflake-resource "Direct link to Step 1: Configure the Snowflake resource") ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- To connect to Snowflake, we'll use the `dagster-snowflake` [`SnowflakeResource`](https://docs.dagster.io/api/libraries/dagster-snowflake#dagster_snowflake.SnowflakeResource) . The [`SnowflakeResource`](https://docs.dagster.io/api/libraries/dagster-snowflake#dagster_snowflake.SnowflakeResource) requires some configuration: * **The `account` and `user` values are required.** * **One method of authentication is required**, either by using a password or a private key. * **Optional**: Using the `warehouse`, `schema`, and `role` attributes, you can specify where data should be stored and a `role` for the resource to use. from dagster_snowflake import SnowflakeResourcefrom snowflake.connector.pandas_tools import write_pandasfrom dagster import Definitions, EnvVar, MaterializeResult, assetsnowflake = SnowflakeResource( account=EnvVar("SNOWFLAKE_ACCOUNT"), # required user=EnvVar("SNOWFLAKE_USER"), # required password=EnvVar("SNOWFLAKE_PASSWORD"), # password or private key required warehouse="PLANTS", schema="IRIS", role="WRITER",) With this configuration, if you materialized an asset named `iris_dataset`, [`SnowflakeResource`](https://docs.dagster.io/api/libraries/dagster-snowflake#dagster_snowflake.SnowflakeResource) would use the role `WRITER` and store the data in the `FLOWERS.IRIS.IRIS_DATASET` table using the `PLANTS` warehouse. For more info about each of the configuration values, refer to the [`SnowflakeResource`](https://docs.dagster.io/api/libraries/dagster-snowflake#dagster_snowflake.SnowflakeResource) API documentation. Step 2: Create tables in Snowflake[​](https://docs.dagster.io/integrations/libraries/snowflake/using-snowflake-with-dagster#step-2-create-tables-in-snowflake "Direct link to Step 2: Create tables in Snowflake") ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- * Create tables in Snowflake from Dagster assets * Make existing tables available in Dagster Using the Snowflake resource, you can create Snowflake tables using the Snowflake Python API: import pandas as pdfrom dagster_snowflake import SnowflakeResourcefrom snowflake.connector.pandas_tools import write_pandasfrom dagster import MaterializeResult, asset@assetdef iris_dataset(snowflake: SnowflakeResource): iris_df = pd.read_csv( "https://docs.dagster.io/assets/iris.csv", names=[ "sepal_length_cm", "species", ], ) with snowflake.get_connection() as conn: table_name = "iris_dataset" database = "flowers" schema = "iris" success, number_chunks, rows_inserted, output = write_pandas( conn, iris_df, table_name=table_name, database=database, schema=schema, auto_create_table=True, overwrite=True, quote_identifiers=False, ) return MaterializeResult( metadata={"rows_inserted": rows_inserted}, ) In this example, we've defined an asset that fetches the Iris dataset as a Pandas DataFrame. Then, using the Snowflake resource, the DataFrame is stored in Snowflake as the `FLOWERS.IRIS.IRIS_DATASET` table. If you have existing tables in Snowflake and other assets defined in Dagster depend on those tables, you may want Dagster to be aware of those upstream dependencies. Making Dagster aware of these tables allows you to track the full data lineage in Dagster. You can accomplish this by defining [external assets](https://docs.dagster.io/guides/build/assets/external-assets) for these tables. For example: from dagster import AssetSpeciris_harvest_data = AssetSpec(key="iris_harvest_data") In this example, we created a [`AssetSpec`](https://docs.dagster.io/api/dagster/assets#dagster.AssetSpec) for a pre-existing table called `iris_harvest_data`. Since we supplied the database and the schema in the resource configuration in [Step 1](https://docs.dagster.io/integrations/libraries/snowflake/using-snowflake-with-dagster#step-1-configure-the-snowflake-resource) , we only need to provide the table name. We did this by using the `key` parameter in our [`AssetSpec`](https://docs.dagster.io/api/dagster/assets#dagster.AssetSpec) . When the `iris_harvest_data` asset needs to be loaded in a downstream asset, the data in the `FLOWERS.IRIS.IRIS_HARVEST_DATA` table will be selected and provided to the asset. Step 3: Define downstream assets[​](https://docs.dagster.io/integrations/libraries/snowflake/using-snowflake-with-dagster#step-3-define-downstream-assets "Direct link to Step 3: Define downstream assets") ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Once you've created an asset that represents a table in Snowflake, you may want to create additional assets that work with the data. In the following example, we've defined an asset that creates a second table, which contains only the data for the _Iris Setosa_ species: from dagster_snowflake import SnowflakeResourcefrom dagster import asset@asset(deps=["iris_dataset"])def iris_setosa(snowflake: SnowflakeResource) -> None: query = """ create or replace table iris.iris_setosa as ( SELECT * FROM iris.iris_dataset WHERE species = 'Iris-setosa' ); """ with snowflake.get_connection() as conn: conn.cursor.execute(query) # pyright: ignore[reportFunctionMemberAccess] To accomplish this, we defined a dependency on the `iris_dataset` asset using the `deps` parameter. Then, the SQL query runs and creates the table of _Iris Setosa_ data. Step 4: Definitions object[​](https://docs.dagster.io/integrations/libraries/snowflake/using-snowflake-with-dagster#step-4-definitions-object "Direct link to Step 4: Definitions object") ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- The last step is to add the [`SnowflakeResource`](https://docs.dagster.io/api/libraries/dagster-snowflake#dagster_snowflake.SnowflakeResource) and the assets to the project's [`Definitions`](https://docs.dagster.io/api/dagster/definitions#dagster.Definitions) object: from dagster import Definitionsdefs = Definitions( assets=[iris_dataset, iris_setosa], resources={"snowflake": snowflake}) This makes the resource and assets available to Dagster tools like the UI and CLI. Completed code example[​](https://docs.dagster.io/integrations/libraries/snowflake/using-snowflake-with-dagster#completed-code-example "Direct link to Completed code example") -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- When finished, your code should look like the following: import pandas as pdfrom dagster_snowflake import SnowflakeResourcefrom snowflake.connector.pandas_tools import write_pandasfrom dagster import Definitions, EnvVar, MaterializeResult, assetsnowflake = SnowflakeResource( account=EnvVar("SNOWFLAKE_ACCOUNT"), # required user=EnvVar("SNOWFLAKE_USER"), # required password=EnvVar("SNOWFLAKE_PASSWORD"), # password or private key required warehouse="PLANTS", schema="IRIS", role="WRITER",)@assetdef iris_dataset(snowflake: SnowflakeResource): iris_df = pd.read_csv( "https://docs.dagster.io/assets/iris.csv", names=[ "sepal_length_cm", "species", ], ) with snowflake.get_connection() as conn: table_name = "iris_dataset" database = "flowers" schema = "iris" success, number_chunks, rows_inserted, output = write_pandas( conn, iris_df, table_name=table_name, database=database, schema=schema, auto_create_table=True, overwrite=True, quote_identifiers=False, ) return MaterializeResult( metadata={"rows_inserted": rows_inserted}, )@asset(deps=["iris_dataset"])def iris_setosa(snowflake: SnowflakeResource) -> None: query = """ create or replace table iris.iris_setosa as ( SELECT * FROM iris.iris_dataset WHERE species = 'Iris-setosa' ); """ with snowflake.get_connection() as conn: conn.cursor.execute(query) # pyright: ignore[reportFunctionMemberAccess]defs = Definitions( assets=[iris_dataset, iris_setosa], resources={"snowflake": snowflake}) * [Prerequisites](https://docs.dagster.io/integrations/libraries/snowflake/using-snowflake-with-dagster#prerequisites) * [Step 1: Configure the Snowflake resource](https://docs.dagster.io/integrations/libraries/snowflake/using-snowflake-with-dagster#step-1-configure-the-snowflake-resource) * [Step 2: Create tables in Snowflake](https://docs.dagster.io/integrations/libraries/snowflake/using-snowflake-with-dagster#step-2-create-tables-in-snowflake) * [Step 3: Define downstream assets](https://docs.dagster.io/integrations/libraries/snowflake/using-snowflake-with-dagster#step-3-define-downstream-assets) * [Step 4: Definitions object](https://docs.dagster.io/integrations/libraries/snowflake/using-snowflake-with-dagster#step-4-definitions-object) * [Completed code example](https://docs.dagster.io/integrations/libraries/snowflake/using-snowflake-with-dagster#completed-code-example) --- # Dagster & Tableau | Dagster Docs [Skip to main content](https://docs.dagster.io/integrations/libraries/tableau#__docusaurus_skipToContent_fallback) On this page info This feature is considered in a beta stage. It is still being tested and may change. For more information, see the [API lifecycle stages documentation](https://docs.dagster.io/api/api-lifecycle/api-lifecycle-stages) . Your Tableau assets, such as data sources, sheets, and dashboards, can be represented in the Dagster asset graph, allowing you to track lineage and dependencies between Tableau assets and upstream data assets you are already modeling in Dagster. What you'll learn[​](https://docs.dagster.io/integrations/libraries/tableau#what-youll-learn "Direct link to What you'll learn") --------------------------------------------------------------------------------------------------------------------------------- * How to represent Tableau assets in the Dagster asset graph. * How to customize asset definition metadata for these Tableau assets. * How to refresh Tableau data sources. * How to materialize Tableau sheets and dashboards. Prerequisites * The `dagster-tableau` library installed in your environment * Familiarity with asset definitions and the Dagster asset graph * Familiarity with Dagster resources * Familiarity with Tableau concepts, like data sources, sheets, and dashboards * A Tableau site, either on Tableau Cloud or Tableau Server * A connected app configured to access Tableau. For more information, see [Use Tableau Connected Apps for Application Integration](https://help.tableau.com/current/online/en-us/connected_apps.htm) in the Tableau documentation. Set up your environment[​](https://docs.dagster.io/integrations/libraries/tableau#set-up-your-environment "Direct link to Set up your environment") ---------------------------------------------------------------------------------------------------------------------------------------------------- To get started, you'll need to install the `dagster` and `dagster-tableau` Python packages: * uv * pip uv add dagster-tableau pip install dagster-tableau Represent Tableau assets in the asset graph[​](https://docs.dagster.io/integrations/libraries/tableau#represent-tableau-assets-in-the-asset-graph "Direct link to Represent Tableau assets in the asset graph") ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- To load Tableau assets into the Dagster asset graph, you must first construct a Tableau resource, which allows Dagster to communicate with your Tableau workspace. The Tableau resource to create depends on your Tableau deployment type - use [`TableauCloudWorkspace`](https://docs.dagster.io/api/libraries/dagster-tableau#dagster_tableau.TableauCloudWorkspace) if you are using Tableau Cloud or [`TableauServerWorkspace`](https://docs.dagster.io/api/libraries/dagster-tableau#dagster_tableau.TableauServerWorkspace) if you are using Tableau Server. To connect to the Tableau workspace, you'll need to [configure a connected app with direct trust](https://help.tableau.com/current/online/en-gb/connected_apps_direct.htm) in Tableau, then supply your Tableau site information and connected app credentials to the resource. The Tableau resource uses the JSON Web Token (JWT) authentication to connect to the Tableau workspace. Dagster can automatically load all data sources, sheets, and dashboards from your Tableau workspace as asset specs. Call the [`load_tableau_asset_specs`](https://docs.dagster.io/api/libraries/dagster-tableau#dagster_tableau.load_tableau_asset_specs) function, which returns a list of [`AssetSpecs`](https://docs.dagster.io/api/dagster/assets#dagster.AssetSpec) representing your Tableau assets. You can then include these asset specs in your [`Definitions`](https://docs.dagster.io/api/dagster/definitions#dagster.Definitions) object: * Using Dagster with Tableau Cloud * Using Dagster with Tableau Server Use [`TableauCloudWorkspace`](https://docs.dagster.io/api/libraries/dagster-tableau#dagster_tableau.TableauCloudWorkspace) to interact with your Tableau Cloud workspace: from dagster_tableau import TableauCloudWorkspace, load_tableau_asset_specsimport dagster as dg# Connect to Tableau Cloud using the connected app credentialstableau_workspace = TableauCloudWorkspace( connected_app_client_id=dg.EnvVar("TABLEAU_CONNECTED_APP_CLIENT_ID"), connected_app_secret_id=dg.EnvVar("TABLEAU_CONNECTED_APP_SECRET_ID"), connected_app_secret_value=dg.EnvVar("TABLEAU_CONNECTED_APP_SECRET_VALUE"), username=dg.EnvVar("TABLEAU_USERNAME"), site_name=dg.EnvVar("TABLEAU_SITE_NAME"), pod_name=dg.EnvVar("TABLEAU_POD_NAME"),)tableau_specs = load_tableau_asset_specs(tableau_workspace)defs = dg.Definitions(assets=[*tableau_specs], resources={"tableau": tableau_workspace}) Use [`TableauServerWorkspace`](https://docs.dagster.io/api/libraries/dagster-tableau#dagster_tableau.TableauServerWorkspace) to interact with your Tableau Server workspace: from dagster_tableau import TableauServerWorkspace, load_tableau_asset_specsimport dagster as dg# Connect to Tableau Server using the connected app credentialstableau_workspace = TableauServerWorkspace( connected_app_client_id=dg.EnvVar("TABLEAU_CONNECTED_APP_CLIENT_ID"), connected_app_secret_id=dg.EnvVar("TABLEAU_CONNECTED_APP_SECRET_ID"), connected_app_secret_value=dg.EnvVar("TABLEAU_CONNECTED_APP_SECRET_VALUE"), username=dg.EnvVar("TABLEAU_USERNAME"), site_name=dg.EnvVar("TABLEAU_SITE_NAME"), server_name=dg.EnvVar("TABLEAU_SERVER_NAME"),)tableau_specs = load_tableau_asset_specs(tableau_workspace)defs = dg.Definitions(assets=[*tableau_specs], resources={"tableau": tableau_workspace}) ### Customize asset definition metadata for Tableau assets[​](https://docs.dagster.io/integrations/libraries/tableau#customize-asset-definition-metadata-for-tableau-assets "Direct link to Customize asset definition metadata for Tableau assets") By default, Dagster will generate asset specs for each Tableau asset based on its type, and populate default metadata. You can further customize asset properties by passing a custom [`DagsterTableauTranslator`](https://docs.dagster.io/api/libraries/dagster-tableau#dagster_tableau.DagsterTableauTranslator) subclass to the [`load_tableau_asset_specs`](https://docs.dagster.io/api/libraries/dagster-tableau#dagster_tableau.load_tableau_asset_specs) function. This subclass can implement methods to customize the asset specs for each Tableau asset type. from dagster_tableau import ( DagsterTableauTranslator, TableauCloudWorkspace, TableauContentType, TableauTranslatorData, load_tableau_asset_specs,)import dagster as dgtableau_workspace = TableauCloudWorkspace( connected_app_client_id=dg.EnvVar("TABLEAU_CONNECTED_APP_CLIENT_ID"), connected_app_secret_id=dg.EnvVar("TABLEAU_CONNECTED_APP_SECRET_ID"), connected_app_secret_value=dg.EnvVar("TABLEAU_CONNECTED_APP_SECRET_VALUE"), username=dg.EnvVar("TABLEAU_USERNAME"), site_name=dg.EnvVar("TABLEAU_SITE_NAME"), pod_name=dg.EnvVar("TABLEAU_POD_NAME"),)# A translator class lets us customize properties of the built# Tableau assets, such as the owners or asset keyclass MyCustomTableauTranslator(DagsterTableauTranslator): def get_asset_spec(self, data: TableauTranslatorData) -> dg.AssetSpec: # We create the default asset spec using super() default_spec = super().get_asset_spec(data) # We customize the metadata and asset key prefix for all assets, including sheets, # and we customize the team owner tag only for sheets. return default_spec.replace_attributes( key=default_spec.key.with_prefix("prefix"), metadata={**default_spec.metadata, "custom": "metadata"}, owners=( ["team:my_team"] if data.content_type == TableauContentType.SHEET else ... ), )tableau_specs = load_tableau_asset_specs( tableau_workspace, dagster_tableau_translator=MyCustomTableauTranslator(),)defs = dg.Definitions(assets=[*tableau_specs], resources={"tableau": tableau_workspace}) Note that `super()` is called in each of the overridden methods to generate the default asset spec. It is best practice to generate the default asset spec before customizing it. ### Load Tableau assets from multiple workspaces[​](https://docs.dagster.io/integrations/libraries/tableau#load-tableau-assets-from-multiple-workspaces "Direct link to Load Tableau assets from multiple workspaces") Definitions from multiple Tableau workspaces can be combined by instantiating multiple Tableau resources and merging their specs. This lets you view all your Tableau assets in a single asset graph: from dagster_tableau import TableauCloudWorkspace, load_tableau_asset_specsimport dagster as dgsales_team_workspace = TableauCloudWorkspace( connected_app_client_id=dg.EnvVar("SALES_TABLEAU_CONNECTED_APP_CLIENT_ID"), connected_app_secret_id=dg.EnvVar("SALES_TABLEAU_CONNECTED_APP_SECRET_ID"), connected_app_secret_value=dg.EnvVar("SALES_TABLEAU_CONNECTED_APP_SECRET_VALUE"), username=dg.EnvVar("TABLEAU_USERNAME"), site_name=dg.EnvVar("SALES_TABLEAU_SITE_NAME"), pod_name=dg.EnvVar("SALES_TABLEAU_POD_NAME"),)marketing_team_workspace = TableauCloudWorkspace( connected_app_client_id=dg.EnvVar("MARKETING_TABLEAU_CONNECTED_APP_CLIENT_ID"), connected_app_secret_id=dg.EnvVar("MARKETING_TABLEAU_CONNECTED_APP_SECRET_ID"), connected_app_secret_value=dg.EnvVar( "MARKETING_TABLEAU_CONNECTED_APP_SECRET_VALUE" ), username=dg.EnvVar("TABLEAU_USERNAME"), site_name=dg.EnvVar("MARKETING_TABLEAU_SITE_NAME"), pod_name=dg.EnvVar("MARKETING_TABLEAU_POD_NAME"),)sales_team_specs = load_tableau_asset_specs(sales_team_workspace)marketing_team_specs = load_tableau_asset_specs(marketing_team_workspace)defs = dg.Definitions( assets=[*sales_team_specs, *marketing_team_specs], resources={ "marketing_tableau": marketing_team_workspace, "sales_tableau": sales_team_workspace, },) from dagster\_tableau import TableauCloudWorkspace, load\_tableau\_asset\_specs ### Refresh and materialize Tableau assets[​](https://docs.dagster.io/integrations/libraries/tableau#refresh-and-materialize-tableau-assets "Direct link to Refresh and materialize Tableau assets") You can use Dagster to refresh Tableau data sources and materialize Tableau sheets and dashboards. from dagster_tableau import TableauCloudWorkspace, tableau_assetsimport dagster as dgtableau_workspace = TableauCloudWorkspace( connected_app_client_id=dg.EnvVar("TABLEAU_CONNECTED_APP_CLIENT_ID"), connected_app_secret_id=dg.EnvVar("TABLEAU_CONNECTED_APP_SECRET_ID"), connected_app_secret_value=dg.EnvVar("TABLEAU_CONNECTED_APP_SECRET_VALUE"), username=dg.EnvVar("TABLEAU_USERNAME"), site_name=dg.EnvVar("TABLEAU_SITE_NAME"), pod_name=dg.EnvVar("TABLEAU_POD_NAME"),)@tableau_assets( workspace=tableau_workspace, name="tableau_workspace_assets", group_name="tableau",)def tableau_workspace_assets( context: dg.AssetExecutionContext, tableau: TableauCloudWorkspace): yield from tableau.refresh_and_poll(context=context)defs = dg.Definitions( assets=[tableau_workspace_assets], resources={"tableau": tableau_workspace},) Note that only data sources created with extracts can be refreshed using this method. See more about [refreshing data sources](https://help.tableau.com/current/pro/desktop/en-us/refreshing_data.htm) in Tableau documentation website. ### Add a Data Quality Warning in Tableau using a sensor[​](https://docs.dagster.io/integrations/libraries/tableau#add-a-data-quality-warning-in-tableau-using-a-sensor "Direct link to Add a Data Quality Warning in Tableau using a sensor") When an upstream dependency of a Tableau asset fails to materialize or to pass the asset checks, it is possible to add a [Data Quality Warning](https://help.tableau.com/current/online/en-us/dm_dqw.htm) to the corresponding data source in Tableau. This can be achieved by leveraging the `add_data_quality_warning_to_data_source` in a sensor. from dagster_tableau import TableauCloudWorkspace, tableau_assetsimport dagster as dg# Connect to Tableau Cloud using the connected app credentialstableau_workspace = TableauCloudWorkspace( connected_app_client_id=dg.EnvVar("TABLEAU_CONNECTED_APP_CLIENT_ID"), connected_app_secret_id=dg.EnvVar("TABLEAU_CONNECTED_APP_SECRET_ID"), connected_app_secret_value=dg.EnvVar("TABLEAU_CONNECTED_APP_SECRET_VALUE"), username=dg.EnvVar("TABLEAU_USERNAME"), site_name=dg.EnvVar("TABLEAU_SITE_NAME"), pod_name=dg.EnvVar("TABLEAU_POD_NAME"),)@dg.asset( # Define which Tableau data source this upstream asset corresponds to metadata={"dagster/tableau_data_source_id": "f5660c7-2b05-4ff0-90ce-3199226956c6"})def upstream_asset(): ...@dg.run_failure_sensordef tableau_run_failure_sensor( context: dg.RunFailureSensorContext, tableau: TableauCloudWorkspace): asset_keys = context.dagster_run.asset_selection or set() for asset_key in asset_keys: data_source_id = upstream_asset.metadata_by_key.get(asset_key, {}).get( "dagster/tableau_data_source_id" ) if data_source_id: with tableau.get_client() as client: client.add_data_quality_warning_to_data_source( data_source_id=data_source_id, message=context.failure_event.message )@tableau_assets( workspace=tableau_workspace, name="tableau_workspace_assets", group_name="tableau",)def tableau_workspace_assets( context: dg.AssetExecutionContext, tableau: TableauCloudWorkspace): yield from tableau.refresh_and_poll(context=context)# Pass the sensor, Tableau resource, upstream asset, Tableau assets definition at oncedefs = dg.Definitions( assets=[ upstream_asset, tableau_workspace_assets, ], sensors=[tableau_run_failure_sensor], resources={"tableau": tableau_workspace},) ### Customize upstream dependencies[​](https://docs.dagster.io/integrations/libraries/tableau#customize-upstream-dependencies "Direct link to Customize upstream dependencies") By default, Dagster sets upstream dependencies when generating asset specs for your Tableau assets. To do so, Dagster parses information about assets that are upstream of specific Tableau sheets and dashboards from the Tableau workspace itself. You can customize how upstream dependencies are set on your Tableau assets by passing an instance of the custom [`DagsterTableauTranslator`](https://docs.dagster.io/api/libraries/dagster-tableau#dagster_tableau.DagsterTableauTranslator) to the [`load_tableau_asset_specs`](https://docs.dagster.io/api/libraries/dagster-tableau#dagster_tableau.load_tableau_asset_specs) function. The below example defines `my_upstream_asset` as an upstream dependency of `my_tableau_sheet`: class MyCustomTableauTranslator(DagsterTableauTranslator): def get_asset_spec(self, data: TableauTranslatorData) -> dg.AssetSpec: # We create the default asset spec using super() default_spec = super().get_asset_spec(data) # We customize upstream dependencies for the Tableau sheet named `my_tableau_sheet` return default_spec.replace_attributes( deps=["my_upstream_asset"] if data.content_type == TableauContentType.SHEET and data.properties.get("name") == "my_tableau_sheet" else ... )tableau_specs = load_tableau_asset_specs( tableau_workspace, dagster_tableau_translator=MyCustomTableauTranslator(),) Note that `super()` is called in each of the overridden methods to generate the default asset spec. It is best practice to generate the default asset spec before customizing it. ### Related[​](https://docs.dagster.io/integrations/libraries/tableau#related "Direct link to Related") * [`dagster-tableau` API reference](https://docs.dagster.io/api/libraries/dagster-tableau) * [Asset definitions](https://docs.dagster.io/guides/build/assets) * [Resources](https://docs.dagster.io/guides/build/external-resources) * [Using environment variables and secrets](https://docs.dagster.io/guides/operate/configuration/using-environment-variables-and-secrets) * [What you'll learn](https://docs.dagster.io/integrations/libraries/tableau#what-youll-learn) * [Set up your environment](https://docs.dagster.io/integrations/libraries/tableau#set-up-your-environment) * [Represent Tableau assets in the asset graph](https://docs.dagster.io/integrations/libraries/tableau#represent-tableau-assets-in-the-asset-graph) * [Customize asset definition metadata for Tableau assets](https://docs.dagster.io/integrations/libraries/tableau#customize-asset-definition-metadata-for-tableau-assets) * [Load Tableau assets from multiple workspaces](https://docs.dagster.io/integrations/libraries/tableau#load-tableau-assets-from-multiple-workspaces) * [Refresh and materialize Tableau assets](https://docs.dagster.io/integrations/libraries/tableau#refresh-and-materialize-tableau-assets) * [Add a Data Quality Warning in Tableau using a sensor](https://docs.dagster.io/integrations/libraries/tableau#add-a-data-quality-warning-in-tableau-using-a-sensor) * [Customize upstream dependencies](https://docs.dagster.io/integrations/libraries/tableau#customize-upstream-dependencies) * [Related](https://docs.dagster.io/integrations/libraries/tableau#related) --- # Dagster & Spark | Dagster Docs [Skip to main content](https://docs.dagster.io/integrations/libraries/spark#__docusaurus_skipToContent_fallback) On this page Running Spark code often requires submitting code to a Databricks or EMR cluster. The Pyspark integration provides a Spark class with methods for configuration and constructing the spark-submit command for a Spark job. This page is focused on using Pipes with specific Spark providers, such as AWS EMR or Databricks. Our guide about [Building pipelines with PySpark](https://docs.dagster.io/guides/build/external-pipelines/pyspark-pipeline) provides more information on using Dagster Pipes to launch & monitor general PySpark jobs. About Apache Spark[​](https://docs.dagster.io/integrations/libraries/spark#about-apache-spark "Direct link to About Apache Spark") ----------------------------------------------------------------------------------------------------------------------------------- **Apache Spark** is an open source unified analytics engine for large-scale data processing. Spark provides an interface for programming clusters with implicit data parallelism and fault tolerance. It also provides libraries for graph computation, SQL for structured data processing, ML, and data science. Using Dagster Pipes to run Spark jobs[​](https://docs.dagster.io/integrations/libraries/spark#using-dagster-pipes-to-run-spark-jobs "Direct link to Using Dagster Pipes to run Spark jobs") -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- [Dagster pipes](https://docs.dagster.io/guides/build/external-pipelines) is our toolkit for orchestrating remote compute from Dagster. It allows you to run code outside of the Dagster process, and stream logs and events back to Dagster. This is the recommended approach for running Spark jobs. With Pipes, the code inside the asset or op definition submits a Spark job to an external system like Databricks or AWS EMR, usually pointing to a jar or zip of Python files that contain the actual Spark data transformations and actions. You can either use one of the available Pipes Clients or make your own. The available Pipes Clients for popular Spark providers are: * [Databricks](https://docs.dagster.io/guides/build/external-pipelines/databricks-pipeline) * [AWS Glue](https://docs.dagster.io/guides/build/external-pipelines/aws/aws-glue-pipeline) * [AWS EMR](https://docs.dagster.io/guides/build/external-pipelines/aws/aws-emr-pipeline) * [AWS EMR on EKS](https://docs.dagster.io/guides/build/external-pipelines/aws/aws-emr-containers-pipeline) * [AWS EMR Serverless](https://docs.dagster.io/guides/build/external-pipelines/aws/aws-emr-serverless-pipeline) Existing Spark jobs can be used with Pipes without any modifications. In this case, Dagster will be receiving logs from the job, but not events like asset checks or attached metadata. Additionally, it's possible to send events to Dagster from the job by utilizing the `dagster_pipes` module. This requires minimal code changes on the job side. This approach also works for Spark jobs written in Java or Scala, although we don't have Pipes implementations for emitting events from those languages yet. * [About Apache Spark](https://docs.dagster.io/integrations/libraries/spark#about-apache-spark) * [Using Dagster Pipes to run Spark jobs](https://docs.dagster.io/integrations/libraries/spark#using-dagster-pipes-to-run-spark-jobs) --- # Tags | Dagster Docs [Skip to main content](https://docs.dagster.io/tags#__docusaurus_skipToContent_fallback) Tags ==== A[​](https://docs.dagster.io/tags#A "Direct link to A") -------------------------------------------------------- * [AI2](https://docs.dagster.io/tags/integrations/ai "AI integrations.") * [alerting4](https://docs.dagster.io/tags/integrations/alerting "Alerting integrations.") * * * B[​](https://docs.dagster.io/tags#B "Direct link to B") -------------------------------------------------------- * [BI5](https://docs.dagster.io/tags/integrations/bi "BI integrations.") * * * C[​](https://docs.dagster.io/tags#C "Direct link to C") -------------------------------------------------------- * [code-example5](https://docs.dagster.io/tags/examples/code-example "Code example.") * [Community supported32](https://docs.dagster.io/tags/integrations/community-supported "Community-supported integrations.") * [compute14](https://docs.dagster.io/tags/integrations/compute "Compute integrations.") * * * D[​](https://docs.dagster.io/tags#D "Direct link to D") -------------------------------------------------------- * [Dagster supported48](https://docs.dagster.io/tags/integrations/dagster-supported "Dagster-supported integrations.") * * * E[​](https://docs.dagster.io/tags#E "Direct link to E") -------------------------------------------------------- * [ETL15](https://docs.dagster.io/tags/integrations/etl "ETL integrations.") * * * M[​](https://docs.dagster.io/tags#M "Direct link to M") -------------------------------------------------------- * [metadata6](https://docs.dagster.io/tags/integrations/metadata "Metadata integrations.") * [monitoring3](https://docs.dagster.io/tags/integrations/monitoring "Monitoring integrations.") * * * O[​](https://docs.dagster.io/tags#O "Direct link to O") -------------------------------------------------------- * [other1](https://docs.dagster.io/tags/integrations/other "Other integrations") * * * R[​](https://docs.dagster.io/tags#R "Direct link to R") -------------------------------------------------------- * [reference-architecture4](https://docs.dagster.io/tags/examples/reference-architecture "Reference architecture.") * * * S[​](https://docs.dagster.io/tags#S "Direct link to S") -------------------------------------------------------- * [storage16](https://docs.dagster.io/tags/integrations/storage "Storage integrations.") * * * --- # 5 docs tagged with "BI" | Dagster Docs [Skip to main content](https://docs.dagster.io/tags/integrations/bi#__docusaurus_skipToContent_fallback) [Dagster & Evidence\ ------------------](https://docs.dagster.io/integrations/libraries/evidence) The Evidence library offers a component to easily generate dashboards from your Evidence project. [Dagster & Looker\ ----------------](https://docs.dagster.io/integrations/libraries/looker) The Looker integration allows you to monitor your Looker project as assets in Dagster, along with other data assets. [Dagster & Power BI\ ------------------](https://docs.dagster.io/integrations/libraries/powerbi) Your Power BI assets, such as semantic models, data sources, reports, and dashboards, can be represented in the Dagster asset graph, allowing you to track lineage and dependencies between Power BI assets and upstream data assets you are already modeling in Dagster. You can also use Dagster to orchestrate Power BI semantic models, allowing you to trigger refreshes of these models on a cadence or based on upstream data changes. [Dagster & Sigma\ ---------------](https://docs.dagster.io/integrations/libraries/sigma) Your Sigma assets, including datasets and workbooks, can be represented in the Dagster asset graph, allowing you to track lineage and dependencies between Sigma assets and upstream data assets you are already modeling in Dagster. [Dagster & Tableau\ -----------------](https://docs.dagster.io/integrations/libraries/tableau) Your Tableau assets, such as data sources, sheets, and dashboards, can be represented in the Dagster asset graph, allowing you to track lineage and dependencies between Tableau assets and upstream data assets you are already modeling in Dagster. --- # 4 docs tagged with "alerting" | Dagster Docs [Skip to main content](https://docs.dagster.io/tags/integrations/alerting#__docusaurus_skipToContent_fallback) [Dagster & Microsoft Teams\ -------------------------](https://docs.dagster.io/integrations/libraries/microsoft-teams) An integration with Microsoft Teams to post messages to MS Teams from any Dagster op or asset. [Dagster & PagerDuty\ -------------------](https://docs.dagster.io/integrations/libraries/pagerduty) This library provides an integration between Dagster and PagerDuty to support creating alerts from your Dagster code. [Dagster & Slack\ ---------------](https://docs.dagster.io/integrations/libraries/slack) This library provides an integration with Slack to support posting messages in your company's Slack workspace. [Dagster & Twilio\ ----------------](https://docs.dagster.io/integrations/libraries/twilio) Use your Twilio Account SID and Auth Token to build Twilio tasks right into your Dagster pipeline. --- # 6 docs tagged with "metadata" | Dagster Docs [Skip to main content](https://docs.dagster.io/tags/integrations/metadata#__docusaurus_skipToContent_fallback) [Dagster & Open Metadata\ -----------------------](https://docs.dagster.io/integrations/libraries/open-metadata) With this integration you can create a Open Metadata service to ingest metadata produced by the Dagster application. View the Ingestion Pipeline running from the Open Metadata Service Page. [Dagster & Pandas\ ----------------](https://docs.dagster.io/integrations/libraries/pandas) Implement validation on pandas DataFrames. [Dagster & Pandera\ -----------------](https://docs.dagster.io/integrations/libraries/pandera) The Pandera integration library provides an API for generating Dagster Types from Pandera dataframe schemas. Like all Dagster types, Pandera-generated types can be used to annotate op inputs and outputs. [Dagster & Patito\ ----------------](https://docs.dagster.io/integrations/libraries/patito) Patito is a data validation framework for Polars, based on Pydantic. [Dagster & Polars\ ----------------](https://docs.dagster.io/integrations/libraries/polars) The Polars integration allows using Polars eager or lazy DataFrames as inputs and outputs with Dagster’s assets and ops. Type annotations are used to control whether to load an eager or lazy DataFrame. Lazy DataFrames can be sinked as output. Multiple serialization formats (Parquet, Delta Lake, BigQuery) and filesystems (local, S3, GCS, …) are supported. [Dagster & Secoda\ ----------------](https://docs.dagster.io/integrations/libraries/secoda) Connect Dagster to Secoda and see metadata related to your Dagster assets, asset groups and jobs right in Secoda. Simplify your team's access, and remove the need to switch between tools. --- # 2 docs tagged with "AI" | Dagster Docs [Skip to main content](https://docs.dagster.io/tags/integrations/ai#__docusaurus_skipToContent_fallback) [Dagster & Gemini\ ----------------](https://docs.dagster.io/integrations/libraries/gemini) The Gemini library allows you to easily interact with the Gemini REST API using the Gemini Python API to build AI steps into your Dagster pipelines. You can also log Gemini API usage metadata in Dagster Insights, giving you detailed observability on API call credit consumption. [Dagster & Not Diamond\ ---------------------](https://docs.dagster.io/integrations/libraries/notdiamond) Leverage the Not Diamond resource to easily determine which LLM provider is most appropriate for your use case. ---