# Table of Contents - [OpenMetadata Documentation: Get Help Instantly](#openmetadata-documentation-get-help-instantly) - [OpenMetadata Documentation: Get Help Instantly](#openmetadata-documentation-get-help-instantly) - [Metadata Ingestion Best Practices](#metadata-ingestion-best-practices) - [Metadata Versioning | OpenMetadata Data History Management](#metadata-versioning-openmetadata-data-history-management) - [Lineage Workflow | OpenMetadata Data Lineage Guide](#lineage-workflow-openmetadata-data-lineage-guide) - [Database Filter Patterns | Official Documentation](#database-filter-patterns-official-documentation) - [Edit Data Lineage Manually](#edit-data-lineage-manually) - [Ingestion Pipeline UI Deployment](#ingestion-pipeline-ui-deployment) - [Metadata Ingestion Workflow | Official Documentation](#metadata-ingestion-workflow-official-documentation) - [Lineage Workflow Through Query Logs | Official Documentation](#lineage-workflow-through-query-logs-official-documentation) - [Usage Workflow Through Query Logs](#usage-workflow-through-query-logs) - [Kubernetes Deployment | Official Documentation](#kubernetes-deployment-official-documentation) - [Metadata Ingestion | OpenMetadata Data Pipeline Overview](#metadata-ingestion-openmetadata-data-pipeline-overview) - [High Level Design | OpenMetadata Architecture Overview](#high-level-design-openmetadata-architecture-overview) - [dbt Workflow | OpenMetadata Data Build Tool Integration](#dbt-workflow-openmetadata-data-build-tool-integration) - [Kubernetes On Premises Deployment | Official Documentation](#kubernetes-on-premises-deployment-official-documentation) - [Kubernetes Helm Values | Official Documentation](#kubernetes-helm-values-official-documentation) - [Azure AKS Deployment | Official Documentation](#azure-aks-deployment-official-documentation) - [AWS EKS Deployment | OpenMetadata Kubernetes Guide](#aws-eks-deployment-openmetadata-kubernetes-guide) - [Kubernetes GKE Deployment | Official Documentation](#kubernetes-gke-deployment-official-documentation) - [Ingestion Workflows | OpenMetadata Pipeline Orchestration](#ingestion-workflows-openmetadata-pipeline-orchestration) - [Lineage Ingestion | OpenMetadata Data Lineage Setup Guide](#lineage-ingestion-openmetadata-data-lineage-setup-guide) - [Usage Workflow Guide | OpenMetadata Ingestion Workflows](#usage-workflow-guide-openmetadata-ingestion-workflows) - [Ingestion Framework Deployment | Official Documentation](#ingestion-framework-deployment-official-documentation) - [JWT validation Troubleshooting | Official Documentation](#jwt-validation-troubleshooting-official-documentation) - [Enable SSL in Airflow | OpenMetadata Security Guide](#enable-ssl-in-airflow-openmetadata-security-guide) - [API Services | OpenMetadata Connector Integration Guide](#api-services-openmetadata-connector-integration-guide) - [Nifi Connector Troubleshooting | Official Documentation](#nifi-connector-troubleshooting-official-documentation) - [Exasol Troubleshooting Guide | OpenMetadata Support](#exasol-troubleshooting-guide-openmetadata-support) - [Kinesis Connector Troubleshooting Guide | OpenMetadata Support](#kinesis-connector-troubleshooting-guide-openmetadata-support) - [Lightdash Troubleshooting Guide | OpenMetadata Support](#lightdash-troubleshooting-guide-openmetadata-support) - [Dagster Connector Troubleshooting Guide | OpenMetadata Support](#dagster-connector-troubleshooting-guide-openmetadata-support) - [Fivetran Troubleshooting Guide | OpenMetadata Support](#fivetran-troubleshooting-guide-openmetadata-support) - [REST Connector Troubleshooting Guide | OpenMetadata Support](#rest-connector-troubleshooting-guide-openmetadata-support) - [Mode Dashboard Troubleshooting Guide | OpenMetadata Support](#mode-dashboard-troubleshooting-guide-openmetadata-support) - [MongoDB Troubleshooting Guide | OpenMetadata Support](#mongodb-troubleshooting-guide-openmetadata-support) - [SAP ERP Troubleshooting Guide | OpenMetadata Support](#sap-erp-troubleshooting-guide-openmetadata-support) - [Redash Troubleshooting Guide | OpenMetadata Support](#redash-troubleshooting-guide-openmetadata-support) - [DeltaLake Troubleshooting Guide | OpenMetadata Support](#deltalake-troubleshooting-guide-openmetadata-support) - [MicroStrategy Troubleshooting Guide | OpenMetadata Support](#microstrategy-troubleshooting-guide-openmetadata-support) - [Databricks Pipeline Troubleshooting Guide | OpenMetadata Support](#databricks-pipeline-troubleshooting-guide-openmetadata-support) - [Oracle Troubleshooting Guide | OpenMetadata Support](#oracle-troubleshooting-guide-openmetadata-support) - [Run the Exasol Connector Externally](#run-the-exasol-connector-externally) - [Spline Connector Troubleshooting Guide | OpenMetadata Support](#spline-connector-troubleshooting-guide-openmetadata-support) - [Superset Troubleshooting Guide | OpenMetadata Support](#superset-troubleshooting-guide-openmetadata-support) - [Profiler Workflow | OpenMetadata Profiling Workflow](#profiler-workflow-openmetadata-profiling-workflow) - [Run the Dagster Connector Externally](#run-the-dagster-connector-externally) - [Run the Redash Connector Externally](#run-the-redash-connector-externally) - [Run the Superset Connector Externally](#run-the-superset-connector-externally) - [Run the Delta Lake Connector Externally](#run-the-delta-lake-connector-externally) - [Great Expectations | OpenMetadata Data Quality Integration](#great-expectations-openmetadata-data-quality-integration) - [SAP HANA Troubleshooting Guide | OpenMetadata Support](#sap-hana-troubleshooting-guide-openmetadata-support) - [Run the Databricks Pipeline Connector Externally](#run-the-databricks-pipeline-connector-externally) - [Run the Kinesis Connector Externally](#run-the-kinesis-connector-externally) - [Run the Grafana Connector Externally](#run-the-grafana-connector-externally) - [Domo Pipeline | OpenMetadata Data Pipeline Services](#domo-pipeline-openmetadata-data-pipeline-services) - [Run the Spline Connector Externally | Official Documentation](#run-the-spline-connector-externally-official-documentation) - [Run the MongoDB Connector Externally](#run-the-mongodb-connector-externally) - [Run the SAP ERP Connector Externally](#run-the-sap-erp-connector-externally) - [Druid Troubleshooting Guide | OpenMetadata Support](#druid-troubleshooting-guide-openmetadata-support) - [How to Deploy a Lineage Workflow](#how-to-deploy-a-lineage-workflow) - [Run the Ingestion Framework Externally](#run-the-ingestion-framework-externally) - [PinotDB Troubleshooting Guide | OpenMetadata Support](#pinotdb-troubleshooting-guide-openmetadata-support) - [GCS Connector Troubleshooting Guide | OpenMetadata Support](#gcs-connector-troubleshooting-guide-openmetadata-support) - [Redpanda Connector Troubleshooting Guide | OpenMetadata Support](#redpanda-connector-troubleshooting-guide-openmetadata-support) - [Setup SAP ERP APIs | OpenMetadata Integration Guide](#setup-sap-erp-apis-openmetadata-integration-guide) - [Run the OpenAPI/REST Connector Externally](#run-the-openapi-rest-connector-externally) - [Configure Data Quality | Official Documentation](#configure-data-quality-official-documentation) - [Run the GCS Connector Externally](#run-the-gcs-connector-externally) - [Run the Fivetran Connector Externally](#run-the-fivetran-connector-externally) - [Google SSO Configuration Guide | Public Client Setup](#google-sso-configuration-guide-public-client-setup) - [Run the Redpanda Connector Externally](#run-the-redpanda-connector-externally) - [Run the Lightdash Connector Externally](#run-the-lightdash-connector-externally) - [Run the MicroStrategy Connector Externally](#run-the-microstrategy-connector-externally) - [Run the Mode Connector Externally](#run-the-mode-connector-externally) - [Run the Nifi Connector Externally | Official Documentation](#run-the-nifi-connector-externally-official-documentation) - [Azure SSO for Docker | OpenMetadata Deployment Guide](#azure-sso-for-docker-openmetadata-deployment-guide) - [Glue Pipeline | OpenMetadata Data Integration Pipeline](#glue-pipeline-openmetadata-data-integration-pipeline) - [Run the Oracle Connector Externally](#run-the-oracle-connector-externally) - [Okta SSO | OpenMetadata Authentication Integration](#okta-sso-openmetadata-authentication-integration) - [Run the PinotDB Connector Externally](#run-the-pinotdb-connector-externally) - [Run the Athena Connector Externally](#run-the-athena-connector-externally) - [Run the SAP HANA Connector Externally](#run-the-sap-hana-connector-externally) - [Domo Dashboard | OpenMetadata Connector Setup Guide](#domo-dashboard-openmetadata-connector-setup-guide) - [Docker Deployment | OpenMetadata Container Setup](#docker-deployment-openmetadata-container-setup) - [AWS Cognito SSO Setup Guide for Public Apps](#aws-cognito-sso-setup-guide-for-public-apps) - [Implicit flow of Keyclock | Official Documentation](#implicit-flow-of-keyclock-official-documentation) - [Enable JWT Tokens | OpenMetadata Security Features](#enable-jwt-tokens-openmetadata-security-features) - [Azure SSO Setup Guide for Public Apps](#azure-sso-setup-guide-for-public-apps) - [Custom OIDC SSO Configuration | OpenMetadata](#custom-oidc-sso-configuration-openmetadata) - [Enable Secrets Manager | Official Documentation](#enable-secrets-manager-official-documentation) - [Advanced Guide for Roles and Policies](#advanced-guide-for-roles-and-policies) - [Metadata Services | OpenMetadata Integration Overview](#metadata-services-openmetadata-integration-overview) - [OIDC Based Authentication](#oidc-based-authentication) - [LDAP SSO Configuration | OpenMetadata](#ldap-sso-configuration-openmetadata) - [Azure SSO Configuration for Confidential Apps](#azure-sso-configuration-for-confidential-apps) - [SAML SSO](#saml-sso) - [Bare Metal Deployment | Official Documentation](#bare-metal-deployment-official-documentation) - [Secrets Manager | OpenMetadata Deployment Integration](#secrets-manager-openmetadata-deployment-integration) - [Dashboard Services | Connect BI Tools with OpenMetadata](#dashboard-services-connect-bi-tools-with-openmetadata) - [Pipeline Services | OpenMetadata Data Pipeline Guide](#pipeline-services-openmetadata-data-pipeline-guide) - [Enable SSL with Nginx | OpenMetadata Security Setup](#enable-ssl-with-nginx-openmetadata-security-setup) - [Run the ingestion from GCP Composer | Official Documentation](#run-the-ingestion-from-gcp-composer-official-documentation) - [Data Quality Tab | OpenMetadata Quality Interface](#data-quality-tab-openmetadata-quality-interface) - [OpenMetadata System Architecture | Developer Guide](#openmetadata-system-architecture-developer-guide) - [Enable SSL at the OpenMetadata Server](#enable-ssl-at-the-openmetadata-server) - [Data Quality as Code](#data-quality-as-code) - [Explore the Lineage View | Official Documentation](#explore-the-lineage-view-official-documentation) - [Domo Dashboard Troubleshooting Guide | OpenMetadata Support](#domo-dashboard-troubleshooting-guide-openmetadata-support) - [Glue Pipeline Troubleshooting Guide | OpenMetadata Support](#glue-pipeline-troubleshooting-guide-openmetadata-support) - [How to Write and Deploy No-Code Test Cases](#how-to-write-and-deploy-no-code-test-cases) - [Run the ingestion from GitHub Actions](#run-the-ingestion-from-github-actions) - [Custom Tests | OpenMetadata Quality Testing Guide](#custom-tests-openmetadata-quality-testing-guide) - [Configuring Okta Public Authentication | OpenMetadata SSO Setup Guide](#configuring-okta-public-authentication-openmetadata-sso-setup-guide) - [SSO for Docker | Official Documentation](#sso-for-docker-official-documentation) - [Run the ingestion from AWS MWAA | Official Documentation](#run-the-ingestion-from-aws-mwaa-official-documentation) - [SSO for Kubernetes | Official Documentation](#sso-for-kubernetes-official-documentation) - [Data Lineage | OpenMetadata Lineage How-To Guide](#data-lineage-openmetadata-lineage-how-to-guide) - [Auth0 SSO Configuration for Confidential Apps](#auth0-sso-configuration-for-confidential-apps) - [Run the Domo Pipeline Connector Externally](#run-the-domo-pipeline-connector-externally) - [Run the Glue Pipeline Connector Externally](#run-the-glue-pipeline-connector-externally) - [Okta SSO Configuration Guide | Confidential Client Setup](#okta-sso-configuration-guide-confidential-client-setup) - [Run the ingestion from your Airflow](#run-the-ingestion-from-your-airflow) - [Tests - UI Config | OpenMetadata Quality Config Guide](#tests-ui-config-openmetadata-quality-config-guide) - [Dimensional Validation | Data Quality Testing by Dimension](#dimensional-validation-data-quality-testing-by-dimension) - [Enable Security (Docker) | OpenMetadata Docker Security](#enable-security-docker-openmetadata-docker-security) - [Data Quality Observability Guide | Official Documentation](#data-quality-observability-guide-official-documentation) - [Data Profiler | OpenMetadata Data Profiling Guide](#data-profiler-openmetadata-data-profiling-guide) - [How to Manually Add or Edit Lineage](#how-to-manually-add-or-edit-lineage) - [Try OpenMetadata in Docker](#try-openmetadata-in-docker) - [SSO for Bare Metal | Official Documentation](#sso-for-bare-metal-official-documentation) - [Domo-Pipeline Troubleshooting Guide | OpenMetadata Support](#domo-pipeline-troubleshooting-guide-openmetadata-support) - [Tests - YAML Config | OpenMetadata Quality Config Guide](#tests-yaml-config-openmetadata-quality-config-guide) - [SAML AZURE SSO](#saml-azure-sso) - [GCP Secret Manager Parameter Store | Official Documentation](#gcp-secret-manager-parameter-store-official-documentation) - [Azure Key Vault | OpenMetadata Secrets Manager Guide](#azure-key-vault-openmetadata-secrets-manager-guide) - [Server Configuration Reference | Official Documentation](#server-configuration-reference-official-documentation) - [Enable Security | OpenMetadata Deployment Security Guide](#enable-security-openmetadata-deployment-security-guide) - [KafkaConnect | OpenMetadata Messaging Pipeline Connector](#kafkaconnect-openmetadata-messaging-pipeline-connector) - [How Column-Level Lineage Works | Official Documentation](#how-column-level-lineage-works-official-documentation) - [SSO | OpenMetadata Security Integration](#sso-openmetadata-security-integration) - [Metrics | OpenMetadata Profiler Metrics Guide](#metrics-openmetadata-profiler-metrics-guide) - [Incident Manager | OpenMetadata Data Quality Management](#incident-manager-openmetadata-data-quality-management) - [Adding test suites through the UI](#adding-test-suites-through-the-ui) - [Profiler Tab | OpenMetadata Data Profiling Interface](#profiler-tab-openmetadata-data-profiling-interface) - [Adding Test Cases to an Entity](#adding-test-cases-to-an-entity) - [External Profiler Workflow | Official Documentation](#external-profiler-workflow-official-documentation) - [Alerts & Notifications | OpenMetadata Guide](#alerts-notifications-openmetadata-guide) - [Data Observability Alerts | OpenMetadata](#data-observability-alerts-openmetadata) - [System & Governance Notifications | OpenMetadata](#system-governance-notifications-openmetadata) - [How to work with the Incident Manager | Official Documentation](#how-to-work-with-the-incident-manager-official-documentation) - [Microsoft Teams Alerts Configuration | OpenMetadata](#microsoft-teams-alerts-configuration-openmetadata) - [Root Cause Analysis | OpenMetadata Incident Management](#root-cause-analysis-openmetadata-incident-management) - [Email Alerts Configuration | OpenMetadata](#email-alerts-configuration-openmetadata) - [Google Chat Alerts Configuration | OpenMetadata](#google-chat-alerts-configuration-openmetadata) - [Run the KafkaConnect Connector Externally](#run-the-kafkaconnect-connector-externally) - [Slack Alerts Configuration | OpenMetadata](#slack-alerts-configuration-openmetadata) - [Generic Webhook Alert Configuration | OpenMetadata](#generic-webhook-alert-configuration-openmetadata) - [OpenMetadata Documentation: Get Help Instantly](#openmetadata-documentation-get-help-instantly) - [OpenMetadata Documentation: Get Help Instantly](#openmetadata-documentation-get-help-instantly) - [Configuring OpenMetadata to Run Under a Subpath](#configuring-openmetadata-to-run-under-a-subpath) - [Minimum Requirements | Official Documentation](#minimum-requirements-official-documentation) - [Database Connection Pooling](#database-connection-pooling) - [Production-Ready Requirements for OpenMetadata Deployment](#production-ready-requirements-for-openmetadata-deployment) - [How to enable Azure Auth](#how-to-enable-azure-auth) - [Azure - Enable Passwordless Database Backend Connection](#azure-enable-passwordless-database-backend-connection) - [How to Visualize Test Results](#how-to-visualize-test-results) - [Metrics](#metrics) - [How to enable AWS RDS IAM Auth | Official Documentation](#how-to-enable-aws-rds-iam-auth-official-documentation) - [OSS Security Best Practices](#oss-security-best-practices) - [Adding Data Quality Test Cases from yaml config](#adding-data-quality-test-cases-from-yaml-config) --- # OpenMetadata Documentation: Get Help Instantly We use cookies to improve site navigation, analyze site usage, and enhance your user experience. Click "Accept" to enable cookies or "Reject" to reject cookies. AcceptReject ![](https://static.scarf.sh/a.png?x-pxid=87848f39-08ad-4fee-a3d6-ab05b717d351) [Home](https://docs.open-metadata.org/latest) [Quickstart](https://docs.open-metadata.org/latest/quick-start) [Deployment](https://docs.open-metadata.org/latest/deployment) [Connectors](https://docs.open-metadata.org/latest/connectors) [How-to Guides](https://docs.open-metadata.org/latest/how-to-guides) [Releases](https://docs.open-metadata.org/latest/releases) [Main Concepts](https://docs.open-metadata.org/latest/main-concepts) [Developers](https://docs.open-metadata.org/latest/developers) [SDK](https://docs.open-metadata.org/latest/sdk) OpenMetadata Documentation ========================== Unlock the value of data assets with an end-to-end metadata management solution that includes data discovery, governance, data quality, observability, and people collaboration. Quick Start Install OpenMetadata to explore its full potential. [Get Started](https://docs.open-metadata.org/latest/quick-start) How-to Guides Get a complete overview of the features in OpenMetadata from our How-to Guides [Explore Features](https://docs.open-metadata.org/latest/how-to-guides) Join the OSS Community Connect with 1000s of OpenMetadata users. Get support for all your questions from data experts. [Join Slack Now!](https://slack.open-metadata.org/) Overview OpenMetadata is a unified platform for discovery, observability, and governance powered by a central metadata repository, in-depth lineage, and seamless team collaboration. It is one of the fastest-growing open-source projects with a vibrant community and adoption by a diverse set of companies in a variety of industry verticals. Based on Open Metadata Standards and APIs, supporting connectors to a wide range of data services, OpenMetadata enables end-to-end metadata management, giving you the freedom to unlock the value of your data assets. Quick Links [Deployment\ \ Deploy in Bare Metal, Docker or Kubernetes on any cloud πŸŽ‰](https://docs.open-metadata.org/latest/deployment) [SaaS\ \ Enjoy 100% of OpenMetadata with 0% of the hassle πŸš€](https://getcollate.io/) [Knowledge Base\ \ Check out some frequent Questions & Answers πŸ’¬](https://github.com/open-metadata/OpenMetadata/discussions/categories/q-a) Connectors All connectorsAPIDatabaseDriveMessagingDashboardPipelineMlmodelSearchStorageMetadataSecurity [![ADLS Datalake-icon](https://docs.open-metadata.org/_next/image?url=%2Fimages%2Fconnectors%2Fadls.webp&w=64&q=75)\ \ ADLS Datalake](https://docs.open-metadata.org/latest/connectors/database/adls-datalake) [![Airbyte-icon](https://docs.open-metadata.org/_next/image?url=%2Fimages%2Fconnectors%2Fairbyte.webp&w=64&q=75)\ \ Airbyte](https://docs.open-metadata.org/latest/connectors/pipeline/airbyte) [![Airflow-icon](https://docs.open-metadata.org/_next/image?url=%2Fimages%2Fconnectors%2Fairflow.webp&w=64&q=75)\ \ Airflow](https://docs.open-metadata.org/latest/connectors/pipeline/airflow) [![AlationSink-icon](https://docs.open-metadata.org/_next/image?url=%2Fimages%2Fconnectors%2Falation.webp&w=64&q=75)\ \ AlationSink](https://docs.open-metadata.org/latest/connectors/metadata/alationsink) [![Athena-icon](https://docs.open-metadata.org/_next/image?url=%2Fimages%2Fconnectors%2Fathena.webp&w=64&q=75)\ \ Athena](https://docs.open-metadata.org/latest/connectors/database/athena) [![Atlas-icon](https://docs.open-metadata.org/_next/image?url=%2Fimages%2Fconnectors%2Fatlas.webp&w=64&q=75)\ \ Atlas](https://docs.open-metadata.org/latest/connectors/metadata/atlas) [![Azure SQL-icon](https://docs.open-metadata.org/_next/image?url=%2Fimages%2Fconnectors%2Fazuresql.webp&w=64&q=75)\ \ Azure SQL](https://docs.open-metadata.org/latest/connectors/database/azuresql) [![BigQuery-icon](https://docs.open-metadata.org/_next/image?url=%2Fimages%2Fconnectors%2Fbigquery.webp&w=64&q=75)\ \ BigQuery](https://docs.open-metadata.org/latest/connectors/database/bigquery) [![BigTable-icon](https://docs.open-metadata.org/_next/image?url=%2Fimages%2Fconnectors%2Fbig-table.webp&w=64&q=75)\ \ BigTable](https://docs.open-metadata.org/latest/connectors/database/bigtable) [![Cassandra-icon](https://docs.open-metadata.org/_next/image?url=%2Fimages%2Fconnectors%2Fcassandra.webp&w=64&q=75)\ \ Cassandra](https://docs.open-metadata.org/latest/connectors/database/cassandra) [![ClickHouse-icon](https://docs.open-metadata.org/_next/image?url=%2Fimages%2Fconnectors%2Fclickhouse.webp&w=64&q=75)\ \ ClickHouse](https://docs.open-metadata.org/latest/connectors/database/clickhouse) [![Cockroach-icon](https://docs.open-metadata.org/_next/image?url=%2Fimages%2Fconnectors%2Fcockroach.webp&w=64&q=75)\ \ Cockroach](https://docs.open-metadata.org/latest/connectors/database/cockroach) [![Collibra-icon](https://docs.open-metadata.org/_next/image?url=%2Fimages%2Fconnectors%2Fcollibra.webp&w=64&q=75)\ \ Collibra](https://docs.open-metadata.org/latest/connectors/metadata/collibra) [![Couchbase-icon](https://docs.open-metadata.org/_next/image?url=%2Fimages%2Fconnectors%2Fcouchbase.webp&w=64&q=75)\ \ Couchbase](https://docs.open-metadata.org/latest/connectors/database/couchbase) [![Dagster-icon](https://docs.open-metadata.org/_next/image?url=%2Fimages%2Fconnectors%2Fdagster.webp&w=64&q=75)\ \ Dagster](https://docs.open-metadata.org/latest/connectors/pipeline/dagster) [![Databricks-icon](https://docs.open-metadata.org/_next/image?url=%2Fimages%2Fconnectors%2Fdatabrick.webp&w=64&q=75)\ \ Databricks](https://docs.open-metadata.org/latest/connectors/database/databricks) [![Databricks Pipeline-icon](https://docs.open-metadata.org/_next/image?url=%2Fimages%2Fconnectors%2Fdatabrick.webp&w=64&q=75)\ \ Databricks Pipeline](https://docs.open-metadata.org/latest/connectors/pipeline/databricks-pipeline) [![DB2-icon](https://docs.open-metadata.org/_next/image?url=%2Fimages%2Fconnectors%2Fibmdb2.webp&w=64&q=75)\ \ DB2](https://docs.open-metadata.org/latest/connectors/database/db2) [![dbt-icon](https://docs.open-metadata.org/_next/image?url=%2Fimages%2Fconnectors%2Fdbtcloud.webp&w=64&q=75)\ \ dbt](https://docs.open-metadata.org/latest/connectors/pipeline/dbtcloud) [![dbt Cloud-icon](https://docs.open-metadata.org/_next/image?url=%2Fimages%2Fconnectors%2Fdbtcloud.webp&w=64&q=75)\ \ dbt Cloud](https://docs.open-metadata.org/latest/connectors/pipeline/dbtcloud) [![Delta Lake-icon](https://docs.open-metadata.org/_next/image?url=%2Fimages%2Fconnectors%2Fdelta-lake.webp&w=64&q=75)\ \ Delta Lake](https://docs.open-metadata.org/latest/connectors/database/deltalake) [![Domo-icon](https://docs.open-metadata.org/_next/image?url=%2Fimages%2Fconnectors%2Fdomo.webp&w=64&q=75)\ \ Domo](https://docs.open-metadata.org/latest/connectors/database/domo-database) [![Doris-icon](https://docs.open-metadata.org/_next/image?url=%2Fimages%2Fconnectors%2Fdoris.webp&w=64&q=75)\ \ Doris](https://docs.open-metadata.org/latest/connectors/database/doris) [![Druid-icon](https://docs.open-metadata.org/_next/image?url=%2Fimages%2Fconnectors%2Fdruid.webp&w=64&q=75)\ \ Druid](https://docs.open-metadata.org/latest/connectors/database/druid) [![DynamoDB-icon](https://docs.open-metadata.org/_next/image?url=%2Fimages%2Fconnectors%2Fdynamodb.webp&w=64&q=75)\ \ DynamoDB](https://docs.open-metadata.org/latest/connectors/database/dynamodb) [![Elasticsearch-icon](https://docs.open-metadata.org/_next/image?url=%2Fimages%2Fconnectors%2Felasticsearch.webp&w=64&q=75)\ \ Elasticsearch](https://docs.open-metadata.org/latest/connectors/search/elasticsearch) [![Exasol-icon](https://docs.open-metadata.org/_next/image?url=%2Fimages%2Fconnectors%2Fexasol.webp&w=64&q=75)\ \ Exasol](https://docs.open-metadata.org/latest/connectors/database/exasol) [![Fivetran-icon](https://docs.open-metadata.org/_next/image?url=%2Fimages%2Fconnectors%2Ffivetran.webp&w=64&q=75)\ \ Fivetran](https://docs.open-metadata.org/latest/connectors/pipeline/fivetran) [![Flink-icon](https://docs.open-metadata.org/_next/image?url=%2Fimages%2Fconnectors%2Fflink.webp&w=64&q=75)\ \ Flink](https://docs.open-metadata.org/latest/connectors/pipeline/flink) [![GCS-icon](https://docs.open-metadata.org/_next/image?url=%2Fimages%2Fconnectors%2Fgcs.webp&w=64&q=75)\ \ GCS](https://docs.open-metadata.org/latest/connectors/storage/gcs) [![GCS Datalake-icon](https://docs.open-metadata.org/_next/image?url=%2Fimages%2Fconnectors%2Fgcs.webp&w=64&q=75)\ \ GCS Datalake](https://docs.open-metadata.org/latest/connectors/database/gcs-datalake) [![Glue-icon](https://docs.open-metadata.org/_next/image?url=%2Fimages%2Fconnectors%2Fglue.webp&w=64&q=75)\ \ Glue](https://docs.open-metadata.org/latest/connectors/database/glue) [![Grafana-icon](https://docs.open-metadata.org/_next/image?url=%2Fimages%2Fconnectors%2Fgrafana.webp&w=64&q=75)\ \ Grafana](https://docs.open-metadata.org/latest/connectors/dashboard/grafana) [![Greenplum-icon](https://docs.open-metadata.org/_next/image?url=%2Fimages%2Fconnectors%2Fgreenplum.webp&w=64&q=75)\ \ Greenplum](https://docs.open-metadata.org/latest/connectors/database/greenplum) [![Hex-icon](https://docs.open-metadata.org/_next/image?url=%2Fimages%2Fconnectors%2Fhex.webp&w=64&q=75)\ \ Hex](https://docs.open-metadata.org/latest/connectors/dashboard/hex) [![Hive-icon](https://docs.open-metadata.org/_next/image?url=%2Fimages%2Fconnectors%2Fhive.webp&w=64&q=75)\ \ Hive](https://docs.open-metadata.org/latest/connectors/database/hive) [![Iceberg-icon](https://docs.open-metadata.org/_next/image?url=%2Fimages%2Fconnectors%2Ficeberg.webp&w=64&q=75)\ \ Iceberg](https://docs.open-metadata.org/latest/connectors/database/iceberg) [![Impala-icon](https://docs.open-metadata.org/_next/image?url=%2Fimages%2Fconnectors%2Fimpala.webp&w=64&q=75)\ \ Impala](https://docs.open-metadata.org/latest/connectors/database/impala) [![Kafka-icon](https://docs.open-metadata.org/_next/image?url=%2Fimages%2Fconnectors%2Fkafka.webp&w=64&q=75)\ \ Kafka](https://docs.open-metadata.org/latest/connectors/messaging/kafka) [![Kinesis-icon](https://docs.open-metadata.org/_next/image?url=%2Fimages%2Fconnectors%2Fkinesis.webp&w=64&q=75)\ \ Kinesis](https://docs.open-metadata.org/latest/connectors/messaging/kinesis) [![Lightdash-icon](https://docs.open-metadata.org/_next/image?url=%2Fimages%2Fconnectors%2Flightdash.webp&w=64&q=75)\ \ Lightdash](https://docs.open-metadata.org/latest/connectors/dashboard/lightdash) [![Looker-icon](https://docs.open-metadata.org/_next/image?url=%2Fimages%2Fconnectors%2Flooker.webp&w=64&q=75)\ \ Looker](https://docs.open-metadata.org/latest/connectors/dashboard/looker) [![MariaDB-icon](https://docs.open-metadata.org/_next/image?url=%2Fimages%2Fconnectors%2Fmariadb.webp&w=64&q=75)\ \ MariaDB](https://docs.open-metadata.org/latest/connectors/database/mariadb) [![Metabase-icon](https://docs.open-metadata.org/_next/image?url=%2Fimages%2Fconnectors%2Fmetabase.webp&w=64&q=75)\ \ Metabase](https://docs.open-metadata.org/latest/connectors/dashboard/metabase) [![MicroStrategy-icon](https://docs.open-metadata.org/_next/image?url=%2Fimages%2Fconnectors%2Fmicrostrategy.webp&w=64&q=75)\ \ MicroStrategy](https://docs.open-metadata.org/latest/connectors/dashboard/microstrategy) [![MLflow-icon](https://docs.open-metadata.org/_next/image?url=%2Fimages%2Fconnectors%2Fmlflow.webp&w=64&q=75)\ \ MLflow](https://docs.open-metadata.org/latest/connectors/ml-model/mlflow) [![Mode-icon](https://docs.open-metadata.org/_next/image?url=%2Fimages%2Fconnectors%2Fmode.webp&w=64&q=75)\ \ Mode](https://docs.open-metadata.org/latest/connectors/dashboard/mode) [![MongoDB-icon](https://docs.open-metadata.org/_next/image?url=%2Fimages%2Fconnectors%2Fmongodb.webp&w=64&q=75)\ \ MongoDB](https://docs.open-metadata.org/latest/connectors/database/mongodb) [![MSSQL-icon](https://docs.open-metadata.org/_next/image?url=%2Fimages%2Fconnectors%2Fmssql.webp&w=64&q=75)\ \ MSSQL](https://docs.open-metadata.org/latest/connectors/database/mssql) [![MySQL-icon](https://docs.open-metadata.org/_next/image?url=%2Fimages%2Fconnectors%2Fsql.webp&w=64&q=75)\ \ MySQL](https://docs.open-metadata.org/latest/connectors/database/mysql) [![NiFi-icon](https://docs.open-metadata.org/_next/image?url=%2Fimages%2Fconnectors%2Fapachenifi.webp&w=64&q=75)\ \ NiFi](https://docs.open-metadata.org/latest/connectors/pipeline/nifi) [![OpenLineage-icon](https://docs.open-metadata.org/_next/image?url=%2Fimages%2Fconnectors%2Fopenlineage.webp&w=64&q=75)\ \ OpenLineage](https://docs.open-metadata.org/latest/connectors/pipeline/openlineage) [![OpenSearch-icon](https://docs.open-metadata.org/_next/image?url=%2Fimages%2Fconnectors%2Fopensearch.webp&w=64&q=75)\ \ OpenSearch](https://docs.open-metadata.org/latest/connectors/search/opensearch) [![Oracle-icon](https://docs.open-metadata.org/_next/image?url=%2Fimages%2Fconnectors%2Foracle.webp&w=64&q=75)\ \ Oracle](https://docs.open-metadata.org/latest/connectors/database/oracle) [![Pinot-icon](https://docs.open-metadata.org/_next/image?url=%2Fimages%2Fconnectors%2Fpinot.webp&w=64&q=75)\ \ Pinot](https://docs.open-metadata.org/latest/connectors/database/pinotdb) [![PostgreSQL-icon](https://docs.open-metadata.org/_next/image?url=%2Fimages%2Fconnectors%2Fpost.webp&w=64&q=75)\ \ PostgreSQL](https://docs.open-metadata.org/latest/connectors/database/postgres) [![Power BI-icon](https://docs.open-metadata.org/_next/image?url=%2Fimages%2Fconnectors%2Fpower-bi.webp&w=64&q=75)\ \ Power BI](https://docs.open-metadata.org/latest/connectors/dashboard/powerbi) [![PowerBI Report Server-icon](https://docs.open-metadata.org/_next/image?url=%2Fimages%2Fconnectors%2Fpower-bi.webp&w=64&q=75)\ \ PowerBI Report Server](https://docs.open-metadata.org/latest/connectors/dashboard/powerbireportserver) [![Presto-icon](https://docs.open-metadata.org/_next/image?url=%2Fimages%2Fconnectors%2Fpresto.webp&w=64&q=75)\ \ Presto](https://docs.open-metadata.org/latest/connectors/database/presto) [![Qlik Cloud-icon](https://docs.open-metadata.org/_next/image?url=%2Fimages%2Fconnectors%2Fqlikcloud.webp&w=64&q=75)\ \ Qlik Cloud](https://docs.open-metadata.org/latest/connectors/dashboard/qlikcloud) [![Qlik Sense-icon](https://docs.open-metadata.org/_next/image?url=%2Fimages%2Fconnectors%2Fqlik-sense.webp&w=64&q=75)\ \ Qlik Sense](https://docs.open-metadata.org/latest/connectors/dashboard/qliksense) [![QuickSight-icon](https://docs.open-metadata.org/_next/image?url=%2Fimages%2Fconnectors%2Fquicksight.webp&w=64&q=75)\ \ QuickSight](https://docs.open-metadata.org/latest/connectors/dashboard/quicksight) [![Ranger-icon](https://docs.open-metadata.org/_next/image?url=%2Fimages%2Fconnectors%2Franger.webp&w=64&q=75)\ \ Ranger](https://docs.open-metadata.org/latest/connectors/security/ranger) [![Redash-icon](https://docs.open-metadata.org/_next/image?url=%2Fimages%2Fconnectors%2Fredash.webp&w=64&q=75)\ \ Redash](https://docs.open-metadata.org/latest/connectors/dashboard/redash) [![Redpanda-icon](https://docs.open-metadata.org/_next/image?url=%2Fimages%2Fconnectors%2Fredpanda.webp&w=64&q=75)\ \ Redpanda](https://docs.open-metadata.org/latest/connectors/messaging/redpanda) [![Redshift-icon](https://docs.open-metadata.org/_next/image?url=%2Fimages%2Fconnectors%2Fredshift.webp&w=64&q=75)\ \ Redshift](https://docs.open-metadata.org/latest/connectors/database/redshift) [![REST-icon](https://docs.open-metadata.org/_next/image?url=%2Fimages%2Fconnectors%2Frest.webp&w=64&q=75)\ \ REST](https://docs.open-metadata.org/latest/connectors/api/rest) [![S3 Datalake-icon](https://docs.open-metadata.org/_next/image?url=%2Fimages%2Fconnectors%2Famazon-s3.webp&w=64&q=75)\ \ S3 Datalake](https://docs.open-metadata.org/latest/connectors/database/s3-datalake) [![S3 Storage-icon](https://docs.open-metadata.org/_next/image?url=%2Fimages%2Fconnectors%2Famazon-s3.webp&w=64&q=75)\ \ S3 Storage](https://docs.open-metadata.org/latest/connectors/storage/s3) [![SageMaker-icon](https://docs.open-metadata.org/_next/image?url=%2Fimages%2Fconnectors%2Fsagemaker.webp&w=64&q=75)\ \ SageMaker](https://docs.open-metadata.org/latest/connectors/ml-model/sagemaker) [![Salesforce-icon](https://docs.open-metadata.org/_next/image?url=%2Fimages%2Fconnectors%2Fsalesforce.webp&w=64&q=75)\ \ Salesforce](https://docs.open-metadata.org/latest/connectors/database/salesforce) [![SAP ERP-icon](https://docs.open-metadata.org/_next/image?url=%2Fimages%2Fconnectors%2Fsap-erp.webp&w=64&q=75)\ \ SAP ERP](https://docs.open-metadata.org/latest/connectors/database/sap-erp) [![SAP HANA-icon](https://docs.open-metadata.org/_next/image?url=%2Fimages%2Fconnectors%2Fsap-hana.webp&w=64&q=75)\ \ SAP HANA](https://docs.open-metadata.org/latest/connectors/database/sap-hana) [![SAS-icon](https://docs.open-metadata.org/_next/image?url=%2Fimages%2Fconnectors%2Fsas.webp&w=64&q=75)\ \ SAS](https://docs.open-metadata.org/latest/connectors/database/sas) [![ServiceNow-icon](https://docs.open-metadata.org/_next/image?url=%2Fimages%2Fconnectors%2Fservicenow.webp&w=64&q=75)\ \ ServiceNow](https://docs.open-metadata.org/latest/connectors/database/servicenow) [![Sigma-icon](https://docs.open-metadata.org/_next/image?url=%2Fimages%2Fconnectors%2Fsigma.webp&w=64&q=75)\ \ Sigma](https://docs.open-metadata.org/latest/connectors/dashboard/sigma) [![SingleStore-icon](https://docs.open-metadata.org/_next/image?url=%2Fimages%2Fconnectors%2Fsinglestore.webp&w=64&q=75)\ \ SingleStore](https://docs.open-metadata.org/latest/connectors/database/singlestore) [![Snowflake-icon](https://docs.open-metadata.org/_next/image?url=%2Fimages%2Fconnectors%2Fsnowflakes.webp&w=64&q=75)\ \ Snowflake](https://docs.open-metadata.org/latest/connectors/database/snowflake) [![Snowplow-icon](https://docs.open-metadata.org/_next/image?url=%2Fimages%2Fconnectors%2Fsnowplow.webp&w=64&q=75)\ \ Snowplow](https://docs.open-metadata.org/latest/connectors/pipeline/snowplow) [![Spline-icon](https://docs.open-metadata.org/_next/image?url=%2Fimages%2Fconnectors%2Fspline.webp&w=64&q=75)\ \ Spline](https://docs.open-metadata.org/latest/connectors/pipeline/spline) [![SQL Lite-icon](https://docs.open-metadata.org/_next/image?url=%2Fimages%2Fconnectors%2Fsqlite.webp&w=64&q=75)\ \ SQL Lite](https://docs.open-metadata.org/latest/connectors/database/sqlite) [![Superset-icon](https://docs.open-metadata.org/_next/image?url=%2Fimages%2Fconnectors%2Fsuperset.webp&w=64&q=75)\ \ Superset](https://docs.open-metadata.org/latest/connectors/dashboard/superset) [![Tableau-icon](https://docs.open-metadata.org/_next/image?url=%2Fimages%2Fconnectors%2Ftableau.webp&w=64&q=75)\ \ Tableau](https://docs.open-metadata.org/latest/connectors/dashboard/tableau) [![Teradata-icon](https://docs.open-metadata.org/_next/image?url=%2Fimages%2Fconnectors%2Fteradata.webp&w=64&q=75)\ \ Teradata](https://docs.open-metadata.org/latest/connectors/database/teradata) [![TimescaleDB-icon](https://docs.open-metadata.org/_next/image?url=%2Fimages%2Fconnectors%2Ftimescale.webp&w=64&q=75)\ \ TimescaleDB](https://docs.open-metadata.org/latest/connectors/database/timescale) [![Trino-icon](https://docs.open-metadata.org/_next/image?url=%2Fimages%2Fconnectors%2Ftrino.webp&w=64&q=75)\ \ Trino](https://docs.open-metadata.org/latest/connectors/database/trino) [![Unity Catalog-icon](https://docs.open-metadata.org/_next/image?url=%2Fimages%2Fconnectors%2Fdatabrick.webp&w=64&q=75)\ \ Unity Catalog](https://docs.open-metadata.org/latest/connectors/database/unity-catalog) [![Vertica-icon](https://docs.open-metadata.org/_next/image?url=%2Fimages%2Fconnectors%2Fvertica.webp&w=64&q=75)\ \ Vertica](https://docs.open-metadata.org/latest/connectors/database/vertica) Blogs [![OpenMetadata Release 1.11.0](https://docs.open-metadata.org/_next/image?url=%2Fblogs%2Fopenmetadata-1.11.0-release.webp&w=2048&q=75)\ \ OpenMetadata Release 1.11.0\ ---------------------------\ \ Shift Left with Data Quality as Code, Data Quality Dimensionality, and more](https://blog.open-metadata.org/announcing-openmetadata-1-11-38e79cfe5e15) [![OpenMetadata Release 1.10](https://docs.open-metadata.org/_next/image?url=%2Fblogs%2Fopenmetadata-1.10.0-release.webp&w=2048&q=75)\ \ OpenMetadata Release 1.10\ -------------------------\ \ Impact analysis, data contract improvements, OpenMetadata SDK 2.0, and more](https://blog.open-metadata.org/announcing-openmetadata-1-10-67250f268d3d) [![OpenMetadata Release 1.9.0](https://docs.open-metadata.org/_next/image?url=%2Fblogs%2Fopenmetadata-1.9.0-release.webp&w=2048&q=75)\ \ OpenMetadata Release 1.9.0\ --------------------------\ \ Data Contracts, Multi-Domain Support, and Enhanced User Experience](https://blog.open-metadata.org/announcing-openmetadata-1-9-68b63005bb25) --- # OpenMetadata Documentation: Get Help Instantly We use cookies to improve site navigation, analyze site usage, and enhance your user experience. Click "Accept" to enable cookies or "Reject" to reject cookies. AcceptReject ![](https://static.scarf.sh/a.png?x-pxid=87848f39-08ad-4fee-a3d6-ab05b717d351) OpenMetadata Documentation ========================== Unlock the value of data assets with an end-to-end metadata management solution that includes data discovery, governance, data quality, observability, and people collaboration. Quick Start Install OpenMetadata to explore its full potential. [Get Started](https://docs.open-metadata.org/latest/quick-start) How-to Guides Get a complete overview of the features in OpenMetadata from our How-to Guides [Explore Features](https://docs.open-metadata.org/latest/how-to-guides) Join the OSS Community Connect with 1000s of OpenMetadata users. Get support for all your questions from data experts. [Join Slack Now!](https://slack.open-metadata.org/) Overview OpenMetadata is a unified platform for discovery, observability, and governance powered by a central metadata repository, in-depth lineage, and seamless team collaboration. It is one of the fastest-growing open-source projects with a vibrant community and adoption by a diverse set of companies in a variety of industry verticals. Based on Open Metadata Standards and APIs, supporting connectors to a wide range of data services, OpenMetadata enables end-to-end metadata management, giving you the freedom to unlock the value of your data assets. Quick Links [Deployment\ \ Deploy in Bare Metal, Docker or Kubernetes on any cloud πŸŽ‰](https://docs.open-metadata.org/latest/deployment) [SaaS\ \ Enjoy 100% of OpenMetadata with 0% of the hassle πŸš€](https://getcollate.io/) [Knowledge Base\ \ Check out some frequent Questions & Answers πŸ’¬](https://github.com/open-metadata/OpenMetadata/discussions/categories/q-a) Connectors All connectorsAPIDatabaseDriveMessagingDashboardPipelineMlmodelSearchStorageMetadataSecurity [ADLS Datalake](https://docs.open-metadata.org/latest/connectors/database/adls-datalake) [Airbyte](https://docs.open-metadata.org/latest/connectors/pipeline/airbyte) [Airflow](https://docs.open-metadata.org/latest/connectors/pipeline/airflow) [AlationSink](https://docs.open-metadata.org/latest/connectors/metadata/alationsink) [Athena](https://docs.open-metadata.org/latest/connectors/database/athena) [Atlas](https://docs.open-metadata.org/latest/connectors/metadata/atlas) [Azure SQL](https://docs.open-metadata.org/latest/connectors/database/azuresql) [BigQuery](https://docs.open-metadata.org/latest/connectors/database/bigquery) [BigTable](https://docs.open-metadata.org/latest/connectors/database/bigtable) [Cassandra](https://docs.open-metadata.org/latest/connectors/database/cassandra) [ClickHouse](https://docs.open-metadata.org/latest/connectors/database/clickhouse) [Cockroach](https://docs.open-metadata.org/latest/connectors/database/cockroach) [Collibra](https://docs.open-metadata.org/latest/connectors/metadata/collibra) [Couchbase](https://docs.open-metadata.org/latest/connectors/database/couchbase) [Dagster](https://docs.open-metadata.org/latest/connectors/pipeline/dagster) [Databricks](https://docs.open-metadata.org/latest/connectors/database/databricks) [Databricks Pipeline](https://docs.open-metadata.org/latest/connectors/pipeline/databricks-pipeline) [DB2](https://docs.open-metadata.org/latest/connectors/database/db2) [dbt](https://docs.open-metadata.org/latest/connectors/pipeline/dbtcloud) [dbt Cloud](https://docs.open-metadata.org/latest/connectors/pipeline/dbtcloud) [Delta Lake](https://docs.open-metadata.org/latest/connectors/database/deltalake) [Domo](https://docs.open-metadata.org/latest/connectors/database/domo-database) [Doris](https://docs.open-metadata.org/latest/connectors/database/doris) [Druid](https://docs.open-metadata.org/latest/connectors/database/druid) [DynamoDB](https://docs.open-metadata.org/latest/connectors/database/dynamodb) [Elasticsearch](https://docs.open-metadata.org/latest/connectors/search/elasticsearch) [Exasol](https://docs.open-metadata.org/latest/connectors/database/exasol) [Fivetran](https://docs.open-metadata.org/latest/connectors/pipeline/fivetran) [Flink](https://docs.open-metadata.org/latest/connectors/pipeline/flink) [GCS](https://docs.open-metadata.org/latest/connectors/storage/gcs) [GCS Datalake](https://docs.open-metadata.org/latest/connectors/database/gcs-datalake) [Glue](https://docs.open-metadata.org/latest/connectors/database/glue) [Grafana](https://docs.open-metadata.org/latest/connectors/dashboard/grafana) [Greenplum](https://docs.open-metadata.org/latest/connectors/database/greenplum) [Hex](https://docs.open-metadata.org/latest/connectors/dashboard/hex) [Hive](https://docs.open-metadata.org/latest/connectors/database/hive) [Iceberg](https://docs.open-metadata.org/latest/connectors/database/iceberg) [Impala](https://docs.open-metadata.org/latest/connectors/database/impala) [Kafka](https://docs.open-metadata.org/latest/connectors/messaging/kafka) [Kinesis](https://docs.open-metadata.org/latest/connectors/messaging/kinesis) [Lightdash](https://docs.open-metadata.org/latest/connectors/dashboard/lightdash) [Looker](https://docs.open-metadata.org/latest/connectors/dashboard/looker) [MariaDB](https://docs.open-metadata.org/latest/connectors/database/mariadb) [Metabase](https://docs.open-metadata.org/latest/connectors/dashboard/metabase) [MicroStrategy](https://docs.open-metadata.org/latest/connectors/dashboard/microstrategy) [MLflow](https://docs.open-metadata.org/latest/connectors/ml-model/mlflow) [Mode](https://docs.open-metadata.org/latest/connectors/dashboard/mode) [MongoDB](https://docs.open-metadata.org/latest/connectors/database/mongodb) [MSSQL](https://docs.open-metadata.org/latest/connectors/database/mssql) [MySQL](https://docs.open-metadata.org/latest/connectors/database/mysql) [NiFi](https://docs.open-metadata.org/latest/connectors/pipeline/nifi) [OpenLineage](https://docs.open-metadata.org/latest/connectors/pipeline/openlineage) [OpenSearch](https://docs.open-metadata.org/latest/connectors/search/opensearch) [Oracle](https://docs.open-metadata.org/latest/connectors/database/oracle) [Pinot](https://docs.open-metadata.org/latest/connectors/database/pinotdb) [PostgreSQL](https://docs.open-metadata.org/latest/connectors/database/postgres) [Power BI](https://docs.open-metadata.org/latest/connectors/dashboard/powerbi) [PowerBI Report Server](https://docs.open-metadata.org/latest/connectors/dashboard/powerbireportserver) [Presto](https://docs.open-metadata.org/latest/connectors/database/presto) [Qlik Cloud](https://docs.open-metadata.org/latest/connectors/dashboard/qlikcloud) [Qlik Sense](https://docs.open-metadata.org/latest/connectors/dashboard/qliksense) [QuickSight](https://docs.open-metadata.org/latest/connectors/dashboard/quicksight) [Ranger](https://docs.open-metadata.org/latest/connectors/security/ranger) [Redash](https://docs.open-metadata.org/latest/connectors/dashboard/redash) [Redpanda](https://docs.open-metadata.org/latest/connectors/messaging/redpanda) [Redshift](https://docs.open-metadata.org/latest/connectors/database/redshift) [REST](https://docs.open-metadata.org/latest/connectors/api/rest) [S3 Datalake](https://docs.open-metadata.org/latest/connectors/database/s3-datalake) [S3 Storage](https://docs.open-metadata.org/latest/connectors/storage/s3) [SageMaker](https://docs.open-metadata.org/latest/connectors/ml-model/sagemaker) [Salesforce](https://docs.open-metadata.org/latest/connectors/database/salesforce) [SAP ERP](https://docs.open-metadata.org/latest/connectors/database/sap-erp) [SAP HANA](https://docs.open-metadata.org/latest/connectors/database/sap-hana) [SAS](https://docs.open-metadata.org/latest/connectors/database/sas) [ServiceNow](https://docs.open-metadata.org/latest/connectors/database/servicenow) [Sigma](https://docs.open-metadata.org/latest/connectors/dashboard/sigma) [SingleStore](https://docs.open-metadata.org/latest/connectors/database/singlestore) [Snowflake](https://docs.open-metadata.org/latest/connectors/database/snowflake) [Snowplow](https://docs.open-metadata.org/latest/connectors/pipeline/snowplow) [Spline](https://docs.open-metadata.org/latest/connectors/pipeline/spline) [SQL Lite](https://docs.open-metadata.org/latest/connectors/database/sqlite) [Superset](https://docs.open-metadata.org/latest/connectors/dashboard/superset) [Tableau](https://docs.open-metadata.org/latest/connectors/dashboard/tableau) [Teradata](https://docs.open-metadata.org/latest/connectors/database/teradata) [TimescaleDB](https://docs.open-metadata.org/latest/connectors/database/timescale) [Trino](https://docs.open-metadata.org/latest/connectors/database/trino) [Unity Catalog](https://docs.open-metadata.org/latest/connectors/database/unity-catalog) [Vertica](https://docs.open-metadata.org/latest/connectors/database/vertica) Blogs [![OpenMetadata Release 1.11.0](https://docs.open-metadata.org/_next/image?url=%2Fblogs%2Fopenmetadata-1.11.0-release.webp&w=2048&q=75)\ \ OpenMetadata Release 1.11.0\ ---------------------------\ \ Shift Left with Data Quality as Code, Data Quality Dimensionality, and more](https://blog.open-metadata.org/announcing-openmetadata-1-11-38e79cfe5e15) [![OpenMetadata Release 1.10](https://docs.open-metadata.org/_next/image?url=%2Fblogs%2Fopenmetadata-1.10.0-release.webp&w=2048&q=75)\ \ OpenMetadata Release 1.10\ -------------------------\ \ Impact analysis, data contract improvements, OpenMetadata SDK 2.0, and more](https://blog.open-metadata.org/announcing-openmetadata-1-10-67250f268d3d) [![OpenMetadata Release 1.9.0](https://docs.open-metadata.org/_next/image?url=%2Fblogs%2Fopenmetadata-1.9.0-release.webp&w=2048&q=75)\ \ OpenMetadata Release 1.9.0\ --------------------------\ \ Data Contracts, Multi-Domain Support, and Enhanced User Experience](https://blog.open-metadata.org/announcing-openmetadata-1-9-68b63005bb25) --- # Metadata Ingestion Best Practices We use cookies to improve site navigation, analyze site usage, and enhance your user experience. Click "Accept" to enable cookies or "Reject" to reject cookies. AcceptReject connectors No menu items for this category [Home](https://docs.open-metadata.org/latest) /[Connectors](https://docs.open-metadata.org/latest/connectors) /[Ingestion](https://docs.open-metadata.org/latest/connectors/ingestion) /[Best Practices](https://docs.open-metadata.org/latest/connectors/ingestion/best-practices) OpenMetadata Documentation Best Practices for Metadata Ingestion ===================================== In this section we are going to present some guidelines that can be useful when preparing metadata ingestion both from the UI or via any custom orchestration system. We will use the generic terms from Airflow, as the most common used tool, but the underlying ideas can be applied anywhere. Generic Practices ----------------- * **DAGs should not have any retry**: If the workflow is marked as failed due to any error (unexpected exception, connectivity issues, individual assets’ errors,...) there is usually no point on running automatic retries. For heavy workflows failing in the middle of processing, it will just incur in extra costs. Note that for internal communications between the Ingestion Workflow and the OpenMetadata APIs, we already have an internal retry in place in case of intermittent networking issues. * **DAGs should not have a catch-up**: Any ingestion will be based on the current state of data and metadata. If old runs were skipped for any reason, there is no point in triggering past executions as they won’t be adding any value. Just the single, most recent run will already be providing all the information available. * **Be mindful of enabled DEBUG logs**: When configuring the ingestion YAML you have the option to control the logging level. Keeping it as INFO (default) is the usual best bet. Only use DEBUG logs when testing out ingestion for the first time * **Test the ingestions using the CLI if you will be building a DAG**: When preparing the first ingestion processes, it is ok to try different configurations (debug logs, enable views, filtering of assets,...). The fastest and easiest way to test the ingestion process that will end up on a DAG is using the CLI (example). Playing with the CLI will help you find the right YAML configuration fast. Note that for OpenMetadata, the process that gets triggered from the CLI, is the same as the one that will eventually run in your DAGs. If you have the possibility to test the CLI first, it will give you fast feedback and will help you isolate your tests. Metadata Ingestion ------------------ * **Apply the right filters**: For example, there is usually no business-related information on schemas such as `INFORMATION_SCHEMA`. You can use OpenMetadata filtering logic on databases, services and tables to opt in/out specific assets. Profiler Ingestion ------------------ * **On filters, scheduling and asset importance**: While OpenMetadata provides sampling and multi-threading, profiling can be a costly and time-consuming process. Then it is important to know which data assets are business critical. * **Deploy multiple profiler ingestions for the same service**: For a given service, prepare different ingestion pipelines, each of them attacking a specific set of assets based on input filters. You can then schedule more important assets to be profiled more often, while keeping the rest of profiles to be executed either on demand, or with lower cadence. * **Apply the right sampling**: Important tables can hold higher sampling, while the rest of assets might be good enough with smaller %. Usage & Lineage Ingestion ------------------------- * **Schedule and log duration should match**: The Log Duration configuration parameter specifies how many days in the past we are going to look for query history data. If we schedule the workflows to run daily, there is no need to look for the past week, as we will be re-analysing data that won’t change. OpenMetadata Ingestion Troubleshooting ====================================== Here we will discuss different errors that you might encounter when running a workflow: * **Connection errors**: When deploying ingestions from the OpenMetadata UI you have the possibility to test the connection when configuring the service. This connection test happens at the Airflow host configured with OpenMetadata. If instead, you are running your ingestion workflows from any external system, you’ll need to validate that the host where the ingestion runs has the proper network settings to reach both the source system and OpenMetadata. * **Processing Errors**: During the workflow process you might see logs like `Cannot ingest X due to Y` or similar statements. They appear for specific assets being ingested, and the origin can be different: * Missing permissions on a specific table or tag (e.g., due to BigQuery policies), * Internal errors when processing specific assets or translating them to the OpenMetadata standard. In these cases, you can reach out to the OpenMetadata team. The workflow itself will continue, and the OpenMetadata team can help analyse the root cause and provide a fix. * **Workflow breaking exceptions**: In rare circumstances there can be exceptions that break the overall workflow processing. The goal of the Ingestion Framework is to be as robust as possible and continue even for specific assets failures (see point above). If there is a scenario not contemplated by the current code, the OpenMetadata team will apply the highest priority to fix the issue and allow the workflow to run end to end. --- # Metadata Versioning | OpenMetadata Data History Management We use cookies to improve site navigation, analyze site usage, and enhance your user experience. Click "Accept" to enable cookies or "Reject" to reject cookies. AcceptReject connectors No menu items for this category [Home](https://docs.open-metadata.org/latest) /[Connectors](https://docs.open-metadata.org/latest/connectors) /[Ingestion](https://docs.open-metadata.org/latest/connectors/ingestion) /[Versioning](https://docs.open-metadata.org/latest/connectors/ingestion/versioning) OpenMetadata Documentation Metadata Versioning =================== OpenMetadata maintains the version history for all entities using a number with the format _major.minor_, starting with 0.1 as the initial version of an entity. Changes in metadata result in version changes as follows: * Backward **compatible** changes result in a Minor version change. A change in the description, tags, or ownership will increase the version of the entity metadata by 0.1 (e.g., from 0.1 to 0.2). * Backward **incompatible** changes result in a Major version change. For example, when a column in a table is deleted, the version increases by 1.0 (e.g., from 0.2 to 1.2). Metadata versioning helps **simplify debugging processes**. View the version history to see if a recent change led to a data issue. Data owners and admins can review changes and revert if necessary. Versioning also helps in **broader collaboration** among consumers and producers of data. Admins can provide access to more users in the organization to change certain fields. Crow sourcing makes metadata the collective responsibility of the entire organization. ![Metadata versioning](https://docs.open-metadata.org/images/v1.11/features/ingestion/versioning/metadata-versioning.gif) --- # Lineage Workflow | OpenMetadata Data Lineage Guide We use cookies to improve site navigation, analyze site usage, and enhance your user experience. Click "Accept" to enable cookies or "Reject" to reject cookies. AcceptReject connectors No menu items for this category [Home](https://docs.open-metadata.org/latest) /[Connectors](https://docs.open-metadata.org/latest/connectors) /[Ingestion](https://docs.open-metadata.org/latest/connectors/ingestion) /[Workflows](https://docs.open-metadata.org/latest/connectors/ingestion/workflows) /[Lineage](https://docs.open-metadata.org/latest/connectors/ingestion/workflows/lineage) OpenMetadata Documentation Lineage Workflow ================ Learn how to configure the Lineage workflow from the UI to ingest Lineage data from your data sources. Checkout the documentation of the connector you are using to know if it supports automated lineage workflow. If your database service is not yet supported, you can use this same workflow by providing a Query Log file! Learn how to do so πŸ‘‡ [Lineage Workflow through Query Logs\ \ Configure the lineage workflow by providing a Query Log file.](https://docs.open-metadata.org/latest/connectors/ingestion/workflows/lineage/lineage-workflow-query-logs) UI Configuration ---------------- Once the metadata ingestion runs correctly and we are able to explore the service Entities, we can add Entity Lineage information. This will populate the Lineage tab from the Table Entity Page. ![table-entity-page](https://docs.open-metadata.org/images/v1.11/features/ingestion/workflows/lineage/table-entity-page.png) Table Entity Page We can create a workflow that will obtain the query log and table creation information from the underlying database and feed it to OpenMetadata. The Lineage Ingestion will be in charge of obtaining this data. ### 1\. Add a Lineage Ingestion From the Service Page, go to the Ingestions tab to add a new ingestion and click on Add Lineage Ingestion. ![add-ingestion](https://docs.open-metadata.org/images/v1.11/features/ingestion/workflows/lineage/add-ingestion.png) Add Ingestion ### 2\. Configure the Lineage Ingestion Here you can enter the Lineage Ingestion details: ![configure-lineage-ingestion](https://docs.open-metadata.org/images/v1.11/features/ingestion/workflows/lineage/configure-lineage-ingestion.png) Configure the Lineage Ingestion ### Lineage Options **Query Log Duration** Specify the duration in days for which the lineage should capture lineage data from the query logs. For example, if you specify 2 as the value for the duration, the data lineage will capture lineage information for 48 hours prior to when the ingestion workflow is run. **Result Limit** Set the limit for the query log results to be run at a time. ### 3\. Schedule and Deploy After clicking Next, you will be redirected to the Scheduling form. This will be the same as the Metadata Ingestion. Select your desired schedule and click on Deploy to find the lineage pipeline being added to the Service Ingestions. ![schedule-and-deploy](https://docs.open-metadata.org/images/v1.11/features/ingestion/workflows/lineage/scheule-and-deploy.png) View Service Ingestion pipelines YAML Configuration ------------------ In the [connectors](https://docs.open-metadata.org/latest/connectors) section we showcase how to run the metadata ingestion from a JSON/YAML file using the Airflow SDK or the CLI via metadata ingest. Running a lineage workflow is also possible using a JSON/YAML configuration file. This is a good option if you wish to execute your workflow via the Airflow SDK or using the CLI; if you use the CLI a lineage workflow can be triggered with the command `metadata ingest -c FILENAME.yaml`. The `serviceConnection` config will be specific to your connector (you can find more information in the [connectors](https://docs.open-metadata.org/latest/connectors) section), though the sourceConfig for the lineage will be similar across all connectors. Lineage ------- After running a Metadata Ingestion workflow, we can run Lineage workflow. While the `serviceName` will be the same to that was used in Metadata Ingestion, so the ingestion bot can get the `serviceConnection` details from the server. ### 1\. Define the YAML Config This is a sample config for BigQuery Lineage: #### Source Configuration - Source Config You can find all the definitions and types for the `sourceConfig` [here](https://github.com/open-metadata/OpenMetadata/blob/main/openmetadata-spec/src/main/resources/json/schema/metadataIngestion/databaseServiceQueryLineagePipeline.json) . **queryLogDuration**: Configuration to tune how far we want to look back in query logs to process lineage data in days. **parsingTimeoutLimit**: Configuration to set the timeout for parsing the query in seconds. **filterCondition**: Condition to filter the query history. **resultLimit**: Configuration to set the limit for query logs. **queryLogFilePath**: Configuration to set the file path for query logs. **databaseFilterPattern**: Regex to only fetch databases that matches the pattern. **schemaFilterPattern**: Regex to only fetch tables or databases that matches the pattern. **tableFilterPattern**: Regex to only fetch tables or databases that matches the pattern. **overrideViewLineage**: Set the 'Override View Lineage' toggle to control whether to override the existing view lineage. **processViewLineage**: Set the 'Process View Lineage' toggle to control whether to process view lineage. **processQueryLineage**: Set the 'Process Query Lineage' toggle to control whether to process query lineage. **processStoredProcedureLineage**: Set the 'Process Stored ProcedureLog Lineage' toggle to control whether to process stored procedure lineage. **threads**: Number of Threads to use in order to parallelize lineage ingestion. #### Sink Configuration To send the metadata to OpenMetadata, it needs to be specified as `type: metadata-rest`. #### Workflow Configuration The main property here is the `openMetadataServerConfig`, where you can define the host and security provider of your OpenMetadata installation. For a simple, local installation using our docker containers, this looks like: filename.yamlCopy * You can learn more about how to configure and run the Lineage Workflow to extract Lineage data from [here](https://docs.open-metadata.org/latest/connectors/ingestion/workflows/lineage) ### 2\. Run with the CLI After saving the YAML config, we will run the command the same way we did for the metadata ingestion: --- # Database Filter Patterns | Official Documentation We use cookies to improve site navigation, analyze site usage, and enhance your user experience. Click "Accept" to enable cookies or "Reject" to reject cookies. AcceptReject connectors No menu items for this category [Home](https://docs.open-metadata.org/latest) /[Connectors](https://docs.open-metadata.org/latest/connectors) /[Ingestion](https://docs.open-metadata.org/latest/connectors/ingestion) /[Workflows](https://docs.open-metadata.org/latest/connectors/ingestion/workflows) /[Metadata](https://docs.open-metadata.org/latest/connectors/ingestion/workflows/metadata) /[Filter Patterns](https://docs.open-metadata.org/latest/connectors/ingestion/workflows/metadata/filter-patterns) /[Database](https://docs.open-metadata.org/latest/connectors/ingestion/workflows/metadata/filter-patterns/database) OpenMetadata Documentation Database Filter Patterns ======================== Configuring Filters ------------------- One can configure the metadata ingestion filter for database source using four configuration fields which are `Database Filter Pattern`, `Schema Filter Pattern`, `Table Filter Pattern` & `Use FQN For Filtering`. In this document we will learn about each field in detail along with many examples. In OpenMetadata v1.5.x, when both include and exclude filters are applied, the system first processes the include filter, followed by the exclude filter. ### Configuring Filters via UI Filters can be configured in UI while adding an ingestion pipeline through `Add Metadata Ingestion` page. ![Database Filter Pattern Fields](https://docs.open-metadata.org/images/v1.11/features/ingestion/workflows/metadata/filter-patterns/database-filter-patterns.webp) Database Filter Pattern Fields ### Configuring Filters via CLI Filters can be configured in CLI in connection configuration within `source.sourceConfig.config` field as described below. ### Use FQN For Filtering This flag set when you want to apply the filter on fully qualified name (e.g service\_name.db\_name.schema\_name.table\_name) instead of applying the filter to raw name of entity (e.g table\_name). This Flag is useful in scenario when you have schema with same name in different databases or table with same name in different schemas and you want to filter out one of them. This will be explained further in detail in this document. ### Database Filter Pattern Database filter patterns to control whether or not to include database as part of metadata ingestion. * **Include**: Explicitly include databases by adding a list of comma-separated regular expressions to the Include field. OpenMetadata will include all databases with names matching one or more of the supplied regular expressions. All other databases will be excluded. * **Exclude**: Explicitly exclude databases by adding a list of comma-separated regular expressions to the Exclude field. OpenMetadata will exclude all databases with names matching one or more of the supplied regular expressions. All other databases will be included. #### Example 1 Let's say we want to ingest metadata from a snowflake instance which contains multiple databases as described above. In this example we want to ingest all databases which contains `SNOWFLAKE` in name, then the filter pattern applied would be `.*SNOWFLAKE.*` in the include field. This will result in ingestion of database `SNOWFLAKE`, `SNOWFLAKE_SAMPLE_DATA` and `TEST_SNOWFLAKEDB`. ### Configuring Filters via UI for Example 1 ![Database Filter Pattern Example 1](https://docs.open-metadata.org/images/v1.11/features/ingestion/workflows/metadata/filter-patterns/database-filter-example-1.webp) Database Filter Pattern Example 1 ### Configuring Filters via CLI for Example 1 #### Example 2 In this example we want to ingest all databases which starts with `SNOWFLAKE` in name, then the filter pattern applied would be `^SNOWFLAKE.*` in the include field. This will result in ingestion of database `SNOWFLAKE` & `SNOWFLAKE_SAMPLE_DATA`. ### Configuring Filters via UI for Example 2 ![Database Filter Pattern Example 2](https://docs.open-metadata.org/images/v1.11/features/ingestion/workflows/metadata/filter-patterns/database-filter-example-2.webp) Database Filter Pattern Example 2 ### Configuring Filters via CLI for Example 2 #### Example 3 In this example we want to ingest all databases for which the name starts with `SNOWFLAKE` OR ends with `DB` , then the filter pattern applied would be `^SNOWFLAKE` & `DB$` in the include field. This will result in ingestion of database `SNOWFLAKE`, `SNOWFLAKE_SAMPLE_DATA`, `TEST_SNOWFLAKEDB` & `DUMMY_DB`. ### Configuring Filters via UI for Example 3 ![Database Filter Pattern Example 3](https://docs.open-metadata.org/images/v1.11/features/ingestion/workflows/metadata/filter-patterns/database-filter-example-3.webp) Database Filter Pattern Example 3 ### Configuring Filters via CLI for Example 3 #### Example 4 In this example we want to ingest only the `SNOWFLAKE` database then the filter pattern applied would be `^SNOWFLAKE$`. ### Configuring Filters via UI for Example 4 ![Database Filter Pattern Example 4](https://docs.open-metadata.org/images/v1.11/features/ingestion/workflows/metadata/filter-patterns/database-filter-example-4.webp) Database Filter Pattern Example 4 ### Configuring Filters via CLI for Example 4 ### Schema Filter Pattern Schema filter patterns are used to control whether or not to include schemas as part of metadata ingestion. * **Include**: Explicitly include schemas by adding a list of comma-separated regular expressions to the Include field. OpenMetadata will include all schemas with names matching one or more of the supplied regular expressions. All other schemas will be excluded. * **Exclude**: Explicitly exclude schemas by adding a list of comma-separated regular expressions to the Exclude field. OpenMetadata will exclude all schemas with names matching one or more of the supplied regular expressions. All other schemas will be included. #### Example 1 In this example we want to ingest all schema within any database with name `PUBLIC`, then the schema filter pattern applied would be `^PUBLIC$` in the include field. This will result in ingestion of schemas `SNOWFLAKE.PUBLIC` & `SNOWFLAKE_SAMPLE_DATA.PUBLIC` ### Configuring Filters via UI for Example 1 ![Schema Filter Pattern Example 1](https://docs.open-metadata.org/images/v1.11/features/ingestion/workflows/metadata/filter-patterns/schema-filter-example-1.webp) Schema Filter Pattern Example 1 ### Configuring Filters via CLI for Example 1 #### Example 2 In this example we want to ingest all schema within any database except schema with name `PUBLIC` available in `SNOWFLAKE_SAMPLE_DATA`. Notice that we have two schemas available with name `PUBLIC` one is available in database `SNOWFLAKE_SAMPLE_DATA.PUBLIC` and other is `SNOWFLAKE.PUBLIC`. As per the constraint of this example all the schemas including `SNOWFLAKE.PUBLIC` but we need to skip `SNOWFLAKE_SAMPLE_DATA.PUBLIC`. to do that we will need to set `useFqnForFiltering` flag to true by doing this the filter pattern will be applied to fully qualified name instead of raw table name. A fully qualified name(FQN) of schema is combination of service name, database name & schema name joined with `.`. In this example fully qualified name of the `SNOWFLAKE_SAMPLE_DATA.PUBLIC` schema will be `Snowflake_Prod.SNOWFLAKE_SAMPLE_DATA.PUBLIC`, so we will need to apply a exclude filter pattern `^Snowflake_Prod\.SNOWFLAKE_SAMPLE_DATA\.PUBLIC$` and set `useFqnForFiltering` to true. ### Configuring Filters via UI for Example 2 ![Schema Filter Pattern Example 2](https://docs.open-metadata.org/images/v1.11/features/ingestion/workflows/metadata/filter-patterns/schema-filter-example-2.webp) Schema Filter Pattern Example 2 ### Configuring Filters via CLI for Example 2 #### Example 3 In this example we want to ingest `SNOWFLAKE.PUBLIC` & all the schemas in `SNOWFLAKE_SAMPLE_DATA` that starts with `TPCH_` i.e `SNOWFLAKE_SAMPLE_DATA.TPCH_1`, `SNOWFLAKE_SAMPLE_DATA.TPCH_10` & `SNOWFLAKE_SAMPLE_DATA.TPCH_100`. To achieve this an include schema filter will be applied with pattern `^Snowflake_Prod\.SNOWFLAKE\.PUBLIC$` & `^Snowflake_Prod\.SNOWFLAKE_SAMPLE_DATA\.TPCH_.*`, we need to set `useFqnForFiltering` as true as we want to apply filter on FQN. ### Configuring Filters via UI for Example 3 ![Schema Filter Pattern Example 3](https://docs.open-metadata.org/images/v1.11/features/ingestion/workflows/metadata/filter-patterns/schema-filter-example-3.webp) Schema Filter Pattern Example 3 ### Configuring Filters via CLI for Example 3 ### Table Filter Pattern Table filter patterns are used to control whether or not to include tables as part of metadata ingestion. * **Include**: Explicitly include tables by adding a list of comma-separated regular expressions to the Include field. OpenMetadata will include all tables with names matching one or more of the supplied regular expressions. All other tables will be excluded. * **Exclude**: Explicitly exclude tables by adding a list of comma-separated regular expressions to the Exclude field. OpenMetadata will exclude all tables with names matching one or more of the supplied regular expressions. All other tables will be included. #### Example 1 #### Example 1 In this example we want to ingest table with name `CUSTOMER` within any schema and database. In this case we need to apply include table filter pattern `^CUSTOMER$`. This will result in ingestion of tables `Snowflake_Prod.SNOWFLAKE_SAMPLE_DATA.PUBLIC.CUSTOMER`, `Snowflake_Prod.SNOWFLAKE_SAMPLE_DATA.INFORMATION.CUSTOMER` & `Snowflake_Prod.SNOWFLAKE.PUBLIC.CUSTOMER` ### Configuring Filters via UI for Example 1 ![Table Filter Pattern Example 1](https://docs.open-metadata.org/images/v1.11/features/ingestion/workflows/metadata/filter-patterns/table-filter-example-1.webp) Table Filter Pattern Example 1 ### Configuring Filters via CLI for Example 1 #### Example 2 In this example we want to ingest table with name `CUSTOMER` within `PUBLIC` schema of any database. In this case we need to apply include table filter pattern `.*\.PUBLIC\.CUSTOMER$` this will also require to set the `useFqnForFiltering` flag as true as we want to apply filter on FQN. This will result in ingestion of tables `Snowflake_Prod.SNOWFLAKE_SAMPLE_DATA.PUBLIC.CUSTOMER` & `Snowflake_Prod.SNOWFLAKE.PUBLIC.CUSTOMER` ### Configuring Filters via UI for Example 2 ![Table Filter Pattern Example 2](https://docs.open-metadata.org/images/v1.11/features/ingestion/workflows/metadata/filter-patterns/table-filter-example-2.webp) Table Filter Pattern Example 2 ### Configuring Filters via CLI for Example 2 #### Example 3 In this example, we aim to ingest a table named `_delta_log/file.json` within the `session` schema of any database. To achieve this, we need to configure the following filter patterns: * Include Table Filter Pattern: `.*_delta_log/file\.json$` * Schema Filter Pattern: `session` The backslash `\` is used as an escape character for the dot (.) in the pattern ### Configuring Filters via UI for Example 3 ![Table Filter Pattern Example 3](https://docs.open-metadata.org/images/v1.11/features/ingestion/workflows/metadata/filter-patterns/table-filter-example-3.webp) Table Filter Pattern Example 3 ### Configuring Filters via CLI for Example 3 --- # Edit Data Lineage Manually We use cookies to improve site navigation, analyze site usage, and enhance your user experience. Click "Accept" to enable cookies or "Reject" to reject cookies. AcceptReject connectors No menu items for this category [Home](https://docs.open-metadata.org/latest) /[Connectors](https://docs.open-metadata.org/latest/connectors) /[Ingestion](https://docs.open-metadata.org/latest/connectors/ingestion) /[Lineage](https://docs.open-metadata.org/latest/connectors/ingestion/lineage) /[Edit Lineage Manually](https://docs.open-metadata.org/latest/connectors/ingestion/lineage/edit-lineage-manually) OpenMetadata Documentation Edit Data Lineage Manually ========================== Edit lineage to provide a richer understanding of the provenance of data. The OpenMetadata no-code editor provides a drag and drop interface. Drop tables, pipelines, and dashboards onto the lineage graph. You may add new edges or delete existing edges to better represent data lineage. ![gif](https://docs.open-metadata.org/images/v1.11/features/ingestion/lineage/edit-lineage-manually.gif) --- # Ingestion Pipeline UI Deployment We use cookies to improve site navigation, analyze site usage, and enhance your user experience. Click "Accept" to enable cookies or "Reject" to reject cookies. AcceptReject connectors No menu items for this category [Home](https://docs.open-metadata.org/latest) /[Connectors](https://docs.open-metadata.org/latest/connectors) /[Ingestion](https://docs.open-metadata.org/latest/connectors/ingestion) /[Deployment](https://docs.open-metadata.org/latest/connectors/ingestion/deployment) OpenMetadata Documentation Ingestion Pipeline UI Deployment ================================ In this page we are going to explain how OpenMetadata internally deploys the workflows that are configured from the UI. As of now, OpenMetadata uses Airflow under the hood as a scheduler for the Ingestion Pipelines. This is the right place if you are curious about our current approach or if you are looking forward to contribute by adding the implementation to deploy workflows to another tool directly from the UI. Here we are talking about an internal implementation detail. Do not be confused about the information that is going to be shared here vs. the pipeline services supported as connectors for metadata extraction. For example, we use Airflow as an internal element to deploy and schedule ingestion workflows, but we can also extract metadata from Airflow. Fivetran, for example, is a possible source, but we are not using it to deploy and schedule workflows. Before Reading -------------- This is a rather deep dive guide. We recommend that you get familiar with the overall OpenMetadata architecture first. You can find that [here](https://docs.open-metadata.org/latest/main-concepts/high-level-design) . System Context -------------- Everything in OpenMetadata is centralized and managed via the API. Then, the Workflow's lifecycle is also fully managed via the OpenMetadata server APIs. More over, the `IngestionPipeline` Entity is also defined in a JSON Schema that you can find [here](https://github.com/open-metadata/OpenMetadata/blob/main/openmetadata-spec/src/main/resources/json/schema/entity/services/ingestionPipelines/ingestionPipeline.json) . ![system context](https://docs.open-metadata.org/images/v1.11/features/ingestion/ingestion-pipeline/ingestion-pipeline-system-context.drawio.png) Note how OpenMetadata here acts as a middleware, connecting the actions being triggered in the UI to external orchestration systems, which will be the ones managing the heavy lifting of getting a workflow created, scheduled and run. Out of the box, OpenMetadata ships with the required logic to manage this connection to Airflow. Any workflow triggered from the UI won't directly run on the server, but instead it will be handled as a DAG in Airflow. The whole process here will describe in further detail how we transform an incoming request from the UI - in the shape of an Ingestion Pipeline Entity - to a DAG that Airflow can fully manage. OpenMetadata Server Container Diagram ------------------------------------- The main difference between an `IngestionPipeline` and any other Entity in OpenMetadata is that we need to bind some logic to an external system. Therefore, we are not done by the time we create the Entity itself, but we need an extra component handling the communication between Server and Orchestrator. We then have the `IngestionPipelineResource` defining not only the endpoints for managing the Entity, but also routes to handle the workflow lifecycle, such as deploying it, triggering it or deleting it from the orchestrator. * The Entity management is still handled by the `EntityRepository` and saved in the Storage Layer as a JSON, * While the communication to the Orchestrator is handled with the `PipelineServiceClient`. While the endpoints are directly defined in the `IngestionPipelineResource`, the `PipelineServiceClient` is an interface that decouples how OpenMetadata communicates with the Orchestrator, as different external systems will need different calls and data to be sent. * You can find the `PipelineServiceClient` abstraction [here](https://github.com/open-metadata/OpenMetadata/blob/main/openmetadata-spec/src/main/java/org/openmetadata/sdk/PipelineServiceClient.java) , * And the `AirflowRESTClient` implementation [here](https://github.com/open-metadata/OpenMetadata/blob/main/openmetadata-service/src/main/java/org/openmetadata/service/clients/pipeline/airflow/AirflowRESTClient.java) . The clients that implement the abstractions from the `PipelineServiceClient` are merely a translation layer between the information received in the shape of an `IngestionPipeline` Entity, and the specific requirements of each Orchestrator. After creating a new workflow from the UI or when editing it, there are two calls happening: * `POST` or `PUT` call to update the `Ingestion Pipeline Entity`, * `/deploy` HTTP call to the `IngestionPipelineResource` to trigger the deployment of the new or updated DAG in the Orchestrator. ![software system](https://docs.open-metadata.org/images/v1.11/features/ingestion/ingestion-pipeline/ingestion-pipeline-software-system.drawio.png) ### Creating the Ingestion Pipeline Based on its [JSON Schema](https://github.com/open-metadata/OpenMetadata/blob/main/openmetadata-spec/src/main/resources/json/schema/entity/services/ingestionPipelines/ingestionPipeline.json) , there are a few properties about the Ingestion Pipeline we can highlight: **1.** `service`: a pipeline is linked via an Entity Reference to a Service Entity or a Test Suite Entity. From the service is **2.** `pipelineType`: which represents the type of workflow to be created. This flag will be used down the line in the Orchestrator logic to properly create the required Python classes (e.g., `Workflow`, `ProfilerWorkflow`, `TestSuiteWorkflow`, etc.). **3.** `sourceConfig`: which is dependent on the pipeline type and define how the pipeline should behave (e.g., marking ingesting views as `False`). **4.** `openMetadataServerConnection`: defining how to reach the OM server with properties such as host, auth configuration, etc. **5.** `airflowConfig`: with Airflow specific configurations about the DAG such as the schedule. While we have yet to update the `airflowConfig` property to be more generic, the only field actually being used is the schedule. You might see this property here, but the whole process can still support other Orchestrators. We will clean this up in future releases. ![container create](https://docs.open-metadata.org/images/v1.11/features/ingestion/ingestion-pipeline/ingestion-pipeline-container-IngestionPipeline.drawio.png) Here, the process of creating an Ingestion Pipeline is then the same as with any other Entity. ### Deploying the Ingestion Pipeline When calling the `/v1/services/ingestionPipelines/deploy` endpoint defined in the `IngestionPipelineResource`, the Pipeline Service Client enters into play. The client needs to be implemented with a separated class, which has the knowledge on how to interact with the Orchestrator. The role of OpenMetadata here is just to pass the required communication to the Orchestrator to trigger a deployment of a new DAG. Basically we need a way to send a call to the Orchestrator that generated a DAG / Workflow object that will be run using the proper functions and classes from the Ingestion Framework. ![deploy](https://docs.open-metadata.org/images/v1.11/features/ingestion/ingestion-pipeline/ingestion-pipeline-pipeline-service-container.drawio.png) Any Orchestration system that is capable to **DYNAMICALLY** create a workflow based on a given input (that can be obtained from the `IngestionPipeline` Entity information) is a potentially valid candidate to be used as a Pipeline Service. Deep Dive - Deploying an Ingestion Pipeline to Airflow ====================================================== Now that we have the big picture in mind, let's go step by step on how we have defined this process in Airflow. The goal here is not to enter so much in Airflow specific details, but to explain what deploying an Ingestion Pipeline entails so that you feel engaged and prepared to contribute a new Pipeline Service Client implementation. In this example I will be deploying an ingestion workflow to get the metadata from a MySQL database. After clicking on the UI to deploy such pipeline, these are the calls that get triggered: **1.** `POST` call to create the `IngestionPipeline` Entity **2.** `POST` call to deploy the newly created pipeline. Create the Ingestion Pipeline ----------------------------- These are the details of such a call with the default parameters. And we receive the created Ingestion Pipeline Entity back: Deploy the Ingestion Pipeline - OpenMetadata -------------------------------------------- Once the Ingestion Pipeline is created in OM, the following request is sent: Notice how we are passing the ID of the created Entity. Based on this ID, the `IngestionPipelineResource` will pick up the Entity and call the Pipeline Service Client Then, the actual deployment logic is handled by the class implementing the Pipeline Service Client. For this example, it will be the [AirflowRESTClient](https://github.com/open-metadata/OpenMetadata/blob/main/openmetadata-service/src/main/java/org/openmetadata/service/clients/pipeline/airflow/AirflowRESTClient.java) . First, let's see what it is needed to instantiate the Airflow REST Client: If we focus on the important properties here, we can see: * username, * password, * and an API Endpoint As we use a Basic Auth to connect to Airflow's API, those all the necessary ingredients. This is just Airflow specific, so any other Pipeline Service can tune this parameters as needed. If we then check what has been implemented in the `deployPipeline` method of the Airflow REST Client, we will see that it is just calling a `/deploy` endpoint from its API root. What is this endpoint? What does it do? ### Airflow Managed APIs Airflow has many benefits, but it does not support to create DAGs dynamically via its API. That is why we have created the [OpenMetadata Airflow Managed APIs](https://github.com/open-metadata/OpenMetadata/tree/main/openmetadata-airflow-apis) Python package. This is a plugin that can be installed in Airflow that adds a set of endpoints (all that is needed for the Pipeline Service implementation), and more specifically, helps us create the bridge between the `IngestionPipeline` Entity and whatever Airflow requires to create a DAG. We know that to create a new DAG in Airflow we need a Python file to be placed under the `AIRFLOW_HOME/dags` directory (by default). Then, calling the `/deploy` endpoint will make the necessary steps to create such a file. What it is important here is to notice that in order to run a metadata ingestion workflow, we just need the following few lines of Python code: Where the YAML config shape is defined in each [Connector](https://docs.open-metadata.org/latest/connectors) and the workflow class depends on our goal: Ingestion, Profiling, Testing... You can follow this logic deeper in the source code of the managed APIs package, but the important thought here is that we need the following logic flow: **1.** An Ingestion Pipeline is created and sent to the Ingestion Pipeline Resource. **2.** We need to transform this Ingestion Pipeline into something capable of running the Python `Workflow`. For Airflow, this something is a `.py` file. **3.** Note that as Airflow required us to build the whole dynamic creation, we shifted all the building logic towards the managed APIs package, but if any orchestrator already has an API capable of creating DAGs dynamically, this process can be directly handled in the Pipeline Service Client implementation as all the necessary data is present in the Ingestion Pipeline Entity. Deep Dive - Pipeline Service Client =================================== Now that we have covered the most important function (the deployment), let's list down what other actions we should be able to do with any Pipeline Service Client. * `getServiceStatus`: to check if we can properly reach the configured Orchestrator. * `testConnection`: as an endpoint that allows us to test the connectivity from the Orchestrator to a specific service. * `deployPipeline`: as explained above, to dynamically created a DAG in the Orchestrator. * `runPipeline`: to trigger a DAG. * `deletePipeline`: to delete a DAG. * `toggleIngestion`: to pause or unpause a DAG from future executions. * `getPipelineStatus`: to check the status of the latest runs of the DAG. * `getLastIngestionLogs`: to pick up the logs of the latest execution. * `killIngestion`: To kill all queued or ongoing runs of a DAG. * `requestGetHostIp`: To get the pipeline service host IP. This can either be statically picked up from the OM YAML configuration or if the Orchestrator supports it, retrieved from there. --- # Metadata Ingestion Workflow | Official Documentation We use cookies to improve site navigation, analyze site usage, and enhance your user experience. Click "Accept" to enable cookies or "Reject" to reject cookies. AcceptReject connectors No menu items for this category [Home](https://docs.open-metadata.org/latest) /[Connectors](https://docs.open-metadata.org/latest/connectors) /[Ingestion](https://docs.open-metadata.org/latest/connectors/ingestion) /[Workflows](https://docs.open-metadata.org/latest/connectors/ingestion/workflows) /[Metadata](https://docs.open-metadata.org/latest/connectors/ingestion/workflows/metadata) OpenMetadata Documentation Metadata Ingestion Workflow =========================== The easiest way to extract metadata is to use any of our connectors! [Metadata Connectors\ \ Configure your automated Metadata extraction.](https://docs.open-metadata.org/latest/connectors) If you want to learn more about how to extract metadata from dbt, we have you covered: [dbt Ingestion\ \ Extract Metadata and ingest your dbt models.](https://docs.open-metadata.org/latest/connectors/ingestion/workflows/dbt) --- # Lineage Workflow Through Query Logs | Official Documentation We use cookies to improve site navigation, analyze site usage, and enhance your user experience. Click "Accept" to enable cookies or "Reject" to reject cookies. AcceptReject connectors No menu items for this category [Home](https://docs.open-metadata.org/latest) /[Connectors](https://docs.open-metadata.org/latest/connectors) /[Ingestion](https://docs.open-metadata.org/latest/connectors/ingestion) /[Workflows](https://docs.open-metadata.org/latest/connectors/ingestion/workflows) /[Lineage](https://docs.open-metadata.org/latest/connectors/ingestion/workflows/lineage) /[Lineage Workflow Query Logs](https://docs.open-metadata.org/latest/connectors/ingestion/workflows/lineage/lineage-workflow-query-logs) OpenMetadata Documentation Lineage Workflow Through Query Logs =================================== In order to extract lineage information, OpenMetadata parses the queries that have run against the database. This query log information is available from WITHIN the database in the following services: * [BigQuery](https://docs.open-metadata.org/latest/connectors/database/bigquery) * [Snowflake](https://docs.open-metadata.org/latest/connectors/database/snowflake) * [MSSQL](https://docs.open-metadata.org/latest/connectors/database/mssql) * [Redshift](https://docs.open-metadata.org/latest/connectors/database/redshift) * [Clickhouse](https://docs.open-metadata.org/latest/connectors/database/clickhouse) * [Databricks](https://docs.open-metadata.org/latest/connectors/database/databricks) * [PostgreSQL](https://docs.open-metadata.org/latest/connectors/database/postgres) If you are using any other database connector, direct execution of lineage workflow is not possible. This is mainly because these database connectors does not maintain query execution logs which is required for lineage workflow. If you are interested in running the lineage workflow for a connector not listed above, this documentation will help you to execute the lineage workflow using a query log file. This can be arbitrarily executed for **any** database connector. Query Log File -------------- A query log file is a standard CSV file which contains the following information. A standard CSV should be comma separated, and each row represented as a single line in the file. * **query\_text:** This field contains the literal query that has been executed in the database. It is quite possible that your query has commas `,` inside. Then, wrap each query in quotes to not have any clashes with the comma as a separator. * **database\_name (optional):** Enter the database name on which the query was executed. * **schema\_name (optional):** Enter the schema name to which the query is associated. Checkout a sample query log file [here](https://github.com/open-metadata/OpenMetadata/blob/main/ingestion/examples/sample_data/glue/query_log.csv) . Lineage Workflow ---------------- In order to run a Lineage Workflow we need to make sure that Metadata Ingestion Workflow for corresponding service has already been executed. We will follow the steps to create a JSON configuration able to collect the query log file and execute the lineage workflow. ### 1\. Create a configuration file using template YAML Create a new file called `query_log_lineage.yaml` in the current directory. Note that the current directory should be the openmetadata directory. Copy and paste the configuration template below into the `query_log_lineage.yaml` the file you created. The `serviceName` should be a service already ingested in OpenMetadata. * **queryLogFilePath**: Enter the file path of query log csv file. ### 2\. Run with the CLI First, we will need to save the YAML file. Afterward, and with all requirements installed, we can run: Note that from connector-to-connector, this recipe will always be the same. By updating the YAML configuration, you will be able to extract metadata from different sources. --- # Usage Workflow Through Query Logs We use cookies to improve site navigation, analyze site usage, and enhance your user experience. Click "Accept" to enable cookies or "Reject" to reject cookies. AcceptReject connectors No menu items for this category [Home](https://docs.open-metadata.org/latest) /[Connectors](https://docs.open-metadata.org/latest/connectors) /[Ingestion](https://docs.open-metadata.org/latest/connectors/ingestion) /[Workflows](https://docs.open-metadata.org/latest/connectors/ingestion/workflows) /[Usage](https://docs.open-metadata.org/latest/connectors/ingestion/workflows/usage) /[Usage Workflow Query Logs](https://docs.open-metadata.org/latest/connectors/ingestion/workflows/usage/usage-workflow-query-logs) OpenMetadata Documentation Usage Workflow Through Query Logs ================================= In order to extract usage information, OpenMetadata parses the queries that have run against the database. This query log information is available from WITHIN the database in the following services: * [Athena](https://docs.open-metadata.org/latest/connectors/database/athena) * [BigQuery](https://docs.open-metadata.org/latest/connectors/database/bigquery) * [Snowflake](https://docs.open-metadata.org/latest/connectors/database/snowflake) * [MSSQL](https://docs.open-metadata.org/latest/connectors/database/mssql) * [Redshift](https://docs.open-metadata.org/latest/connectors/database/redshift) * [Clickhouse](https://docs.open-metadata.org/latest/connectors/database/clickhouse) * [Databricks](https://docs.open-metadata.org/latest/connectors/database/databricks) * [PostgreSQL](https://docs.open-metadata.org/latest/connectors/database/postgres) If you are using any other database connector, direct execution of Usage Workflow is not possible. This is mainly because these database connectors does not maintain query execution logs which is required for Usage Workflow. If you are interested in running the usage workflow for a connector not listed above, this documentation will help you to execute the Usage Workflow using a query log file. This can be arbitrarily executed for **any** database connector. Query Log File -------------- A query log file is a standard CSV file which contains the following information. A standard CSV should be comma separated, and each row represented as a single line in the file. * **query\_text:** This field contains the literal query that has been executed in the database. It is quite possible that your query has commas `,` inside. Then, wrap each query in quotes to not have any clashes with the comma as a separator. * **user\_name (optional):** Enter the database user name which has executed this query. * **start\_time (optional):** Enter the query execution start time in YYYY-MM-DD HH:MM:SS format. * **end\_time (optional):** Enter the query execution end time in YYYY-MM-DD HH:MM:SS format. * **aborted (optional):** This field accepts values as true or false and indicates whether the query was aborted during execution * **database\_name (optional):** Enter the database name on which the query was executed. * **schema\_name (optional):** Enter the schema name to which the query is associated. Checkout a sample query log file [here](https://github.com/open-metadata/OpenMetadata/blob/main/ingestion/examples/sample_data/glue/query_log.csv) . Usage Workflow -------------- In order to run a Usage Workflow we need to make sure that Metadata Ingestion Workflow for corresponding service has already been executed. We will follow the steps to create a JSON configuration able to collect the query log file and execute the usage workflow. ### 1\. Create a configuration file using template YAML Create a new file called `query_log_usage.yaml` in the current directory. Note that the current directory should be the openmetadata directory. Copy and paste the configuration template below into the `query_log_usage.yaml` the file you created. The `serviceName` should be a service already ingested in OpenMetadata. * **queryLogFilePath**: Enter the file path of query log csv file. ### 2\. Run with the CLI First, we will need to save the YAML file. Afterward, and with all requirements installed, we can run: Note that from connector-to-connector, this recipe will always be the same. By updating the YAML configuration, you will be able to extract metadata from different sources. --- # Kubernetes Deployment | Official Documentation We use cookies to improve site navigation, analyze site usage, and enhance your user experience. Click "Accept" to enable cookies or "Reject" to reject cookies. AcceptReject deployment No menu items for this category [Home](https://docs.open-metadata.org/latest) /[Deployment](https://docs.open-metadata.org/latest/deployment) /[Kubernetes](https://docs.open-metadata.org/latest/deployment/kubernetes) OpenMetadata Documentation Kubernetes Deployment ===================== OpenMetadata supports the Installation and Running of Application on kubernetes through Helm Charts. Kubernetes Deployment Architecture ---------------------------------- Below is the expected Kubernetes Deployment Architecture for OpenMetadata Application in **Production**. ![Kubernetes Deployment Architecture](https://docs.open-metadata.org/images/v1.11/deployment/kubernetes/kubernetes-architecture-prod.png) In the above architecture diagram, OpenMetadata Application is deployed using Helm Charts. The various kubernetes manifests that supports the installation. With the above architecture, OpenMetadata Application Connects with external dependencies which is Database, ElasticSearch and Orchestration tools like airflow. The OpenMetadata Helm Charts Exposes the Application from Kubernetes Service at Port `8585` and `8586`. The Health Checks and Metrics endpoints are available on port `8586`. Network Policies and Ingresses are optional manifests and disabled by default. These can be installed / enabled using the [Helm Values](https://docs.open-metadata.org/latest/deployment/kubernetes/helm-values) . Links ----- [Helm Values\ \ For customizing OpenMetadata Helm Deployments](https://docs.open-metadata.org/latest/deployment/kubernetes/helm-values) [Deploy in AWS EKS\ \ Deploy OpenMetadata in AWS Kubernetes](https://docs.open-metadata.org/latest/deployment/kubernetes/eks) [Deploy in GCP GKE\ \ Deploy OpenMetadata in GCP Kubernetes](https://docs.open-metadata.org/latest/deployment/kubernetes/gke) [Deploy in Azure AKS\ \ Deploy OpenMetadata in Azure Kubernetes](https://docs.open-metadata.org/latest/deployment/kubernetes/aks) [Deploy in OnPremises Kubernetes\ \ Deploy OpenMetadata in On Premises Kubernetes](https://docs.open-metadata.org/latest/deployment/kubernetes/on-prem) --- # Metadata Ingestion | OpenMetadata Data Pipeline Overview We use cookies to improve site navigation, analyze site usage, and enhance your user experience. Click "Accept" to enable cookies or "Reject" to reject cookies. AcceptReject connectors No menu items for this category [Home](https://docs.open-metadata.org/latest) /[Connectors](https://docs.open-metadata.org/latest/connectors) /[Ingestion](https://docs.open-metadata.org/latest/connectors/ingestion) OpenMetadata Documentation Metadata Ingestion ================== The goal of OpenMetadata is to serve as a centralised platform where users can gather and collaborate around data. This is possible thanks for different workflows that users can deploy and schedule, which will connect to the data sources to extract metadata. Different metadata being ingested to OpenMetadata can be: * Entities metadata, such as Tables, Dashboards, Topics... * Query usage to rank the most used tables, * Lineage between Entities, * Data Profiles and Quality Tests. In this section we will explore the different workflows, how they work and how to use them. [Metadata Ingestion\ \ Learn more about how to ingest metadata from dozens of connectors.](https://docs.open-metadata.org/latest/connectors/ingestion/workflows/metadata) [Metadata Profiler\ \ To get metrics from your Tables=](https://docs.open-metadata.org/latest/how-to-guides/data-quality-observability/profiler/workflow) [Metadata Data Quality Tests\ \ To run automated Quality Tests on your Tables.](https://docs.open-metadata.org/latest/how-to-guides/data-quality-observability/quality) [Metadata Usage\ \ To analyze popular entities.](https://docs.open-metadata.org/latest/connectors/ingestion/workflows/usage) [Metadata Lineage\ \ To analyze relationships in your data platform.](https://docs.open-metadata.org/latest/connectors/ingestion/workflows/lineage) Great Expectations ------------------ [Great Expectations](https://greatexpectations.io/) is a shared, open standard for data quality. It helps data teams eliminate pipeline debt, through data testing, documentation, and profiling. Learn how to configure Great Expectations to integrate with OpenMetadata and ingest your test results to your table service page. [Great Expectations\ \ Ingest your test results from Great Expectations.](https://docs.open-metadata.org/latest/connectors/ingestion/great-expectations) Metadata Versioning ------------------- One fundamental aspect of Metadata Ingestion is being able to analyze the evolution of your metadata. OpenMetadata support Metadata Versioning, maintaining the history of changes of all your assets. [Metadata Versioning\ \ Learn how OpenMetadata keeps track of your metadata evolution.](https://docs.open-metadata.org/latest/connectors/ingestion/versioning) Best Practices -------------- You want to know some of the best practices around metadata ingestion? This is the right place! [Best Practices\ \ Learn the best practices to ingest metadata, both from the UI and using any custom orchestrator.](https://docs.open-metadata.org/latest/connectors/ingestion/best-practices) Deep Dive --------- Understand how OpenMetadata deploys the workflows that are created from the UI. [Ingestion Pipeline UI Deployment\ \ Learn about the Pipeline Service interface and how OpenMetadata handles workflow deployments.](https://docs.open-metadata.org/latest/connectors/ingestion/deployment) --- # High Level Design | OpenMetadata Architecture Overview We use cookies to improve site navigation, analyze site usage, and enhance your user experience. Click "Accept" to enable cookies or "Reject" to reject cookies. AcceptReject main-concepts No menu items for this category [Home](https://docs.open-metadata.org/latest) /[Main Concepts](https://docs.open-metadata.org/latest/main-concepts) /[High Level Design](https://docs.open-metadata.org/latest/main-concepts/high-level-design) OpenMetadata Documentation High Level Design ================= This Solution Design document will help us explore and understand the internals of OpenMetadata services, how are they built and their interactions. We will start by describing the big picture of the software design of the application. Bit by bit we will get inside specific components, describing their behaviour and showing examples on how to use them. System Context -------------- The goal of this first section is to get familiar with the high-level concepts and technologies involved. The learning objectives here are: * Describe the elements that compose OpenMetadata and their relationships. * How end-users and external applications can communicate with the system. Here we have the main actors of the solution: ![system-context](https://docs.open-metadata.org/images/v1.11/main-concepts/high-level-design/system-context.png) * **API**: This is the main pillar of OpenMetadata. Here we have defined how we can interact with the metadata Entities. It powers all the other components of the solution. * **UI**: Discovery-focused tool that helps users keep track of all the data assets in the organisation. Its goal is enabling and fueling collaboration. * **Ingestion Framework**: Based on the API specifications, this system is the foundation of all the Connectors, i.e., the components that define the interaction between OpenMetadata and external systems containing the metadata we want to integrate. * **Entity Store**: MySQL storage that contains real-time information on the state of all the Entities and their Relationships. * **Search Engine**: Powered by ElasticSearch, it is the indexing system for the UI to help users discover the metadata. JSON Schemas ------------ If we abstract away from the Storage Layer for a moment, we then realize that the OpenMetadata implementation is the integration of three blocks: * The core **API**, unifying and centralising the communication with internal and external systems. * The **UI** for a team-centric metadata Serving Layer. * The **Ingestion Framework** as an Interface between OpenMetadata and external sources. The only thing these components have in common is the **vocabulary** -> All of them are shaping, describing, and moving around metadata Entities. OpenMetadata is based on a **standard definition** for metadata. Therefore, we need to make sure that in our implementation of this standard we share this definition in the end-to-end workflow. To this end, the main lexicon is defined as JSON Schemas, a readable and language-agnostic solution. Then, when packaging the main components, we generate the specific programming classes for all the Entities. What we achieve is three views from the same source: * Java Classes for the API, * Python Classes for the Ingestion Framework and * TypeScript Types for the UI, each of them modeled after a single source of truth. Thanks to this approach we can be sure that it does not matter at which point we zoom in throughout the whole process, we are always going to find a univocal well-defined Entity. API Container Diagram --------------------- Now we are going to zoom inside the API Container. As the central Software System of the solution, its goal is to manage calls (both from internal and external sources, e.g., Ingestion Framework or any custom integration) and update the state of the metadata Entities. While the data is stored in the MySQL container, the API will be the one fetching it and completing the necessary information, validating the Entities data and all the relationships. Having a Serving Layer (API) decoupled from the Storage Layer allows users and integrations to ask for what they need in a simple language (REST), without the learning curve of diving into specific data models and design choices. ![api-container-diagram](https://docs.open-metadata.org/images/v1.11/main-concepts/high-level-design/api-container-diagram.png) Entity Resource --------------- When we interact with most of our Entities, we follow the same endpoint structure. For example: * `GET /api/v1//` to retrieve an Entity instance by ID, or * `GET /api/v1//name/` to query by its fully qualified domain name. Similarly, we support other CRUD operations, each of them expecting a specific incoming data structure, and returning the Entity's class. As the foundations of OpenMetadata are the Entities definitions, we have this data contract with any consumer, where the backend will validate the received data, as well as the outputs. The endpoint definition and datatype setting are what happens at the Entity Resource. Each metadata Entity is packed with a Resource class, which builds the API definition for the given Entity. This logic is what then surfaces in the [API docs](https://docs.open-metadata.org/swagger.html) . Entity Repository ----------------- The goal of the Entity Repository is to perform Read & Write operations to the **backend database** to Create, Retrieve, Update and Delete Entities. While the Entity Resource handles external communication, the Repository is in charge of managing how the whole process interacts with the Storage Layer, making sure that incoming and outgoing Entities are valid and hold proper and complete information. This means that here is where we define our **DAO** (Data Access Object), with all the validation and data storage logic. As there are processes repeated across all Entities (e.g., listing entities in a collection or getting a specific version from an Entity), the Entity Repository extends an **Interface** that implements some basic functionalities and abstracts Entity specific logic. Each Entity then needs to implement its **server-side processes** such as building the FQN based on the Entity hierarchy, how the Entity stores and retrieves **Relationship** information with other Entities or how the Entity reacts to **Change Events**. Entity Storage Layer -------------------- In the API Container Diagram, we showed how the Entity Repository interacts with three different Storage Containers (tables) depending on what type of information is being processed. To fully understand this decision, we should first talk about the information contained by Entities instances. An Entity has two types of fields: **attributes** (JSON Schema properties) and **relationships** (JSON Schema href): * **Attributes** are the core properties of the Entity: the name and id, the columns for a table, or the algorithm for an ML Model. Those are intrinsic pieces of information of an Entity and their existence and values are what help us differentiate both Entity instances (Table A vs. Table B) and Entity definitions (Dashboard vs. Topic). * **Relationships** are associations between two Entities. For example, a Table belongs to a Database, a User owns a Dashboard, etc. Relationships are a special type of attribute that is captured using Entity References. Entity and Relationship Store ----------------------------- Entities are stored as JSON documents in the database. Each entity has an associated table (`_entity`) which contains the JSON defining the Entity attributes and other metadata fields, such as the id, `updatedAt` or `updatedBy`. This JSON does not store any Relationship. E.g., a User owning a Dashboard is a piece of information that is materialised in a separate table entity\_relationship as graph nodes, where the edge holds the type of the Relationship (e.g., `contains`, `uses`, `follows`...). This separation helps us decouple concerns. We can process related entities independently and validate at runtime what information needs to be updated and/or retrieved. For example, if we delete a Dashboard being owned by a User, we will then clean up this row in `entity_relationship`, but that won't alter the information from the User. Another trickier example would be trying to delete a Database that contains Tables. In this case, the process would check that the Database Entity is not empty, and therefore we cannot continue with the removal. Change Events Store ------------------- You might have already noticed that in all Entities definitions we have a `changeDescription` field. It is defined as "Change that leads to this version of the entity". If we inspect further the properties of `changeDescription`, we can see how it stores the differences between the current and last versions of an Entity. This results in giving visibility on the last update step of each Entity instance. However, there might be times when this level of tracking is not enough. One of the greatest features of OpenMetadata is the ability to track all Entity versions. Each operation that leads to a change (`PUT`, `POST`, `PATCH`) will generate a trace that is going to be stored in the table `change_event`. Using the API to get events data, or directly exploring the different versions of each entity gives great debugging power to both data consumers and producers. API Component Diagram --------------------- Now that we have a clear picture of the main pieces and their roles, we will analyze the logical flow of a `POST` and a `PUT` calls to the API. The main goal of this section is to get familiar with the code organisation and its main steps. To take the most out of this section, it is recommended to follow the source code as well, from the Entity JSON you'd like to use as an example to its implementation of Resource and Repository. ### Create a new Entity - POST We will start with the simplest scenario: Creating a new Entity via a `POST` call. This is a great first point to review as part of the logic and methods are reused during updates. ![create-new-entity](https://docs.open-metadata.org/images/v1.11/main-concepts/high-level-design/create-new-entity.png) #### Create As we already know, the recipient of the HTTP call will be the `EntityResource`. In there, we have the create function with the @POST annotation and the description of the API endpoint and expected schemas. The role of this first component is to receive the call and validate the request body and headers, but the real implementation happens in the `EntityRepository`, which we already described as the **DAO**. For the `POST` operation, the internal flow is rather simple and is composed of two steps: * **Prepare**: Which validates the Entity data and computes some attributes at the server-side. * **Store**: This saves the Entity JSON and its Relationships to the backend DB. #### Prepare This method is used for validating an entity to be created during `POST`, `PUT`, and `PATCH` operations and preparing the entity with all the required attributes and relationships. Here we handle, for example, the process of setting up the FQN of an Entity based on its hierarchy. While all Entities require an FQN, this is not an attribute we expect to receive in a request. Moreover, this checks that the received attributes are being correctly informed, e.g., we have a valid `User` as an `owner` or a valid `Database` for a `Table`. #### Store The storing process is divided into two different steps (as we have two tables holding the information). We strip the validated Entity from any `href` attribute (such as `owner` or `tags`) in order to just store a JSON document with the Entity intrinsic values. We then store the graph representation of the Relationships for the attributes omitted above. At the end of these calls, we end up with a validated Entity holding all the required attributes, which have been validated and stored accordingly. We can then return the created Entity to the caller. ### Create or Update an Entity - PUT Let's now build on top of what we learned during the `POST` discussion, expanding the example to a `PUT` request handling. ![create-update-entity](https://docs.open-metadata.org/images/v1.11/main-concepts/high-level-design/create-or-update.png) The first steps are fairly similar: 1. We have a function in our `Resource` annotated as `@PUT` and handling headers, auth and schemas. 2. The `Resource` then calls the DAO at the Repository, bootstrapping the data-related logic. 3. We validate the Entity and cook some attributes during the prepare step. After processing and validating the Entity request, we then check if the Entity instance has already been stored, querying the backend database by its FQN. If it has not, then we proceed with the same logic as the `POST` operation -> simple creation. Otherwise, we need to validate the updated fields. #### Set Fields We cannot allow all fields to be updated for a given Entity instance. For example, the `id` or `name` stay immutable once the instance is created, and the same thing happens to the `Database` of a `Table`. The list of specified fields that can change is defined at each Entity's Repository, and we should only allow changes on those attributes that can naturally evolve throughout the lifecycle of the object. At this step, we set the fields to the Entity that are either required by the JSON schema definition (e.g., the algorithm for an `MlModel`) or, in the case of a `GET` operation, that are requested as `GET /api/v1//?fields=field1,field2...` #### Update In the `EntityRepository` there is an abstract implementation of the `EntityUpdater` interface, which is in charge of defining the generic update logic flow common for all the Entities. The main steps handled in the update calls are: **1.** Update the Entity **generic** fields, such as the description or the owner. **2.** Run Entity **specific** updates, which are implemented by each Entity's `EntityUpdater` extension. **3.** **Store** the updated Entity JSON doc to the Entity Table in MySQL. #### Entity Specific Updates Each Entity has a set of attributes that define it. These attributes are going to have a very specific behaviour, so the implementation of the `update` logic falls to each Entity Repository. For example, we can update the `Columns` of a `Table`, or the `Dashboard` holding the performance metrics of an `MlModel`. Both of these changes are going to be treated differently, in terms of how the Entity performs internally the update, how the Entity version gets affected, or the impact on the **Relationship** data. For the sake of discussion, we'll follow a couple of update scenarios. #### Example 1 - Updating Columns of a Table When updating `Columns`, we need to compare the existing set of columns in the original Entity vs. the incoming columns of the `PUT` request. If we are receiving an existing column, we might need to update its description or tags. This change will be considered a minor change. Therefore, the version of the Entity will be bumped by 0.1, following the software release specification model. However, what happens if a stored column is not received in the updated instance? That would mean that such a column has been deleted. This is a type of change that could possibly break integrations on top of the `Table`'s data. Therefore, we can mark this scenario as a major update. In this case, the version of the Entity will increase by `1.0`. Checking the Change Events or visiting the Entity history will easily show us the evolution of an Entity instance, which will be immensely valuable when debugging data issues. #### Example 2 - Updating the Dashboard of an ML Model One of the attributes for an MlModel is the `EntityReference` to a `Dashboard` holding its performance metrics evolution. As this attribute is a reference to another existing Entity, this data is not directly stored in the `MlModel` JSON doc, but rather as a Relationship graph, as we have been discussing previously. Therefore, during the update step we will need to: **1.** Insert the relationship, if the original Entity had no `Dashboard` informed, **2.** Delete the relationship if the `Dashboard` has been removed, or **3.** Update the relationship if we now point to a different `Dashboard`. Note how during the `POST` operation we needed to always call the `storeRelationship` function, as it was the first time we were storing the instance's information. During an update, we will just modify the Relationship data if the Entity's specific attributes require it. Handling Events --------------- During all these discussions and examples we've been showing how the backend API handles HTTP requests and what the Entities' data lifecycle is. Not only we've been focusing on the JSON docs and **Relationships**, but from time to time we have talked about Change Events. Moreover, In the API Container Diagram we drew a Container representing the `Table` holding the Change Event data, but yet, we have not found any Component accessing it. This is because the API server is powered by Jetty, which means that luckily we do not need to make those calls ourselves! By defining a `ChangeEventHandler` and registering it during the creation of the server, this postprocessing of the calls happens transparently. Our `ChangeEventHandler` will check if the Entity has been `Created`, `Updated` or `Deleted` and will store the appropriate `ChangeEvent` data from our response to the backend DB. --- # dbt Workflow | OpenMetadata Data Build Tool Integration We use cookies to improve site navigation, analyze site usage, and enhance your user experience. Click "Accept" to enable cookies or "Reject" to reject cookies. AcceptReject connectors No menu items for this category [Home](https://docs.open-metadata.org/latest) /[Connectors](https://docs.open-metadata.org/latest/connectors) /[Ingestion](https://docs.open-metadata.org/latest/connectors/ingestion) /[Workflows](https://docs.open-metadata.org/latest/connectors/ingestion/workflows) /[Dbt](https://docs.open-metadata.org/latest/connectors/ingestion/workflows/dbt) OpenMetadata Documentation dbt Workflow ============ [Configure dbt workflow from OpenMetadata UI\ \ Configure the dbt Workflow from the UI.](https://docs.open-metadata.org/latest/connectors/ingestion/workflows/dbt/configure-dbt-workflow-from-ui) [Run dbt Workflow Externally\ \ Configure the dbt Workflow from the CLI.](https://docs.open-metadata.org/latest/connectors/ingestion/workflows/dbt/run-dbt-workflow-externally) [Auto Ingest dbt Artifacts (dbt-core)\ \ Configure the auto dbt ingestion for dbt-core.](https://docs.open-metadata.org/latest/connectors/ingestion/workflows/dbt/auto-ingest-dbt-core) dbt Integration =============== | Feature | Status | | --- | --- | | Stage | PROD | | dbt Queries | | | dbt Lineage | | | dbt Tags | | | dbt Tiers | | | dbt Glossary | | | dbt Owner | | | dbt Descriptions | | | dbt Tests | | | dbt Exposures | | | dbt Domains | | | dbt Custom Properties | | | Supported dbt Core Versions | `v1.2` `v1.3` `v1.5` `v1.5` `v1.6` `v1.7` `v1.8` `v1.9` | Requirements ------------ ### AWS S3 If we have the artifacts on the bucket `MyBucket`, the user running the ingestion should have, at least, the permissions from the following policy: Note that it's not enough to point the resource to `arn:aws:s3:::MyBucket`. We need its contents as well! OpenMetadata integrates the below metadata from dbt --------------------------------------------------- ### 1\. dbt Queries Queries used to create the dbt models can be viewed in the dbt tab ![dbt-query](https://docs.open-metadata.org/images/v1.11/features/ingestion/workflows/dbt/dbt-features/dbt-query.webp) dbt Query ### 2\. dbt Lineage Lineage from dbt models can be viewed in the Lineage tab. For more information on how lineage is extracted from dbt take a look [here](https://docs.open-metadata.org/latest/connectors/ingestion/workflows/dbt/ingest-dbt-lineage) ![dbt-lineage](https://docs.open-metadata.org/images/v1.11/features/ingestion/workflows/dbt/dbt-features/dbt-lineage.webp) dbt Lineage To capture lineage, the `compiled_code` field must be present in the `manifest.json` file. * If `compiled_code` is missing, lineage will **not** be captured for that node. * To ensure `compiled_code` is populated in your dbt manifest, run the following commands in your dbt project: * `dbt compile` * `dbt docs generate` ### 3\. dbt Tags Table and column level tags can be imported from dbt Please refer [here](https://docs.open-metadata.org/latest/connectors/ingestion/workflows/dbt/ingest-dbt-tags) for adding dbt tags ![dbt-tags](https://docs.open-metadata.org/images/v1.11/features/ingestion/workflows/dbt/dbt-features/dbt-tags.png) dbt Tags ### 4\. dbt Owner Owner from dbt models can be imported and assigned to respective tables Please refer [here](https://docs.open-metadata.org/latest/connectors/ingestion/workflows/dbt/ingest-dbt-owner) for adding dbt owner ![dbt-owner](https://docs.open-metadata.org/images/v1.11/features/ingestion/workflows/dbt/dbt-features/dbt-owner.webp) dbt Owner ### 5\. dbt Descriptions Descriptions from dbt `manifest.json` and `catalog.json` can be imported and assigned to respective tables and columns. For more information and to control how the table and column descriptions are updated from dbt please take a look [here](https://docs.open-metadata.org/latest/connectors/ingestion/workflows/dbt/ingest-dbt-descriptions) ![dbt-descriptions](https://docs.open-metadata.org/images/v1.11/features/ingestion/workflows/dbt/dbt-features/dbt-descriptions.webp) dbt Descriptions ### 6\. dbt Tests and Test Results Tests from dbt will only be imported if the `run_results.json` file is passed. ![dbt-tests](https://docs.open-metadata.org/images/v1.11/features/ingestion/workflows/dbt/dbt-features/dbt-tests.webp) dbt Tests ### 7\. dbt Tiers Table and column level Tiers can be imported from dbt Please refer [here](https://docs.open-metadata.org/latest/connectors/ingestion/workflows/dbt/ingest-dbt-tier) for adding dbt tiers ![dbt-tiers](https://docs.open-metadata.org/images/v1.11/features/ingestion/workflows/dbt/dbt-features/dbt-tier.png) dbt Tiers ### 8\. dbt Glossary Table and column level Glossary can be imported from dbt Please refer [here](https://docs.open-metadata.org/latest/connectors/ingestion/workflows/dbt/ingest-dbt-glossary) for adding dbt glossary ![dbt-glossary](https://docs.open-metadata.org/images/v1.11/features/ingestion/workflows/dbt/dbt-features/dbt-glossary.png) dbt Glossary ### 9\. dbt Domains Domains from dbt models can be assigned to tables for better data organization and governance Please refer [here](https://docs.open-metadata.org/latest/connectors/ingestion/workflows/dbt/ingest-dbt-domain) for adding dbt domains ### 10\. dbt Custom Properties Custom property values can be assigned to tables from dbt to enrich metadata with organization-specific attributes Please refer [here](https://docs.open-metadata.org/latest/connectors/ingestion/workflows/dbt/ingest-dbt-custom-properties) for adding dbt custom properties Troubleshooting --------------- For any issues please refer to the troubleshooting documentation [here](https://docs.open-metadata.org/latest/connectors/ingestion/workflows/dbt/dbt-troubleshooting) --- # Kubernetes On Premises Deployment | Official Documentation We use cookies to improve site navigation, analyze site usage, and enhance your user experience. Click "Accept" to enable cookies or "Reject" to reject cookies. AcceptReject deployment No menu items for this category [Home](https://docs.open-metadata.org/latest) /[Deployment](https://docs.open-metadata.org/latest/deployment) /[Kubernetes](https://docs.open-metadata.org/latest/deployment/kubernetes) /[On Prem](https://docs.open-metadata.org/latest/deployment/kubernetes/on-prem) OpenMetadata Documentation On Premises Kubernetes Deployment ================================= OpenMetadata supports the Installation and Running of application on OnPremises Kubernetes through Helm Charts. However, there are some additional configurations which needs to be done as prerequisites for the same. This guide presumes you have an on premises Kubernetes cluster setup, and you are installing OpenMetadata in `default` namespace. Prerequisites ------------- ### External Database and Search Engine as ElasticSearch / OpenSearch We support * MySQL engine version 8 or higher * PostgreSQL engine version 12 or higher * ElasticSearch version 8.X (upto 8.11.4) or OpenSearch Version 2.X (upto 2.19) Once you have the External Database and Search Engine configured, you can update the environment variables below for OpenMetadata kubernetes deployments to connect with Database and ElasticSearch. Make sure to create database and search engine credentials as Kubernetes Secrets mentioned [here](https://docs.open-metadata.org/latest/quick-start/local-kubernetes-deployment#2.-create-kubernetes-secrets-required-for-helm-charts) . Also, disable MySQL and ElasticSearch from OpenMetadata Dependencies Helm Charts as mentioned in the FAQs [here](https://docs.open-metadata.org/latest/deployment/kubernetes/on-prem#how-to-disable-mysql-and-elasticsearch-from-openmetadata-dependencies-helm-charts) . ### Persistent Volumes with ReadWriteMany Access Modes OpenMetadata helm chart depends on Airflow and Airflow expects a persistent disk that support ReadWriteMany (the volume can be mounted as read-write by many nodes). The workaround is to create nfs-share and use that as the persistent claim to deploy OpenMetadata by implementing the following steps in order. This guide assumes you have NFS Server already setup with Hostname or IP Address which is reachable from your on premises Kubernetes cluster, and you have configured a path to be used for OpenMetadata Airflow Helm Dependency. ### Dynamic Provisioning using StorageClass To provision PersistentVolume dynamically using the StorageClass, you need to install the NFS provisioner. It is recommended to use [nfs-subdir-external-provisioner](https://github.com/kubernetes-sigs/nfs-subdir-external-provisioner) helm charts for this case. Replace the `NFS_HOSTNAME_OR_IP` with your NFS Server value and run the commands. This will create a new StorageClass with `nfs-subdir-external-provisioner`. You can view the same using the kubectl command `kubectl get storageclass -n nfs-provisioner`. Provision NFS backed PVC for Airflow DAGs and Airflow Logs ---------------------------------------------------------- ### Code Samples for PVC for Airflow DAGs Create Persistent Volumes and Persistent Volume claims with the below command. ### Code Samples for PVC for Airflow Logs Create Persistent Volumes and Persistent Volume claims with the below command. Change owner and permission manually on disks --------------------------------------------- Since airflow pods run as non-root users, they would not have write access on the nfs server volumes. In order to fix the permission here, spin up a pod with persistent volumes attached and run it once. Airflow runs the pods with linux username as airflow and linux user id as 50000. Run the below command to create the pod and fix the permissions Create OpenMetadata dependencies Values --------------------------------------- Override openmetadata dependencies airflow helm values to bind the nfs persistent volumes for DAGs and logs. For more information on airflow helm chart values, please refer to [airflow-helm](https://artifacthub.io/packages/helm/airflow-helm/airflow/8.8.0) . When deploying openmetadata dependencies helm chart, use the below command - The above command uses configurations defined [here](https://raw.githubusercontent.com/open-metadata/openmetadata-helm-charts/main/charts/deps/values.yaml) . You can modify any configuration and deploy by passing your own `values.yaml` Once the openmetadata dependencies helm chart deployed, you can then run the below command to install the openmetadata helm chart - Again, this uses the values defined [here](https://github.com/open-metadata/openmetadata-helm-charts/blob/main/charts/openmetadata/values.yaml) . Use the `--values` flag to point to your own YAML configuration if needed. Troubleshooting --------------- Starting with **OpenMetadata v1.11.4**, the dependency Helm chart no longer supports passing database passwords using individual Kubernetes secret keys (for example, `passwordSecret` and `passwordSecretKey`). Instead, database credentials must be provided via a **single Kubernetes Secret** referenced using `metadataSecretName`. This secret must contain the **full database connection string**, including the password. This change applies **only to the dependency `values.yml` configuration** and aligns with the database configuration approach used by the **Airflow Helm chart**. FAQs ==== Java Memory Heap Issue ---------------------- If your openmetadata pods are not in ready state at any point in time and the openmetadata pod logs speaks about the below issue - This is due to the default JVM Heap Space configuration (1 GiB) being not enough for your workloads. In order to resolve this issue, head over to your custom openmetadata helm values and append the below environment variable The flag `Xmx` specifies the maximum memory allocation pool for a Java virtual machine (JVM), while `Xms` specifies the initial memory allocation pool. Upgrade the helm charts with the above changes using the following command `helm upgrade --install openmetadata open-metadata/openmetadata --values --namespace `. Update this command your `values.yml` filename and `namespaceName` where you have deployed OpenMetadata in Kubernetes. PostgreSQL Issue permission denied to create extension "pgcrypto" ----------------------------------------------------------------- If you are facing the below issue with PostgreSQL as Database Backend for OpenMetadata Application, It seems the Database User does not have sufficient privileges. In order to resolve the above issue, grant usage permissions to the PSQL User. In the above command, replace `` with the sql user used by OpenMetadata Application to connect to PostgreSQL Database. How to extend and use custom docker images with OpenMetadata Helm Charts ? -------------------------------------------------------------------------- Extending OpenMetadata Server Docker Image ------------------------------------------ ### 1\. Create a `Dockerfile` based on `docker.getcollate.io/openmetadata/server` OpenMetadata helm charts uses official published docker images from [DockerHub](https://hub.docker.com/u/openmetadata) . A typical scenario will be to install organization certificates for connecting with inhouse systems. For Example - where `docker.getcollate.io/openmetadata/server:x.y.z` needs to point to the same version of the OpenMetadata server, for example `docker.getcollate.io/openmetadata/server:1.3.1`. This image needs to be built and published to the container registry of your choice. ### 2\. Update your openmetadata helm values yaml The OpenMetadata Application gets installed as part of `openmetadata` helm chart. In this step, update the custom helm values using YAML file to point the image created in the previous step. For example, create a helm values file named `values.yaml` with the following contents - ### 3\. Install / Upgrade your helm release Upgrade/Install your openmetadata helm charts with the below single command: Extending OpenMetadata Ingestion Docker Image --------------------------------------------- One possible use case where you would need to use a custom image for the ingestion is because you have developed your own custom connectors. You can find a complete working example of this [here](https://github.com/open-metadata/openmetadata-demo/tree/main/custom-connector) . After you have your code ready, the steps would be the following: ### 1\. Create a `Dockerfile` based on `docker.getcollate.io/openmetadata/ingestion`: For example - where `docker.getcollate.io/openmetadata/ingestion:x.y.z` needs to point to the same version of the OpenMetadata server, for example `docker.getcollate.io/openmetadata/ingestion:1.3.1`. This image needs to be built and published to the container registry of your choice. ### 2\. Update the airflow in openmetadata dependencies values YAML The ingestion containers (which is the one shipping Airflow) gets installed in the `openmetadata-dependencies` helm chart. In this step, we use our own custom values YAML file to point to the image we just created on the previous step. You can create a file named `values.deps.yaml` with the following contents: ### 3\. Install / Upgrade helm release Upgrade/Install your openmetadata-dependencies helm charts with the below single command: How to disable MySQL and ElasticSearch from OpenMetadata Dependencies Helm Charts ? ----------------------------------------------------------------------------------- If you are using MySQL and ElasticSearch externally, you would want to disable the local installation of mysql and elasticsearch while installing OpenMetadata Dependencies Helm Chart. You can disable the MySQL and ElasticSearch Helm Dependencies by setting `enabled: false` value for each dependency. Below is the command to set helm values from Helm CLI - Alternatively, you can create a custom YAML file named `values.deps.yaml` to disable installation of MySQL and Elasticsearch . How to configure external database like PostgreSQL with OpenMetadata Helm Charts ? ---------------------------------------------------------------------------------- OpenMetadata Supports PostgreSQL as one of the Database Dependencies. OpenMetadata Helm Charts by default does not include PostgreSQL as Database Dependencies. In order to configure Helm Charts with External Database like PostgreSQL, follow the below guide to make the helm values change and upgrade / install OpenMetadata helm charts with the same. Upgrade Airflow Helm Dependencies Helm Charts to connect to External Database like PostgreSQL --------------------------------------------------------------------------------------------- We ship [airflow-helm](https://github.com/airflow-helm/charts/tree/main/charts/airflow) as one of OpenMetadata Dependencies with default values to connect to MySQL Database as part of `externalDatabase` configurations. You can find more information on setting the `externalDatabase` as part of helm values [here](https://github.com/airflow-helm/charts/blob/main/charts/airflow/docs/faq/database/external-database.md) . With OpenMetadata Dependencies Helm Charts, your helm values would look something like below - For the above code, it is assumed you are creating a kubernetes secret for storing Airflow Database login Credentials. A sample command to create the secret will be `kubectl create secret generic airflow-postgresql-secrets --from-literal=airflow-postgresql-password=`. Upgrade OpenMetadata Helm Charts to connect to External Database like PostgreSQL -------------------------------------------------------------------------------- Update the `openmetadata.config.database.*` helm values for OpenMetadata Application to connect to External Database like PostgreSQL. With OpenMetadata Helm Charts, your helm values would look something like below - For the above code, it is assumed you are creating a kubernetes secret for storing OpenMetadata Database login Credentials. A sample command to create the secret will be `kubectl create secret generic openmetadata-postgresql-secrets --from-literal=openmetadata-postgresql-password=`. Once you make the above changes to your helm values, run the below command to install/upgrade helm charts - How to customize OpenMetadata Dependencies Helm Chart with custom helm values ----------------------------------------------------------------------------- Our OpenMetadata Dependencies Helm Charts are internally depends on three sub-charts - * [Bitnami MySQL](https://artifacthub.io/packages/helm/bitnami/mysql/9.7.2) (helm chart version 9.7.2) * [OpenSearch](https://artifacthub.io/packages/helm/opensearch-project-helm-charts/opensearch/2.12.2) (helm chart version 2.12.2) * [Airflow](https://artifacthub.io/packages/helm/airflow-helm/airflow/8.8.0) (helm chart version 8.8.0) If you are looking to customize the deployments of any of the above dependencies, please refer to the above links for customizations of helm values for further references. By default, OpenMetadata Dependencies helm chart provides initial generic customization of these helm values in order to get you started quickly. You can refer to the openmetadata-dependencies helm charts default values [here](https://github.com/open-metadata/openmetadata-helm-charts/blob/main/charts/deps/values.yaml) . --- # Kubernetes Helm Values | Official Documentation We use cookies to improve site navigation, analyze site usage, and enhance your user experience. Click "Accept" to enable cookies or "Reject" to reject cookies. AcceptReject deployment No menu items for this category [Home](https://docs.open-metadata.org/latest) /[Deployment](https://docs.open-metadata.org/latest/deployment) /[Kubernetes](https://docs.open-metadata.org/latest/deployment/kubernetes) /[Helm Values](https://docs.open-metadata.org/latest/deployment/kubernetes/helm-values) OpenMetadata Documentation Kubernetes Helm Values ====================== This page list all the supported helm values for OpenMetadata Helm Charts. Openmetadata Config Chart Values -------------------------------- | Key | Type | Default | Environment Variable from openmetadata.yaml | | --- | --- | --- | --- | | openmetadata.config.authentication.enabled | bool | `true` | | | openmetadata.config.authentication.clientType | string | `public` | AUTHENTICATION\_CLIENT\_TYPE | | openmetadata.config.authentication.provider | string | `basic` | AUTHENTICATION\_PROVIDER | | openmetadata.config.authentication.publicKeys | list | `[http://openmetadata:8585/api/v1/system/config/jwks]` | AUTHENTICATION\_PUBLIC\_KEYS | | openmetadata.config.authentication.authority | string | `https://accounts.google.com` | AUTHENTICATION\_AUTHORITY | | openmetadata.config.authentication.clientId | string | `Empty String` | AUTHENTICATION\_CLIENT\_ID | | openmetadata.config.authentication.callbackUrl | string | `Empty String` | AUTHENTICATION\_CALLBACK\_URL | | openmetadata.config.authentication.enableSelfSignup | bool | `true` | AUTHENTICATION\_ENABLE\_SELF\_SIGNUP | | openmetadata.config.authentication.jwtPrincipalClaims | list | `[email,preferred_username,sub]` | AUTHENTICATION\_JWT\_PRINCIPAL\_CLAIMS | | openmetadata.config.authentication.ldapConfiguration.host | string | `localhost` | AUTHENTICATION\_LDAP\_HOST | | openmetadata.config.authentication.ldapConfiguration.port | int | 10636 | AUTHENTICATION\_LDAP\_PORT | | openmetadata.config.authentication.ldapConfiguration.dnAdminPrincipal | string | `cn=admin,dc=example,dc=com` | AUTHENTICATION\_LOOKUP\_ADMIN\_DN | | openmetadata.config.authentication.ldapConfiguration.dnAdminPassword.secretRef | string | `ldap-secret` | AUTHENTICATION\_LOOKUP\_ADMIN\_PWD | | openmetadata.config.authentication.ldapConfiguration.dnAdminPassword.secretKey | string | `openmetadata-ldap-secret` | AUTHENTICATION\_LOOKUP\_ADMIN\_PWD | | openmetadata.config.authentication.ldapConfiguration.userBaseDN | string | `ou=people,dc=example,dc=com` | AUTHENTICATION\_USER\_LOOKUP\_BASEDN | | openmetadata.config.authentication.ldapConfiguration.groupBaseDN | string | `Empty String` | AUTHENTICATION\_GROUP\_LOOKUP\_BASEDN | | openmetadata.config.authentication.ldapConfiguration.roleAdminName | string | `Empty String` | AUTHENTICATION\_USER\_ROLE\_ADMIN\_NAME | | openmetadata.config.authentication.ldapConfiguration.allAttributeName | string | `Empty String` | AUTHENTICATION\_USER\_ALL\_ATTR | | openmetadata.config.authentication.ldapConfiguration.usernameAttributeName | string | `Empty String` | AUTHENTICATION\_USER\_NAME\_ATTR | | openmetadata.config.authentication.ldapConfiguration.groupAttributeName | string | `Empty String` | AUTHENTICATION\_USER\_GROUP\_ATTR | | openmetadata.config.authentication.ldapConfiguration.groupAttributeValue | string | `Empty String` | AUTHENTICATION\_USER\_GROUP\_ATTR\_VALUE | | openmetadata.config.authentication.ldapConfiguration.groupMemberAttributeName | string | `Empty String` | AUTHENTICATION\_USER\_GROUP\_MEMBER\_ATTR | | openmetadata.config.authentication.ldapConfiguration.authRolesMapping | string | `Empty String` | AUTH\_ROLES\_MAPPING | | openmetadata.config.authentication.ldapConfiguration.authReassignRoles | string | `Empty String` | AUTH\_REASSIGN\_ROLES | | openmetadata.config.authentication.ldapConfiguration.mailAttributeName | string | `email` | AUTHENTICATION\_USER\_MAIL\_ATTR | | openmetadata.config.authentication.ldapConfiguration.maxPoolSize | int | 3 | AUTHENTICATION\_LDAP\_POOL\_SIZE | | openmetadata.config.authentication.ldapConfiguration.sslEnabled | bool | `true` | AUTHENTICATION\_LDAP\_SSL\_ENABLED | | openmetadata.config.authentication.ldapConfiguration.truststoreConfigType | string | `TrustAll` | AUTHENTICATION\_LDAP\_TRUSTSTORE\_TYPE | | openmetadata.config.authentication.ldapConfiguration.trustStoreConfig.customTrustManagerConfig.trustStoreFilePath | string | `Empty String` | AUTHENTICATION\_LDAP\_TRUSTSTORE\_PATH | | openmetadata.config.authentication.ldapConfiguration.trustStoreConfig.customTrustManagerConfig.trustStoreFilePassword.secretRef | string | `Empty String` | AUTHENTICATION\_LDAP\_KEYSTORE\_PASSWORD | | openmetadata.config.authentication.ldapConfiguration.trustStoreConfig.customTrustManagerConfig.trustStoreFilePassword.secretKey | string | `Empty String` | AUTHENTICATION\_LDAP\_KEYSTORE\_PASSWORD | | openmetadata.config.authentication.ldapConfiguration.trustStoreConfig.customTrustManagerConfig.trustStoreFileFormat | string | `Empty String` | AUTHENTICATION\_LDAP\_SSL\_KEY\_FORMAT | | openmetadata.config.authentication.ldapConfiguration.trustStoreConfig.customTrustManagerConfig.verifyHostname | string | `Empty String` | AUTHENTICATION\_LDAP\_SSL\_VERIFY\_CERT\_HOST | | openmetadata.config.authentication.ldapConfiguration.trustStoreConfig.customTrustManagerConfig.examineValidityDate | bool | `true` | AUTHENTICATION\_LDAP\_EXAMINE\_VALIDITY\_DATES | | openmetadata.config.authentication.ldapConfiguration.trustStoreConfig.hostNameConfig.allowWildCards | bool | `false` | AUTHENTICATION\_LDAP\_ALLOW\_WILDCARDS | | openmetadata.config.authentication.ldapConfiguration.trustStoreConfig.hostNameConfig.acceptableHostNames | string | `[Empty String]` | AUTHENTICATION\_LDAP\_ALLOWED\_HOSTNAMES | | openmetadata.config.authentication.ldapConfiguration.trustStoreConfig.jvmDefaultConfig.verifyHostname | string | `Empty String` | AUTHENTICATION\_LDAP\_SSL\_VERIFY\_CERT\_HOST | | openmetadata.config.authentication.ldapConfiguration.trustStoreConfig.trustAllConfig.examineValidityDates | bool | `true` | AUTHENTICATION\_LDAP\_EXAMINE\_VALIDITY\_DATES | | openmetadata.config.authentication.oidcConfiguration.callbackUrl | string | `http://openmetadata:8585/callback` | OIDC\_CALLBACK | | openmetadata.config.authentication.oidcConfiguration.clientAuthenticationMethod | string | `client_secret_post` | OIDC\_CLIENT\_AUTH\_METHOD | | openmetadata.config.authentication.oidcConfiguration.clientId.secretKey | string | `openmetadata-oidc-client-id` | OIDC\_CLIENT\_ID | | openmetadata.config.authentication.oidcConfiguration.clientId.secretRef | string | `oidc-secrets` | OIDC\_CLIENT\_ID | | openmetadata.config.authentication.oidcConfiguration.clientSecret.secretKey | string | `openmetadata-oidc-client-secret` | OIDC\_CLIENT\_SECRET | | openmetadata.config.authentication.oidcConfiguration.clientSecret.secretRef | string | `oidc-secrets` | OIDC\_CLIENT\_SECRET | | openmetadata.config.authentication.oidcConfiguration.customParams | string | `Empty` | OIDC\_CUSTOM\_PARAMS | | openmetadata.config.authentication.oidcConfiguration.disablePkce | bool | true | OIDC\_DISABLE\_PKCE | | openmetadata.config.authentication.oidcConfiguration.discoveryUri | string | `Empty` | OIDC\_DISCOVERY\_URI | | openmetadata.config.authentication.oidcConfiguration.enabled | bool | false | | | openmetadata.config.authentication.oidcConfiguration.maxClockSkew | string | `Empty` | OIDC\_MAX\_CLOCK\_SKEW | | openmetadata.config.authentication.oidcConfiguration.oidcType | string | `Empty` | OIDC\_TYPE | | openmetadata.config.authentication.oidcConfiguration.preferredJwsAlgorithm | string | `RS256` | OIDC\_PREFERRED\_JWS | | openmetadata.config.authentication.oidcConfiguration.responseType | string | `code` | OIDC\_RESPONSE\_TYPE | | openmetadata.config.authentication.oidcConfiguration.scope | string | `openid email profile` | OIDC\_SCOPE | | openmetadata.config.authentication.oidcConfiguration.serverUrl | string | `http://openmetadata:8585` | OIDC\_SERVER\_URL | | openmetadata.config.authentication.oidcConfiguration.tenant | string | `Empty` | OIDC\_TENANT | | openmetadata.config.authentication.oidcConfiguration.useNonce | bool | `true` | OIDC\_USE\_NONCE | | openmetadata.config.authentication.saml.debugMode | bool | false | SAML\_DEBUG\_MODE | | openmetadata.config.authentication.saml.idp.entityId | string | `Empty` | SAML\_IDP\_ENTITY\_ID | | openmetadata.config.authentication.saml.idp.ssoLoginUrl | string | `Empty` | SAML\_IDP\_SSO\_LOGIN\_URL | | openmetadata.config.authentication.saml.idp.idpX509Certificate.secretRef | string | `Empty` | SAML\_IDP\_CERTIFICATE | | openmetadata.config.authentication.saml.idp.idpX509Certificate.secretKey | string | `Empty` | SAML\_IDP\_CERTIFICATE | | openmetadata.config.authentication.saml.idp.authorityUrl | string | `http://openmetadata:8585/api/v1/saml/login` | SAML\_AUTHORITY\_URL | | openmetadata.config.authentication.saml.idp.nameId | string | `urn:oasis:names:tc:SAML:2.0:nameid-format:emailAddress` | SAML\_IDP\_NAME\_ID | | openmetadata.config.authentication.saml.sp.entityId | string | `http://openmetadata:8585/api/v1/saml/acs` | SAML\_SP\_ENTITY\_ID | | openmetadata.config.authentication.saml.sp.acs | string | `http://openmetadata:8585/api/v1/saml/acs` | SAML\_SP\_ACS | | openmetadata.config.authentication.saml.sp.spX509Certificate.secretRef | string | `Empty` | SAML\_SP\_CERTIFICATE | | openmetadata.config.authentication.saml.sp.spX509Certificate.secretKey | string | `Empty` | SAML\_SP\_CERTIFICATE | | openmetadata.config.authentication.saml.sp.callback | string | `http://openmetadata:8585/saml/callback` | SAML\_SP\_CALLBACK | | openmetadata.config.authentication.saml.security.strictMode | bool | false | SAML\_STRICT\_MODE | | openmetadata.config.authentication.saml.security.tokenValidity | int | 3600 | SAML\_SP\_TOKEN\_VALIDITY | | openmetadata.config.authentication.saml.security.sendEncryptedNameId | bool | false | SAML\_SEND\_ENCRYPTED\_NAME\_ID | | openmetadata.config.authentication.saml.security.sendSignedAuthRequest | bool | false | SAML\_SEND\_SIGNED\_AUTH\_REQUEST | | openmetadata.config.authentication.saml.security.signSpMetadata | bool | false | SAML\_SIGNED\_SP\_METADATA | | openmetadata.config.authentication.saml.security.wantMessagesSigned | bool | false | SAML\_WANT\_MESSAGE\_SIGNED | | openmetadata.config.authentication.saml.security.wantAssertionsSigned | bool | false | SAML\_WANT\_ASSERTION\_SIGNED | | openmetadata.config.authentication.saml.security.wantAssertionEncrypted | bool | false | SAML\_WANT\_ASSERTION\_ENCRYPTED | | openmetadata.config.authentication.saml.security.wantNameIdEncrypted | bool | false | SAML\_WANT\_NAME\_ID\_ENCRYPTED | | openmetadata.config.authentication.saml.security.keyStoreFilePath | string | `Empty` | SAML\_KEYSTORE\_FILE\_PATH | | openmetadata.config.authentication.saml.security.keyStoreAlias.secretRef | string | `Empty` | SAML\_KEYSTORE\_ALIAS | | openmetadata.config.authentication.saml.security.keyStoreAlias.secretKey | string | `Empty` | SAML\_KEYSTORE\_ALIAS | | openmetadata.config.authentication.saml.security.keyStorePassword.secretRef | string | `Empty` | SAML\_KEYSTORE\_PASSWORD | | openmetadata.config.authentication.saml.security.keyStorePassword.secretKey | string | `Empty` | SAML\_KEYSTORE\_PASSWORD | | openmetadata.config.authorizer.enabled | bool | `true` | | | openmetadata.config.authorizer.allowedEmailRegistrationDomains | list | `[all]` | AUTHORIZER\_ALLOWED\_REGISTRATION\_DOMAIN | | openmetadata.config.authorizer.className | string | `org.openmetadata.service.security.DefaultAuthorizer` | AUTHORIZER\_CLASS\_NAME | | openmetadata.config.authorizer.containerRequestFilter | string | `org.openmetadata.service.security.JwtFilter` | AUTHORIZER\_REQUEST\_FILTER | | openmetadata.config.authorizer.enforcePrincipalDomain | bool | `false` | AUTHORIZER\_ENFORCE\_PRINCIPAL\_DOMAIN | | openmetadata.config.authorizer.enableSecureSocketConnection | bool | `false` | AUTHORIZER\_ENABLE\_SECURE\_SOCKET | | openmetadata.config.authorizer.initialAdmins | list | `[admin]` | AUTHORIZER\_ADMIN\_PRINCIPALS | | openmetadata.config.authorizer.principalDomain | string | `open-metadata.org` | AUTHORIZER\_PRINCIPAL\_DOMAIN | | openmetadata.config.airflow.auth.password.secretRef | string | `airflow-secrets` | AIRFLOW\_PASSWORD | | openmetadata.config.airflow.auth.password.secretKey | string | `openmetadata-airflow-password` | AIRFLOW\_PASSWORD | | openmetadata.config.airflow.auth.username | string | `admin` | AIRFLOW\_USERNAME | | openmetadata.config.airflow.enabled | bool | `true` | | | openmetadata.config.airflow.host | string | `http://openmetadata-dependencies-web:8080` | PIPELINE\_SERVICE\_CLIENT\_ENDPOINT | | openmetadata.config.airflow.openmetadata.serverHostApiUrl | string | `http://openmetadata:8585/api` | SERVER\_HOST\_API\_URL | | openmetadata.config.airflow.sslCertificatePath | string | `/no/path` | PIPELINE\_SERVICE\_CLIENT\_SSL\_CERT\_PATH | | openmetadata.config.airflow.verifySsl | string | `no-ssl` | PIPELINE\_SERVICE\_CLIENT\_VERIFY\_SSL | | openmetadata.config.clusterName | string | `openmetadata` | OPENMETADATA\_CLUSTER\_NAME | | openmetadata.config.database.enabled | bool | `true` | | | openmetadata.config.database.auth.password.secretRef | string | `mysql-secrets` | DB\_USER\_PASSWORD | | openmetadata.config.database.auth.password.secretKey | string | `openmetadata-mysql-password` | DB\_USER\_PASSWORD | | openmetadata.config.database.auth.username | string | `openmetadata_user` | DB\_USER | | openmetadata.config.database.databaseName | string | `openmetadata_db` | OM\_DATABASE | | openmetadata.config.database.dbParams | string | `allowPublicKeyRetrieval=true&useSSL=false&serverTimezone=UTC` | DB\_PARAMS | | openmetadata.config.database.dbScheme | string | `mysql` | DB\_SCHEME | | openmetadata.config.database.driverClass | string | `com.mysql.cj.jdbc.Driver` | DB\_DRIVER\_CLASS | | openmetadata.config.database.host | string | `mysql` | DB\_HOST | | openmetadata.config.database.port | int | 3306 | DB\_PORT | | openmetadata.config.elasticsearch.enabled | bool | `true` | | | openmetadata.config.elasticsearch.auth.enabled | bool | `false` | | | openmetadata.config.elasticsearch.auth.username | string | `elasticsearch` | ELASTICSEARCH\_USER | | openmetadata.config.elasticsearch.auth.password.secretRef | string | `elasticsearch-secrets` | ELASTICSEARCH\_PASSWORD | | openmetadata.config.elasticsearch.auth.password.secretKey | string | `openmetadata-elasticsearch-password` | ELASTICSEARCH\_PASSWORD | | openmetadata.config.elasticsearch.host | string | `opensearch` | ELASTICSEARCH\_HOST | | openmetadata.config.elasticsearch.keepAliveTimeoutSecs | int | `600` | ELASTICSEARCH\_KEEP\_ALIVE\_TIMEOUT\_SECS | | openmetadata.config.elasticsearch.port | int | 9200 | ELASTICSEARCH\_PORT | | openmetadata.config.elasticsearch.searchType | string | `opensearch` | SEARCH\_TYPE | | openmetadata.config.elasticsearch.scheme | string | `http` | ELASTICSEARCH\_SCHEME | | openmetadata.config.elasticsearch.clusterAlias | string | `Empty String` | ELASTICSEARCH\_CLUSTER\_ALIAS | | openmetadata.config.elasticsearch.searchIndexMappingLanguage | string | `EN` | ELASTICSEARCH\_INDEX\_MAPPING\_LANG | | openmetadata.config.elasticsearch.trustStore.enabled | bool | `false` | | | openmetadata.config.elasticsearch.trustStore.path | string | `Empty String` | ELASTICSEARCH\_TRUST\_STORE\_PATH | | openmetadata.config.elasticsearch.trustStore.password.secretRef | string | `elasticsearch-truststore-secrets` | ELASTICSEARCH\_TRUST\_STORE\_PASSWORD | | openmetadata.config.elasticsearch.trustStore.password.secretKey | string | `openmetadata-elasticsearch-truststore-password` | ELASTICSEARCH\_TRUST\_STORE\_PASSWORD | | openmetadata.config.eventMonitor.enabled | bool | `true` | | | openmetadata.config.eventMonitor.type | string | `prometheus` | EVENT\_MONITOR | | openmetadata.config.eventMonitor.batchSize | int | `10` | EVENT\_MONITOR\_BATCH\_SIZE | | openmetadata.config.eventMonitor.pathPattern | list | `[/api/v1/tables/*,/api/v1/health-check]` | EVENT\_MONITOR\_PATH\_PATTERN | | openmetadata.config.eventMonitor.latency | list | `[]` | EVENT\_MONITOR\_LATENCY | | openmetadata.config.fernetkey.value | string | `jJ/9sz0g0OHxsfxOoSfdFdmk3ysNmPRnH3TUAbz3IHA=` | FERNET\_KEY | | openmetadata.config.fernetkey.secretRef | string | \`\` | FERNET\_KEY | | openmetadata.config.fernetkey.secretKef | string | \`\` | FERNET\_KEY | | openmetadata.config.jwtTokenConfiguration.enabled | bool | `true` | | | openmetadata.config.jwtTokenConfiguration.rsapublicKeyFilePath | string | `./conf/public_key.der` | RSA\_PUBLIC\_KEY\_FILE\_PATH | | openmetadata.config.jwtTokenConfiguration.rsaprivateKeyFilePath | string | `./conf/private_key.der` | RSA\_PRIVATE\_KEY\_FILE\_PATH | | openmetadata.config.jwtTokenConfiguration.jwtissuer | string | `open-metadata.org` | JWT\_ISSUER | | openmetadata.config.jwtTokenConfiguration.keyId | string | `Gb389a-9f76-gdjs-a92j-0242bk94356` | JWT\_KEY\_ID | | openmetadata.config.logLevel | string | `INFO` | LOG\_LEVEL | | openmetadata.config.openmetadata.adminPort | int | 8586 | SERVER\_ADMIN\_PORT | | openmetadata.config.openmetadata.host | string | `openmetadata` | OPENMETADATA\_SERVER\_URL | | openmetadata.config.openmetadata.port | int | 8585 | SERVER\_PORT | | openmetadata.config.pipelineServiceClientConfig.auth.password.secretRef | string | `airflow-secrets` | AIRFLOW\_PASSWORD | | openmetadata.config.pipelineServiceClientConfig.auth.password.secretKey | string | `openmetadata-airflow-password` | AIRFLOW\_PASSWORD | | openmetadata.config.pipelineServiceClientConfig.auth.username | string | `admin` | AIRFLOW\_USERNAME | | openmetadata.config.pipelineServiceClientConfig.auth.trustStorePath | string | \`\` | AIRFLOW\_TRUST\_STORE\_PATH | | openmetadata.config.pipelineServiceClientConfig.auth.trustStorePassword.secretRef | string | \`\` | AIRFLOW\_TRUST\_STORE\_PASSWORD | | openmetadata.config.pipelineServiceClientConfig.auth.trustStorePassword.secretKey | string | \`\` | AIRFLOW\_TRUST\_STORE\_PASSWORD | | openmetadata.config.pipelineServiceClientConfig.apiEndpoint | string | `http://openmetadata-dependencies-web:8080` | PIPELINE\_SERVICE\_CLIENT\_ENDPOINT | | openmetadata.config.pipelineServiceClientConfig.className | string | `org.openmetadata.service.clients.pipeline.airflow.AirflowRESTClient` | PIPELINE\_SERVICE\_CLIENT\_CLASS\_NAME | | openmetadata.config.pipelineServiceClientConfig.enabled | bool | `true` | PIPELINE\_SERVICE\_CLIENT\_ENABLED | | openmetadata.config.pipelineServiceClientConfig.healthCheckInterval | int | `300` | PIPELINE\_SERVICE\_CLIENT\_HEALTH\_CHECK\_INTERVAL | | openmetadata.config.pipelineServiceClientConfig.ingestionIpInfoEnabled | bool | `false` | PIPELINE\_SERVICE\_IP\_INFO\_ENABLED | | openmetadata.config.pipelineServiceClientConfig.metadataApiEndpoint | string | `http://openmetadata:8585/api` | SERVER\_HOST\_API\_URL | | openmetadata.config.pipelineServiceClientConfig.sslCertificatePath | string | `/no/path` | PIPELINE\_SERVICE\_CLIENT\_SSL\_CERT\_PATH | | openmetadata.config.pipelineServiceClientConfig.verifySsl | string | `no-ssl` | PIPELINE\_SERVICE\_CLIENT\_VERIFY\_SSL | | openmetadata.config.pipelineServiceClientConfig.hostIp | string | `Empty` | PIPELINE\_SERVICE\_CLIENT\_HOST\_IP | | openmetadata.config.secretsManager.enabled | bool | `true` | | | openmetadata.config.secretsManager.provider | string | `Empty String` | SECRET\_MANAGER | | openmetadata.config.secretsManager.prefix | string | `Empty String` | SECRET\_MANAGER\_PREFIX | | openmetadata.config.secretsManager.tags | list | `[]` | SECRET\_MANAGER\_TAGS | | openmetadata.config.secretsManager.additionalParameters.enabled | bool | `false` | | | openmetadata.config.secretsManager.additionalParameters.accessKeyId.secretRef | string | `aws-access-key-secret` | OM\_SM\_ACCESS\_KEY\_ID | | openmetadata.config.secretsManager.additionalParameters.accessKeyId.secretKey | string | `aws-key-secret` | OM\_SM\_ACCESS\_KEY\_ID | | openmetadata.config.secretsManager.additionalParameters.clientId.secretRef | string | `azure-client-id-secret` | OM\_SM\_CLIENT\_ID | | openmetadata.config.secretsManager.additionalParameters.clientId.secretKey | string | `azure-key-secret` | OM\_SM\_CLIENT\_ID | | openmetadata.config.secretsManager.additionalParameters.clientSecret.secretRef | string | `azure-client-secret` | OM\_SM\_CLIENT\_SECRET | | openmetadata.config.secretsManager.additionalParameters.clientSecret.secretKey | string | `azure-key-secret` | OM\_SM\_CLIENT\_SECRET | | openmetadata.config.secretsManager.additionalParameters.tenantId.secretRef | string | `azure-tenant-id-secret` | OM\_SM\_TENANT\_ID | | openmetadata.config.secretsManager.additionalParameters.tenantId.secretKey | string | `azure-key-secret` | OM\_SM\_TENANT\_ID | | openmetadata.config.secretsManager.additionalParameters.vaultName.secretRef | string | `azure-vault-name-secret` | OM\_SM\_VAULT\_NAME | | openmetadata.config.secretsManager.additionalParameters.vaultName.secretKey | string | `azure-key-secret` | OM\_SM\_VAULT\_NAME | | openmetadata.config.secretsManager.additionalParameters.region | string | `Empty String` | OM\_SM\_REGION | | openmetadata.config.secretsManager.additionalParameters.secretAccessKey.secretRef | string | `aws-secret-access-key-secret` | OM\_SM\_ACCESS\_KEY | | openmetadata.config.secretsManager.additionalParameters.secretAccessKey.secretKey | string | `aws-key-secret` | OM\_SM\_ACCESS\_KEY | | openmetadata.config.smtpConfig.enableSmtpServer | bool | `false` | AUTHORIZER\_ENABLE\_SMTP | | openmetadata.config.smtpConfig.emailingEntity | string | `OpenMetadata` | OM\_EMAIL\_ENTITY | | openmetadata.config.smtpConfig.openMetadataUrl | string | `Empty String` | OPENMETADATA\_SERVER\_URL | | openmetadata.config.smtpConfig.password.secretKey | string | `Empty String` | SMTP\_SERVER\_PWD | | openmetadata.config.smtpConfig.password.secretRef | string | `Empty String` | SMTP\_SERVER\_PWD | | openmetadata.config.smtpConfig.serverEndpoint | string | `Empty String` | SMTP\_SERVER\_ENDPOINT | | openmetadata.config.smtpConfig.serverPort | string | `Empty String` | SMTP\_SERVER\_PORT | | openmetadata.config.smtpConfig.supportUrl | string | `https://slack.open-metadata.org` | OM\_SUPPORT\_URL | | openmetadata.config.smtpConfig.transportationStrategy | string | `SMTP_TLS` | SMTP\_SERVER\_STRATEGY | | openmetadata.config.smtpConfig.username | string | `Empty String` | SMTP\_SERVER\_USERNAME | | openmetadata.config.upgradeMigrationConfigs.debug | bool | `false` | | | openmetadata.config.upgradeMigrationConfigs.additionalArgs | string | `Empty String` | | | openmetadata.config.web.enabled | bool | `true` | | | openmetadata.config.web.contentTypeOptions.enabled | bool | `false` | WEB\_CONF\_CONTENT\_TYPE\_OPTIONS\_ENABLED | | openmetadata.config.web.csp.enabled | bool | `false` | WEB\_CONF\_XSS\_CSP\_ENABLED | | openmetadata.config.web.csp.policy | string | `default-src 'self` | WEB\_CONF\_XSS\_CSP\_POLICY | | openmetadata.config.web.csp.reportOnlyPolicy | string | `Empty String` | WEB\_CONF\_XSS\_CSP\_REPORT\_ONLY\_POLICY | | openmetadata.config.web.frameOptions.enabled | bool | `false` | WEB\_CONF\_FRAME\_OPTION\_ENABLED | | openmetadata.config.web.frameOptions.option | string | `SAMEORIGIN` | WEB\_CONF\_FRAME\_OPTION | | openmetadata.config.web.frameOptions.origin | string | `Empty String` | WEB\_CONF\_FRAME\_ORIGIN | | openmetadata.config.web.hsts.enabled | bool | `false` | WEB\_CONF\_HSTS\_ENABLED | | openmetadata.config.web.hsts.includeSubDomains | bool | `true` | WEB\_CONF\_HSTS\_INCLUDE\_SUBDOMAINS | | openmetadata.config.web.hsts.maxAge | string | `365 days` | WEB\_CONF\_HSTS\_MAX\_AGE | | openmetadata.config.web.hsts.preload | bool | `true` | WEB\_CONF\_HSTS\_PRELOAD | | openmetadata.config.web.uriPath | string | `/api` | WEB\_CONF\_URI\_PATH | | openmetadata.config.web.xssProtection.block | bool | `true` | WEB\_CONF\_XSS\_PROTECTION\_BLOCK | | openmetadata.config.web.xssProtection.enabled | bool | `false` | WEB\_CONF\_XSS\_PROTECTION\_ENABLED | | openmetadata.config.web.xssProtection.onXss | bool | `true` | WEB\_CONF\_XSS\_PROTECTION\_ON | | openmetadata.config.web.referrer-policy.enabled | bool | `false` | WEB\_CONF\_REFERRER\_POLICY\_ENABLED | | openmetadata.config.web.referrer-policy.option | string | `SAME_ORIGIN'` | WEB\_CONF\_REFERRER\_POLICY\_OPTION | | openmetadata.config.web.permission-policy.enabled | bool | `false` | WEB\_CONF\_PERMISSION\_POLICY\_ENABLED | | openmetadata.config.web.permission-policy.option | string | `Empty String` | WEB\_CONF\_PERMISSION\_POLICY\_OPTION | Chart Values ------------ | Key | Type | Default | | --- | --- | --- | | affinity | object | `{}` | | commonLabels | object | `{}` | | extraEnvs | Extra \[environment variables\]\[\] which will be appended to the `env:` definition for the container | `[]` | | extraInitContainers | Templatable string of additional `initContainers` to be passed to `tpl` function | `[]` | | extraVolumes | Templatable string of additional `volumes` to be passed to the `tpl` function | `[]` | | extraVolumeMounts | Templatable string of additional `volumeMounts` to be passed to the `tpl` function | `[]` | | fullnameOverride | string | `"openmetadata"` | | image.pullPolicy | string | `"Always"` | | image.repository | string | `"docker.getcollate.io/openmetadata/server"` | | image.tag | string | `1.3.4` | | imagePullSecrets | list | `[]` | | ingress.annotations | object | `{}` | | ingress.className | string | `""` | | ingress.enabled | bool | `false` | | ingress.hosts\[0\].host | string | `"open-metadata.local"` | | ingress.hosts\[0\].paths\[0\].path | string | `"/"` | | ingress.hosts\[0\].paths\[0\].pathType | string | `"ImplementationSpecific"` | | ingress.tls | list | `[]` | | livenessProbe.initialDelaySeconds | int | `60` | | livenessProbe.periodSeconds | int | `30` | | livenessProbe.failureThreshold | int | `5` | | livenessProbe.httpGet.path | string | `/healthcheck` | | livenessProbe.httpGet.port | string | `http-admin` | | nameOverride | string | `""` | | nodeSelector | object | `{}` | | podAnnotations | object | `{}` | | podSecurityContext | object | `{}` | | readinessProbe.initialDelaySeconds | int | `60` | | readinessProbe.periodSeconds | int | `30` | | readinessProbe.failureThreshold | int | `5` | | readinessProbe.httpGet.path | string | `/` | | readinessProbe.httpGet.port | string | `http` | | replicaCount | int | `1` | | resources | object | `{}` | | securityContext | object | `{}` | | service.adminPort | string | `8586` | | service.annotations | object | `{}` | | service.port | int | `8585` | | service.type | string | `"ClusterIP"` | | serviceAccount.annotations | object | `{}` | | serviceAccount.create | bool | `true` | | serviceAccount.name | string | `nil` | | automountServiceAccountToken | bool | `true` | | serviceMonitor.annotations | object | `{}` | | serviceMonitor.enabled | bool | `false` | | serviceMonitor.interval | string | `30s` | | serviceMonitor.labels | object | `{}` | | sidecars | list | `[]` | | startupProbe.periodSeconds | int | `60` | | startupProbe.failureThreshold | int | `5` | | startupProbe.httpGet.path | string | `/healthcheck` | | startupProbe.httpGet.port | string | `http-admin` | | startupProbe.successThreshold | int | `1` | | tolerations | list | `[]` | | networkPolicy.enabled | bool | `false` | | podDisruptionBudget.enabled | bool | `false` | | podDisruptionBudget.config.maxUnavailable | String | `1` | | podDisruptionBudget.config.minAvailable | String | `1` | --- # Azure AKS Deployment | Official Documentation We use cookies to improve site navigation, analyze site usage, and enhance your user experience. Click "Accept" to enable cookies or "Reject" to reject cookies. AcceptReject deployment No menu items for this category [Home](https://docs.open-metadata.org/latest) /[Deployment](https://docs.open-metadata.org/latest/deployment) /[Kubernetes](https://docs.open-metadata.org/latest/deployment/kubernetes) /[Aks](https://docs.open-metadata.org/latest/deployment/kubernetes/aks) OpenMetadata Documentation Openmetadata Deployment on Azure Kubernetes Service Cluster =========================================================== Openmetadata can be deployed on Azure Kubernetes Service. It however requires certain cloud specific configurations with regards to setting up storage accounts for Airflow which is one of its dependencies. Prerequisites ------------- ### Azure Services for Database and Search Engine as Elastic Cloud It is recommended to use [Azure SQL](https://azure.microsoft.com/en-in/products/azure-sql/database) and [Elastic Cloud on Azure](https://www.elastic.co/partners/microsoft-azure) for Production Deployments. We support * Azure SQL (MySQL) engine version 8 or higher * Azure SQL (PostgreSQL) engine version 12 or higher * Elastic Cloud (ElasticSearch version 8.11.4) Once you have the Azure SQL and Elastic Cloud on Azure configured, you can update the environment variables below for OpenMetadata kubernetes deployments to connect with Database and ElasticSearch. We recommend - * Azure SQL to be Multi Zone Available and Production Workload Environment * Elastic Cloud Environment with multiple zones and minimum 2 nodes Make sure to create database and elastic cloud credentials as Kubernetes Secrets mentioned [here](https://docs.open-metadata.org/latest/quick-start/local-kubernetes-deployment#2.-create-kubernetes-secrets-required-for-helm-charts) . Also, disable MySQL and ElasticSearch from OpenMetadata Dependencies Helm Charts as mentioned in the FAQs [here](https://docs.open-metadata.org/latest/deployment/kubernetes/aks#how-to-disable-mysql-and-elasticsearch-from-openmetadata-dependencies-helm-charts) . ### Step 1 - Create a AKS cluster If you are deploying on a new cluster set the `EnableAzureDiskFileCSIDriver=true` to enable container storage interface storage drivers. For existing cluster it is important to enable the CSI storage drivers ### Step 2 - Create a Namespace (optional) ### Step 3 - Create Persistent Volumes OpenMetadata helm chart depends on Airflow and Airflow expects a persistent disk that support ReadWriteMany (the volume can be mounted as read-write by many nodes). The Azure CSI storage drivers we enabled earlier support the provisioning of the disks in ReadWriteMany mode,. Create the volume claims by applying the manifest. ### Step 4 - Change owner and update permission for persistent volumes Airflow pods run as non-root user and lack write access to our persistent volumes. To fix this we create a job permissions\_pod.yaml that runs a pod that mounts volumnes into the persistent volume claim and updates the owner of the mounted folders /airflow-dags and /airflow-logs to user id 5000, which is the default linux user id of Airflow pods. Start the job by applying the manifest in permissions\_pod.yaml. ### Step 5 - Add the Helm Openmetadata repo and set-up secrets #### Add Helm Repo #### Create secrets It is recommended to use external database and search for production deployments. The following implementation uses external postgresql DB from Azure Database. Any of the popular databases can be used. The default implementation uses mysql. For production deployments connecting external postgresql database provide external database connection details by settings up appropriate secrets as below to use in manifests. ### Step 6 - Install Openmetadata dependencies The values-dependencies-yaml is used to override default values in the official helm chart and must be configured for customizing for use cases. Uncomment the externalDatabase section with meaningful values to connect to external database for production deployments. We set sensitive information like host address, DB name and DB username through the CLI. We overwrite some of the default values in the official openmetadata-dependencies helm chart with the values-dependencies.yaml to include an external postgresql db. And it's important to turn the mysql.enable flag to false if you are not using the default mysql db. This can be done both through the yaml file or as shown by setting variable values in the helm install command. For more information on airflow helm chart values, please refer to [airflow-helm](https://artifacthub.io/packages/helm/airflow-helm/airflow/8.5.3) It takes a few minutes for all the pods to be correctly set-up and running. ### Step 7 - Install Openmetadata Finally install Openmetadata optionally customizing the values provided in the official chart [here](https://github.com/open-metadata/openmetadata-helm-charts/blob/main/charts/openmetadata/values.yaml) using the values.yaml file. Give it again a few seconds for the pod to get ready. And when its ready, the service can be accessed by forwarding port 8585 of the cluster ip to you local host port. Troubleshooting Airflow ----------------------- ### JSONDecodeError: Unterminated string starting If you are using Airflow with Azure Blob Storage as `PersistentVolume` as explained in [Storage class using blobfuse](https://learn.microsoft.com/en-us/azure/aks/azure-csi-blob-storage-provision?tabs=mount-nfs%2Csecret) , you may encounter the following error after a few days: Moreover, the Executor pods would actually be using old files. This behaviour is caused by the recommended config by the mentioned documentation: **Disabling the cache** will help here. In this case it won't have any negative impact, since the `.py` and `.json` files are small enough and not heavily used. The same configuration without cache: You can find more information about this error [here](https://github.com/open-metadata/OpenMetadata/issues/15321) , and similar discussions [here](https://github.com/Azure/azure-storage-fuse/issues/1171) and [here](https://github.com/Azure/azure-storage-fuse/issues/1139) . FAQs ==== Java Memory Heap Issue ---------------------- If your openmetadata pods are not in ready state at any point in time and the openmetadata pod logs speaks about the below issue - This is due to the default JVM Heap Space configuration (1 GiB) being not enough for your workloads. In order to resolve this issue, head over to your custom openmetadata helm values and append the below environment variable The flag `Xmx` specifies the maximum memory allocation pool for a Java virtual machine (JVM), while `Xms` specifies the initial memory allocation pool. Upgrade the helm charts with the above changes using the following command `helm upgrade --install openmetadata open-metadata/openmetadata --values --namespace `. Update this command your `values.yml` filename and `namespaceName` where you have deployed OpenMetadata in Kubernetes. PostgreSQL Issue permission denied to create extension "pgcrypto" ----------------------------------------------------------------- If you are facing the below issue with PostgreSQL as Database Backend for OpenMetadata Application, It seems the Database User does not have sufficient privileges. In order to resolve the above issue, grant usage permissions to the PSQL User. In the above command, replace `` with the sql user used by OpenMetadata Application to connect to PostgreSQL Database. How to extend and use custom docker images with OpenMetadata Helm Charts ? -------------------------------------------------------------------------- Extending OpenMetadata Server Docker Image ------------------------------------------ ### 1\. Create a `Dockerfile` based on `docker.getcollate.io/openmetadata/server` OpenMetadata helm charts uses official published docker images from [DockerHub](https://hub.docker.com/u/openmetadata) . A typical scenario will be to install organization certificates for connecting with inhouse systems. For Example - where `docker.getcollate.io/openmetadata/server:x.y.z` needs to point to the same version of the OpenMetadata server, for example `docker.getcollate.io/openmetadata/server:1.3.1`. This image needs to be built and published to the container registry of your choice. ### 2\. Update your openmetadata helm values yaml The OpenMetadata Application gets installed as part of `openmetadata` helm chart. In this step, update the custom helm values using YAML file to point the image created in the previous step. For example, create a helm values file named `values.yaml` with the following contents - ### 3\. Install / Upgrade your helm release Upgrade/Install your openmetadata helm charts with the below single command: Extending OpenMetadata Ingestion Docker Image --------------------------------------------- One possible use case where you would need to use a custom image for the ingestion is because you have developed your own custom connectors. You can find a complete working example of this [here](https://github.com/open-metadata/openmetadata-demo/tree/main/custom-connector) . After you have your code ready, the steps would be the following: ### 1\. Create a `Dockerfile` based on `docker.getcollate.io/openmetadata/ingestion`: For example - where `docker.getcollate.io/openmetadata/ingestion:x.y.z` needs to point to the same version of the OpenMetadata server, for example `docker.getcollate.io/openmetadata/ingestion:1.3.1`. This image needs to be built and published to the container registry of your choice. ### 2\. Update the airflow in openmetadata dependencies values YAML The ingestion containers (which is the one shipping Airflow) gets installed in the `openmetadata-dependencies` helm chart. In this step, we use our own custom values YAML file to point to the image we just created on the previous step. You can create a file named `values.deps.yaml` with the following contents: ### 3\. Install / Upgrade helm release Upgrade/Install your openmetadata-dependencies helm charts with the below single command: How to disable MySQL and ElasticSearch from OpenMetadata Dependencies Helm Charts ? ----------------------------------------------------------------------------------- If you are using MySQL and ElasticSearch externally, you would want to disable the local installation of mysql and elasticsearch while installing OpenMetadata Dependencies Helm Chart. You can disable the MySQL and ElasticSearch Helm Dependencies by setting `enabled: false` value for each dependency. Below is the command to set helm values from Helm CLI - Alternatively, you can create a custom YAML file named `values.deps.yaml` to disable installation of MySQL and Elasticsearch . How to configure external database like PostgreSQL with OpenMetadata Helm Charts ? ---------------------------------------------------------------------------------- OpenMetadata Supports PostgreSQL as one of the Database Dependencies. OpenMetadata Helm Charts by default does not include PostgreSQL as Database Dependencies. In order to configure Helm Charts with External Database like PostgreSQL, follow the below guide to make the helm values change and upgrade / install OpenMetadata helm charts with the same. Upgrade Airflow Helm Dependencies Helm Charts to connect to External Database like PostgreSQL --------------------------------------------------------------------------------------------- We ship [airflow-helm](https://github.com/airflow-helm/charts/tree/main/charts/airflow) as one of OpenMetadata Dependencies with default values to connect to MySQL Database as part of `externalDatabase` configurations. You can find more information on setting the `externalDatabase` as part of helm values [here](https://github.com/airflow-helm/charts/blob/main/charts/airflow/docs/faq/database/external-database.md) . With OpenMetadata Dependencies Helm Charts, your helm values would look something like below - For the above code, it is assumed you are creating a kubernetes secret for storing Airflow Database login Credentials. A sample command to create the secret will be `kubectl create secret generic airflow-postgresql-secrets --from-literal=airflow-postgresql-password=`. Upgrade OpenMetadata Helm Charts to connect to External Database like PostgreSQL -------------------------------------------------------------------------------- Update the `openmetadata.config.database.*` helm values for OpenMetadata Application to connect to External Database like PostgreSQL. With OpenMetadata Helm Charts, your helm values would look something like below - For the above code, it is assumed you are creating a kubernetes secret for storing OpenMetadata Database login Credentials. A sample command to create the secret will be `kubectl create secret generic openmetadata-postgresql-secrets --from-literal=openmetadata-postgresql-password=`. Once you make the above changes to your helm values, run the below command to install/upgrade helm charts - How to customize OpenMetadata Dependencies Helm Chart with custom helm values ----------------------------------------------------------------------------- Our OpenMetadata Dependencies Helm Charts are internally depends on three sub-charts - * [Bitnami MySQL](https://artifacthub.io/packages/helm/bitnami/mysql/9.7.2) (helm chart version 9.7.2) * [OpenSearch](https://artifacthub.io/packages/helm/opensearch-project-helm-charts/opensearch/2.12.2) (helm chart version 2.12.2) * [Airflow](https://artifacthub.io/packages/helm/airflow-helm/airflow/8.8.0) (helm chart version 8.8.0) If you are looking to customize the deployments of any of the above dependencies, please refer to the above links for customizations of helm values for further references. By default, OpenMetadata Dependencies helm chart provides initial generic customization of these helm values in order to get you started quickly. You can refer to the openmetadata-dependencies helm charts default values [here](https://github.com/open-metadata/openmetadata-helm-charts/blob/main/charts/deps/values.yaml) . --- # AWS EKS Deployment | OpenMetadata Kubernetes Guide We use cookies to improve site navigation, analyze site usage, and enhance your user experience. Click "Accept" to enable cookies or "Reject" to reject cookies. AcceptReject deployment No menu items for this category [Home](https://docs.open-metadata.org/latest) /[Deployment](https://docs.open-metadata.org/latest/deployment) /[Kubernetes](https://docs.open-metadata.org/latest/deployment/kubernetes) /[Eks](https://docs.open-metadata.org/latest/deployment/kubernetes/eks) OpenMetadata Documentation EKS on Amazon Web Services Deployment ===================================== OpenMetadata supports the Installation and Running of Application on Elastic Kubernetes Services (EKS) through Helm Charts. However, there are some additional configurations which needs to be done as prerequisites for the same. All the code snippets in this section assume the `default` namespace for kubernetes. This guide presumes you have AWS EKS Cluster already available. Prerequisites ------------- ### AWS Services for Database as RDS and Search Engine as ElasticSearch It is recommended to use [Amazon RDS](https://docs.aws.amazon.com/rds/index.html) and [Amazon OpenSearch Service](https://docs.aws.amazon.com/opensearch-service/?id=docs_gateway) for Production Deployments. We support * Amazon RDS (MySQL) engine version 8 or higher * Amazon RDS (PostgreSQL) engine version 12 or higher * Amazon OpenSearch engine version 2.X (upto 2.19) When using AWS Services the SearchType Configuration for elastic search should be `opensearch`, for both cases ElasticSearch and OpenSearch, as you can see in the ElasticSearch configuration example below. We recommend * Amazon RDS to be in Multiple Availability Zones. * Amazon OpenSearch (or ElasticSearch) Service with Multiple Availability Zones with minimum 2 Nodes. Make sure to increase `sort_buffer_size` (for MySQL) or `work_mem` (for PostgreSQL) to the recommended value of **20MB** or more using the [database parameter group setting](https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/USER_WorkingWithParamGroups.html) . This is especially important when running migrations to prevent **Out of Sort Memory Error**. You can revert the setting once the migrations are complete. Once you have the RDS and OpenSearch Services Setup, you can update the environment variables below for OpenMetadata kubernetes deployments to connect with Database and ElasticSearch. Make sure to create RDS and OpenSearch credentials as Kubernetes Secrets mentioned [here](https://docs.open-metadata.org/latest/quick-start/local-kubernetes-deployment#2.-create-kubernetes-secrets-required-for-helm-charts) . Also, disable MySQL and ElasticSearch from OpenMetadata Dependencies Helm Charts as mentioned in the FAQs [here](https://docs.open-metadata.org/latest/deployment/kubernetes/eks#how-to-disable-mysql-and-elasticsearch-from-openmetadata-dependencies-helm-charts) . ### Create Elastic File System in AWS You can follow official AWS Guides [here](https://docs.aws.amazon.com/efs/latest/ug/gs-step-two-create-efs-resources.html) to provision EFS File System in the same VPC which is associated with your EKS Cluster. ### Persistent Volumes with ReadWriteMany Access Modes OpenMetadata helm chart depends on Airflow and Airflow expects a persistent disk that support ReadWriteMany (the volume can be mounted as read-write by many nodes). In AWS, this is achieved by Elastic File System (EFS) service. AWS Elastic Block Store (EBS) does not provide ReadWriteMany Volume access mode as EBS will only be attached to one Kubernetes Node at any given point of time. In order to provision persistent volumes from AWS EFS, you will need to setup and install [aws-efs-csi-driver](https://docs.aws.amazon.com/eks/latest/userguide/efs-csi.html) . Note that this is required for Airflow as One OpenMetadata Dependencies. Also, [aws-ebs-csi-driver](https://docs.aws.amazon.com/eks/latest/userguide/ebs-csi.html) might be required for Persistent Volumes that are to be used for MySQL and ElasticSearch as OpenMetadata Dependencies. The below guide provides Persistent Volumes provisioning as static volumes (meaning you will be responsible to create, maintain and destroy Persistent Volumes). Provision EFS backed PVs, PVCs for Airflow DAGs and Airflow Logs ---------------------------------------------------------------- Please note that we are using one AWS Elastic File System (EFS) service with subdirectories as `airflow-dags` and `airflow-logs` with the reference in this documentation. Also, it is presumed that `airflow-dags` and `airflow-logs` directories are already available on that file system. In order to create directories inside the AWS Elastic File System (EFS) you would need to follow these [steps](https://docs.aws.amazon.com/efs/latest/ug/accessing-fs-nfs-permissions-per-user-subdirs.html) . ### Code Samples for PV and PVC for Airflow DAGs Create Persistent Volumes and Persistent Volume claims with the below command. ### Code Samples for PV and PVC for Airflow Logs Create Persistent Volumes and Persistent Volume claims with the below command. Change owner and permission manually on disks --------------------------------------------- Since airflow pods run as non root users, they would not have write access on the nfs server volumes. In order to fix the permission here, spin up a pod with persistent volumes attached and run it once. You can find more reference on AWS EFS permissions in docs [here](https://docs.aws.amazon.com/efs/latest/ug/using-fs.html) . Airflow runs the pods with linux user name as airflow and linux user id as 50000. Run the below command to create the pod and fix the permissions Create OpenMetadata dependencies Values --------------------------------------- Override openmetadata dependencies airflow helm values to bind the efs persistent volumes for DAGs and logs. For more information on airflow helm chart values, please refer to [airflow-helm](https://artifacthub.io/packages/helm/airflow-helm/airflow/8.5.3) . When deploying openmetadata dependencies helm chart, use the below command - The above command uses configurations defined [here](https://raw.githubusercontent.com/open-metadata/openmetadata-helm-charts/main/charts/deps/values.yaml) . You can modify any configuration and deploy by passing your own `values.yaml` Once the openmetadata dependencies helm chart deployed, you can then run the below command to install the openmetadata helm chart - Again, this uses the values defined [here](https://github.com/open-metadata/openmetadata-helm-charts/blob/main/charts/openmetadata/values.yaml) . Use the `--values` flag to point to your own YAML configuration if needed. FAQs ==== Getting an error when install OpenMetadata Dependencies Helm Charts on EKS with EFS ----------------------------------------------------------------------------------- If you are facing the below issue - This error is typically related to EKS Cluster not able to reach to EFS File systems. You can check the security groups associated between the connectivity EFS and EKS. [Here is an article](https://github.com/kubernetes-sigs/aws-efs-csi-driver/blob/master/docs/efs-create-filesystem.md) which further describes the steps required to create Security Group Rules for EKS to use EFS over `port 2049`. It can also happen if the mount targets are already available for EKS Nodes but the Nodes do not pick that up. In such cases, you can do an [AWS AutoScaling Group instance refresh](https://docs.aws.amazon.com/autoscaling/ec2/userguide/start-instance-refresh.html) in order for EKS nodes to get the available mount targets. Java Memory Heap Issue ---------------------- If your openmetadata pods are not in ready state at any point in time and the openmetadata pod logs speaks about the below issue - This is due to the default JVM Heap Space configuration (1 GiB) being not enough for your workloads. In order to resolve this issue, head over to your custom openmetadata helm values and append the below environment variable The flag `Xmx` specifies the maximum memory allocation pool for a Java virtual machine (JVM), while `Xms` specifies the initial memory allocation pool. Upgrade the helm charts with the above changes using the following command `helm upgrade --install openmetadata open-metadata/openmetadata --values --namespace `. Update this command your `values.yml` filename and `namespaceName` where you have deployed OpenMetadata in Kubernetes. PostgreSQL Issue permission denied to create extension "pgcrypto" ----------------------------------------------------------------- If you are facing the below issue with PostgreSQL as Database Backend for OpenMetadata Application, It seems the Database User does not have sufficient privileges. In order to resolve the above issue, grant usage permissions to the PSQL User. In the above command, replace `` with the sql user used by OpenMetadata Application to connect to PostgreSQL Database. How to extend and use custom docker images with OpenMetadata Helm Charts ? -------------------------------------------------------------------------- Extending OpenMetadata Server Docker Image ------------------------------------------ ### 1\. Create a `Dockerfile` based on `docker.getcollate.io/openmetadata/server` OpenMetadata helm charts uses official published docker images from [DockerHub](https://hub.docker.com/u/openmetadata) . A typical scenario will be to install organization certificates for connecting with inhouse systems. For Example - where `docker.getcollate.io/openmetadata/server:x.y.z` needs to point to the same version of the OpenMetadata server, for example `docker.getcollate.io/openmetadata/server:1.3.1`. This image needs to be built and published to the container registry of your choice. ### 2\. Update your openmetadata helm values yaml The OpenMetadata Application gets installed as part of `openmetadata` helm chart. In this step, update the custom helm values using YAML file to point the image created in the previous step. For example, create a helm values file named `values.yaml` with the following contents - ### 3\. Install / Upgrade your helm release Upgrade/Install your openmetadata helm charts with the below single command: Extending OpenMetadata Ingestion Docker Image --------------------------------------------- One possible use case where you would need to use a custom image for the ingestion is because you have developed your own custom connectors. You can find a complete working example of this [here](https://github.com/open-metadata/openmetadata-demo/tree/main/custom-connector) . After you have your code ready, the steps would be the following: ### 1\. Create a `Dockerfile` based on `docker.getcollate.io/openmetadata/ingestion`: For example - where `docker.getcollate.io/openmetadata/ingestion:x.y.z` needs to point to the same version of the OpenMetadata server, for example `docker.getcollate.io/openmetadata/ingestion:1.3.1`. This image needs to be built and published to the container registry of your choice. ### 2\. Update the airflow in openmetadata dependencies values YAML The ingestion containers (which is the one shipping Airflow) gets installed in the `openmetadata-dependencies` helm chart. In this step, we use our own custom values YAML file to point to the image we just created on the previous step. You can create a file named `values.deps.yaml` with the following contents: ### 3\. Install / Upgrade helm release Upgrade/Install your openmetadata-dependencies helm charts with the below single command: How to disable MySQL and ElasticSearch from OpenMetadata Dependencies Helm Charts ? ----------------------------------------------------------------------------------- If you are using MySQL and ElasticSearch externally, you would want to disable the local installation of mysql and elasticsearch while installing OpenMetadata Dependencies Helm Chart. You can disable the MySQL and ElasticSearch Helm Dependencies by setting `enabled: false` value for each dependency. Below is the command to set helm values from Helm CLI - Alternatively, you can create a custom YAML file named `values.deps.yaml` to disable installation of MySQL and Elasticsearch . How to configure external database like PostgreSQL with OpenMetadata Helm Charts ? ---------------------------------------------------------------------------------- OpenMetadata Supports PostgreSQL as one of the Database Dependencies. OpenMetadata Helm Charts by default does not include PostgreSQL as Database Dependencies. In order to configure Helm Charts with External Database like PostgreSQL, follow the below guide to make the helm values change and upgrade / install OpenMetadata helm charts with the same. Upgrade Airflow Helm Dependencies Helm Charts to connect to External Database like PostgreSQL --------------------------------------------------------------------------------------------- We ship [airflow-helm](https://github.com/airflow-helm/charts/tree/main/charts/airflow) as one of OpenMetadata Dependencies with default values to connect to MySQL Database as part of `externalDatabase` configurations. You can find more information on setting the `externalDatabase` as part of helm values [here](https://github.com/airflow-helm/charts/blob/main/charts/airflow/docs/faq/database/external-database.md) . With OpenMetadata Dependencies Helm Charts, your helm values would look something like below - For the above code, it is assumed you are creating a kubernetes secret for storing Airflow Database login Credentials. A sample command to create the secret will be `kubectl create secret generic airflow-postgresql-secrets --from-literal=airflow-postgresql-password=`. Upgrade OpenMetadata Helm Charts to connect to External Database like PostgreSQL -------------------------------------------------------------------------------- Update the `openmetadata.config.database.*` helm values for OpenMetadata Application to connect to External Database like PostgreSQL. With OpenMetadata Helm Charts, your helm values would look something like below - For the above code, it is assumed you are creating a kubernetes secret for storing OpenMetadata Database login Credentials. A sample command to create the secret will be `kubectl create secret generic openmetadata-postgresql-secrets --from-literal=openmetadata-postgresql-password=`. Once you make the above changes to your helm values, run the below command to install/upgrade helm charts - How to customize OpenMetadata Dependencies Helm Chart with custom helm values ----------------------------------------------------------------------------- Our OpenMetadata Dependencies Helm Charts are internally depends on three sub-charts - * [Bitnami MySQL](https://artifacthub.io/packages/helm/bitnami/mysql/9.7.2) (helm chart version 9.7.2) * [OpenSearch](https://artifacthub.io/packages/helm/opensearch-project-helm-charts/opensearch/2.12.2) (helm chart version 2.12.2) * [Airflow](https://artifacthub.io/packages/helm/airflow-helm/airflow/8.8.0) (helm chart version 8.8.0) If you are looking to customize the deployments of any of the above dependencies, please refer to the above links for customizations of helm values for further references. By default, OpenMetadata Dependencies helm chart provides initial generic customization of these helm values in order to get you started quickly. You can refer to the openmetadata-dependencies helm charts default values [here](https://github.com/open-metadata/openmetadata-helm-charts/blob/main/charts/deps/values.yaml) . --- # Kubernetes GKE Deployment | Official Documentation We use cookies to improve site navigation, analyze site usage, and enhance your user experience. Click "Accept" to enable cookies or "Reject" to reject cookies. AcceptReject deployment No menu items for this category [Home](https://docs.open-metadata.org/latest) /[Deployment](https://docs.open-metadata.org/latest/deployment) /[Kubernetes](https://docs.open-metadata.org/latest/deployment/kubernetes) /[Gke](https://docs.open-metadata.org/latest/deployment/kubernetes/gke) OpenMetadata Documentation GKE on Google Cloud Platform Deployment ======================================= OpenMetadata supports the Installation and Running of Application on Google Kubernetes Engine through Helm Charts. However, there are some additional configurations which needs to be done as prerequisites for the same. Google Kubernetes Engine (GKE) Auto Pilot Mode is not compatible with one of OpenMetadata Dependencies - ElasticSearch. The reason being that ElasticSearch Pods require Elevated permissions to run initContainers for changing configurations which is not allowed by GKE AutoPilot PodSecurityPolicy. All the code snippets in this section assume the `default` namespace for kubernetes. Prerequisites ------------- ### Cloud Database with CloudSQL and ElasticCloud for GCP as Search Engine It is recommended to use GCP [Cloud SQL](https://cloud.google.com/sql/) services for Database and [Elastic Cloud GCP](https://www.elastic.co/partners/google-cloud) for Search Engine for Production. We support - * Cloud SQL (MySQL) engine version 8 or higher * Cloud SQL (postgreSQL) engine version 12 or higher * ElasticSearch Engine version 8.X (upto 8.10.X) We recommend - * CloudSQL to be Multi Zone Available * Elastic Cloud Environment with multiple zones and minimum 2 nodes Make sure to increase `sort_buffer_size` ([for MySQL](https://cloud.google.com/sql/docs/mysql/flags) ) or `work_mem` ([for PostgreSQL](https://cloud.google.com/sql/docs/postgres/flags) ) to the recommended value of **20MB** or more using flags. This is especially important when running migrations to prevent **Out of Sort Memory Error**. You can revert the setting once the migrations are complete. Once you have the Database and Search Engine configured and available, update the helm values below for OpenMetadata kubernetes deployments to connect with Database and ElasticSearch. For Database as PostgreSQL, the use the below config for database values - Make sure to create CloudSQL and ElasticSearch credentials as Kubernetes Secrets mentioned [here](https://docs.open-metadata.org/latest/quick-start/local-kubernetes-deployment#2.-create-kubernetes-secrets-required-for-helm-charts) . Also, disable MySQL and ElasticSearch from OpenMetadata Dependencies Helm Charts as mentioned in the FAQs [here](https://docs.open-metadata.org/latest/deployment/kubernetes/gke#how-to-disable-mysql-and-elasticsearch-from-openmetadata-dependencies-helm-charts) . ### Persistent Volumes with ReadWriteMany Access Modes OpenMetadata helm chart depends on Airflow and Airflow expects a persistent disk that support ReadWriteMany (the volume can be mounted as read-write by many nodes). The workaround is to create nfs-server disk on Google Kubernetes Engine and use that as the persistent claim and deploy OpenMetadata by implementing the following steps in order. Create NFS Share ---------------- ### Provision GCP Persistent Disk for Google Kubernetes Engine Run the below command to create a gcloud compute zonal disk. For more information on Google Cloud Disk Options, please visit [here](https://cloud.google.com/compute/docs/disks) . ### Deploy NFS Server in GKE ### Code Samples Run the commands below and ensure the pods are running. We create a ClusterIP Service for pods to access NFS within the cluster at a fixed IP/DNS. ### Provision NFS backed PV and PVC for Airflow DAGs and Airflow Logs Update `` with the NFS Service Cluster IP Address for below code snippets. You can get the clusterIP using the following command ### Code Samples for PV and PVC for Airflow DAGs Create Persistent Volumes and Persistent Volume claims with the below command. ### Code Samples for PV and PVC for Airflow Logs Create Persistent Volumes and Persistent Volume claims with the below command. Change owner and permission manually on disks --------------------------------------------- Since airflow pods run as non root users, they would not have write access on the nfs server volumes. In order to fix the permission here, spin up a pod with persistent volumes attached and run it once. Airflow runs the pods with linux user name as airflow and linux user id as 50000. Run the below command to create the pod and fix the permissions Once the permissions pod is up and running, execute the below commands within the container. Create OpenMetadata dependencies Values --------------------------------------- Override openmetadata dependencies airflow helm values to bind the nfs persistent volumes for DAGs and logs. For more information on airflow helm chart values, please refer to [airflow-helm](https://artifacthub.io/packages/helm/airflow-helm/airflow/8.8.0) . When deploying openmeteadata dependencies helm chart, use the below command - The above command uses configurations defined [here](https://raw.githubusercontent.com/open-metadata/openmetadata-helm-charts/main/charts/deps/values.yaml) . You can modify any configuration and deploy by passing your own `values.yaml` Once the openmetadata dependencies helm chart deployed, you can then run the below command to install the openmetadata helm chart - Troubleshooting =============== Pods are stuck in Pending State due to Persistent Volume Creation Failure ------------------------------------------------------------------------- If you came across `invalid access type while creating the pvc`, and the permission pod is stuck in "pending" state. The above error might have occurred due to the pvc volumes not setup or pvc volumes are not mounted properly. ![dag-log](https://docs.open-metadata.org/images/v1.11/deployment/troubleshoot/dag-log.png) ![permission-pod-events](https://docs.open-metadata.org/images/v1.11/deployment/troubleshoot/permission-pod-events.png) Permission pod events Please validate: * all the prerequisites mentioned in this [section](https://docs.open-metadata.org/latest/deployment/kubernetes/gke#prerequisites) * the configuration of `dags_pv_pvc.yml` file * `storageClassName` field in YAML file FAQs ==== Java Memory Heap Issue ---------------------- If your openmetadata pods are not in ready state at any point in time and the openmetadata pod logs speaks about the below issue - This is due to the default JVM Heap Space configuration (1 GiB) being not enough for your workloads. In order to resolve this issue, head over to your custom openmetadata helm values and append the below environment variable The flag `Xmx` specifies the maximum memory allocation pool for a Java virtual machine (JVM), while `Xms` specifies the initial memory allocation pool. Upgrade the helm charts with the above changes using the following command `helm upgrade --install openmetadata open-metadata/openmetadata --values --namespace `. Update this command your `values.yml` filename and `namespaceName` where you have deployed OpenMetadata in Kubernetes. PostgreSQL Issue permission denied to create extension "pgcrypto" ----------------------------------------------------------------- If you are facing the below issue with PostgreSQL as Database Backend for OpenMetadata Application, It seems the Database User does not have sufficient privileges. In order to resolve the above issue, grant usage permissions to the PSQL User. In the above command, replace `` with the sql user used by OpenMetadata Application to connect to PostgreSQL Database. How to extend and use custom docker images with OpenMetadata Helm Charts ? -------------------------------------------------------------------------- Extending OpenMetadata Server Docker Image ------------------------------------------ ### 1\. Create a `Dockerfile` based on `docker.getcollate.io/openmetadata/server` OpenMetadata helm charts uses official published docker images from [DockerHub](https://hub.docker.com/u/openmetadata) . A typical scenario will be to install organization certificates for connecting with inhouse systems. For Example - where `docker.getcollate.io/openmetadata/server:x.y.z` needs to point to the same version of the OpenMetadata server, for example `docker.getcollate.io/openmetadata/server:1.3.1`. This image needs to be built and published to the container registry of your choice. ### 2\. Update your openmetadata helm values yaml The OpenMetadata Application gets installed as part of `openmetadata` helm chart. In this step, update the custom helm values using YAML file to point the image created in the previous step. For example, create a helm values file named `values.yaml` with the following contents - ### 3\. Install / Upgrade your helm release Upgrade/Install your openmetadata helm charts with the below single command: Extending OpenMetadata Ingestion Docker Image --------------------------------------------- One possible use case where you would need to use a custom image for the ingestion is because you have developed your own custom connectors. You can find a complete working example of this [here](https://github.com/open-metadata/openmetadata-demo/tree/main/custom-connector) . After you have your code ready, the steps would be the following: ### 1\. Create a `Dockerfile` based on `docker.getcollate.io/openmetadata/ingestion`: For example - where `docker.getcollate.io/openmetadata/ingestion:x.y.z` needs to point to the same version of the OpenMetadata server, for example `docker.getcollate.io/openmetadata/ingestion:1.3.1`. This image needs to be built and published to the container registry of your choice. ### 2\. Update the airflow in openmetadata dependencies values YAML The ingestion containers (which is the one shipping Airflow) gets installed in the `openmetadata-dependencies` helm chart. In this step, we use our own custom values YAML file to point to the image we just created on the previous step. You can create a file named `values.deps.yaml` with the following contents: ### 3\. Install / Upgrade helm release Upgrade/Install your openmetadata-dependencies helm charts with the below single command: How to disable MySQL and ElasticSearch from OpenMetadata Dependencies Helm Charts ? ----------------------------------------------------------------------------------- If you are using MySQL and ElasticSearch externally, you would want to disable the local installation of mysql and elasticsearch while installing OpenMetadata Dependencies Helm Chart. You can disable the MySQL and ElasticSearch Helm Dependencies by setting `enabled: false` value for each dependency. Below is the command to set helm values from Helm CLI - Alternatively, you can create a custom YAML file named `values.deps.yaml` to disable installation of MySQL and Elasticsearch . How to configure external database like PostgreSQL with OpenMetadata Helm Charts ? ---------------------------------------------------------------------------------- OpenMetadata Supports PostgreSQL as one of the Database Dependencies. OpenMetadata Helm Charts by default does not include PostgreSQL as Database Dependencies. In order to configure Helm Charts with External Database like PostgreSQL, follow the below guide to make the helm values change and upgrade / install OpenMetadata helm charts with the same. Upgrade Airflow Helm Dependencies Helm Charts to connect to External Database like PostgreSQL --------------------------------------------------------------------------------------------- We ship [airflow-helm](https://github.com/airflow-helm/charts/tree/main/charts/airflow) as one of OpenMetadata Dependencies with default values to connect to MySQL Database as part of `externalDatabase` configurations. You can find more information on setting the `externalDatabase` as part of helm values [here](https://github.com/airflow-helm/charts/blob/main/charts/airflow/docs/faq/database/external-database.md) . With OpenMetadata Dependencies Helm Charts, your helm values would look something like below - For the above code, it is assumed you are creating a kubernetes secret for storing Airflow Database login Credentials. A sample command to create the secret will be `kubectl create secret generic airflow-postgresql-secrets --from-literal=airflow-postgresql-password=`. Upgrade OpenMetadata Helm Charts to connect to External Database like PostgreSQL -------------------------------------------------------------------------------- Update the `openmetadata.config.database.*` helm values for OpenMetadata Application to connect to External Database like PostgreSQL. With OpenMetadata Helm Charts, your helm values would look something like below - For the above code, it is assumed you are creating a kubernetes secret for storing OpenMetadata Database login Credentials. A sample command to create the secret will be `kubectl create secret generic openmetadata-postgresql-secrets --from-literal=openmetadata-postgresql-password=`. Once you make the above changes to your helm values, run the below command to install/upgrade helm charts - How to customize OpenMetadata Dependencies Helm Chart with custom helm values ----------------------------------------------------------------------------- Our OpenMetadata Dependencies Helm Charts are internally depends on three sub-charts - * [Bitnami MySQL](https://artifacthub.io/packages/helm/bitnami/mysql/9.7.2) (helm chart version 9.7.2) * [OpenSearch](https://artifacthub.io/packages/helm/opensearch-project-helm-charts/opensearch/2.12.2) (helm chart version 2.12.2) * [Airflow](https://artifacthub.io/packages/helm/airflow-helm/airflow/8.8.0) (helm chart version 8.8.0) If you are looking to customize the deployments of any of the above dependencies, please refer to the above links for customizations of helm values for further references. By default, OpenMetadata Dependencies helm chart provides initial generic customization of these helm values in order to get you started quickly. You can refer to the openmetadata-dependencies helm charts default values [here](https://github.com/open-metadata/openmetadata-helm-charts/blob/main/charts/deps/values.yaml) . --- # Ingestion Workflows | OpenMetadata Pipeline Orchestration We use cookies to improve site navigation, analyze site usage, and enhance your user experience. Click "Accept" to enable cookies or "Reject" to reject cookies. AcceptReject connectors No menu items for this category [Home](https://docs.open-metadata.org/latest) /[Connectors](https://docs.open-metadata.org/latest/connectors) /[Ingestion](https://docs.open-metadata.org/latest/connectors/ingestion) /[Workflows](https://docs.open-metadata.org/latest/connectors/ingestion/workflows) OpenMetadata Documentation Ingestion Workflows =================== OpenMetadata uses workflows to ingest different kinds of metadata: [Metadata Ingestion\ \ Ingest metadata from any of our connectors](https://docs.open-metadata.org/latest/connectors/ingestion/workflows/metadata) [Usage Information\ \ Find out how your users interact with your data.](https://docs.open-metadata.org/latest/connectors/ingestion/workflows/usage) [Data Lineage\ \ Understand dependencies in your Data Platform.](https://docs.open-metadata.org/latest/connectors/ingestion/workflows/lineage) [dbt Ingestion\ \ Configure dbt metadata](https://docs.open-metadata.org/latest/connectors/ingestion/workflows/dbt) [Data Profiler\ \ Compute metrics and ingest sample data.](https://docs.open-metadata.org/latest/how-to-guides/data-quality-observability/profiler/workflow) [Data Quality\ \ Monitor your data and avoid surprises.](https://docs.open-metadata.org/latest/how-to-guides/data-quality-observability/quality) --- # Lineage Ingestion | OpenMetadata Data Lineage Setup Guide We use cookies to improve site navigation, analyze site usage, and enhance your user experience. Click "Accept" to enable cookies or "Reject" to reject cookies. AcceptReject connectors No menu items for this category [Home](https://docs.open-metadata.org/latest) /[Connectors](https://docs.open-metadata.org/latest/connectors) /[Ingestion](https://docs.open-metadata.org/latest/connectors/ingestion) /[Lineage](https://docs.open-metadata.org/latest/connectors/ingestion/lineage) OpenMetadata Documentation Lineage Ingestion ================= A large subset of connectors distributed with OpenMetadata include support for lineage ingestion. Lineage ingestion processes queries to determine upstream and downstream entities for data assets. Lineage is published to the OpenMetadata catalog when metadata is ingested. Using the OpenMetadata user interface and API, you may trace the path of data across Tables, Pipelines, and Dashboards. ![gif](https://docs.open-metadata.org/images/v1.11/features/ingestion/lineage/lineage-ingestion.gif) Lineage ingestion is specific to the type of the Entity that we are processing. We are going to explain the ingestion process for the supported services. The team is continuously working to increase the lineage coverage of the available services. Do not hesitate to [reach out](https://slack.open-metadata.org/) if you have any questions, issues or requests! Database Services ----------------- Here we have 3 lineage sources, divided in different workflows, but mostly built around a **Query Parser**. ### View Lineage During the Metadata Ingestion workflow we differentiate if a Table is a View. For those sources where we can obtain the query that generates the View (e.g., Snowflake allows us to pick up the View query from the DDL). After all Tables have been ingested in the workflow, it's time to [parse](https://sqllineage.readthedocs.io/en/latest/) all the queries generating Views. During the query parsing, we will obtain the source and target tables, search if the Tables exist in OpenMetadata, and finally create the lineage relationship between the involved Entities. Let's go over this process with an example. Suppose have the following DDL: From this query we will extract the following information: **1.** There are two `source` tables, represented by the string `schema.table_a` as `another_schema.table_b` **2.** There is a `target` table `schema.my_view`. In this case we suppose that the database connection requires us to write the table names as `.`. However, there are other possible options. Sometimes we can find just `
` in a query, or even `..
`. The point here is that we have limited information that we can use to identify the Table Entity that represents the table written down in SQL. To close this gap, we run a query against ElasticSearch using the Table FQN. Once we have identified all the ingredients in OpenMetadata as Entities, we can run the Lineage API to add the relationship between the nodes. ![query-parser](https://docs.open-metadata.org/images/v1.11/features/ingestion/lineage/query-parser.png) What we just described is the core process of identifying and ingesting lineage, and it will be reused (or partially reused) for the rest of the options as well. ### dbt When configuring an Ingestion adding dbt information we can parse the nodes on the Manifest JSON to get the data model lineage. Here we don't need to parse a query to obtain the source and target elements, but we still rely on querying ElasticSearch to identify the graph nodes as OpenMetadata Entities. Note that if a Model is not materialized, its data won't be ingested. #### How to run? The main difference here is between those sources that provide internal access to query logs and those that do not. For services such as [BigQuery](https://docs.open-metadata.org/latest/connectors/database/bigquery) , [Snowflake](https://docs.open-metadata.org/latest/connectors/database/snowflake) etc. There are specific workflows (Usage & Lineage) that will use the query log information. An alternative for sources not listed here is to run the workflow by providing the Query Logs that you have previously exported and then running the [workflow](https://docs.open-metadata.org/latest/connectors/ingestion/workflows/usage/usage-workflow-query-logs) . Checkout the documentation of the connector you are using to know if it supports lineage & usage. #### Process That being said, this process is the same as the one shown in the View Lineage above. By obtaining a set of queries to parse, we will obtain the `source` and `target` information, use ElasticSearch to identify the Entities in OpenMetadata and then send the lineage to the API. When running any query from within OpenMetadata we add an information comment to the query text Note that queries with this text as well as the ones containing headers from dbt (which follow a similar structure), will be filtered out when building the query log internally. #### Troubleshooting Make sure that the tables that you are trying to add lineage for are present in OpenMetadata (and their upstream/downstream as well). You might also need to validate if the query logs are available in the tables for each service. You can check the queries being used here: * [BigQuery](https://github.com/open-metadata/OpenMetadata/blob/main/ingestion/src/metadata/ingestion/source/database/bigquery/queries.py) * [Snowflake](https://github.com/open-metadata/OpenMetadata/blob/main/ingestion/src/metadata/ingestion/source/database/snowflake/queries.py) * [MSSQL](https://github.com/open-metadata/OpenMetadata/blob/main/ingestion/src/metadata/ingestion/source/database/mssql/queries.py) * [Redshift](https://github.com/open-metadata/OpenMetadata/blob/main/ingestion/src/metadata/ingestion/source/database/redshift/queries.py) * [Clickhouse](https://github.com/open-metadata/OpenMetadata/blob/main/ingestion/src/metadata/ingestion/source/database/clickhouse/queries.py) * [PostgreSQL](https://github.com/open-metadata/OpenMetadata/blob/main/ingestion/src/metadata/ingestion/source/database/postgres/queries.py) By default, we apply a result limit of 1000 records. You might also need to increase that for databases with big volumes of queries. Dashboard Services ------------------ When configuring the Ingestion Workflow for Dashboard Services you can select which Database Services are hosting the data feeding the Dashboards and Charts. When ingesting the Dashboards metadata, the workflow will pick up the origin tables (or database, in the case of PowerBI), and prepare the lineage information. ![Dashboard Lineage](https://docs.open-metadata.org/images/v1.11/features/ingestion/lineage/dashboard-ingestion-lineage.png) Dashboard Lineage Pipeline Services ----------------- The supported services here are [Airflow](https://docs.open-metadata.org/latest/connectors/pipeline/airflow) , [Fivetran](https://docs.open-metadata.org/latest/connectors/pipeline/fivetran) , [Dagster](https://docs.open-metadata.org/latest/connectors/pipeline/dagster) and [Airbyte](https://docs.open-metadata.org/latest/connectors/pipeline/airbyte) . All of them ingest the lineage information out of the box. The only special case is Airflow, where one needs to setup `inlets` and `outlets`. You can find more information about it [here](https://docs.open-metadata.org/latest/connectors/pipeline/airflow/lineage-backend#adding-lineage) . Manual Lineage -------------- Sometimes there is information that is shared among people but not present in the sources. To enable capturing all the possible knowledge, you can also add lineage manually with our UI editor. [Manual Lineage\ \ Capture Lineage knowledge with the UI editor.](https://docs.open-metadata.org/latest/connectors/ingestion/lineage/edit-lineage-manually) --- # Usage Workflow Guide | OpenMetadata Ingestion Workflows We use cookies to improve site navigation, analyze site usage, and enhance your user experience. Click "Accept" to enable cookies or "Reject" to reject cookies. AcceptReject connectors No menu items for this category [Home](https://docs.open-metadata.org/latest) /[Connectors](https://docs.open-metadata.org/latest/connectors) /[Ingestion](https://docs.open-metadata.org/latest/connectors/ingestion) /[Workflows](https://docs.open-metadata.org/latest/connectors/ingestion/workflows) /[Usage](https://docs.open-metadata.org/latest/connectors/ingestion/workflows/usage) OpenMetadata Documentation Usage Workflow ============== Learn how to configure the Usage workflow from the UI to ingest Query history data from your data sources. This workflow is available ONLY for the following connectors: * [BigQuery](https://docs.open-metadata.org/latest/connectors/database/bigquery) * [Snowflake](https://docs.open-metadata.org/latest/connectors/database/snowflake) * [MSSQL](https://docs.open-metadata.org/latest/connectors/database/mssql) * [Redshift](https://docs.open-metadata.org/latest/connectors/database/redshift) * [Clickhouse](https://docs.open-metadata.org/latest/connectors/database/clickhouse) * [PostgreSQL](https://docs.open-metadata.org/latest/connectors/database/postgres) * [Databricks](https://docs.open-metadata.org/latest/connectors/database/databricks) If your database service is not yet supported, you can use this same workflow by providing a Query Log file! Learn how to do so πŸ‘‡ [Usage Workflow through Query Logs\ \ Configure the usage workflow by providing a Query Log file.](https://docs.open-metadata.org/latest/connectors/ingestion/workflows/usage/usage-workflow-query-logs) UI Configuration ---------------- Once the metadata ingestion runs correctly and we are able to explore the service Entities, we can add Query Usage information. This will populate the Queries tab from the Table Entity Page. ![table-entity-page](https://docs.open-metadata.org/images/v1.11/features/ingestion/workflows/usage/table-entity-page.png) Table Entity Page We can create a workflow that will obtain the query log and table creation information from the underlying database and feed it to OpenMetadata. The Usage Ingestion will be in charge of obtaining this data. ### 1\. Add a Usage Ingestion From the Service Page, go to the Ingestions tab to add a new ingestion and click on Add Usage Ingestion. ![add-ingestion](https://docs.open-metadata.org/images/v1.11/features/ingestion/workflows/usage/add-ingestion.png) Add Ingestion ### 2\. Configure the Usage Ingestion Here you can enter the Usage Ingestion details: ![configure-usage-ingestion](https://docs.open-metadata.org/images/v1.11/features/ingestion/workflows/usage/configure-usage-ingestion.png) Configure the Usage Ingestion ### Usage Options **Query Log Duration** Specify the duration in days for which the usage should capture usage data from the query logs. For example, if you specify 2 as the value for the duration, the data usage will capture usage information for 48 hours prior to when the ingestion workflow is run. **Stage File Location** Mention the absolute file path of the temporary file name to store the query logs before processing. **Result Limit** Set the limit for the query log results to be run at a time. ### 3\. Schedule and Deploy After clicking Next, you will be redirected to the Scheduling form. This will be the same as the Metadata Ingestion. Select your desired schedule and click on Deploy to find the usage pipeline being added to the Service Ingestions. ![schedule-and-deploy](https://docs.open-metadata.org/images/v1.11/features/ingestion/workflows/usage/scheule-and-deploy.png) View Service Ingestion pipelines YAML Configuration ------------------ In the [connectors](https://docs.open-metadata.org/latest/connectors) section we showcase how to run the metadata ingestion from a JSON/YAML file using the Airflow SDK or the CLI via metadata ingest. Running a usage workflow is also possible using a JSON/YAML configuration file. This is a good option if you wish to execute your workflow via the Airflow SDK or using the CLI; if you use the CLI a usage workflow can be triggered with the command `metadata usage -c FILENAME.yaml`. The `serviceConnection` config will be specific to your connector (you can find more information in the [connectors](https://docs.open-metadata.org/latest/connectors) section), though the sourceConfig for the usage will be similar across all connectors. Query Usage ----------- The Query Usage workflow will be using the `query-parser` processor. After running a Metadata Ingestion workflow, we can run Query Usage workflow. While the `serviceName` will be the same to that was used in Metadata Ingestion, so the ingestion bot can get the `serviceConnection` details from the server. ### 1\. Define the YAML Config This is a sample config for BigQuery Usage: #### Source Configuration - Source Config You can find all the definitions and types for the `sourceConfig` [here](https://github.com/open-metadata/OpenMetadata/blob/main/openmetadata-spec/src/main/resources/json/schema/metadataIngestion/databaseServiceQueryUsagePipeline.json) . **queryLogDuration**: Configuration to tune how far we want to look back in query logs to process usage data. **stageFileLocation**: Temporary file name to store the query logs before processing. Absolute file path required. **resultLimit**: Configuration to set the limit for query logs **queryLogFilePath**: Configuration to set the file path for query logs #### Processor, Stage and Bulk Sink Configuration To specify where the staging files will be located. Note that the location is a directory that will be cleaned at the end of the ingestion. filename.yamlCopy ### 2\. Run with the CLI After saving the YAML config, we will run the command the same way we did for the metadata ingestion: --- # Ingestion Framework Deployment | Official Documentation We use cookies to improve site navigation, analyze site usage, and enhance your user experience. Click "Accept" to enable cookies or "Reject" to reject cookies. AcceptReject deployment No menu items for this category [Home](https://docs.open-metadata.org/latest) /[Deployment](https://docs.open-metadata.org/latest/deployment) /[Ingestion](https://docs.open-metadata.org/latest/deployment/ingestion) OpenMetadata Documentation Ingestion Framework Deployment ============================== The Ingestion Framework is the module that takes care of bringing metadata in to OpenMetadata. It is used for any type of workflow that is supported in the platform: Metadata, Lineage, Usage, Profiler, Data Quality,... Manage & Schedule the Ingestion Framework ----------------------------------------- In this guide, we will present the different alternatives to run and manage your ingestion workflows. There are mainly 2 ways of running the ingestion: 1. Internally, by managing the workflows from OpenMetadata. 2. Externally, by using any other tool capable of running Python code. Note that the end result is going to be the same. The only difference is that running the workflows internally, OpenMetadata will dynamically generate the processes that will perform the metadata extraction. If configuring the ingestion externally, you will be managing this processes directly on your platform of choice. Option 1 - From OpenMetadata ---------------------------- If you want to learn how to configure your setup to run them from OpenMetadata, follow this guide: [OpenMetadata UI\ \ Deploy, configure and manage the ingestion workflows directly from the OpenMetadata UI.](https://docs.open-metadata.org/latest/deployment/ingestion/openmetadata) Option 2 - Externally --------------------- Any tool capable of running Python code can be used to configure the metadata extraction from your sources. In this section, we are going to give you some background on how the Ingestion Framework works, how to configure the metadata extraction, and some examples on how to host the ingestion in different platforms. [External Ingestion\ \ Manage the Ingestion Framework from anywhere!](https://docs.open-metadata.org/latest/deployment/ingestion/external) --- # JWT validation Troubleshooting | Official Documentation We use cookies to improve site navigation, analyze site usage, and enhance your user experience. Click "Accept" to enable cookies or "Reject" to reject cookies. AcceptReject ![](https://static.scarf.sh/a.png?x-pxid=8472f2b3-d307-4935-a5ed-3f9d3982c15d) deployment No menu items for this category [Home](https://docs.open-metadata.org/latest) /[Deployment](https://docs.open-metadata.org/latest/deployment) /[Security](https://docs.open-metadata.org/latest/deployment/security) /[Jwt Troubleshooting](https://docs.open-metadata.org/latest/deployment/security/jwt-troubleshooting) OpenMetadata Documentation JWT Troubleshooting =================== Add the `{domain}:{port}/api/v1/system/config/jwks` in the list of publicKeys This config with `"http://localhost:8585/api/v1/system/config/jwks"` is the default behavior. If you are configuring and expecting a JWT token to work, configuring with that extra URL is required. JWT Tokens are issued by private certificates. We need public keys to decrypt it and get that token's user name, expiry time, etc. In OpenMetadata users can enable SSO for users to login and use JWT tokens issued by OpenMetadata for bots The way OpenMetadata issues a JWT Token is using this [config](https://github.com/open-metadata/OpenMetadata/blob/main/conf/openmetadata.yaml#L155) . It uses the `rsapublicKeyFilePath` file to generate a token. When the ingestion workflow uses this token, we use `rsapublicKeyPath` to decrypt it. The way we do this is using the response from this endpoint `http://localhost:8585/api/v1/system/config/jwks`. Get JWT token from UI. ---------------------- First Open Open-Metadata UI than go to settings > Bots > Ingestion Bot ![jwt-token](https://docs.open-metadata.org/images/v1.11/deployment/troubleshoot/jwt-token.png) JWT token in OpenMetadata UI You can validate that in [jwt.io](https://jwt.io/) . if there's something wrong on how the JWT token was generated. ![jwt.io](https://docs.open-metadata.org/images/v1.11/deployment/troubleshoot/jwt-validation.png) jwt.io tool for validating JWT claims ### Resolving the "Failed in filtering request: Not Authorized! Token not present" Error If you encounter the error message **"Failed in filtering request: Not Authorized! Token not present"**, verify the **`enableSecureSocketConnection`** environment setting. Ensure that **`enableSecureSocketConnection: ${AUTHORIZER_ENABLE_SECURE_SOCKET:-false}`** is set to `false` if it is currently set to `true`. --- # Enable SSL in Airflow | OpenMetadata Security Guide We use cookies to improve site navigation, analyze site usage, and enhance your user experience. Click "Accept" to enable cookies or "Reject" to reject cookies. AcceptReject ![](https://static.scarf.sh/a.png?x-pxid=8472f2b3-d307-4935-a5ed-3f9d3982c15d) deployment No menu items for this category [Home](https://docs.open-metadata.org/latest) /[Deployment](https://docs.open-metadata.org/latest/deployment) /[Security](https://docs.open-metadata.org/latest/deployment/security) /[Enable Ssl](https://docs.open-metadata.org/latest/deployment/security/enable-ssl) /[Airflow](https://docs.open-metadata.org/latest/deployment/security/enable-ssl/airflow) OpenMetadata Documentation Configure OpenMetadata certificates in Airflow ============================================== Follow this section if you added SSL certs in the OpenMetadata server. The OpenMetadata configuration related to Airflow (or in general, the Pipeline Service Client) is the following: Then, in order to add this, you can either update the `openmetadata.yaml` config if your deployment is Bare Metal, or update the following environment variables: * `PIPELINE_SERVICE_CLIENT_VERIFY_SSL=validate` * `PIPELINE_SERVICE_CLIENT_SSL_CERT_PATH="path/to/cert` Note that the `PIPELINE_SERVICE_CLIENT_SSL_CERT_PATH` should be the path to the certificate you generated [here](https://docs.open-metadata.org/latest/deployment/security/enable-ssl) , and it should be the local path in your Airflow deployment. Enable SSL in Airflow ===================== Follow this section if you want to add SSL certificates in Airflow. This will secure the connection from the OpenMetadata to Airflow. Airflow has two [configurations](https://airflow.apache.org/docs/apache-airflow/stable/configurations-ref.html#web-server-ssl-cert) to be added in `airflow.cfg` to enable SSL: * `AIRFLOW__WEBSERVER__WEB_SERVER_SSL_CERT` * `AIRFLOW__WEBSERVER__WEB_SERVER_SSL_KEY` Those are files that will need to be local to the Airflow deployment. Generate Certs -------------- We can generate these files following this [SO](https://stackoverflow.com/questions/47883769/how-to-enable-ssl-on-apache-airflow) thread: and we can provide the following answers to try this locally: It is important that the `Common Name` is the host name that will be hosting Airflow. This command will generate the pair `airflow.key` and `airflow.crt`. Include Certificates -------------------- Once the files are generated we need to add them to the Airflow deployment. For example, if using the `openmetadata-ingestion` image, you can update it to add the following lines: If you now start Airflow with these changes, it will be running at `https://localhost:8080`. Update the OpenMetadata configuration ------------------------------------- Since Airflow will be using SSL, we need to update the OpenMetadata Server configuration to use the certificates when preparing the connection to the Airflow Webserver. The `pipelineServiceClientConfiguration` will look like the following: Update the `truststorePath` and `truststorePassword` accordingly, pointing to the `keystore` in your server host holding the certificates we created. For docker deployments, you will provide OpenMetadata Server Application with the self signed certificates of Airflow bundled in JVM keystore. These will be passed to the application using `AIRFLOW_TRUST_STORE_PATH` and `AIRFLOW_TRUST_STORE_PASSWORD` environment variable. Please make sure to have the the truststore file mounted and available as part of Docker Deployments. For kubernetes deployments, update the helm values as below - In the above code snippet, we are mounting the volumes of truststore file from a kubernetes secret. You can create the secret from `truststore.jks` file from the below `kubectl` command - Next, restart or redeploy openmetadata application to take the above configs in effect. ### Example: Setting it locally For example, if we are running the server locally, we need to add the certificate to the JVM `cacerts` store: Then, the values of the YAML config would be something similar to: Make sure to update these values to the ones in your host. Also, it's always preferred to use environment variables instead of hardcoding sensitive information. --- # API Services | OpenMetadata Connector Integration Guide We use cookies to improve site navigation, analyze site usage, and enhance your user experience. Click "Accept" to enable cookies or "Reject" to reject cookies. AcceptReject connectors No menu items for this category [Home](https://docs.open-metadata.org/latest) /[Connectors](https://docs.open-metadata.org/latest/connectors) /[Api](https://docs.open-metadata.org/latest/connectors/api) OpenMetadata Documentation API Services ============ Overview -------- The OpenMetadata API service facilitates metadata ingestion from RESTful APIs that expose OpenAPI (Swagger) specifications. This connector is particularly useful for integrating custom services or third-party tools that are not natively supported by OpenMetadata. This is the supported list of connectors for API Services: [![REST](https://docs.open-metadata.org/_next/image?url=%2Fimages%2Fconnectors%2Frest.webp&w=64&q=75)\ \ REST\ \ PROD\ \ Available In](https://docs.open-metadata.org/latest/connectors/api/rest) Supported Features ------------------ * Metadata Ingestion: Extracts metadata from services exposing OpenAPI JSON schemas. * Custom Integration: Allows integration with bespoke systems through standardized API definitions. * Flexible Deployment: Supports both UI-based and CLI-based ingestion workflows. Use Cases --------- * Custom Application Integration: Ingest metadata from proprietary applications exposing OpenAPI specifications. * Third-Party Tools: Integrate with external tools and platforms that provide RESTful APIs. * Extended Metadata Management: Enhance metadata coverage by incorporating services beyond the default connectors. Best Practices -------------- * Schema Validation: Ensure the OpenAPI JSON schema is valid and accessible. * Authentication Management: Securely store and manage authentication tokens required for API access. * Regular Updates: Periodically update the OpenAPI JSON schema URL if the API definitions change. If you have a request for a new connector, don't hesitate to reach out in [Slack](https://slack.open-metadata.org/) or open a [feature request](https://github.com/open-metadata/OpenMetadata/issues/new/choose) in our GitHub repo. --- # Nifi Connector Troubleshooting | Official Documentation We use cookies to improve site navigation, analyze site usage, and enhance your user experience. Click "Accept" to enable cookies or "Reject" to reject cookies. AcceptReject connectors No menu items for this category [Home](https://docs.open-metadata.org/latest) /[Connectors](https://docs.open-metadata.org/latest/connectors) /[Pipeline](https://docs.open-metadata.org/latest/connectors/pipeline) /[Nifi](https://docs.open-metadata.org/latest/connectors/pipeline/nifi) /[Troubleshooting](https://docs.open-metadata.org/latest/connectors/pipeline/nifi/troubleshooting) OpenMetadata Documentation Troubleshooting =============== Workflow Deployment Error ------------------------- If there were any errors during the workflow deployment process, the Ingestion Pipeline Entity will still be created, but no workflow will be present in the Ingestion container. * You can then Edit the Ingestion Pipeline and **Deploy** it again. * From the Connection tab, you can also Edit the Service if needed. Connector Debug Troubleshooting ------------------------------- This section provides instructions to help resolve common issues encountered during connector setup and metadata ingestion in OpenMetadata. Below are some of the most frequently observed troubleshooting scenarios. How to Enable Debug Logging for Any Ingestion --------------------------------------------- To enable debug logging for any ingestion workflow in OpenMetadata: 1. **Navigate to Services** Go to **Settings > Services > Service Type** (e.g., Database) in the OpenMetadata UI. 2. **Select a Service** Choose the specific service for which you want to enable debug logging. ![Select a Service](https://docs.open-metadata.org/images/v1.11/connectors/debug/debug1.png) Select a Service 3. **Access Ingestion Tab** Go to the **Ingestion tab** and click the three-dot menu on the right-hand side of the ingestion type, and select Edit. ![Access Agents Tab](https://docs.open-metadata.org/images/v1.11/connectors/debug/debug2.png) Access Agents Tab 4. **Enable Debug Logging** In the configuration dialog, enable the **Debug Log** option and click **Next**. ![Enable Debug Logging](https://docs.open-metadata.org/images/v1.11/connectors/debug/debug3.png) Enable Debug Logging 5. **Schedule and Submit** Configure the schedule if needed and click **Submit** to apply the changes. ![Schedule and Submit](https://docs.open-metadata.org/images/v1.11/connectors/debug/debug4.png) Schedule and Submit Permission Issues ----------------- If you encounter permission-related errors during connector setup or metadata ingestion, ensure that all the prerequisites and access configurations specified for each connector are properly implemented. Refer to the connector-specific documentation to verify the required permissions. Learn how to resolve the most common problems people encounter in the Nifi connector. No applicable policies could be found ------------------------------------- If you see the error `No applicable policies could be found. Contact the system administrator` during the Test Connection or when running the ingestion, you will need to add the missing policies in the Nifi instance. You can find more information in this [link](https://community.cloudera.com/t5/Support-Questions/API-call-to-nifi-api-resources-results-in-quot-No-applicable/td-p/363534) . The accepted answer is to add a policy to `authorizations.xml` as follows: --- # Exasol Troubleshooting Guide | OpenMetadata Support We use cookies to improve site navigation, analyze site usage, and enhance your user experience. Click "Accept" to enable cookies or "Reject" to reject cookies. AcceptReject connectors No menu items for this category [Home](https://docs.open-metadata.org/latest) /[Connectors](https://docs.open-metadata.org/latest/connectors) /[Database](https://docs.open-metadata.org/latest/connectors/database) /[Exasol](https://docs.open-metadata.org/latest/connectors/database/exasol) /[Troubleshooting](https://docs.open-metadata.org/latest/connectors/database/exasol/troubleshooting) OpenMetadata Documentation Troubleshooting =============== Workflow Deployment Error ------------------------- If there were any errors during the workflow deployment process, the Ingestion Pipeline Entity will still be created, but no workflow will be present in the Ingestion container. * You can then Edit the Ingestion Pipeline and **Deploy** it again. * From the Connection tab, you can also Edit the Service if needed. Connector Debug Troubleshooting ------------------------------- This section provides instructions to help resolve common issues encountered during connector setup and metadata ingestion in OpenMetadata. Below are some of the most frequently observed troubleshooting scenarios. How to Enable Debug Logging for Any Ingestion --------------------------------------------- To enable debug logging for any ingestion workflow in OpenMetadata: 1. **Navigate to Services** Go to **Settings > Services > Service Type** (e.g., Database) in the OpenMetadata UI. 2. **Select a Service** Choose the specific service for which you want to enable debug logging. ![Select a Service](https://docs.open-metadata.org/images/v1.11/connectors/debug/debug1.png) Select a Service 3. **Access Ingestion Tab** Go to the **Ingestion tab** and click the three-dot menu on the right-hand side of the ingestion type, and select Edit. ![Access Agents Tab](https://docs.open-metadata.org/images/v1.11/connectors/debug/debug2.png) Access Agents Tab 4. **Enable Debug Logging** In the configuration dialog, enable the **Debug Log** option and click **Next**. ![Enable Debug Logging](https://docs.open-metadata.org/images/v1.11/connectors/debug/debug3.png) Enable Debug Logging 5. **Schedule and Submit** Configure the schedule if needed and click **Submit** to apply the changes. ![Schedule and Submit](https://docs.open-metadata.org/images/v1.11/connectors/debug/debug4.png) Schedule and Submit Permission Issues ----------------- If you encounter permission-related errors during connector setup or metadata ingestion, ensure that all the prerequisites and access configurations specified for each connector are properly implemented. Refer to the connector-specific documentation to verify the required permissions. --- # Kinesis Connector Troubleshooting Guide | OpenMetadata Support We use cookies to improve site navigation, analyze site usage, and enhance your user experience. Click "Accept" to enable cookies or "Reject" to reject cookies. AcceptReject connectors No menu items for this category [Home](https://docs.open-metadata.org/latest) /[Connectors](https://docs.open-metadata.org/latest/connectors) /[Messaging](https://docs.open-metadata.org/latest/connectors/messaging) /[Kinesis](https://docs.open-metadata.org/latest/connectors/messaging/kinesis) /[Troubleshooting](https://docs.open-metadata.org/latest/connectors/messaging/kinesis/troubleshooting) OpenMetadata Documentation Troubleshooting =============== Workflow Deployment Error ------------------------- If there were any errors during the workflow deployment process, the Ingestion Pipeline Entity will still be created, but no workflow will be present in the Ingestion container. * You can then Edit the Ingestion Pipeline and **Deploy** it again. * From the Connection tab, you can also Edit the Service if needed. Connector Debug Troubleshooting ------------------------------- This section provides instructions to help resolve common issues encountered during connector setup and metadata ingestion in OpenMetadata. Below are some of the most frequently observed troubleshooting scenarios. How to Enable Debug Logging for Any Ingestion --------------------------------------------- To enable debug logging for any ingestion workflow in OpenMetadata: 1. **Navigate to Services** Go to **Settings > Services > Service Type** (e.g., Database) in the OpenMetadata UI. 2. **Select a Service** Choose the specific service for which you want to enable debug logging. ![Select a Service](https://docs.open-metadata.org/images/v1.11/connectors/debug/debug1.png) Select a Service 3. **Access Ingestion Tab** Go to the **Ingestion tab** and click the three-dot menu on the right-hand side of the ingestion type, and select Edit. ![Access Agents Tab](https://docs.open-metadata.org/images/v1.11/connectors/debug/debug2.png) Access Agents Tab 4. **Enable Debug Logging** In the configuration dialog, enable the **Debug Log** option and click **Next**. ![Enable Debug Logging](https://docs.open-metadata.org/images/v1.11/connectors/debug/debug3.png) Enable Debug Logging 5. **Schedule and Submit** Configure the schedule if needed and click **Submit** to apply the changes. ![Schedule and Submit](https://docs.open-metadata.org/images/v1.11/connectors/debug/debug4.png) Schedule and Submit Permission Issues ----------------- If you encounter permission-related errors during connector setup or metadata ingestion, ensure that all the prerequisites and access configurations specified for each connector are properly implemented. Refer to the connector-specific documentation to verify the required permissions. --- # Lightdash Troubleshooting Guide | OpenMetadata Support We use cookies to improve site navigation, analyze site usage, and enhance your user experience. Click "Accept" to enable cookies or "Reject" to reject cookies. AcceptReject connectors No menu items for this category [Home](https://docs.open-metadata.org/latest) /[Connectors](https://docs.open-metadata.org/latest/connectors) /[Dashboard](https://docs.open-metadata.org/latest/connectors/dashboard) /[Lightdash](https://docs.open-metadata.org/latest/connectors/dashboard/lightdash) /[Troubleshooting](https://docs.open-metadata.org/latest/connectors/dashboard/lightdash/troubleshooting) OpenMetadata Documentation Troubleshooting =============== Workflow Deployment Error ------------------------- If there were any errors during the workflow deployment process, the Ingestion Pipeline Entity will still be created, but no workflow will be present in the Ingestion container. * You can then Edit the Ingestion Pipeline and **Deploy** it again. * From the Connection tab, you can also Edit the Service if needed. Connector Debug Troubleshooting ------------------------------- This section provides instructions to help resolve common issues encountered during connector setup and metadata ingestion in OpenMetadata. Below are some of the most frequently observed troubleshooting scenarios. How to Enable Debug Logging for Any Ingestion --------------------------------------------- To enable debug logging for any ingestion workflow in OpenMetadata: 1. **Navigate to Services** Go to **Settings > Services > Service Type** (e.g., Database) in the OpenMetadata UI. 2. **Select a Service** Choose the specific service for which you want to enable debug logging. ![Select a Service](https://docs.open-metadata.org/images/v1.11/connectors/debug/debug1.png) Select a Service 3. **Access Ingestion Tab** Go to the **Ingestion tab** and click the three-dot menu on the right-hand side of the ingestion type, and select Edit. ![Access Agents Tab](https://docs.open-metadata.org/images/v1.11/connectors/debug/debug2.png) Access Agents Tab 4. **Enable Debug Logging** In the configuration dialog, enable the **Debug Log** option and click **Next**. ![Enable Debug Logging](https://docs.open-metadata.org/images/v1.11/connectors/debug/debug3.png) Enable Debug Logging 5. **Schedule and Submit** Configure the schedule if needed and click **Submit** to apply the changes. ![Schedule and Submit](https://docs.open-metadata.org/images/v1.11/connectors/debug/debug4.png) Schedule and Submit Permission Issues ----------------- If you encounter permission-related errors during connector setup or metadata ingestion, ensure that all the prerequisites and access configurations specified for each connector are properly implemented. Refer to the connector-specific documentation to verify the required permissions. --- # Dagster Connector Troubleshooting Guide | OpenMetadata Support We use cookies to improve site navigation, analyze site usage, and enhance your user experience. Click "Accept" to enable cookies or "Reject" to reject cookies. AcceptReject connectors No menu items for this category [Home](https://docs.open-metadata.org/latest) /[Connectors](https://docs.open-metadata.org/latest/connectors) /[Pipeline](https://docs.open-metadata.org/latest/connectors/pipeline) /[Dagster](https://docs.open-metadata.org/latest/connectors/pipeline/dagster) /[Troubleshooting](https://docs.open-metadata.org/latest/connectors/pipeline/dagster/troubleshooting) OpenMetadata Documentation Troubleshooting =============== Workflow Deployment Error ------------------------- If there were any errors during the workflow deployment process, the Ingestion Pipeline Entity will still be created, but no workflow will be present in the Ingestion container. * You can then Edit the Ingestion Pipeline and **Deploy** it again. * From the Connection tab, you can also Edit the Service if needed. Connector Debug Troubleshooting ------------------------------- This section provides instructions to help resolve common issues encountered during connector setup and metadata ingestion in OpenMetadata. Below are some of the most frequently observed troubleshooting scenarios. How to Enable Debug Logging for Any Ingestion --------------------------------------------- To enable debug logging for any ingestion workflow in OpenMetadata: 1. **Navigate to Services** Go to **Settings > Services > Service Type** (e.g., Database) in the OpenMetadata UI. 2. **Select a Service** Choose the specific service for which you want to enable debug logging. ![Select a Service](https://docs.open-metadata.org/images/v1.11/connectors/debug/debug1.png) Select a Service 3. **Access Ingestion Tab** Go to the **Ingestion tab** and click the three-dot menu on the right-hand side of the ingestion type, and select Edit. ![Access Agents Tab](https://docs.open-metadata.org/images/v1.11/connectors/debug/debug2.png) Access Agents Tab 4. **Enable Debug Logging** In the configuration dialog, enable the **Debug Log** option and click **Next**. ![Enable Debug Logging](https://docs.open-metadata.org/images/v1.11/connectors/debug/debug3.png) Enable Debug Logging 5. **Schedule and Submit** Configure the schedule if needed and click **Submit** to apply the changes. ![Schedule and Submit](https://docs.open-metadata.org/images/v1.11/connectors/debug/debug4.png) Schedule and Submit Permission Issues ----------------- If you encounter permission-related errors during connector setup or metadata ingestion, ensure that all the prerequisites and access configurations specified for each connector are properly implemented. Refer to the connector-specific documentation to verify the required permissions. --- # Fivetran Troubleshooting Guide | OpenMetadata Support We use cookies to improve site navigation, analyze site usage, and enhance your user experience. Click "Accept" to enable cookies or "Reject" to reject cookies. AcceptReject connectors No menu items for this category [Home](https://docs.open-metadata.org/latest) /[Connectors](https://docs.open-metadata.org/latest/connectors) /[Pipeline](https://docs.open-metadata.org/latest/connectors/pipeline) /[Fivetran](https://docs.open-metadata.org/latest/connectors/pipeline/fivetran) /[Troubleshooting](https://docs.open-metadata.org/latest/connectors/pipeline/fivetran/troubleshooting) OpenMetadata Documentation Troubleshooting =============== Workflow Deployment Error ------------------------- If there were any errors during the workflow deployment process, the Ingestion Pipeline Entity will still be created, but no workflow will be present in the Ingestion container. * You can then Edit the Ingestion Pipeline and **Deploy** it again. * From the Connection tab, you can also Edit the Service if needed. Connector Debug Troubleshooting ------------------------------- This section provides instructions to help resolve common issues encountered during connector setup and metadata ingestion in OpenMetadata. Below are some of the most frequently observed troubleshooting scenarios. How to Enable Debug Logging for Any Ingestion --------------------------------------------- To enable debug logging for any ingestion workflow in OpenMetadata: 1. **Navigate to Services** Go to **Settings > Services > Service Type** (e.g., Database) in the OpenMetadata UI. 2. **Select a Service** Choose the specific service for which you want to enable debug logging. ![Select a Service](https://docs.open-metadata.org/images/v1.11/connectors/debug/debug1.png) Select a Service 3. **Access Ingestion Tab** Go to the **Ingestion tab** and click the three-dot menu on the right-hand side of the ingestion type, and select Edit. ![Access Agents Tab](https://docs.open-metadata.org/images/v1.11/connectors/debug/debug2.png) Access Agents Tab 4. **Enable Debug Logging** In the configuration dialog, enable the **Debug Log** option and click **Next**. ![Enable Debug Logging](https://docs.open-metadata.org/images/v1.11/connectors/debug/debug3.png) Enable Debug Logging 5. **Schedule and Submit** Configure the schedule if needed and click **Submit** to apply the changes. ![Schedule and Submit](https://docs.open-metadata.org/images/v1.11/connectors/debug/debug4.png) Schedule and Submit Permission Issues ----------------- If you encounter permission-related errors during connector setup or metadata ingestion, ensure that all the prerequisites and access configurations specified for each connector are properly implemented. Refer to the connector-specific documentation to verify the required permissions. --- # REST Connector Troubleshooting Guide | OpenMetadata Support We use cookies to improve site navigation, analyze site usage, and enhance your user experience. Click "Accept" to enable cookies or "Reject" to reject cookies. AcceptReject connectors No menu items for this category [Home](https://docs.open-metadata.org/latest) /[Connectors](https://docs.open-metadata.org/latest/connectors) /[Api](https://docs.open-metadata.org/latest/connectors/api) /[Rest](https://docs.open-metadata.org/latest/connectors/api/rest) /[Troubleshooting](https://docs.open-metadata.org/latest/connectors/api/rest/troubleshooting) OpenMetadata Documentation Troubleshooting =============== Workflow Deployment Error ------------------------- If there were any errors during the workflow deployment process, the Ingestion Pipeline Entity will still be created, but no workflow will be present in the Ingestion container. * You can then Edit the Ingestion Pipeline and **Deploy** it again. * From the Connection tab, you can also Edit the Service if needed. Connector Debug Troubleshooting ------------------------------- This section provides instructions to help resolve common issues encountered during connector setup and metadata ingestion in OpenMetadata. Below are some of the most frequently observed troubleshooting scenarios. How to Enable Debug Logging for Any Ingestion --------------------------------------------- To enable debug logging for any ingestion workflow in OpenMetadata: 1. **Navigate to Services** Go to **Settings > Services > Service Type** (e.g., Database) in the OpenMetadata UI. 2. **Select a Service** Choose the specific service for which you want to enable debug logging. ![Select a Service](https://docs.open-metadata.org/images/v1.11/connectors/debug/debug1.png) Select a Service 3. **Access Ingestion Tab** Go to the **Ingestion tab** and click the three-dot menu on the right-hand side of the ingestion type, and select Edit. ![Access Agents Tab](https://docs.open-metadata.org/images/v1.11/connectors/debug/debug2.png) Access Agents Tab 4. **Enable Debug Logging** In the configuration dialog, enable the **Debug Log** option and click **Next**. ![Enable Debug Logging](https://docs.open-metadata.org/images/v1.11/connectors/debug/debug3.png) Enable Debug Logging 5. **Schedule and Submit** Configure the schedule if needed and click **Submit** to apply the changes. ![Schedule and Submit](https://docs.open-metadata.org/images/v1.11/connectors/debug/debug4.png) Schedule and Submit Permission Issues ----------------- If you encounter permission-related errors during connector setup or metadata ingestion, ensure that all the prerequisites and access configurations specified for each connector are properly implemented. Refer to the connector-specific documentation to verify the required permissions. --- # Mode Dashboard Troubleshooting Guide | OpenMetadata Support We use cookies to improve site navigation, analyze site usage, and enhance your user experience. Click "Accept" to enable cookies or "Reject" to reject cookies. AcceptReject connectors No menu items for this category [Home](https://docs.open-metadata.org/latest) /[Connectors](https://docs.open-metadata.org/latest/connectors) /[Dashboard](https://docs.open-metadata.org/latest/connectors/dashboard) /[Mode](https://docs.open-metadata.org/latest/connectors/dashboard/mode) /[Troubleshooting](https://docs.open-metadata.org/latest/connectors/dashboard/mode/troubleshooting) OpenMetadata Documentation Troubleshooting =============== Workflow Deployment Error ------------------------- If there were any errors during the workflow deployment process, the Ingestion Pipeline Entity will still be created, but no workflow will be present in the Ingestion container. * You can then Edit the Ingestion Pipeline and **Deploy** it again. * From the Connection tab, you can also Edit the Service if needed. Connector Debug Troubleshooting ------------------------------- This section provides instructions to help resolve common issues encountered during connector setup and metadata ingestion in OpenMetadata. Below are some of the most frequently observed troubleshooting scenarios. How to Enable Debug Logging for Any Ingestion --------------------------------------------- To enable debug logging for any ingestion workflow in OpenMetadata: 1. **Navigate to Services** Go to **Settings > Services > Service Type** (e.g., Database) in the OpenMetadata UI. 2. **Select a Service** Choose the specific service for which you want to enable debug logging. ![Select a Service](https://docs.open-metadata.org/images/v1.11/connectors/debug/debug1.png) Select a Service 3. **Access Ingestion Tab** Go to the **Ingestion tab** and click the three-dot menu on the right-hand side of the ingestion type, and select Edit. ![Access Agents Tab](https://docs.open-metadata.org/images/v1.11/connectors/debug/debug2.png) Access Agents Tab 4. **Enable Debug Logging** In the configuration dialog, enable the **Debug Log** option and click **Next**. ![Enable Debug Logging](https://docs.open-metadata.org/images/v1.11/connectors/debug/debug3.png) Enable Debug Logging 5. **Schedule and Submit** Configure the schedule if needed and click **Submit** to apply the changes. ![Schedule and Submit](https://docs.open-metadata.org/images/v1.11/connectors/debug/debug4.png) Schedule and Submit Permission Issues ----------------- If you encounter permission-related errors during connector setup or metadata ingestion, ensure that all the prerequisites and access configurations specified for each connector are properly implemented. Refer to the connector-specific documentation to verify the required permissions. --- # MongoDB Troubleshooting Guide | OpenMetadata Support We use cookies to improve site navigation, analyze site usage, and enhance your user experience. Click "Accept" to enable cookies or "Reject" to reject cookies. AcceptReject connectors No menu items for this category [Home](https://docs.open-metadata.org/latest) /[Connectors](https://docs.open-metadata.org/latest/connectors) /[Database](https://docs.open-metadata.org/latest/connectors/database) /[Mongodb](https://docs.open-metadata.org/latest/connectors/database/mongodb) /[Troubleshooting](https://docs.open-metadata.org/latest/connectors/database/mongodb/troubleshooting) OpenMetadata Documentation Troubleshooting =============== Workflow Deployment Error ------------------------- If there were any errors during the workflow deployment process, the Ingestion Pipeline Entity will still be created, but no workflow will be present in the Ingestion container. * You can then Edit the Ingestion Pipeline and **Deploy** it again. * From the Connection tab, you can also Edit the Service if needed. Connector Debug Troubleshooting ------------------------------- This section provides instructions to help resolve common issues encountered during connector setup and metadata ingestion in OpenMetadata. Below are some of the most frequently observed troubleshooting scenarios. How to Enable Debug Logging for Any Ingestion --------------------------------------------- To enable debug logging for any ingestion workflow in OpenMetadata: 1. **Navigate to Services** Go to **Settings > Services > Service Type** (e.g., Database) in the OpenMetadata UI. 2. **Select a Service** Choose the specific service for which you want to enable debug logging. ![Select a Service](https://docs.open-metadata.org/images/v1.11/connectors/debug/debug1.png) Select a Service 3. **Access Ingestion Tab** Go to the **Ingestion tab** and click the three-dot menu on the right-hand side of the ingestion type, and select Edit. ![Access Agents Tab](https://docs.open-metadata.org/images/v1.11/connectors/debug/debug2.png) Access Agents Tab 4. **Enable Debug Logging** In the configuration dialog, enable the **Debug Log** option and click **Next**. ![Enable Debug Logging](https://docs.open-metadata.org/images/v1.11/connectors/debug/debug3.png) Enable Debug Logging 5. **Schedule and Submit** Configure the schedule if needed and click **Submit** to apply the changes. ![Schedule and Submit](https://docs.open-metadata.org/images/v1.11/connectors/debug/debug4.png) Schedule and Submit Permission Issues ----------------- If you encounter permission-related errors during connector setup or metadata ingestion, ensure that all the prerequisites and access configurations specified for each connector are properly implemented. Refer to the connector-specific documentation to verify the required permissions. Resolving MongoDB Ingestion Failures ------------------------------------ When attempting to connect OpenMetadata to MongoDB, particularly cloud-hosted or replica set deployments, you might encounter timeout errors or connection failures. Follow the steps below to identify and resolve these issues. 1\. Verify Connection Scheme ---------------------------- ### Cloud MongoDB (MongoDB Atlas) Use the connection scheme `mongodb+srv` in the **Advanced Config > Connection Scheme** field. Do **not** include the port in the host field. **Example:** 2\. Enable SSL for Encrypted Connections ---------------------------------------- If your connection string includes `ssl=true`, you must explicitly set SSL in the Connection Options. This is mandatory when connecting to clusters that enforce SSL/TLS encryption. 3\. Inspect Debug Logs ---------------------- Enable and review debug logs for more detailed error messages: * Navigate to the ingestion workflow in the OpenMetadata UI * Enable **Debug Log** in the configuration settings * Check logs using the Docker CLI: 4. Check Docker Networking Ensure that the OpenMetadata container is on the same network as the ingestion container, especially when running locally via Docker Compose. --- # SAP ERP Troubleshooting Guide | OpenMetadata Support We use cookies to improve site navigation, analyze site usage, and enhance your user experience. Click "Accept" to enable cookies or "Reject" to reject cookies. AcceptReject connectors No menu items for this category [Home](https://docs.open-metadata.org/latest) /[Connectors](https://docs.open-metadata.org/latest/connectors) /[Database](https://docs.open-metadata.org/latest/connectors/database) /[Sap Erp](https://docs.open-metadata.org/latest/connectors/database/sap-erp) /[Troubleshooting](https://docs.open-metadata.org/latest/connectors/database/sap-erp/troubleshooting) OpenMetadata Documentation Troubleshooting =============== Workflow Deployment Error ------------------------- If there were any errors during the workflow deployment process, the Ingestion Pipeline Entity will still be created, but no workflow will be present in the Ingestion container. * You can then Edit the Ingestion Pipeline and **Deploy** it again. * From the Connection tab, you can also Edit the Service if needed. Connector Debug Troubleshooting ------------------------------- This section provides instructions to help resolve common issues encountered during connector setup and metadata ingestion in OpenMetadata. Below are some of the most frequently observed troubleshooting scenarios. How to Enable Debug Logging for Any Ingestion --------------------------------------------- To enable debug logging for any ingestion workflow in OpenMetadata: 1. **Navigate to Services** Go to **Settings > Services > Service Type** (e.g., Database) in the OpenMetadata UI. 2. **Select a Service** Choose the specific service for which you want to enable debug logging. ![Select a Service](https://docs.open-metadata.org/images/v1.11/connectors/debug/debug1.png) Select a Service 3. **Access Ingestion Tab** Go to the **Ingestion tab** and click the three-dot menu on the right-hand side of the ingestion type, and select Edit. ![Access Agents Tab](https://docs.open-metadata.org/images/v1.11/connectors/debug/debug2.png) Access Agents Tab 4. **Enable Debug Logging** In the configuration dialog, enable the **Debug Log** option and click **Next**. ![Enable Debug Logging](https://docs.open-metadata.org/images/v1.11/connectors/debug/debug3.png) Enable Debug Logging 5. **Schedule and Submit** Configure the schedule if needed and click **Submit** to apply the changes. ![Schedule and Submit](https://docs.open-metadata.org/images/v1.11/connectors/debug/debug4.png) Schedule and Submit Permission Issues ----------------- If you encounter permission-related errors during connector setup or metadata ingestion, ensure that all the prerequisites and access configurations specified for each connector are properly implemented. Refer to the connector-specific documentation to verify the required permissions. --- # Redash Troubleshooting Guide | OpenMetadata Support We use cookies to improve site navigation, analyze site usage, and enhance your user experience. Click "Accept" to enable cookies or "Reject" to reject cookies. AcceptReject connectors No menu items for this category [Home](https://docs.open-metadata.org/latest) /[Connectors](https://docs.open-metadata.org/latest/connectors) /[Dashboard](https://docs.open-metadata.org/latest/connectors/dashboard) /[Redash](https://docs.open-metadata.org/latest/connectors/dashboard/redash) /[Troubleshooting](https://docs.open-metadata.org/latest/connectors/dashboard/redash/troubleshooting) OpenMetadata Documentation Troubleshooting =============== Workflow Deployment Error ------------------------- If there were any errors during the workflow deployment process, the Ingestion Pipeline Entity will still be created, but no workflow will be present in the Ingestion container. * You can then Edit the Ingestion Pipeline and **Deploy** it again. * From the Connection tab, you can also Edit the Service if needed. Connector Debug Troubleshooting ------------------------------- This section provides instructions to help resolve common issues encountered during connector setup and metadata ingestion in OpenMetadata. Below are some of the most frequently observed troubleshooting scenarios. How to Enable Debug Logging for Any Ingestion --------------------------------------------- To enable debug logging for any ingestion workflow in OpenMetadata: 1. **Navigate to Services** Go to **Settings > Services > Service Type** (e.g., Database) in the OpenMetadata UI. 2. **Select a Service** Choose the specific service for which you want to enable debug logging. ![Select a Service](https://docs.open-metadata.org/images/v1.11/connectors/debug/debug1.png) Select a Service 3. **Access Ingestion Tab** Go to the **Ingestion tab** and click the three-dot menu on the right-hand side of the ingestion type, and select Edit. ![Access Agents Tab](https://docs.open-metadata.org/images/v1.11/connectors/debug/debug2.png) Access Agents Tab 4. **Enable Debug Logging** In the configuration dialog, enable the **Debug Log** option and click **Next**. ![Enable Debug Logging](https://docs.open-metadata.org/images/v1.11/connectors/debug/debug3.png) Enable Debug Logging 5. **Schedule and Submit** Configure the schedule if needed and click **Submit** to apply the changes. ![Schedule and Submit](https://docs.open-metadata.org/images/v1.11/connectors/debug/debug4.png) Schedule and Submit Permission Issues ----------------- If you encounter permission-related errors during connector setup or metadata ingestion, ensure that all the prerequisites and access configurations specified for each connector are properly implemented. Refer to the connector-specific documentation to verify the required permissions. --- # DeltaLake Troubleshooting Guide | OpenMetadata Support We use cookies to improve site navigation, analyze site usage, and enhance your user experience. Click "Accept" to enable cookies or "Reject" to reject cookies. AcceptReject connectors No menu items for this category [Home](https://docs.open-metadata.org/latest) /[Connectors](https://docs.open-metadata.org/latest/connectors) /[Database](https://docs.open-metadata.org/latest/connectors/database) /[Deltalake](https://docs.open-metadata.org/latest/connectors/database/deltalake) /[Troubleshooting](https://docs.open-metadata.org/latest/connectors/database/deltalake/troubleshooting) OpenMetadata Documentation Troubleshooting =============== Workflow Deployment Error ------------------------- If there were any errors during the workflow deployment process, the Ingestion Pipeline Entity will still be created, but no workflow will be present in the Ingestion container. * You can then Edit the Ingestion Pipeline and **Deploy** it again. * From the Connection tab, you can also Edit the Service if needed. Connector Debug Troubleshooting ------------------------------- This section provides instructions to help resolve common issues encountered during connector setup and metadata ingestion in OpenMetadata. Below are some of the most frequently observed troubleshooting scenarios. How to Enable Debug Logging for Any Ingestion --------------------------------------------- To enable debug logging for any ingestion workflow in OpenMetadata: 1. **Navigate to Services** Go to **Settings > Services > Service Type** (e.g., Database) in the OpenMetadata UI. 2. **Select a Service** Choose the specific service for which you want to enable debug logging. ![Select a Service](https://docs.open-metadata.org/images/v1.11/connectors/debug/debug1.png) Select a Service 3. **Access Ingestion Tab** Go to the **Ingestion tab** and click the three-dot menu on the right-hand side of the ingestion type, and select Edit. ![Access Agents Tab](https://docs.open-metadata.org/images/v1.11/connectors/debug/debug2.png) Access Agents Tab 4. **Enable Debug Logging** In the configuration dialog, enable the **Debug Log** option and click **Next**. ![Enable Debug Logging](https://docs.open-metadata.org/images/v1.11/connectors/debug/debug3.png) Enable Debug Logging 5. **Schedule and Submit** Configure the schedule if needed and click **Submit** to apply the changes. ![Schedule and Submit](https://docs.open-metadata.org/images/v1.11/connectors/debug/debug4.png) Schedule and Submit Permission Issues ----------------- If you encounter permission-related errors during connector setup or metadata ingestion, ensure that all the prerequisites and access configurations specified for each connector are properly implemented. Refer to the connector-specific documentation to verify the required permissions. --- # MicroStrategy Troubleshooting Guide | OpenMetadata Support We use cookies to improve site navigation, analyze site usage, and enhance your user experience. Click "Accept" to enable cookies or "Reject" to reject cookies. AcceptReject connectors No menu items for this category [Home](https://docs.open-metadata.org/latest) /[Connectors](https://docs.open-metadata.org/latest/connectors) /[Dashboard](https://docs.open-metadata.org/latest/connectors/dashboard) /[Microstrategy](https://docs.open-metadata.org/latest/connectors/dashboard/microstrategy) /[Troubleshooting](https://docs.open-metadata.org/latest/connectors/dashboard/microstrategy/troubleshooting) OpenMetadata Documentation Troubleshooting =============== Workflow Deployment Error ------------------------- If there were any errors during the workflow deployment process, the Ingestion Pipeline Entity will still be created, but no workflow will be present in the Ingestion container. * You can then Edit the Ingestion Pipeline and **Deploy** it again. * From the Connection tab, you can also Edit the Service if needed. Connector Debug Troubleshooting ------------------------------- This section provides instructions to help resolve common issues encountered during connector setup and metadata ingestion in OpenMetadata. Below are some of the most frequently observed troubleshooting scenarios. How to Enable Debug Logging for Any Ingestion --------------------------------------------- To enable debug logging for any ingestion workflow in OpenMetadata: 1. **Navigate to Services** Go to **Settings > Services > Service Type** (e.g., Database) in the OpenMetadata UI. 2. **Select a Service** Choose the specific service for which you want to enable debug logging. ![Select a Service](https://docs.open-metadata.org/images/v1.11/connectors/debug/debug1.png) Select a Service 3. **Access Ingestion Tab** Go to the **Ingestion tab** and click the three-dot menu on the right-hand side of the ingestion type, and select Edit. ![Access Agents Tab](https://docs.open-metadata.org/images/v1.11/connectors/debug/debug2.png) Access Agents Tab 4. **Enable Debug Logging** In the configuration dialog, enable the **Debug Log** option and click **Next**. ![Enable Debug Logging](https://docs.open-metadata.org/images/v1.11/connectors/debug/debug3.png) Enable Debug Logging 5. **Schedule and Submit** Configure the schedule if needed and click **Submit** to apply the changes. ![Schedule and Submit](https://docs.open-metadata.org/images/v1.11/connectors/debug/debug4.png) Schedule and Submit Permission Issues ----------------- If you encounter permission-related errors during connector setup or metadata ingestion, ensure that all the prerequisites and access configurations specified for each connector are properly implemented. Refer to the connector-specific documentation to verify the required permissions. --- # Databricks Pipeline Troubleshooting Guide | OpenMetadata Support We use cookies to improve site navigation, analyze site usage, and enhance your user experience. Click "Accept" to enable cookies or "Reject" to reject cookies. AcceptReject connectors No menu items for this category [Home](https://docs.open-metadata.org/latest) /[Connectors](https://docs.open-metadata.org/latest/connectors) /[Pipeline](https://docs.open-metadata.org/latest/connectors/pipeline) /[Databricks Pipeline](https://docs.open-metadata.org/latest/connectors/pipeline/databricks-pipeline) /[Troubleshooting](https://docs.open-metadata.org/latest/connectors/pipeline/databricks-pipeline/troubleshooting) OpenMetadata Documentation Troubleshooting =============== Workflow Deployment Error ------------------------- If there were any errors during the workflow deployment process, the Ingestion Pipeline Entity will still be created, but no workflow will be present in the Ingestion container. * You can then Edit the Ingestion Pipeline and **Deploy** it again. * From the Connection tab, you can also Edit the Service if needed. Connector Debug Troubleshooting ------------------------------- This section provides instructions to help resolve common issues encountered during connector setup and metadata ingestion in OpenMetadata. Below are some of the most frequently observed troubleshooting scenarios. How to Enable Debug Logging for Any Ingestion --------------------------------------------- To enable debug logging for any ingestion workflow in OpenMetadata: 1. **Navigate to Services** Go to **Settings > Services > Service Type** (e.g., Database) in the OpenMetadata UI. 2. **Select a Service** Choose the specific service for which you want to enable debug logging. ![Select a Service](https://docs.open-metadata.org/images/v1.11/connectors/debug/debug1.png) Select a Service 3. **Access Ingestion Tab** Go to the **Ingestion tab** and click the three-dot menu on the right-hand side of the ingestion type, and select Edit. ![Access Agents Tab](https://docs.open-metadata.org/images/v1.11/connectors/debug/debug2.png) Access Agents Tab 4. **Enable Debug Logging** In the configuration dialog, enable the **Debug Log** option and click **Next**. ![Enable Debug Logging](https://docs.open-metadata.org/images/v1.11/connectors/debug/debug3.png) Enable Debug Logging 5. **Schedule and Submit** Configure the schedule if needed and click **Submit** to apply the changes. ![Schedule and Submit](https://docs.open-metadata.org/images/v1.11/connectors/debug/debug4.png) Schedule and Submit Permission Issues ----------------- If you encounter permission-related errors during connector setup or metadata ingestion, ensure that all the prerequisites and access configurations specified for each connector are properly implemented. Refer to the connector-specific documentation to verify the required permissions. --- # Oracle Troubleshooting Guide | OpenMetadata Support We use cookies to improve site navigation, analyze site usage, and enhance your user experience. Click "Accept" to enable cookies or "Reject" to reject cookies. AcceptReject connectors No menu items for this category [Home](https://docs.open-metadata.org/latest) /[Connectors](https://docs.open-metadata.org/latest/connectors) /[Database](https://docs.open-metadata.org/latest/connectors/database) /[Oracle](https://docs.open-metadata.org/latest/connectors/database/oracle) /[Troubleshooting](https://docs.open-metadata.org/latest/connectors/database/oracle/troubleshooting) OpenMetadata Documentation Troubleshooting =============== Workflow Deployment Error ------------------------- If there were any errors during the workflow deployment process, the Ingestion Pipeline Entity will still be created, but no workflow will be present in the Ingestion container. * You can then Edit the Ingestion Pipeline and **Deploy** it again. * From the Connection tab, you can also Edit the Service if needed. Connector Debug Troubleshooting ------------------------------- This section provides instructions to help resolve common issues encountered during connector setup and metadata ingestion in OpenMetadata. Below are some of the most frequently observed troubleshooting scenarios. How to Enable Debug Logging for Any Ingestion --------------------------------------------- To enable debug logging for any ingestion workflow in OpenMetadata: 1. **Navigate to Services** Go to **Settings > Services > Service Type** (e.g., Database) in the OpenMetadata UI. 2. **Select a Service** Choose the specific service for which you want to enable debug logging. ![Select a Service](https://docs.open-metadata.org/images/v1.11/connectors/debug/debug1.png) Select a Service 3. **Access Ingestion Tab** Go to the **Ingestion tab** and click the three-dot menu on the right-hand side of the ingestion type, and select Edit. ![Access Agents Tab](https://docs.open-metadata.org/images/v1.11/connectors/debug/debug2.png) Access Agents Tab 4. **Enable Debug Logging** In the configuration dialog, enable the **Debug Log** option and click **Next**. ![Enable Debug Logging](https://docs.open-metadata.org/images/v1.11/connectors/debug/debug3.png) Enable Debug Logging 5. **Schedule and Submit** Configure the schedule if needed and click **Submit** to apply the changes. ![Schedule and Submit](https://docs.open-metadata.org/images/v1.11/connectors/debug/debug4.png) Schedule and Submit Permission Issues ----------------- If you encounter permission-related errors during connector setup or metadata ingestion, ensure that all the prerequisites and access configurations specified for each connector are properly implemented. Refer to the connector-specific documentation to verify the required permissions. --- # Run the Exasol Connector Externally We use cookies to improve site navigation, analyze site usage, and enhance your user experience. Click "Accept" to enable cookies or "Reject" to reject cookies. AcceptReject connectors No menu items for this category [Home](https://docs.open-metadata.org/latest) /[Connectors](https://docs.open-metadata.org/latest/connectors) /[Database](https://docs.open-metadata.org/latest/connectors/database) /[Exasol](https://docs.open-metadata.org/latest/connectors/database/exasol) /[Yaml](https://docs.open-metadata.org/latest/connectors/database/exasol/yaml) OpenMetadata Documentation ![Exasol](https://docs.open-metadata.org/_next/image?url=%2Fimages%2Fconnectors%2Fexasol.webp&w=64&q=75) Exasol ====== PROD Available In Feature List Metadata Query Usage Lineage Column-level Lineage Data Profiler Data Quality Owners dbt Tags Stored Procedures Sample Data Auto-Classification In this section, we provide guides and references to use the Exasol connector. Configure and schedule Exasol metadata and profiler workflows from the OpenMetadata UI: * [Requirements](https://docs.open-metadata.org/latest/connectors/database/exasol/yaml#requirements) * [Metadata Ingestion](https://docs.open-metadata.org/latest/connectors/database/exasol/yaml#metadata-ingestion) How to Run the Connector Externally ----------------------------------- To run the Ingestion via the UI you'll need to use the OpenMetadata Ingestion Container, which comes shipped with custom Airflow plugins to handle the workflow deployment. If, instead, you want to manage your workflows externally on your preferred orchestrator, you can check the following docs to run the Ingestion Framework **anywhere**. [#### External Schedulers\ \ Get more information about running the Ingestion Framework Externally](https://docs.open-metadata.org/latest/deployment/ingestion) Requirements ------------ ### Python Requirements We have support for Python versions 3.9-3.11 To run the Exasol ingestion, you will need to install: Metadata Ingestion ------------------ ### 1\. Define the YAML Config This is a sample config for Exasol: #### Source Configuration - Service Connection **`username`** The username required to authenticate and connect to the Exasol database. The user must have sufficient privileges to access and read all the metadata available in Exasol. **`password`** The password associated with the user account used to connect to the Exasol database. Ensure this password corresponds to the specified username and is stored securely. Avoid sharing passwords in plain text and use secure methods for managing sensitive credentials. **`hostPort`** Provide the fully qualified hostname and port number of your Exasol deployment in the "Host and Port" field. **`SSL/TLS Settings`** Mode/setting for SSL validation: * **`validate-certificate`**: Uses Transport Layer Security (TLS) and validates the server certificate using system certificate stores. * **`ignore-certificate`**: Uses Transport Layer Security (TLS) but disables the validation of the server certificate. This should not be used in production. It can be useful during testing with self-signed certificates. * **`disable-tls`**: Does not use any Transport Layer Security (TLS). Data will be sent in plain text (no encryption). While this may be helpful in rare cases of debugging, make sure you do not use this in production. #### Advanced Configuration **Connection Options (Optional)**: Enter the details for any additional connection options that can be sent to database during the connection. These details must be added as Key-Value pairs. **Connection Arguments (Optional)**: Enter the details for any additional connection arguments such as security or protocol configs that can be sent to database during the connection. These details must be added as Key-Value pairs. filename.yamlCopy ### 2\. Run with the CLI First, we will need to save the YAML file. Afterward, and with all requirements installed, we can run: Note that from connector to connector, this recipe will always be the same. By updating the YAML configuration, you will be able to extract metadata from different sources. --- # Spline Connector Troubleshooting Guide | OpenMetadata Support We use cookies to improve site navigation, analyze site usage, and enhance your user experience. Click "Accept" to enable cookies or "Reject" to reject cookies. AcceptReject connectors No menu items for this category [Home](https://docs.open-metadata.org/latest) /[Connectors](https://docs.open-metadata.org/latest/connectors) /[Pipeline](https://docs.open-metadata.org/latest/connectors/pipeline) /[Spline](https://docs.open-metadata.org/latest/connectors/pipeline/spline) /[Troubleshooting](https://docs.open-metadata.org/latest/connectors/pipeline/spline/troubleshooting) OpenMetadata Documentation Troubleshooting =============== Workflow Deployment Error ------------------------- If there were any errors during the workflow deployment process, the Ingestion Pipeline Entity will still be created, but no workflow will be present in the Ingestion container. * You can then Edit the Ingestion Pipeline and **Deploy** it again. * From the Connection tab, you can also Edit the Service if needed. Connector Debug Troubleshooting ------------------------------- This section provides instructions to help resolve common issues encountered during connector setup and metadata ingestion in OpenMetadata. Below are some of the most frequently observed troubleshooting scenarios. How to Enable Debug Logging for Any Ingestion --------------------------------------------- To enable debug logging for any ingestion workflow in OpenMetadata: 1. **Navigate to Services** Go to **Settings > Services > Service Type** (e.g., Database) in the OpenMetadata UI. 2. **Select a Service** Choose the specific service for which you want to enable debug logging. ![Select a Service](https://docs.open-metadata.org/images/v1.11/connectors/debug/debug1.png) Select a Service 3. **Access Ingestion Tab** Go to the **Ingestion tab** and click the three-dot menu on the right-hand side of the ingestion type, and select Edit. ![Access Agents Tab](https://docs.open-metadata.org/images/v1.11/connectors/debug/debug2.png) Access Agents Tab 4. **Enable Debug Logging** In the configuration dialog, enable the **Debug Log** option and click **Next**. ![Enable Debug Logging](https://docs.open-metadata.org/images/v1.11/connectors/debug/debug3.png) Enable Debug Logging 5. **Schedule and Submit** Configure the schedule if needed and click **Submit** to apply the changes. ![Schedule and Submit](https://docs.open-metadata.org/images/v1.11/connectors/debug/debug4.png) Schedule and Submit Permission Issues ----------------- If you encounter permission-related errors during connector setup or metadata ingestion, ensure that all the prerequisites and access configurations specified for each connector are properly implemented. Refer to the connector-specific documentation to verify the required permissions. --- # Superset Troubleshooting Guide | OpenMetadata Support We use cookies to improve site navigation, analyze site usage, and enhance your user experience. Click "Accept" to enable cookies or "Reject" to reject cookies. AcceptReject connectors No menu items for this category [Home](https://docs.open-metadata.org/latest) /[Connectors](https://docs.open-metadata.org/latest/connectors) /[Dashboard](https://docs.open-metadata.org/latest/connectors/dashboard) /[Superset](https://docs.open-metadata.org/latest/connectors/dashboard/superset) /[Troubleshooting](https://docs.open-metadata.org/latest/connectors/dashboard/superset/troubleshooting) OpenMetadata Documentation Troubleshooting =============== Workflow Deployment Error ------------------------- If there were any errors during the workflow deployment process, the Ingestion Pipeline Entity will still be created, but no workflow will be present in the Ingestion container. * You can then Edit the Ingestion Pipeline and **Deploy** it again. * From the Connection tab, you can also Edit the Service if needed. Connector Debug Troubleshooting ------------------------------- This section provides instructions to help resolve common issues encountered during connector setup and metadata ingestion in OpenMetadata. Below are some of the most frequently observed troubleshooting scenarios. How to Enable Debug Logging for Any Ingestion --------------------------------------------- To enable debug logging for any ingestion workflow in OpenMetadata: 1. **Navigate to Services** Go to **Settings > Services > Service Type** (e.g., Database) in the OpenMetadata UI. 2. **Select a Service** Choose the specific service for which you want to enable debug logging. ![Select a Service](https://docs.open-metadata.org/images/v1.11/connectors/debug/debug1.png) Select a Service 3. **Access Ingestion Tab** Go to the **Ingestion tab** and click the three-dot menu on the right-hand side of the ingestion type, and select Edit. ![Access Agents Tab](https://docs.open-metadata.org/images/v1.11/connectors/debug/debug2.png) Access Agents Tab 4. **Enable Debug Logging** In the configuration dialog, enable the **Debug Log** option and click **Next**. ![Enable Debug Logging](https://docs.open-metadata.org/images/v1.11/connectors/debug/debug3.png) Enable Debug Logging 5. **Schedule and Submit** Configure the schedule if needed and click **Submit** to apply the changes. ![Schedule and Submit](https://docs.open-metadata.org/images/v1.11/connectors/debug/debug4.png) Schedule and Submit Permission Issues ----------------- If you encounter permission-related errors during connector setup or metadata ingestion, ensure that all the prerequisites and access configurations specified for each connector are properly implemented. Refer to the connector-specific documentation to verify the required permissions. --- # Profiler Workflow | OpenMetadata Profiling Workflow We use cookies to improve site navigation, analyze site usage, and enhance your user experience. Click "Accept" to enable cookies or "Reject" to reject cookies. AcceptReject ![](https://static.scarf.sh/a.png?x-pxid=6814879a-96c0-40bd-be69-ddfeb14694aa) how-to-guides No menu items for this category [Home](https://docs.open-metadata.org/latest) /[How To Guides](https://docs.open-metadata.org/latest/how-to-guides) /[Data Quality Observability](https://docs.open-metadata.org/latest/how-to-guides/data-quality-observability) /[Profiler](https://docs.open-metadata.org/latest/how-to-guides/data-quality-observability/profiler) /[Workflow](https://docs.open-metadata.org/latest/how-to-guides/data-quality-observability/profiler/workflow) OpenMetadata Documentation Profiler Workflow ================= Learn how to configure and run the Profiler Workflow to extract Profiler data and execute the Data Quality. For Datalake Profiling, we drop NaN (Not a Number) values from the DataFrame using the dropna() method to allow metric computation. However, we make an exception for null values, which are retained. This ensures that our computations are accurate while handling missing data UI configuration ---------------- After the metadata ingestion has been done correctly, we can configure and deploy the Profiler Workflow. This Pipeline will be in charge of feeding the Profiler tab of the Table Entity, as well as running any tests configured in the Entity. ![Table profile summary page](https://docs.open-metadata.org/images/v1.11/features/ingestion/workflows/profiler/profiler-summary-table.png) Table profile summary page ![Column profile summary page](https://docs.open-metadata.org/images/v1.11/features/ingestion/workflows/profiler/profiler-summary-column.png) Column profile summary page ### 1\. Add a Profiler Agent From the Service Page, go to the Agents tab to add a new ingestion and click on Add Profiler Agent. ![Add a profiler service](https://docs.open-metadata.org/images/v1.11/features/ingestion/workflows/profiler/add-profiler-workflow.png) Add a profiler service ### 2\. Configure the Profiler Agent Here you can enter the Profiler Agent details. ![Set profiler configuration](https://docs.open-metadata.org/images/v1.11/features/ingestion/workflows/profiler/configure-profiler-workflow.png) Set profiler configuration #### Profiler Options **Name** Define the name of the Profiler Workflow. While we only support a single workflow for the Metadata and Usage ingestion, users can define different schedules and filters for Profiler workflows. As profiling is a costly task, this enables a fine-grained approach to profiling and running tests by specifying different filters for each pipeline. **Database filter pattern (Optional)** regex expression to filter databases. **Schema filter pattern (Optional)** regex expression to filter schemas. **Table filter pattern (Optional)** regex expression to filter tables. **Profile Sample (Optional)** Set the sample to be use by the profiler for the specific table. * `Percentage`: Value must be between 0 and 100 exclusive (0 < percentage < 100). This will sample the table based on a percentage * `Row Count`: The table will be sampled based on a number of rows (i.e. `1,000`, `2,000`), etc. ⚠️ This option is currently not support for Druid. Sampling leverage `RANDOM` functions in most database (some have specific sampling functions) and Druid provides neither of these option. We recommend using the partitioning or sample query option if you need to limit the amount of data scanned. **Enable Debug Log** Set the Enable Debug Log toggle to set the logging level of the process to debug. You can check these logs in the Ingestion tab of the service and dig deeper into any errors you might find. **Include Views** If activated the profiler will compute metric for view entity types. Note that it can have a negative impact on the profiler performance. **Use FQN For Filtering Views** Set this flag when you want to apply the filters on Fully Qualified Names (e.g service\_name.db\_name.schema\_name.table\_name) instead of applying them to the raw name of the asset (e.g table\_name). This Flag is useful in scenarios when you have different schemas with same name in multiple databases, or tables with same name in different schemas, and you want to filter out only one of them. **Compute Metrics** Set the Compute Metrics toggle off to not perform any metric computation during the profiler ingestion workflow. Used in combination with Ingest Sample Data toggle on allows you to only ingest sample data. **Advanced Configuration** **Sample Data Rows Count** Set the number of rows to ingest when Ingest Sample Data toggle is on. Defaults to 50. **Thread Count (Optional)** Number of thread to use when computing metrics for the profiler. For Snowflake users we recommend setting it to 1. There is a known issue with one of the dependency (`snowflake-connector-python`) affecting projects with certain environments. **Timeout in Seconds (Optional)** This will set the duration a profiling job against a table should wait before interrupting its execution and moving on to profiling the next table. It is important to note that the profiler will wait for the hanging query to terminate before killing the execution. If there is a risk for your profiling job to hang, it is important to also set a query/connection timeout on your database engine. The default value for the profiler timeout is 12-hours. ### 3\. Schedule and Deploy After clicking Next, you will be redirected to the Scheduling form. This will be the same as the Metadata and Usage Ingestions. Select your desired schedule and click on Deploy to find the usage pipeline being added to the Service Ingestions. ### 4\. Updating Profiler setting at the table level Once you have created your profiler you can adjust some behavior at the table level by going to the table and clicking on the profiler tab ![table profile settings](https://docs.open-metadata.org/images/v1.11/features/ingestion/workflows/profiler/accessing-table-profile-settings.png) table profile settings ![table profile settings](https://docs.open-metadata.org/images/v1.11/features/ingestion/workflows/profiler/table-profile-summary-view.png) table profile settings #### Profiler Options **Profile Sample** Set the sample to be use by the profiler for the specific table. * `Percentage`: Value must be between 0 and 100 exclusive (0 < percentage < 100). This will sample the table based on a percentage * `Row Count`: The table will be sampled based on a number of rows (i.e. `1,000`, `2,000`), etc. ⚠️ This option is currently not support for Druid. Sampling leverage `RANDOM` functions in most database (some have specific sampling functions) and Druid provides neither of these option. We recommend using the partitioning or sample query option if you need to limit the amount of data scanned. **Enable Column Profile** This setting allows user to exclude or include specific columns and metrics from the profiler. _Note: for Google BigQuery tables partitioned on timestamp/datetime column type, month and year interval are not supported. You will need to set the `Interval Unit` to `DAY` or `HOUR`._ **Enable Partition** When enabled, the profiler will fetch the data based on your profiler settings. Note that if "profile sample" is set, this configuration will be used against the partitioned data and not the whole table. * `Column Name`: this is the name of the column that will be used as the partition field * `Interval Type`: * `TIME-UNIT`: a business logic timestamp/date/datetime (e.g. order date, sign up datetime, etc.) * `INGESTION-TIME`: a process logic timestamp/date/datetime (i.e. when was my data ingested in my table) * `COLUMN-VALUE`: a value representing a chunk of data (e.g. Product Type A, B, C, etc.) * `INTEGER-RANGE`: a range of integer that will be used as the partition (e.g. Customer ID between 1 and 10) Once you have picked the `Interval Type` you will need to define the configuration specific to your `Interval Type`. `INGESTION-TIME` or `INTEGER-RANGE` * `Interval`: the interval value (e.g. `1`, `2`, etc.) * `Interval Unit`: * `HOUR` * `DAY` * `MONTH` * `YEAR` `COLUMN-VALUE` * `Value`: a list of value to use for the partitioning logic `INTEGER-RANGE` * `Start Range`: the start of the range (inclusive) * `End Range`: the end of the range (inclusive) ### 5\. Updating Profiler setting at the platform level The behavior of the profiler can be configured at the platform level. Navigating to `Settings > Preferences > Profiler Configuration` you will find settings to adjust the behavior of the profiler. ![table profile global settings](https://docs.open-metadata.org/images/v1.11/features/ingestion/workflows/profiler/profiler-global-configuration.png) table profile global settings **Disabling All Metric Computation for a Data Type** Select the data type you want to disable all metric for. Then toggle disable on. When running the profiler all metric computation will be skipped for the data type. ![table profile global settings](https://docs.open-metadata.org/images/v1.11/features/ingestion/workflows/profiler/disable-metric-computation.png) table profile global settings **Disabling Specific Metric Computation for a Data Type** Select the data type you want to disable a metric for. Then in the `Metric Type` section select the metric you to compute (or unselect the ones you don't want to compute). When running the profiler the unselected metric will not be computed. ![table profile global settings](https://docs.open-metadata.org/images/v1.11/features/ingestion/workflows/profiler/disable-specific-metric-computation.png) table profile global settings YAML Configuration ------------------ In the [connectors](https://docs.open-metadata.org/latest/connectors) section we showcase how to run the metadata ingestion from a JSON file using the Airflow SDK or the CLI via metadata ingest. Running a profiler workflow is also possible using a JSON configuration file. This is a good option if you which to execute your workflow via the Airflow SDK or using the CLI; if you use the CLI a profile workflow can be triggered with the command `metadata profile -c FILENAME.yaml`. The `serviceConnection` config will be specific to your connector (you can find more information in the [connectors](https://docs.open-metadata.org/latest/connectors) section), though the sourceConfig for the profiler will be similar across all connectors. This is a sample config for the profiler: **computeMetrics**: Option to turn on/off computing profiler metrics. This flag is useful when you want to only ingest the sample data with the profiler workflow and not any other information. **profileSample**: Percentage of data or no. of rows we want to execute the profiler and tests on. **threadCount**: Number of threads to use during metric computations. **timeoutSeconds**: Profiler Timeout in Seconds **databaseFilterPattern**: Regex to only fetch databases that matches the pattern. **schemaFilterPattern**: Regex to only fetch tables or databases that matches the pattern. **tableFilterPattern**: Regex to only fetch tables or databases that matches the pattern. #### Processor Configuration Choose the `orm-profiler`. Its config can also be updated to define tests from the YAML itself instead of the UI: **tableConfig**: `tableConfig` allows you to set up some configuration at the table level. #### Sink Configuration To send the metadata to OpenMetadata, it needs to be specified as `type: metadata-rest`. #### Workflow Configuration The main property here is the `openMetadataServerConfig`, where you can define the host and security provider of your OpenMetadata installation. For a simple, local installation using our docker containers, this looks like: filename.yamlCopy ### 2\. Run with the CLI After saving the YAML config, we will run the command the same way we did for the metadata ingestion: Note now instead of running `ingest`, we are using the `profile` command to select the Profiler workflow. Profiler Best Practices ----------------------- When setting a profiler workflow it is important to keep in mind that queries will be running against your database. Depending on your database engine, you may incur costs (e.g., Google BigQuery, Snowflake). Execution time will also vary depending on your database engine computing power, the size of the table, and the number of columns. Given these elements, there are a few best practices we recommend you follow. ### 1\. Profile what you Need Profiling all the tables in your data platform might not be the most optimized approach. Profiled tables give an indication of the structure of the table, which is most useful for tables where this information is valuable (e.g., tables used by analysts or data scientists, etc.). When setting up a profiler workflow, you have the possibility to filter out/in certain databases, schemas, or tables. Using this feature will greatly help you narrow down which table you want to profile. ### 2\. Sampling and Partitioning your Tables On a table asset, you have the possibility to add a sample percentage/rows and a partitioning logic. Doing so will significantly reduce the amount of data scanned and the computing power required to perform the different operations. For sampling, you can set a sampling percentage at the workflow level. ### 3\. Excluding/Including Specific Columns/Metrics By default, the profiler will compute all the metrics against all the columns. This behavior can be fine-tuned only to include or exclude specific columns and specific metrics. For example, excluding `id` columns will reduce the number of columns against which the metrics are computed. ### 4\. Set Up Multiple Workflow If you have a large number of tables you would like to profile, setting up multiple workflows will help distribute the load. It is important though to monitor your instance CPU, and memory as having a large amount of workflow running simultaneously will require an adapted amount of resources. --- # Run the Dagster Connector Externally We use cookies to improve site navigation, analyze site usage, and enhance your user experience. Click "Accept" to enable cookies or "Reject" to reject cookies. AcceptReject connectors No menu items for this category [Home](https://docs.open-metadata.org/latest) /[Connectors](https://docs.open-metadata.org/latest/connectors) /[Pipeline](https://docs.open-metadata.org/latest/connectors/pipeline) /[Dagster](https://docs.open-metadata.org/latest/connectors/pipeline/dagster) /[Yaml](https://docs.open-metadata.org/latest/connectors/pipeline/dagster/yaml) OpenMetadata Documentation ![Dagster](https://docs.open-metadata.org/_next/image?url=%2Fimages%2Fconnectors%2Fdagster.webp&w=64&q=75) Dagster ======= PROD Available In Feature List Pipelines Pipeline Status Tags Usage Lineage Owners In this section, we provide guides and references to use the Dagster connector. Configure and schedule Dagster metadata and profiler workflows from the OpenMetadata UI: * [Requirements](https://docs.open-metadata.org/latest/connectors/pipeline/dagster/yaml#requirements) * [Metadata Ingestion](https://docs.open-metadata.org/latest/connectors/pipeline/dagster/yaml#metadata-ingestion) How to Run the Connector Externally ----------------------------------- To run the Ingestion via the UI you'll need to use the OpenMetadata Ingestion Container, which comes shipped with custom Airflow plugins to handle the workflow deployment. If, instead, you want to manage your workflows externally on your preferred orchestrator, you can check the following docs to run the Ingestion Framework **anywhere**. [#### External Schedulers\ \ Get more information about running the Ingestion Framework Externally](https://docs.open-metadata.org/latest/deployment/ingestion) Requirements ------------ ### Python Requirements We have support for Python versions 3.9-3.11 To run the Dagster ingestion, you will need to install: Metadata Ingestion ------------------ All connectors are defined as JSON Schemas. [Here](https://github.com/open-metadata/OpenMetadata/blob/main/openmetadata-spec/src/main/resources/json/schema/entity/services/connections/pipeline/dagsterConnection.json) you can find the structure to create a connection to Dagster. In order to create and run a Metadata Ingestion workflow, we will follow the steps to create a YAML configuration able to connect to the source, process the Entities if needed, and reach the OpenMetadata server. The workflow is modeled around the following [JSON Schema](https://github.com/open-metadata/OpenMetadata/blob/main/openmetadata-spec/src/main/resources/json/schema/metadataIngestion/workflow.json) ### 1\. Define the YAML Config This is a sample config for Dagster: #### Source Configuration - Service Connection * **host**: host and port for dagster pipeline **Note**: If dagster is deployed on `localhost` and entering `https://localhost:3000` into hostPort gives a connection refused error, please enter `https://127.0.0.1:3000` into the hostPort and try again. **Token** : Need pass token if connecting to `dagster cloud` instance **timeout** : Connection Time Limit Between OM and Dagster Graphql API in second #### Source Configuration - Lineage **lineageInformation**: Configure lineage extraction settings. * **dbServiceNames**: List of database service names to search for tables when creating lineage. If not specified, OpenMetadata searches all database services. Specifying services improves performance and accuracy. For lineage to work, ensure: * Your Dagster assets use [Software-Defined Assets](https://docs.dagster.io/concepts/assets/software-defined-assets) * Asset dependencies are declared explicitly using `deps` * Tables exist in OpenMetadata (run database ingestion first) * Asset keys match table names (use `["database", "schema", "table"]` format) #### Source Configuration - Source Config The `sourceConfig` is defined [here](https://github.com/open-metadata/OpenMetadata/blob/main/openmetadata-spec/src/main/resources/json/schema/metadataIngestion/pipelineServiceMetadataPipeline.json) : * **dbServiceNames**: Database Service Name for the creation of lineage, if the source supports it. * **includeTags**: Set the 'Include Tags' toggle to control whether to include tags as part of metadata ingestion. * **includeUnDeployedPipelines**: Set the 'Include UnDeployed Pipelines' toggle to control whether to include un-deployed pipelines as part of metadata ingestion. By default it is set to `true` * **markDeletedPipelines**: Set the Mark Deleted Pipelines toggle to flag pipelines as soft-deleted if they are not present anymore in the source system. * **pipelineFilterPattern** and **chartFilterPattern**: Note that the `pipelineFilterPattern` and `chartFilterPattern` both support regex as include or exclude. * **includeOwners**: Set the 'Include Owners' toggle to control whether to include owners to the ingested entity if the owner email matches with a user stored in the OM server as part of metadata ingestion. If the ingested entity already exists and has an owner, the owner will not be overwritten.It supports boolean values either `true` or `false`. * **overrideLineage**: Set the 'Override Lineage' toggle to control whether to override the existing lineage. It supports boolean values either `true` or `false`. * **overrideMetadata**: Set the 'Override Metadata' toggle to control whether to override the existing metadata in the OpenMetadata server with the metadata fetched from the source. If the toggle is set to true, the metadata fetched from the source will override the existing metadata in the OpenMetadata server. If the toggle is set to false, the metadata fetched from the source will not override the existing metadata in the OpenMetadata server. This is applicable for fields like description, tags, owner and displayName. It supports boolean values either `true` or `false`. #### Sink Configuration To send the metadata to OpenMetadata, it needs to be specified as `type: metadata-rest`. #### Workflow Configuration The main property here is the `openMetadataServerConfig`, where you can define the host and security provider of your OpenMetadata installation. **Logger Level** You can specify the `loggerLevel` depending on your needs. If you are trying to troubleshoot an ingestion, running with `DEBUG` will give you far more traces for identifying issues. **JWT Token** JWT tokens will allow your clients to authenticate against the OpenMetadata server. To enable JWT Tokens, you will get more details [here](https://docs.open-metadata.org/latest/deployment/security/enable-jwt-tokens) . You can refer to the JWT Troubleshooting section [link](https://docs.open-metadata.org/latest/deployment/security/jwt-troubleshooting) for any issues in your JWT configuration. **Store Service Connection** If set to `true` (default), we will store the sensitive information either encrypted via the Fernet Key in the database or externally, if you have configured any [Secrets Manager](https://docs.open-metadata.org/latest/deployment/secrets-manager) . If set to `false`, the service will be created, but the service connection information will only be used by the Ingestion Framework at runtime, and won't be sent to the OpenMetadata server. **Store Service Connection** If set to `true` (default), we will store the sensitive information either encrypted via the Fernet Key in the database or externally, if you have configured any [Secrets Manager](https://docs.open-metadata.org/latest/deployment/secrets-manager) . If set to `false`, the service will be created, but the service connection information will only be used by the Ingestion Framework at runtime, and won't be sent to the OpenMetadata server. **SSL Configuration** If you have added SSL to the [OpenMetadata server](https://docs.open-metadata.org/latest/deployment/security/enable-ssl) , then you will need to handle the certificates when running the ingestion too. You can either set `verifySSL` to `ignore`, or have it as `validate`, which will require you to set the `sslConfig.caCertificate` with a local path where your ingestion runs that points to the server certificate file. Find more information on how to troubleshoot SSL issues [here](https://docs.open-metadata.org/latest/deployment/security/enable-ssl/ssl-troubleshooting) . **ingestionPipelineFQN** Fully qualified name of ingestion pipeline, used to identify the current ingestion pipeline. filename.yamlCopy ### 2\. Run with the CLI First, we will need to save the YAML file. Afterward, and with all requirements installed, we can run: Note that from connector to connector, this recipe will always be the same. By updating the YAML configuration, you will be able to extract metadata from different sources. --- # Run the Redash Connector Externally We use cookies to improve site navigation, analyze site usage, and enhance your user experience. Click "Accept" to enable cookies or "Reject" to reject cookies. AcceptReject connectors No menu items for this category [Home](https://docs.open-metadata.org/latest) /[Connectors](https://docs.open-metadata.org/latest/connectors) /[Dashboard](https://docs.open-metadata.org/latest/connectors/dashboard) /[Redash](https://docs.open-metadata.org/latest/connectors/dashboard/redash) /[Yaml](https://docs.open-metadata.org/latest/connectors/dashboard/redash/yaml) OpenMetadata Documentation ![Redash](https://docs.open-metadata.org/_next/image?url=%2Fimages%2Fconnectors%2Fredash.webp&w=64&q=75) Redash ====== PROD Available In Feature List Dashboards Charts Lineage Owners Tags Datamodels Projects In this section, we provide guides and references to use the Redash connector. Configure and schedule Redash metadata and profiler workflows from the OpenMetadata UI: * [Requirements](https://docs.open-metadata.org/latest/connectors/dashboard/redash/yaml#requirements) * [Metadata Ingestion](https://docs.open-metadata.org/latest/connectors/dashboard/redash/yaml#metadata-ingestion) How to Run the Connector Externally ----------------------------------- To run the Ingestion via the UI you'll need to use the OpenMetadata Ingestion Container, which comes shipped with custom Airflow plugins to handle the workflow deployment. If, instead, you want to manage your workflows externally on your preferred orchestrator, you can check the following docs to run the Ingestion Framework **anywhere**. [#### External Schedulers\ \ Get more information about running the Ingestion Framework Externally](https://docs.open-metadata.org/latest/deployment/ingestion) Requirements ------------ ### Python Requirements We have support for Python versions 3.9-3.11 To run the Redash ingestion, you will need to install: Metadata Ingestion ------------------ All connectors are defined as JSON Schemas. [Here](https://github.com/open-metadata/OpenMetadata/blob/main/openmetadata-spec/src/main/resources/json/schema/entity/services/connections/dashboard/redashConnection.json) you can find the structure to create a connection to Redash. In order to create and run a Metadata Ingestion workflow, we will follow the steps to create a YAML configuration able to connect to the source, process the Entities if needed, and reach the OpenMetadata server. The workflow is modeled around the following [JSON Schema](https://github.com/open-metadata/OpenMetadata/blob/main/openmetadata-spec/src/main/resources/json/schema/metadataIngestion/workflow.json) ### 1\. Define the YAML Config #### Source Configuration - Service Connection **hostPort**: URL to the Redash instance. **username**: Specify the User to connect to Redash. It should have enough privileges to read all the metadata. **apiKey**: API key of the redash instance to access. It has the same permissions as the user who owns it. Can be found on a user profile page. **Redash Version**: Redash version of your redash instance. Enter the numerical value from the [Redash Releases](https://github.com/getredash/redash/releases) page. Default: `10.0.0`. #### Source Configuration - Source Config The `sourceConfig` is defined [here](https://github.com/open-metadata/OpenMetadata/blob/main/openmetadata-spec/src/main/resources/json/schema/metadataIngestion/dashboardServiceMetadataPipeline.json) : * **dbServicePrefixes**: List of service path prefixes for lineage matching. Supported formats: DBServiceName, DBServiceName.DatabaseName, DBServiceName.DatabaseName.SchemaName, or DBServiceName.DatabaseName.SchemaName.TableName * **dashboardFilterPattern**, **chartFilterPattern**, **dataModelFilterPattern**: Note that all of them support regex as include or exclude. E.g., "My dashboard, My dash.\*, .\*Dashboard". * **projectFilterPattern**: Filter the dashboards, charts and data sources by projects. Note that all of them support regex as include or exclude. E.g., "My project, My proj.\*, .\*Project". * **includeOwners**: Set the 'Include Owners' toggle to control whether to include owners to the ingested entity if the owner email matches with a user stored in the OM server as part of metadata ingestion. If the ingested entity already exists and has an owner, the owner will not be overwritten. * **includeTags**: Set the 'Include Tags' toggle to control whether to include tags in metadata ingestion. * **includeDataModels**: Set the 'Include Data Models' toggle to control whether to include tags as part of metadata ingestion. * **markDeletedDashboards**: Set the 'Mark Deleted Dashboards' toggle to flag dashboards as soft-deleted if they are not present anymore in the source system. * **Include Draft Dashboard (toggle)**: Set the 'Include Draft Dashboard' toggle to include draft dashboards. By default it will include draft dashboards. * **dataModelFilterPattern**: Regex exclude or include data models that matches the pattern. * **includeOwners**:Enabling a flag will replace the current owner with a new owner from the source during metadata ingestion, if the current owner is null. It is recommended to keep the flag enabled to obtain the owner information during the first metadata ingestion.`includeOwners` supports boolean value either true or false. * **markDeletedDashboards**: Optional configuration to soft delete dashboards in OpenMetadata if the source dashboards are deleted. Also, if the dashboard is deleted, all the associated entities like lineage, etc., with that dashboard will be deleted.`markDeletedDashboards` supports boolean value either true or false. * **markDeletedDataModels**: Optional configuration to soft delete data models in OpenMetadata if the source data models are deleted. Also, if the data models is deleted, all the associated entities like lineage, etc., with that data models will be deleted.`includeOwners` supports boolean value either true or false. * **includeTags**:Optional configuration to toggle the tags ingestion.`markDeletedDataModels` supports boolean value either true or false. * **includeDataModels**: Optional configuration to toggle the ingestion of data models.`includeDataModels` supports boolean value either true or false. * **includeDraftDashboard**: Optional Configuration to include/exclude draft dashboards. By default it will include draft dashboards.`includeDraftDashboard` supports boolean value either true or false. * **overrideMetadata**: Set the 'Override Metadata' toggle to control whether to override the existing metadata in the OpenMetadata server with the metadata fetched from the source. If the toggle is set to true, the metadata fetched from the source will override the existing metadata in the OpenMetadata server. If the toggle is set to false, the metadata fetched from the source will not override the existing metadata in the OpenMetadata server. This is applicable for fields like description, tags, owner and displayName.`overrideMetadata` supports boolean value either true or false. * **overrideLineage**: Set the 'Override Lineage' toggle to control whether to override the existing lineage.`overrideLineage` supports boolean value either true or false. #### Sink Configuration To send the metadata to OpenMetadata, it needs to be specified as `type: metadata-rest`. #### Workflow Configuration The main property here is the `openMetadataServerConfig`, where you can define the host and security provider of your OpenMetadata installation. **Logger Level** You can specify the `loggerLevel` depending on your needs. If you are trying to troubleshoot an ingestion, running with `DEBUG` will give you far more traces for identifying issues. **JWT Token** JWT tokens will allow your clients to authenticate against the OpenMetadata server. To enable JWT Tokens, you will get more details [here](https://docs.open-metadata.org/latest/deployment/security/enable-jwt-tokens) . You can refer to the JWT Troubleshooting section [link](https://docs.open-metadata.org/latest/deployment/security/jwt-troubleshooting) for any issues in your JWT configuration. **Store Service Connection** If set to `true` (default), we will store the sensitive information either encrypted via the Fernet Key in the database or externally, if you have configured any [Secrets Manager](https://docs.open-metadata.org/latest/deployment/secrets-manager) . If set to `false`, the service will be created, but the service connection information will only be used by the Ingestion Framework at runtime, and won't be sent to the OpenMetadata server. **Store Service Connection** If set to `true` (default), we will store the sensitive information either encrypted via the Fernet Key in the database or externally, if you have configured any [Secrets Manager](https://docs.open-metadata.org/latest/deployment/secrets-manager) . If set to `false`, the service will be created, but the service connection information will only be used by the Ingestion Framework at runtime, and won't be sent to the OpenMetadata server. **SSL Configuration** If you have added SSL to the [OpenMetadata server](https://docs.open-metadata.org/latest/deployment/security/enable-ssl) , then you will need to handle the certificates when running the ingestion too. You can either set `verifySSL` to `ignore`, or have it as `validate`, which will require you to set the `sslConfig.caCertificate` with a local path where your ingestion runs that points to the server certificate file. Find more information on how to troubleshoot SSL issues [here](https://docs.open-metadata.org/latest/deployment/security/enable-ssl/ssl-troubleshooting) . **ingestionPipelineFQN** Fully qualified name of ingestion pipeline, used to identify the current ingestion pipeline. filename.yamlCopy ### 2\. Run with the CLI First, we will need to save the YAML file. Afterward, and with all requirements installed, we can run: Note that from connector to connector, this recipe will always be the same. By updating the YAML configuration, you will be able to extract metadata from different sources. --- # Run the Superset Connector Externally We use cookies to improve site navigation, analyze site usage, and enhance your user experience. Click "Accept" to enable cookies or "Reject" to reject cookies. AcceptReject connectors No menu items for this category [Home](https://docs.open-metadata.org/latest) /[Connectors](https://docs.open-metadata.org/latest/connectors) /[Dashboard](https://docs.open-metadata.org/latest/connectors/dashboard) /[Superset](https://docs.open-metadata.org/latest/connectors/dashboard/superset) /[Yaml](https://docs.open-metadata.org/latest/connectors/dashboard/superset/yaml) OpenMetadata Documentation ![Superset](https://docs.open-metadata.org/_next/image?url=%2Fimages%2Fconnectors%2Fsuperset.webp&w=64&q=75) Superset ======== PROD Available In Feature List Dashboards Charts Lineage Owners Datamodels Tags Projects In this section, we provide guides and references to use the Superset connector. Configure and schedule Superset metadata and profiler workflows from the OpenMetadata UI: * [Requirements](https://docs.open-metadata.org/latest/connectors/dashboard/superset/yaml#requirements) * [Metadata Ingestion](https://docs.open-metadata.org/latest/connectors/dashboard/superset/yaml#metadata-ingestion) * [Enable Security](https://docs.open-metadata.org/latest/connectors/dashboard/superset/yaml#securing-superset-connection-with-ssl-in-openmetadata) How to Run the Connector Externally ----------------------------------- To run the Ingestion via the UI you'll need to use the OpenMetadata Ingestion Container, which comes shipped with custom Airflow plugins to handle the workflow deployment. If, instead, you want to manage your workflows externally on your preferred orchestrator, you can check the following docs to run the Ingestion Framework **anywhere**. [#### External Schedulers\ \ Get more information about running the Ingestion Framework Externally](https://docs.open-metadata.org/latest/deployment/ingestion) Requirements ------------ The ingestion also works with Superset 2.0.0 πŸŽ‰ **Note:** **API Connection**: To extract metadata from Superset via API, user must have at least `can read on Chart` & `can read on Dashboard` permissions. **Database Connection**: To extract metadata from Superset via MySQL or Postgres database, database user must have at least `SELECT` privilege on `dashboards` & `slices` tables within superset schema. ### Python Requirements We have support for Python versions 3.9-3.11 To run the Superset ingestion, you will need to install: Metadata Ingestion ------------------ All connectors are defined as JSON Schemas. [Here](https://github.com/open-metadata/OpenMetadata/blob/main/openmetadata-spec/src/main/resources/json/schema/entity/services/connections/dashboard/supersetConnection.json) you can find the structure to create a connection to Superset. In order to create and run a Metadata Ingestion workflow, we will follow the steps to create a YAML configuration able to connect to the source, process the Entities if needed, and reach the OpenMetadata server. The workflow is modeled around the following [JSON Schema](https://github.com/open-metadata/OpenMetadata/blob/main/openmetadata-spec/src/main/resources/json/schema/metadataIngestion/workflow.json) ### 1\. Define the YAML Config This is a sample config for Superset: #### Source Configuration - Service Connection **hostPort**: The `Host and Post` parameter is common for all three modes of authentication which specifies the host and port of the Superset instance. This should be specified as a string in the format `http://hostname:port` or `https://hostname:port`. For example, you might set the hostPort parameter to `https://org.superset.com:8088`. **connection**: Add the connection details to fetch metadata from Superset either through APIs or Database. #### For Superset API Connection: Superset API connection is the default mode of authentication where we fetch the metadata using [Superset APIs](https://superset.apache.org/docs/api/) . **Note**: Superset only supports basic or ldap authentication through APIs so if you have SSO enabled on your Superset instance then this mode of authentication will not work for you and you can opt for MySQL or Postgres Connection to fetch metadata directly from the database in the backend of Superset. **username**: Username to connect to Superset, for ex. `user@organization.com`. This user should have access to relevant dashboards and charts in Superset to fetch the metadata. **password**: Password of the user account to connect with Superset. **provider**: Choose between `db`(default) or `ldap` mode of Authentication provider for the Superset service. This parameter is used internally to connect to Superset's REST API. #### For MySQL Connection: You can use Mysql Connection when you have SSO enabled and your Superset is backed by Mysql database. **username**: Specify the User to connect to MySQL. It should have enough privileges to read all the metadata. Make sure the user has select privileges on `dashboards`, `tables` & `slices` tables of superset schema. **password**: Password to connect to MySQL. **hostPort**: Enter the fully qualified hostname and port number for your MySQL deployment in the Host and Port field. * **databaseSchema**: Enter the database schema which is associated with the Superset instance.. * **caCertificate**: Provide the path to ssl ca file. * **sslCertificate**: Provide the path to ssl client certificate file (ssl\_cert). * **sslKey**: Provide the path to ssl client certificate file (ssl\_key). **Connection Options (Optional)**: Enter the details for any additional connection options that can be sent to MySQL during the connection. These details must be added as Key-Value pairs. **Connection Arguments (Optional)**: Enter the details for any additional connection arguments such as security or protocol configs that can be sent to MySQL during the connection. These details must be added as Key-Value pairs. * In case you are using Single-Sign-On (SSO) for authentication, add the `authenticator` details in the Connection Arguments as a Key-Value pair as follows: `"authenticator" : "sso_login_url"` #### For Postgres Connection: You can use Postgres Connection when you have SSO enabled and your Superset is backed by Postgres database. * **username**: Specify the User to connect to Postgres. Make sure the user has select privileges on `dashboards`, `tables` & `slices` tables of superset schema. **password**: Password to connect to Postgres. **hostPort**: Enter the fully qualified hostname and port number for your Postgres deployment in the Host and Port field. * **database**: Initial Postgres database to connect to. Specify the name of database associated with Superset instance. * **caCertificate**: Provide the path to ssl ca file. **Connection Options (Optional)**: Enter the details for any additional connection options that can be sent to Postgres during the connection. These details must be added as Key-Value pairs. **Connection Arguments (Optional)**: Enter the details for any additional connection arguments such as security or protocol configs that can be sent to Postgres during the connection. These details must be added as Key-Value pairs. * In case you are using Single-Sign-On (SSO) for authentication, add the `authenticator` details in the Connection Arguments as a Key-Value pair as follows: `"authenticator" : "sso_login_url"` #### Source Configuration - Source Config The `sourceConfig` is defined [here](https://github.com/open-metadata/OpenMetadata/blob/main/openmetadata-spec/src/main/resources/json/schema/metadataIngestion/dashboardServiceMetadataPipeline.json) : * **dbServicePrefixes**: List of service path prefixes for lineage matching. Supported formats: DBServiceName, DBServiceName.DatabaseName, DBServiceName.DatabaseName.SchemaName, or DBServiceName.DatabaseName.SchemaName.TableName * **dashboardFilterPattern**, **chartFilterPattern**, **dataModelFilterPattern**: Note that all of them support regex as include or exclude. E.g., "My dashboard, My dash.\*, .\*Dashboard". * **projectFilterPattern**: Filter the dashboards, charts and data sources by projects. Note that all of them support regex as include or exclude. E.g., "My project, My proj.\*, .\*Project". * **includeOwners**: Set the 'Include Owners' toggle to control whether to include owners to the ingested entity if the owner email matches with a user stored in the OM server as part of metadata ingestion. If the ingested entity already exists and has an owner, the owner will not be overwritten. * **includeTags**: Set the 'Include Tags' toggle to control whether to include tags in metadata ingestion. * **includeDataModels**: Set the 'Include Data Models' toggle to control whether to include tags as part of metadata ingestion. * **markDeletedDashboards**: Set the 'Mark Deleted Dashboards' toggle to flag dashboards as soft-deleted if they are not present anymore in the source system. * **Include Draft Dashboard (toggle)**: Set the 'Include Draft Dashboard' toggle to include draft dashboards. By default it will include draft dashboards. * **dataModelFilterPattern**: Regex exclude or include data models that matches the pattern. * **includeOwners**:Enabling a flag will replace the current owner with a new owner from the source during metadata ingestion, if the current owner is null. It is recommended to keep the flag enabled to obtain the owner information during the first metadata ingestion.`includeOwners` supports boolean value either true or false. * **markDeletedDashboards**: Optional configuration to soft delete dashboards in OpenMetadata if the source dashboards are deleted. Also, if the dashboard is deleted, all the associated entities like lineage, etc., with that dashboard will be deleted.`markDeletedDashboards` supports boolean value either true or false. * **markDeletedDataModels**: Optional configuration to soft delete data models in OpenMetadata if the source data models are deleted. Also, if the data models is deleted, all the associated entities like lineage, etc., with that data models will be deleted.`includeOwners` supports boolean value either true or false. * **includeTags**:Optional configuration to toggle the tags ingestion.`markDeletedDataModels` supports boolean value either true or false. * **includeDataModels**: Optional configuration to toggle the ingestion of data models.`includeDataModels` supports boolean value either true or false. * **includeDraftDashboard**: Optional Configuration to include/exclude draft dashboards. By default it will include draft dashboards.`includeDraftDashboard` supports boolean value either true or false. * **overrideMetadata**: Set the 'Override Metadata' toggle to control whether to override the existing metadata in the OpenMetadata server with the metadata fetched from the source. If the toggle is set to true, the metadata fetched from the source will override the existing metadata in the OpenMetadata server. If the toggle is set to false, the metadata fetched from the source will not override the existing metadata in the OpenMetadata server. This is applicable for fields like description, tags, owner and displayName.`overrideMetadata` supports boolean value either true or false. * **overrideLineage**: Set the 'Override Lineage' toggle to control whether to override the existing lineage.`overrideLineage` supports boolean value either true or false. #### Sink Configuration To send the metadata to OpenMetadata, it needs to be specified as `type: metadata-rest`. #### Workflow Configuration The main property here is the `openMetadataServerConfig`, where you can define the host and security provider of your OpenMetadata installation. **Logger Level** You can specify the `loggerLevel` depending on your needs. If you are trying to troubleshoot an ingestion, running with `DEBUG` will give you far more traces for identifying issues. **JWT Token** JWT tokens will allow your clients to authenticate against the OpenMetadata server. To enable JWT Tokens, you will get more details [here](https://docs.open-metadata.org/latest/deployment/security/enable-jwt-tokens) . You can refer to the JWT Troubleshooting section [link](https://docs.open-metadata.org/latest/deployment/security/jwt-troubleshooting) for any issues in your JWT configuration. **Store Service Connection** If set to `true` (default), we will store the sensitive information either encrypted via the Fernet Key in the database or externally, if you have configured any [Secrets Manager](https://docs.open-metadata.org/latest/deployment/secrets-manager) . If set to `false`, the service will be created, but the service connection information will only be used by the Ingestion Framework at runtime, and won't be sent to the OpenMetadata server. **Store Service Connection** If set to `true` (default), we will store the sensitive information either encrypted via the Fernet Key in the database or externally, if you have configured any [Secrets Manager](https://docs.open-metadata.org/latest/deployment/secrets-manager) . If set to `false`, the service will be created, but the service connection information will only be used by the Ingestion Framework at runtime, and won't be sent to the OpenMetadata server. **SSL Configuration** If you have added SSL to the [OpenMetadata server](https://docs.open-metadata.org/latest/deployment/security/enable-ssl) , then you will need to handle the certificates when running the ingestion too. You can either set `verifySSL` to `ignore`, or have it as `validate`, which will require you to set the `sslConfig.caCertificate` with a local path where your ingestion runs that points to the server certificate file. Find more information on how to troubleshoot SSL issues [here](https://docs.open-metadata.org/latest/deployment/security/enable-ssl/ssl-troubleshooting) . **ingestionPipelineFQN** Fully qualified name of ingestion pipeline, used to identify the current ingestion pipeline. filename.yamlCopy Securing Superset Connection with SSL in OpenMetadata ----------------------------------------------------- 1. To establish secure connections between OpenMetadata and Superset, in the `YAML` under `sslConfig`, we need to add `caCertificate` and update the certificate path. Ensure that the certificates are accessible from the Airflow Server. 2. To establish secure connections between OpenMetadata and Superset's MySQL database, you need to configure SSL certificates appropriately. If you only require SSL validation, specify the `caCertificate` to use the CA certificate for validating the server's certificate. For mutual authentication, where both client and server need to authenticate each other, you must provide all three parameters: `ssl_key` for the client’s private key, `ssl_cert` for the client’s SSL certificate, and `ssl_ca` for the CA certificate to validate the server’s certificate. 3. To establish secure connxxwections between OpenMetadata and Superset's PostgreSQL database, you can configure SSL using different SSL modes provided by PostgreSQL, each offering varying levels of security.Under `PostgresConnection Advanced Config`, specify the SSL mode appropriate for your connection, such as `prefer`, `verify-ca`, `allow`, and others. After selecting the SSL mode, provide the CA certificate used for SSL validation (`caCertificate`). Note that PostgreSQL requires only the CA certificate for SSL validation. ### 2\. Run with the CLI First, we will need to save the YAML file. Afterward, and with all requirements installed, we can run: Note that from connector to connector, this recipe will always be the same. By updating the YAML configuration, you will be able to extract metadata from different sources. --- # Run the Delta Lake Connector Externally We use cookies to improve site navigation, analyze site usage, and enhance your user experience. Click "Accept" to enable cookies or "Reject" to reject cookies. AcceptReject connectors No menu items for this category [Home](https://docs.open-metadata.org/latest) /[Connectors](https://docs.open-metadata.org/latest/connectors) /[Database](https://docs.open-metadata.org/latest/connectors/database) /[Deltalake](https://docs.open-metadata.org/latest/connectors/database/deltalake) /[Yaml](https://docs.open-metadata.org/latest/connectors/database/deltalake/yaml) OpenMetadata Documentation ![Delta Lake](https://docs.open-metadata.org/_next/image?url=%2Fimages%2Fconnectors%2Fdelta-lake.webp&w=64&q=75) Delta Lake ========== PROD Available In Feature List Metadata dbt Query Usage Data Profiler Data Quality Lineage Column-level Lineage Owners Tags Stored Procedures Sample Data Auto-Classification In this section, we provide guides and references to use the Delta Lake connector. Configure and schedule Delta Lake metadata and profiler workflows from the OpenMetadata UI: * [Requirements](https://docs.open-metadata.org/latest/connectors/database/deltalake/yaml#requirements) * [Metadata Ingestion](https://docs.open-metadata.org/latest/connectors/database/deltalake/yaml#metadata-ingestion) * [dbt Integration](https://docs.open-metadata.org/latest/connectors/database/deltalake/yaml#dbt-integration) How to Run the Connector Externally ----------------------------------- To run the Ingestion via the UI you'll need to use the OpenMetadata Ingestion Container, which comes shipped with custom Airflow plugins to handle the workflow deployment. If, instead, you want to manage your workflows externally on your preferred orchestrator, you can check the following docs to run the Ingestion Framework **anywhere**. [#### External Schedulers\ \ Get more information about running the Ingestion Framework Externally](https://docs.open-metadata.org/latest/deployment/ingestion) Requirements ------------ Delta Lake requires to run with Python 3.9 or 3.10. We do not yet support the Delta connector for Python 3.11 ### Python Requirements We have support for Python versions 3.9-3.11 To run the Delta Lake ingestion, you will need to install: * If extracting from a metastore * If extracting directly from the storage Metadata Ingestion ------------------ All connectors are defined as JSON Schemas. [Here](https://github.com/open-metadata/OpenMetadata/blob/main/openmetadata-spec/src/main/resources/json/schema/entity/services/connections/database/deltaLakeConnection.json) you can find the structure to create a connection to Delta Lake. In order to create and run a Metadata Ingestion workflow, we will follow the steps to create a YAML configuration able to connect to the source, process the Entities if needed, and reach the OpenMetadata server. The workflow is modeled around the following [JSON Schema](https://github.com/open-metadata/OpenMetadata/blob/main/openmetadata-spec/src/main/resources/json/schema/metadataIngestion/workflow.json) ### 1\. Define the YAML Config #### Source Configuration - From Metastore ##### Source Configuration - Service Connection **Metastore Host Port**: Enter the Host & Port of Hive Metastore Service to configure the Spark Session. Either of `metastoreHostPort`, `metastoreDb` or `metastoreFilePath` is required. **Metastore File Path**: Enter the file path to local Metastore in case Spark cluster is running locally. Either of `metastoreHostPort`, `metastoreDb` or `metastoreFilePath` is required. **Metastore DB**: The JDBC connection to the underlying Hive metastore DB. Either of `metastoreHostPort`, `metastoreDb` or `metastoreFilePath` is required. **appName (Optional)**: Enter the app name of spark session. **Connection Arguments (Optional)**: Key-Value pairs that will be used to pass extra `config` elements to the Spark Session builder. We are internally running with `pyspark` 3.X and `delta-lake` 2.0.0. This means that we need to consider Spark configuration options for 3.X. ###### Metastore Host Port When connecting to an External Metastore passing the parameter `Metastore Host Port`, we will be preparing a Spark Session with the configuration Then, we will be using the `catalog` functions from the Spark Session to pick up the metadata exposed by the Hive Metastore. ###### Metastore File Path If instead we use a local file path that contains the metastore information (e.g., for local testing with the default `metastore_db` directory), we will set To update the `Derby` information. More information about this in a great [SO thread](https://stackoverflow.com/questions/38377188/how-to-get-rid-of-derby-log-metastore-db-from-spark-shell) . * You can find all supported configurations [here](https://spark.apache.org/docs/latest/configuration.html) * If you need further information regarding the Hive metastore, you can find it [here](https://spark.apache.org/docs/latest/sql-data-sources-hive-tables.html) , and in The Internals of Spark SQL [book](https://jaceklaskowski.gitbooks.io/mastering-spark-sql/content/spark-sql-hive-metastore.html) . ###### Metastore Database You can also connect to the metastore by directly pointing to the Hive Metastore db, e.g., `jdbc:mysql://localhost:3306/demo_hive`. Here, we will need to inform all the common database settings (url, username, password), and the driver class name for JDBC metastore. You will need to provide the driver to the ingestion image, and pass the `classpath` which will be used in the Spark Configuration under `spark.driver.extraClassPath`. #### Source Configuration - Source Config The `sourceConfig` is defined [here](https://github.com/open-metadata/OpenMetadata/blob/main/openmetadata-spec/src/main/resources/json/schema/metadataIngestion/databaseServiceMetadataPipeline.json) : **markDeletedTables**: To flag tables as soft-deleted if they are not present anymore in the source system. **markDeletedStoredProcedures**: Optional configuration to soft delete stored procedures in OpenMetadata if the source stored procedures are deleted. Also, if the stored procedures is deleted, all the associated entities like lineage, etc., with that stored procedures will be deleted. **markDeletedSchemas**: Optional configuration to soft delete schemas stored in OpenMetadata if the source schema is deleted. Setting this flag to true will only keep filtered schema and delete any other schemas that do not match schemaFilterPattern or do not exist at source. **markDeletedDatabases**: Optional configuration to soft delete databases stored in OpenMetadata if the source database is deleted. Setting this flag to true will only keep filtered databases and delete any other databases that do not match databaseFilterPattern or do not exist at source. **includeTables**: true or false, to ingest table data. Default is true. **includeViews**: true or false, to ingest views definitions. **includeTags**: Optional configuration to toggle the tags ingestion. **includeOwners**: Set the 'Include Owners' toggle to control whether to include owners to the ingested entity if the owner email matches with a user stored in the OM server as part of metadata ingestion. If the ingested entity already exists and has an owner, the owner will not be overwritten. **includeStoredProcedures**: Optional configuration to toggle the Stored Procedures ingestion. **includeDDL**: Optional configuration to toggle the DDL Statements ingestion. **overrideMetadata** _(boolean)_: Set the 'Override Metadata' toggle to control whether to override the existing metadata in the OpenMetadata server with the metadata fetched from the source. If the toggle is set to true, the metadata fetched from the source will override the existing metadata in the OpenMetadata server. If the toggle is set to false, the metadata fetched from the source will not override the existing metadata in the OpenMetadata server. This is applicable for fields like description, tags, owner and displayName. **queryLogDuration**: Configuration to tune how far we want to look back in query logs to process Stored Procedures results. **queryParsingTimeoutLimit**: Configuration to set the timeout for parsing the query in seconds. **useFqnForFiltering**: Regex will be applied on fully qualified name (e.g service\_name.db\_name.schema\_name.table\_name) instead of raw name (e.g. table\_name). **databaseFilterPattern**, **schemaFilterPattern**, **tableFilterPattern**: Note that the filter supports regex as include or exclude. You can find examples [here](https://docs.open-metadata.org/latest/connectors/ingestion/workflows/metadata/filter-patterns/database) **threads (beta)**: The number of threads to use when extracting the metadata using multithreading. Please take a look [here](https://docs.open-metadata.org/latest/connectors/ingestion/workflows/metadata/multithreading) before configuring this. **databaseMetadataConfigType** _(string)_: Database Source Config Metadata Pipeline type. **incremental (beta)**: Incremental Extraction configuration. Currently implemented for: * [BigQuery](https://docs.open-metadata.org/latest/connectors/ingestion/workflows/metadata/incremental-extraction/bigquery) * [Redshift](https://docs.open-metadata.org/latest/connectors/ingestion/workflows/metadata/incremental-extraction/redshift) * [Snowflake](https://docs.open-metadata.org/latest/connectors/ingestion/workflows/metadata/incremental-extraction/snowflake) #### Sink Configuration To send the metadata to OpenMetadata, it needs to be specified as `type: metadata-rest`. #### Workflow Configuration The main property here is the `openMetadataServerConfig`, where you can define the host and security provider of your OpenMetadata installation. **Logger Level** You can specify the `loggerLevel` depending on your needs. If you are trying to troubleshoot an ingestion, running with `DEBUG` will give you far more traces for identifying issues. **JWT Token** JWT tokens will allow your clients to authenticate against the OpenMetadata server. To enable JWT Tokens, you will get more details [here](https://docs.open-metadata.org/latest/deployment/security/enable-jwt-tokens) . You can refer to the JWT Troubleshooting section [link](https://docs.open-metadata.org/latest/deployment/security/jwt-troubleshooting) for any issues in your JWT configuration. **Store Service Connection** If set to `true` (default), we will store the sensitive information either encrypted via the Fernet Key in the database or externally, if you have configured any [Secrets Manager](https://docs.open-metadata.org/latest/deployment/secrets-manager) . If set to `false`, the service will be created, but the service connection information will only be used by the Ingestion Framework at runtime, and won't be sent to the OpenMetadata server. **Store Service Connection** If set to `true` (default), we will store the sensitive information either encrypted via the Fernet Key in the database or externally, if you have configured any [Secrets Manager](https://docs.open-metadata.org/latest/deployment/secrets-manager) . If set to `false`, the service will be created, but the service connection information will only be used by the Ingestion Framework at runtime, and won't be sent to the OpenMetadata server. **SSL Configuration** If you have added SSL to the [OpenMetadata server](https://docs.open-metadata.org/latest/deployment/security/enable-ssl) , then you will need to handle the certificates when running the ingestion too. You can either set `verifySSL` to `ignore`, or have it as `validate`, which will require you to set the `sslConfig.caCertificate` with a local path where your ingestion runs that points to the server certificate file. Find more information on how to troubleshoot SSL issues [here](https://docs.open-metadata.org/latest/deployment/security/enable-ssl/ssl-troubleshooting) . **ingestionPipelineFQN** Fully qualified name of ingestion pipeline, used to identify the current ingestion pipeline. ##### Advanced Configuration **Connection Options (Optional)**: Enter the details for any additional connection options that can be sent to database during the connection. These details must be added as Key-Value pairs. **Connection Arguments (Optional)**: Enter the details for any additional connection arguments such as security or protocol configs that can be sent to database during the connection. These details must be added as Key-Value pairs. * In case you are using Single-Sign-On (SSO) for authentication, add the `authenticator` details in the Connection Arguments as a Key-Value pair as follows: `"authenticator" : "sso_login_url"` filename.yamlCopy #### Source Configuration - From Storage - S3 ##### Source Configuration - Service Connection * **awsAccessKeyId**: Enter your secure access key ID for your DynamoDB connection. The specified key ID should be authorized to read all databases you want to include in the metadata ingestion workflow. * **awsSecretAccessKey**: Enter the Secret Access Key (the passcode key pair to the key ID from above). * **awsRegion**: Specify the region in which your DynamoDB is located. This setting is required even if you have configured a local AWS profile. * **schemaFilterPattern** and **tableFilterPattern**: Note that the `schemaFilterPattern` and `tableFilterPattern` both support regex as `include` or `exclude`. E.g., #### Source Configuration - Source Config The `sourceConfig` is defined [here](https://github.com/open-metadata/OpenMetadata/blob/main/openmetadata-spec/src/main/resources/json/schema/metadataIngestion/databaseServiceMetadataPipeline.json) : **markDeletedTables**: To flag tables as soft-deleted if they are not present anymore in the source system. **markDeletedStoredProcedures**: Optional configuration to soft delete stored procedures in OpenMetadata if the source stored procedures are deleted. Also, if the stored procedures is deleted, all the associated entities like lineage, etc., with that stored procedures will be deleted. **markDeletedSchemas**: Optional configuration to soft delete schemas stored in OpenMetadata if the source schema is deleted. Setting this flag to true will only keep filtered schema and delete any other schemas that do not match schemaFilterPattern or do not exist at source. **markDeletedDatabases**: Optional configuration to soft delete databases stored in OpenMetadata if the source database is deleted. Setting this flag to true will only keep filtered databases and delete any other databases that do not match databaseFilterPattern or do not exist at source. **includeTables**: true or false, to ingest table data. Default is true. **includeViews**: true or false, to ingest views definitions. **includeTags**: Optional configuration to toggle the tags ingestion. **includeOwners**: Set the 'Include Owners' toggle to control whether to include owners to the ingested entity if the owner email matches with a user stored in the OM server as part of metadata ingestion. If the ingested entity already exists and has an owner, the owner will not be overwritten. **includeStoredProcedures**: Optional configuration to toggle the Stored Procedures ingestion. **includeDDL**: Optional configuration to toggle the DDL Statements ingestion. **overrideMetadata** _(boolean)_: Set the 'Override Metadata' toggle to control whether to override the existing metadata in the OpenMetadata server with the metadata fetched from the source. If the toggle is set to true, the metadata fetched from the source will override the existing metadata in the OpenMetadata server. If the toggle is set to false, the metadata fetched from the source will not override the existing metadata in the OpenMetadata server. This is applicable for fields like description, tags, owner and displayName. **queryLogDuration**: Configuration to tune how far we want to look back in query logs to process Stored Procedures results. **queryParsingTimeoutLimit**: Configuration to set the timeout for parsing the query in seconds. **useFqnForFiltering**: Regex will be applied on fully qualified name (e.g service\_name.db\_name.schema\_name.table\_name) instead of raw name (e.g. table\_name). **databaseFilterPattern**, **schemaFilterPattern**, **tableFilterPattern**: Note that the filter supports regex as include or exclude. You can find examples [here](https://docs.open-metadata.org/latest/connectors/ingestion/workflows/metadata/filter-patterns/database) **threads (beta)**: The number of threads to use when extracting the metadata using multithreading. Please take a look [here](https://docs.open-metadata.org/latest/connectors/ingestion/workflows/metadata/multithreading) before configuring this. **databaseMetadataConfigType** _(string)_: Database Source Config Metadata Pipeline type. **incremental (beta)**: Incremental Extraction configuration. Currently implemented for: * [BigQuery](https://docs.open-metadata.org/latest/connectors/ingestion/workflows/metadata/incremental-extraction/bigquery) * [Redshift](https://docs.open-metadata.org/latest/connectors/ingestion/workflows/metadata/incremental-extraction/redshift) * [Snowflake](https://docs.open-metadata.org/latest/connectors/ingestion/workflows/metadata/incremental-extraction/snowflake) #### Sink Configuration To send the metadata to OpenMetadata, it needs to be specified as `type: metadata-rest`. #### Workflow Configuration The main property here is the `openMetadataServerConfig`, where you can define the host and security provider of your OpenMetadata installation. **Logger Level** You can specify the `loggerLevel` depending on your needs. If you are trying to troubleshoot an ingestion, running with `DEBUG` will give you far more traces for identifying issues. **JWT Token** JWT tokens will allow your clients to authenticate against the OpenMetadata server. To enable JWT Tokens, you will get more details [here](https://docs.open-metadata.org/latest/deployment/security/enable-jwt-tokens) . You can refer to the JWT Troubleshooting section [link](https://docs.open-metadata.org/latest/deployment/security/jwt-troubleshooting) for any issues in your JWT configuration. **Store Service Connection** If set to `true` (default), we will store the sensitive information either encrypted via the Fernet Key in the database or externally, if you have configured any [Secrets Manager](https://docs.open-metadata.org/latest/deployment/secrets-manager) . If set to `false`, the service will be created, but the service connection information will only be used by the Ingestion Framework at runtime, and won't be sent to the OpenMetadata server. **Store Service Connection** If set to `true` (default), we will store the sensitive information either encrypted via the Fernet Key in the database or externally, if you have configured any [Secrets Manager](https://docs.open-metadata.org/latest/deployment/secrets-manager) . If set to `false`, the service will be created, but the service connection information will only be used by the Ingestion Framework at runtime, and won't be sent to the OpenMetadata server. **SSL Configuration** If you have added SSL to the [OpenMetadata server](https://docs.open-metadata.org/latest/deployment/security/enable-ssl) , then you will need to handle the certificates when running the ingestion too. You can either set `verifySSL` to `ignore`, or have it as `validate`, which will require you to set the `sslConfig.caCertificate` with a local path where your ingestion runs that points to the server certificate file. Find more information on how to troubleshoot SSL issues [here](https://docs.open-metadata.org/latest/deployment/security/enable-ssl/ssl-troubleshooting) . **ingestionPipelineFQN** Fully qualified name of ingestion pipeline, used to identify the current ingestion pipeline. filename.yamlCopy ### 2\. Run with the CLI First, we will need to save the YAML file. Afterward, and with all requirements installed, we can run: Note that from connector to connector, this recipe will always be the same. By updating the YAML configuration, you will be able to extract metadata from different sources. dbt Integration --------------- [#### dbt Integration\ \ Learn more about how to ingest dbt models' definitions and their lineage.](https://docs.open-metadata.org/latest/connectors/ingestion/workflows/dbt) --- # Great Expectations | OpenMetadata Data Quality Integration We use cookies to improve site navigation, analyze site usage, and enhance your user experience. Click "Accept" to enable cookies or "Reject" to reject cookies. AcceptReject connectors No menu items for this category [Home](https://docs.open-metadata.org/latest) /[Connectors](https://docs.open-metadata.org/latest/connectors) /[Ingestion](https://docs.open-metadata.org/latest/connectors/ingestion) /[Great Expectations](https://docs.open-metadata.org/latest/connectors/ingestion/great-expectations) OpenMetadata Documentation Great Expectations ================== For Data Quality tests the open source python package Great Expectations stands out from the crowd. For those of you who don't know, [Great Expectations](https://greatexpectations.io/) is a shared, open standard for data quality. It helps data teams eliminate pipeline debt, through data testing, documentation, and profiling. Learn more about the product in [their documentation](https://docs.greatexpectations.io/docs/) . With this tutorial, we show you how to configure Great Expectations to integrate with OpenMetadata and ingest your test results to your table service page. Requirements ------------ ### OpenMetadata Requirements You will to have OpenMetadata version 0.12 or later. To deploy OpenMetadata, follow the procedure to [Try OpenMetadata in Docker](https://docs.open-metadata.org/latest/quick-start/local-docker-deployment) . Before ingesting your tests results from Great Expectations you will need to have your table metadata ingested into OpenMetadata. Follow the instruction in the [Connectors](https://docs.open-metadata.org/latest/connectors) section to learn more. ### Python Requirements We have support for Python versions 3.9-3.11 You will need to install our Great Expectations submodule Great Expectations Setup ------------------------ ### Configure your checkpoint file Great Expectations integration leverage custom actions. To be able to execute custom actions you will need to run a checkpoint file. In your checkpoint yaml file, you will need to add the above code block in `action_list` section. **Properties**: * `module_name`: this is OpenMetadata submodule name * `class_name`: this is the name of the class that will be used to execute the custom action * `config_file_path`: this is the path to your `config.yaml` file that holds the configuration of your OpenMetadata server * `database_service_name`: \[Optional\] this is an optional parameter. If not specified and 2 tables have the same name in 2 different OpenMetadata services, the custom action will fail * `database_name`: \[Optional\] The database name as it appears in OpenMetadata. For table-based validations (`SqlAlchemyDatasourceBatchSpec`), this is inferred from the batch spec. **Required** for query-based or dataframe validations (`RuntimeQueryBatchSpec`, `RuntimeDataBatchSpec`) where the table context must be explicitly specified. * `schema_name`: \[Optional\] The schema name as it appears in OpenMetadata. For table-based validations, this is inferred from the batch spec. **Required** for query-based or dataframe validations. Defaults to _default_ if not specified. * `table_name`: \[Optional\] The table name as it appears in OpenMetadata. For table-based validations, this is inferred from the batch spec. **Required** for query-based or dataframe validations where the table cannot be automatically determined. * `expectation_suite_table_config_map`: \[Optional\] A dictionary mapping expectation suite names to their target OpenMetadata tables. Required when running multi-table checkpoints, where different expectation suites should send results to different tables. Each entry specifies the `database_name`, `schema_name` and `table_name` for routing validation results. **Multi-Table Checkpoints** When validating **multiple tables in a single checkpoint**, use the `expectation_suite_table_config_map` parameter to route validation results to the correct OpenMetadata tables. This is necessary because: * Each expectation suite may target a different table * The checkpoint action needs to know where to send each suite's results * Without the mapping, all results would attempt to go to the same default table **Example scenario:** You have a checkpoint validating both `users` and `orders` tables with separate expectation suites (`users_suite` and `orders_suite`). The `expectation_suite_table_config_map` ensures `users_suite` results go to the `users` table and the `orders_suite` go to the `orders` table. For single-table checkpoints, this parameter is not needed - the table information is provided directly or inferred from the batch spec. **Note** If you are using Great Expectation `DataContext` instance in Python to run your tests, you can use the `run_checkpoint` method as follows: ### Create your `config.yaml` file To ingest Great Expectations results in OpenMetadata, you will need to specify your OpenMetadata security configuration for the REST endpoint. This configuration file needs to be located inside the `config_file_path` referenced in step 2 and named `config.yaml`. You can use environment variables in your configuration file by simply using `{{ env('') }}`. These will be parsed and rendered at runtime allowing you to securely create your configuration and commit it to your favorite version control tool. As we support multiple security configurations, you can check out the [Enable Security](https://docs.open-metadata.org/latest/deployment/security) section for more details on how to set the `securityConfig` part of the `yaml` file. ![Great Expectations config file](https://docs.open-metadata.org/images/v1.11/features/integrations/ge-config-yaml.gif) ### Run your Great Expectations Checkpoint File With everything set up, it is now time to run your checkpoint file. ![Run Great Expectations checkpoint](https://docs.open-metadata.org/images/v1.11/features/integrations/ge-run-checkpoint.gif) ### Running GX using the Python SDK? If you are running GX using their Python SDK below is a full example of how to add the action to your code #### Multi-Table Checkpoint Example Validate multiple tables in a single checkpoint run: ### Working with GX 1.x.x? In v1.x.x GX introduced significant changes to their SDK. One notable change was the removal of the `great_expectations` CLI. OpenMetadata introduced support for 1.x.x version through its `OpenMetadataValidationAction1xx` class. You will need to first \`pip install 'open-metadata\[great-expectations-1xx\]'. Below is a complete example #### Multi-Table Checkpoint Example Validate multiple tables in a single checkpoint run: ### List of Great Expectations Supported Test We currently only support a certain number of Great Expectations tests. The full list can be found in the [Tests](https://docs.open-metadata.org/latest/how-to-guides/data-quality-observability/quality/tests-yaml) section. If a test is not supported, there is no need to worry about the execution of your Great Expectations test. We will simply skip the tests that are not supported and continue the execution of your test suite. --- # SAP HANA Troubleshooting Guide | OpenMetadata Support We use cookies to improve site navigation, analyze site usage, and enhance your user experience. Click "Accept" to enable cookies or "Reject" to reject cookies. AcceptReject connectors No menu items for this category [Home](https://docs.open-metadata.org/latest) /[Connectors](https://docs.open-metadata.org/latest/connectors) /[Database](https://docs.open-metadata.org/latest/connectors/database) /[Sap Hana](https://docs.open-metadata.org/latest/connectors/database/sap-hana) /[Troubleshooting](https://docs.open-metadata.org/latest/connectors/database/sap-hana/troubleshooting) OpenMetadata Documentation Troubleshooting =============== Workflow Deployment Error ------------------------- If there were any errors during the workflow deployment process, the Ingestion Pipeline Entity will still be created, but no workflow will be present in the Ingestion container. * You can then Edit the Ingestion Pipeline and **Deploy** it again. * From the Connection tab, you can also Edit the Service if needed. Connector Debug Troubleshooting ------------------------------- This section provides instructions to help resolve common issues encountered during connector setup and metadata ingestion in OpenMetadata. Below are some of the most frequently observed troubleshooting scenarios. How to Enable Debug Logging for Any Ingestion --------------------------------------------- To enable debug logging for any ingestion workflow in OpenMetadata: 1. **Navigate to Services** Go to **Settings > Services > Service Type** (e.g., Database) in the OpenMetadata UI. 2. **Select a Service** Choose the specific service for which you want to enable debug logging. ![Select a Service](https://docs.open-metadata.org/images/v1.11/connectors/debug/debug1.png) Select a Service 3. **Access Ingestion Tab** Go to the **Ingestion tab** and click the three-dot menu on the right-hand side of the ingestion type, and select Edit. ![Access Agents Tab](https://docs.open-metadata.org/images/v1.11/connectors/debug/debug2.png) Access Agents Tab 4. **Enable Debug Logging** In the configuration dialog, enable the **Debug Log** option and click **Next**. ![Enable Debug Logging](https://docs.open-metadata.org/images/v1.11/connectors/debug/debug3.png) Enable Debug Logging 5. **Schedule and Submit** Configure the schedule if needed and click **Submit** to apply the changes. ![Schedule and Submit](https://docs.open-metadata.org/images/v1.11/connectors/debug/debug4.png) Schedule and Submit Permission Issues ----------------- If you encounter permission-related errors during connector setup or metadata ingestion, ensure that all the prerequisites and access configurations specified for each connector are properly implemented. Refer to the connector-specific documentation to verify the required permissions. --- # Run the Databricks Pipeline Connector Externally We use cookies to improve site navigation, analyze site usage, and enhance your user experience. Click "Accept" to enable cookies or "Reject" to reject cookies. AcceptReject connectors No menu items for this category [Home](https://docs.open-metadata.org/latest) /[Connectors](https://docs.open-metadata.org/latest/connectors) /[Pipeline](https://docs.open-metadata.org/latest/connectors/pipeline) /[Databricks Pipeline](https://docs.open-metadata.org/latest/connectors/pipeline/databricks-pipeline) /[Yaml](https://docs.open-metadata.org/latest/connectors/pipeline/databricks-pipeline/yaml) OpenMetadata Documentation ![Databricks Pipeline](https://docs.open-metadata.org/_next/image?url=%2Fimages%2Fconnectors%2Fdatabrick.webp&w=64&q=75) Databricks Pipeline =================== PROD Available In Feature List Pipelines Pipeline Status Usage Owners Tags Lineage In this section, we provide guides and references to use the Databricks Pipeline connector. Configure and schedule Databricks Pipeline metadata and profiler workflows from the OpenMetadata UI: * [Requirements](https://docs.open-metadata.org/latest/connectors/pipeline/databricks-pipeline/yaml#requirements) * [Metadata Ingestion](https://docs.open-metadata.org/latest/connectors/pipeline/databricks-pipeline/yaml#metadata-ingestion) How to Run the Connector Externally ----------------------------------- To run the Ingestion via the UI you'll need to use the OpenMetadata Ingestion Container, which comes shipped with custom Airflow plugins to handle the workflow deployment. If, instead, you want to manage your workflows externally on your preferred orchestrator, you can check the following docs to run the Ingestion Framework **anywhere**. [#### External Schedulers\ \ Get more information about running the Ingestion Framework Externally](https://docs.open-metadata.org/latest/deployment/ingestion) Requirements ------------ ### Python Requirements We have support for Python versions 3.9-3.11 To run the Databricks Pipeline ingestion, you will need to install: Metadata Ingestion ------------------ All connectors are defined as JSON Schemas. [Here](https://github.com/open-metadata/OpenMetadata/blob/main/openmetadata-spec/src/main/resources/json/schema/entity/services/connections/pipeline/databricksPipelineConnection.json) you can find the structure to create a connection to Databricks Pipeline. In order to create and run a Metadata Ingestion workflow, we will follow the steps to create a YAML configuration able to connect to the source, process the Entities if needed, and reach the OpenMetadata server. The workflow is modeled around the following [JSON Schema](https://github.com/open-metadata/OpenMetadata/blob/main/openmetadata-spec/src/main/resources/json/schema/metadataIngestion/workflow.json) ### 1\. Define the YAML Config This is a sample config for Databricks Pipeline: #### Source Configuration - Service Connection **Host and Port**: Enter the fully qualified hostname and port number for your Databricks Pipeline deployment in the Host and Port field. **Token**: Generated Token to connect to Databricks Pipeline. **Connection Arguments (Optional)**: Enter the details for any additional connection arguments such as security or protocol configs that can be sent to Databricks during the connection. These details must be added as Key-Value pairs. * In case you are using Single-Sign-On (SSO) for authentication, add the `authenticator` details in the Connection Arguments as a Key-Value pair as follows: `"authenticator" : "sso_login_url"` **HTTP Path**: Databricks Pipeline compute resources URL. #### Source Configuration - Source Config The `sourceConfig` is defined [here](https://github.com/open-metadata/OpenMetadata/blob/main/openmetadata-spec/src/main/resources/json/schema/metadataIngestion/pipelineServiceMetadataPipeline.json) : * **dbServiceNames**: Database Service Name for the creation of lineage, if the source supports it. * **includeTags**: Set the 'Include Tags' toggle to control whether to include tags as part of metadata ingestion. * **includeUnDeployedPipelines**: Set the 'Include UnDeployed Pipelines' toggle to control whether to include un-deployed pipelines as part of metadata ingestion. By default it is set to `true` * **markDeletedPipelines**: Set the Mark Deleted Pipelines toggle to flag pipelines as soft-deleted if they are not present anymore in the source system. * **pipelineFilterPattern** and **chartFilterPattern**: Note that the `pipelineFilterPattern` and `chartFilterPattern` both support regex as include or exclude. * **includeOwners**: Set the 'Include Owners' toggle to control whether to include owners to the ingested entity if the owner email matches with a user stored in the OM server as part of metadata ingestion. If the ingested entity already exists and has an owner, the owner will not be overwritten.It supports boolean values either `true` or `false`. * **overrideLineage**: Set the 'Override Lineage' toggle to control whether to override the existing lineage. It supports boolean values either `true` or `false`. * **overrideMetadata**: Set the 'Override Metadata' toggle to control whether to override the existing metadata in the OpenMetadata server with the metadata fetched from the source. If the toggle is set to true, the metadata fetched from the source will override the existing metadata in the OpenMetadata server. If the toggle is set to false, the metadata fetched from the source will not override the existing metadata in the OpenMetadata server. This is applicable for fields like description, tags, owner and displayName. It supports boolean values either `true` or `false`. #### Sink Configuration To send the metadata to OpenMetadata, it needs to be specified as `type: metadata-rest`. #### Workflow Configuration The main property here is the `openMetadataServerConfig`, where you can define the host and security provider of your OpenMetadata installation. **Logger Level** You can specify the `loggerLevel` depending on your needs. If you are trying to troubleshoot an ingestion, running with `DEBUG` will give you far more traces for identifying issues. **JWT Token** JWT tokens will allow your clients to authenticate against the OpenMetadata server. To enable JWT Tokens, you will get more details [here](https://docs.open-metadata.org/latest/deployment/security/enable-jwt-tokens) . You can refer to the JWT Troubleshooting section [link](https://docs.open-metadata.org/latest/deployment/security/jwt-troubleshooting) for any issues in your JWT configuration. **Store Service Connection** If set to `true` (default), we will store the sensitive information either encrypted via the Fernet Key in the database or externally, if you have configured any [Secrets Manager](https://docs.open-metadata.org/latest/deployment/secrets-manager) . If set to `false`, the service will be created, but the service connection information will only be used by the Ingestion Framework at runtime, and won't be sent to the OpenMetadata server. **Store Service Connection** If set to `true` (default), we will store the sensitive information either encrypted via the Fernet Key in the database or externally, if you have configured any [Secrets Manager](https://docs.open-metadata.org/latest/deployment/secrets-manager) . If set to `false`, the service will be created, but the service connection information will only be used by the Ingestion Framework at runtime, and won't be sent to the OpenMetadata server. **SSL Configuration** If you have added SSL to the [OpenMetadata server](https://docs.open-metadata.org/latest/deployment/security/enable-ssl) , then you will need to handle the certificates when running the ingestion too. You can either set `verifySSL` to `ignore`, or have it as `validate`, which will require you to set the `sslConfig.caCertificate` with a local path where your ingestion runs that points to the server certificate file. Find more information on how to troubleshoot SSL issues [here](https://docs.open-metadata.org/latest/deployment/security/enable-ssl/ssl-troubleshooting) . **ingestionPipelineFQN** Fully qualified name of ingestion pipeline, used to identify the current ingestion pipeline. filename.yamlCopy ### 2\. Run with the CLI First, we will need to save the YAML file. Afterward, and with all requirements installed, we can run: Note that from connector to connector, this recipe will always be the same. By updating the YAML configuration, you will be able to extract metadata from different sources. --- # Run the Kinesis Connector Externally We use cookies to improve site navigation, analyze site usage, and enhance your user experience. Click "Accept" to enable cookies or "Reject" to reject cookies. AcceptReject connectors No menu items for this category [Home](https://docs.open-metadata.org/latest) /[Connectors](https://docs.open-metadata.org/latest/connectors) /[Messaging](https://docs.open-metadata.org/latest/connectors/messaging) /[Kinesis](https://docs.open-metadata.org/latest/connectors/messaging/kinesis) /[Yaml](https://docs.open-metadata.org/latest/connectors/messaging/kinesis/yaml) OpenMetadata Documentation ![Kinesis](https://docs.open-metadata.org/_next/image?url=%2Fimages%2Fconnectors%2Fkinesis.webp&w=64&q=75) Kinesis ======= PROD Available In Feature List Topics Sample Data In this section, we provide guides and references to use the Kinesis connector. Configure and schedule Kinesis metadata workflows from the OpenMetadata UI: * [Requirements](https://docs.open-metadata.org/latest/connectors/messaging/kinesis/yaml#requirements) * [Metadata Ingestion](https://docs.open-metadata.org/latest/connectors/messaging/kinesis/yaml#metadata-ingestion) How to Run the Connector Externally ----------------------------------- To run the Ingestion via the UI you'll need to use the OpenMetadata Ingestion Container, which comes shipped with custom Airflow plugins to handle the workflow deployment. If, instead, you want to manage your workflows externally on your preferred orchestrator, you can check the following docs to run the Ingestion Framework **anywhere**. [#### External Schedulers\ \ Get more information about running the Ingestion Framework Externally](https://docs.open-metadata.org/latest/deployment/ingestion) Requirements ------------ OpenMetadata retrieves information about streams and sample data from the streams in the AWS account. The user must have the following policy set to access the metadata from Kinesis. For more information on Kinesis permissions visit the [AWS Kinesis official documentation](https://docs.aws.amazon.com/streams/latest/dev/controlling-access.html) . ### Python Requirements We have support for Python versions 3.9-3.11 To run the Kinesis ingestion, you will need to install: Metadata Ingestion ------------------ All connectors are defined as JSON Schemas. [Here](https://github.com/open-metadata/OpenMetadata/blob/main/openmetadata-spec/src/main/resources/json/schema/entity/services/connections/messaging/kinesisConnection.json) you can find the structure to create a connection to Kinesis. In order to create and run a Metadata Ingestion workflow, we will follow the steps to create a YAML configuration able to connect to the source, process the Entities if needed, and reach the OpenMetadata server. The workflow is modeled around the following [JSON Schema](https://github.com/open-metadata/OpenMetadata/blob/main/openmetadata-spec/src/main/resources/json/schema/metadataIngestion/workflow.json) ### 1\. Define the YAML Config This is a sample config for Kinesis: #### Source Configuration - Service Connection * **awsAccessKeyId** & **awsSecretAccessKey**: When you interact with AWS, you specify your AWS security credentials to verify who you are and whether you have permission to access the resources that you are requesting. AWS uses the security credentials to authenticate and authorize your requests ([docs](https://docs.aws.amazon.com/IAM/latest/UserGuide/security-creds.html) ). Access keys consist of two parts: An **access key ID** (for example, `AKIAIOSFODNN7EXAMPLE`), and a **secret access key** (for example, `wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY`). You must use both the access key ID and secret access key together to authenticate your requests. You can find further information on how to manage your access keys [here](https://docs.aws.amazon.com/IAM/latest/UserGuide/id_credentials_access-keys.html) . **awsSessionToken**: If you are using temporary credentials to access your services, you will need to inform the AWS Access Key ID and AWS Secrets Access Key. Also, these will include an AWS Session Token. **awsRegion**: Each AWS Region is a separate geographic area in which AWS clusters data centers ([docs](https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/Concepts.RegionsAndAvailabilityZones.html) ). As AWS can have instances in multiple regions, we need to know the region the service you want reach belongs to. Note that the AWS Region is the only required parameter when configuring a connection. When connecting to the services programmatically, there are different ways in which we can extract and use the rest of AWS configurations. You can find further information about configuring your credentials [here](https://boto3.amazonaws.com/v1/documentation/api/latest/guide/credentials.html#configuring-credentials) . **endPointURL**: To connect programmatically to an AWS service, you use an endpoint. An _endpoint_ is the URL of the entry point for an AWS web service. The AWS SDKs and the AWS Command Line Interface (AWS CLI) automatically use the default endpoint for each service in an AWS Region. But you can specify an alternate endpoint for your API requests. Find more information on [AWS service endpoints](https://docs.aws.amazon.com/general/latest/gr/rande.html) . **profileName**: A named profile is a collection of settings and credentials that you can apply to a AWS CLI command. When you specify a profile to run a command, the settings and credentials are used to run that command. Multiple named profiles can be stored in the config and credentials files. You can inform this field if you'd like to use a profile other than `default`. Find here more information about [Named profiles for the AWS CLI](https://docs.aws.amazon.com/cli/latest/userguide/cli-configure-profiles.html) . **assumeRoleArn**: Typically, you use `AssumeRole` within your account or for cross-account access. In this field you'll set the `ARN` (Amazon Resource Name) of the policy of the other account. A user who wants to access a role in a different account must also have permissions that are delegated from the account administrator. The administrator must attach a policy that allows the user to call `AssumeRole` for the `ARN` of the role in the other account. This is a required field if you'd like to `AssumeRole`. Find more information on [AssumeRole](https://docs.aws.amazon.com/STS/latest/APIReference/API_AssumeRole.html) . When using Assume Role authentication, ensure you provide the following details: * **AWS Region**: Specify the AWS region for your deployment. * **Assume Role ARN**: Provide the ARN of the role in your AWS account that OpenMetadata will assume. **assumeRoleSessionName**: An identifier for the assumed role session. Use the role session name to uniquely identify a session when the same role is assumed by different principals or for different reasons. By default, we'll use the name `OpenMetadataSession`. Find more information about the [Role Session Name](https://docs.aws.amazon.com/STS/latest/APIReference/API_AssumeRole.html#:~:text=An%20identifier%20for%20the%20assumed%20role%20session.) . **assumeRoleSourceIdentity**: The source identity specified by the principal that is calling the `AssumeRole` operation. You can use source identity information in AWS CloudTrail logs to determine who took actions with a role. Find more information about [Source Identity](https://docs.aws.amazon.com/STS/latest/APIReference/API_AssumeRole.html#:~:text=Required%3A%20No-,SourceIdentity,-The%20source%20identity) . #### Source Configuration - Source Config The sourceConfig is defined [here](https://github.com/open-metadata/OpenMetadata/blob/main/openmetadata-spec/src/main/resources/json/schema/metadataIngestion/messagingServiceMetadataPipeline.json) : * **generateSampleData:** Option to turn on/off generating sample data during metadata extraction. * **topicFilterPattern:** Note that the `topicFilterPattern` supports regex as include or exclude. * **generateSampleData:** Option to turn on/off generating sample data during metadata extraction. `generateSampleData` supports boolean value either `true` or `false`. * **markDeletedTopics:** Optional configuration to soft delete topics in OpenMetadata if the source topics are deleted. Also, if the topic is deleted, all the associated entities like sample data, lineage, etc., with that topic will be deleted. `markDeletedTopics` supports boolean value either `true` or `false`. * **overrideMetadata:** Set the 'Override Metadata' toggle to control whether to override the existing metadata in the OpenMetadata server with the metadata fetched from the source. If the toggle is set to true, the metadata fetched from the source will override the existing metadata in the OpenMetadata server. If the toggle is set to false, the metadata fetched from the source will not override the existing metadata in the OpenMetadata server. This is applicable for fields like description, tags, owner and displayName. `overrideMetadata` supports boolean value either `true` or `false`. #### Sink Configuration To send the metadata to OpenMetadata, it needs to be specified as `type: metadata-rest`. #### Workflow Configuration The main property here is the `openMetadataServerConfig`, where you can define the host and security provider of your OpenMetadata installation. **Logger Level** You can specify the `loggerLevel` depending on your needs. If you are trying to troubleshoot an ingestion, running with `DEBUG` will give you far more traces for identifying issues. **JWT Token** JWT tokens will allow your clients to authenticate against the OpenMetadata server. To enable JWT Tokens, you will get more details [here](https://docs.open-metadata.org/latest/deployment/security/enable-jwt-tokens) . You can refer to the JWT Troubleshooting section [link](https://docs.open-metadata.org/latest/deployment/security/jwt-troubleshooting) for any issues in your JWT configuration. **Store Service Connection** If set to `true` (default), we will store the sensitive information either encrypted via the Fernet Key in the database or externally, if you have configured any [Secrets Manager](https://docs.open-metadata.org/latest/deployment/secrets-manager) . If set to `false`, the service will be created, but the service connection information will only be used by the Ingestion Framework at runtime, and won't be sent to the OpenMetadata server. **Store Service Connection** If set to `true` (default), we will store the sensitive information either encrypted via the Fernet Key in the database or externally, if you have configured any [Secrets Manager](https://docs.open-metadata.org/latest/deployment/secrets-manager) . If set to `false`, the service will be created, but the service connection information will only be used by the Ingestion Framework at runtime, and won't be sent to the OpenMetadata server. **SSL Configuration** If you have added SSL to the [OpenMetadata server](https://docs.open-metadata.org/latest/deployment/security/enable-ssl) , then you will need to handle the certificates when running the ingestion too. You can either set `verifySSL` to `ignore`, or have it as `validate`, which will require you to set the `sslConfig.caCertificate` with a local path where your ingestion runs that points to the server certificate file. Find more information on how to troubleshoot SSL issues [here](https://docs.open-metadata.org/latest/deployment/security/enable-ssl/ssl-troubleshooting) . **ingestionPipelineFQN** Fully qualified name of ingestion pipeline, used to identify the current ingestion pipeline. filename.yamlCopy ### 2\. Run with the CLI First, we will need to save the YAML file. Afterward, and with all requirements installed, we can run: Note that from connector to connector, this recipe will always be the same. By updating the YAML configuration, you will be able to extract metadata from different sources. --- # Run the Grafana Connector Externally We use cookies to improve site navigation, analyze site usage, and enhance your user experience. Click "Accept" to enable cookies or "Reject" to reject cookies. AcceptReject connectors No menu items for this category [Home](https://docs.open-metadata.org/latest) /[Connectors](https://docs.open-metadata.org/latest/connectors) /[Dashboard](https://docs.open-metadata.org/latest/connectors/dashboard) /[Grafana](https://docs.open-metadata.org/latest/connectors/dashboard/grafana) /[Yaml](https://docs.open-metadata.org/latest/connectors/dashboard/grafana/yaml) OpenMetadata Documentation ![Grafana](https://docs.open-metadata.org/_next/image?url=%2Fimages%2Fconnectors%2Fgrafana.webp&w=64&q=75) Grafana ======= BETA Available In Feature List Dashboards Charts Owners Tags Lineage In this section, we provide guides and references to use the Grafana connector. Configure and schedule Grafana metadata and profiler workflows from the OpenMetadata UI: * [Requirements](https://docs.open-metadata.org/latest/connectors/dashboard/grafana/yaml#requirements) * [Metadata Ingestion](https://docs.open-metadata.org/latest/connectors/dashboard/grafana/yaml#metadata-ingestion) How to Run the Connector Externally ----------------------------------- To run the Ingestion via the UI you'll need to use the OpenMetadata Ingestion Container, which comes shipped with custom Airflow plugins to handle the workflow deployment. If, instead, you want to manage your workflows externally on your preferred orchestrator, you can check the following docs to run the Ingestion Framework **anywhere**. [#### External Schedulers\ \ Get more information about running the Ingestion Framework Externally](https://docs.open-metadata.org/latest/deployment/ingestion) Requirements ------------ You will need: * Grafana 9.0+ (Service Account Tokens) * Service Account Token with Admin role (for full metadata extraction) * Network access to Grafana API endpoints ### Python Requirements We have support for Python versions 3.9-3.11 To run the Grafana ingestion, install: Metadata Ingestion ------------------ All connectors are defined as JSON Schemas. [Here](https://github.com/open-metadata/OpenMetadata/blob/main/openmetadata-spec/src/main/resources/json/schema/entity/services/connections/dashboard/grafanaConnection.json) you can find the structure to create a connection to Grafana. In order to create and run a Metadata Ingestion workflow, we will follow the steps to create a YAML configuration able to connect to the source, process the Entities if needed, and reach the OpenMetadata server. ### 1\. Define the YAML Config This is a sample config for Grafana: #### Source Configuration - Service Connection **hostPort**: URL or IP address of your Grafana instance. **apiKey**: Service Account Token for authentication (format: `glsa_xxxxx`). Admin role recommended. **verifySSL**: (Optional) Whether to verify SSL certificates. Default: true **pageSize**: (Optional) Page size for Grafana API pagination. Default: 100 **includeTags**: When set to true, imports Grafana tags as OpenMetadata tags. #### Sink Configuration To send the metadata to OpenMetadata, specify `type: metadata-rest`. #### Workflow Configuration The main property here is the `openMetadataServerConfig`, where you can define the host and security provider of your OpenMetadata installation. **Logger Level** You can specify the `loggerLevel` depending on your needs. If you are trying to troubleshoot an ingestion, running with `DEBUG` will give you far more traces for identifying issues. **JWT Token** JWT tokens will allow your clients to authenticate against the OpenMetadata server. To enable JWT Tokens, you will get more details [here](https://docs.open-metadata.org/latest/deployment/security/enable-jwt-tokens) . You can refer to the JWT Troubleshooting section [link](https://docs.open-metadata.org/latest/deployment/security/jwt-troubleshooting) for any issues in your JWT configuration. **Store Service Connection** If set to `true` (default), we will store the sensitive information either encrypted via the Fernet Key in the database or externally, if you have configured any [Secrets Manager](https://docs.open-metadata.org/latest/deployment/secrets-manager) . If set to `false`, the service will be created, but the service connection information will only be used by the Ingestion Framework at runtime, and won't be sent to the OpenMetadata server. **Store Service Connection** If set to `true` (default), we will store the sensitive information either encrypted via the Fernet Key in the database or externally, if you have configured any [Secrets Manager](https://docs.open-metadata.org/latest/deployment/secrets-manager) . If set to `false`, the service will be created, but the service connection information will only be used by the Ingestion Framework at runtime, and won't be sent to the OpenMetadata server. **SSL Configuration** If you have added SSL to the [OpenMetadata server](https://docs.open-metadata.org/latest/deployment/security/enable-ssl) , then you will need to handle the certificates when running the ingestion too. You can either set `verifySSL` to `ignore`, or have it as `validate`, which will require you to set the `sslConfig.caCertificate` with a local path where your ingestion runs that points to the server certificate file. Find more information on how to troubleshoot SSL issues [here](https://docs.open-metadata.org/latest/deployment/security/enable-ssl/ssl-troubleshooting) . **ingestionPipelineFQN** Fully qualified name of ingestion pipeline, used to identify the current ingestion pipeline. grafana-workflow.yamlCopy Securing Grafana Connection with SSL in OpenMetadata ---------------------------------------------------- ### 2\. Run with the CLI First, we will need to save the YAML file. Afterward, and with all requirements installed, we can run: Note that from connector to connector, this recipe will always be the same. By updating the YAML configuration, you will be able to extract metadata from different sources. --- # Domo Pipeline | OpenMetadata Data Pipeline Services We use cookies to improve site navigation, analyze site usage, and enhance your user experience. Click "Accept" to enable cookies or "Reject" to reject cookies. AcceptReject connectors No menu items for this category [Home](https://docs.open-metadata.org/latest) /[Connectors](https://docs.open-metadata.org/latest/connectors) /[Pipeline](https://docs.open-metadata.org/latest/connectors/pipeline) /[Domo Pipeline](https://docs.open-metadata.org/latest/connectors/pipeline/domo-pipeline) OpenMetadata Documentation ![Domo](https://docs.open-metadata.org/_next/image?url=%2Fimages%2Fconnectors%2Fdomo.webp&w=64&q=75) Domo ==== PROD Available In Feature List Pipelines Pipeline Status Lineage Usage Owners Tags In this section, we provide guides and references to use the Domo-Pipeline connector. Configure and schedule Domo-Pipeline metadata and profiler workflows from the OpenMetadata UI: * [Requirements](https://docs.open-metadata.org/latest/connectors/pipeline/domo-pipeline#requirements) * [Metadata Ingestion](https://docs.open-metadata.org/latest/connectors/pipeline/domo-pipeline#metadata-ingestion) * [Troubleshooting](https://docs.open-metadata.org/latest/connectors/pipeline/domo-pipeline/troubleshooting) Ingestion Deployment -------------------- To run the Ingestion via the UI you'll need to use the OpenMetadata Ingestion Container, which comes shipped with custom Airflow plugins to handle the workflow deployment. If you want to install it manually in an already existing Airflow host, you can follow [this](https://docs.open-metadata.org/latest/deployment/ingestion/openmetadata) guide. If you don't want to use the OpenMetadata Ingestion container to configure the workflows via the UI, then you can check the following docs to run the Ingestion Framework in any orchestrator externally. [#### Run Connectors from the OpenMetadata UI\ \ Learn how to manage your deployment to run connectors from the UI](https://docs.open-metadata.org/latest/deployment/ingestion/openmetadata) [#### Run the Connector Externally\ \ Get the YAML to run the ingestion externally](https://docs.open-metadata.org/latest/connectors/pipeline/domo-pipeline/yaml) [#### External Schedulers\ \ Get more information about running the Ingestion Framework Externally](https://docs.open-metadata.org/latest/deployment/ingestion) Requirements ------------ For metadata ingestion, make sure to add at least `data` scopes to the clientId provided. For questions related to scopes, click [here](https://developer.domo.com/portal/1845fc11bbe5d-api-authentication) . Metadata Ingestion ------------------ #### 1\. Visit the Services Page Click `Settings` in the side navigation bar and then `Services`. The first step is to ingest the metadata from your sources. To do that, you first need to create a Service connection first. This Service will be the bridge between OpenMetadata and your source system. Once a Service is created, it can be used to configure your ingestion workflows. ![Visit Services Page](https://docs.open-metadata.org/images/v1.11/connectors/visit-services-page.png) Select your Service Type and Add a New Service #### 2\. Create a New Service Click on _Add New Service_ to start the Service creation. ![Create a new Service](https://docs.open-metadata.org/images/v1.11/connectors/create-new-service.png) Add a new Service from the Services page #### 3\. Select the Service Type Select Domo Pipeline as the Service type and click _Next_. ![Select Service](https://docs.open-metadata.org/images/v1.11/connectors/domopipeline/select-service.png) Select your Service from the list #### 4\. Name and Describe your Service Provide a name and description for your Service. #### Service Name OpenMetadata uniquely identifies Services by their **Service Name**. Provide a name that distinguishes your deployment from other Services, including the other Domo Pipeline Services that you might be ingesting metadata from. Note that when the name is set, it cannot be changed. ![Add New Service](https://docs.open-metadata.org/images/v1.11/connectors/domopipeline/add-new-service.png) Provide a Name and description for your Service #### 5\. Configure the Service Connection In this step, we will configure the connection settings required for Domo Pipeline. Please follow the instructions below to properly configure the Service to read from your sources. You will also find helper documentation on the right-hand side panel in the UI. ![Configure Service connection](https://docs.open-metadata.org/images/v1.11/connectors/domopipeline/service-connection.png) Configure the Service connection by filling the form #### Connection Details * **Client ID**: Client Id for DOMO Pipeline. * **Secret Token**: Secret Token to Connect to DOMO Pipeline. * **Access Token**: Access to Connect to DOMO Pipeline. * **API Host**: API Host to Connect to DOMO Pipeline. * **Instance Domain**: URL to connect to your Domo instance UI. For example `https://.domo.com`. #### 6\. Test the Connection Once the credentials have been added, click on _Test Connection_ and _Save_ the changes. ![Test Connection](https://docs.open-metadata.org/images/v1.11/connectors/test-connection.png) Test the connection and save the Service #### 7\. Configure Metadata Ingestion In this step we will configure the metadata ingestion pipeline, Please follow the instructions below ![Configure Metadata Ingestion](https://docs.open-metadata.org/images/v1.11/connectors/configure-metadata-ingestion-pipeline.png) Configure Metadata Ingestion Page #### Metadata Ingestion Options * **Name**: This field refers to the name of ingestion pipeline, you can customize the name or use the generated name. * **Pipeline Filter Pattern (Optional)**: Use to pipeline filter patterns to control whether or not to include pipeline as part of metadata ingestion. * **Include**: Explicitly include pipeline by adding a list of comma-separated regular expressions to the Include field. OpenMetadata will include all pipeline with names matching one or more of the supplied regular expressions. All other schemas will be excluded. * **Exclude**: Explicitly exclude pipeline by adding a list of comma-separated regular expressions to the Exclude field. OpenMetadata will exclude all pipeline with names matching one or more of the supplied regular expressions. All other schemas will be included. * **Include lineage (toggle)**: Set the Include lineage toggle to control whether to include lineage between pipelines and data sources as part of metadata ingestion. * **Enable Debug Log (toggle)**: Set the Enable Debug Log toggle to set the default log level to debug. * **Mark Deleted Pipelines (toggle)**: Set the Mark Deleted Pipelines toggle to flag pipelines as soft-deleted if they are not present anymore in the source system. #### 8\. Schedule the Ingestion and Deploy Scheduling can be set up at an hourly, daily, weekly, or manual cadence. The timezone is in UTC. Select a Start Date to schedule for ingestion. It is optional to add an End Date. Review your configuration settings. If they match what you intended, click Deploy to create the service and schedule metadata ingestion. If something doesn't look right, click the Back button to return to the appropriate step and change the settings as needed. After configuring the workflow, you can click on Deploy to create the pipeline. ![Schedule the Workflow](https://docs.open-metadata.org/images/v1.11/connectors/schedule.png) Schedule the Ingestion Pipeline and Deploy #### 9\. View the Ingestion Pipeline Once the workflow has been successfully deployed, you can view the Ingestion Pipeline running from the Service Page. ![View Ingestion Pipeline](https://docs.open-metadata.org/images/v1.11/connectors/view-ingestion-pipeline.png) View the Ingestion Pipeline from the Service Page --- # Run the Spline Connector Externally | Official Documentation We use cookies to improve site navigation, analyze site usage, and enhance your user experience. Click "Accept" to enable cookies or "Reject" to reject cookies. AcceptReject connectors No menu items for this category [Home](https://docs.open-metadata.org/latest) /[Connectors](https://docs.open-metadata.org/latest/connectors) /[Pipeline](https://docs.open-metadata.org/latest/connectors/pipeline) /[Spline](https://docs.open-metadata.org/latest/connectors/pipeline/spline) /[Yaml](https://docs.open-metadata.org/latest/connectors/pipeline/spline/yaml) OpenMetadata Documentation ![Spline](https://docs.open-metadata.org/_next/image?url=%2Fimages%2Fconnectors%2Fspline.webp&w=64&q=75) Spline ====== BETA Available In Feature List Pipelines Pipeline Status Usage Owners Tags Lineage In this section, we provide guides and references to use the Spline connector. Configure and schedule Spline metadata and profiler workflows from the OpenMetadata UI: * [Requirements](https://docs.open-metadata.org/latest/connectors/pipeline/spline/yaml#requirements) * [Metadata Ingestion](https://docs.open-metadata.org/latest/connectors/pipeline/spline/yaml#metadata-ingestion) How to Run the Connector Externally ----------------------------------- To run the Ingestion via the UI you'll need to use the OpenMetadata Ingestion Container, which comes shipped with custom Airflow plugins to handle the workflow deployment. If, instead, you want to manage your workflows externally on your preferred orchestrator, you can check the following docs to run the Ingestion Framework **anywhere**. [#### External Schedulers\ \ Get more information about running the Ingestion Framework Externally](https://docs.open-metadata.org/latest/deployment/ingestion) Requirements ------------ The Spline connector support lineage of data source of type `jdbc` or `dbfs` i.e. The spline connector would be able to extract lineage if the data source is either a jdbc connection or the data source is databricks instance. Currently we do not support data source of type aws s3 or any other cloud storage, which also means that the lineage for external tables from databricks will not be extracted. You can refer [this](https://github.com/AbsaOSS/spline-getting-started/tree/main/spline-on-databricks) documentation on how to configure databricks with spline. ### Python Requirements We have support for Python versions 3.9-3.11 To run the Spline ingestion, you will need to install: Metadata Ingestion ------------------ All connectors are defined as JSON Schemas. [Here](https://github.com/open-metadata/OpenMetadata/blob/main/openmetadata-spec/src/main/resources/json/schema/entity/services/connections/pipeline/splineConnection.json) you can find the structure to create a connection to Spline. In order to create and run a Metadata Ingestion workflow, we will follow the steps to create a YAML configuration able to connect to the source, process the Entities if needed, and reach the OpenMetadata server. The workflow is modeled around the following [JSON Schema](https://github.com/open-metadata/OpenMetadata/blob/main/openmetadata-spec/src/main/resources/json/schema/metadataIngestion/workflow.json) ### 1\. Define the YAML Config This is a sample config for Spline: #### Source Configuration - Service Connection **hostPort**: Spline REST Server API Host & Port, OpenMetadata uses Spline REST Server APIs to extract the execution details from spline to generate lineage. This should be specified as a URI string in the format `scheme://hostname:port`. E.g., `http://localhost:8080`, `http://host.docker.internal:8080`. **uiHostPort**: Spline UI Host & Port is an optional field which is used for generating redirection URL from OpenMetadata to Spline Portal. This should be specified as a URI string in the format `scheme://hostname:port`. E.g., `http://localhost:9090`, `http://host.docker.internal:9090`. #### Source Configuration - Source Config The `sourceConfig` is defined [here](https://github.com/open-metadata/OpenMetadata/blob/main/openmetadata-spec/src/main/resources/json/schema/metadataIngestion/pipelineServiceMetadataPipeline.json) : * **dbServiceNames**: Database Service Name for the creation of lineage, if the source supports it. * **includeTags**: Set the 'Include Tags' toggle to control whether to include tags as part of metadata ingestion. * **includeUnDeployedPipelines**: Set the 'Include UnDeployed Pipelines' toggle to control whether to include un-deployed pipelines as part of metadata ingestion. By default it is set to `true` * **markDeletedPipelines**: Set the Mark Deleted Pipelines toggle to flag pipelines as soft-deleted if they are not present anymore in the source system. * **pipelineFilterPattern** and **chartFilterPattern**: Note that the `pipelineFilterPattern` and `chartFilterPattern` both support regex as include or exclude. * **includeOwners**: Set the 'Include Owners' toggle to control whether to include owners to the ingested entity if the owner email matches with a user stored in the OM server as part of metadata ingestion. If the ingested entity already exists and has an owner, the owner will not be overwritten.It supports boolean values either `true` or `false`. * **overrideLineage**: Set the 'Override Lineage' toggle to control whether to override the existing lineage. It supports boolean values either `true` or `false`. * **overrideMetadata**: Set the 'Override Metadata' toggle to control whether to override the existing metadata in the OpenMetadata server with the metadata fetched from the source. If the toggle is set to true, the metadata fetched from the source will override the existing metadata in the OpenMetadata server. If the toggle is set to false, the metadata fetched from the source will not override the existing metadata in the OpenMetadata server. This is applicable for fields like description, tags, owner and displayName. It supports boolean values either `true` or `false`. #### Sink Configuration To send the metadata to OpenMetadata, it needs to be specified as `type: metadata-rest`. #### Workflow Configuration The main property here is the `openMetadataServerConfig`, where you can define the host and security provider of your OpenMetadata installation. **Logger Level** You can specify the `loggerLevel` depending on your needs. If you are trying to troubleshoot an ingestion, running with `DEBUG` will give you far more traces for identifying issues. **JWT Token** JWT tokens will allow your clients to authenticate against the OpenMetadata server. To enable JWT Tokens, you will get more details [here](https://docs.open-metadata.org/latest/deployment/security/enable-jwt-tokens) . You can refer to the JWT Troubleshooting section [link](https://docs.open-metadata.org/latest/deployment/security/jwt-troubleshooting) for any issues in your JWT configuration. **Store Service Connection** If set to `true` (default), we will store the sensitive information either encrypted via the Fernet Key in the database or externally, if you have configured any [Secrets Manager](https://docs.open-metadata.org/latest/deployment/secrets-manager) . If set to `false`, the service will be created, but the service connection information will only be used by the Ingestion Framework at runtime, and won't be sent to the OpenMetadata server. **Store Service Connection** If set to `true` (default), we will store the sensitive information either encrypted via the Fernet Key in the database or externally, if you have configured any [Secrets Manager](https://docs.open-metadata.org/latest/deployment/secrets-manager) . If set to `false`, the service will be created, but the service connection information will only be used by the Ingestion Framework at runtime, and won't be sent to the OpenMetadata server. **SSL Configuration** If you have added SSL to the [OpenMetadata server](https://docs.open-metadata.org/latest/deployment/security/enable-ssl) , then you will need to handle the certificates when running the ingestion too. You can either set `verifySSL` to `ignore`, or have it as `validate`, which will require you to set the `sslConfig.caCertificate` with a local path where your ingestion runs that points to the server certificate file. Find more information on how to troubleshoot SSL issues [here](https://docs.open-metadata.org/latest/deployment/security/enable-ssl/ssl-troubleshooting) . **ingestionPipelineFQN** Fully qualified name of ingestion pipeline, used to identify the current ingestion pipeline. filename.yamlCopy ### 2\. Run with the CLI First, we will need to save the YAML file. Afterward, and with all requirements installed, we can run: Note that from connector to connector, this recipe will always be the same. By updating the YAML configuration, you will be able to extract metadata from different sources. --- # Run the MongoDB Connector Externally We use cookies to improve site navigation, analyze site usage, and enhance your user experience. Click "Accept" to enable cookies or "Reject" to reject cookies. AcceptReject connectors No menu items for this category [Home](https://docs.open-metadata.org/latest) /[Connectors](https://docs.open-metadata.org/latest/connectors) /[Database](https://docs.open-metadata.org/latest/connectors/database) /[Mongodb](https://docs.open-metadata.org/latest/connectors/database/mongodb) /[Yaml](https://docs.open-metadata.org/latest/connectors/database/mongodb/yaml) OpenMetadata Documentation ![MongoDB](https://docs.open-metadata.org/_next/image?url=%2Fimages%2Fconnectors%2Fmongodb.webp&w=64&q=75) MongoDB ======= PROD Available In Feature List Metadata Data Profiler Sample Data Auto-Classification Query Usage Data Quality dbt Owners Lineage Column-level Lineage Tags Stored Procedures In this section, we provide guides and references to use the MongoDB connector. Configure and schedule MongoDB metadata workflows from the OpenMetadata UI: * [Requirements](https://docs.open-metadata.org/latest/connectors/database/mongodb/yaml#requirements) * [Metadata Ingestion](https://docs.open-metadata.org/latest/connectors/database/mongodb/yaml#metadata-ingestion) * [Data Profiler](https://docs.open-metadata.org/latest/connectors/database/mongodb/yaml#data-profiler) Ingestion Deployment -------------------- To run the Ingestion via the UI you'll need to use the OpenMetadata Ingestion Container, which comes shipped with custom Airflow plugins to handle the workflow deployment. If you want to install it manually in an already existing Airflow host, you can follow [this](https://docs.open-metadata.org/latest/deployment/ingestion/openmetadata) guide. If you don't want to use the OpenMetadata Ingestion container to configure the workflows via the UI, then you can check the following docs to run the Ingestion Framework in any orchestrator externally. [#### Run Connectors from the OpenMetadata UI\ \ Learn how to manage your deployment to run connectors from the UI](https://docs.open-metadata.org/latest/deployment/ingestion/openmetadata) [#### Run the Connector Externally\ \ Get the YAML to run the ingestion externally](https://docs.open-metadata.org/latest/connectors/database/mongodb/yaml) [#### External Schedulers\ \ Get more information about running the Ingestion Framework Externally](https://docs.open-metadata.org/latest/deployment/ingestion) How to Run the Connector Externally ----------------------------------- To run the Ingestion via the UI you'll need to use the OpenMetadata Ingestion Container, which comes shipped with custom Airflow plugins to handle the workflow deployment. If, instead, you want to manage your workflows externally on your preferred orchestrator, you can check the following docs to run the Ingestion Framework **anywhere**. [#### External Schedulers\ \ Get more information about running the Ingestion Framework Externally](https://docs.open-metadata.org/latest/deployment/ingestion) Requirements ------------ To fetch the metadata from MongoDB to OpenMetadata, the MongoDB user must have access to perform `find` operation on collection and `listCollection` operations on database available in MongoDB. ### Python Requirements We have support for Python versions 3.9-3.11 To run the MongoDB ingestion, you will need to install: Metadata Ingestion ------------------ All connectors are defined as JSON Schemas. [Here](https://github.com/open-metadata/OpenMetadata/blob/main/openmetadata-spec/src/main/resources/json/schema/entity/services/connections/database/mongoDBConnection.json) you can find the structure to create a connection to MongoDB. In order to create and run a Metadata Ingestion workflow, we will follow the steps to create a YAML configuration able to connect to the source, process the Entities if needed, and reach the OpenMetadata server. The workflow is modeled around the following [JSON Schema](https://github.com/open-metadata/OpenMetadata/blob/main/openmetadata-spec/src/main/resources/json/schema/metadataIngestion/workflow.json) ### 1\. Define the YAML Config This is a sample config for MongoDB: #### Source Configuration - Service Connection **username**: Username to connect to Mongodb. This user must have access to perform `find` operation on collection and `listCollection` operations on database available in MongoDB. **password**: Password to connect to MongoDB. **hostPort**: When using the `mongodb` connection schema, the hostPort parameter specifies the host and port of the MongoDB. This should be specified as a string in the format `hostname:port`. E.g., `localhost:27017`. When using the `mongodb+srv` connection schema, the hostPort parameter specifies the host and port of the MongoDB. This should be specified as a string in the format `hostname`. E.g., `cluster0-abcde.mongodb.net`. Using Atlas? Follow [this guide](https://www.mongodb.com/docs/guides/atlas/connection-string/) to get the connection string. **databaseName**: Optional name to give to the database in OpenMetadata. If left blank, we will use default as the database name. #### Source Configuration - Source Config The `sourceConfig` is defined [here](https://github.com/open-metadata/OpenMetadata/blob/main/openmetadata-spec/src/main/resources/json/schema/metadataIngestion/databaseServiceMetadataPipeline.json) : **markDeletedTables**: To flag tables as soft-deleted if they are not present anymore in the source system. **markDeletedStoredProcedures**: Optional configuration to soft delete stored procedures in OpenMetadata if the source stored procedures are deleted. Also, if the stored procedures is deleted, all the associated entities like lineage, etc., with that stored procedures will be deleted. **markDeletedSchemas**: Optional configuration to soft delete schemas stored in OpenMetadata if the source schema is deleted. Setting this flag to true will only keep filtered schema and delete any other schemas that do not match schemaFilterPattern or do not exist at source. **markDeletedDatabases**: Optional configuration to soft delete databases stored in OpenMetadata if the source database is deleted. Setting this flag to true will only keep filtered databases and delete any other databases that do not match databaseFilterPattern or do not exist at source. **includeTables**: true or false, to ingest table data. Default is true. **includeViews**: true or false, to ingest views definitions. **includeTags**: Optional configuration to toggle the tags ingestion. **includeOwners**: Set the 'Include Owners' toggle to control whether to include owners to the ingested entity if the owner email matches with a user stored in the OM server as part of metadata ingestion. If the ingested entity already exists and has an owner, the owner will not be overwritten. **includeStoredProcedures**: Optional configuration to toggle the Stored Procedures ingestion. **includeDDL**: Optional configuration to toggle the DDL Statements ingestion. **overrideMetadata** _(boolean)_: Set the 'Override Metadata' toggle to control whether to override the existing metadata in the OpenMetadata server with the metadata fetched from the source. If the toggle is set to true, the metadata fetched from the source will override the existing metadata in the OpenMetadata server. If the toggle is set to false, the metadata fetched from the source will not override the existing metadata in the OpenMetadata server. This is applicable for fields like description, tags, owner and displayName. **queryLogDuration**: Configuration to tune how far we want to look back in query logs to process Stored Procedures results. **queryParsingTimeoutLimit**: Configuration to set the timeout for parsing the query in seconds. **useFqnForFiltering**: Regex will be applied on fully qualified name (e.g service\_name.db\_name.schema\_name.table\_name) instead of raw name (e.g. table\_name). **databaseFilterPattern**, **schemaFilterPattern**, **tableFilterPattern**: Note that the filter supports regex as include or exclude. You can find examples [here](https://docs.open-metadata.org/latest/connectors/ingestion/workflows/metadata/filter-patterns/database) **threads (beta)**: The number of threads to use when extracting the metadata using multithreading. Please take a look [here](https://docs.open-metadata.org/latest/connectors/ingestion/workflows/metadata/multithreading) before configuring this. **databaseMetadataConfigType** _(string)_: Database Source Config Metadata Pipeline type. **incremental (beta)**: Incremental Extraction configuration. Currently implemented for: * [BigQuery](https://docs.open-metadata.org/latest/connectors/ingestion/workflows/metadata/incremental-extraction/bigquery) * [Redshift](https://docs.open-metadata.org/latest/connectors/ingestion/workflows/metadata/incremental-extraction/redshift) * [Snowflake](https://docs.open-metadata.org/latest/connectors/ingestion/workflows/metadata/incremental-extraction/snowflake) #### Sink Configuration To send the metadata to OpenMetadata, it needs to be specified as `type: metadata-rest`. #### Workflow Configuration The main property here is the `openMetadataServerConfig`, where you can define the host and security provider of your OpenMetadata installation. **Logger Level** You can specify the `loggerLevel` depending on your needs. If you are trying to troubleshoot an ingestion, running with `DEBUG` will give you far more traces for identifying issues. **JWT Token** JWT tokens will allow your clients to authenticate against the OpenMetadata server. To enable JWT Tokens, you will get more details [here](https://docs.open-metadata.org/latest/deployment/security/enable-jwt-tokens) . You can refer to the JWT Troubleshooting section [link](https://docs.open-metadata.org/latest/deployment/security/jwt-troubleshooting) for any issues in your JWT configuration. **Store Service Connection** If set to `true` (default), we will store the sensitive information either encrypted via the Fernet Key in the database or externally, if you have configured any [Secrets Manager](https://docs.open-metadata.org/latest/deployment/secrets-manager) . If set to `false`, the service will be created, but the service connection information will only be used by the Ingestion Framework at runtime, and won't be sent to the OpenMetadata server. **Store Service Connection** If set to `true` (default), we will store the sensitive information either encrypted via the Fernet Key in the database or externally, if you have configured any [Secrets Manager](https://docs.open-metadata.org/latest/deployment/secrets-manager) . If set to `false`, the service will be created, but the service connection information will only be used by the Ingestion Framework at runtime, and won't be sent to the OpenMetadata server. **SSL Configuration** If you have added SSL to the [OpenMetadata server](https://docs.open-metadata.org/latest/deployment/security/enable-ssl) , then you will need to handle the certificates when running the ingestion too. You can either set `verifySSL` to `ignore`, or have it as `validate`, which will require you to set the `sslConfig.caCertificate` with a local path where your ingestion runs that points to the server certificate file. Find more information on how to troubleshoot SSL issues [here](https://docs.open-metadata.org/latest/deployment/security/enable-ssl/ssl-troubleshooting) . **ingestionPipelineFQN** Fully qualified name of ingestion pipeline, used to identify the current ingestion pipeline. #### Advanced Configuration **Connection Options (Optional)**: Enter the details for any additional connection options that can be sent to database during the connection. These details must be added as Key-Value pairs. filename.yamlCopy ### 2\. Run with the CLI First, we will need to save the YAML file. Afterward, and with all requirements installed, we can run: Note that from connector to connector, this recipe will always be the same. By updating the YAML configuration, you will be able to extract metadata from different sources. Data Profiler ------------- The Data Profiler workflow will be using the `orm-profiler` processor. After running a Metadata Ingestion workflow, we can run Data Profiler workflow. While the `serviceName` will be the same to that was used in Metadata Ingestion, so the ingestion bot can get the `serviceConnection` details from the server. ### Limitations The MongodDB data profiler current supports only the following features: 1. **Row count**: The number of rows in the collection. Sampling or custom query is not supported. 2. **Sample data:** If a custom query is defined it will be used for sample data. ### 1\. Define the YAML Config This is a sample config for the profiler: #### Source Configuration - Source Config You can find all the definitions and types for the `sourceConfig` [here](https://github.com/open-metadata/OpenMetadata/blob/main/openmetadata-spec/src/main/resources/json/schema/metadataIngestion/databaseServiceProfilerPipeline.json) . **generateSampleData**: Option to turn on/off generating sample data. **processPiiSensitive**: Optional configuration to automatically tag columns that might contain sensitive information. **timeoutSeconds**: Profiler Timeout in Seconds **schemaFilterPattern**: Regex to only fetch tables or databases that matches the pattern. **tableFilterPattern**: Regex to only fetch tables or databases that matches the pattern. #### Processor Configuration Choose the `orm-profiler`. Its config can also be updated to define tests from the YAML itself instead of the UI: **tableConfig**: `tableConfig` allows you to set up some configuration at the table level. #### Sink Configuration To send the metadata to OpenMetadata, it needs to be specified as `type: metadata-rest`. #### Workflow Configuration The main property here is the `openMetadataServerConfig`, where you can define the host and security provider of your OpenMetadata installation. For a simple, local installation using our docker containers, this looks like: filename.yamlCopy * You can learn more about how to configure and run the Profiler Workflow to extract Profiler data and execute the Data Quality from [here](https://docs.open-metadata.org/latest/how-to-guides/data-quality-observability/profiler/workflow) ### 2\. Prepare the Profiler DAG Here, we follow a similar approach as with the metadata and usage pipelines, although we will use a different Workflow class: #### Import necessary modules The `ProfilerWorkflow` class that is being imported is a part of a metadata orm\_profiler framework, which defines a process of extracting Profiler data. Here we are also importing all the basic requirements to parse YAMLs, handle dates and build our DAG. **Default arguments for all tasks in the Airflow DAG.** * Default arguments dictionary contains default arguments for tasks in the DAG, including the owner's name, email address, number of retries, retry delay, and execution timeout. * **config**: Specifies config for the profiler as we prepare above. * **metadata\_ingestion\_workflow()**: This code defines a function `metadata_ingestion_workflow()` that loads a YAML configuration, creates a `ProfilerWorkflow` object, executes the workflow, checks its status, prints the status to the console, and stops the workflow. * **DAG**: creates a DAG using the Airflow framework, and tune the DAG configurations to whatever fits with your requirements * For more Airflow DAGs creation details visit [here](https://airflow.apache.org/docs/apache-airflow/stable/core-concepts/dags.html#declaring-a-dag) . filename.pyCopy dbt Integration --------------- [#### dbt Integration\ \ Learn more about how to ingest dbt models' definitions and their lineage.](https://docs.open-metadata.org/latest/connectors/ingestion/workflows/dbt) --- # Run the SAP ERP Connector Externally We use cookies to improve site navigation, analyze site usage, and enhance your user experience. Click "Accept" to enable cookies or "Reject" to reject cookies. AcceptReject connectors No menu items for this category [Home](https://docs.open-metadata.org/latest) /[Connectors](https://docs.open-metadata.org/latest/connectors) /[Database](https://docs.open-metadata.org/latest/connectors/database) /[Sap Erp](https://docs.open-metadata.org/latest/connectors/database/sap-erp) /[Yaml](https://docs.open-metadata.org/latest/connectors/database/sap-erp/yaml) OpenMetadata Documentation ![SAP ERP](https://docs.open-metadata.org/_next/image?url=%2Fimages%2Fconnectors%2Fsap-erp.webp&w=64&q=75) SAP ERP ======= PROD Available In Feature List Metadata Query Usage Stored Procedures Owners Tags Data Profiler Data Quality View Lineage View Column-level Lineage dbt Sample Data Auto-Classification In this section, we provide guides and references to use the SAP ERP connector. Configure and schedule SAP ERP metadata workflow externally: * [Requirements](https://docs.open-metadata.org/latest/connectors/database/sap-erp/yaml#requirements) * [Metadata Ingestion](https://docs.open-metadata.org/latest/connectors/database/sap-erp/yaml#metadata-ingestion) * [dbt Integration](https://docs.open-metadata.org/latest/connectors/database/sap-erp/yaml#dbt-integration) How to Run the Connector Externally ----------------------------------- To run the Ingestion via the UI you'll need to use the OpenMetadata Ingestion Container, which comes shipped with custom Airflow plugins to handle the workflow deployment. If, instead, you want to manage your workflows externally on your preferred orchestrator, you can check the following docs to run the Ingestion Framework **anywhere**. [#### External Schedulers\ \ Get more information about running the Ingestion Framework Externally](https://docs.open-metadata.org/latest/deployment/ingestion) Requirements ------------ [OpenMetadata 1.1 or later\ \ To deploy OpenMetadata, check the Deployment guides.](https://docs.open-metadata.org/latest/deployment) To ingest the SAP ERP metadata, CDS Views and OData services need to be setup to efficiently expose SAP data. To achieve this, data must be exposed via RESTful interfaces. Follow the guide [here](https://docs.open-metadata.org/latest/connectors/database/sap-erp/setup-sap-apis) to setup the APIs. ### Python Requirements We have support for Python versions 3.9-3.11 To run the SAP ERP ingestion, you will need to install: Metadata Ingestion ------------------ All connectors are defined as JSON Schemas. [Here](https://github.com/open-metadata/OpenMetadata/blob/main/openmetadata-spec/src/main/resources/json/schema/entity/services/connections/database/sapErpConnection.json) you can find the structure to create a connection to SAP ERP. In order to create and run a Metadata Ingestion workflow, we will follow the steps to create a YAML configuration able to connect to the source, process the Entities if needed, and reach the OpenMetadata server. The workflow is modeled around the following [JSON Schema](https://github.com/open-metadata/OpenMetadata/blob/main/openmetadata-spec/src/main/resources/json/schema/metadataIngestion/workflow.json) ### 1\. Define the YAML Config This is a sample config for SAP ERP: #### Source Configuration - Service Connection **hostPort**: Host and port of the SAP ERP service. This specifies the host and port of the SAP ERP instance. It should be specified as a string in the format `https://hostname.com`. **apiKey**: Api Key to authenticate the SAP ERP Apis **databaseName**: In OpenMetadata, the Database Service hierarchy works as follows: `Database Service > Database > Schema > Table` In the case of SAP ERP, we won't have a Database as such. If you'd like to see your data in a database named something other than `default`, you can specify the name in this field. **databaseSchema**: In OpenMetadata, the Database Service hierarchy works as follows: `Database Service > Database > Schema > Table` In the case of SAP ERP, we won't have a Database Schema as such. If you'd like to see your data in a database schema named something other than `default`, you can specify the name in this field. **paginationLimit**: Pagination limit used while querying the SAP ERP API for fetching the entities. #### Source Configuration - Source Config The `sourceConfig` is defined [here](https://github.com/open-metadata/OpenMetadata/blob/main/openmetadata-spec/src/main/resources/json/schema/metadataIngestion/databaseServiceMetadataPipeline.json) : **markDeletedTables**: To flag tables as soft-deleted if they are not present anymore in the source system. **markDeletedStoredProcedures**: Optional configuration to soft delete stored procedures in OpenMetadata if the source stored procedures are deleted. Also, if the stored procedures is deleted, all the associated entities like lineage, etc., with that stored procedures will be deleted. **markDeletedSchemas**: Optional configuration to soft delete schemas stored in OpenMetadata if the source schema is deleted. Setting this flag to true will only keep filtered schema and delete any other schemas that do not match schemaFilterPattern or do not exist at source. **markDeletedDatabases**: Optional configuration to soft delete databases stored in OpenMetadata if the source database is deleted. Setting this flag to true will only keep filtered databases and delete any other databases that do not match databaseFilterPattern or do not exist at source. **includeTables**: true or false, to ingest table data. Default is true. **includeViews**: true or false, to ingest views definitions. **includeTags**: Optional configuration to toggle the tags ingestion. **includeOwners**: Set the 'Include Owners' toggle to control whether to include owners to the ingested entity if the owner email matches with a user stored in the OM server as part of metadata ingestion. If the ingested entity already exists and has an owner, the owner will not be overwritten. **includeStoredProcedures**: Optional configuration to toggle the Stored Procedures ingestion. **includeDDL**: Optional configuration to toggle the DDL Statements ingestion. **overrideMetadata** _(boolean)_: Set the 'Override Metadata' toggle to control whether to override the existing metadata in the OpenMetadata server with the metadata fetched from the source. If the toggle is set to true, the metadata fetched from the source will override the existing metadata in the OpenMetadata server. If the toggle is set to false, the metadata fetched from the source will not override the existing metadata in the OpenMetadata server. This is applicable for fields like description, tags, owner and displayName. **queryLogDuration**: Configuration to tune how far we want to look back in query logs to process Stored Procedures results. **queryParsingTimeoutLimit**: Configuration to set the timeout for parsing the query in seconds. **useFqnForFiltering**: Regex will be applied on fully qualified name (e.g service\_name.db\_name.schema\_name.table\_name) instead of raw name (e.g. table\_name). **databaseFilterPattern**, **schemaFilterPattern**, **tableFilterPattern**: Note that the filter supports regex as include or exclude. You can find examples [here](https://docs.open-metadata.org/latest/connectors/ingestion/workflows/metadata/filter-patterns/database) **threads (beta)**: The number of threads to use when extracting the metadata using multithreading. Please take a look [here](https://docs.open-metadata.org/latest/connectors/ingestion/workflows/metadata/multithreading) before configuring this. **databaseMetadataConfigType** _(string)_: Database Source Config Metadata Pipeline type. **incremental (beta)**: Incremental Extraction configuration. Currently implemented for: * [BigQuery](https://docs.open-metadata.org/latest/connectors/ingestion/workflows/metadata/incremental-extraction/bigquery) * [Redshift](https://docs.open-metadata.org/latest/connectors/ingestion/workflows/metadata/incremental-extraction/redshift) * [Snowflake](https://docs.open-metadata.org/latest/connectors/ingestion/workflows/metadata/incremental-extraction/snowflake) #### Sink Configuration To send the metadata to OpenMetadata, it needs to be specified as `type: metadata-rest`. #### Workflow Configuration The main property here is the `openMetadataServerConfig`, where you can define the host and security provider of your OpenMetadata installation. **Logger Level** You can specify the `loggerLevel` depending on your needs. If you are trying to troubleshoot an ingestion, running with `DEBUG` will give you far more traces for identifying issues. **JWT Token** JWT tokens will allow your clients to authenticate against the OpenMetadata server. To enable JWT Tokens, you will get more details [here](https://docs.open-metadata.org/latest/deployment/security/enable-jwt-tokens) . You can refer to the JWT Troubleshooting section [link](https://docs.open-metadata.org/latest/deployment/security/jwt-troubleshooting) for any issues in your JWT configuration. **Store Service Connection** If set to `true` (default), we will store the sensitive information either encrypted via the Fernet Key in the database or externally, if you have configured any [Secrets Manager](https://docs.open-metadata.org/latest/deployment/secrets-manager) . If set to `false`, the service will be created, but the service connection information will only be used by the Ingestion Framework at runtime, and won't be sent to the OpenMetadata server. **Store Service Connection** If set to `true` (default), we will store the sensitive information either encrypted via the Fernet Key in the database or externally, if you have configured any [Secrets Manager](https://docs.open-metadata.org/latest/deployment/secrets-manager) . If set to `false`, the service will be created, but the service connection information will only be used by the Ingestion Framework at runtime, and won't be sent to the OpenMetadata server. **SSL Configuration** If you have added SSL to the [OpenMetadata server](https://docs.open-metadata.org/latest/deployment/security/enable-ssl) , then you will need to handle the certificates when running the ingestion too. You can either set `verifySSL` to `ignore`, or have it as `validate`, which will require you to set the `sslConfig.caCertificate` with a local path where your ingestion runs that points to the server certificate file. Find more information on how to troubleshoot SSL issues [here](https://docs.open-metadata.org/latest/deployment/security/enable-ssl/ssl-troubleshooting) . **ingestionPipelineFQN** Fully qualified name of ingestion pipeline, used to identify the current ingestion pipeline. #### Advanced Configuration **Connection Options (Optional)**: Enter the details for any additional connection options that can be sent to database during the connection. These details must be added as Key-Value pairs. **Connection Arguments (Optional)**: Enter the details for any additional connection arguments such as security or protocol configs that can be sent to database during the connection. These details must be added as Key-Value pairs. * In case you are using Single-Sign-On (SSO) for authentication, add the `authenticator` details in the Connection Arguments as a Key-Value pair as follows: `"authenticator" : "sso_login_url"` filename.yamlCopy ### 2\. Run with the CLI First, we will need to save the YAML file. Afterward, and with all requirements installed, we can run: Note that from connector to connector, this recipe will always be the same. By updating the YAML configuration, you will be able to extract metadata from different sources. Securing SAP ERP Connection with SSL in OpenMetadata ---------------------------------------------------- To configure SSL for secure connections between OpenMetadata and a Redshift database, Redshift offers various SSL modes, each providing different levels of connection security. When running the ingestion process externally, specify the SSL mode to be used for the Redshift connection, such as `prefer`, `verify-ca`, `allow`, and others. Once you've chosen the SSL mode, provide the CA certificate for SSL validation (`caCertificate`). Only the CA certificate is required for SSL validation in Redshift. dbt Integration --------------- [#### dbt Integration\ \ Learn more about how to ingest dbt models' definitions and their lineage.](https://docs.open-metadata.org/latest/connectors/ingestion/workflows/dbt) --- # Druid Troubleshooting Guide | OpenMetadata Support We use cookies to improve site navigation, analyze site usage, and enhance your user experience. Click "Accept" to enable cookies or "Reject" to reject cookies. AcceptReject connectors No menu items for this category [Home](https://docs.open-metadata.org/latest) /[Connectors](https://docs.open-metadata.org/latest/connectors) /[Database](https://docs.open-metadata.org/latest/connectors/database) /[Druid](https://docs.open-metadata.org/latest/connectors/database/druid) /[Troubleshooting](https://docs.open-metadata.org/latest/connectors/database/druid/troubleshooting) OpenMetadata Documentation Troubleshooting =============== Workflow Deployment Error ------------------------- If there were any errors during the workflow deployment process, the Ingestion Pipeline Entity will still be created, but no workflow will be present in the Ingestion container. * You can then Edit the Ingestion Pipeline and **Deploy** it again. * From the Connection tab, you can also Edit the Service if needed. Connector Debug Troubleshooting ------------------------------- This section provides instructions to help resolve common issues encountered during connector setup and metadata ingestion in OpenMetadata. Below are some of the most frequently observed troubleshooting scenarios. How to Enable Debug Logging for Any Ingestion --------------------------------------------- To enable debug logging for any ingestion workflow in OpenMetadata: 1. **Navigate to Services** Go to **Settings > Services > Service Type** (e.g., Database) in the OpenMetadata UI. 2. **Select a Service** Choose the specific service for which you want to enable debug logging. ![Select a Service](https://docs.open-metadata.org/images/v1.11/connectors/debug/debug1.png) Select a Service 3. **Access Ingestion Tab** Go to the **Ingestion tab** and click the three-dot menu on the right-hand side of the ingestion type, and select Edit. ![Access Agents Tab](https://docs.open-metadata.org/images/v1.11/connectors/debug/debug2.png) Access Agents Tab 4. **Enable Debug Logging** In the configuration dialog, enable the **Debug Log** option and click **Next**. ![Enable Debug Logging](https://docs.open-metadata.org/images/v1.11/connectors/debug/debug3.png) Enable Debug Logging 5. **Schedule and Submit** Configure the schedule if needed and click **Submit** to apply the changes. ![Schedule and Submit](https://docs.open-metadata.org/images/v1.11/connectors/debug/debug4.png) Schedule and Submit Permission Issues ----------------- If you encounter permission-related errors during connector setup or metadata ingestion, ensure that all the prerequisites and access configurations specified for each connector are properly implemented. Refer to the connector-specific documentation to verify the required permissions. --- # How to Deploy a Lineage Workflow We use cookies to improve site navigation, analyze site usage, and enhance your user experience. Click "Accept" to enable cookies or "Reject" to reject cookies. AcceptReject ![](https://static.scarf.sh/a.png?x-pxid=6814879a-96c0-40bd-be69-ddfeb14694aa) how-to-guides No menu items for this category [Home](https://docs.open-metadata.org/latest) /[How To Guides](https://docs.open-metadata.org/latest/how-to-guides) /[Data Lineage](https://docs.open-metadata.org/latest/how-to-guides/data-lineage) /[Workflow](https://docs.open-metadata.org/latest/how-to-guides/data-lineage/workflow) OpenMetadata Documentation How to Deploy a Lineage Workflow ================================ Lineage data can be ingested from your data sources right from the OpenMetadata UI. Currently, the lineage workflow is supported for a limited set of connectors, like [BigQuery](https://docs.open-metadata.org/latest/connectors/database/bigquery) , [Snowflake](https://docs.open-metadata.org/latest/connectors/database/snowflake) , [MSSQL](https://docs.open-metadata.org/latest/connectors/database/mssql) , [Redshift](https://docs.open-metadata.org/latest/connectors/database/redshift) , [Clickhouse](https://docs.open-metadata.org/latest/connectors/database/clickhouse) , [PostgreSQL](https://docs.open-metadata.org/latest/connectors/database/postgres) , [Databricks](https://docs.open-metadata.org/latest/connectors/database/databricks) . **Tip:** Trace the upstream and downstream dependencies with Lineage. View Lineage from Metadata Ingestion ------------------------------------ Once the metadata ingestion runs correctly, and we are able to explore the service Entities, we can add the view lineage information for the data assets. This will populate the Lineage tab in the data asset page. During the Metadata Ingestion workflow we differentiate if a Table is a View. For those sources, where we can obtain the query that generates the View, we bring in the view lineage along with the metadata. After all Tables have been ingested in the workflow, it's time to parse all the queries generating Views. During the query parsing, we will obtain the source and target tables, search if the Tables exist in OpenMetadata, and finally create the lineage relationship between the involved Entities. If the database has views, then the view lineage would be generated automatically, along with the column-level lineage. In such a case, the table type is **View** as shown in the example below. ![View Lineage through Metadata Ingestion](https://docs.open-metadata.org/images/v1.11/how-to-guides/lineage/view.png) View Lineage through Metadata Ingestion Lineage Agent from UI --------------------- Apart from the Metadata ingestion, we can create a workflow that will obtain the query log and table creation information from the underlying database and feed it to OpenMetadata. The Lineage Agent will be in charge of obtaining this data. The metadata ingestion will only bring in the View lineage queries, whereas the Lineage Agent workflow will be bring in all those queries that can be used to generate lineage information. ### 1\. Add a Lineage Agent Navigate to **Settings >> Services >> Databases**. Select the required service ![Select a Service](https://docs.open-metadata.org/images/v1.11/how-to-guides/lineage/wkf1.png) Select a Service ![Click on Databases](https://docs.open-metadata.org/images/v1.11/how-to-guides/lineage/wkf1.1.png) Click on Databases ![Select the Database](https://docs.open-metadata.org/images/v1.11/how-to-guides/lineage/wkf1.2.png) Select the Database Go the the **Ingestions** tab. Click on **Add Ingestion** and select **Add Lineage Agent**. ![Add a Lineage Agent](https://docs.open-metadata.org/images/v1.11/how-to-guides/lineage/wkf2.png) Add a Lineage Agent ### 2\. Configure the Lineage Agent Here you can enter the Lineage Agent details: ![Configure the Lineage Agent](https://docs.open-metadata.org/images/v1.11/how-to-guides/lineage/wkf3.png) Configure the Lineage Agent ### Lineage Options **Query Log Duration:** Specify the duration in days for which the profiler should capture lineage data from the query logs. For example, if you specify 2 as the value for the duration, the data profiler will capture lineage information for 2 **days** or 48 hours prior to when the ingestion workflow is run. **Parsing Timeout Limit:** Specify the timeout limit for parsing the sql queries to perform the lineage analysis. This must be specified in **seconds**. **Result Limit:** Set the limit for the query log results to be run at a time. This is the **number of rows**. **Filter Condition:** We execute a query on query history table of the respective data source to perform the query analysis and extract the lineage and usage information. This field will be useful when you want to restrict some queries from being part of this analysis. In this field you can specify a sql condition that will be applied on the query history result set. You can check more about [Usage Query Filtering here](https://docs.open-metadata.org/latest/connectors/ingestion/workflows/usage/filter-query-set) . ### 3\. Schedule and Deploy After clicking Next, you will be redirected to the Scheduling form. This will be the same as the Metadata Ingestion. Select your desired schedule and click on Deploy to find the lineage pipeline being added to the Service Ingestions. ![Schedule and Deploy the Lineage Agent](https://docs.open-metadata.org/images/v1.11/how-to-guides/lineage/wkf4.png) Schedule and Deploy the Lineage Agent Run Lineage Workflow Externally ------------------------------- Lineage ------- After running a Metadata Ingestion workflow, we can run Lineage workflow. While the `serviceName` will be the same to that was used in Metadata Ingestion, so the ingestion bot can get the `serviceConnection` details from the server. ### 1\. Define the YAML Config This is a sample config for BigQuery Lineage: #### Source Configuration - Source Config You can find all the definitions and types for the `sourceConfig` [here](https://github.com/open-metadata/OpenMetadata/blob/main/openmetadata-spec/src/main/resources/json/schema/metadataIngestion/databaseServiceQueryLineagePipeline.json) . **queryLogDuration**: Configuration to tune how far we want to look back in query logs to process lineage data in days. **parsingTimeoutLimit**: Configuration to set the timeout for parsing the query in seconds. **filterCondition**: Condition to filter the query history. **resultLimit**: Configuration to set the limit for query logs. **queryLogFilePath**: Configuration to set the file path for query logs. **databaseFilterPattern**: Regex to only fetch databases that matches the pattern. **schemaFilterPattern**: Regex to only fetch tables or databases that matches the pattern. **tableFilterPattern**: Regex to only fetch tables or databases that matches the pattern. **overrideViewLineage**: Set the 'Override View Lineage' toggle to control whether to override the existing view lineage. **processViewLineage**: Set the 'Process View Lineage' toggle to control whether to process view lineage. **processQueryLineage**: Set the 'Process Query Lineage' toggle to control whether to process query lineage. **processStoredProcedureLineage**: Set the 'Process Stored ProcedureLog Lineage' toggle to control whether to process stored procedure lineage. **threads**: Number of Threads to use in order to parallelize lineage ingestion. #### Sink Configuration To send the metadata to OpenMetadata, it needs to be specified as `type: metadata-rest`. #### Workflow Configuration The main property here is the `openMetadataServerConfig`, where you can define the host and security provider of your OpenMetadata installation. For a simple, local installation using our docker containers, this looks like: filename.yamlCopy * You can learn more about how to configure and run the Lineage Workflow to extract Lineage data from [here](https://docs.open-metadata.org/latest/connectors/ingestion/workflows/lineage) ### 2\. Run with the CLI After saving the YAML config, we will run the command the same way we did for the metadata ingestion: dbt Ingestion ------------- We can also generate lineage through [dbt ingestion](https://docs.open-metadata.org/latest/connectors/ingestion/workflows/dbt/configure-dbt-workflow-from-ui) . The dbt workflow can fetch queries that carry lineage information. For a dbt ingestion pipeline, the path to the Catalog and Manifest files must be specified. We also fetch the column level lineage through dbt. You can learn more about [lineage ingestion here](https://docs.open-metadata.org/latest/connectors/ingestion/lineage) . Query Logs using CSV File ------------------------- Lineage ingestion is supported for a few connectors as mentioned earlier. For the unsupported connectors, you can set up [Lineage Workflows using Query Logs](https://docs.open-metadata.org/latest/connectors/ingestion/workflows/lineage/lineage-workflow-query-logs) using a CSV file. Manual Lineage -------------- Lineage can also be added and edited manually in OpenMetadata. Refer for more information on [adding lineage manually](https://docs.open-metadata.org/latest/how-to-guides/data-lineage/manual) . [Explore the Lineage View\ \ Explore the rich lineage view in OpenMetadata.](https://docs.open-metadata.org/latest/how-to-guides/data-lineage/explore) --- # Run the Ingestion Framework Externally We use cookies to improve site navigation, analyze site usage, and enhance your user experience. Click "Accept" to enable cookies or "Reject" to reject cookies. AcceptReject deployment No menu items for this category [Home](https://docs.open-metadata.org/latest) /[Deployment](https://docs.open-metadata.org/latest/deployment) /[Ingestion](https://docs.open-metadata.org/latest/deployment/ingestion) /[External](https://docs.open-metadata.org/latest/deployment/ingestion/external) OpenMetadata Documentation Ingestion Framework External Deployment ======================================= Any tool capable of running Python code can be used to configure the metadata extraction from your sources. We have support for Python versions 3.9-3.11 1\. How does the Ingestion Framework work? ------------------------------------------ The Ingestion Framework contains all the logic about how to connect to the sources, extract their metadata and send it to the OpenMetadata server. We have built it from scratch with the main idea of making it an independent component that can be run from - **literally** - anywhere. In order to install it, you just need to get it from [PyPI](https://pypi.org/project/openmetadata-ingestion/) . We will show further examples later, but a piece of code is the best showcase for its simplicity. In order to run a full ingestion process, you just need to execute a single function. For example, if we wanted to run the metadata ingestion from within a simple Python script: Where this function runs is completely up to you, and you can adapt it to what makes the most sense within your organization and engineering context. Below you'll see some examples of different orchestrators you can leverage to execute the ingestion process. 2\. Ingestion Configuration --------------------------- In the example above, the `Workflow` class got created from a YAML configuration. Any Workflow that you execute (ingestion, profiler, lineage,...) will have its own YAML representation. You can think about this configuration as the recipe you want to execute: where is your source, which pieces do you extract, how are they processed and where are they sent. An example YAML config for extracting MySQL metadata looks like this: You will find examples of all the workflow's YAML files at each Connector [page](https://docs.open-metadata.org/latest/connectors) . We will now show you examples on how to configure and run every workflow externally by using Snowflake as an example. But first, let's digest some information that will be common everywhere, the `workflowConfig`. ### Workflow Config Here you will define information such as where are you hosting the OpenMetadata server, and the JWT token to authenticate. Review this section carefully to ensure you are properly managing service credentials and other security configurations. **Logger Level** You can specify the `loggerLevel` depending on your needs. If you are trying to troubleshoot an ingestion, running with `DEBUG` will give you far more traces for identifying issues. **JWT Token** JWT tokens will allow your clients to authenticate against the OpenMetadata server. To enable JWT Tokens, you will get more details [here](https://docs.open-metadata.org/latest/deployment/security/enable-jwt-tokens) . You can refer to the JWT Troubleshooting section [link](https://docs.open-metadata.org/latest/deployment/security/jwt-troubleshooting) for any issues in your JWT configuration. **Store Service Connection** If set to `true` (default), we will store the sensitive information either encrypted via the Fernet Key in the database or externally, if you have configured any [Secrets Manager](https://docs.open-metadata.org/latest/deployment/secrets-manager) . If set to `false`, the service will be created, but the service connection information will only be used by the Ingestion Framework at runtime, and won't be sent to the OpenMetadata server. **Secrets Manager Configuration** If you have configured any [Secrets Manager](https://docs.open-metadata.org/latest/deployment/secrets-manager) , you need to let the Ingestion Framework know how to retrieve the credentials securely. Follow the [docs](https://docs.open-metadata.org/latest/deployment/secrets-manager) to configure the secret retrieval based on your environment. **SSL Configuration** If you have added SSL to the [OpenMetadata server](https://docs.open-metadata.org/latest/deployment/security/enable-ssl) , then you will need to handle the certificates when running the ingestion too. You can either set `verifySSL` to `ignore`, or have it as `validate`, which will require you to set the `sslConfig.caCertificate` with a local path where your ingestion runs that points to the server certificate file. Find more information on how to troubleshoot SSL issues [here](https://docs.open-metadata.org/latest/deployment/security/enable-ssl/ssl-troubleshooting) . #### JWT Token with Secrets Manager If you are using the [Secrets Manager](https://docs.open-metadata.org/latest/deployment/secrets-manager) , you can let the Ingestion client to pick up the JWT Token dynamically from the Secrets Manager at runtime. Let's show an example: We have an OpenMetadata server running with the `managed-aws` Secrets Manager. Since we used the `OPENMETADATA_CLUSTER_NAME` env var as `test`, our `ingestion-bot` JWT Token is safely stored under the secret ID `/test/bot/ingestion-bot/config/jwttoken`. Now, we can use the following workflow config to run the ingestion without having to pass the token, but just pointing to the secret itself: Notice how: 1. We specify the `secretsManagerProvider` pointing to `aws`, since that's the manager we are using. 2. We set `secretsManagerLoader` as `env`. Since we're running this from our local, we'll let the AWS credentials to be loaded from the local env vars. (When running this using the UI, note that the generated workflows will have this value set as `airflow`!) 3. We set the `jwtToken` value as `secret:/test/bot/ingestion-bot/config/jwttoken`, which tells the client that this value is a `secret` located under `/test/bot/ingestion-bot/config/jwttoken`. Those are our env vars: And we can run this normally with `metadata ingest -c `. Note that **even if you are not using the Secrets Manager for the OpenMetadata Server**, you can still apply the same approach by storing the JWT token manually to the secrets manager, and let the Ingestion client pick it up from there automatically. 3\. (Optional) Ingestion Pipeline --------------------------------- Additionally, if you want to see your runs logged in the `Ingestions` tab of the connectors page in the UI as you would when running the connectors natively with OpenMetadata, you can add the following configuration on your YAMLs: Adding the `ingestionPipelineFQN` - the Ingestion Pipeline Fully Qualified Name - will tell the Ingestion Framework to log the executions and update the ingestion status, which will appear on the UI. Note that the action buttons will be disabled, since OpenMetadata won't be able to interact with external systems. 4\. (Optional) Disable the Pipeline Service Client -------------------------------------------------- If you want to run your workflows **ONLY externally** without relying on OpenMetadata for any workflow management or scheduling, you can update the following server configuration: by setting `enabled: false` or setting the `PIPELINE_SERVICE_CLIENT_ENABLED=false` as an environment variable. This will stop certain APIs and monitors related to the Pipeline Service Client (e.g., Airflow) from being operative. Examples -------- This is not an exhaustive list, and it will keep growing over time. Not because the orchestrators X or Y are not supported, but just because we did not have the time yet to add it here. If you'd like to chip in and help us expand these guides and examples, don't hesitate to reach to us in [Slack](https://slack.open-metadata.org/) or directly open a PR in [GitHub](https://github.com/open-metadata/docs-v1/tree/main/content) . [Airflow\ \ Run the ingestion process externally from Airflow](https://docs.open-metadata.org/latest/deployment/ingestion/external/airflow) [MWAA\ \ Run the ingestion process externally using AWS MWAA](https://docs.open-metadata.org/latest/deployment/ingestion/external/mwaa) [GCP Composer\ \ Run the ingestion process externally from GCP Composer](https://docs.open-metadata.org/latest/deployment/ingestion/external/gcp-composer) [GitHub Actions\ \ Run the ingestion process externally from GitHub Actions](https://docs.open-metadata.org/latest/deployment/ingestion/external/github-actions) Let's jump now into some examples on how you could create the function the run the different workflows. Note that this code can then be executed inside a DAG, a GitHub action, or a vanilla Python script. It will work for any environment. ### Testing You can easily test every YAML configuration using the `metadata` CLI from the Ingestion Framework. In order to install it, you just need to get it from [PyPI](https://pypi.org/project/openmetadata-ingestion/) . In each of the examples below, we'll showcase how to run the CLI, assuming you have a YAML file that contains the workflow configuration. ### Metadata Workflow This is the first workflow you have to configure and run. It will take care of fetching the metadata from your sources, be it Database Services, Dashboard Services, Pipelines, etc. The rest of the workflows (Lineage, Profiler,...) will be executed on top of the metadata already available in the platform. **Adding the imports** The first step is to import the `MetadataWorkflow` class, which will take care of the full ingestion logic. We'll add the import for printing the results at the end. **Defining the YAML** Then, we need to pass the YAML configuration. For this simple example we are defining a variable, but you can read from a file, parse secrets from your environment, or any other approach you'd need. In the end, it's just Python code. You can find complete YAMLs in each connector [docs](https://docs.open-metadata.org/latest/connectors) and find more information about the available configurations. **Preparing the Workflow** Finally, we'll prepare a function that we can execute anywhere. It will take care of instantiating the workflow, executing it and giving us the results. ingestion.pyCopy You can test the workflow via `metadata ingest -c `. ### Lineage Workflow This workflow will take care of scanning your query history and defining lineage relationships between your tables. You can find more information about this workflow [here](https://docs.open-metadata.org/latest/connectors/ingestion/lineage) . **Adding the imports** The first step is to import the `MetadataWorkflow` class, which will take care of the full ingestion logic. We'll add the import for printing the results at the end. Note that we are using the same class as in the Metadata Ingestion. **Defining the YAML** Then, we need to pass the YAML configuration. For this simple example we are defining a variable, but you can read from a file, parse secrets from your environment, or any other approach you'd need. Note how we have not added here the `serviceConnection`. Since the service would have been created during the metadata ingestion, we can let the Ingestion Framework dynamically fetch the Service Connection information. If, however, you are configuring the workflow with `storeServiceConnection: false`, you'll need to explicitly define the `serviceConnection`. You can find complete YAMLs in each connector [docs](https://docs.open-metadata.org/latest/connectors) and find more information about the available configurations. **Preparing the Workflow** Finally, we'll prepare a function that we can execute anywhere. It will take care of instantiating the workflow, executing it and giving us the results. ingestion.pyCopy You can test the workflow via `metadata ingest -c `. ### Usage Workflow As with the lineage workflow, we'll scan the query history for any DML statements. The goal is to ingest queries into the platform, figure out the relevancy of your assets and frequently joined tables. **Adding the imports** The first step is to import the `UsageWorkflow` class, which will take care of the full ingestion logic. We'll add the import for printing the results at the end. **Defining the YAML** Then, we need to pass the YAML configuration. For this simple example we are defining a variable, but you can read from a file, parse secrets from your environment, or any other approach you'd need. Note how we have not added here the `serviceConnection`. Since the service would have been created during the metadata ingestion, we can let the Ingestion Framework dynamically fetch the Service Connection information. If, however, you are configuring the workflow with `storeServiceConnection: false`, you'll need to explicitly define the `serviceConnection`. You can find complete YAMLs in each connector [docs](https://docs.open-metadata.org/latest/connectors) and find more information about the available configurations. **Preparing the Workflow** Finally, we'll prepare a function that we can execute anywhere. It will take care of instantiating the workflow, executing it and giving us the results. ingestion.pyCopy You can test the workflow via `metadata usage -c `. ### Profiler Workflow This workflow will execute queries against your database and send the results into OpenMetadata. The goal is to compute metrics about your data and give you a high-level view of its shape, together with the sample data. This is an interesting previous step before creating Data Quality Workflows. You can find more information about this workflow [here](https://docs.open-metadata.org/latest/how-to-guides/data-quality-observability/profiler/workflow) . **Adding the imports** The first step is to import the `ProfilerWorkflow` class, which will take care of the full ingestion logic. We'll add the import for printing the results at the end. **Defining the YAML** Then, we need to pass the YAML configuration. For this simple example we are defining a variable, but you can read from a file, parse secrets from your environment, or any other approach you'd need. Note how we have not added here the `serviceConnection`. Since the service would have been created during the metadata ingestion, we can let the Ingestion Framework dynamically fetch the Service Connection information. If, however, you are configuring the workflow with `storeServiceConnection: false`, you'll need to explicitly define the `serviceConnection`. You can find complete YAMLs in each connector [docs](https://docs.open-metadata.org/latest/connectors) and find more information about the available configurations. **Preparing the Workflow** Finally, we'll prepare a function that we can execute anywhere. It will take care of instantiating the workflow, executing it and giving us the results. ingestion.pyCopy You can test the workflow via `metadata profile -c `. ### Data Quality Workflow This workflow will execute queries against your database and send the results into OpenMetadata. The goal is to compute metrics about your data and give you a high-level view of its shape, together with the sample data. This is an interesting previous step before creating Data Quality Workflows. You can find more information about this workflow [here](https://docs.open-metadata.org/latest/how-to-guides/data-quality-observability/quality/configure) . **Adding the imports** The first step is to import the `TestSuiteWorkflow` class, which will take care of the full ingestion logic. We'll add the import for printing the results at the end. **Defining the YAML** Then, we need to pass the YAML configuration. For this simple example we are defining a variable, but you can read from a file, parse secrets from your environment, or any other approach you'd need. Note how we have not added here the `serviceConnection`. Since the service would have been created during the metadata ingestion, we can let the Ingestion Framework dynamically fetch the Service Connection information. If, however, you are configuring the workflow with `storeServiceConnection: false`, you'll need to explicitly define the `serviceConnection`. Moreover, see how we are not configuring any tests in the `processor`. You can do [that](https://docs.open-metadata.org/latest/how-to-guides/data-quality-observability/quality/configure#full-yaml-config-example) , but even if nothing gets defined in the YAML, we will execute all the tests configured against the table. You can find complete YAMLs in each connector [docs](https://docs.open-metadata.org/latest/connectors) and find more information about the available configurations. **Preparing the Workflow** Finally, we'll prepare a function that we can execute anywhere. It will take care of instantiating the workflow, executing it and giving us the results. ingestion.pyCopy You can test the workflow via `metadata test -c `. --- # PinotDB Troubleshooting Guide | OpenMetadata Support We use cookies to improve site navigation, analyze site usage, and enhance your user experience. Click "Accept" to enable cookies or "Reject" to reject cookies. AcceptReject connectors No menu items for this category [Home](https://docs.open-metadata.org/latest) /[Connectors](https://docs.open-metadata.org/latest/connectors) /[Database](https://docs.open-metadata.org/latest/connectors/database) /[Pinotdb](https://docs.open-metadata.org/latest/connectors/database/pinotdb) /[Troubleshooting](https://docs.open-metadata.org/latest/connectors/database/pinotdb/troubleshooting) OpenMetadata Documentation Troubleshooting =============== Workflow Deployment Error ------------------------- If there were any errors during the workflow deployment process, the Ingestion Pipeline Entity will still be created, but no workflow will be present in the Ingestion container. * You can then Edit the Ingestion Pipeline and **Deploy** it again. * From the Connection tab, you can also Edit the Service if needed. Connector Debug Troubleshooting ------------------------------- This section provides instructions to help resolve common issues encountered during connector setup and metadata ingestion in OpenMetadata. Below are some of the most frequently observed troubleshooting scenarios. How to Enable Debug Logging for Any Ingestion --------------------------------------------- To enable debug logging for any ingestion workflow in OpenMetadata: 1. **Navigate to Services** Go to **Settings > Services > Service Type** (e.g., Database) in the OpenMetadata UI. 2. **Select a Service** Choose the specific service for which you want to enable debug logging. ![Select a Service](https://docs.open-metadata.org/images/v1.11/connectors/debug/debug1.png) Select a Service 3. **Access Ingestion Tab** Go to the **Ingestion tab** and click the three-dot menu on the right-hand side of the ingestion type, and select Edit. ![Access Agents Tab](https://docs.open-metadata.org/images/v1.11/connectors/debug/debug2.png) Access Agents Tab 4. **Enable Debug Logging** In the configuration dialog, enable the **Debug Log** option and click **Next**. ![Enable Debug Logging](https://docs.open-metadata.org/images/v1.11/connectors/debug/debug3.png) Enable Debug Logging 5. **Schedule and Submit** Configure the schedule if needed and click **Submit** to apply the changes. ![Schedule and Submit](https://docs.open-metadata.org/images/v1.11/connectors/debug/debug4.png) Schedule and Submit Permission Issues ----------------- If you encounter permission-related errors during connector setup or metadata ingestion, ensure that all the prerequisites and access configurations specified for each connector are properly implemented. Refer to the connector-specific documentation to verify the required permissions. --- # GCS Connector Troubleshooting Guide | OpenMetadata Support We use cookies to improve site navigation, analyze site usage, and enhance your user experience. Click "Accept" to enable cookies or "Reject" to reject cookies. AcceptReject connectors No menu items for this category [Home](https://docs.open-metadata.org/latest) /[Connectors](https://docs.open-metadata.org/latest/connectors) /[Storage](https://docs.open-metadata.org/latest/connectors/storage) /[Gcs](https://docs.open-metadata.org/latest/connectors/storage/gcs) /[Troubleshooting](https://docs.open-metadata.org/latest/connectors/storage/gcs/troubleshooting) OpenMetadata Documentation Troubleshooting =============== Workflow Deployment Error ------------------------- If there were any errors during the workflow deployment process, the Ingestion Pipeline Entity will still be created, but no workflow will be present in the Ingestion container. * You can then Edit the Ingestion Pipeline and **Deploy** it again. * From the Connection tab, you can also Edit the Service if needed. Connector Debug Troubleshooting ------------------------------- This section provides instructions to help resolve common issues encountered during connector setup and metadata ingestion in OpenMetadata. Below are some of the most frequently observed troubleshooting scenarios. How to Enable Debug Logging for Any Ingestion --------------------------------------------- To enable debug logging for any ingestion workflow in OpenMetadata: 1. **Navigate to Services** Go to **Settings > Services > Service Type** (e.g., Database) in the OpenMetadata UI. 2. **Select a Service** Choose the specific service for which you want to enable debug logging. ![Select a Service](https://docs.open-metadata.org/images/v1.11/connectors/debug/debug1.png) Select a Service 3. **Access Ingestion Tab** Go to the **Ingestion tab** and click the three-dot menu on the right-hand side of the ingestion type, and select Edit. ![Access Agents Tab](https://docs.open-metadata.org/images/v1.11/connectors/debug/debug2.png) Access Agents Tab 4. **Enable Debug Logging** In the configuration dialog, enable the **Debug Log** option and click **Next**. ![Enable Debug Logging](https://docs.open-metadata.org/images/v1.11/connectors/debug/debug3.png) Enable Debug Logging 5. **Schedule and Submit** Configure the schedule if needed and click **Submit** to apply the changes. ![Schedule and Submit](https://docs.open-metadata.org/images/v1.11/connectors/debug/debug4.png) Schedule and Submit Permission Issues ----------------- If you encounter permission-related errors during connector setup or metadata ingestion, ensure that all the prerequisites and access configurations specified for each connector are properly implemented. Refer to the connector-specific documentation to verify the required permissions. --- # Redpanda Connector Troubleshooting Guide | OpenMetadata Support We use cookies to improve site navigation, analyze site usage, and enhance your user experience. Click "Accept" to enable cookies or "Reject" to reject cookies. AcceptReject connectors No menu items for this category [Home](https://docs.open-metadata.org/latest) /[Connectors](https://docs.open-metadata.org/latest/connectors) /[Messaging](https://docs.open-metadata.org/latest/connectors/messaging) /[Redpanda](https://docs.open-metadata.org/latest/connectors/messaging/redpanda) /[Troubleshooting](https://docs.open-metadata.org/latest/connectors/messaging/redpanda/troubleshooting) OpenMetadata Documentation Troubleshooting =============== Workflow Deployment Error ------------------------- If there were any errors during the workflow deployment process, the Ingestion Pipeline Entity will still be created, but no workflow will be present in the Ingestion container. * You can then Edit the Ingestion Pipeline and **Deploy** it again. * From the Connection tab, you can also Edit the Service if needed. Connector Debug Troubleshooting ------------------------------- This section provides instructions to help resolve common issues encountered during connector setup and metadata ingestion in OpenMetadata. Below are some of the most frequently observed troubleshooting scenarios. How to Enable Debug Logging for Any Ingestion --------------------------------------------- To enable debug logging for any ingestion workflow in OpenMetadata: 1. **Navigate to Services** Go to **Settings > Services > Service Type** (e.g., Database) in the OpenMetadata UI. 2. **Select a Service** Choose the specific service for which you want to enable debug logging. ![Select a Service](https://docs.open-metadata.org/images/v1.11/connectors/debug/debug1.png) Select a Service 3. **Access Ingestion Tab** Go to the **Ingestion tab** and click the three-dot menu on the right-hand side of the ingestion type, and select Edit. ![Access Agents Tab](https://docs.open-metadata.org/images/v1.11/connectors/debug/debug2.png) Access Agents Tab 4. **Enable Debug Logging** In the configuration dialog, enable the **Debug Log** option and click **Next**. ![Enable Debug Logging](https://docs.open-metadata.org/images/v1.11/connectors/debug/debug3.png) Enable Debug Logging 5. **Schedule and Submit** Configure the schedule if needed and click **Submit** to apply the changes. ![Schedule and Submit](https://docs.open-metadata.org/images/v1.11/connectors/debug/debug4.png) Schedule and Submit Permission Issues ----------------- If you encounter permission-related errors during connector setup or metadata ingestion, ensure that all the prerequisites and access configurations specified for each connector are properly implemented. Refer to the connector-specific documentation to verify the required permissions. --- # Setup SAP ERP APIs | OpenMetadata Integration Guide We use cookies to improve site navigation, analyze site usage, and enhance your user experience. Click "Accept" to enable cookies or "Reject" to reject cookies. AcceptReject connectors No menu items for this category [Home](https://docs.open-metadata.org/latest) /[Connectors](https://docs.open-metadata.org/latest/connectors) /[Database](https://docs.open-metadata.org/latest/connectors/database) /[Sap Erp](https://docs.open-metadata.org/latest/connectors/database/sap-erp) /[Setup Sap Apis](https://docs.open-metadata.org/latest/connectors/database/sap-erp/setup-sap-apis) OpenMetadata Documentation Setup SAP ERP APIs ================== In this section, we provide guides and references to use setup the SAP ERP APIs needed for the connector. This document details the integration of Open Metadata with SAP systems, emphasizing the use of CDS Views and OData services to efficiently expose SAP data. To achieve this, data must be exposed via RESTful interfaces. Key concepts include: * **SAP Gateway**: A software component that bridges RFC and RESTful interfaces. * **RAP (Restful Application Programming)**: A coding framework designed to expose services via RESTful interfaces. * **CDS (Core Data Services)**: A layer that describes data objects and annotates them with desired functionalities, which are converted into code upon activation. * **OData V2 or V4**: A RESTful standard that simplifies interaction with database backends. Steps ----- ### 1\. ABAP Development Tools (ADT) Using the Eclipse based [ABAP Development Tools (ADT)](https://tools.hana.ondemand.com/#abap) the Restful interfaces are built. ### 2\. CDS Views After creating a new ABAP Project for the connected SAP system, a new Data Definition object is to be created. ![Data Definition Object](https://docs.open-metadata.org/images/v1.11/connectors/sap-erp/data-definition-object.png) Create data definition object * Create the first view that gets the table metadata * Then create the second view for table columns ### 3\. SAP Gateway Using the transaction `/nsegw` in SAPGUI, open the configuration screen for the SAP Gateway and create a new project with default project type. ![Create Project](https://docs.open-metadata.org/images/v1.11/connectors/sap-erp/create-project.png) Create Project Create a reference to the CDS views under Data Model and import the views. This is all that is needed to configure the OData details thanks to the CDS view annotations. ![Add Reference](https://docs.open-metadata.org/images/v1.11/connectors/sap-erp/add-reference.png) Add Reference The final step is to expose the generated code as OData service. This is the Register step. ![Register odata Service](https://docs.open-metadata.org/images/v1.11/connectors/sap-erp/register-odata-service.png) Register odata Service In the next screen click on Add Service and add the service as new OData endpoint. The service alias is the location where the SAP Gateway is installed. ![Add Service As Endpoint](https://docs.open-metadata.org/images/v1.11/connectors/sap-erp/add-service-as-endpoint.png) Add Service As Endpoint --- # Run the OpenAPI/REST Connector Externally We use cookies to improve site navigation, analyze site usage, and enhance your user experience. Click "Accept" to enable cookies or "Reject" to reject cookies. AcceptReject connectors No menu items for this category [Home](https://docs.open-metadata.org/latest) /[Connectors](https://docs.open-metadata.org/latest/connectors) /[Api](https://docs.open-metadata.org/latest/connectors/api) /[Rest](https://docs.open-metadata.org/latest/connectors/api/rest) /[Yaml](https://docs.open-metadata.org/latest/connectors/api/rest/yaml) OpenMetadata Documentation ![REST](https://docs.open-metadata.org/_next/image?url=%2Fimages%2Fconnectors%2Frest.webp&w=64&q=75) REST ==== BETA Available In Feature List API Endpoint Request Schema Response Schema In this section, we provide guides and references to use the OpenAPI/REST connector. Configure and schedule REST metadata workflows from the OpenMetadata UI: * [Requirements](https://docs.open-metadata.org/latest/connectors/api/rest/yaml#requirements) * [Metadata Ingestion](https://docs.open-metadata.org/latest/connectors/api/rest/yaml#metadata-ingestion) How to Run the Connector Externally ----------------------------------- To run the Ingestion via the UI you'll need to use the OpenMetadata Ingestion Container, which comes shipped with custom Airflow plugins to handle the workflow deployment. If, instead, you want to manage your workflows externally on your preferred orchestrator, you can check the following docs to run the Ingestion Framework **anywhere**. [#### External Schedulers\ \ Get more information about running the Ingestion Framework Externally](https://docs.open-metadata.org/latest/deployment/ingestion) Requirements ------------ ### Python Requirements We have support for Python versions 3.9-3.11 ### Generate OpenAPI Schema URL * Generate OpenAPI schema url for your service[OpenAPI spec](https://swagger.io/specification/#openapi-document) Metadata Ingestion ------------------ ### 1\. Define the YAML Config This is a sample config for OpenAPI: #### Source Configuration - Service Connection **OpenAPI Schema URL**: An OpenAPI schema URL typically refers to the URL where the OpenAPI Specification (OAS) document of a web service is hosted. The document defines the service's API, including available endpoints, request/response formats, authentication methods, etc. It is usually in JSON format. for e.g. `https://petstore3.swagger.io/api/v3/openapi.json` **Token**: An authentication token to connect to an OpenAPI schema URL. It is only required if the API schema is protected or secured. filename.yamlCopy ### 2\. Run with the CLI First, we will need to save the YAML file. Afterward, and with all requirements installed, we can run: Note that from connector to connector, this recipe will always be the same. By updating the YAML configuration, you will be able to extract metadata from different sources. --- # Configure Data Quality | Official Documentation We use cookies to improve site navigation, analyze site usage, and enhance your user experience. Click "Accept" to enable cookies or "Reject" to reject cookies. AcceptReject ![](https://static.scarf.sh/a.png?x-pxid=6814879a-96c0-40bd-be69-ddfeb14694aa) how-to-guides No menu items for this category [Home](https://docs.open-metadata.org/latest) /[How To Guides](https://docs.open-metadata.org/latest/how-to-guides) /[Data Quality Observability](https://docs.open-metadata.org/latest/how-to-guides/data-quality-observability) /[Quality](https://docs.open-metadata.org/latest/how-to-guides/data-quality-observability/quality) /[Configure](https://docs.open-metadata.org/latest/how-to-guides/data-quality-observability/quality/configure) OpenMetadata Documentation Configure Data Quality ====================== Learn how you can use OpenMetadata to define Data Quality tests and measure your data reliability. Requirements ------------ ### OpenMetadata You must have a running deployment of OpenMetadata to use this guide. OpenMetadata includes the following services: * OpenMetadata server supporting the metadata APIs and user interface * Elasticsearch for metadata search and discovery * MySQL as the backing store for all metadata * Airflow for metadata ingestion workflows To deploy OpenMetadata checkout the [deployment guide](https://docs.open-metadata.org/latest/deployment) ### Python (version 3.9.0 or later) Please use the following command to check the version of Python you have. Building Trust with Data Quality -------------------------------- OpenMetadata is where all users share and collaborate around data. It is where you make your assets discoverable; with data quality you make these assets **trustable**. This section will show you how to configure and run Data Quality pipelines with the OpenMetadata built-in tests. Main Concepts ------------- ### Test Suite Test Suites are logical container allowing you to group related Test Cases together from different tables. This is a great approach to group test case alerts and reduce alerting overload. Logical Test Suite ------------------ A Logical Test Suite is a collection of various test cases, which may pertain to different tables, grouped together under a single framework. Unlike Executable Test Suites, Logical Test Suites do not have an associated pipeline to execute the tests. Their primary purpose is to provide a consolidated view of related test cases, facilitating easier management and visualization without the need to run them as a single unit. Executable Test Suite --------------------- An Executable Test Suite is specifically associated with a single table, ensuring that all test cases within this suite are relevant to that particular table. The term "executable entity reference" refers to the specific table that the test suite is connected to, signifying that the tests can be run directly on this table. This suite is designed for execution, allowing for direct testing of the table's data integrity and functionality. ### Test Definition Test Definitions are generic tests definition elements specific to a test such as: * test name * column name * data type ### Test Cases Test Cases specify a Test Definition. It will define what condition a test must meet to be successful (e.g. `max=n`, etc.). One Test Definition can be linked to multiple Test Cases. --- # Run the GCS Connector Externally We use cookies to improve site navigation, analyze site usage, and enhance your user experience. Click "Accept" to enable cookies or "Reject" to reject cookies. AcceptReject connectors No menu items for this category [Home](https://docs.open-metadata.org/latest) /[Connectors](https://docs.open-metadata.org/latest/connectors) /[Storage](https://docs.open-metadata.org/latest/connectors/storage) /[Gcs](https://docs.open-metadata.org/latest/connectors/storage/gcs) /[Yaml](https://docs.open-metadata.org/latest/connectors/storage/gcs/yaml) OpenMetadata Documentation ![GCS](https://docs.open-metadata.org/_next/image?url=%2Fimages%2Fconnectors%2Fgcs.webp&w=64&q=75) GCS === PROD Available In Feature List Metadata This page contains the setup guide and reference information for the GCS connector. Configure and schedule GCS metadata workflows from the CLI: * [Requirements](https://docs.open-metadata.org/latest/connectors/storage/gcs/yaml#requirements) * [Metadata Ingestion](https://docs.open-metadata.org/latest/connectors/storage/gcs/yaml#metadata-ingestion) How to Run the Connector Externally ----------------------------------- To run the Ingestion via the UI you'll need to use the OpenMetadata Ingestion Container, which comes shipped with custom Airflow plugins to handle the workflow deployment. If, instead, you want to manage your workflows externally on your preferred orchestrator, you can check the following docs to run the Ingestion Framework **anywhere**. [#### External Schedulers\ \ Get more information about running the Ingestion Framework Externally](https://docs.open-metadata.org/latest/deployment/ingestion) Requirements ------------ To run the GCS ingestion, you will need to install: [OpenMetadata 1.0 or later\ \ To deploy OpenMetadata, check the Deployment guides.](https://docs.open-metadata.org/latest/deployment) We need the following permissions in GCP: ### GCS Permissions For all the buckets that we want to ingest, we need to provide the following: * `storage.buckets.get` * `storage.buckets.list` * `storage.objects.get` * `storage.objects.list` ### OpenMetadata Manifest In any other connector, extracting metadata happens automatically. In this case, we will be able to extract high-level metadata from buckets, but in order to understand their internal structure we need users to provide an `openmetadata.json` file at the bucket root. `Supported File Formats: [ "csv", "tsv", "avro", "parquet", "json", "json.gz", "json.zip" ]` You can learn more about this [here](https://docs.open-metadata.org/latest/connectors/storage) . Keep reading for an example on the shape of the manifest file. OpenMetadata Manifest --------------------- Our manifest file is defined as a [JSON Schema](https://github.com/open-metadata/OpenMetadata/blob/main/openmetadata-spec/src/main/resources/json/schema/metadataIngestion/storage/containerMetadataConfig.json) , and can look like this: **Entries**: We need to add a list of `entries`. Each inner JSON structure will be ingested as a child container of the top-level one. In this case, we will be ingesting 4 children. **Simple Container**: The simplest container we can have would be structured, but without partitions. Note that we still need to bring information about: * **dataPath**: Where we can find the data. This should be a path relative to the top-level container. * **structureFormat**: What is the format of the data we are going to find. This information will be used to read the data. * **separator**: Optionally, for delimiter-separated formats such as CSV, you can specify the separator to use when reading the file. If you don't, we will use `,` for CSV and `/t` for TSV files. After ingesting this container, we will bring in the schema of the data in the `dataPath`. **Partitioned Container**: We can ingest partitioned data without bringing in any further details. By informing the `isPartitioned` field as `true`, we'll flag the container as `Partitioned`. We will be reading the source files schemas', but won't add any other information. **Single-Partition Container**: We can bring partition information by specifying the `partitionColumns`. Their definition is based on the [JSON Schema](https://github.com/open-metadata/OpenMetadata/blob/main/openmetadata-spec/src/main/resources/json/schema/entity/data/table.json#L232) definition for table columns. The minimum required information is the `name` and `dataType`. When passing `partitionColumns`, these values will be added to the schema, on top of the inferred information from the files. **Multiple-Partition Container**: We can add multiple columns as partitions. Note how in the example we even bring our custom `displayName` for the column `dataTypeDisplay` for its type. Again, this information will be added on top of the inferred schema from the data files. **Automated Container Ingestion**: Registering all the data paths one by one can be a time consuming job, to make the automated structure container ingestion you can provide the depth at which all the data is available. Let us understand this with the example, suppose following is the file hierarchy within my bucket. all my tables folders which contains the actual data are available at depth 3, hence when you specify the `depth: 3` in manifest entry all following path will get registered as container in OpenMetadata with this single entry saving efforts to add 4 individual entries compared to 1 **Unstructured Container**: OpenMetadata supports ingesting unstructured files like images, pdf's etc. We support fetching the file names, size and tags associates to such files. In case you want to ingest a single unstructured file, then just specifying the full path of the unstructured file in `datapath` would be enough for ingestion. In case you want to ingest all unstructured files with a specific extension for example `pdf` & `png` then you can provide the folder name containing such files in `dataPath` and list of extensions in the `unstructuredFormats` field. In case you want to ingest all unstructured files with irrespective of their file type or extension then you can provide the folder name containing such files in `dataPath` and `["*"]` in the `unstructuredFormats` field. openmetadata.jsonCopy ### Global Manifest You can also manage a **single** manifest file to centralize the ingestion process for any container, named `openmetadata_storage_manifest.json`. For example: In that case, you will need to add a `containerName` entry to the structure above. For example: The fields shown above (`dataPath`, `structureFormat`, `isPartitioned`, etc.) are still valid. **Container Name**: Since we are using a single manifest for all your containers, the field `containerName` will help us identify which container (or Bucket in S3, etc.), contains the presented information. openmetadata-global.jsonCopy You can also keep local manifests `openmetadata.json` in each container, but if possible, we will always try to pick up the global manifest during the ingestion. Metadata Ingestion ------------------ All connectors are defined as JSON Schemas. [Here](https://github.com/open-metadata/OpenMetadata/blob/main/openmetadata-spec/src/main/resources/json/schema/entity/services/connections/storage/GCSConnection.json) you can find the structure to create a connection to Athena. In order to create and run a Metadata Ingestion workflow, we will follow the steps to create a YAML configuration able to connect to the source, process the Entities if needed, and reach the OpenMetadata server. The workflow is modeled around the following [JSON Schema](https://github.com/open-metadata/OpenMetadata/blob/main/openmetadata-spec/src/main/resources/json/schema/metadataIngestion/workflow.json) ### 1\. Define the YAML Config This is a sample config for Athena: #### Source Configuration - Service Connection **gcpConfig:** **1.** Passing the raw credential values provided by GCP. This requires us to provide the following information, all provided by GCP: **gcpConfig:** * **type**: Credentials Type is the type of the account, for a service account the value of this field is `service_account`. To fetch this key, look for the value associated with the `type` key in the service account key file. * **projectId**: A project ID is a unique string used to differentiate your project from all others in Google Cloud. To fetch this key, look for the value associated with the `project_id` key in the service account key file. You can also pass multiple project id to ingest metadata from different BigQuery projects into one service. * **privateKeyId**: This is a unique identifier for the private key associated with the service account. To fetch this key, look for the value associated with the `private_key_id` key in the service account file. * **privateKey**: This is the private key associated with the service account that is used to authenticate and authorize access to BigQuery. To fetch this key, look for the value associated with the `private_key` key in the service account file. * **clientEmail**: This is the email address associated with the service account. To fetch this key, look for the value associated with the `client_email` key in the service account key file. * **clientId**: This is a unique identifier for the service account. To fetch this key, look for the value associated with the `client_id` key in the service account key file. * **authUri**: This is the URI for the authorization server. To fetch this key, look for the value associated with the `auth_uri` key in the service account key file. The default value to Auth URI is https://accounts.google.com/o/oauth2/auth. * **tokenUri**: The Google Cloud Token URI is a specific endpoint used to obtain an OAuth 2.0 access token from the Google Cloud IAM service. This token allows you to authenticate and access various Google Cloud resources and APIs that require authorization. To fetch this key, look for the value associated with the `token_uri` key in the service account credentials file. Default Value to Token URI is https://oauth2.googleapis.com/token. * **authProviderX509CertUrl**: This is the URL of the certificate that verifies the authenticity of the authorization server. To fetch this key, look for the value associated with the `auth_provider_x509_cert_url` key in the service account key file. The Default value for Auth Provider X509Cert URL is https://www.googleapis.com/oauth2/v1/certs * **clientX509CertUrl**: This is the URL of the certificate that verifies the authenticity of the service account. To fetch this key, look for the value associated with the `client_x509_cert_url` key in the service account key file. **2.** Passing a local file path that contains the credentials: * **gcpCredentialsPath** * If you prefer to pass the credentials file, you can do so as follows: * If you want to use [ADC authentication](https://cloud.google.com/docs/authentication#adc) for GCP you can just leave the GCP credentials empty. This is why they are not marked as required. #### Source Configuration - Source Config The `sourceConfig` is defined [here](https://github.com/open-metadata/OpenMetadata/blob/main/openmetadata-spec/src/main/resources/json/schema/metadataIngestion/storageServiceMetadataPipeline.json) : **containerFilterPattern**: Note that the filter supports regex as include or exclude. You can find examples [here](https://docs.open-metadata.org/latest/connectors/ingestion/workflows/metadata/filter-patterns/database) . **storageMetadataConfigSource**: Path to the `openmetadata_storage_manifest.json` global manifest file. It can be located in S3, a local path or as a URL to the file. #### Sink Configuration To send the metadata to OpenMetadata, it needs to be specified as `type: metadata-rest`. #### Workflow Configuration The main property here is the `openMetadataServerConfig`, where you can define the host and security provider of your OpenMetadata installation. **Logger Level** You can specify the `loggerLevel` depending on your needs. If you are trying to troubleshoot an ingestion, running with `DEBUG` will give you far more traces for identifying issues. **JWT Token** JWT tokens will allow your clients to authenticate against the OpenMetadata server. To enable JWT Tokens, you will get more details [here](https://docs.open-metadata.org/latest/deployment/security/enable-jwt-tokens) . You can refer to the JWT Troubleshooting section [link](https://docs.open-metadata.org/latest/deployment/security/jwt-troubleshooting) for any issues in your JWT configuration. **Store Service Connection** If set to `true` (default), we will store the sensitive information either encrypted via the Fernet Key in the database or externally, if you have configured any [Secrets Manager](https://docs.open-metadata.org/latest/deployment/secrets-manager) . If set to `false`, the service will be created, but the service connection information will only be used by the Ingestion Framework at runtime, and won't be sent to the OpenMetadata server. **Store Service Connection** If set to `true` (default), we will store the sensitive information either encrypted via the Fernet Key in the database or externally, if you have configured any [Secrets Manager](https://docs.open-metadata.org/latest/deployment/secrets-manager) . If set to `false`, the service will be created, but the service connection information will only be used by the Ingestion Framework at runtime, and won't be sent to the OpenMetadata server. **SSL Configuration** If you have added SSL to the [OpenMetadata server](https://docs.open-metadata.org/latest/deployment/security/enable-ssl) , then you will need to handle the certificates when running the ingestion too. You can either set `verifySSL` to `ignore`, or have it as `validate`, which will require you to set the `sslConfig.caCertificate` with a local path where your ingestion runs that points to the server certificate file. Find more information on how to troubleshoot SSL issues [here](https://docs.open-metadata.org/latest/deployment/security/enable-ssl/ssl-troubleshooting) . **ingestionPipelineFQN** Fully qualified name of ingestion pipeline, used to identify the current ingestion pipeline. #### Advanced Configuration **Connection Options (Optional)**: Enter the details for any additional connection options that can be sent to storage service during the connection. These details must be added as Key-Value pairs. **Connection Arguments (Optional)**: Enter the details for any additional connection arguments such as security or protocol configs that can be sent to storage service during the connection. These details must be added as Key-Value pairs. filename.yamlCopy --- # Run the Fivetran Connector Externally We use cookies to improve site navigation, analyze site usage, and enhance your user experience. Click "Accept" to enable cookies or "Reject" to reject cookies. AcceptReject connectors No menu items for this category [Home](https://docs.open-metadata.org/latest) /[Connectors](https://docs.open-metadata.org/latest/connectors) /[Pipeline](https://docs.open-metadata.org/latest/connectors/pipeline) /[Fivetran](https://docs.open-metadata.org/latest/connectors/pipeline/fivetran) /[Yaml](https://docs.open-metadata.org/latest/connectors/pipeline/fivetran/yaml) OpenMetadata Documentation ![Fivetran](https://docs.open-metadata.org/_next/image?url=%2Fimages%2Fconnectors%2Ffivetran.webp&w=64&q=75) Fivetran ======== PROD Available In Feature List Pipelines Pipeline Status Lineage Usage Owners Tags In this section, we provide guides and references to use the Fivetran connector. Configure and schedule Fivetran metadata and profiler workflows from the OpenMetadata UI: * [Requirements](https://docs.open-metadata.org/latest/connectors/pipeline/fivetran/yaml#requirements) * [Metadata Ingestion](https://docs.open-metadata.org/latest/connectors/pipeline/fivetran/yaml#metadata-ingestion) How to Run the Connector Externally ----------------------------------- To run the Ingestion via the UI you'll need to use the OpenMetadata Ingestion Container, which comes shipped with custom Airflow plugins to handle the workflow deployment. If, instead, you want to manage your workflows externally on your preferred orchestrator, you can check the following docs to run the Ingestion Framework **anywhere**. [#### External Schedulers\ \ Get more information about running the Ingestion Framework Externally](https://docs.open-metadata.org/latest/deployment/ingestion) Requirements ------------ To access Fivetran APIs, a Fivetran account on a Standard, Enterprise, or Business Critical plan is required. ### Python Requirements We have support for Python versions 3.9-3.11 To run the Fivetran ingestion, you will need to install: Metadata Ingestion ------------------ All connectors are defined as JSON Schemas. [Here](https://github.com/open-metadata/OpenMetadata/blob/main/openmetadata-spec/src/main/resources/json/schema/entity/services/connections/pipeline/fivetranConnection.json) you can find the structure to create a connection to Fivetran. In order to create and run a Metadata Ingestion workflow, we will follow the steps to create a YAML configuration able to connect to the source, process the Entities if needed, and reach the OpenMetadata server. The workflow is modeled around the following [JSON Schema](https://github.com/open-metadata/OpenMetadata/blob/main/openmetadata-spec/src/main/resources/json/schema/metadataIngestion/workflow.json) ### 1\. Define the YAML Config This is a sample config for Fivetran: #### Source Configuration - Service Connection **apiKey**: Fivetran API Key. Follow the steps mentioned below to generate the Fivetran API key and API secret: * Click your user name in your Fivetran dashboard. * Click API Key. * Click Generate API key. (If you already have an API key, then the button text is Generate new API key.) * Make a note of the key and secret as they won't be displayed once you close the page or navigate away. For more detailed documentation visit [here](https://fivetran.com/docs/rest-api/getting-started) . **apiSecret**: Fivetran API Secret. From the above step where the API key is generated copy the the API secret **hostPort**: HostPort of the Fivetran instance. Hostport of the Fivetran instance that the connection will be made to By default OpenMetadata will use `https://api.fivetran.com` to connect to the Fivetran APIs. **limit**: Fivetran API Limit For Pagination. This refers to the maximum number of records that can be returned in a single page of results when using Fivetran's API for pagination. #### Source Configuration - Source Config The `sourceConfig` is defined [here](https://github.com/open-metadata/OpenMetadata/blob/main/openmetadata-spec/src/main/resources/json/schema/metadataIngestion/pipelineServiceMetadataPipeline.json) : * **dbServiceNames**: Database Service Name for the creation of lineage, if the source supports it. * **includeTags**: Set the 'Include Tags' toggle to control whether to include tags as part of metadata ingestion. * **includeUnDeployedPipelines**: Set the 'Include UnDeployed Pipelines' toggle to control whether to include un-deployed pipelines as part of metadata ingestion. By default it is set to `true` * **markDeletedPipelines**: Set the Mark Deleted Pipelines toggle to flag pipelines as soft-deleted if they are not present anymore in the source system. * **pipelineFilterPattern** and **chartFilterPattern**: Note that the `pipelineFilterPattern` and `chartFilterPattern` both support regex as include or exclude. * **includeOwners**: Set the 'Include Owners' toggle to control whether to include owners to the ingested entity if the owner email matches with a user stored in the OM server as part of metadata ingestion. If the ingested entity already exists and has an owner, the owner will not be overwritten.It supports boolean values either `true` or `false`. * **overrideLineage**: Set the 'Override Lineage' toggle to control whether to override the existing lineage. It supports boolean values either `true` or `false`. * **overrideMetadata**: Set the 'Override Metadata' toggle to control whether to override the existing metadata in the OpenMetadata server with the metadata fetched from the source. If the toggle is set to true, the metadata fetched from the source will override the existing metadata in the OpenMetadata server. If the toggle is set to false, the metadata fetched from the source will not override the existing metadata in the OpenMetadata server. This is applicable for fields like description, tags, owner and displayName. It supports boolean values either `true` or `false`. #### Sink Configuration To send the metadata to OpenMetadata, it needs to be specified as `type: metadata-rest`. #### Workflow Configuration The main property here is the `openMetadataServerConfig`, where you can define the host and security provider of your OpenMetadata installation. **Logger Level** You can specify the `loggerLevel` depending on your needs. If you are trying to troubleshoot an ingestion, running with `DEBUG` will give you far more traces for identifying issues. **JWT Token** JWT tokens will allow your clients to authenticate against the OpenMetadata server. To enable JWT Tokens, you will get more details [here](https://docs.open-metadata.org/latest/deployment/security/enable-jwt-tokens) . You can refer to the JWT Troubleshooting section [link](https://docs.open-metadata.org/latest/deployment/security/jwt-troubleshooting) for any issues in your JWT configuration. **Store Service Connection** If set to `true` (default), we will store the sensitive information either encrypted via the Fernet Key in the database or externally, if you have configured any [Secrets Manager](https://docs.open-metadata.org/latest/deployment/secrets-manager) . If set to `false`, the service will be created, but the service connection information will only be used by the Ingestion Framework at runtime, and won't be sent to the OpenMetadata server. **Store Service Connection** If set to `true` (default), we will store the sensitive information either encrypted via the Fernet Key in the database or externally, if you have configured any [Secrets Manager](https://docs.open-metadata.org/latest/deployment/secrets-manager) . If set to `false`, the service will be created, but the service connection information will only be used by the Ingestion Framework at runtime, and won't be sent to the OpenMetadata server. **SSL Configuration** If you have added SSL to the [OpenMetadata server](https://docs.open-metadata.org/latest/deployment/security/enable-ssl) , then you will need to handle the certificates when running the ingestion too. You can either set `verifySSL` to `ignore`, or have it as `validate`, which will require you to set the `sslConfig.caCertificate` with a local path where your ingestion runs that points to the server certificate file. Find more information on how to troubleshoot SSL issues [here](https://docs.open-metadata.org/latest/deployment/security/enable-ssl/ssl-troubleshooting) . **ingestionPipelineFQN** Fully qualified name of ingestion pipeline, used to identify the current ingestion pipeline. filename.yamlCopy ### 2\. Run with the CLI First, we will need to save the YAML file. Afterward, and with all requirements installed, we can run: Note that from connector to connector, this recipe will always be the same. By updating the YAML configuration, you will be able to extract metadata from different sources. --- # Google SSO Configuration Guide | Public Client Setup We use cookies to improve site navigation, analyze site usage, and enhance your user experience. Click "Accept" to enable cookies or "Reject" to reject cookies. AcceptReject ![](https://static.scarf.sh/a.png?x-pxid=8472f2b3-d307-4935-a5ed-3f9d3982c15d) deployment No menu items for this category [Home](https://docs.open-metadata.org/latest) /[Deployment](https://docs.open-metadata.org/latest/deployment) /[Security](https://docs.open-metadata.org/latest/deployment/security) /[Google](https://docs.open-metadata.org/latest/deployment/security/google) /[Public Client](https://docs.open-metadata.org/latest/deployment/security/google/public-client) OpenMetadata Documentation Google SSO Authentication Documentation - Public Client ======================================================= * [Troubleshooting](https://docs.open-metadata.org/latest/deployment/security/google/public-client#troubleshooting) Overview -------- OpenMetadata supports Single Sign-On (SSO) integration with various identity providers, enabling secure, centralized user authentication. * **Navigate to:** `Settings > SSO` ![SSO Authentication](https://docs.open-metadata.org/images/v1.11/deployment/security/google/sso1.png) * Select Google as the service Provider ![Supported Providers](https://docs.open-metadata.org/images/v1.11/deployment/security/google/sso2.png) Google SSO Configuration ------------------------ This configuration is recommended for **public applications**, such as **SPAs (Single Page Applications)** and **mobile apps**. It does **not require a client secret**. ![Google SSO Configuration - Public Client](https://docs.open-metadata.org/images/v1.11/deployment/security/google/google1.png) ### 1\. Client Type * **Definition:** Defines whether the application is `Public` (no client secret) or `Confidential` (requires client secret). * **Options:** `Public` | `Confidential` * **Example:** `Public` * **Why it matters:** Determines the security level and OAuth flow type. * **Note:** * Choose `Public` for SPAs and mobile apps. * Google typically uses `Confidential` for server-side apps. ### 2\. Callback URL * **Definition:** Redirect URL where Google sends authentication responses. * **Example:** `https://yourapp.company.com/callback` * **Why it matters:** Must match exactly with Google Cloud Console’s registered redirect URL. * **Note:** * Must be registered under **OAuth 2.0 Client > Authorized Redirect URLs**. * Always use **HTTPS** in production. ### 3\. Enable Self Signup * **Definition:** Allows users to automatically create accounts on first login. * **Options:** `Enabled` | `Disabled` * **Example:** `Enabled` * **Why it matters:** Controls whether new users are auto-created. * **Note:** Set to `Disabled` for stricter access control. ### 4\. Authority * **Definition:** Google’s OAuth 2.0 authorization server. * **Default & Example:** `https://accounts.google.com` * **Why it matters:** Specifies where OpenMetadata should send auth requests. * **Note:** Usually does not need to be changed. ### 5\. Public Key URLs * **Definition:** URL(s) where Google publishes its JWT signing keys. * **Example:** `["https://www.googleapis.com/oauth2/v3/certs"]` * **Why it matters:** Required to verify JWT token signatures. * **Note:** Typically auto-discovered via OIDC discovery endpoint. ### 6\. Token Validation Algorithm * **Definition:** Algorithm used to validate JWT tokens. * **Options:** `RS256` | `RS384` | `RS512` * **Default & Example:** `RS256` * **Why it matters:** Must match Google’s signing algorithm. * **Note:** Google typically uses `RS256`. ### 7\. JWT Principal Claims * **Definition:** JWT fields used to identify the user. * **Example:** `["email", "sub"]` * **Why it matters:** Identifies the user in OpenMetadata. * **Note:** Use `email` for consistency and compatibility. For domain scoping, use the `hd` claim. ### 8\. JWT Principal Claims Mapping * **Definition:** Maps JWT claims to OpenMetadata user attributes. * **Example:** * **Why it matters:** Maps identity information to OpenMetadata user profiles. * **Note:** Use the format `openmetadata_field:jwt_claim`. ### 9\. Admin Principals * **Definition:** List of email addresses with admin access. * **Example:** `["admin@company.com", "superuser@company.com"]` * **Why it matters:** Grants admin-level privileges in the system. * **Note:** Email must match a JWT claim. ### 10\. Principal Domain * **Definition:** Default domain for constructing user emails. * **Example:** `company.com` * **Why it matters:** Helps form complete user identifiers from partial input. * **Note:** Matches your Google Workspace domain. ### 11\. Enforce Principal Domain * **Definition:** Restrict login to users from a specific domain. * **Default:** false * **Example:** `true` * **Why it matters:** Adds domain-level access control. * **Note:** Use with hd parameter in custom OIDC config. ### 12\. Enable Secure Socket Connection * **Definition:** Enables SSL/TLS for secure communications. * **Default:** false * **Example:** `true` * **Why it matters:** Ensures secure token exchange and communication. * **Note:** Must be true in production. ### Troubleshooting If users are automatically logged out and unable to log in again due to a bad authentication configuration, you can reset the security setup using the following command: After executing the command, **restart the server**. The authentication values from your YAML or Helm chart will then be reapplied on startup. The following tiles detail how to apply this configuration across Docker, Kubernetes, and Bare Metal deployments: [Docker Security\ \ Configure SSO for your Docker Deployment.](https://docs.open-metadata.org/latest/deployment/security/configuration/docker) [Bare Metal Security\ \ Configure SSO for your Bare Metal Deployment.](https://docs.open-metadata.org/latest/deployment/security/configuration/bare-metal) [Kubernetes Security\ \ Configure SSO for your Kubernetes Deployment.](https://docs.open-metadata.org/latest/deployment/security/configuration/kubernetes) --- # Run the Redpanda Connector Externally We use cookies to improve site navigation, analyze site usage, and enhance your user experience. Click "Accept" to enable cookies or "Reject" to reject cookies. AcceptReject connectors No menu items for this category [Home](https://docs.open-metadata.org/latest) /[Connectors](https://docs.open-metadata.org/latest/connectors) /[Messaging](https://docs.open-metadata.org/latest/connectors/messaging) /[Redpanda](https://docs.open-metadata.org/latest/connectors/messaging/redpanda) /[Yaml](https://docs.open-metadata.org/latest/connectors/messaging/redpanda/yaml) OpenMetadata Documentation ![Redpanda](https://docs.open-metadata.org/_next/image?url=%2Fimages%2Fconnectors%2Fredpanda.webp&w=64&q=75) Redpanda ======== PROD Available In Feature List Topics Sample Data In this section, we provide guides and references to use the Redpanda connector. Configure and schedule Redpanda metadata and profiler workflows from the OpenMetadata UI: * [Requirements](https://docs.open-metadata.org/latest/connectors/messaging/redpanda/yaml#requirements) * [Metadata Ingestion](https://docs.open-metadata.org/latest/connectors/messaging/redpanda/yaml#metadata-ingestion) How to Run the Connector Externally ----------------------------------- To run the Ingestion via the UI you'll need to use the OpenMetadata Ingestion Container, which comes shipped with custom Airflow plugins to handle the workflow deployment. If, instead, you want to manage your workflows externally on your preferred orchestrator, you can check the following docs to run the Ingestion Framework **anywhere**. [#### External Schedulers\ \ Get more information about running the Ingestion Framework Externally](https://docs.open-metadata.org/latest/deployment/ingestion) Requirements ------------ ### Python Requirements We have support for Python versions 3.9-3.11 To run the Redpanda ingestion, you will need to install: Metadata Ingestion ------------------ All connectors are defined as JSON Schemas. [Here](https://github.com/open-metadata/OpenMetadata/blob/main/openmetadata-spec/src/main/resources/json/schema/entity/services/connections/messaging/redpandaConnection.json) you can find the structure to create a connection to Redpanda. In order to create and run a Metadata Ingestion workflow, we will follow the steps to create a YAML configuration able to connect to the source, process the Entities if needed, and reach the OpenMetadata server. The workflow is modeled around the following [JSON Schema](https://github.com/open-metadata/OpenMetadata/blob/main/openmetadata-spec/src/main/resources/json/schema/metadataIngestion/workflow.json) ### 1\. Define the YAML Config This is a sample config for Redpanda: #### Source Configuration - Service Connection **bootstrapServers**: List of brokers as comma separated values of broker `host` or `host:port`. Example: `host1:9092,host2:9092` **schemaRegistryURL**: URL of the Schema Registry used to ingest the schemas of the topics. **NOTE**: For now, the schema will be the last version found for the schema name `{topic-name}-value`. An [issue](https://github.com/open-metadata/OpenMetadata/issues/10399) to improve how it currently works has been opened. **saslUsername**: SASL username for use with the PLAIN and SASL-SCRAM mechanisms. **saslPassword**: SASL password for use with the PLAIN and SASL-SCRAM mechanisms. **saslMechanism**: SASL mechanism to use for authentication. Supported: _GSSAPI, PLAIN, SCRAM-SHA-256, SCRAM-SHA-512, OAUTHBEARER_. **NOTE**: Despite the name only one mechanism must be configured. **basicAuthUserInfo**: Schema Registry Client HTTP credentials in the form of `username:password`. By default, user info is extracted from the URL if present. **consumerConfig**: The accepted additional values for the consumer configuration can be found in the following [link](https://github.com/edenhill/librdkafka/blob/master/CONFIGURATION.md) . **schemaRegistryConfig**: The accepted additional values for the Schema Registry configuration can be found in the following [link](https://docs.confluent.io/platform/current/clients/confluent-kafka-python/html/index.html#schemaregistryclient) . **Note:** To ingest the topic schema, `schemaRegistryURL` must be passed. **securityProtocol**: security.protocol consumer config property. It accepts `PLAINTEXT`,`SASL_PLAINTEXT`, `SASL_SSL`, `SSL`. **supportsMetadataExtraction**: Supports Metadata Extraction. `supportsMetadataExtraction` supports boolean value either true or false. #### Source Configuration - Source Config The sourceConfig is defined [here](https://github.com/open-metadata/OpenMetadata/blob/main/openmetadata-spec/src/main/resources/json/schema/metadataIngestion/messagingServiceMetadataPipeline.json) : * **generateSampleData:** Option to turn on/off generating sample data during metadata extraction. * **topicFilterPattern:** Note that the `topicFilterPattern` supports regex as include or exclude. * **generateSampleData:** Option to turn on/off generating sample data during metadata extraction. `generateSampleData` supports boolean value either `true` or `false`. * **markDeletedTopics:** Optional configuration to soft delete topics in OpenMetadata if the source topics are deleted. Also, if the topic is deleted, all the associated entities like sample data, lineage, etc., with that topic will be deleted. `markDeletedTopics` supports boolean value either `true` or `false`. * **overrideMetadata:** Set the 'Override Metadata' toggle to control whether to override the existing metadata in the OpenMetadata server with the metadata fetched from the source. If the toggle is set to true, the metadata fetched from the source will override the existing metadata in the OpenMetadata server. If the toggle is set to false, the metadata fetched from the source will not override the existing metadata in the OpenMetadata server. This is applicable for fields like description, tags, owner and displayName. `overrideMetadata` supports boolean value either `true` or `false`. #### Sink Configuration To send the metadata to OpenMetadata, it needs to be specified as `type: metadata-rest`. #### Workflow Configuration The main property here is the `openMetadataServerConfig`, where you can define the host and security provider of your OpenMetadata installation. **Logger Level** You can specify the `loggerLevel` depending on your needs. If you are trying to troubleshoot an ingestion, running with `DEBUG` will give you far more traces for identifying issues. **JWT Token** JWT tokens will allow your clients to authenticate against the OpenMetadata server. To enable JWT Tokens, you will get more details [here](https://docs.open-metadata.org/latest/deployment/security/enable-jwt-tokens) . You can refer to the JWT Troubleshooting section [link](https://docs.open-metadata.org/latest/deployment/security/jwt-troubleshooting) for any issues in your JWT configuration. **Store Service Connection** If set to `true` (default), we will store the sensitive information either encrypted via the Fernet Key in the database or externally, if you have configured any [Secrets Manager](https://docs.open-metadata.org/latest/deployment/secrets-manager) . If set to `false`, the service will be created, but the service connection information will only be used by the Ingestion Framework at runtime, and won't be sent to the OpenMetadata server. **Store Service Connection** If set to `true` (default), we will store the sensitive information either encrypted via the Fernet Key in the database or externally, if you have configured any [Secrets Manager](https://docs.open-metadata.org/latest/deployment/secrets-manager) . If set to `false`, the service will be created, but the service connection information will only be used by the Ingestion Framework at runtime, and won't be sent to the OpenMetadata server. **SSL Configuration** If you have added SSL to the [OpenMetadata server](https://docs.open-metadata.org/latest/deployment/security/enable-ssl) , then you will need to handle the certificates when running the ingestion too. You can either set `verifySSL` to `ignore`, or have it as `validate`, which will require you to set the `sslConfig.caCertificate` with a local path where your ingestion runs that points to the server certificate file. Find more information on how to troubleshoot SSL issues [here](https://docs.open-metadata.org/latest/deployment/security/enable-ssl/ssl-troubleshooting) . **ingestionPipelineFQN** Fully qualified name of ingestion pipeline, used to identify the current ingestion pipeline. filename.yamlCopy ### 2\. Run with the CLI First, we will need to save the YAML file. Afterward, and with all requirements installed, we can run: Note that from connector to connector, this recipe will always be the same. By updating the YAML configuration, you will be able to extract metadata from different sources. --- # Run the Lightdash Connector Externally We use cookies to improve site navigation, analyze site usage, and enhance your user experience. Click "Accept" to enable cookies or "Reject" to reject cookies. AcceptReject connectors No menu items for this category [Home](https://docs.open-metadata.org/latest) /[Connectors](https://docs.open-metadata.org/latest/connectors) /[Dashboard](https://docs.open-metadata.org/latest/connectors/dashboard) /[Lightdash](https://docs.open-metadata.org/latest/connectors/dashboard/lightdash) /[Yaml](https://docs.open-metadata.org/latest/connectors/dashboard/lightdash/yaml) OpenMetadata Documentation ![Lightdash](https://docs.open-metadata.org/_next/image?url=%2Fimages%2Fconnectors%2Flightdash.webp&w=64&q=75) Lightdash ========= PROD Available In Feature List Dashboards Charts Owners Datamodels Lineage Tags Projects In this section, we provide guides and references to use the Lightdash connector. Configure and schedule Lightdash metadata and profiler workflows from the OpenMetadata UI: * [Requirements](https://docs.open-metadata.org/latest/connectors/dashboard/lightdash/yaml#requirements) * [Metadata Ingestion](https://docs.open-metadata.org/latest/connectors/dashboard/lightdash/yaml#metadata-ingestion) How to Run the Connector Externally ----------------------------------- To run the Ingestion via the UI you'll need to use the OpenMetadata Ingestion Container, which comes shipped with custom Airflow plugins to handle the workflow deployment. If, instead, you want to manage your workflows externally on your preferred orchestrator, you can check the following docs to run the Ingestion Framework **anywhere**. [#### External Schedulers\ \ Get more information about running the Ingestion Framework Externally](https://docs.open-metadata.org/latest/deployment/ingestion) Requirements ------------ To integrate Lightdash, ensure you are using OpenMetadata version 1.2.x or higher. ### Python Requirements We have support for Python versions 3.9-3.11 To run the Lightdash ingestion, you will need to install: Metadata Ingestion ------------------ All connectors are defined as JSON Schemas. [Here](https://github.com/open-metadata/OpenMetadata/blob/main/openmetadata-spec/src/main/resources/json/schema/entity/services/connections/dashboard/lightdashConnection.json) you can find the structure to create a connection to Lightdash. In order to create and run a Metadata Ingestion workflow, we will follow the steps to create a YAML configuration able to connect to the source, process the Entities if needed, and reach the OpenMetadata server. The workflow is modeled around the following [JSON Schema](https://github.com/open-metadata/OpenMetadata/blob/main/openmetadata-spec/src/main/resources/json/schema/metadataIngestion/workflow.json) ### 1\. Define the YAML Config This is a sample config for Lightdash: #### Source Configuration - Service Connection * **Host and Port**: Specify the network location where your Lightdash instance is accessible, combining both hostname and port in a URI format: either `http://hostname:port` or `https://hostname:port`, based on your security needs. **Example**: For a local setup, use `http://localhost:8080`; for a server deployment, it might be `https://lightdash.example.com:3000`. Ensure the specified port is open and accessible through network firewall settings. * **API Key**: This key authenticates requests to your Lightdash instance. Keep the API Key secure, sharing it only with authorized applications or users. * **Project UUID**: This unique identifier links API requests or configurations to a specific project in Lightdash. * **Space UUID**: Identifies a specific "Space" in Lightdash, used to organize dashboards, charts, and assets. * **Proxy Authentication**: If your Lightdash instance requires authentication through a proxy server, provide proxy credentials. Proxy authentication controls access to external resources and Lightdash. #### Source Configuration - Source Config The `sourceConfig` is defined [here](https://github.com/open-metadata/OpenMetadata/blob/main/openmetadata-spec/src/main/resources/json/schema/metadataIngestion/dashboardServiceMetadataPipeline.json) : * **dbServicePrefixes**: List of service path prefixes for lineage matching. Supported formats: DBServiceName, DBServiceName.DatabaseName, DBServiceName.DatabaseName.SchemaName, or DBServiceName.DatabaseName.SchemaName.TableName * **dashboardFilterPattern**, **chartFilterPattern**, **dataModelFilterPattern**: Note that all of them support regex as include or exclude. E.g., "My dashboard, My dash.\*, .\*Dashboard". * **projectFilterPattern**: Filter the dashboards, charts and data sources by projects. Note that all of them support regex as include or exclude. E.g., "My project, My proj.\*, .\*Project". * **includeOwners**: Set the 'Include Owners' toggle to control whether to include owners to the ingested entity if the owner email matches with a user stored in the OM server as part of metadata ingestion. If the ingested entity already exists and has an owner, the owner will not be overwritten. * **includeTags**: Set the 'Include Tags' toggle to control whether to include tags in metadata ingestion. * **includeDataModels**: Set the 'Include Data Models' toggle to control whether to include tags as part of metadata ingestion. * **markDeletedDashboards**: Set the 'Mark Deleted Dashboards' toggle to flag dashboards as soft-deleted if they are not present anymore in the source system. * **Include Draft Dashboard (toggle)**: Set the 'Include Draft Dashboard' toggle to include draft dashboards. By default it will include draft dashboards. * **dataModelFilterPattern**: Regex exclude or include data models that matches the pattern. * **includeOwners**:Enabling a flag will replace the current owner with a new owner from the source during metadata ingestion, if the current owner is null. It is recommended to keep the flag enabled to obtain the owner information during the first metadata ingestion.`includeOwners` supports boolean value either true or false. * **markDeletedDashboards**: Optional configuration to soft delete dashboards in OpenMetadata if the source dashboards are deleted. Also, if the dashboard is deleted, all the associated entities like lineage, etc., with that dashboard will be deleted.`markDeletedDashboards` supports boolean value either true or false. * **markDeletedDataModels**: Optional configuration to soft delete data models in OpenMetadata if the source data models are deleted. Also, if the data models is deleted, all the associated entities like lineage, etc., with that data models will be deleted.`includeOwners` supports boolean value either true or false. * **includeTags**:Optional configuration to toggle the tags ingestion.`markDeletedDataModels` supports boolean value either true or false. * **includeDataModels**: Optional configuration to toggle the ingestion of data models.`includeDataModels` supports boolean value either true or false. * **includeDraftDashboard**: Optional Configuration to include/exclude draft dashboards. By default it will include draft dashboards.`includeDraftDashboard` supports boolean value either true or false. * **overrideMetadata**: Set the 'Override Metadata' toggle to control whether to override the existing metadata in the OpenMetadata server with the metadata fetched from the source. If the toggle is set to true, the metadata fetched from the source will override the existing metadata in the OpenMetadata server. If the toggle is set to false, the metadata fetched from the source will not override the existing metadata in the OpenMetadata server. This is applicable for fields like description, tags, owner and displayName.`overrideMetadata` supports boolean value either true or false. * **overrideLineage**: Set the 'Override Lineage' toggle to control whether to override the existing lineage.`overrideLineage` supports boolean value either true or false. #### Sink Configuration To send the metadata to OpenMetadata, it needs to be specified as `type: metadata-rest`. #### Workflow Configuration The main property here is the `openMetadataServerConfig`, where you can define the host and security provider of your OpenMetadata installation. **Logger Level** You can specify the `loggerLevel` depending on your needs. If you are trying to troubleshoot an ingestion, running with `DEBUG` will give you far more traces for identifying issues. **JWT Token** JWT tokens will allow your clients to authenticate against the OpenMetadata server. To enable JWT Tokens, you will get more details [here](https://docs.open-metadata.org/latest/deployment/security/enable-jwt-tokens) . You can refer to the JWT Troubleshooting section [link](https://docs.open-metadata.org/latest/deployment/security/jwt-troubleshooting) for any issues in your JWT configuration. **Store Service Connection** If set to `true` (default), we will store the sensitive information either encrypted via the Fernet Key in the database or externally, if you have configured any [Secrets Manager](https://docs.open-metadata.org/latest/deployment/secrets-manager) . If set to `false`, the service will be created, but the service connection information will only be used by the Ingestion Framework at runtime, and won't be sent to the OpenMetadata server. **Store Service Connection** If set to `true` (default), we will store the sensitive information either encrypted via the Fernet Key in the database or externally, if you have configured any [Secrets Manager](https://docs.open-metadata.org/latest/deployment/secrets-manager) . If set to `false`, the service will be created, but the service connection information will only be used by the Ingestion Framework at runtime, and won't be sent to the OpenMetadata server. **SSL Configuration** If you have added SSL to the [OpenMetadata server](https://docs.open-metadata.org/latest/deployment/security/enable-ssl) , then you will need to handle the certificates when running the ingestion too. You can either set `verifySSL` to `ignore`, or have it as `validate`, which will require you to set the `sslConfig.caCertificate` with a local path where your ingestion runs that points to the server certificate file. Find more information on how to troubleshoot SSL issues [here](https://docs.open-metadata.org/latest/deployment/security/enable-ssl/ssl-troubleshooting) . **ingestionPipelineFQN** Fully qualified name of ingestion pipeline, used to identify the current ingestion pipeline. filename.yamlCopy ### 2\. Run with the CLI First, we will need to save the YAML file. Afterward, and with all requirements installed, we can run: Note that from connector to connector, this recipe will always be the same. By updating the YAML configuration, you will be able to extract metadata from different sources. --- # Run the MicroStrategy Connector Externally We use cookies to improve site navigation, analyze site usage, and enhance your user experience. Click "Accept" to enable cookies or "Reject" to reject cookies. AcceptReject connectors No menu items for this category [Home](https://docs.open-metadata.org/latest) /[Connectors](https://docs.open-metadata.org/latest/connectors) /[Dashboard](https://docs.open-metadata.org/latest/connectors/dashboard) /[Microstrategy](https://docs.open-metadata.org/latest/connectors/dashboard/microstrategy) /[Yaml](https://docs.open-metadata.org/latest/connectors/dashboard/microstrategy/yaml) OpenMetadata Documentation ![MicroStrategy](https://docs.open-metadata.org/_next/image?url=%2Fimages%2Fconnectors%2Fmicrostrategy.webp&w=64&q=75) MicroStrategy ============= PROD Available In Feature List Dashboards Charts Owners Datamodels Lineage Tags Projects In this section, we provide guides and references to use the MicroStrategy connector. Configure and schedule MicroStrategy metadata and profiler workflows from the OpenMetadata UI: * [Requirements](https://docs.open-metadata.org/latest/connectors/dashboard/microstrategy/yaml#requirements) * [Metadata Ingestion](https://docs.open-metadata.org/latest/connectors/dashboard/microstrategy/yaml#metadata-ingestion) How to Run the Connector Externally ----------------------------------- To run the Ingestion via the UI you'll need to use the OpenMetadata Ingestion Container, which comes shipped with custom Airflow plugins to handle the workflow deployment. If, instead, you want to manage your workflows externally on your preferred orchestrator, you can check the following docs to run the Ingestion Framework **anywhere**. [#### External Schedulers\ \ Get more information about running the Ingestion Framework Externally](https://docs.open-metadata.org/latest/deployment/ingestion) Requirements ------------ To integrate MicroStrategy, ensure you are using OpenMetadata version 1.2.x or higher. ### Python Requirements We have support for Python versions 3.9-3.11 To run the MicroStrategy ingestion, you will need to install: Metadata Ingestion ------------------ All connectors are defined as JSON Schemas. [Here](https://github.com/open-metadata/OpenMetadata/blob/main/openmetadata-spec/src/main/resources/json/schema/entity/services/connections/dashboard/mstrConnection.json) you can find the structure to create a connection to MicroStrategy. In order to create and run a Metadata Ingestion workflow, we will follow the steps to create a YAML configuration able to connect to the source, process the Entities if needed, and reach the OpenMetadata server. The workflow is modeled around the following [JSON Schema](https://github.com/open-metadata/OpenMetadata/blob/main/openmetadata-spec/src/main/resources/json/schema/metadataIngestion/workflow.json) ### 1\. Define the YAML Config This is a sample config for MicroStrategy: #### Source Configuration - Service Connection * **Username**: Username to connect to MicroStrategy, e.g., user@organization.com. This user should have access to relevant dashboards and charts in MicroStrategy to fetch the metadata. * **Password**: Password of the user account to connect with MicroStrategy. * **Host Port**: This parameter specifies the host of the MicroStrategy instance. This should be specified as a URI string in the format http://hostname or https://hostname. For example, you might set it to https://demo.microstrategy.com. * **Project Name**: The name of the project within MicroStrategy that OpenMetadata will connect to, linking to the relevant dashboards and reports for metadata retrieval. * **Login Mode**: Login Mode for Microstrategy's REST API connection. You can authenticate with one of the following authentication modes: `Standard (1)`, `Anonymous (8)`. Default will be `Standard (1)`. If you're using demo account for Microstrategy, it will be needed to authenticate through loginMode `8`. #### Source Configuration - Source Config The `sourceConfig` is defined [here](https://github.com/open-metadata/OpenMetadata/blob/main/openmetadata-spec/src/main/resources/json/schema/metadataIngestion/dashboardServiceMetadataPipeline.json) : * **dbServicePrefixes**: List of service path prefixes for lineage matching. Supported formats: DBServiceName, DBServiceName.DatabaseName, DBServiceName.DatabaseName.SchemaName, or DBServiceName.DatabaseName.SchemaName.TableName * **dashboardFilterPattern**, **chartFilterPattern**, **dataModelFilterPattern**: Note that all of them support regex as include or exclude. E.g., "My dashboard, My dash.\*, .\*Dashboard". * **projectFilterPattern**: Filter the dashboards, charts and data sources by projects. Note that all of them support regex as include or exclude. E.g., "My project, My proj.\*, .\*Project". * **includeOwners**: Set the 'Include Owners' toggle to control whether to include owners to the ingested entity if the owner email matches with a user stored in the OM server as part of metadata ingestion. If the ingested entity already exists and has an owner, the owner will not be overwritten. * **includeTags**: Set the 'Include Tags' toggle to control whether to include tags in metadata ingestion. * **includeDataModels**: Set the 'Include Data Models' toggle to control whether to include tags as part of metadata ingestion. * **markDeletedDashboards**: Set the 'Mark Deleted Dashboards' toggle to flag dashboards as soft-deleted if they are not present anymore in the source system. * **Include Draft Dashboard (toggle)**: Set the 'Include Draft Dashboard' toggle to include draft dashboards. By default it will include draft dashboards. * **dataModelFilterPattern**: Regex exclude or include data models that matches the pattern. * **includeOwners**:Enabling a flag will replace the current owner with a new owner from the source during metadata ingestion, if the current owner is null. It is recommended to keep the flag enabled to obtain the owner information during the first metadata ingestion.`includeOwners` supports boolean value either true or false. * **markDeletedDashboards**: Optional configuration to soft delete dashboards in OpenMetadata if the source dashboards are deleted. Also, if the dashboard is deleted, all the associated entities like lineage, etc., with that dashboard will be deleted.`markDeletedDashboards` supports boolean value either true or false. * **markDeletedDataModels**: Optional configuration to soft delete data models in OpenMetadata if the source data models are deleted. Also, if the data models is deleted, all the associated entities like lineage, etc., with that data models will be deleted.`includeOwners` supports boolean value either true or false. * **includeTags**:Optional configuration to toggle the tags ingestion.`markDeletedDataModels` supports boolean value either true or false. * **includeDataModels**: Optional configuration to toggle the ingestion of data models.`includeDataModels` supports boolean value either true or false. * **includeDraftDashboard**: Optional Configuration to include/exclude draft dashboards. By default it will include draft dashboards.`includeDraftDashboard` supports boolean value either true or false. * **overrideMetadata**: Set the 'Override Metadata' toggle to control whether to override the existing metadata in the OpenMetadata server with the metadata fetched from the source. If the toggle is set to true, the metadata fetched from the source will override the existing metadata in the OpenMetadata server. If the toggle is set to false, the metadata fetched from the source will not override the existing metadata in the OpenMetadata server. This is applicable for fields like description, tags, owner and displayName.`overrideMetadata` supports boolean value either true or false. * **overrideLineage**: Set the 'Override Lineage' toggle to control whether to override the existing lineage.`overrideLineage` supports boolean value either true or false. #### Sink Configuration To send the metadata to OpenMetadata, it needs to be specified as `type: metadata-rest`. #### Workflow Configuration The main property here is the `openMetadataServerConfig`, where you can define the host and security provider of your OpenMetadata installation. **Logger Level** You can specify the `loggerLevel` depending on your needs. If you are trying to troubleshoot an ingestion, running with `DEBUG` will give you far more traces for identifying issues. **JWT Token** JWT tokens will allow your clients to authenticate against the OpenMetadata server. To enable JWT Tokens, you will get more details [here](https://docs.open-metadata.org/latest/deployment/security/enable-jwt-tokens) . You can refer to the JWT Troubleshooting section [link](https://docs.open-metadata.org/latest/deployment/security/jwt-troubleshooting) for any issues in your JWT configuration. **Store Service Connection** If set to `true` (default), we will store the sensitive information either encrypted via the Fernet Key in the database or externally, if you have configured any [Secrets Manager](https://docs.open-metadata.org/latest/deployment/secrets-manager) . If set to `false`, the service will be created, but the service connection information will only be used by the Ingestion Framework at runtime, and won't be sent to the OpenMetadata server. **Store Service Connection** If set to `true` (default), we will store the sensitive information either encrypted via the Fernet Key in the database or externally, if you have configured any [Secrets Manager](https://docs.open-metadata.org/latest/deployment/secrets-manager) . If set to `false`, the service will be created, but the service connection information will only be used by the Ingestion Framework at runtime, and won't be sent to the OpenMetadata server. **SSL Configuration** If you have added SSL to the [OpenMetadata server](https://docs.open-metadata.org/latest/deployment/security/enable-ssl) , then you will need to handle the certificates when running the ingestion too. You can either set `verifySSL` to `ignore`, or have it as `validate`, which will require you to set the `sslConfig.caCertificate` with a local path where your ingestion runs that points to the server certificate file. Find more information on how to troubleshoot SSL issues [here](https://docs.open-metadata.org/latest/deployment/security/enable-ssl/ssl-troubleshooting) . **ingestionPipelineFQN** Fully qualified name of ingestion pipeline, used to identify the current ingestion pipeline. filename.yamlCopy ### 2\. Run with the CLI First, we will need to save the YAML file. Afterward, and with all requirements installed, we can run: Note that from connector to connector, this recipe will always be the same. By updating the YAML configuration, you will be able to extract metadata from different sources. --- # Run the Mode Connector Externally We use cookies to improve site navigation, analyze site usage, and enhance your user experience. Click "Accept" to enable cookies or "Reject" to reject cookies. AcceptReject connectors No menu items for this category [Home](https://docs.open-metadata.org/latest) /[Connectors](https://docs.open-metadata.org/latest/connectors) /[Dashboard](https://docs.open-metadata.org/latest/connectors/dashboard) /[Mode](https://docs.open-metadata.org/latest/connectors/dashboard/mode) /[Yaml](https://docs.open-metadata.org/latest/connectors/dashboard/mode/yaml) OpenMetadata Documentation ![Mode](https://docs.open-metadata.org/_next/image?url=%2Fimages%2Fconnectors%2Fmode.webp&w=64&q=75) Mode ==== PROD Available In Feature List Dashboards Charts Lineage Owners Tags Datamodels Projects In this section, we provide guides and references to use the Mode connector. Configure and schedule Mode metadata and profiler workflows from the OpenMetadata UI: * [Requirements](https://docs.open-metadata.org/latest/connectors/dashboard/mode/yaml#requirements) * [Metadata Ingestion](https://docs.open-metadata.org/latest/connectors/dashboard/mode/yaml#metadata-ingestion) How to Run the Connector Externally ----------------------------------- To run the Ingestion via the UI you'll need to use the OpenMetadata Ingestion Container, which comes shipped with custom Airflow plugins to handle the workflow deployment. If, instead, you want to manage your workflows externally on your preferred orchestrator, you can check the following docs to run the Ingestion Framework **anywhere**. [#### External Schedulers\ \ Get more information about running the Ingestion Framework Externally](https://docs.open-metadata.org/latest/deployment/ingestion) Requirements ------------ OpenMetadata relies on Mode's API, which is exclusive to members of the Mode Business Workspace. This means that only resources that belong to a Mode Business Workspace can be accessed via the API. ### Python Requirements We have support for Python versions 3.9-3.11 To run the Mode ingestion, you will need to install: Metadata Ingestion ------------------ All connectors are defined as JSON Schemas. [Here](https://github.com/open-metadata/OpenMetadata/blob/main/openmetadata-spec/src/main/resources/json/schema/entity/services/connections/dashboard/modeConnection.json) you can find the structure to create a connection to Mode. In order to create and run a Metadata Ingestion workflow, we will follow the steps to create a YAML configuration able to connect to the source, process the Entities if needed, and reach the OpenMetadata server. The workflow is modeled around the following [JSON Schema](https://github.com/open-metadata/OpenMetadata/blob/main/openmetadata-spec/src/main/resources/json/schema/metadataIngestion/workflow.json) ### 1\. Define the YAML Config This is a sample config for Mode: #### Source Configuration - Service Connection **hostPort**: Host and Port Mode Dashboard. The hostPort parameter specifies the host and port of the Mode server. This should be specified as a string in the format `https://app.mode.com`. **accessToken**: Access Token for Mode Dashboard. Get the Access Token by following below mentioned steps: * Navigate to your Mode homepage. * Click on your name in the upper left corner and click My Account. * Click on API Tokens on the left side. * To generate a new API token and password, enter a token name and click `Create token`. * Copy the generated access token and password. For detailed information visit [here](https://mode.com/developer/api-reference/introduction/) . **accessTokenPassword**: Access Token Password for Mode Dashboard. Copy the access token password from the step above where a new token is generated. For detailed information visit [here](https://mode.com/developer/api-reference/introduction/) . **workspaceName**: Mode Workspace Name. Name of the mode workspace from where the metadata is to be fetched. **filterQueryParam**: Filter query parameter for some of the Mode API calls. #### Source Configuration - Source Config The `sourceConfig` is defined [here](https://github.com/open-metadata/OpenMetadata/blob/main/openmetadata-spec/src/main/resources/json/schema/metadataIngestion/dashboardServiceMetadataPipeline.json) : * **dbServicePrefixes**: List of service path prefixes for lineage matching. Supported formats: DBServiceName, DBServiceName.DatabaseName, DBServiceName.DatabaseName.SchemaName, or DBServiceName.DatabaseName.SchemaName.TableName * **dashboardFilterPattern**, **chartFilterPattern**, **dataModelFilterPattern**: Note that all of them support regex as include or exclude. E.g., "My dashboard, My dash.\*, .\*Dashboard". * **projectFilterPattern**: Filter the dashboards, charts and data sources by projects. Note that all of them support regex as include or exclude. E.g., "My project, My proj.\*, .\*Project". * **includeOwners**: Set the 'Include Owners' toggle to control whether to include owners to the ingested entity if the owner email matches with a user stored in the OM server as part of metadata ingestion. If the ingested entity already exists and has an owner, the owner will not be overwritten. * **includeTags**: Set the 'Include Tags' toggle to control whether to include tags in metadata ingestion. * **includeDataModels**: Set the 'Include Data Models' toggle to control whether to include tags as part of metadata ingestion. * **markDeletedDashboards**: Set the 'Mark Deleted Dashboards' toggle to flag dashboards as soft-deleted if they are not present anymore in the source system. * **Include Draft Dashboard (toggle)**: Set the 'Include Draft Dashboard' toggle to include draft dashboards. By default it will include draft dashboards. * **dataModelFilterPattern**: Regex exclude or include data models that matches the pattern. * **includeOwners**:Enabling a flag will replace the current owner with a new owner from the source during metadata ingestion, if the current owner is null. It is recommended to keep the flag enabled to obtain the owner information during the first metadata ingestion.`includeOwners` supports boolean value either true or false. * **markDeletedDashboards**: Optional configuration to soft delete dashboards in OpenMetadata if the source dashboards are deleted. Also, if the dashboard is deleted, all the associated entities like lineage, etc., with that dashboard will be deleted.`markDeletedDashboards` supports boolean value either true or false. * **markDeletedDataModels**: Optional configuration to soft delete data models in OpenMetadata if the source data models are deleted. Also, if the data models is deleted, all the associated entities like lineage, etc., with that data models will be deleted.`includeOwners` supports boolean value either true or false. * **includeTags**:Optional configuration to toggle the tags ingestion.`markDeletedDataModels` supports boolean value either true or false. * **includeDataModels**: Optional configuration to toggle the ingestion of data models.`includeDataModels` supports boolean value either true or false. * **includeDraftDashboard**: Optional Configuration to include/exclude draft dashboards. By default it will include draft dashboards.`includeDraftDashboard` supports boolean value either true or false. * **overrideMetadata**: Set the 'Override Metadata' toggle to control whether to override the existing metadata in the OpenMetadata server with the metadata fetched from the source. If the toggle is set to true, the metadata fetched from the source will override the existing metadata in the OpenMetadata server. If the toggle is set to false, the metadata fetched from the source will not override the existing metadata in the OpenMetadata server. This is applicable for fields like description, tags, owner and displayName.`overrideMetadata` supports boolean value either true or false. * **overrideLineage**: Set the 'Override Lineage' toggle to control whether to override the existing lineage.`overrideLineage` supports boolean value either true or false. #### Sink Configuration To send the metadata to OpenMetadata, it needs to be specified as `type: metadata-rest`. #### Workflow Configuration The main property here is the `openMetadataServerConfig`, where you can define the host and security provider of your OpenMetadata installation. **Logger Level** You can specify the `loggerLevel` depending on your needs. If you are trying to troubleshoot an ingestion, running with `DEBUG` will give you far more traces for identifying issues. **JWT Token** JWT tokens will allow your clients to authenticate against the OpenMetadata server. To enable JWT Tokens, you will get more details [here](https://docs.open-metadata.org/latest/deployment/security/enable-jwt-tokens) . You can refer to the JWT Troubleshooting section [link](https://docs.open-metadata.org/latest/deployment/security/jwt-troubleshooting) for any issues in your JWT configuration. **Store Service Connection** If set to `true` (default), we will store the sensitive information either encrypted via the Fernet Key in the database or externally, if you have configured any [Secrets Manager](https://docs.open-metadata.org/latest/deployment/secrets-manager) . If set to `false`, the service will be created, but the service connection information will only be used by the Ingestion Framework at runtime, and won't be sent to the OpenMetadata server. **Store Service Connection** If set to `true` (default), we will store the sensitive information either encrypted via the Fernet Key in the database or externally, if you have configured any [Secrets Manager](https://docs.open-metadata.org/latest/deployment/secrets-manager) . If set to `false`, the service will be created, but the service connection information will only be used by the Ingestion Framework at runtime, and won't be sent to the OpenMetadata server. **SSL Configuration** If you have added SSL to the [OpenMetadata server](https://docs.open-metadata.org/latest/deployment/security/enable-ssl) , then you will need to handle the certificates when running the ingestion too. You can either set `verifySSL` to `ignore`, or have it as `validate`, which will require you to set the `sslConfig.caCertificate` with a local path where your ingestion runs that points to the server certificate file. Find more information on how to troubleshoot SSL issues [here](https://docs.open-metadata.org/latest/deployment/security/enable-ssl/ssl-troubleshooting) . **ingestionPipelineFQN** Fully qualified name of ingestion pipeline, used to identify the current ingestion pipeline. filename.yamlCopy ### 2\. Run with the CLI First, we will need to save the YAML file. Afterward, and with all requirements installed, we can run: Note that from connector to connector, this recipe will always be the same. By updating the YAML configuration, you will be able to extract metadata from different sources. --- # Run the Nifi Connector Externally | Official Documentation We use cookies to improve site navigation, analyze site usage, and enhance your user experience. Click "Accept" to enable cookies or "Reject" to reject cookies. AcceptReject connectors No menu items for this category [Home](https://docs.open-metadata.org/latest) /[Connectors](https://docs.open-metadata.org/latest/connectors) /[Pipeline](https://docs.open-metadata.org/latest/connectors/pipeline) /[Nifi](https://docs.open-metadata.org/latest/connectors/pipeline/nifi) /[Yaml](https://docs.open-metadata.org/latest/connectors/pipeline/nifi/yaml) OpenMetadata Documentation ![Nifi](https://docs.open-metadata.org/_next/image?url=%2Fimages%2Fconnectors%2Fdefault-service-icon.webp&w=64&q=75) Nifi ==== PROD Available In Feature List Pipelines Usage Lineage Pipeline Status Owners Tags In this section, we provide guides and references to use the NiFi connector. Configure and schedule NiFi metadata and profiler workflows from the OpenMetadata UI: * [Requirements](https://docs.open-metadata.org/latest/connectors/pipeline/nifi/yaml#requirements) * [Metadata Ingestion](https://docs.open-metadata.org/latest/connectors/pipeline/nifi/yaml#metadata-ingestion) How to Run the Connector Externally ----------------------------------- To run the Ingestion via the UI you'll need to use the OpenMetadata Ingestion Container, which comes shipped with custom Airflow plugins to handle the workflow deployment. If, instead, you want to manage your workflows externally on your preferred orchestrator, you can check the following docs to run the Ingestion Framework **anywhere**. [#### External Schedulers\ \ Get more information about running the Ingestion Framework Externally](https://docs.open-metadata.org/latest/deployment/ingestion) Requirements ------------ ### Python Requirements We have support for Python versions 3.9-3.11 To run the NiFi ingestion, you will need to install: Metadata Ingestion ------------------ All connectors are defined as JSON Schemas. [Here](https://github.com/open-metadata/OpenMetadata/blob/main/openmetadata-spec/src/main/resources/json/schema/entity/services/connections/pipeline/nifiConnection.json) you can find the structure to create a connection to NiFi. In order to create and run a Metadata Ingestion workflow, we will follow the steps to create a YAML configuration able to connect to the source, process the Entities if needed, and reach the OpenMetadata server. The workflow is modeled around the following [JSON Schema](https://github.com/open-metadata/OpenMetadata/blob/main/openmetadata-spec/src/main/resources/json/schema/metadataIngestion/workflow.json) ### 1\. Define the YAML Config This is a sample config for NiFi: #### Source Configuration - Service Connection **hostPort**: Pipeline Service Management UI URL **NiFiConfig**: one of **1.** Using Basic authentication \- **username**: Username to connect to NiFi. This user should be able to send request to the NiFi API and access the `Resources` endpoint. \- **password**: Password to connect to NiFi. \- **verifySSL**: Whether SSL verification should be perform when authenticating. **2.** Using client certificate authentication \- **certificateAuthorityPath**: Path to the certificate authority (CA) file. This is the certificate used to store and issue your digital certificate. This is an optional parameter. If omitted SSL verification will be skipped; this can present some sever security issue. **important**: This file should be accessible from where the ingestion workflow is running. For example, if you are using OpenMetadata Ingestion Docker container, this file should be in this container. \- **clientCertificatePath**: Path to the certificate client file. **important**: This file should be accessible from where the ingestion workflow is running. For example, if you are using OpenMetadata Ingestion Docker container, this file should be in this container. \- **clientkeyPath**: Path to the client key file. **important**: This file should be accessible from where the ingestion workflow is running. For example, if you are using OpenMetadata Ingestion Docker container, this file should be in this container. #### Source Configuration - Source Config The `sourceConfig` is defined [here](https://github.com/open-metadata/OpenMetadata/blob/main/openmetadata-spec/src/main/resources/json/schema/metadataIngestion/pipelineServiceMetadataPipeline.json) : * **dbServiceNames**: Database Service Name for the creation of lineage, if the source supports it. * **includeTags**: Set the 'Include Tags' toggle to control whether to include tags as part of metadata ingestion. * **includeUnDeployedPipelines**: Set the 'Include UnDeployed Pipelines' toggle to control whether to include un-deployed pipelines as part of metadata ingestion. By default it is set to `true` * **markDeletedPipelines**: Set the Mark Deleted Pipelines toggle to flag pipelines as soft-deleted if they are not present anymore in the source system. * **pipelineFilterPattern** and **chartFilterPattern**: Note that the `pipelineFilterPattern` and `chartFilterPattern` both support regex as include or exclude. * **includeOwners**: Set the 'Include Owners' toggle to control whether to include owners to the ingested entity if the owner email matches with a user stored in the OM server as part of metadata ingestion. If the ingested entity already exists and has an owner, the owner will not be overwritten.It supports boolean values either `true` or `false`. * **overrideLineage**: Set the 'Override Lineage' toggle to control whether to override the existing lineage. It supports boolean values either `true` or `false`. * **overrideMetadata**: Set the 'Override Metadata' toggle to control whether to override the existing metadata in the OpenMetadata server with the metadata fetched from the source. If the toggle is set to true, the metadata fetched from the source will override the existing metadata in the OpenMetadata server. If the toggle is set to false, the metadata fetched from the source will not override the existing metadata in the OpenMetadata server. This is applicable for fields like description, tags, owner and displayName. It supports boolean values either `true` or `false`. #### Sink Configuration To send the metadata to OpenMetadata, it needs to be specified as `type: metadata-rest`. #### Workflow Configuration The main property here is the `openMetadataServerConfig`, where you can define the host and security provider of your OpenMetadata installation. **Logger Level** You can specify the `loggerLevel` depending on your needs. If you are trying to troubleshoot an ingestion, running with `DEBUG` will give you far more traces for identifying issues. **JWT Token** JWT tokens will allow your clients to authenticate against the OpenMetadata server. To enable JWT Tokens, you will get more details [here](https://docs.open-metadata.org/latest/deployment/security/enable-jwt-tokens) . You can refer to the JWT Troubleshooting section [link](https://docs.open-metadata.org/latest/deployment/security/jwt-troubleshooting) for any issues in your JWT configuration. **Store Service Connection** If set to `true` (default), we will store the sensitive information either encrypted via the Fernet Key in the database or externally, if you have configured any [Secrets Manager](https://docs.open-metadata.org/latest/deployment/secrets-manager) . If set to `false`, the service will be created, but the service connection information will only be used by the Ingestion Framework at runtime, and won't be sent to the OpenMetadata server. **Store Service Connection** If set to `true` (default), we will store the sensitive information either encrypted via the Fernet Key in the database or externally, if you have configured any [Secrets Manager](https://docs.open-metadata.org/latest/deployment/secrets-manager) . If set to `false`, the service will be created, but the service connection information will only be used by the Ingestion Framework at runtime, and won't be sent to the OpenMetadata server. **SSL Configuration** If you have added SSL to the [OpenMetadata server](https://docs.open-metadata.org/latest/deployment/security/enable-ssl) , then you will need to handle the certificates when running the ingestion too. You can either set `verifySSL` to `ignore`, or have it as `validate`, which will require you to set the `sslConfig.caCertificate` with a local path where your ingestion runs that points to the server certificate file. Find more information on how to troubleshoot SSL issues [here](https://docs.open-metadata.org/latest/deployment/security/enable-ssl/ssl-troubleshooting) . **ingestionPipelineFQN** Fully qualified name of ingestion pipeline, used to identify the current ingestion pipeline. filename.yamlCopy ### 2\. Run with the CLI First, we will need to save the YAML file. Afterward, and with all requirements installed, we can run: Note that from connector to connector, this recipe will always be the same. By updating the YAML configuration, you will be able to extract metadata from different sources. --- # Azure SSO for Docker | OpenMetadata Deployment Guide We use cookies to improve site navigation, analyze site usage, and enhance your user experience. Click "Accept" to enable cookies or "Reject" to reject cookies. AcceptReject ![](https://static.scarf.sh/a.png?x-pxid=8472f2b3-d307-4935-a5ed-3f9d3982c15d) deployment No menu items for this category [Home](https://docs.open-metadata.org/latest) /[Deployment](https://docs.open-metadata.org/latest/deployment) /[Security](https://docs.open-metadata.org/latest/deployment/security) /[Auth 0](https://docs.open-metadata.org/latest/deployment/security/auth0) /[Implicit Flow](https://docs.open-metadata.org/latest/deployment/security/auth0/implicit-flow) OpenMetadata Documentation Implicit Flow ============= ### Step 1: Create a New Application * Once you are on the Dashboard page, click on `Applications > Applications` available on the left-hand side panel. ![create-app](https://docs.open-metadata.org/images/v1.11/deployment/security/auth0/create-new-app-1.png) * Click on `Create Application`. ![create-app](https://docs.open-metadata.org/images/v1.11/deployment/security/auth0/create-new-app-2.png) * Enter the Application name. * Choose an application type and click on `Create`. ![create-app](https://docs.open-metadata.org/images/v1.11/deployment/security/auth0/create-new-app-3.png) ### Step 2: Where to Find the Credentials * Navigate to the Settings tab. * You will find your `Client ID` and `Domain`. ![credentials](https://docs.open-metadata.org/images/v1.11/deployment/security/auth0/credentials.png) After the applying these steps, you can update the configuration of your deployment: [Auth0 SSO - Public\ \ Configure Auth0 SSO using the Public client type.](https://docs.open-metadata.org/latest/deployment/security/auth0/public-client) [Auth0 SSO - Confidential\ \ Configure Auth0 SSO using the Confidential client type.](https://docs.open-metadata.org/latest/deployment/security/auth0/confidential-client) [Auth\ \ Go to Auth0 Configuration](https://docs.open-metadata.org/latest/deployment/security/auth0) --- # Glue Pipeline | OpenMetadata Data Integration Pipeline We use cookies to improve site navigation, analyze site usage, and enhance your user experience. Click "Accept" to enable cookies or "Reject" to reject cookies. AcceptReject connectors No menu items for this category [Home](https://docs.open-metadata.org/latest) /[Connectors](https://docs.open-metadata.org/latest/connectors) /[Pipeline](https://docs.open-metadata.org/latest/connectors/pipeline) /[Glue Pipeline](https://docs.open-metadata.org/latest/connectors/pipeline/glue-pipeline) OpenMetadata Documentation ![Glue](https://docs.open-metadata.org/_next/image?url=%2Fimages%2Fconnectors%2Fglue.webp&w=64&q=75) Glue ==== PROD Available In Feature List Pipelines Pipeline Status Usage Lineage Owners Tags In this section, we provide guides and references to use the Glue connector. Configure and schedule Glue metadata and profiler workflows from the OpenMetadata UI: * [Requirements](https://docs.open-metadata.org/latest/connectors/pipeline/glue-pipeline#requirements) * [Metadata Ingestion](https://docs.open-metadata.org/latest/connectors/pipeline/glue-pipeline#metadata-ingestion) * [Troubleshooting](https://docs.open-metadata.org/latest/connectors/pipeline/glue-pipeline/troubleshooting) Ingestion Deployment -------------------- To run the Ingestion via the UI you'll need to use the OpenMetadata Ingestion Container, which comes shipped with custom Airflow plugins to handle the workflow deployment. If you want to install it manually in an already existing Airflow host, you can follow [this](https://docs.open-metadata.org/latest/deployment/ingestion/openmetadata) guide. If you don't want to use the OpenMetadata Ingestion container to configure the workflows via the UI, then you can check the following docs to run the Ingestion Framework in any orchestrator externally. [#### Run Connectors from the OpenMetadata UI\ \ Learn how to manage your deployment to run connectors from the UI](https://docs.open-metadata.org/latest/deployment/ingestion/openmetadata) [#### Run the Connector Externally\ \ Get the YAML to run the ingestion externally](https://docs.open-metadata.org/latest/connectors/pipeline/glue-pipeline/yaml) [#### External Schedulers\ \ Get more information about running the Ingestion Framework Externally](https://docs.open-metadata.org/latest/deployment/ingestion) Requirements ------------ The Glue connector ingests metadata through AWS [Boto3](https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/glue.html) Client. We will ingest Workflows, its jobs and their run status. The user must have the following permissions for the ingestion to run successfully: * `glue:ListWorkflows` * `glue:GetWorkflow` * `glue:GetJobRuns` Metadata Ingestion ------------------ #### 1\. Visit the Services Page Click `Settings` in the side navigation bar and then `Services`. The first step is to ingest the metadata from your sources. To do that, you first need to create a Service connection first. This Service will be the bridge between OpenMetadata and your source system. Once a Service is created, it can be used to configure your ingestion workflows. ![Visit Services Page](https://docs.open-metadata.org/images/v1.11/connectors/visit-services-page.png) Select your Service Type and Add a New Service #### 2\. Create a New Service Click on _Add New Service_ to start the Service creation. ![Create a new Service](https://docs.open-metadata.org/images/v1.11/connectors/create-new-service.png) Add a new Service from the Services page #### 3\. Select the Service Type Select Glue Pipeline as the Service type and click _Next_. ![Select Service](https://docs.open-metadata.org/images/v1.11/connectors/gluepipeline/select-service.png) Select your Service from the list #### 4\. Name and Describe your Service Provide a name and description for your Service. #### Service Name OpenMetadata uniquely identifies Services by their **Service Name**. Provide a name that distinguishes your deployment from other Services, including the other Glue Pipeline Services that you might be ingesting metadata from. Note that when the name is set, it cannot be changed. ![Add New Service](https://docs.open-metadata.org/images/v1.11/connectors/gluepipeline/add-new-service.png) Provide a Name and description for your Service #### 5\. Configure the Service Connection In this step, we will configure the connection settings required for Glue Pipeline. Please follow the instructions below to properly configure the Service to read from your sources. You will also find helper documentation on the right-hand side panel in the UI. ![Configure Service connection](https://docs.open-metadata.org/images/v1.11/connectors/gluepipeline/service-connection.png) Configure the Service connection by filling the form #### Connection Details * **AWS Access Key ID** & **AWS Secret Access Key**: When you interact with AWS, you specify your AWS security credentials to verify who you are and whether you have permission to access the resources that you are requesting. AWS uses the security credentials to authenticate and authorize your requests ([docs](https://docs.aws.amazon.com/IAM/latest/UserGuide/security-creds.html) ). Access keys consist of two parts: An **access key ID** (for example, `AKIAIOSFODNN7EXAMPLE`), and a **secret access key** (for example, `wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY`). You must use both the access key ID and secret access key together to authenticate your requests. You can find further information on how to manage your access keys [here](https://docs.aws.amazon.com/IAM/latest/UserGuide/id_credentials_access-keys.html) . * **AWS Region**: Each AWS Region is a separate geographic area in which AWS clusters data centers ([docs](https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/Concepts.RegionsAndAvailabilityZones.html) ). As AWS can have instances in multiple regions, we need to know the region the service you want reach belongs to. Note that the AWS Region is the only required parameter when configuring a connection. When connecting to the services programmatically, there are different ways in which we can extract and use the rest of AWS configurations. You can find further information about configuring your credentials [here](https://boto3.amazonaws.com/v1/documentation/api/latest/guide/credentials.html#configuring-credentials) . * **AWS Session Token (optional)**: If you are using temporary credentials to access your services, you will need to inform the AWS Access Key ID and AWS Secrets Access Key. Also, these will include an AWS Session Token. You can find more information on [Using temporary credentials with AWS resources](https://docs.aws.amazon.com/IAM/latest/UserGuide/id_credentials_temp_use-resources.html) . * **Endpoint URL (optional)**: To connect programmatically to an AWS service, you use an endpoint. An _endpoint_ is the URL of the entry point for an AWS web service. The AWS SDKs and the AWS Command Line Interface (AWS CLI) automatically use the default endpoint for each service in an AWS Region. But you can specify an alternate endpoint for your API requests. Find more information on [AWS service endpoints](https://docs.aws.amazon.com/general/latest/gr/rande.html) . * **Profile Name**: A named profile is a collection of settings and credentials that you can apply to a AWS CLI command. When you specify a profile to run a command, the settings and credentials are used to run that command. Multiple named profiles can be stored in the config and credentials files. You can inform this field if you'd like to use a profile other than `default`. Find here more information about [Named profiles for the AWS CLI](https://docs.aws.amazon.com/cli/latest/userguide/cli-configure-profiles.html) . * **Assume Role Arn**: Typically, you use `AssumeRole` within your account or for cross-account access. In this field you'll set the `ARN` (Amazon Resource Name) of the policy of the other account. A user who wants to access a role in a different account must also have permissions that are delegated from the account administrator. The administrator must attach a policy that allows the user to call `AssumeRole` for the `ARN` of the role in the other account. This is a required field if you'd like to `AssumeRole`. Find more information on [AssumeRole](https://docs.aws.amazon.com/STS/latest/APIReference/API_AssumeRole.html) . When using Assume Role authentication, ensure you provide the following details: * **AWS Region**: Specify the AWS region for your deployment. * **Assume Role ARN**: Provide the ARN of the role in your AWS account that OpenMetadata will assume. * **Assume Role Session Name**: An identifier for the assumed role session. Use the role session name to uniquely identify a session when the same role is assumed by different principals or for different reasons. By default, we'll use the name `OpenMetadataSession`. Find more information about the [Role Session Name](https://docs.aws.amazon.com/STS/latest/APIReference/API_AssumeRole.html#:~:text=An%20identifier%20for%20the%20assumed%20role%20session.) . * **Assume Role Source Identity**: The source identity specified by the principal that is calling the `AssumeRole` operation. You can use source identity information in AWS CloudTrail logs to determine who took actions with a role. Find more information about [Source Identity](https://docs.aws.amazon.com/STS/latest/APIReference/API_AssumeRole.html#:~:text=Required%3A%20No-,SourceIdentity,-The%20source%20identity) . #### 6\. Test the Connection Once the credentials have been added, click on _Test Connection_ and _Save_ the changes. ![Test Connection](https://docs.open-metadata.org/images/v1.11/connectors/test-connection.png) Test the connection and save the Service #### 7\. Configure Metadata Ingestion In this step we will configure the metadata ingestion pipeline, Please follow the instructions below ![Configure Metadata Ingestion](https://docs.open-metadata.org/images/v1.11/connectors/configure-metadata-ingestion-pipeline.png) Configure Metadata Ingestion Page #### Metadata Ingestion Options * **Name**: This field refers to the name of ingestion pipeline, you can customize the name or use the generated name. * **Pipeline Filter Pattern (Optional)**: Use to pipeline filter patterns to control whether or not to include pipeline as part of metadata ingestion. * **Include**: Explicitly include pipeline by adding a list of comma-separated regular expressions to the Include field. OpenMetadata will include all pipeline with names matching one or more of the supplied regular expressions. All other schemas will be excluded. * **Exclude**: Explicitly exclude pipeline by adding a list of comma-separated regular expressions to the Exclude field. OpenMetadata will exclude all pipeline with names matching one or more of the supplied regular expressions. All other schemas will be included. * **Include lineage (toggle)**: Set the Include lineage toggle to control whether to include lineage between pipelines and data sources as part of metadata ingestion. * **Enable Debug Log (toggle)**: Set the Enable Debug Log toggle to set the default log level to debug. * **Mark Deleted Pipelines (toggle)**: Set the Mark Deleted Pipelines toggle to flag pipelines as soft-deleted if they are not present anymore in the source system. #### 8\. Schedule the Ingestion and Deploy Scheduling can be set up at an hourly, daily, weekly, or manual cadence. The timezone is in UTC. Select a Start Date to schedule for ingestion. It is optional to add an End Date. Review your configuration settings. If they match what you intended, click Deploy to create the service and schedule metadata ingestion. If something doesn't look right, click the Back button to return to the appropriate step and change the settings as needed. After configuring the workflow, you can click on Deploy to create the pipeline. ![Schedule the Workflow](https://docs.open-metadata.org/images/v1.11/connectors/schedule.png) Schedule the Ingestion Pipeline and Deploy #### 9\. View the Ingestion Pipeline Once the workflow has been successfully deployed, you can view the Ingestion Pipeline running from the Service Page. ![View Ingestion Pipeline](https://docs.open-metadata.org/images/v1.11/connectors/view-ingestion-pipeline.png) View the Ingestion Pipeline from the Service Page --- # Run the Oracle Connector Externally We use cookies to improve site navigation, analyze site usage, and enhance your user experience. Click "Accept" to enable cookies or "Reject" to reject cookies. AcceptReject connectors No menu items for this category [Home](https://docs.open-metadata.org/latest) /[Connectors](https://docs.open-metadata.org/latest/connectors) /[Database](https://docs.open-metadata.org/latest/connectors/database) /[Oracle](https://docs.open-metadata.org/latest/connectors/database/oracle) /[Yaml](https://docs.open-metadata.org/latest/connectors/database/oracle/yaml) OpenMetadata Documentation ![Oracle](https://docs.open-metadata.org/_next/image?url=%2Fimages%2Fconnectors%2Foracle.webp&w=64&q=75) Oracle ====== PROD Available In Feature List Metadata Query Usage Data Profiler Data Quality dbt Lineage Column-level Lineage Stored Procedures Sample Data Reverse Metadata (Collate Only) Auto-Classification Owners Tags In this section, we provide guides and references to use the Oracle connector. Configure and schedule Oracle metadata and profiler workflows from the OpenMetadata UI: * [Requirements](https://docs.open-metadata.org/latest/connectors/database/oracle/yaml#requirements) * [Metadata Ingestion](https://docs.open-metadata.org/latest/connectors/database/oracle/yaml#metadata-ingestion) * [Data Profiler](https://docs.open-metadata.org/latest/connectors/database/oracle/yaml#data-profiler) * [Data Quality](https://docs.open-metadata.org/latest/connectors/database/oracle/yaml#data-quality) * [Lineage](https://docs.open-metadata.org/latest/connectors/database/oracle/yaml#lineage) * [dbt Integration](https://docs.open-metadata.org/latest/connectors/database/oracle/yaml#dbt-integration) How to Run the Connector Externally ----------------------------------- To run the Ingestion via the UI you'll need to use the OpenMetadata Ingestion Container, which comes shipped with custom Airflow plugins to handle the workflow deployment. If, instead, you want to manage your workflows externally on your preferred orchestrator, you can check the following docs to run the Ingestion Framework **anywhere**. [#### External Schedulers\ \ Get more information about running the Ingestion Framework Externally](https://docs.open-metadata.org/latest/deployment/ingestion) Requirements ------------ **Note**: To retrieve metadata from an Oracle database, we use the `python-oracledb` library, which provides support for versions 12c, 18c, 19c, and 21c. To ingest metadata from oracle user must have `CREATE SESSION` privilege for the user. If you don't want to create a role, and directly give permissions to the user, you can take a look at an example given below. **Note**: With just these permissions, your user should be able to ingest the metadata, but not the `Profiler & Data Quality`, you should grant `SELECT` permissions to the tables you are interested in for the `Profiler & Data Quality` features to work. You can find further information [here](https://docs.oracle.com/javadb/10.8.3.0/ref/rrefsqljgrant.html) . Note that there is no routine out of the box in Oracle to grant SELECT to a full schema. ### Python Requirements We have support for Python versions 3.9-3.11 To run the Oracle ingestion, you will need to install: Metadata Ingestion ------------------ All connectors are defined as JSON Schemas. [Here](https://github.com/open-metadata/OpenMetadata/blob/main/openmetadata-spec/src/main/resources/json/schema/entity/services/connections/database/oracleConnection.json) you can find the structure to create a connection to Oracle. In order to create and run a Metadata Ingestion workflow, we will follow the steps to create a YAML configuration able to connect to the source, process the Entities if needed, and reach the OpenMetadata server. The workflow is modeled around the following [JSON Schema](https://github.com/open-metadata/OpenMetadata/blob/main/openmetadata-spec/src/main/resources/json/schema/metadataIngestion/workflow.json) ### 1\. Define the YAML Config This is a sample config for Oracle: #### Source Configuration - Service Connection **username**: Specify the User to connect to Oracle. It should have enough privileges to read all the metadata. **password**: Password to connect to Oracle. **hostPort**: Enter the fully qualified hostname and port number for your Oracle deployment in the Host and Port field. **oracleConnectionType** : * **oracleServiceName**: The Oracle Service name is the TNS alias that you give when you remotely connect to your database and this Service name is recorded in tnsnames. * **databaseSchema**: The name of the database schema available in Oracle that you want to connect with. * **Oracle instant client directory**: The directory pointing to where the `instantclient` binaries for Oracle are located. In the ingestion Docker image we provide them by default at `/instantclient`. If this parameter is informed (it is by default), we will run the [thick oracle client](https://python-oracledb.readthedocs.io/en/latest/user_guide/initialization.html#initializing-python-oracledb) . We are shipping the binaries for ARM and AMD architectures from [here](https://www.oracle.com/database/technologies/instant-client/linux-x86-64-downloads.html) and [here](https://www.oracle.com/database/technologies/instant-client/linux-arm-aarch64-downloads.html) for the instant client version 19. **databaseName**: Optional name to give to the database in OpenMetadata. If left blank, we will use default as the database name. It is recommended to use the database name same as the SID, This ensures accurate results and proper identification of tables during profiling, data quality checks and dbt workflow. #### Source Configuration - Source Config The `sourceConfig` is defined [here](https://github.com/open-metadata/OpenMetadata/blob/main/openmetadata-spec/src/main/resources/json/schema/metadataIngestion/databaseServiceMetadataPipeline.json) : **markDeletedTables**: To flag tables as soft-deleted if they are not present anymore in the source system. **markDeletedStoredProcedures**: Optional configuration to soft delete stored procedures in OpenMetadata if the source stored procedures are deleted. Also, if the stored procedures is deleted, all the associated entities like lineage, etc., with that stored procedures will be deleted. **markDeletedSchemas**: Optional configuration to soft delete schemas stored in OpenMetadata if the source schema is deleted. Setting this flag to true will only keep filtered schema and delete any other schemas that do not match schemaFilterPattern or do not exist at source. **markDeletedDatabases**: Optional configuration to soft delete databases stored in OpenMetadata if the source database is deleted. Setting this flag to true will only keep filtered databases and delete any other databases that do not match databaseFilterPattern or do not exist at source. **includeTables**: true or false, to ingest table data. Default is true. **includeViews**: true or false, to ingest views definitions. **includeTags**: Optional configuration to toggle the tags ingestion. **includeOwners**: Set the 'Include Owners' toggle to control whether to include owners to the ingested entity if the owner email matches with a user stored in the OM server as part of metadata ingestion. If the ingested entity already exists and has an owner, the owner will not be overwritten. **includeStoredProcedures**: Optional configuration to toggle the Stored Procedures ingestion. **includeDDL**: Optional configuration to toggle the DDL Statements ingestion. **overrideMetadata** _(boolean)_: Set the 'Override Metadata' toggle to control whether to override the existing metadata in the OpenMetadata server with the metadata fetched from the source. If the toggle is set to true, the metadata fetched from the source will override the existing metadata in the OpenMetadata server. If the toggle is set to false, the metadata fetched from the source will not override the existing metadata in the OpenMetadata server. This is applicable for fields like description, tags, owner and displayName. **queryLogDuration**: Configuration to tune how far we want to look back in query logs to process Stored Procedures results. **queryParsingTimeoutLimit**: Configuration to set the timeout for parsing the query in seconds. **useFqnForFiltering**: Regex will be applied on fully qualified name (e.g service\_name.db\_name.schema\_name.table\_name) instead of raw name (e.g. table\_name). **databaseFilterPattern**, **schemaFilterPattern**, **tableFilterPattern**: Note that the filter supports regex as include or exclude. You can find examples [here](https://docs.open-metadata.org/latest/connectors/ingestion/workflows/metadata/filter-patterns/database) **threads (beta)**: The number of threads to use when extracting the metadata using multithreading. Please take a look [here](https://docs.open-metadata.org/latest/connectors/ingestion/workflows/metadata/multithreading) before configuring this. **databaseMetadataConfigType** _(string)_: Database Source Config Metadata Pipeline type. **incremental (beta)**: Incremental Extraction configuration. Currently implemented for: * [BigQuery](https://docs.open-metadata.org/latest/connectors/ingestion/workflows/metadata/incremental-extraction/bigquery) * [Redshift](https://docs.open-metadata.org/latest/connectors/ingestion/workflows/metadata/incremental-extraction/redshift) * [Snowflake](https://docs.open-metadata.org/latest/connectors/ingestion/workflows/metadata/incremental-extraction/snowflake) #### Sink Configuration To send the metadata to OpenMetadata, it needs to be specified as `type: metadata-rest`. #### Workflow Configuration The main property here is the `openMetadataServerConfig`, where you can define the host and security provider of your OpenMetadata installation. **Logger Level** You can specify the `loggerLevel` depending on your needs. If you are trying to troubleshoot an ingestion, running with `DEBUG` will give you far more traces for identifying issues. **JWT Token** JWT tokens will allow your clients to authenticate against the OpenMetadata server. To enable JWT Tokens, you will get more details [here](https://docs.open-metadata.org/latest/deployment/security/enable-jwt-tokens) . You can refer to the JWT Troubleshooting section [link](https://docs.open-metadata.org/latest/deployment/security/jwt-troubleshooting) for any issues in your JWT configuration. **Store Service Connection** If set to `true` (default), we will store the sensitive information either encrypted via the Fernet Key in the database or externally, if you have configured any [Secrets Manager](https://docs.open-metadata.org/latest/deployment/secrets-manager) . If set to `false`, the service will be created, but the service connection information will only be used by the Ingestion Framework at runtime, and won't be sent to the OpenMetadata server. **Store Service Connection** If set to `true` (default), we will store the sensitive information either encrypted via the Fernet Key in the database or externally, if you have configured any [Secrets Manager](https://docs.open-metadata.org/latest/deployment/secrets-manager) . If set to `false`, the service will be created, but the service connection information will only be used by the Ingestion Framework at runtime, and won't be sent to the OpenMetadata server. **SSL Configuration** If you have added SSL to the [OpenMetadata server](https://docs.open-metadata.org/latest/deployment/security/enable-ssl) , then you will need to handle the certificates when running the ingestion too. You can either set `verifySSL` to `ignore`, or have it as `validate`, which will require you to set the `sslConfig.caCertificate` with a local path where your ingestion runs that points to the server certificate file. Find more information on how to troubleshoot SSL issues [here](https://docs.open-metadata.org/latest/deployment/security/enable-ssl/ssl-troubleshooting) . **ingestionPipelineFQN** Fully qualified name of ingestion pipeline, used to identify the current ingestion pipeline. #### Advanced Configuration **Connection Options (Optional)**: Enter the details for any additional connection options that can be sent to database during the connection. These details must be added as Key-Value pairs. **Connection Arguments (Optional)**: Enter the details for any additional connection arguments such as security or protocol configs that can be sent to database during the connection. These details must be added as Key-Value pairs. * In case you are using Single-Sign-On (SSO) for authentication, add the `authenticator` details in the Connection Arguments as a Key-Value pair as follows: `"authenticator" : "sso_login_url"` filename.yamlCopy ### 2\. Run with the CLI First, we will need to save the YAML file. Afterward, and with all requirements installed, we can run: Note that from connector to connector, this recipe will always be the same. By updating the YAML configuration, you will be able to extract metadata from different sources. Data Profiler ------------- The Data Profiler workflow will be using the `orm-profiler` processor. After running a Metadata Ingestion workflow, we can run the Data Profiler workflow. While the `serviceName` will be the same to that was used in Metadata Ingestion, so the ingestion bot can get the `serviceConnection` details from the server. ### 1\. Define the YAML Config This is a sample config for the profiler: #### Source Configuration - Source Config You can find all the definitions and types for the `sourceConfig` [here](https://github.com/open-metadata/OpenMetadata/blob/main/openmetadata-spec/src/main/resources/json/schema/metadataIngestion/databaseServiceProfilerPipeline.json) . **profileSample**: Percentage of data or no. of rows we want to execute the profiler and tests on. **threadCount**: Number of threads to use during metric computations. **timeoutSeconds**: Profiler Timeout in Seconds **databaseFilterPattern**: Regex to only fetch databases that matches the pattern. **schemaFilterPattern**: Regex to only fetch tables or databases that matches the pattern. **tableFilterPattern**: Regex to only fetch tables or databases that matches the pattern. #### Processor Configuration Choose the `orm-profiler`. Its config can also be updated to define tests from the YAML itself instead of the UI: **tableConfig**: `tableConfig` allows you to set up some configuration at the table level. #### Sink Configuration To send the metadata to OpenMetadata, it needs to be specified as `type: metadata-rest`. #### Workflow Configuration The main property here is the `openMetadataServerConfig`, where you can define the host and security provider of your OpenMetadata installation. **Logger Level** You can specify the `loggerLevel` depending on your needs. If you are trying to troubleshoot an ingestion, running with `DEBUG` will give you far more traces for identifying issues. **JWT Token** JWT tokens will allow your clients to authenticate against the OpenMetadata server. To enable JWT Tokens, you will get more details [here](https://docs.open-metadata.org/latest/deployment/security/enable-jwt-tokens) . You can refer to the JWT Troubleshooting section [link](https://docs.open-metadata.org/latest/deployment/security/jwt-troubleshooting) for any issues in your JWT configuration. **Store Service Connection** If set to `true` (default), we will store the sensitive information either encrypted via the Fernet Key in the database or externally, if you have configured any [Secrets Manager](https://docs.open-metadata.org/latest/deployment/secrets-manager) . If set to `false`, the service will be created, but the service connection information will only be used by the Ingestion Framework at runtime, and won't be sent to the OpenMetadata server. **Store Service Connection** If set to `true` (default), we will store the sensitive information either encrypted via the Fernet Key in the database or externally, if you have configured any [Secrets Manager](https://docs.open-metadata.org/latest/deployment/secrets-manager) . If set to `false`, the service will be created, but the service connection information will only be used by the Ingestion Framework at runtime, and won't be sent to the OpenMetadata server. **SSL Configuration** If you have added SSL to the [OpenMetadata server](https://docs.open-metadata.org/latest/deployment/security/enable-ssl) , then you will need to handle the certificates when running the ingestion too. You can either set `verifySSL` to `ignore`, or have it as `validate`, which will require you to set the `sslConfig.caCertificate` with a local path where your ingestion runs that points to the server certificate file. Find more information on how to troubleshoot SSL issues [here](https://docs.open-metadata.org/latest/deployment/security/enable-ssl/ssl-troubleshooting) . **ingestionPipelineFQN** Fully qualified name of ingestion pipeline, used to identify the current ingestion pipeline. filename.yamlCopy * You can learn more about how to configure and run the Profiler Workflow to extract Profiler data and execute the Data Quality from [here](https://docs.open-metadata.org/latest/how-to-guides/data-quality-observability/profiler/workflow) ### 2\. Run with the CLI After saving the YAML config, we will run the command the same way we did for the metadata ingestion: Note now instead of running `ingest`, we are using the `profile` command to select the Profiler workflow. [#### Data Profiler\ \ Find more information about the Data Profiler here](https://docs.open-metadata.org/latest/how-to-guides/data-quality-observability/profiler/workflow) Auto Classification ------------------- The Auto Classification workflow will be using the `orm-profiler` processor. After running a Metadata Ingestion workflow, we can run the Auto Classification workflow. While the `serviceName` will be the same to that was used in Metadata Ingestion, so the ingestion bot can get the `serviceConnection` details from the server. ### 1\. Define the YAML Config This is a sample config for the Auto Classification Workflow: #### Source Configuration - Source Config You can find all the definitions and types for the `sourceConfig` [here](https://github.com/open-metadata/OpenMetadata/blob/main/openmetadata-spec/src/main/resources/json/schema/metadataIngestion/databaseServiceAutoClassificationPipeline.json) . **storeSampleData**: Option to turn on/off storing sample data. If enabled, we will ingest sample data for each table. **enableAutoClassification**: Optional configuration to automatically tag columns that might contain sensitive information. **confidence**: Set the Confidence value for which you want the column to be tagged as PII. Confidence value ranges from 0 to 100. A higher number will yield less false positives but more false negatives. A lower number will yield more false positives but less false negatives. **databaseFilterPattern**: Regex to only fetch databases that matches the pattern. **schemaFilterPattern**: Regex to only fetch tables or databases that matches the pattern. **tableFilterPattern**: Regex to only fetch tables or databases that matches the pattern. #### Processor Configuration Choose the `orm-profiler`. Its config can also be updated to define tests from the YAML itself instead of the UI: **tableConfig**: `tableConfig` allows you to set up some configuration at the table level. #### Sink Configuration To send the metadata to OpenMetadata, it needs to be specified as `type: metadata-rest`. #### Workflow Configuration The main property here is the `openMetadataServerConfig`, where you can define the host and security provider of your OpenMetadata installation. **Logger Level** You can specify the `loggerLevel` depending on your needs. If you are trying to troubleshoot an ingestion, running with `DEBUG` will give you far more traces for identifying issues. **JWT Token** JWT tokens will allow your clients to authenticate against the OpenMetadata server. To enable JWT Tokens, you will get more details [here](https://docs.open-metadata.org/latest/deployment/security/enable-jwt-tokens) . You can refer to the JWT Troubleshooting section [link](https://docs.open-metadata.org/latest/deployment/security/jwt-troubleshooting) for any issues in your JWT configuration. **Store Service Connection** If set to `true` (default), we will store the sensitive information either encrypted via the Fernet Key in the database or externally, if you have configured any [Secrets Manager](https://docs.open-metadata.org/latest/deployment/secrets-manager) . If set to `false`, the service will be created, but the service connection information will only be used by the Ingestion Framework at runtime, and won't be sent to the OpenMetadata server. **Store Service Connection** If set to `true` (default), we will store the sensitive information either encrypted via the Fernet Key in the database or externally, if you have configured any [Secrets Manager](https://docs.open-metadata.org/latest/deployment/secrets-manager) . If set to `false`, the service will be created, but the service connection information will only be used by the Ingestion Framework at runtime, and won't be sent to the OpenMetadata server. **SSL Configuration** If you have added SSL to the [OpenMetadata server](https://docs.open-metadata.org/latest/deployment/security/enable-ssl) , then you will need to handle the certificates when running the ingestion too. You can either set `verifySSL` to `ignore`, or have it as `validate`, which will require you to set the `sslConfig.caCertificate` with a local path where your ingestion runs that points to the server certificate file. Find more information on how to troubleshoot SSL issues [here](https://docs.open-metadata.org/latest/deployment/security/enable-ssl/ssl-troubleshooting) . **ingestionPipelineFQN** Fully qualified name of ingestion pipeline, used to identify the current ingestion pipeline. filename.yamlCopy ### 2\. Run with the CLI After saving the YAML config, we will run the command the same way we did for the metadata ingestion: Now instead of running `ingest`, we are using the `classify` command to select the Auto Classification workflow. Data Quality ------------ ### Adding Data Quality Test Cases from yaml config When creating a JSON config for a test workflow the source configuration is very simple. The only sections you need to modify here are the `serviceName` (this name needs to be unique) and `entityFullyQualifiedName` (the entity for which we'll be executing tests against) keys. Once you have defined your source configuration you'll need to define te processor configuration. The processor type should be set to `"orm-test-runner"`. For accepted test definition names and parameter value names refer to the [tests page](https://docs.open-metadata.org/latest/how-to-guides/data-quality-observability/quality/tests-yaml) . Note that while you can define tests directly in this YAML configuration, running the workflow will execute ALL THE TESTS present in the table, regardless of what you are defining in the YAML. This makes it easy for any user to contribute tests via the UI, while maintaining the test execution external. You can keep your YAML config as simple as follows if the table already has tests. ### Key reference: * `forceUpdate`: if the test case exists (base on the test case name) for the entity, implements the strategy to follow when running the test (i.e. whether or not to update parameters) * `testCases`: list of test cases to add to the entity referenced. Note that we will execute all the tests present in the Table. * `name`: test case name * `testDefinitionName`: test definition * `columnName`: only applies to column test. The name of the column to run the test against * `parameterValues`: parameter values of the test The `sink` and `workflowConfig` will have the same settings as the ingestion and profiler workflow. ### Full `yaml` config example ### How to Run Tests To run the tests from the CLI execute the following command Lineage ------- After running a Metadata Ingestion workflow, we can run Lineage workflow. While the `serviceName` will be the same to that was used in Metadata Ingestion, so the ingestion bot can get the `serviceConnection` details from the server. ### 1\. Define the YAML Config This is a sample config for BigQuery Lineage: #### Source Configuration - Source Config You can find all the definitions and types for the `sourceConfig` [here](https://github.com/open-metadata/OpenMetadata/blob/main/openmetadata-spec/src/main/resources/json/schema/metadataIngestion/databaseServiceQueryLineagePipeline.json) . **queryLogDuration**: Configuration to tune how far we want to look back in query logs to process lineage data in days. **parsingTimeoutLimit**: Configuration to set the timeout for parsing the query in seconds. **filterCondition**: Condition to filter the query history. **resultLimit**: Configuration to set the limit for query logs. **queryLogFilePath**: Configuration to set the file path for query logs. **databaseFilterPattern**: Regex to only fetch databases that matches the pattern. **schemaFilterPattern**: Regex to only fetch tables or databases that matches the pattern. **tableFilterPattern**: Regex to only fetch tables or databases that matches the pattern. **overrideViewLineage**: Set the 'Override View Lineage' toggle to control whether to override the existing view lineage. **processViewLineage**: Set the 'Process View Lineage' toggle to control whether to process view lineage. **processQueryLineage**: Set the 'Process Query Lineage' toggle to control whether to process query lineage. **processStoredProcedureLineage**: Set the 'Process Stored ProcedureLog Lineage' toggle to control whether to process stored procedure lineage. **threads**: Number of Threads to use in order to parallelize lineage ingestion. #### Sink Configuration To send the metadata to OpenMetadata, it needs to be specified as `type: metadata-rest`. #### Workflow Configuration The main property here is the `openMetadataServerConfig`, where you can define the host and security provider of your OpenMetadata installation. For a simple, local installation using our docker containers, this looks like: filename.yamlCopy * You can learn more about how to configure and run the Lineage Workflow to extract Lineage data from [here](https://docs.open-metadata.org/latest/connectors/ingestion/workflows/lineage) ### 2\. Run with the CLI After saving the YAML config, we will run the command the same way we did for the metadata ingestion: dbt Integration --------------- [#### dbt Integration\ \ Learn more about how to ingest dbt models' definitions and their lineage.](https://docs.open-metadata.org/latest/connectors/ingestion/workflows/dbt) --- # Okta SSO | OpenMetadata Authentication Integration We use cookies to improve site navigation, analyze site usage, and enhance your user experience. Click "Accept" to enable cookies or "Reject" to reject cookies. AcceptReject ![](https://static.scarf.sh/a.png?x-pxid=8472f2b3-d307-4935-a5ed-3f9d3982c15d) deployment No menu items for this category [Home](https://docs.open-metadata.org/latest) /[Deployment](https://docs.open-metadata.org/latest/deployment) /[Security](https://docs.open-metadata.org/latest/deployment/security) /[Okta](https://docs.open-metadata.org/latest/deployment/security/okta) /[Implicit Flow](https://docs.open-metadata.org/latest/deployment/security/okta/implicit-flow) OpenMetadata Documentation Implicit Flow ============= ### Step 1: Configuring the App * Once you are in the **Create a new app integration** page, select **OIDC - OpenID Connect**. * Next, select the **Application type -> Single-Page Application**. * Once selected, click **Next**. ![configuring-the-app](https://docs.open-metadata.org/images/v1.11/deployment/security/okta/configuring-the-app.png) * From the **General Settings** page, * Enter an **App integration name** * Select the following in **Grant type**: * **Authorization Code** * **Refresh Token** - For the refresh token behavior, it is recommended to select the option to 'Rotate token after every use'. * **Implicit (hybrid)** - Select the options to allow ID Token and Access Token with implicit grant type. * Enter the **Sign-in redirect URIs** * http://localhost:8585/callback * http://localhost:8585/silent-callback * Enter the **Sign-out redirect URIs** * Enter the **Base URIs** * Select the required option for **Controlled access** * Click **Save**. ![general-settings-click-save](https://docs.open-metadata.org/images/v1.11/deployment/security/okta/general-settings-click-save.png) * The app is now configured. ![app-is-configured](https://docs.open-metadata.org/images/v1.11/deployment/security/okta/app-is-configured.png) ### Step 2: Add Authorization Server to get the Issuer URL #### New Authorization Server It is recommended to create a separate authorization server for different applications. The authorization server needs an endpoint, which'll be the Issuer URL. * Click on **Security -> API** in the left navigation panel. ![click-security-api](https://docs.open-metadata.org/images/v1.11/deployment/security/okta/click-security-api.png) * From the **Authorization Servers** tab, click on **Add Authorization Server** button. ![click-add-authorization-server](https://docs.open-metadata.org/images/v1.11/deployment/security/okta/click-add-authorization-server.png) * Enter a Name and Description. * While creating the authorization server, an **Audience** must be provided for the server. The Audience is the **Client ID** of the single page application that was created. Refer the next Step 7 to locate the Client ID. * **Save** the changes. ![add-auth-server-save-changes](https://docs.open-metadata.org/images/v1.11/deployment/security/okta/add-auth-server-save-changes.png) This will generate the Issuer URL. #### Default Authorization Server (not recommended ) It is recommended to create a separate authorization server for different applications. The authorization server needs an endpoint, which'll be the Issuer URL. * Click on **Security -> API** in the left navigation panel. ![click-security-api](https://docs.open-metadata.org/images/v1.11/deployment/security/okta/click-security-api.png) * From the **Authorization Servers** tab, click on **default** server. ![default-server](https://docs.open-metadata.org/images/v1.11/deployment/security/okta/default-server.png) ### Step 3: Change the Issuer URL from Dynamic to Okta URL Once the Authorization Server has been added, navigate to Security >> API >> Authorization Servers and click on the authorization server created in the previous step. ![click-auth-server-from-prev-step](https://docs.open-metadata.org/images/v1.11/deployment/security/okta/click-auth-server-from-prev-step.png) The Issuer URL shows up as Dynamic by default. Change the Issuer URL to Okta URL and save the changes. ![change-issuer-url](https://docs.open-metadata.org/images/v1.11/deployment/security/okta/change-issuer-url.png) ### Step 4: Create a Default Scope * To create a default scope from **Security -> API**, click on the required **Authorization Server**. ![click-req-auth-server](https://docs.open-metadata.org/images/v1.11/deployment/security/okta/click-req-auth-server.png) * In the resulting page, click on the **Scopes** tab * Click on **Add Scope** ![add-scope](https://docs.open-metadata.org/images/v1.11/deployment/security/okta/add-scope.png) * Set as a **Default Scope**. ![set-default-scope](https://docs.open-metadata.org/images/v1.11/deployment/security/okta/set-default-scope.png) ### Step 5: Add New Access Policy and Rule * From **Security -> API**, click on the required **Authorization Server** * Navigate to the **Access Policies Tab** * Click on **Add New Access Policy** ![add-new-access-policy](https://docs.open-metadata.org/images/v1.11/deployment/security/okta/add-new-access-policy.png) * To create a policy, add a Name and Description. * Assign the policy to the required clients. ![](https://docs.open-metadata.org/images/v1.11/deployment/security/okta/assign-policy.png) * Add a new **Rule** inside the policy as required. Rules can be created with just a few grant type details, such as Client Credentials, Authorization Code, Device Authorization, and Token Exchange. * Click on **Create Rule** to save the changes. ![add-rule](https://docs.open-metadata.org/images/v1.11/deployment/security/okta/add-rule.png) ### Step 6: Where to Find the Credentials * Once the app is configured, the **Client ID** can be used. * You can also go to **Application -> Application** as in step 2. * You should be able to see your application in the list. ![see-your-application](https://docs.open-metadata.org/images/v1.11/deployment/security/okta/see-your-application.png) * Click on your application. * You will find your **Client ID** and **Okta domain**. * The **Client authentication** is enabled by default. * By clicking on the Edit \*\*\*\* option for General Settings, you can deselect the option for **User consent**. Save the changes. ![deselect-user-consent](https://docs.open-metadata.org/images/v1.11/deployment/security/okta/deselect-user-consent.png) * Click on the **Sign On** tab from the top navigation bar. * Click on Edit for **OpenID Connect ID Token**. * For **Issuer**, change from the Dynamic (based on request domain) option to the **Okta URL** option. * The **Audience** is the same as the Client ID. ![click-edit-token](https://docs.open-metadata.org/images/v1.11/deployment/security/okta/click-edit-token.png) After the applying these steps, you can update the configuration of your deployment: [Public Client\ \ Configure Okta SSO using the Public client type.](https://docs.open-metadata.org/latest/deployment/security/okta/public-client) [Confidential Client\ \ Configure Okta SSO using the Confidential client type.](https://docs.open-metadata.org/latest/deployment/security/okta/confidential-client) [OKTA\ \ Go to okta Configuration](https://docs.open-metadata.org/latest/deployment/security/okta) --- # Run the PinotDB Connector Externally We use cookies to improve site navigation, analyze site usage, and enhance your user experience. Click "Accept" to enable cookies or "Reject" to reject cookies. AcceptReject connectors No menu items for this category [Home](https://docs.open-metadata.org/latest) /[Connectors](https://docs.open-metadata.org/latest/connectors) /[Database](https://docs.open-metadata.org/latest/connectors/database) /[Pinotdb](https://docs.open-metadata.org/latest/connectors/database/pinotdb) /[Yaml](https://docs.open-metadata.org/latest/connectors/database/pinotdb/yaml) OpenMetadata Documentation ![PinotDB](https://docs.open-metadata.org/_next/image?url=%2Fimages%2Fconnectors%2Fpinot.webp&w=64&q=75) PinotDB ======= PROD Available In Feature List Metadata Data Profiler Data Quality dbt View Lineage View Column-level Lineage Sample Data Auto-Classification Query Usage Owners Tags Stored Procedures In this section, we provide guides and references to use the PinotDB connector. Configure and schedule PinotDB metadata and profiler workflows from the OpenMetadata UI: * [Requirements](https://docs.open-metadata.org/latest/connectors/database/pinotdb/yaml#requirements) * [Metadata Ingestion](https://docs.open-metadata.org/latest/connectors/database/pinotdb/yaml#metadata-ingestion) * [Lineage](https://docs.open-metadata.org/latest/connectors/database/pinotdb/yaml#lineage) * [Data Profiler](https://docs.open-metadata.org/latest/connectors/database/pinotdb/yaml#data-profiler) * [Data Quality](https://docs.open-metadata.org/latest/connectors/database/pinotdb/yaml#data-quality) * [dbt Integration](https://docs.open-metadata.org/latest/connectors/database/pinotdb/yaml#dbt-integration) How to Run the Connector Externally ----------------------------------- To run the Ingestion via the UI you'll need to use the OpenMetadata Ingestion Container, which comes shipped with custom Airflow plugins to handle the workflow deployment. If, instead, you want to manage your workflows externally on your preferred orchestrator, you can check the following docs to run the Ingestion Framework **anywhere**. [#### External Schedulers\ \ Get more information about running the Ingestion Framework Externally](https://docs.open-metadata.org/latest/deployment/ingestion) Requirements ------------ ### Python Requirements We have support for Python versions 3.9-3.11 To run the PinotDB ingestion, you will need to install: Metadata Ingestion ------------------ All connectors are defined as JSON Schemas. [Here](https://github.com/open-metadata/OpenMetadata/blob/main/openmetadata-spec/src/main/resources/json/schema/entity/services/connections/database/pinotDBConnection.json) you can find the structure to create a connection to PinotDB. In order to create and run a Metadata Ingestion workflow, we will follow the steps to create a YAML configuration able to connect to the source, process the Entities if needed, and reach the OpenMetadata server. The workflow is modeled around the following [JSON Schema](https://github.com/open-metadata/OpenMetadata/blob/main/openmetadata-spec/src/main/resources/json/schema/metadataIngestion/workflow.json) ### 1\. Define the YAML Config This is a sample config for PinotDB: #### Source Configuration - Service Connection **username**: Specify the User to connect to PinotDB. It should have enough privileges to read all the metadata. **password**: Password to connect to PinotDB. **Host and Port**: Enter the fully qualified hostname and port number for your PinotDB deployment in the Host and Port field. **databaseSchema**: databaseSchema of the data source. This is optional parameter, if you would like to restrict the metadata reading to a single databaseSchema. When left blank, OpenMetadata Ingestion attempts to scan all the databaseSchema. #### Source Configuration - Source Config The `sourceConfig` is defined [here](https://github.com/open-metadata/OpenMetadata/blob/main/openmetadata-spec/src/main/resources/json/schema/metadataIngestion/databaseServiceMetadataPipeline.json) : **markDeletedTables**: To flag tables as soft-deleted if they are not present anymore in the source system. **markDeletedStoredProcedures**: Optional configuration to soft delete stored procedures in OpenMetadata if the source stored procedures are deleted. Also, if the stored procedures is deleted, all the associated entities like lineage, etc., with that stored procedures will be deleted. **markDeletedSchemas**: Optional configuration to soft delete schemas stored in OpenMetadata if the source schema is deleted. Setting this flag to true will only keep filtered schema and delete any other schemas that do not match schemaFilterPattern or do not exist at source. **markDeletedDatabases**: Optional configuration to soft delete databases stored in OpenMetadata if the source database is deleted. Setting this flag to true will only keep filtered databases and delete any other databases that do not match databaseFilterPattern or do not exist at source. **includeTables**: true or false, to ingest table data. Default is true. **includeViews**: true or false, to ingest views definitions. **includeTags**: Optional configuration to toggle the tags ingestion. **includeOwners**: Set the 'Include Owners' toggle to control whether to include owners to the ingested entity if the owner email matches with a user stored in the OM server as part of metadata ingestion. If the ingested entity already exists and has an owner, the owner will not be overwritten. **includeStoredProcedures**: Optional configuration to toggle the Stored Procedures ingestion. **includeDDL**: Optional configuration to toggle the DDL Statements ingestion. **overrideMetadata** _(boolean)_: Set the 'Override Metadata' toggle to control whether to override the existing metadata in the OpenMetadata server with the metadata fetched from the source. If the toggle is set to true, the metadata fetched from the source will override the existing metadata in the OpenMetadata server. If the toggle is set to false, the metadata fetched from the source will not override the existing metadata in the OpenMetadata server. This is applicable for fields like description, tags, owner and displayName. **queryLogDuration**: Configuration to tune how far we want to look back in query logs to process Stored Procedures results. **queryParsingTimeoutLimit**: Configuration to set the timeout for parsing the query in seconds. **useFqnForFiltering**: Regex will be applied on fully qualified name (e.g service\_name.db\_name.schema\_name.table\_name) instead of raw name (e.g. table\_name). **databaseFilterPattern**, **schemaFilterPattern**, **tableFilterPattern**: Note that the filter supports regex as include or exclude. You can find examples [here](https://docs.open-metadata.org/latest/connectors/ingestion/workflows/metadata/filter-patterns/database) **threads (beta)**: The number of threads to use when extracting the metadata using multithreading. Please take a look [here](https://docs.open-metadata.org/latest/connectors/ingestion/workflows/metadata/multithreading) before configuring this. **databaseMetadataConfigType** _(string)_: Database Source Config Metadata Pipeline type. **incremental (beta)**: Incremental Extraction configuration. Currently implemented for: * [BigQuery](https://docs.open-metadata.org/latest/connectors/ingestion/workflows/metadata/incremental-extraction/bigquery) * [Redshift](https://docs.open-metadata.org/latest/connectors/ingestion/workflows/metadata/incremental-extraction/redshift) * [Snowflake](https://docs.open-metadata.org/latest/connectors/ingestion/workflows/metadata/incremental-extraction/snowflake) #### Sink Configuration To send the metadata to OpenMetadata, it needs to be specified as `type: metadata-rest`. #### Workflow Configuration The main property here is the `openMetadataServerConfig`, where you can define the host and security provider of your OpenMetadata installation. **Logger Level** You can specify the `loggerLevel` depending on your needs. If you are trying to troubleshoot an ingestion, running with `DEBUG` will give you far more traces for identifying issues. **JWT Token** JWT tokens will allow your clients to authenticate against the OpenMetadata server. To enable JWT Tokens, you will get more details [here](https://docs.open-metadata.org/latest/deployment/security/enable-jwt-tokens) . You can refer to the JWT Troubleshooting section [link](https://docs.open-metadata.org/latest/deployment/security/jwt-troubleshooting) for any issues in your JWT configuration. **Store Service Connection** If set to `true` (default), we will store the sensitive information either encrypted via the Fernet Key in the database or externally, if you have configured any [Secrets Manager](https://docs.open-metadata.org/latest/deployment/secrets-manager) . If set to `false`, the service will be created, but the service connection information will only be used by the Ingestion Framework at runtime, and won't be sent to the OpenMetadata server. **Store Service Connection** If set to `true` (default), we will store the sensitive information either encrypted via the Fernet Key in the database or externally, if you have configured any [Secrets Manager](https://docs.open-metadata.org/latest/deployment/secrets-manager) . If set to `false`, the service will be created, but the service connection information will only be used by the Ingestion Framework at runtime, and won't be sent to the OpenMetadata server. **SSL Configuration** If you have added SSL to the [OpenMetadata server](https://docs.open-metadata.org/latest/deployment/security/enable-ssl) , then you will need to handle the certificates when running the ingestion too. You can either set `verifySSL` to `ignore`, or have it as `validate`, which will require you to set the `sslConfig.caCertificate` with a local path where your ingestion runs that points to the server certificate file. Find more information on how to troubleshoot SSL issues [here](https://docs.open-metadata.org/latest/deployment/security/enable-ssl/ssl-troubleshooting) . **ingestionPipelineFQN** Fully qualified name of ingestion pipeline, used to identify the current ingestion pipeline. #### Advanced Configuration **Connection Options (Optional)**: Enter the details for any additional connection options that can be sent to database during the connection. These details must be added as Key-Value pairs. **Connection Arguments (Optional)**: Enter the details for any additional connection arguments such as security or protocol configs that can be sent to database during the connection. These details must be added as Key-Value pairs. * In case you are using Single-Sign-On (SSO) for authentication, add the `authenticator` details in the Connection Arguments as a Key-Value pair as follows: `"authenticator" : "sso_login_url"` filename.yamlCopy ### 2\. Run with the CLI First, we will need to save the YAML file. Afterward, and with all requirements installed, we can run: Note that from connector to connector, this recipe will always be the same. By updating the YAML configuration, you will be able to extract metadata from different sources. Lineage ------- After running a Metadata Ingestion workflow, we can run Lineage workflow. While the `serviceName` will be the same to that was used in Metadata Ingestion, so the ingestion bot can get the `serviceConnection` details from the server. ### 1\. Define the YAML Config This is a sample config for BigQuery Lineage: #### Source Configuration - Source Config You can find all the definitions and types for the `sourceConfig` [here](https://github.com/open-metadata/OpenMetadata/blob/main/openmetadata-spec/src/main/resources/json/schema/metadataIngestion/databaseServiceQueryLineagePipeline.json) . **queryLogDuration**: Configuration to tune how far we want to look back in query logs to process lineage data in days. **parsingTimeoutLimit**: Configuration to set the timeout for parsing the query in seconds. **filterCondition**: Condition to filter the query history. **resultLimit**: Configuration to set the limit for query logs. **queryLogFilePath**: Configuration to set the file path for query logs. **databaseFilterPattern**: Regex to only fetch databases that matches the pattern. **schemaFilterPattern**: Regex to only fetch tables or databases that matches the pattern. **tableFilterPattern**: Regex to only fetch tables or databases that matches the pattern. **overrideViewLineage**: Set the 'Override View Lineage' toggle to control whether to override the existing view lineage. **processViewLineage**: Set the 'Process View Lineage' toggle to control whether to process view lineage. **processQueryLineage**: Set the 'Process Query Lineage' toggle to control whether to process query lineage. **processStoredProcedureLineage**: Set the 'Process Stored ProcedureLog Lineage' toggle to control whether to process stored procedure lineage. **threads**: Number of Threads to use in order to parallelize lineage ingestion. #### Sink Configuration To send the metadata to OpenMetadata, it needs to be specified as `type: metadata-rest`. #### Workflow Configuration The main property here is the `openMetadataServerConfig`, where you can define the host and security provider of your OpenMetadata installation. For a simple, local installation using our docker containers, this looks like: filename.yamlCopy * You can learn more about how to configure and run the Lineage Workflow to extract Lineage data from [here](https://docs.open-metadata.org/latest/connectors/ingestion/workflows/lineage) ### 2\. Run with the CLI After saving the YAML config, we will run the command the same way we did for the metadata ingestion: Data Profiler ------------- The Data Profiler workflow will be using the `orm-profiler` processor. After running a Metadata Ingestion workflow, we can run the Data Profiler workflow. While the `serviceName` will be the same to that was used in Metadata Ingestion, so the ingestion bot can get the `serviceConnection` details from the server. ### 1\. Define the YAML Config This is a sample config for the profiler: #### Source Configuration - Source Config You can find all the definitions and types for the `sourceConfig` [here](https://github.com/open-metadata/OpenMetadata/blob/main/openmetadata-spec/src/main/resources/json/schema/metadataIngestion/databaseServiceProfilerPipeline.json) . **profileSample**: Percentage of data or no. of rows we want to execute the profiler and tests on. **threadCount**: Number of threads to use during metric computations. **timeoutSeconds**: Profiler Timeout in Seconds **databaseFilterPattern**: Regex to only fetch databases that matches the pattern. **schemaFilterPattern**: Regex to only fetch tables or databases that matches the pattern. **tableFilterPattern**: Regex to only fetch tables or databases that matches the pattern. #### Processor Configuration Choose the `orm-profiler`. Its config can also be updated to define tests from the YAML itself instead of the UI: **tableConfig**: `tableConfig` allows you to set up some configuration at the table level. #### Sink Configuration To send the metadata to OpenMetadata, it needs to be specified as `type: metadata-rest`. #### Workflow Configuration The main property here is the `openMetadataServerConfig`, where you can define the host and security provider of your OpenMetadata installation. **Logger Level** You can specify the `loggerLevel` depending on your needs. If you are trying to troubleshoot an ingestion, running with `DEBUG` will give you far more traces for identifying issues. **JWT Token** JWT tokens will allow your clients to authenticate against the OpenMetadata server. To enable JWT Tokens, you will get more details [here](https://docs.open-metadata.org/latest/deployment/security/enable-jwt-tokens) . You can refer to the JWT Troubleshooting section [link](https://docs.open-metadata.org/latest/deployment/security/jwt-troubleshooting) for any issues in your JWT configuration. **Store Service Connection** If set to `true` (default), we will store the sensitive information either encrypted via the Fernet Key in the database or externally, if you have configured any [Secrets Manager](https://docs.open-metadata.org/latest/deployment/secrets-manager) . If set to `false`, the service will be created, but the service connection information will only be used by the Ingestion Framework at runtime, and won't be sent to the OpenMetadata server. **Store Service Connection** If set to `true` (default), we will store the sensitive information either encrypted via the Fernet Key in the database or externally, if you have configured any [Secrets Manager](https://docs.open-metadata.org/latest/deployment/secrets-manager) . If set to `false`, the service will be created, but the service connection information will only be used by the Ingestion Framework at runtime, and won't be sent to the OpenMetadata server. **SSL Configuration** If you have added SSL to the [OpenMetadata server](https://docs.open-metadata.org/latest/deployment/security/enable-ssl) , then you will need to handle the certificates when running the ingestion too. You can either set `verifySSL` to `ignore`, or have it as `validate`, which will require you to set the `sslConfig.caCertificate` with a local path where your ingestion runs that points to the server certificate file. Find more information on how to troubleshoot SSL issues [here](https://docs.open-metadata.org/latest/deployment/security/enable-ssl/ssl-troubleshooting) . **ingestionPipelineFQN** Fully qualified name of ingestion pipeline, used to identify the current ingestion pipeline. filename.yamlCopy * You can learn more about how to configure and run the Profiler Workflow to extract Profiler data and execute the Data Quality from [here](https://docs.open-metadata.org/latest/how-to-guides/data-quality-observability/profiler/workflow) ### 2\. Run with the CLI After saving the YAML config, we will run the command the same way we did for the metadata ingestion: Note now instead of running `ingest`, we are using the `profile` command to select the Profiler workflow. [#### Data Profiler\ \ Find more information about the Data Profiler here](https://docs.open-metadata.org/latest/how-to-guides/data-quality-observability/profiler/workflow) Auto Classification ------------------- The Auto Classification workflow will be using the `orm-profiler` processor. After running a Metadata Ingestion workflow, we can run the Auto Classification workflow. While the `serviceName` will be the same to that was used in Metadata Ingestion, so the ingestion bot can get the `serviceConnection` details from the server. ### 1\. Define the YAML Config This is a sample config for the Auto Classification Workflow: #### Source Configuration - Source Config You can find all the definitions and types for the `sourceConfig` [here](https://github.com/open-metadata/OpenMetadata/blob/main/openmetadata-spec/src/main/resources/json/schema/metadataIngestion/databaseServiceAutoClassificationPipeline.json) . **storeSampleData**: Option to turn on/off storing sample data. If enabled, we will ingest sample data for each table. **enableAutoClassification**: Optional configuration to automatically tag columns that might contain sensitive information. **confidence**: Set the Confidence value for which you want the column to be tagged as PII. Confidence value ranges from 0 to 100. A higher number will yield less false positives but more false negatives. A lower number will yield more false positives but less false negatives. **databaseFilterPattern**: Regex to only fetch databases that matches the pattern. **schemaFilterPattern**: Regex to only fetch tables or databases that matches the pattern. **tableFilterPattern**: Regex to only fetch tables or databases that matches the pattern. #### Processor Configuration Choose the `orm-profiler`. Its config can also be updated to define tests from the YAML itself instead of the UI: **tableConfig**: `tableConfig` allows you to set up some configuration at the table level. #### Sink Configuration To send the metadata to OpenMetadata, it needs to be specified as `type: metadata-rest`. #### Workflow Configuration The main property here is the `openMetadataServerConfig`, where you can define the host and security provider of your OpenMetadata installation. **Logger Level** You can specify the `loggerLevel` depending on your needs. If you are trying to troubleshoot an ingestion, running with `DEBUG` will give you far more traces for identifying issues. **JWT Token** JWT tokens will allow your clients to authenticate against the OpenMetadata server. To enable JWT Tokens, you will get more details [here](https://docs.open-metadata.org/latest/deployment/security/enable-jwt-tokens) . You can refer to the JWT Troubleshooting section [link](https://docs.open-metadata.org/latest/deployment/security/jwt-troubleshooting) for any issues in your JWT configuration. **Store Service Connection** If set to `true` (default), we will store the sensitive information either encrypted via the Fernet Key in the database or externally, if you have configured any [Secrets Manager](https://docs.open-metadata.org/latest/deployment/secrets-manager) . If set to `false`, the service will be created, but the service connection information will only be used by the Ingestion Framework at runtime, and won't be sent to the OpenMetadata server. **Store Service Connection** If set to `true` (default), we will store the sensitive information either encrypted via the Fernet Key in the database or externally, if you have configured any [Secrets Manager](https://docs.open-metadata.org/latest/deployment/secrets-manager) . If set to `false`, the service will be created, but the service connection information will only be used by the Ingestion Framework at runtime, and won't be sent to the OpenMetadata server. **SSL Configuration** If you have added SSL to the [OpenMetadata server](https://docs.open-metadata.org/latest/deployment/security/enable-ssl) , then you will need to handle the certificates when running the ingestion too. You can either set `verifySSL` to `ignore`, or have it as `validate`, which will require you to set the `sslConfig.caCertificate` with a local path where your ingestion runs that points to the server certificate file. Find more information on how to troubleshoot SSL issues [here](https://docs.open-metadata.org/latest/deployment/security/enable-ssl/ssl-troubleshooting) . **ingestionPipelineFQN** Fully qualified name of ingestion pipeline, used to identify the current ingestion pipeline. filename.yamlCopy ### 2\. Run with the CLI After saving the YAML config, we will run the command the same way we did for the metadata ingestion: Now instead of running `ingest`, we are using the `classify` command to select the Auto Classification workflow. Data Quality ------------ ### Adding Data Quality Test Cases from yaml config When creating a JSON config for a test workflow the source configuration is very simple. The only sections you need to modify here are the `serviceName` (this name needs to be unique) and `entityFullyQualifiedName` (the entity for which we'll be executing tests against) keys. Once you have defined your source configuration you'll need to define te processor configuration. The processor type should be set to `"orm-test-runner"`. For accepted test definition names and parameter value names refer to the [tests page](https://docs.open-metadata.org/latest/how-to-guides/data-quality-observability/quality/tests-yaml) . Note that while you can define tests directly in this YAML configuration, running the workflow will execute ALL THE TESTS present in the table, regardless of what you are defining in the YAML. This makes it easy for any user to contribute tests via the UI, while maintaining the test execution external. You can keep your YAML config as simple as follows if the table already has tests. ### Key reference: * `forceUpdate`: if the test case exists (base on the test case name) for the entity, implements the strategy to follow when running the test (i.e. whether or not to update parameters) * `testCases`: list of test cases to add to the entity referenced. Note that we will execute all the tests present in the Table. * `name`: test case name * `testDefinitionName`: test definition * `columnName`: only applies to column test. The name of the column to run the test against * `parameterValues`: parameter values of the test The `sink` and `workflowConfig` will have the same settings as the ingestion and profiler workflow. ### Full `yaml` config example ### How to Run Tests To run the tests from the CLI execute the following command dbt Integration --------------- [#### dbt Integration\ \ Learn more about how to ingest dbt models' definitions and their lineage.](https://docs.open-metadata.org/latest/connectors/ingestion/workflows/dbt) --- # Run the Athena Connector Externally We use cookies to improve site navigation, analyze site usage, and enhance your user experience. Click "Accept" to enable cookies or "Reject" to reject cookies. AcceptReject connectors No menu items for this category [Home](https://docs.open-metadata.org/latest) /[Connectors](https://docs.open-metadata.org/latest/connectors) /[Database](https://docs.open-metadata.org/latest/connectors/database) /[Athena](https://docs.open-metadata.org/latest/connectors/database/athena) /[Yaml](https://docs.open-metadata.org/latest/connectors/database/athena/yaml) OpenMetadata Documentation ![Athena](https://docs.open-metadata.org/_next/image?url=%2Fimages%2Fconnectors%2Fathena.webp&w=64&q=75) Athena ====== PROD Available In Feature List Metadata Query Usage Lineage Column-level Lineage Data Profiler Auto-Classification Data Quality Tags dbt Sample Data Reverse Metadata (Collate Only) Owners Stored Procedures In this section, we provide guides and references to use the Athena connector. Configure and schedule Athena metadata and profiler workflows from the OpenMetadata UI: * [Requirements](https://docs.open-metadata.org/latest/connectors/database/athena/yaml#requirements) * [Metadata Ingestion](https://docs.open-metadata.org/latest/connectors/database/athena/yaml#metadata-ingestion) * [Query Usage](https://docs.open-metadata.org/latest/connectors/database/athena/yaml#query-usage) * [Lineage](https://docs.open-metadata.org/latest/connectors/database/athena/yaml#lineage) * [Data Profiler](https://docs.open-metadata.org/latest/connectors/database/athena/yaml#data-profiler) * [Data Quality](https://docs.open-metadata.org/latest/connectors/database/athena/yaml#data-quality) * [dbt Integration](https://docs.open-metadata.org/latest/connectors/database/athena/yaml#dbt-integration) How to Run the Connector Externally ----------------------------------- To run the Ingestion via the UI you'll need to use the OpenMetadata Ingestion Container, which comes shipped with custom Airflow plugins to handle the workflow deployment. If, instead, you want to manage your workflows externally on your preferred orchestrator, you can check the following docs to run the Ingestion Framework **anywhere**. [#### External Schedulers\ \ Get more information about running the Ingestion Framework Externally](https://docs.open-metadata.org/latest/deployment/ingestion) Requirements ------------ The Athena connector ingests metadata through JDBC connections. According to AWS's official [documentation](https://docs.aws.amazon.com/athena/latest/ug/policy-actions.html) : _If you are using the JDBC or ODBC driver, ensure that the IAM permissions policy includes all of the actions listed in [AWS managed policy: AWSQuicksightAthenaAccess](https://docs.aws.amazon.com/athena/latest/ug/managed-policies.html#awsquicksightathenaaccess-managed-policy) ._ This policy groups the following permissions: * `athena` – Allows the principal to run queries on Athena resources. * `glue` – Allows principals access to AWS Glue databases, tables, and partitions. This is required so that the principal can use the AWS Glue Data Catalog with Athena. Resources of each table and database needs to be added as resource for each database user wants to ingest. * `lakeformation` – Allows principals to request temporary credentials to access data in a data lake location that is registered with Lake Formation and allows access to the LF-tags linked to databases, tables and columns. And is defined as: ### LF-Tags Athena connector ingests and creates LF-tags in OpenMetadata with LF-tag key mapped to OpenMetadata's classification and the values mapped to tag labels. To ingest LF-tags provide the appropriate permissions as to the resources as mentioned above and enable the `includeTags` toggle in the ingestion config. If you have external services other than glue and facing permission issues, add the permissions to the list above. You can find further information on the Athena connector in the [docs](https://docs.open-metadata.org/latest/connectors/database/athena) . ### Python Requirements We have support for Python versions 3.9-3.11 To run the Athena ingestion, you will need to install: Metadata Ingestion ------------------ All connectors are defined as JSON Schemas. [Here](https://github.com/open-metadata/OpenMetadata/blob/main/openmetadata-spec/src/main/resources/json/schema/entity/services/connections/database/athenaConnection.json) you can find the structure to create a connection to Athena. In order to create and run a Metadata Ingestion workflow, we will follow the steps to create a YAML configuration able to connect to the source, process the Entities if needed, and reach the OpenMetadata server. The workflow is modeled around the following [JSON Schema](https://github.com/open-metadata/OpenMetadata/blob/main/openmetadata-spec/src/main/resources/json/schema/metadataIngestion/workflow.json) ### 1\. Define the YAML Config This is a sample config for Athena: #### Source Configuration - Service Connection * **awsAccessKeyId** & **awsSecretAccessKey**: When you interact with AWS, you specify your AWS security credentials to verify who you are and whether you have permission to access the resources that you are requesting. AWS uses the security credentials to authenticate and authorize your requests ([docs](https://docs.aws.amazon.com/IAM/latest/UserGuide/security-creds.html) ). Access keys consist of two parts: An **access key ID** (for example, `AKIAIOSFODNN7EXAMPLE`), and a **secret access key** (for example, `wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY`). You must use both the access key ID and secret access key together to authenticate your requests. You can find further information on how to manage your access keys [here](https://docs.aws.amazon.com/IAM/latest/UserGuide/id_credentials_access-keys.html) . **awsSessionToken**: If you are using temporary credentials to access your services, you will need to inform the AWS Access Key ID and AWS Secrets Access Key. Also, these will include an AWS Session Token. **awsRegion**: Each AWS Region is a separate geographic area in which AWS clusters data centers ([docs](https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/Concepts.RegionsAndAvailabilityZones.html) ). As AWS can have instances in multiple regions, we need to know the region the service you want reach belongs to. Note that the AWS Region is the only required parameter when configuring a connection. When connecting to the services programmatically, there are different ways in which we can extract and use the rest of AWS configurations. You can find further information about configuring your credentials [here](https://boto3.amazonaws.com/v1/documentation/api/latest/guide/credentials.html#configuring-credentials) . **endPointURL**: To connect programmatically to an AWS service, you use an endpoint. An _endpoint_ is the URL of the entry point for an AWS web service. The AWS SDKs and the AWS Command Line Interface (AWS CLI) automatically use the default endpoint for each service in an AWS Region. But you can specify an alternate endpoint for your API requests. Find more information on [AWS service endpoints](https://docs.aws.amazon.com/general/latest/gr/rande.html) . **profileName**: A named profile is a collection of settings and credentials that you can apply to a AWS CLI command. When you specify a profile to run a command, the settings and credentials are used to run that command. Multiple named profiles can be stored in the config and credentials files. You can inform this field if you'd like to use a profile other than `default`. Find here more information about [Named profiles for the AWS CLI](https://docs.aws.amazon.com/cli/latest/userguide/cli-configure-profiles.html) . **assumeRoleArn**: Typically, you use `AssumeRole` within your account or for cross-account access. In this field you'll set the `ARN` (Amazon Resource Name) of the policy of the other account. A user who wants to access a role in a different account must also have permissions that are delegated from the account administrator. The administrator must attach a policy that allows the user to call `AssumeRole` for the `ARN` of the role in the other account. This is a required field if you'd like to `AssumeRole`. Find more information on [AssumeRole](https://docs.aws.amazon.com/STS/latest/APIReference/API_AssumeRole.html) . When using Assume Role authentication, ensure you provide the following details: * **AWS Region**: Specify the AWS region for your deployment. * **Assume Role ARN**: Provide the ARN of the role in your AWS account that OpenMetadata will assume. **assumeRoleSessionName**: An identifier for the assumed role session. Use the role session name to uniquely identify a session when the same role is assumed by different principals or for different reasons. By default, we'll use the name `OpenMetadataSession`. Find more information about the [Role Session Name](https://docs.aws.amazon.com/STS/latest/APIReference/API_AssumeRole.html#:~:text=An%20identifier%20for%20the%20assumed%20role%20session.) . **assumeRoleSourceIdentity**: The source identity specified by the principal that is calling the `AssumeRole` operation. You can use source identity information in AWS CloudTrail logs to determine who took actions with a role. Find more information about [Source Identity](https://docs.aws.amazon.com/STS/latest/APIReference/API_AssumeRole.html#:~:text=Required%3A%20No-,SourceIdentity,-The%20source%20identity) . **s3StagingDir**: The S3 staging directory is an optional parameter. Enter a staging directory to override the default staging directory for AWS Athena. **workgroup**: The Athena workgroup is an optional parameter. If you wish to have your Athena connection related to an existing AWS workgroup add your workgroup name here. #### Source Configuration - Source Config The `sourceConfig` is defined [here](https://github.com/open-metadata/OpenMetadata/blob/main/openmetadata-spec/src/main/resources/json/schema/metadataIngestion/databaseServiceMetadataPipeline.json) : **markDeletedTables**: To flag tables as soft-deleted if they are not present anymore in the source system. **markDeletedStoredProcedures**: Optional configuration to soft delete stored procedures in OpenMetadata if the source stored procedures are deleted. Also, if the stored procedures is deleted, all the associated entities like lineage, etc., with that stored procedures will be deleted. **markDeletedSchemas**: Optional configuration to soft delete schemas stored in OpenMetadata if the source schema is deleted. Setting this flag to true will only keep filtered schema and delete any other schemas that do not match schemaFilterPattern or do not exist at source. **markDeletedDatabases**: Optional configuration to soft delete databases stored in OpenMetadata if the source database is deleted. Setting this flag to true will only keep filtered databases and delete any other databases that do not match databaseFilterPattern or do not exist at source. **includeTables**: true or false, to ingest table data. Default is true. **includeViews**: true or false, to ingest views definitions. **includeTags**: Optional configuration to toggle the tags ingestion. **includeOwners**: Set the 'Include Owners' toggle to control whether to include owners to the ingested entity if the owner email matches with a user stored in the OM server as part of metadata ingestion. If the ingested entity already exists and has an owner, the owner will not be overwritten. **includeStoredProcedures**: Optional configuration to toggle the Stored Procedures ingestion. **includeDDL**: Optional configuration to toggle the DDL Statements ingestion. **overrideMetadata** _(boolean)_: Set the 'Override Metadata' toggle to control whether to override the existing metadata in the OpenMetadata server with the metadata fetched from the source. If the toggle is set to true, the metadata fetched from the source will override the existing metadata in the OpenMetadata server. If the toggle is set to false, the metadata fetched from the source will not override the existing metadata in the OpenMetadata server. This is applicable for fields like description, tags, owner and displayName. **queryLogDuration**: Configuration to tune how far we want to look back in query logs to process Stored Procedures results. **queryParsingTimeoutLimit**: Configuration to set the timeout for parsing the query in seconds. **useFqnForFiltering**: Regex will be applied on fully qualified name (e.g service\_name.db\_name.schema\_name.table\_name) instead of raw name (e.g. table\_name). **databaseFilterPattern**, **schemaFilterPattern**, **tableFilterPattern**: Note that the filter supports regex as include or exclude. You can find examples [here](https://docs.open-metadata.org/latest/connectors/ingestion/workflows/metadata/filter-patterns/database) **threads (beta)**: The number of threads to use when extracting the metadata using multithreading. Please take a look [here](https://docs.open-metadata.org/latest/connectors/ingestion/workflows/metadata/multithreading) before configuring this. **databaseMetadataConfigType** _(string)_: Database Source Config Metadata Pipeline type. **incremental (beta)**: Incremental Extraction configuration. Currently implemented for: * [BigQuery](https://docs.open-metadata.org/latest/connectors/ingestion/workflows/metadata/incremental-extraction/bigquery) * [Redshift](https://docs.open-metadata.org/latest/connectors/ingestion/workflows/metadata/incremental-extraction/redshift) * [Snowflake](https://docs.open-metadata.org/latest/connectors/ingestion/workflows/metadata/incremental-extraction/snowflake) #### Sink Configuration To send the metadata to OpenMetadata, it needs to be specified as `type: metadata-rest`. #### Workflow Configuration The main property here is the `openMetadataServerConfig`, where you can define the host and security provider of your OpenMetadata installation. **Logger Level** You can specify the `loggerLevel` depending on your needs. If you are trying to troubleshoot an ingestion, running with `DEBUG` will give you far more traces for identifying issues. **JWT Token** JWT tokens will allow your clients to authenticate against the OpenMetadata server. To enable JWT Tokens, you will get more details [here](https://docs.open-metadata.org/latest/deployment/security/enable-jwt-tokens) . You can refer to the JWT Troubleshooting section [link](https://docs.open-metadata.org/latest/deployment/security/jwt-troubleshooting) for any issues in your JWT configuration. **Store Service Connection** If set to `true` (default), we will store the sensitive information either encrypted via the Fernet Key in the database or externally, if you have configured any [Secrets Manager](https://docs.open-metadata.org/latest/deployment/secrets-manager) . If set to `false`, the service will be created, but the service connection information will only be used by the Ingestion Framework at runtime, and won't be sent to the OpenMetadata server. **Store Service Connection** If set to `true` (default), we will store the sensitive information either encrypted via the Fernet Key in the database or externally, if you have configured any [Secrets Manager](https://docs.open-metadata.org/latest/deployment/secrets-manager) . If set to `false`, the service will be created, but the service connection information will only be used by the Ingestion Framework at runtime, and won't be sent to the OpenMetadata server. **SSL Configuration** If you have added SSL to the [OpenMetadata server](https://docs.open-metadata.org/latest/deployment/security/enable-ssl) , then you will need to handle the certificates when running the ingestion too. You can either set `verifySSL` to `ignore`, or have it as `validate`, which will require you to set the `sslConfig.caCertificate` with a local path where your ingestion runs that points to the server certificate file. Find more information on how to troubleshoot SSL issues [here](https://docs.open-metadata.org/latest/deployment/security/enable-ssl/ssl-troubleshooting) . **ingestionPipelineFQN** Fully qualified name of ingestion pipeline, used to identify the current ingestion pipeline. #### Advanced Configuration **Connection Options (Optional)**: Enter the details for any additional connection options that can be sent to database during the connection. These details must be added as Key-Value pairs. **Connection Arguments (Optional)**: Enter the details for any additional connection arguments such as security or protocol configs that can be sent to database during the connection. These details must be added as Key-Value pairs. filename.yamlCopy ### 2\. Run with the CLI First, we will need to save the YAML file. Afterward, and with all requirements installed, we can run: Note that from connector to connector, this recipe will always be the same. By updating the YAML configuration, you will be able to extract metadata from different sources. Query Usage ----------- The Query Usage workflow will be using the `query-parser` processor. After running a Metadata Ingestion workflow, we can run Query Usage workflow. While the `serviceName` will be the same to that was used in Metadata Ingestion, so the ingestion bot can get the `serviceConnection` details from the server. ### 1\. Define the YAML Config This is a sample config for BigQuery Usage: #### Source Configuration - Source Config You can find all the definitions and types for the `sourceConfig` [here](https://github.com/open-metadata/OpenMetadata/blob/main/openmetadata-spec/src/main/resources/json/schema/metadataIngestion/databaseServiceQueryUsagePipeline.json) . **queryLogDuration**: Configuration to tune how far we want to look back in query logs to process usage data. **stageFileLocation**: Temporary file name to store the query logs before processing. Absolute file path required. **resultLimit**: Configuration to set the limit for query logs **queryLogFilePath**: Configuration to set the file path for query logs #### Processor, Stage and Bulk Sink Configuration To specify where the staging files will be located. Note that the location is a directory that will be cleaned at the end of the ingestion. filename.yamlCopy ### 2\. Run with the CLI After saving the YAML config, we will run the command the same way we did for the metadata ingestion: Lineage ------- After running a Metadata Ingestion workflow, we can run Lineage workflow. While the `serviceName` will be the same to that was used in Metadata Ingestion, so the ingestion bot can get the `serviceConnection` details from the server. ### 1\. Define the YAML Config This is a sample config for BigQuery Lineage: #### Source Configuration - Source Config You can find all the definitions and types for the `sourceConfig` [here](https://github.com/open-metadata/OpenMetadata/blob/main/openmetadata-spec/src/main/resources/json/schema/metadataIngestion/databaseServiceQueryLineagePipeline.json) . **queryLogDuration**: Configuration to tune how far we want to look back in query logs to process lineage data in days. **parsingTimeoutLimit**: Configuration to set the timeout for parsing the query in seconds. **filterCondition**: Condition to filter the query history. **resultLimit**: Configuration to set the limit for query logs. **queryLogFilePath**: Configuration to set the file path for query logs. **databaseFilterPattern**: Regex to only fetch databases that matches the pattern. **schemaFilterPattern**: Regex to only fetch tables or databases that matches the pattern. **tableFilterPattern**: Regex to only fetch tables or databases that matches the pattern. **overrideViewLineage**: Set the 'Override View Lineage' toggle to control whether to override the existing view lineage. **processViewLineage**: Set the 'Process View Lineage' toggle to control whether to process view lineage. **processQueryLineage**: Set the 'Process Query Lineage' toggle to control whether to process query lineage. **processStoredProcedureLineage**: Set the 'Process Stored ProcedureLog Lineage' toggle to control whether to process stored procedure lineage. **threads**: Number of Threads to use in order to parallelize lineage ingestion. #### Sink Configuration To send the metadata to OpenMetadata, it needs to be specified as `type: metadata-rest`. #### Workflow Configuration The main property here is the `openMetadataServerConfig`, where you can define the host and security provider of your OpenMetadata installation. For a simple, local installation using our docker containers, this looks like: filename.yamlCopy * You can learn more about how to configure and run the Lineage Workflow to extract Lineage data from [here](https://docs.open-metadata.org/latest/connectors/ingestion/workflows/lineage) ### 2\. Run with the CLI After saving the YAML config, we will run the command the same way we did for the metadata ingestion: Data Profiler ------------- The Data Profiler workflow will be using the `orm-profiler` processor. After running a Metadata Ingestion workflow, we can run the Data Profiler workflow. While the `serviceName` will be the same to that was used in Metadata Ingestion, so the ingestion bot can get the `serviceConnection` details from the server. ### 1\. Define the YAML Config This is a sample config for the profiler: #### Source Configuration - Source Config You can find all the definitions and types for the `sourceConfig` [here](https://github.com/open-metadata/OpenMetadata/blob/main/openmetadata-spec/src/main/resources/json/schema/metadataIngestion/databaseServiceProfilerPipeline.json) . **profileSample**: Percentage of data or no. of rows we want to execute the profiler and tests on. **threadCount**: Number of threads to use during metric computations. **timeoutSeconds**: Profiler Timeout in Seconds **databaseFilterPattern**: Regex to only fetch databases that matches the pattern. **schemaFilterPattern**: Regex to only fetch tables or databases that matches the pattern. **tableFilterPattern**: Regex to only fetch tables or databases that matches the pattern. #### Processor Configuration Choose the `orm-profiler`. Its config can also be updated to define tests from the YAML itself instead of the UI: **tableConfig**: `tableConfig` allows you to set up some configuration at the table level. #### Sink Configuration To send the metadata to OpenMetadata, it needs to be specified as `type: metadata-rest`. #### Workflow Configuration The main property here is the `openMetadataServerConfig`, where you can define the host and security provider of your OpenMetadata installation. **Logger Level** You can specify the `loggerLevel` depending on your needs. If you are trying to troubleshoot an ingestion, running with `DEBUG` will give you far more traces for identifying issues. **JWT Token** JWT tokens will allow your clients to authenticate against the OpenMetadata server. To enable JWT Tokens, you will get more details [here](https://docs.open-metadata.org/latest/deployment/security/enable-jwt-tokens) . You can refer to the JWT Troubleshooting section [link](https://docs.open-metadata.org/latest/deployment/security/jwt-troubleshooting) for any issues in your JWT configuration. **Store Service Connection** If set to `true` (default), we will store the sensitive information either encrypted via the Fernet Key in the database or externally, if you have configured any [Secrets Manager](https://docs.open-metadata.org/latest/deployment/secrets-manager) . If set to `false`, the service will be created, but the service connection information will only be used by the Ingestion Framework at runtime, and won't be sent to the OpenMetadata server. **Store Service Connection** If set to `true` (default), we will store the sensitive information either encrypted via the Fernet Key in the database or externally, if you have configured any [Secrets Manager](https://docs.open-metadata.org/latest/deployment/secrets-manager) . If set to `false`, the service will be created, but the service connection information will only be used by the Ingestion Framework at runtime, and won't be sent to the OpenMetadata server. **SSL Configuration** If you have added SSL to the [OpenMetadata server](https://docs.open-metadata.org/latest/deployment/security/enable-ssl) , then you will need to handle the certificates when running the ingestion too. You can either set `verifySSL` to `ignore`, or have it as `validate`, which will require you to set the `sslConfig.caCertificate` with a local path where your ingestion runs that points to the server certificate file. Find more information on how to troubleshoot SSL issues [here](https://docs.open-metadata.org/latest/deployment/security/enable-ssl/ssl-troubleshooting) . **ingestionPipelineFQN** Fully qualified name of ingestion pipeline, used to identify the current ingestion pipeline. filename.yamlCopy * You can learn more about how to configure and run the Profiler Workflow to extract Profiler data and execute the Data Quality from [here](https://docs.open-metadata.org/latest/how-to-guides/data-quality-observability/profiler/workflow) ### 2\. Run with the CLI After saving the YAML config, we will run the command the same way we did for the metadata ingestion: Note now instead of running `ingest`, we are using the `profile` command to select the Profiler workflow. [#### Data Profiler\ \ Find more information about the Data Profiler here](https://docs.open-metadata.org/latest/how-to-guides/data-quality-observability/profiler/workflow) Auto Classification ------------------- The Auto Classification workflow will be using the `orm-profiler` processor. After running a Metadata Ingestion workflow, we can run the Auto Classification workflow. While the `serviceName` will be the same to that was used in Metadata Ingestion, so the ingestion bot can get the `serviceConnection` details from the server. ### 1\. Define the YAML Config This is a sample config for the Auto Classification Workflow: #### Source Configuration - Source Config You can find all the definitions and types for the `sourceConfig` [here](https://github.com/open-metadata/OpenMetadata/blob/main/openmetadata-spec/src/main/resources/json/schema/metadataIngestion/databaseServiceAutoClassificationPipeline.json) . **storeSampleData**: Option to turn on/off storing sample data. If enabled, we will ingest sample data for each table. **enableAutoClassification**: Optional configuration to automatically tag columns that might contain sensitive information. **confidence**: Set the Confidence value for which you want the column to be tagged as PII. Confidence value ranges from 0 to 100. A higher number will yield less false positives but more false negatives. A lower number will yield more false positives but less false negatives. **databaseFilterPattern**: Regex to only fetch databases that matches the pattern. **schemaFilterPattern**: Regex to only fetch tables or databases that matches the pattern. **tableFilterPattern**: Regex to only fetch tables or databases that matches the pattern. #### Processor Configuration Choose the `orm-profiler`. Its config can also be updated to define tests from the YAML itself instead of the UI: **tableConfig**: `tableConfig` allows you to set up some configuration at the table level. #### Sink Configuration To send the metadata to OpenMetadata, it needs to be specified as `type: metadata-rest`. #### Workflow Configuration The main property here is the `openMetadataServerConfig`, where you can define the host and security provider of your OpenMetadata installation. **Logger Level** You can specify the `loggerLevel` depending on your needs. If you are trying to troubleshoot an ingestion, running with `DEBUG` will give you far more traces for identifying issues. **JWT Token** JWT tokens will allow your clients to authenticate against the OpenMetadata server. To enable JWT Tokens, you will get more details [here](https://docs.open-metadata.org/latest/deployment/security/enable-jwt-tokens) . You can refer to the JWT Troubleshooting section [link](https://docs.open-metadata.org/latest/deployment/security/jwt-troubleshooting) for any issues in your JWT configuration. **Store Service Connection** If set to `true` (default), we will store the sensitive information either encrypted via the Fernet Key in the database or externally, if you have configured any [Secrets Manager](https://docs.open-metadata.org/latest/deployment/secrets-manager) . If set to `false`, the service will be created, but the service connection information will only be used by the Ingestion Framework at runtime, and won't be sent to the OpenMetadata server. **Store Service Connection** If set to `true` (default), we will store the sensitive information either encrypted via the Fernet Key in the database or externally, if you have configured any [Secrets Manager](https://docs.open-metadata.org/latest/deployment/secrets-manager) . If set to `false`, the service will be created, but the service connection information will only be used by the Ingestion Framework at runtime, and won't be sent to the OpenMetadata server. **SSL Configuration** If you have added SSL to the [OpenMetadata server](https://docs.open-metadata.org/latest/deployment/security/enable-ssl) , then you will need to handle the certificates when running the ingestion too. You can either set `verifySSL` to `ignore`, or have it as `validate`, which will require you to set the `sslConfig.caCertificate` with a local path where your ingestion runs that points to the server certificate file. Find more information on how to troubleshoot SSL issues [here](https://docs.open-metadata.org/latest/deployment/security/enable-ssl/ssl-troubleshooting) . **ingestionPipelineFQN** Fully qualified name of ingestion pipeline, used to identify the current ingestion pipeline. filename.yamlCopy ### 2\. Run with the CLI After saving the YAML config, we will run the command the same way we did for the metadata ingestion: Now instead of running `ingest`, we are using the `classify` command to select the Auto Classification workflow. Data Quality ------------ ### Adding Data Quality Test Cases from yaml config When creating a JSON config for a test workflow the source configuration is very simple. The only sections you need to modify here are the `serviceName` (this name needs to be unique) and `entityFullyQualifiedName` (the entity for which we'll be executing tests against) keys. Once you have defined your source configuration you'll need to define te processor configuration. The processor type should be set to `"orm-test-runner"`. For accepted test definition names and parameter value names refer to the [tests page](https://docs.open-metadata.org/latest/how-to-guides/data-quality-observability/quality/tests-yaml) . Note that while you can define tests directly in this YAML configuration, running the workflow will execute ALL THE TESTS present in the table, regardless of what you are defining in the YAML. This makes it easy for any user to contribute tests via the UI, while maintaining the test execution external. You can keep your YAML config as simple as follows if the table already has tests. ### Key reference: * `forceUpdate`: if the test case exists (base on the test case name) for the entity, implements the strategy to follow when running the test (i.e. whether or not to update parameters) * `testCases`: list of test cases to add to the entity referenced. Note that we will execute all the tests present in the Table. * `name`: test case name * `testDefinitionName`: test definition * `columnName`: only applies to column test. The name of the column to run the test against * `parameterValues`: parameter values of the test The `sink` and `workflowConfig` will have the same settings as the ingestion and profiler workflow. ### Full `yaml` config example ### How to Run Tests To run the tests from the CLI execute the following command dbt Integration --------------- [#### dbt Integration\ \ Learn more about how to ingest dbt models' definitions and their lineage.](https://docs.open-metadata.org/latest/connectors/ingestion/workflows/dbt) --- # Run the SAP HANA Connector Externally We use cookies to improve site navigation, analyze site usage, and enhance your user experience. Click "Accept" to enable cookies or "Reject" to reject cookies. AcceptReject connectors No menu items for this category [Home](https://docs.open-metadata.org/latest) /[Connectors](https://docs.open-metadata.org/latest/connectors) /[Database](https://docs.open-metadata.org/latest/connectors/database) /[Sap Hana](https://docs.open-metadata.org/latest/connectors/database/sap-hana) /[Yaml](https://docs.open-metadata.org/latest/connectors/database/sap-hana/yaml) OpenMetadata Documentation ![SAP HANA](https://docs.open-metadata.org/_next/image?url=%2Fimages%2Fconnectors%2Fsap-hana.webp&w=64&q=75) SAP HANA ======== PROD Available In Feature List Metadata Data Profiler Data Quality Lineage Column-level Lineage dbt Sample Data Auto-Classification Query Usage Stored Procedures Owners Tags In this section, we provide guides and references to use the SAP HANA connector. Configure and schedule SAP HANA metadata and profiler workflows from the OpenMetadata UI: * [Requirements](https://docs.open-metadata.org/latest/connectors/database/sap-hana/yaml#requirements) * [Metadata Ingestion](https://docs.open-metadata.org/latest/connectors/database/sap-hana/yaml#metadata-ingestion) * [Lineage](https://docs.open-metadata.org/latest/connectors/database/sap-hana/yaml#lineage) * [Data Profiler](https://docs.open-metadata.org/latest/connectors/database/sap-hana/yaml#data-profiler) * [Data Quality](https://docs.open-metadata.org/latest/connectors/database/sap-hana/yaml#data-quality) * [dbt Integration](https://docs.open-metadata.org/latest/connectors/database/sap-hana/yaml#dbt-integration) How to Run the Connector Externally ----------------------------------- To run the Ingestion via the UI you'll need to use the OpenMetadata Ingestion Container, which comes shipped with custom Airflow plugins to handle the workflow deployment. If, instead, you want to manage your workflows externally on your preferred orchestrator, you can check the following docs to run the Ingestion Framework **anywhere**. [#### External Schedulers\ \ Get more information about running the Ingestion Framework Externally](https://docs.open-metadata.org/latest/deployment/ingestion) Requirements ------------ [OpenMetadata 1.1 or later\ \ To deploy OpenMetadata, check the Deployment guides.](https://docs.open-metadata.org/latest/deployment) The connector is compatible with HANA or HANA express versions since HANA SPS 2. ### Python Requirements We have support for Python versions 3.9-3.11 To run the SAP HANA ingestion, you will need to install: ### Metadata To extract metadata the user used in the connection needs to have access to the `SYS` schema. You can create a new user to run the ingestion with: And, if you have password policies forcing users to reset the password, you can disable that policy for this technical user with: Note that in order to get the metadata for **Calculation Views**, **Analytics Views** and **Attribute Views**, you need to have enough permissions on the `_SYS_BIC` schema. You can grant the required permissions to the user by running the following SQL commands: The same applies to the `_SYS_REPO` schema, required for lineage extraction. ### Profiler & Data Quality Executing the profiler Workflow or data quality tests, will require the user to have `SELECT` permission on the tables/schemas where the profiler/tests will be executed. The user should also be allowed to view information in `tables` for all objects in the database. More information on the profiler workflow setup can be found [here](https://docs.open-metadata.org/latest/how-to-guides/data-quality-observability/profiler/workflow) and data quality tests [here](https://docs.open-metadata.org/latest/how-to-guides/data-quality-observability/quality) . Metadata Ingestion ------------------ All connectors are defined as JSON Schemas. [Here](https://github.com/open-metadata/OpenMetadata/blob/main/openmetadata-spec/src/main/resources/json/schema/entity/services/connections/database/sapHanaConnection.json) you can find the structure to create a connection to SAP HANA. In order to create and run a Metadata Ingestion workflow, we will follow the steps to create a YAML configuration able to connect to the source, process the Entities if needed, and reach the OpenMetadata server. The workflow is modeled around the following [JSON Schema](https://github.com/open-metadata/OpenMetadata/blob/main/openmetadata-spec/src/main/resources/json/schema/metadataIngestion/workflow.json) ### 1\. Define the YAML Config This is a sample config for SAP HANA: #### Source Configuration - Service Connection We support two possible connection types: 1. **SQL Connection**, where you will the username, password and host. 2. **HDB User Store** [connection](https://help.sap.com/docs/SAP_HANA_PLATFORM/b3ee5778bc2e4a089d3299b82ec762a7/dd95ac9dbb571014a7d7f0234d762fdb.html?version=2.0.05&locale=en-US) . Note that the HDB Store will need to be locally available to the instance running the ingestion process. If you are unsure about this setting, you can run the ingestion process passing the usual SQL connection details. ##### SQL Connection If using the SQL Connection, inform: **hostPort**: Host and port of the SAP HANA service. This should be specified as a string in the format `hostname:port`. E.g., `localhost:39041`, `host.docker.internal:39041`. **username**: Specify the User to connect to SAP HANA. It should have enough privileges to read all the metadata. **password**: Password to connect to SAP HANA. **database**: Optional parameter to connect to a specific database. **databaseSchema**: databaseSchema of the data source. This is an optional parameter, if you would like to restrict the metadata reading to a single schema. When left blank, OpenMetadata Ingestion attempts to scan all the schemas. ##### HDB User Store If you have a User Store configured, then: **userKey**: HDB Store User Key generated from the command `hdbuserstore SET `. #### Source Configuration - Source Config The `sourceConfig` is defined [here](https://github.com/open-metadata/OpenMetadata/blob/main/openmetadata-spec/src/main/resources/json/schema/metadataIngestion/databaseServiceMetadataPipeline.json) : **markDeletedTables**: To flag tables as soft-deleted if they are not present anymore in the source system. **markDeletedStoredProcedures**: Optional configuration to soft delete stored procedures in OpenMetadata if the source stored procedures are deleted. Also, if the stored procedures is deleted, all the associated entities like lineage, etc., with that stored procedures will be deleted. **markDeletedSchemas**: Optional configuration to soft delete schemas stored in OpenMetadata if the source schema is deleted. Setting this flag to true will only keep filtered schema and delete any other schemas that do not match schemaFilterPattern or do not exist at source. **markDeletedDatabases**: Optional configuration to soft delete databases stored in OpenMetadata if the source database is deleted. Setting this flag to true will only keep filtered databases and delete any other databases that do not match databaseFilterPattern or do not exist at source. **includeTables**: true or false, to ingest table data. Default is true. **includeViews**: true or false, to ingest views definitions. **includeTags**: Optional configuration to toggle the tags ingestion. **includeOwners**: Set the 'Include Owners' toggle to control whether to include owners to the ingested entity if the owner email matches with a user stored in the OM server as part of metadata ingestion. If the ingested entity already exists and has an owner, the owner will not be overwritten. **includeStoredProcedures**: Optional configuration to toggle the Stored Procedures ingestion. **includeDDL**: Optional configuration to toggle the DDL Statements ingestion. **overrideMetadata** _(boolean)_: Set the 'Override Metadata' toggle to control whether to override the existing metadata in the OpenMetadata server with the metadata fetched from the source. If the toggle is set to true, the metadata fetched from the source will override the existing metadata in the OpenMetadata server. If the toggle is set to false, the metadata fetched from the source will not override the existing metadata in the OpenMetadata server. This is applicable for fields like description, tags, owner and displayName. **queryLogDuration**: Configuration to tune how far we want to look back in query logs to process Stored Procedures results. **queryParsingTimeoutLimit**: Configuration to set the timeout for parsing the query in seconds. **useFqnForFiltering**: Regex will be applied on fully qualified name (e.g service\_name.db\_name.schema\_name.table\_name) instead of raw name (e.g. table\_name). **databaseFilterPattern**, **schemaFilterPattern**, **tableFilterPattern**: Note that the filter supports regex as include or exclude. You can find examples [here](https://docs.open-metadata.org/latest/connectors/ingestion/workflows/metadata/filter-patterns/database) **threads (beta)**: The number of threads to use when extracting the metadata using multithreading. Please take a look [here](https://docs.open-metadata.org/latest/connectors/ingestion/workflows/metadata/multithreading) before configuring this. **databaseMetadataConfigType** _(string)_: Database Source Config Metadata Pipeline type. **incremental (beta)**: Incremental Extraction configuration. Currently implemented for: * [BigQuery](https://docs.open-metadata.org/latest/connectors/ingestion/workflows/metadata/incremental-extraction/bigquery) * [Redshift](https://docs.open-metadata.org/latest/connectors/ingestion/workflows/metadata/incremental-extraction/redshift) * [Snowflake](https://docs.open-metadata.org/latest/connectors/ingestion/workflows/metadata/incremental-extraction/snowflake) #### Sink Configuration To send the metadata to OpenMetadata, it needs to be specified as `type: metadata-rest`. #### Workflow Configuration The main property here is the `openMetadataServerConfig`, where you can define the host and security provider of your OpenMetadata installation. **Logger Level** You can specify the `loggerLevel` depending on your needs. If you are trying to troubleshoot an ingestion, running with `DEBUG` will give you far more traces for identifying issues. **JWT Token** JWT tokens will allow your clients to authenticate against the OpenMetadata server. To enable JWT Tokens, you will get more details [here](https://docs.open-metadata.org/latest/deployment/security/enable-jwt-tokens) . You can refer to the JWT Troubleshooting section [link](https://docs.open-metadata.org/latest/deployment/security/jwt-troubleshooting) for any issues in your JWT configuration. **Store Service Connection** If set to `true` (default), we will store the sensitive information either encrypted via the Fernet Key in the database or externally, if you have configured any [Secrets Manager](https://docs.open-metadata.org/latest/deployment/secrets-manager) . If set to `false`, the service will be created, but the service connection information will only be used by the Ingestion Framework at runtime, and won't be sent to the OpenMetadata server. **Store Service Connection** If set to `true` (default), we will store the sensitive information either encrypted via the Fernet Key in the database or externally, if you have configured any [Secrets Manager](https://docs.open-metadata.org/latest/deployment/secrets-manager) . If set to `false`, the service will be created, but the service connection information will only be used by the Ingestion Framework at runtime, and won't be sent to the OpenMetadata server. **SSL Configuration** If you have added SSL to the [OpenMetadata server](https://docs.open-metadata.org/latest/deployment/security/enable-ssl) , then you will need to handle the certificates when running the ingestion too. You can either set `verifySSL` to `ignore`, or have it as `validate`, which will require you to set the `sslConfig.caCertificate` with a local path where your ingestion runs that points to the server certificate file. Find more information on how to troubleshoot SSL issues [here](https://docs.open-metadata.org/latest/deployment/security/enable-ssl/ssl-troubleshooting) . **ingestionPipelineFQN** Fully qualified name of ingestion pipeline, used to identify the current ingestion pipeline. #### Advanced Configuration **Connection Options (Optional)**: Enter the details for any additional connection options that can be sent to database during the connection. These details must be added as Key-Value pairs. **Connection Arguments (Optional)**: Enter the details for any additional connection arguments such as security or protocol configs that can be sent to database during the connection. These details must be added as Key-Value pairs. * In case you are using Single-Sign-On (SSO) for authentication, add the `authenticator` details in the Connection Arguments as a Key-Value pair as follows: `"authenticator" : "sso_login_url"` filename.yamlCopy ### 2\. Run with the CLI First, we will need to save the YAML file. Afterward, and with all requirements installed, we can run: Note that from connector to connector, this recipe will always be the same. By updating the YAML configuration, you will be able to extract metadata from different sources. Lineage ------- After running a Metadata Ingestion workflow, we can run Lineage workflow. While the `serviceName` will be the same to that was used in Metadata Ingestion, so the ingestion bot can get the `serviceConnection` details from the server. ### 1\. Define the YAML Config This is a sample config for BigQuery Lineage: #### Source Configuration - Source Config You can find all the definitions and types for the `sourceConfig` [here](https://github.com/open-metadata/OpenMetadata/blob/main/openmetadata-spec/src/main/resources/json/schema/metadataIngestion/databaseServiceQueryLineagePipeline.json) . **queryLogDuration**: Configuration to tune how far we want to look back in query logs to process lineage data in days. **parsingTimeoutLimit**: Configuration to set the timeout for parsing the query in seconds. **filterCondition**: Condition to filter the query history. **resultLimit**: Configuration to set the limit for query logs. **queryLogFilePath**: Configuration to set the file path for query logs. **databaseFilterPattern**: Regex to only fetch databases that matches the pattern. **schemaFilterPattern**: Regex to only fetch tables or databases that matches the pattern. **tableFilterPattern**: Regex to only fetch tables or databases that matches the pattern. **overrideViewLineage**: Set the 'Override View Lineage' toggle to control whether to override the existing view lineage. **processViewLineage**: Set the 'Process View Lineage' toggle to control whether to process view lineage. **processQueryLineage**: Set the 'Process Query Lineage' toggle to control whether to process query lineage. **processStoredProcedureLineage**: Set the 'Process Stored ProcedureLog Lineage' toggle to control whether to process stored procedure lineage. **threads**: Number of Threads to use in order to parallelize lineage ingestion. #### Sink Configuration To send the metadata to OpenMetadata, it needs to be specified as `type: metadata-rest`. #### Workflow Configuration The main property here is the `openMetadataServerConfig`, where you can define the host and security provider of your OpenMetadata installation. For a simple, local installation using our docker containers, this looks like: filename.yamlCopy * You can learn more about how to configure and run the Lineage Workflow to extract Lineage data from [here](https://docs.open-metadata.org/latest/connectors/ingestion/workflows/lineage) ### 2\. Run with the CLI After saving the YAML config, we will run the command the same way we did for the metadata ingestion: Data Profiler ------------- The Data Profiler workflow will be using the `orm-profiler` processor. After running a Metadata Ingestion workflow, we can run the Data Profiler workflow. While the `serviceName` will be the same to that was used in Metadata Ingestion, so the ingestion bot can get the `serviceConnection` details from the server. ### 1\. Define the YAML Config This is a sample config for the profiler: #### Source Configuration - Source Config You can find all the definitions and types for the `sourceConfig` [here](https://github.com/open-metadata/OpenMetadata/blob/main/openmetadata-spec/src/main/resources/json/schema/metadataIngestion/databaseServiceProfilerPipeline.json) . **profileSample**: Percentage of data or no. of rows we want to execute the profiler and tests on. **threadCount**: Number of threads to use during metric computations. **timeoutSeconds**: Profiler Timeout in Seconds **databaseFilterPattern**: Regex to only fetch databases that matches the pattern. **schemaFilterPattern**: Regex to only fetch tables or databases that matches the pattern. **tableFilterPattern**: Regex to only fetch tables or databases that matches the pattern. #### Processor Configuration Choose the `orm-profiler`. Its config can also be updated to define tests from the YAML itself instead of the UI: **tableConfig**: `tableConfig` allows you to set up some configuration at the table level. #### Sink Configuration To send the metadata to OpenMetadata, it needs to be specified as `type: metadata-rest`. #### Workflow Configuration The main property here is the `openMetadataServerConfig`, where you can define the host and security provider of your OpenMetadata installation. **Logger Level** You can specify the `loggerLevel` depending on your needs. If you are trying to troubleshoot an ingestion, running with `DEBUG` will give you far more traces for identifying issues. **JWT Token** JWT tokens will allow your clients to authenticate against the OpenMetadata server. To enable JWT Tokens, you will get more details [here](https://docs.open-metadata.org/latest/deployment/security/enable-jwt-tokens) . You can refer to the JWT Troubleshooting section [link](https://docs.open-metadata.org/latest/deployment/security/jwt-troubleshooting) for any issues in your JWT configuration. **Store Service Connection** If set to `true` (default), we will store the sensitive information either encrypted via the Fernet Key in the database or externally, if you have configured any [Secrets Manager](https://docs.open-metadata.org/latest/deployment/secrets-manager) . If set to `false`, the service will be created, but the service connection information will only be used by the Ingestion Framework at runtime, and won't be sent to the OpenMetadata server. **Store Service Connection** If set to `true` (default), we will store the sensitive information either encrypted via the Fernet Key in the database or externally, if you have configured any [Secrets Manager](https://docs.open-metadata.org/latest/deployment/secrets-manager) . If set to `false`, the service will be created, but the service connection information will only be used by the Ingestion Framework at runtime, and won't be sent to the OpenMetadata server. **SSL Configuration** If you have added SSL to the [OpenMetadata server](https://docs.open-metadata.org/latest/deployment/security/enable-ssl) , then you will need to handle the certificates when running the ingestion too. You can either set `verifySSL` to `ignore`, or have it as `validate`, which will require you to set the `sslConfig.caCertificate` with a local path where your ingestion runs that points to the server certificate file. Find more information on how to troubleshoot SSL issues [here](https://docs.open-metadata.org/latest/deployment/security/enable-ssl/ssl-troubleshooting) . **ingestionPipelineFQN** Fully qualified name of ingestion pipeline, used to identify the current ingestion pipeline. filename.yamlCopy * You can learn more about how to configure and run the Profiler Workflow to extract Profiler data and execute the Data Quality from [here](https://docs.open-metadata.org/latest/how-to-guides/data-quality-observability/profiler/workflow) ### 2\. Run with the CLI After saving the YAML config, we will run the command the same way we did for the metadata ingestion: Note now instead of running `ingest`, we are using the `profile` command to select the Profiler workflow. [#### Data Profiler\ \ Find more information about the Data Profiler here](https://docs.open-metadata.org/latest/how-to-guides/data-quality-observability/profiler/workflow) Auto Classification ------------------- The Auto Classification workflow will be using the `orm-profiler` processor. After running a Metadata Ingestion workflow, we can run the Auto Classification workflow. While the `serviceName` will be the same to that was used in Metadata Ingestion, so the ingestion bot can get the `serviceConnection` details from the server. ### 1\. Define the YAML Config This is a sample config for the Auto Classification Workflow: #### Source Configuration - Source Config You can find all the definitions and types for the `sourceConfig` [here](https://github.com/open-metadata/OpenMetadata/blob/main/openmetadata-spec/src/main/resources/json/schema/metadataIngestion/databaseServiceAutoClassificationPipeline.json) . **storeSampleData**: Option to turn on/off storing sample data. If enabled, we will ingest sample data for each table. **enableAutoClassification**: Optional configuration to automatically tag columns that might contain sensitive information. **confidence**: Set the Confidence value for which you want the column to be tagged as PII. Confidence value ranges from 0 to 100. A higher number will yield less false positives but more false negatives. A lower number will yield more false positives but less false negatives. **databaseFilterPattern**: Regex to only fetch databases that matches the pattern. **schemaFilterPattern**: Regex to only fetch tables or databases that matches the pattern. **tableFilterPattern**: Regex to only fetch tables or databases that matches the pattern. #### Processor Configuration Choose the `orm-profiler`. Its config can also be updated to define tests from the YAML itself instead of the UI: **tableConfig**: `tableConfig` allows you to set up some configuration at the table level. #### Sink Configuration To send the metadata to OpenMetadata, it needs to be specified as `type: metadata-rest`. #### Workflow Configuration The main property here is the `openMetadataServerConfig`, where you can define the host and security provider of your OpenMetadata installation. **Logger Level** You can specify the `loggerLevel` depending on your needs. If you are trying to troubleshoot an ingestion, running with `DEBUG` will give you far more traces for identifying issues. **JWT Token** JWT tokens will allow your clients to authenticate against the OpenMetadata server. To enable JWT Tokens, you will get more details [here](https://docs.open-metadata.org/latest/deployment/security/enable-jwt-tokens) . You can refer to the JWT Troubleshooting section [link](https://docs.open-metadata.org/latest/deployment/security/jwt-troubleshooting) for any issues in your JWT configuration. **Store Service Connection** If set to `true` (default), we will store the sensitive information either encrypted via the Fernet Key in the database or externally, if you have configured any [Secrets Manager](https://docs.open-metadata.org/latest/deployment/secrets-manager) . If set to `false`, the service will be created, but the service connection information will only be used by the Ingestion Framework at runtime, and won't be sent to the OpenMetadata server. **Store Service Connection** If set to `true` (default), we will store the sensitive information either encrypted via the Fernet Key in the database or externally, if you have configured any [Secrets Manager](https://docs.open-metadata.org/latest/deployment/secrets-manager) . If set to `false`, the service will be created, but the service connection information will only be used by the Ingestion Framework at runtime, and won't be sent to the OpenMetadata server. **SSL Configuration** If you have added SSL to the [OpenMetadata server](https://docs.open-metadata.org/latest/deployment/security/enable-ssl) , then you will need to handle the certificates when running the ingestion too. You can either set `verifySSL` to `ignore`, or have it as `validate`, which will require you to set the `sslConfig.caCertificate` with a local path where your ingestion runs that points to the server certificate file. Find more information on how to troubleshoot SSL issues [here](https://docs.open-metadata.org/latest/deployment/security/enable-ssl/ssl-troubleshooting) . **ingestionPipelineFQN** Fully qualified name of ingestion pipeline, used to identify the current ingestion pipeline. filename.yamlCopy ### 2\. Run with the CLI After saving the YAML config, we will run the command the same way we did for the metadata ingestion: Now instead of running `ingest`, we are using the `classify` command to select the Auto Classification workflow. Data Quality ------------ ### Adding Data Quality Test Cases from yaml config When creating a JSON config for a test workflow the source configuration is very simple. The only sections you need to modify here are the `serviceName` (this name needs to be unique) and `entityFullyQualifiedName` (the entity for which we'll be executing tests against) keys. Once you have defined your source configuration you'll need to define te processor configuration. The processor type should be set to `"orm-test-runner"`. For accepted test definition names and parameter value names refer to the [tests page](https://docs.open-metadata.org/latest/how-to-guides/data-quality-observability/quality/tests-yaml) . Note that while you can define tests directly in this YAML configuration, running the workflow will execute ALL THE TESTS present in the table, regardless of what you are defining in the YAML. This makes it easy for any user to contribute tests via the UI, while maintaining the test execution external. You can keep your YAML config as simple as follows if the table already has tests. ### Key reference: * `forceUpdate`: if the test case exists (base on the test case name) for the entity, implements the strategy to follow when running the test (i.e. whether or not to update parameters) * `testCases`: list of test cases to add to the entity referenced. Note that we will execute all the tests present in the Table. * `name`: test case name * `testDefinitionName`: test definition * `columnName`: only applies to column test. The name of the column to run the test against * `parameterValues`: parameter values of the test The `sink` and `workflowConfig` will have the same settings as the ingestion and profiler workflow. ### Full `yaml` config example ### How to Run Tests To run the tests from the CLI execute the following command dbt Integration --------------- [#### dbt Integration\ \ Learn more about how to ingest dbt models' definitions and their lineage.](https://docs.open-metadata.org/latest/connectors/ingestion/workflows/dbt) --- # Domo Dashboard | OpenMetadata Connector Setup Guide We use cookies to improve site navigation, analyze site usage, and enhance your user experience. Click "Accept" to enable cookies or "Reject" to reject cookies. AcceptReject connectors No menu items for this category [Home](https://docs.open-metadata.org/latest) /[Connectors](https://docs.open-metadata.org/latest/connectors) /[Dashboard](https://docs.open-metadata.org/latest/connectors/dashboard) /[Domo Dashboard](https://docs.open-metadata.org/latest/connectors/dashboard/domo-dashboard) OpenMetadata Documentation ![Domo](https://docs.open-metadata.org/_next/image?url=%2Fimages%2Fconnectors%2Fdomo.webp&w=64&q=75) Domo ==== PROD Available In Feature List Dashboards Charts Owners Tags Datamodels Projects Lineage In this section, we provide guides and references to use the DomoDashboard connector. Configure and schedule DomoDashboard metadata and profiler workflows from the OpenMetadata UI: * [Requirements](https://docs.open-metadata.org/latest/connectors/dashboard/domo-dashboard#requirements) * [Metadata Ingestion](https://docs.open-metadata.org/latest/connectors/dashboard/domo-dashboard#metadata-ingestion) * [Troubleshooting](https://docs.open-metadata.org/latest/connectors/dashboard/domo-dashboard/troubleshooting) Ingestion Deployment -------------------- To run the Ingestion via the UI you'll need to use the OpenMetadata Ingestion Container, which comes shipped with custom Airflow plugins to handle the workflow deployment. If you want to install it manually in an already existing Airflow host, you can follow [this](https://docs.open-metadata.org/latest/deployment/ingestion/openmetadata) guide. If you don't want to use the OpenMetadata Ingestion container to configure the workflows via the UI, then you can check the following docs to run the Ingestion Framework in any orchestrator externally. [#### Run Connectors from the OpenMetadata UI\ \ Learn how to manage your deployment to run connectors from the UI](https://docs.open-metadata.org/latest/deployment/ingestion/openmetadata) [#### Run the Connector Externally\ \ Get the YAML to run the ingestion externally](https://docs.open-metadata.org/latest/connectors/dashboard/domo-dashboard/yaml) [#### External Schedulers\ \ Get more information about running the Ingestion Framework Externally](https://docs.open-metadata.org/latest/deployment/ingestion) Requirements ------------ For metadata ingestion, make sure to add at least `data` scopes to the clientId provided. For questions related to scopes, click [here](https://developer.domo.com/portal/1845fc11bbe5d-api-authentication) . Metadata Ingestion ------------------ #### 1\. Visit the Services Page Click `Settings` in the side navigation bar and then `Services`. The first step is to ingest the metadata from your sources. To do that, you first need to create a Service connection first. This Service will be the bridge between OpenMetadata and your source system. Once a Service is created, it can be used to configure your ingestion workflows. ![Visit Services Page](https://docs.open-metadata.org/images/v1.11/connectors/visit-services-page.png) Select your Service Type and Add a New Service #### 2\. Create a New Service Click on _Add New Service_ to start the Service creation. ![Create a new Service](https://docs.open-metadata.org/images/v1.11/connectors/create-new-service.png) Add a new Service from the Services page #### 3\. Select the Service Type Select Domo Dashboard as the Service type and click _Next_. ![Select Service](https://docs.open-metadata.org/images/v1.11/connectors/domodashboard/select-service.png) Select your Service from the list #### 4\. Name and Describe your Service Provide a name and description for your Service. #### Service Name OpenMetadata uniquely identifies Services by their **Service Name**. Provide a name that distinguishes your deployment from other Services, including the other Domo Dashboard Services that you might be ingesting metadata from. Note that when the name is set, it cannot be changed. ![Add New Service](https://docs.open-metadata.org/images/v1.11/connectors/domodashboard/add-new-service.png) Provide a Name and description for your Service #### 5\. Configure the Service Connection In this step, we will configure the connection settings required for Domo Dashboard. Please follow the instructions below to properly configure the Service to read from your sources. You will also find helper documentation on the right-hand side panel in the UI. ![Configure Service connection](https://docs.open-metadata.org/images/v1.11/connectors/domodashboard/service-connection.png) Configure the Service connection by filling the form #### Connection Details * **Client ID**: Client ID to Connect to DOMO Dashboard. * **Secret Token**: Secret Token to Connect DOMO Dashboard. * **Access Token**: Access to Connect to DOMO Dashboard. * **API Host**: API Host to Connect to DOMO Dashboard instance. * **Instance Domain**: URL to connect to your Domo instance UI. For example `https://.domo.com`. #### 6\. Test the Connection Once the credentials have been added, click on _Test Connection_ and _Save_ the changes. ![Test Connection](https://docs.open-metadata.org/images/v1.11/connectors/test-connection.png) Test the connection and save the Service #### 7\. Configure Metadata Ingestion In this step we will configure the metadata ingestion pipeline, Please follow the instructions below ![Configure Metadata Ingestion](https://docs.open-metadata.org/images/v1.11/connectors/configure-metadata-ingestion-dashboard.png) Configure Metadata Ingestion Page #### Metadata Ingestion Options * **Name**: This field refers to the name of ingestion pipeline, you can customize the name or use the generated name. * **Dashboard Filter Pattern (Optional)**: Use it to control whether to include dashboard as part of metadata ingestion. * **Include**: Explicitly include dashboards by adding a list of comma-separated regular expressions to the 'Include' field. OpenMetadata will include all dashboards with names matching one or more of the supplied regular expressions. All other dashboards will be excluded. * **Exclude**: Explicitly exclude dashboards by adding a list of comma-separated regular expressions to the 'Exclude' field. OpenMetadata will exclude all dashboards with names matching one or more of the supplied regular expressions. All other dashboards will be included. * **projectFilterPattern**: Filter the dashboards, charts and data sources by projects. Note that all of them support regex as include or exclude. E.g., "My project, My proj.\*, .\*Project". We filter the projects by concatenating the entire project hierarchy using dot notation (e.g., `Project1.NestedProjectA.OtherProject`). Make sure the regex filter pattern accounts for this fully-qualified format. * **Chart Pattern (Optional)**: Use it to control whether to include charts as part of metadata ingestion. * **Include**: Explicitly include charts by adding a list of comma-separated regular expressions to the 'Include' field. OpenMetadata will include all charts with names matching one or more of the supplied regular expressions. All other charts will be excluded. * **Exclude**: Explicitly exclude charts by adding a list of comma-separated regular expressions to the 'Exclude' field. OpenMetadata will exclude all charts with names matching one or more of the supplied regular expressions. All other charts will be included. * **Data Model Pattern (Optional)**: Use it to control whether to include data modes as part of metadata ingestion. * **Include**: Explicitly include data models by adding a list of comma-separated regular expressions to the 'Include' field. OpenMetadata will include all data models with names matching one or more of the supplied regular expressions. All other data models will be excluded. * **Exclude**: Explicitly exclude data models by adding a list of comma-separated regular expressions to the 'Exclude' field. OpenMetadata will exclude all data models with names matching one or more of the supplied regular expressions. All other data models will be included. * **Db Service Prefixes (Optional)**: Enter the names of Database Services which are already ingested in OpenMetadata to create lineage between dashboards and database tables. * **Enable Debug Log (toggle)**: Set the 'Enable Debug Log' toggle to set the default log level to debug. * **Include Owners (toggle)**: Set the 'Include Owners' toggle to control whether to include owners to the ingested entity if the owner email matches with a user stored in the OM server as part of metadata ingestion. If the ingested entity already exists and has an owner, the owner will not be overwritten. * **Include Tags (toggle)**: Set the 'Include Tags' toggle to control whether to include tags in metadata ingestion. * **Include Data Models (toggle)**: Set the 'Include Data Models' toggle to control whether to include tags as part of metadata ingestion. * **Mark Deleted Dashboards (toggle)**: Set the 'Mark Deleted Dashboards' toggle to flag dashboards as soft-deleted if they are not present anymore in the source system. * **Include Draft Dashboard (toggle)**: Set the 'Include Draft Dashboard' toggle to include draft dashboards. By default it will include draft dashboards. #### 8\. Schedule the Ingestion and Deploy Scheduling can be set up at an hourly, daily, weekly, or manual cadence. The timezone is in UTC. Select a Start Date to schedule for ingestion. It is optional to add an End Date. Review your configuration settings. If they match what you intended, click Deploy to create the service and schedule metadata ingestion. If something doesn't look right, click the Back button to return to the appropriate step and change the settings as needed. After configuring the workflow, you can click on Deploy to create the pipeline. ![Schedule the Workflow](https://docs.open-metadata.org/images/v1.11/connectors/schedule.png) Schedule the Ingestion Pipeline and Deploy #### 9\. View the Ingestion Pipeline Once the workflow has been successfully deployed, you can view the Ingestion Pipeline running from the Service Page. ![View Ingestion Pipeline](https://docs.open-metadata.org/images/v1.11/connectors/view-ingestion-pipeline.png) View the Ingestion Pipeline from the Service Page --- # Docker Deployment | OpenMetadata Container Setup We use cookies to improve site navigation, analyze site usage, and enhance your user experience. Click "Accept" to enable cookies or "Reject" to reject cookies. AcceptReject deployment No menu items for this category [Home](https://docs.open-metadata.org/latest) /[Deployment](https://docs.open-metadata.org/latest/deployment) /[Docker](https://docs.open-metadata.org/latest/deployment/docker) OpenMetadata Documentation Docker Deployment ================= This guide will help you set up the OpenMetadata Application using Docker Deployment. Before starting with the deployment make sure you follow all the below Prerequisites. Docker Deployment Architecture ------------------------------ ![Docker Deployment Architecture](https://docs.open-metadata.org/images/v1.11/deployment/docker/om_docker_architecture.png) Prerequisites ------------- ### Configure OpenMetadata to use External Database and Search Engine For Production Deployment using Docker, we recommend bringing your own Databases and ElasticSearch Engine and not rely on quickstart packages. ### Configure External Orchestrator Service (Ingestion Service) OpenMetadata requires connectors to be scheduled to periodically fetch the metadata, or you can use the OpenMetadata APIs to push the metadata as well 1. OpenMetadata Ingestion Framework is flexible to run on any orchestrator. However, we built an ability to deploy and manage connectors as pipelines from the UI. This requires the Airflow container we ship. 2. If your team prefers to run on any other orchestrator such as prefect, dagster or even GitHub workflows. Please refer to our recent webinar on [How Ingestion Framework works](https://www.youtube.com/watch?v=i7DhG_gZMmE&list=PLa1l-WDhLreslIS_96s_DT_KdcDyU_Itv&index=10) ### Docker (version 20.10.0 or higher) [Docker](https://docs.docker.com/get-started/overview/) is an open-source platform for developing, shipping, and running applications. It enables you to separate your applications from your infrastructure, so you can deliver software quickly using OS-level virtualization. It helps deliver software in packages called Containers. To check what version of Docker you have, please use the following command. If you need to install Docker, please visit [Get Docker](https://docs.docker.com/get-docker/) . ### Docker Compose (version v2.2.3 or greater) The Docker compose package enables you to define and run multi-container Docker applications. The compose command integrates compose functions into the Docker platform, making them available from the Docker command-line interface ( CLI). The Python packages you will install in the procedure below use compose to deploy OpenMetadata. * **MacOS X**: Docker on MacOS X ships with compose already available in the Docker CLI. * **Linux**: To install compose on Linux systems, please visit the Docker CLI command documentation and follow the instructions. To verify that the docker compose command is installed and accessible on your system, run the following command. Upon running this command you should see output similar to the following. #### Install Docker Compose Version 2 on Linux Follow the instructions [here](https://docs.docker.com/compose/cli-command/#install-on-linux) to install docker compose version 2 1. Run the following command to download the current stable release of Docker Compose This command installs Compose V2 for the active user under $HOME directory. To install Docker Compose for all users on your system, replace `~/.docker/cli-plugins` with `/usr/local/lib/docker/cli-plugins`. 2. Apply executable permissions to the binary 3. Test your installation Steps for Deploying OpenMetadata using Docker --------------------------------------------- ### 1\. Create a directory for OpenMetadata Create a new directory for OpenMetadata and navigate into that directory. ### 2\. Download Docker Compose Files from GitHub Releases Download the Docker Compose files from the [Latest GitHub Releases](https://github.com/open-metadata/OpenMetadata/releases/latest) . The Docker compose file name will be `docker-compose-openmetadata.yml`. This docker compose file contains only the docker compose services for OpenMetadata Server. Bring up the dependencies as mentioned in the [prerequisites](https://docs.open-metadata.org/latest/deployment/docker#configure-openmetadata-to-use-external-database-and-search-engine) section. You can also run the below command to fetch the docker compose file directly from the terminal - ### 3\. Update Environment Variables required for OpenMetadata Dependencies In the previous [step](https://docs.open-metadata.org/latest/deployment/docker#2.-download-docker-compose-files-from-github-releases) , we download the `docker-compose` file. Identify and update the environment variables in the file to prepare openmetadata configurations. For MySQL Configurations, update the below environment variables - For ElasticSearch Configurations, update the below environment variables - For OpenSearch Configurations, update the below environment variables - If you want to separate indexes for production and non-production environments, you can set the `clusterAlias` in the configuration file. For Ingestion Configurations, update the below environment variables - When setting up environment file if your custom password includes any special characters then make sure to follow the steps [here](https://github.com/open-metadata/OpenMetadata/issues/12110#issuecomment-1611341650) . ### 4\. Start the Docker Compose Services Run the below command to deploy the OpenMetadata - You can validate that all containers are up by running with command `docker ps`. In a few seconds, you should be able to access the OpenMetadata UI at [http://localhost:8585](http://localhost:8585/) Port Mapping / Port Forwarding ------------------------------ We are shipping the OpenMetadata server and UI at container port and host port `8585`. You can change the host port number according to your requirement. As an example, You could update the ports to serve OpenMetadata Server and UI at port `80` To achieve this - * You just have to update the ports mapping of the openmetadata-server in the `docker-compose.yml` file under `openmetadata-server` docker service section. * Once the port is updated if there are any containers running remove them first using `docker compose down` command and then recreate the containers once again by below command Run OpenMetadata with a load balancer ------------------------------------- You may put one or more OpenMetadata instances behind a load balancer for reverse proxying. To do this you will need to add one or more entries to the configuration file for your reverse proxy. ### Nginx To use OpenMetadata behind Nginx reverse proxy, add an entry resembling the following the http context of your Nginx configuration file for each OpenMetadata instance. Run OpenMetadata with AWS Services ---------------------------------- If you are running OpenMetadata in AWS, it is recommended to use [Amazon RDS](https://docs.aws.amazon.com/rds/index.html) and [Amazon OpenSearch Service](https://docs.aws.amazon.com/opensearch-service/?id=docs_gateway) . We support * Amazon RDS (MySQL) engine version 8 or higher * Amazon OpenSearch (ElasticSearch) engine version up to 8.11.4 or Amazon OpenSearch engine version up to 2.19 * Amazon RDS (PostgreSQL) engine version 12 or higher Note:- When using AWS Services the SearchType Configuration for elastic search should be `opensearch`, for both cases ElasticSearch and OpenSearch, as you can see in the ElasticSearch configuration example. For Production Systems, we recommend Amazon RDS to be in Multiple Availability Zones. For Amazon OpenSearch (or ElasticSearch) Service, we recommend Multiple Availability Zones with minimum 3 Master Nodes. Once you have the RDS and OpenSearch Services Setup, you can update the environment variables below for OpenMetadata Docker Compose backed systems to connect with Database and ElasticSearch. Replace the environment variables values with the RDS and OpenSearch Service ones and then provide this environment variable file as part of docker compose command. Advanced -------- ### Add Docker Volumes for OpenMetadata Server Compose Service There are many scenarios where you would want to provide additional files to the OpenMetadata Server and serve while running the application. In such scenarios, it is recommended to provision docker volumes for OpenMetadata Application. If you are not familiar with Docker Volumes with Docker Compose Services, Please refer to [official documentation](https://docs.docker.com/storage/volumes/#use-a-volume-with-docker-compose) for more information. For example, we would like to provide custom JWT Configuration Keys to be served to OpenMetadata Application. This requires the OpenMetadata Containers to have docker volumes sharing the private and public keys. Let's assume you have the keys available in `jwtkeys` directory in the same directory where your `docker-compose` file is available in the host machine. In scenarios where you need to provide a custom `openmetadata.yaml` configuration file to the OpenMetadata application, you can do so by mounting the file as a volume in the Docker container. This is especially useful for configurations that cannot be controlled through environment variables. We add the volumes section to mount the keys or `openmetadata.yaml` onto the docker containers create with docker compose as follows - The above example uses [bind mounts](https://docs.docker.com/storage/bind-mounts/#use-a-bind-mount-with-compose) to share files and directories between host machine and openmetadata container. Next, in your environment file, update the jwt configurations to use the right path from inside the container. Ensure that the default environment variables are set appropriately to complement the settings in your `openmetadata.yaml`. Once the changes are updated, if there are any containers running remove them first using `docker compose down` command and then recreate the containers once again by below command Troubleshooting --------------- ### Java Memory Heap Issue If your openmetadata Docker Compose logs speaks about the below issue - This is due to the default JVM Heap Space configuration (1 GiB) being not enough for your workloads. In order to resolve this issue, head over to your custom openmetadata environment variable file and append the below environment variable The flag `Xmx` specifies the maximum memory allocation pool for a Java virtual machine (JVM), while `Xms` specifies the initial memory allocation pool. Restart the OpenMetadata Docker Compose Application using `docker compose --env-file -f docker-compose.yml up --detach` which will recreate the containers with new environment variable values you have provided. ### PostgreSQL Issue permission denied to create extension "pgcrypto" If you are facing the below issue with PostgreSQL as Database Backend for OpenMetadata Application, It seems the Database User does not have sufficient privileges. In order to resolve the above issue, grant usage permissions to the PSQL User. In the above command, replace `` with the sql user used by OpenMetadata Application to connect to PostgreSQL Database. In the above command, replace `` with the sql user used by OpenMetadata Application to connect to PostgreSQL Database. Security -------- Please follow our [Enable Security Guide](https://docs.open-metadata.org/latest/deployment/docker/security) to configure security for your OpenMetadata installation. Next Steps ---------- 1. Refer the [How-to Guides](https://docs.open-metadata.org/latest/how-to-guides) for an overview of all the features in OpenMetadata. 2. Visit the [Connectors](https://docs.open-metadata.org/latest/connectors) documentation to see what services you can integrate with OpenMetadata. 3. Visit the [API](https://docs.open-metadata.org/swagger.html) documentation and explore the rich set of OpenMetadata APIs. --- # AWS Cognito SSO Setup Guide for Public Apps We use cookies to improve site navigation, analyze site usage, and enhance your user experience. Click "Accept" to enable cookies or "Reject" to reject cookies. AcceptReject ![](https://static.scarf.sh/a.png?x-pxid=8472f2b3-d307-4935-a5ed-3f9d3982c15d) deployment No menu items for this category [Home](https://docs.open-metadata.org/latest) /[Deployment](https://docs.open-metadata.org/latest/deployment) /[Security](https://docs.open-metadata.org/latest/deployment/security) /[Amazon Cognito](https://docs.open-metadata.org/latest/deployment/security/amazon-cognito) /[Public Client](https://docs.open-metadata.org/latest/deployment/security/amazon-cognito/public-client) OpenMetadata Documentation AWS Cognito SSO Configuration (Public) ====================================== * [Troubleshooting](https://docs.open-metadata.org/latest/deployment/security/amazon-cognito/public-client#troubleshooting) AWS Cognito SSO enables users to log in using their credentials from a Cognito User Pool through OAuth 2.0 and OpenID Connect (OIDC). This guide walks you through configuring AWS Cognito as an authentication provider in OpenMetadata. Public Configuration Fields --------------------------- ![AWS Cognito SSO Configuration - Public Client](https://docs.open-metadata.org/images/v1.11/deployment/security/amazon-cognito-sso/cognito1.png) ### Provider Name * **Definition:** Human-readable name for this Cognito SSO instance. * **Example:** `AWS Cognito SSO`, `Company Cognito` * **Note:** Used only for display and logging purposes. ### Client Type * **Definition:** Defines whether the app is public (no secret) or confidential (requires client secret). * **Options:** Public | Confidential * **Example:** Confidential * **Note:** * Use **Public** for SPAs or mobile apps * Use **Confidential** for web apps or backends ### OIDC Client ID * **Definition:** Client ID from the Cognito User Pool App. * **Example:** `1a2b3c4d5e6f7g8h9i0j1k2l3m4n5o6p` * **Note:** Found under **Cognito > User Pools > App Integration > App Clients** ### OIDC Callback URL * **Definition:** URI where Cognito redirects after authentication. * **Example:** `https://yourapp.company.com/callback` * **Note:** * Must be registered in **Cognito > Allowed Callback URLs** * Use HTTPS in production ### Enable Self Signup * **Definition:** Allows new users to auto-create accounts upon first login. * **Example:** Enabled * **Note:** Cognito must also allow sign-ups. ### Authority * **Definition:** AWS Cognito token-issuing domain. * **Example:** `https://cognito-idp.us-east-1.amazonaws.com/us-east-1_ABC123DEF` * **Note:** Replace with your region and User Pool ID ### Public Key URLs * **Definition:** JWKS URLs used to verify token signatures. * **Example:** `["https://cognito-idp.us-east-1.amazonaws.com/us-east-1_ABC123DEF/.well-known/jwks.json"]` ### Token Validation Algorithm * **Definition:** Algorithm to validate JWT tokens. * **Options:** RS256 | RS384 | RS512 * **Default:** RS256 ### JWT Principal Claims * **Definition:** Claims used to identify the user. * **Example:** `["cognito:username", "email", "sub"]` * **Note:** Typical Cognito claims include `cognito:username`, `email`, `sub`, `preferred_username` ### JWT Principal Claims Mapping * **Definition:** Maps claims to OpenMetadata user fields. * **Example:** `["email:email", "name:name", "firstName:given_name"]` * **Note:** Format - `"openmetadata_field:jwt_claim"` ### Admin Principals * **Definition:** List of users with admin access. * **Example:** `["admin@company.com", "superuser@company.com"]` ### Principal Domain * **Definition:** Default domain for users. * **Example:** `company.com` * **Note:** Helps construct full identity from usernames. ### Enforce Principal Domain * **Definition:** Restrict access to users from a specific domain. * **Example:** true * **Default:** false ### Enable Secure Socket Connection * **Definition:** Enables SSL/TLS for secure communication. * **Example:** true * **Default:** false * **Note:** Recommended in production Summary Table ------------- | **Field** | **Example / Default** | | --- | --- | | Provider Name | AWS Cognito SSO | | Client Type | Confidential | | OIDC Client ID | 1a2b3c4d5e6f7g8h9i0j1k2l3m4n5o6p | | OIDC Callback URL | https://yourapp.company.com/callback | | Enable Self Signup | Enabled | | Authority | https://cognito-idp.us-east-1.amazonaws.com/us-east-1\_ABC123DEF | | Public Key URLs | https://cognito-idp.us-east-1.amazonaws.com/us-east-1\_ABC123DEF/.well-known/jwks.json | | Token Validation Algorithm | RS256 | | JWT Principal Claims | \["cognito:username", "email", "sub"\] | | JWT Mapping | \["email:email", "name:name", "firstName:given\_name"\] | | Admin Principals | \["admin@company.com"\] | | Principal Domain | company.com | | Enforce Principal Domain | false | | SSL/TLS | true | ### Troubleshooting If users are automatically logged out and unable to log in again due to a bad authentication configuration, you can reset the security setup using the following command: After executing the command, **restart the server**. The authentication values from your YAML or Helm chart will then be reapplied on startup. The following tiles detail how to apply this configuration across Docker, Kubernetes, and Bare Metal deployments: [Docker Security\ \ Configure SSO for your Docker Deployment.](https://docs.open-metadata.org/latest/deployment/security/configuration/docker) [Bare Metal Security\ \ Configure SSO for your Bare Metal Deployment.](https://docs.open-metadata.org/latest/deployment/security/configuration/bare-metal) [Kubernetes Security\ \ Configure SSO for your Kubernetes Deployment.](https://docs.open-metadata.org/latest/deployment/security/configuration/kubernetes) --- # Implicit flow of Keyclock | Official Documentation We use cookies to improve site navigation, analyze site usage, and enhance your user experience. Click "Accept" to enable cookies or "Reject" to reject cookies. AcceptReject ![](https://static.scarf.sh/a.png?x-pxid=8472f2b3-d307-4935-a5ed-3f9d3982c15d) deployment No menu items for this category [Home](https://docs.open-metadata.org/latest) /[Deployment](https://docs.open-metadata.org/latest/deployment) /[Security](https://docs.open-metadata.org/latest/deployment/security) /[Keycloak](https://docs.open-metadata.org/latest/deployment/security/keycloak) /[Implicit Flow](https://docs.open-metadata.org/latest/deployment/security/keycloak/implicit-flow) OpenMetadata Documentation Implicit Flow ============= ### Step 1: Create OpenMetadata as a new Client * Click on `Clients` in the menu. * Click on `Create Client` button. * Select the `Client type`. * Enter the `Client ID`. * Enter the Name and Description `(Optional)`. * Click on `Next` button. ![add-client](https://docs.open-metadata.org/images/v1.11/deployment/security/keycloak/keycloak-step-3.png) ### Step 2: Edit Configs of the client * Select `Standard flow` and `Implicit flow` as an `Authentication flow`. * Click `Next`. ![compatibility configs](https://docs.open-metadata.org/images/v1.11/deployment/security/keycloak/implicit-keycloak-step-4.png) ### Step 3: Add Login Settings * fill the required options ![edit-settings-url.png](https://docs.open-metadata.org/images/v1.11/deployment/security/keycloak/keycloak-step-5.png) * Click on `Save` button. Note: Scopes `openid`, `email` & `profile` are required to fetch the user details so you will have to add these scopes in your client. After the applying these steps, the users in your realm are able to login in the openmetadata, as a suggestion create a user called "admin-user". Now you can update the configuration of your deployment: [Docker Security\ \ Configure Keycloak SSO for your Docker Deployment.](https://docs.open-metadata.org/latest/deployment/security/keycloak/docker) [Bare Metal Security\ \ Configure Keycloak SSO for your Bare Metal Deployment.](https://docs.open-metadata.org/latest/deployment/security/keycloak/bare-metal) [Kubernetes Security\ \ Configure Keycloak SSO for your Kubernetes Deployment.](https://docs.open-metadata.org/latest/deployment/security/keycloak/kubernetes) A dockerized demo for showing how this SSO works with OpenMetadata can be found [here](https://github.com/open-metadata/openmetadata-demo/tree/main/keycloak-sso) . [KeyCloak\ \ Go to KeyCloak Configuration](https://docs.open-metadata.org/latest/deployment/security/keycloak) --- # Enable JWT Tokens | OpenMetadata Security Features We use cookies to improve site navigation, analyze site usage, and enhance your user experience. Click "Accept" to enable cookies or "Reject" to reject cookies. AcceptReject ![](https://static.scarf.sh/a.png?x-pxid=8472f2b3-d307-4935-a5ed-3f9d3982c15d) deployment No menu items for this category [Home](https://docs.open-metadata.org/latest) /[Deployment](https://docs.open-metadata.org/latest/deployment) /[Security](https://docs.open-metadata.org/latest/deployment/security) /[Enable Jwt Tokens](https://docs.open-metadata.org/latest/deployment/security/enable-jwt-tokens) OpenMetadata Documentation Enable JWT Tokens ================= When we [enable SSO security](https://docs.open-metadata.org/latest/deployment/security) on OpenMetadata, it will restrict access to all the APIs. Users who want to access the UI will be redirected to configured SSO to log in, and SSO will provide the token to continue to make OpenMetadata REST API calls. However, metadata ingestion or any other services which use OpenMetadata APIs to create entities or update them requires a token as well to authenticate. Typically, SSO offers service accounts for this very reason. OpenMetadata supports service accounts that the SSO provider supports. Please read the [docs](https://docs.open-metadata.org/latest/deployment/security) to enable them. In some cases, either creating a service account is not feasible, or the SSO provider itself doesn't support the service account. To address this gap, we shipped JWT token generation and authentication within OpenMetadata. Security requirements for your **production** environment: * **DELETE** the admin default account shipped by OM in case you have [Basic Authentication](https://docs.open-metadata.org/latest/deployment/security/basic-auth) enabled. * **UPDATE** the Private / Public keys used for the [JWT Tokens](https://docs.open-metadata.org/latest/deployment/security/enable-jwt-tokens) . The keys we provide by default are aimed only for quickstart and testing purposes. They should NEVER be used in a production installation. Create Private / Public key --------------------------- ### For local/testing deployment You can work with the existing configuration or generate private/public keys. By default, the `jwtTokenConfiguration` is shipped with OM. ### For production deployment It is a **MUST** to update the JWT configuration. To create private/public key use the following commands can be used: Copy the `private_key.der` and `public_key.der` in OpenMetadata server `conf` directory. Make sure the permissions can only be readable by the user who is starting OpenMetadata server. Configure OpenMetadata Server ----------------------------- To enable JWT token generation. Please add the following to the OpenMetadata server If you are using helm charts or docker use the env variables to override the configs above. Please use absolute path for public and private key files that we generated in previous steps. Update the `JWT_ISSUER` to be the domain where you are running the OpenMetadata server. Generate `UUID64` id to configure `JWT_KEY_ID`. This should be generated once and keep it static even when you are updating the versions. Any change in this id will result in all the tokens issued so far to be invalid. ### Add public key URIS add `{your domain}/api/v1/system/config/jwks` to `publicKeyUrls`. You should append to the existing configuration such that your SSO and JWTToken auth verification will work. Once you configure the above settings, restart OpenMetadata server .

Note on JWKS url Network Reachbility

Make sure the above JWKS URI - `{your domain}/api/v1/system/config/jwks` is reachable from OpenMetadata Server Instance (VM or Docker Container or Kubernetes Pod). You can run the below command from the OpenMetadata Server to test it's reachility - Generate Token -------------- Once the above configuration is updated, the server is restarted. Admin can go to Settings -> Bots page. ![Settings Page](https://docs.open-metadata.org/images/v1.11/deployment/security/enable-jwt/settings-bot.png) Settings Page ![Bot settings page](https://docs.open-metadata.org/images/v1.11/deployment/security/enable-jwt/bot.png) Bot settings page Click on the `ingestion-bot`. The current token can be revoked, or you can create a new one. ![Bot credentials edition](https://docs.open-metadata.org/images/v1.11/deployment/security/enable-jwt/bot-jwt-token.png) Edit JWT Token for ingestion-bot Configure Ingestion ------------------- The generated token from the above page should pass onto the ingestion framework so that the ingestion can make calls securely to OpenMetadata. Make sure this token is not shared and stored securely. ### Running Ingestion from CLI If you are running the ingestion from CLI. Add the below configuration to the workflow configuration you pass: In the above section, under the `workflowConfig`, configure `authProvider` to be "openmetadata" and under `securityConfig` section, add `jwtToken` and its value from the ingestion bot page. Configure JWT Key Pairs for Docker ---------------------------------- Following the above documentation, you will have private key and public key pair available as mentioned [here](https://docs.open-metadata.org/latest/deployment/security/enable-jwt-tokens#create-private-public-key) . Next, will proceed with the below section which will configure JWT token with docker environment. ### Create docker compose host volume mappings Create a host directory which will be mapped as docker volumes to docker compose. This step will require you to update existing docker compose files that comes up with [OpenMetadata Releases](https://github.com/open-metadata/OpenMetadata/releases) . It is presumed with the above code snippet that you have `docker-volume` directory available on host where the docker-compose file is. ### Update the docker compose environment variables with jwtkeys Update the docker environment variables either directly in the docker-compose files or in a separate docker env files. Below is a code snippet for how the docker env file will look like. ### Run the docker compose command to start the services Run the docker compose CLI command to start the docker services with the configured jwt keys. Configure JWT Key Pairs for Kubernetes -------------------------------------- Following the above documentation, you will have private key and public key pair available as mentioned [here](https://docs.open-metadata.org/latest/deployment/security/enable-jwt-tokens#create-private-public-key) . Next, will proceed with the below section which will configure JWT token with kubernetes environment. ### Create Kubernetes Secrets for the Key Pairs Create Kubernetes Secrets from file using the kubernetes imperative commands below. ### Update Helm Values to mount Kubernetes secrets and configure JWT Token Configuration Update your helm values to mount Kubernetes Secrets as Volumes and update the Jwt Token Configuration to point the Key File Paths to mounted path (absolute file path). It is recommended to consider new directory paths for mounting the secrets as volumes to OpenMetadata Server Pod. With OpenMetadata Helm Charts, you will be able to add volumes and volumeMounts with `extraVolumes` and `extraVolumeMounts` helm values. ### Install / Upgrade Helm Chart Release Run the below command to make sure the update helm values are available to OpenMetadata. --- # Azure SSO Setup Guide for Public Apps We use cookies to improve site navigation, analyze site usage, and enhance your user experience. Click "Accept" to enable cookies or "Reject" to reject cookies. AcceptReject ![](https://static.scarf.sh/a.png?x-pxid=8472f2b3-d307-4935-a5ed-3f9d3982c15d) deployment No menu items for this category [Home](https://docs.open-metadata.org/latest) /[Deployment](https://docs.open-metadata.org/latest/deployment) /[Security](https://docs.open-metadata.org/latest/deployment/security) /[Azure](https://docs.open-metadata.org/latest/deployment/security/azure) /[Public Client](https://docs.open-metadata.org/latest/deployment/security/azure/public-client) OpenMetadata Documentation Azure AD SSO Authentication – Public Client Configuration ========================================================= * [Troubleshooting](https://docs.open-metadata.org/latest/deployment/security/azure/public-client#troubleshooting) Overview -------- **Azure Active Directory (Azure AD) Single Sign-On (SSO)** enables users to authenticate using their **Microsoft 365 / Entra ID** accounts via **OAuth 2.0** and **OpenID Connect (OIDC)** protocols. This guide covers the **Public Client** setup, intended for applications that cannot securely store client secrets. Public Configuration Fields --------------------------- ![Azure AD SSO Configuration - Public Client](https://docs.open-metadata.org/images/v1.11/deployment/security/azure/azure1.png) ### 1\. Client Type * **Definition:** Defines whether the application is public (no client secret) or confidential (requires client secret). * **Options:** `Public` | `Confidential` * **Example:** `Public` * **Why it matters:** Determines the OAuth flow and security level. * **Note:** * Use `Public` for clients that cannot store secrets. * Azure typically recommends `Confidential` for secure apps. ### 2\. Client ID * **Definition:** The Application (Client) ID assigned to your app in Azure AD. * **Example:** `12345678-1234-1234-1234-123456789012` * **Why it matters:** Azure AD uses this to identify your application. * **Note:** Found in Azure Portal β†’ **Azure Active Directory** β†’ **App registrations** β†’ Your App β†’ **Overview** β†’ _Application (client) ID_ ### 3\. Callback URL * **Definition:** The redirect URI where Azure AD sends authentication responses. * **Example:** `https://yourapp.company.com/callback` * **Why it matters:** Must exactly match what's registered in Azure AD; mismatches will cause authentication failures. * **Note:** * Configure in Azure AD β†’ **App registrations** β†’ **Authentication** β†’ _Redirect URIs_ * Always use **HTTPS** in production environments. ### 4\. Authority * **Definition:** Azure AD endpoint that issues tokens for your tenant. * **Example:** `https://login.microsoftonline.com/your-tenant-id` * **Why it matters:** Tells OpenMetadata which Azure tenant to use for authentication. * **Note:** * Replace `your-tenant-id` with your actual tenant GUID. * For multi-tenant apps, use `common` instead of the tenant ID. ### 5\. Public Key URLs * **Definition:** URL(s) where Azure AD publishes public keys used for verifying JWT tokens. * **Example:** `["https://login.microsoftonline.com/common/discovery/v2.0/keys"]` * **Why it matters:** Required to validate token signatures. * **Note:** Typically auto-discovered from Azure’s OIDC metadata; manual configuration rarely needed. ### 6\. Token Validation Algorithm * **Definition:** The algorithm used to validate JWT token signatures. * **Options:** `RS256`, `RS384`, `RS512` * **Default:** `RS256` * **Example:** `RS256` * **Why it matters:** Must match Azure AD’s signing algorithm. * **Note:** Azure AD typically uses `RS256`. ### 7\. JWT Principal Claims * **Definition:** Claims in the JWT token used to identify users. * **Example:** `["preferred_username", "email", "sub"]` * **Why it matters:** These claims are used to recognize and map users in OpenMetadata. * **Note:** Common claims include: `email`, `preferred_username`, `upn`, `sub` ### 8\. Admin Principals * **Definition:** A list of users (by email or UPN) granted admin access. * **Example:** `["admin@company.com", "superuser@company.com"]` * **Why it matters:** Grants administrative permissions within OpenMetadata. * **Note:** Entries must match the value from the selected JWT principal claim. ### 9\. Bot Principals * **Definition:** A list of service accounts or bot users for automated operations. * **Example:** `["metadata-bot@company.com"]` * **Why it matters:** Designates non-human principals for running background jobs or automation tasks. ### 10\. Principal Domain * **Definition:** Default domain used for user principal resolution. * **Example:** `company.com` * **Why it matters:** Used to complete email addresses or usernames if only the prefix is provided. * **Note:** Typically matches your organization’s primary domain. ### 11\. Enforce Principal Domain * **Definition:** Whether to restrict login to users from a specific domain. * **Default:** `false` * **Example:** `true` * **Why it matters:** Adds security by limiting access to a known domain space. ### 12\. Enable Secure Socket Connection * **Definition:** Enables TLS/SSL for all SSO communication. * **Default:** `false` * **Example:** `true` * **Why it matters:** Encrypts all communication with Azure AD for added security. * **Note:** Should be enabled for production environments. Summary ------- | Key Field | Example / Default | | --- | --- | | Client Type | `Public` | | Client ID | `12345678-1234-1234-1234-123456789012` | | Callback URL | `https://yourapp.company.com/callback` | | Authority | `https://login.microsoftonline.com/your-tenant-id` | | Public Key URLs | `https://login.microsoftonline.com/common/discovery/v2.0/keys` | | Token Validation | `RS256` | | JWT Claims | `["preferred_username", "email", "sub"]` | | Admin Principals | `["admin@company.com"]` | | Enforce Domain | `true` | | Use TLS (SSL) | `true` | ### Troubleshooting If users are automatically logged out and unable to log in again due to a bad authentication configuration, you can reset the security setup using the following command: After executing the command, **restart the server**. The authentication values from your YAML or Helm chart will then be reapplied on startup. The following tiles detail how to apply this configuration across Docker, Kubernetes, and Bare Metal deployments: [Docker Security\ \ Configure SSO for your Docker Deployment.](https://docs.open-metadata.org/latest/deployment/security/configuration/docker) [Bare Metal Security\ \ Configure SSO for your Bare Metal Deployment.](https://docs.open-metadata.org/latest/deployment/security/configuration/bare-metal) [Kubernetes Security\ \ Configure SSO for your Kubernetes Deployment.](https://docs.open-metadata.org/latest/deployment/security/configuration/kubernetes) --- # Custom OIDC SSO Configuration | OpenMetadata We use cookies to improve site navigation, analyze site usage, and enhance your user experience. Click "Accept" to enable cookies or "Reject" to reject cookies. AcceptReject ![](https://static.scarf.sh/a.png?x-pxid=8472f2b3-d307-4935-a5ed-3f9d3982c15d) deployment No menu items for this category [Home](https://docs.open-metadata.org/latest) /[Deployment](https://docs.open-metadata.org/latest/deployment) /[Security](https://docs.open-metadata.org/latest/deployment/security) /[Custom Oidc](https://docs.open-metadata.org/latest/deployment/security/custom-oidc) /[Custom Oidc Configuration](https://docs.open-metadata.org/latest/deployment/security/custom-oidc/custom-oidc-configuration) OpenMetadata Documentation Custom OIDC SSO Configuration ============================= * [Troubleshooting](https://docs.open-metadata.org/latest/deployment/security/custom-oidc/custom-oidc-configuration#troubleshooting) Custom OIDC authentication enables integration with any OpenID Connect (OIDC) compliant identity provider such as Auth0, Google, Azure AD, PingIdentity, or internal enterprise solutions. This guide walks you through configuring OpenMetadata with a Custom OIDC provider using **Confidential client** settings (Client ID + Client Secret). ![Custom OIDC Configuration](https://docs.open-metadata.org/images/v1.11/deployment/security/custom-oidc/custom1.png) Configuration Fields -------------------- ### Custom Provider Name * **Definition:** A display name for your OIDC provider shown to users during login. * **Example:** `"Company SSO"`, `"Internal Auth"` * **Why it matters:** Used in UI and logs for easy identification. * **Optional:** Defaults to `"Custom OIDC"` if not specified. ### Enable Self Signup * **Definition:** Allows new users to create accounts on first login via OIDC. * **Default:** false * **Why it matters:** Controls auto-provisioning of user accounts. * **Security Consideration:** Enable only if all OIDC users are trusted. ### Authority / Issuer URL * **Definition:** The base URL of your OIDC provider’s authentication server. * **Example:** `https://auth.yourcompany.com` * **Why it matters:** Used for discovering OIDC metadata and validating tokens. * **Required:** Yes * **Note:** Must return a valid discovery document from `/.well-known/openid-configuration`. ### Public Key / JWK URL * **Definition:** URL to the JSON Web Key Set (JWKS) used to validate tokens. * **Example:** `https://auth.yourcompany.com/.well-known/jwks.json` * **Why it matters:** Validates the signature of JWT tokens. * **Note:** Usually auto-resolved from the discovery document. ### Token Validation Algorithm * **Definition:** Algorithm used to validate JWTs. * **Options:** RS256 | RS384 | RS512 | HS256 | HS384 | HS512 * **Default:** RS256 * **Why it matters:** Must match your OIDC provider’s signing algorithm. * **Note:** RS256 is recommended. ### Client Type * **Value:** `custom-oidc` * **Definition:** Identifies this integration type. ### Client ID * **Definition:** OAuth2 client ID issued by your OIDC provider. * **Example:** `my-custom-oidc-client-12345` * **Required:** Yes ### Client Secret * **Definition:** OAuth2 client secret from your OIDC provider. * **Example:** `abc123-secret-xyz789` * **Required:** Yes * **Note:** Keep this value secure. Never expose in frontend code. ### Scopes * **Definition:** OAuth2 scopes requested from your provider. * **Default:** `openid profile email` * **Example:** `openid profile email groups` * **Why it matters:** Controls what user data OpenMetadata can access. * **Common scopes:** * `openid` – Required for OIDC * `profile` – Access basic user profile * `email` – Access user email * `groups` – Access group membership (if supported) ### OIDC Discovery URI * **Definition:** URL to your OIDC provider's discovery document. * **Example:** `https://auth.yourcompany.com/.well-known/openid-configuration` * **Why it matters:** Used to auto-configure token and auth endpoints. ### Use Nonce * **Definition:** Prevents replay attacks in OIDC authentication. * **Default:** false * **Example:** true ### Preferred JWS Algorithm * **Definition:** Signature algorithm for JWT validation. * **Default:** RS256 ### Response Type * **Definition:** OAuth response type. * **Default:** `code` * **Options:** `id_token`, `code` * **Why it matters:** Authorization code flow is recommended for backend services. ### Disable PKCE * **Definition:** Whether to disable Proof Key for Code Exchange. * **Default:** false ### Max Clock Skew * **Definition:** Allowed time difference (in seconds) between client and server. * **Example:** `0` ### Client Authentication Method * **Definition:** How your app authenticates to the OIDC provider. * **Options:** `client_secret_basic` | `client_secret_post` | `client_secret_jwt` | `private_key_jwt` * **Default:** `client_secret_basic` ### Token Validity * **Definition:** Duration (in seconds) for which the token is valid. * **Example:** `3600` * **Note:** Use `0` to inherit provider’s default. ### Tenant * **Definition:** Optional identifier for your OIDC tenant. * **Example:** `company-idp` ### Server URL * **Definition:** Base server URL of the OIDC provider. * **Example:** `https://auth.yourcompany.com` ### Callback URL * **Definition:** Redirect URI where users land after authentication. * **Example:** `https://yourapp.company.com/callback` * **Required:** Yes * **Note:** This must be registered in your OIDC provider's allowed redirect URIs. ### Max Age * **Definition:** Max age (in seconds) since user last authenticated. * **Example:** `3600` ### Prompt * **Definition:** Controls login experience. * **Options:** `none`, `login`, `consent`, `select_account` * **Example:** `login` ### Session Expiry * **Definition:** How long the user session lasts in seconds. * **Default:** 604800 (7 days) ### JWT Principal Claims * **Definition:** Claims in the JWT used to identify the user. * **Default:** `["email", "preferred_username", "sub"]` * **Example:** `["email", "username", "sub"]` ### JWT Principal Claims Mapping * **Definition:** Maps JWT claims to OpenMetadata user profile fields. * **Example:** `["email:email", "name:name", "firstName:given_name"]` * **Format:** `"openmetadata_field:jwt_claim"` ### Admin Principals * **Definition:** Users granted admin rights. * **Example:** `["admin@company.com", "security@company.com"]` ### Bot Principals * **Definition:** Service account(s) used for automation. * **Example:** `["ingestion-bot@example.com"]` ### Principal Domain * **Definition:** Default domain appended to usernames. * **Example:** `company.com` ### Enforce Principal Domain * **Definition:** Restrict user logins to a specific domain. * **Default:** false * **Example:** true ### Enable Secure Socket Connection * **Definition:** Use SSL/TLS for secure communications. * **Default:** false * **Example:** true Summary Table ------------- | **Field** | **Example / Default** | | --- | --- | | Type | custom-oidc | | Client Type | Confidential | | Client ID | my-custom-oidc-client-12345 | | Client Secret | abc123-secret-xyz789 | | Authority / Issuer URL | https://auth.yourcompany.com | | Discovery URI | https://auth.yourcompany.com/.well-known/openid-configuration | | Callback URL | https://yourapp.company.com/callback | | Token Validation Algorithm | RS256 | | Response Type | code | | Scopes | openid profile email groups | | JWT Principal Claims | \["email", "preferred\_username", "sub"\] | | JWT Mapping | \["email:email", "name:name", "firstName:given\_name"\] | | Admin Principals | \["admin@company.com"\] | | Bot Principals | \["ingestion-bot@example.com"\] | | Principal Domain | company.com | | Enforce Principal Domain | false | | SSL/TLS | true | ### Troubleshooting If users are automatically logged out and unable to log in again due to a bad authentication configuration, you can reset the security setup using the following command: After executing the command, **restart the server**. The authentication values from your YAML or Helm chart will then be reapplied on startup. The following tiles detail how to apply this configuration across Docker, Kubernetes, and Bare Metal deployments: [Docker Security\ \ Configure SSO for your Docker Deployment.](https://docs.open-metadata.org/latest/deployment/security/configuration/docker) [Bare Metal Security\ \ Configure SSO for your Bare Metal Deployment.](https://docs.open-metadata.org/latest/deployment/security/configuration/bare-metal) [Kubernetes Security\ \ Configure SSO for your Kubernetes Deployment.](https://docs.open-metadata.org/latest/deployment/security/configuration/kubernetes) --- # Enable Secrets Manager | Official Documentation We use cookies to improve site navigation, analyze site usage, and enhance your user experience. Click "Accept" to enable cookies or "Reject" to reject cookies. AcceptReject deployment No menu items for this category [Home](https://docs.open-metadata.org/latest) /[Deployment](https://docs.open-metadata.org/latest/deployment) /[Secrets Manager](https://docs.open-metadata.org/latest/deployment/secrets-manager) OpenMetadata Documentation Enable Secrets Manager ====================== Secret Manager integrations allow you to use your existing third-party **Key Management Store** (KMS) with OpenMetadata. Your credentials and sensitive information are stored in a tool that you control, and the KMS will mediate between any OpenMetadata internal requirement and sensitive information. Without a secret manager configured in OpenMetadata, all your sensitive data, any password field of a service connection parameters, bot credentials configuration or dbt configuration of an ingestion pipeline, were stored in MySQL (or Postgres) encrypted. The following diagram shows how is the process between the OM server and Airflow workflows: ![om-secrets-manager-disabled](https://docs.open-metadata.org/images/v1.11/deployment/secrets-manager/om-secrets-manager-disabled.png) As you can see, the `Workflow` consumed by Airflow contains the service information as an `EntityReference`. We use that reference to read the Service information, including its connection details. This information goes from `Database > OM > Airflow`. When the Secrets Manager is enabled, sensitive information stop being stored in any system from OpenMetadata. Instead, the KMS will act as a mediator, as we can observe in the diagram below: ![om-secrets-manager-enabled](https://docs.open-metadata.org/images/v1.11/deployment/secrets-manager/om-secrets-manager-enabled.png) In 0.13 and up, OpenMetadata will communicate through an interface to read/write sensitive information -- removing the need to store sensitive data in OM systems. This new interface works whether users keep using the underlying database of OpenMetadata to store credentials (as it was set up thus far) or any external system such as AWS Secrets Manager or AWS SSM Parameter Store. In future releases, we will add support for additional Key Management Stores, such as Azure Key Vault or Kubernetes Secrets. If you’d like to contribute by creating the interface, check the implementation guide, or if you want to see a new one on the supported list, please reach out to us on [Slack](https://slack.open-metadata.org/) . If you are interested in enabling the secrets' manager feature, this is our list of supported Secrets Manager implementations: * [AWS Secrets Manager](https://docs.open-metadata.org/latest/deployment/secrets-manager/supported-implementations/aws-secrets-manager) * [AWS Systems Manager Parameter Store](https://docs.open-metadata.org/latest/deployment/secrets-manager/supported-implementations/aws-ssm-parameter-store) Things to take into account when enabling the Secrets Manager feature: 1. The migration of all the sensitive data will be done automatically after restarting the OpenMetadata server, which can not be undone for the time being. 2. Only users with permissions can edit and retrieve the service connections. The connection parameters will be hidden for all other users. How it works ------------ There are two types of secrets manager implementations. ### Managed secrets manager All the sensitive data will be held automatically in the configured secrets manager, i.e., any password field stored in the connection parameters of a service, in a bot credentials configuration, or a dbt configuration of an ingestion pipeline. For example, suppose we create a MySQL service with the name `mysql-test`. In that case, the connection password will be stored in the secrets manager using the secret id `/openmetadata/database/mysql-test/password`. When we retrieve the connection parameters from the service, the password field will have the value `secret:/openmetadata/database/mysql-test/password`. We can also use secrets already stored in our secrets vault using the same convention `secret:{secret_id}`. All the sensitive data (the secrets ids in this case) values will be encrypted using the Fernet algorithm as extra security protection. ### Non-managed secrets manager On the other hand, the non-managed configuration allows flexibility on how we want to use our secrets vault. Instead of automatically storing all the sensitive data, we can use the secrets ids from our secrets vault following the convention `secret:{secret_id}` when filling in password fields of the connection parameters of a service, in a bot configuration, or a dbt configuration of an ingestion pipeline. The rest of the values which don't follow the convention for using a secret will be encrypted using the Fernet algorithm as extra security protection. --- # Advanced Guide for Roles and Policies We use cookies to improve site navigation, analyze site usage, and enhance your user experience. Click "Accept" to enable cookies or "Reject" to reject cookies. AcceptReject ![](https://static.scarf.sh/a.png?x-pxid=6814879a-96c0-40bd-be69-ddfeb14694aa) how-to-guides No menu items for this category [Home](https://docs.open-metadata.org/latest) /[How To Guides](https://docs.open-metadata.org/latest/how-to-guides) /[Admin Guide](https://docs.open-metadata.org/latest/how-to-guides/admin-guide) /[Roles Policies](https://docs.open-metadata.org/latest/how-to-guides/admin-guide/roles-policies) OpenMetadata Documentation Advanced Guide for Roles and Policies ===================================== Users and Teams --------------- OpenMetadata introduces a versatile hierarchical team structure that aligns with your organization's setup. Administrators can mirror their organizational hierarchy by creating various team types. **Organization** serves as the foundation of the team hierarchy representing the entire company. Under Organization, you can add Business Units, Divisions, Departments, Groups, and Users. For instance, if your company is Facebook, then the Organization represents entire Facebook itself, which further houses diverse teams like Engineering, Sales, Finance, and Marketing. ![Teams Hierarchy](https://docs.open-metadata.org/images/v1.11/how-to-guides/roles-policies/all-teams.png) Teams Hierarchy **BusinessUnit** is positioned one level below the Organization and can contain other Business Units, Divisions, Departments, and Groups. To illustrate, the Engineering Business Unit could be one of the top-tier Business Units in the Organization. It contains other teams like Groups and additional Business Units. ![Business Unit](https://docs.open-metadata.org/images/v1.11/how-to-guides/roles-policies/b-u.png) Business Unit **Division** is positioned below Business Unit and can include Divisions, Departments, and Groups. For example, a Division named 'Product Development' under the Engineering Business Unit. It can have teams like 'Software Division,' 'Hardware Division,' and 'QA Division.' **Department** is positioned below Division and can include other Departments and Groups. For example, a 'Data Engineering Department could include specialized teams like 'Infrastructure,' 'Data Science,' and 'Platform.' **Group** represents the final tier in this hierarchy. It contains a group of users that reflect finite teams within your organization. _**Notably, only Groups have the privilege of owning Data Assets within the OpenMetadata platform.**_ This structured hierarchy enhances your control over team management and resource ownership. By creating a dynamic model mirroring your organization's functions, OpenMetadata empowers you to effortlessly manage permissions, access controls, and data ownership at different levels of granularity. Access Control Design: Roles and Policies ----------------------------------------- ![Policy Evaluation](https://docs.open-metadata.org/images/v1.11/how-to-guides/roles-policies/evaluation.png) Policy Evaluation OpenMetadata incorporates a robust Access Control framework that merges Role-Based Access Control (RBAC) with Attribute-Based Access Control (ABAC) in a powerful hybrid model. This security design is reinforced by **Authentication with SSO Integration:** OpenMetadata seamlessly integrates with various Single Sign-On (SSO) providers, including Azure AD, Google, Okta, Auth0, OneLogin, and more. This ensures a unified and secure authentication experience for users. **Team Hierarchy:** OpenMetadata offers a structured team hierarchy that mirrors your organization's structure, enhancing manageability and granularity in access control. **Roles and Policies:** Policies and Roles are pivotal in determining who can access what resources and perform what actions. These policies are based on a combination of user attributes, roles, and resource attributes. **User and Bots Authentication:** OpenMetadata accommodates human users and automated applications (bots). For human users, logging into the OpenMetadata UI mandates SSO authentication. Upon successful authentication, a JWT token is issued. Bots, on the other hand, are equipped with a JWT token generated based on SSL certificates. This token serves as their identity and authorization mechanism when interacting with the OpenMetadata server APIs. Authentication Flow ------------------- ![Authentication Flow](https://docs.open-metadata.org/images/v1.11/how-to-guides/roles-policies/auth.png) Authentication Flow **User Authentication:** When users access the OpenMetadata UI, they authenticate with their SSO provider. Upon successful authentication, a JWT token is generated. This token validates the user's session and permits them to authenticate requests to the OpenMetadata server. **Bot Authentication:** Automated applications like the ingestion connector are equipped with a pre-generated JWT Token. OpenMetadata, with its configured SSL Certificates, authenticates the JWT token, establishing the bot's identity. This token authorizes the bot to interact with OpenMetadata server APIs. Authorization Framework ----------------------- OpenMetadata's authorization is a result of evaluating three crucial factors: ![Authorization Framework](https://docs.open-metadata.org/images/v1.11/how-to-guides/roles-policies/access.png) Authorization Framework **Who is the User (Authentication):** This aspect is determined by the authentication process – whether it a user or a bot – ensuring that only authorized entities access the system. **What Resource (Resource Attributes):** Based on the API calls being made, OpenMetadata identifies the target resource and its associated attributes. Below is a list of resources that correspond to Entities such as Table, Topic, Pipeline, etc. ![Resources Correspond to Entities](https://docs.open-metadata.org/images/v1.11/how-to-guides/roles-policies/rules1.png) Resources Correspond to Entities **What Operation (API Call):** Each API call is linked to a specific operation, such as editing descriptions, deleting tags, changing ownership, etc. There are common operations such as Create, Delete, and ViewAll that apply to all the resources. Each resource can also have its specific operation, such as ViewTests, ViewQueries for Table. Difference Between ViewBasic and ViewAll in OpenMetadata -------------------------------------------------------- The operations **ViewBasic** and **ViewAll** in OpenMetadata differ in the level of detail they provide access to. Below is a detailed explanation of each operation: ### ViewBasic * Provides access to the **basic details** of an asset. * Includes information such as: * Description * Tags * Owner * Fundamental metadata * **Excludes** more detailed information, including: * Profile data * Sample data * Data profile * Tests * Queries ### Key Points: * Suitable for viewing foundational asset metadata. * Limited access for users who do not require in-depth technical details. ### ViewAll * Provides access to **all details** of an asset. * Includes everything available in **ViewBasic**, along with: * Profile data * Sample data * Data profile * Tests * Queries ### Key Points: * Designed for users who need a complete view of the asset. * Offers comprehensive insights and detailed metadata. Summary Table ------------- | Feature | **ViewBasic** | **ViewAll** | | --- | --- | --- | | Basic Details | βœ… Included | βœ… Included | | Profile Data | ❌ Not Included | βœ… Included | | Sample Data | ❌ Not Included | βœ… Included | | Data Profile | ❌ Not Included | βœ… Included | | Tests & Queries | ❌ Not Included | βœ… Included | ### Overview: * **ViewBasic**: Focused on essential metadata. * **ViewAll**: Provides a complete view, including advanced details. Choose the appropriate operation based on the level of access required. ![Each Resource has its Own Set of Granular Operations](https://docs.open-metadata.org/images/v1.11/how-to-guides/roles-policies/rules2.png) Each Resource has its Own Set of Granular Operations By synthesizing these components, OpenMetadata dynamically ascertains whether a user or bot can perform a particular action on a specific resource. This **fusion of RBAC and ABAC** in the hybrid model contributes to a robust and flexible access control mechanism, bolstering the security and control of your OpenMetadata environment. [Building Blocks of Authorization: Rules, Policies, and Roles\ \ Learn all the details of Rules, Policies, and Roles](https://docs.open-metadata.org/latest/how-to-guides/admin-guide/roles-policies/authorization) [Use Cases: Creating Roles & Policies in OpenMetadata\ \ Tailor you policies to meet your organizational and team needs.](https://docs.open-metadata.org/latest/how-to-guides/admin-guide/roles-policies/use-cases) --- # Metadata Services | OpenMetadata Integration Overview We use cookies to improve site navigation, analyze site usage, and enhance your user experience. Click "Accept" to enable cookies or "Reject" to reject cookies. AcceptReject connectors No menu items for this category [Home](https://docs.open-metadata.org/latest) /[Connectors](https://docs.open-metadata.org/latest/connectors) /[Metadata](https://docs.open-metadata.org/latest/connectors/metadata) OpenMetadata Documentation Metadata Services ================= This is the supported list of connectors for Metadata Services: [![AlationSink](https://docs.open-metadata.org/_next/image?url=%2Fimages%2Fconnectors%2Falation.webp&w=64&q=75)\ \ AlationSink\ \ PROD\ \ Available In](https://docs.open-metadata.org/latest/connectors/metadata/alationsink) [![Amundsen](https://docs.open-metadata.org/_next/image?url=%2Fimages%2Fconnectors%2Famundsen.webp&w=64&q=75)\ \ Amundsen\ \ PROD\ \ Available In](https://docs.open-metadata.org/latest/connectors/metadata/amundsen) [![Atlas](https://docs.open-metadata.org/_next/image?url=%2Fimages%2Fconnectors%2Fatlas.webp&w=64&q=75)\ \ Atlas\ \ PROD\ \ Available In](https://docs.open-metadata.org/latest/connectors/metadata/atlas) If you have a request for a new connector, don't hesitate to reach out in [Slack](https://slack.open-metadata.org/) or open a [feature request](https://github.com/open-metadata/OpenMetadata/issues/new/choose) in our GitHub repo. --- # OIDC Based Authentication We use cookies to improve site navigation, analyze site usage, and enhance your user experience. Click "Accept" to enable cookies or "Reject" to reject cookies. AcceptReject ![](https://static.scarf.sh/a.png?x-pxid=8472f2b3-d307-4935-a5ed-3f9d3982c15d) deployment No menu items for this category [Home](https://docs.open-metadata.org/latest) /[Deployment](https://docs.open-metadata.org/latest/deployment) /[Security](https://docs.open-metadata.org/latest/deployment/security) /[Configuration Parameters](https://docs.open-metadata.org/latest/deployment/security/configuration-parameters) OpenMetadata Documentation Configuration Reference Parameters ================================== Public Key Url (publicKeyUrls): ------------------------------- This needs to be updated as per different SSO providers. The default value is `http://localhost:8585/api/v1/system/config/jwks`. This is the URL where the public keys are stored. The public keys are used to verify the signature of the JWT token. **Google**: https://www.googleapis.com/oauth2/v3/certs **Okta**: https://dev-19259000.okta.com/oauth2/aus5836ihy7o8ivuJ5d7/v1/keys **Auth0**: https://dev-3e0nwcqx.us.auth0.com/.well-known/jwks.json **Azure**: https://login.microsoftonline.com/{tenant}/discovery/v2.0/keys Also if you have enabled [JWT Tokens](https://docs.open-metadata.org/latest/deployment/security/enable-jwt-tokens) then http://localhost:8585/api/v1/system/config/jwks also needs to be there in the list with proper server url. Client ID (id): --------------- The client ID provided by your OIDC provider. This is typically obtained when you register your application with the OIDC provider. Type (type): ------------ Specify the type of OIDC provider you are using (e.g., google, azure). This value is same as `provider` in `authenticationConfiguration`. Client Secret (secret): ----------------------- Replace with the client secret provided by your OIDC provider. Scope (scope): -------------- Define the scopes that your application requests during authentication. Update ${OIDC\_SCOPE:-"openid email profile"} with the desired scopes. It does not need to be changed in most cases. The default scopes are `openid email profile`. The openid scope is required for OIDC authentication. The email and profile scopes are used to retrieve the user's email address and profile information. Although, some provider only give Refresh Token if `offline_access` scope is provided. So, if you want to use Refresh Token, you need to add `offline_access` scope, like below: `offline_access openid email profile`. Discovery URI (discoveryUri): ----------------------------- Provide the URL of the OIDC provider's discovery document. This document contains metadata about the provider's configuration. It is mostly in the format as below: https://accounts.google.com/.well-known/openid-configuration **Google**: https://accounts.google.com/.well-known/openid-configuration **Okta**: https://dev-19259000.okta.com/oauth2/aus5836ihy7o8ivuJ5d7/.well-known/openid-configuration **Auth0**: https://dev-3e0nwcqx.us.auth0.com/.well-known/openid-configuration **Azure**: https://login.microsoftonline.com/{tenant}/v2.0/.well-known/openid-configuration Normally it's some initial SSO provider URL followed by `.well-known/openid-configuration` Use Nonce (useNonce): --------------------- Set to true by Default, if you want to use nonce for replay attack protection during authentication. This does not need to be changed. Preferred JWS Algorithm (preferredJwsAlgorithm): ------------------------------------------------ Specify the preferred JSON Web Signature (JWS) algorithm. Default is RS256 and need not be changed . Response Type (responseType): ----------------------------- Define the response type for the authentication request. Default is code and need not be changed. Disable PKCE (disablePkce): --------------------------- Set ${OIDC\_DISABLE\_PKCE:-true} to true if you want to disable Proof Key for Code Exchange (PKCE). If you want to send CodeVerifier and CodeChallenge in the request, set it to false. Callback URL (callbackUrl): --------------------------- Provide the callback URL where the OIDC provider redirects after authentication. Update ${OIDC\_CALLBACK:-"http://localhost:8585/callback"} with your actual callback URL. The only initial part of the URL should be changed, the rest of the URL should be the same as the default one. The default URL is `http://localhost:8585/callback`. Also, this should match what you have configured in your OIDC provider. Server URL (serverUrl): ----------------------- Specify the URL of your OM Server. Default is http://localhost:8585. Client Authentication Method (clientAuthenticationMethod): ---------------------------------------------------------- Define the method used for client authentication. Default is client\_secret\_post. This does not need to be changed in most cases. The default value is `client_secret_post`. This method is used to send the client ID and client secret in the request body. Another possible value is `client_secret_basic`, which sends the client ID and client secret in the Authorization header. Depending on the OIDC provider, you may need to change this value if only one of them is supported. Tenant (tenant): ---------------- If applicable, specify the tenant ID for multi-tenant applications. Example in case of Azure. This is only applicable for multi-tenant applications. If you are using a single tenant application, you can leave this field empty. For Azure SSO Provider this may be needed. Max Clock Skew (maxClockSkew): ------------------------------ Define the maximum acceptable clock skew between your application server and the OIDC server. Custom Parameters (customParams): --------------------------------- If you have any additional custom parameters required for OIDC configuration, specify them here. Config (config): ---------------- The central configuration block for OpenMetadata. Provider (provider): -------------------- Specifies the authentication method to be used. The default is `ldap`, but you can change it to another supported provider. Example: `google`, `azure`. Entity Id (entityId): --------------------- The unique identifier for the SAML Identity Provider. Example: `"https://mocksaml.com/api/saml/sso"` SSO Login URL (ssoLoginUrl): ---------------------------- The URL to which users are redirected for Single Sign-On (SSO) authentication. Example: `"https://saml.example.com/entityid"` IPDX509 Certificate (idpX509Certificate): ----------------------------------------- The public certificate used by the IdP to sign SAML assertions. Example: `""` (empty string means no certificate provided, needs to be set with actual certificate) Authority URL (authorityUrl): ----------------------------- The URL used for SAML login, typically a custom endpoint for your SAML provider. Example: `"http://localhost:8585/api/v1/saml/login"` Name ID (nameId): ----------------- The format for the NameID element in the SAML response, usually representing the unique identifier of the user. Example: `"urn:oasis:names:tc:SAML:1.1:nameid-format:emailAddress"` ACS (acs): ---------- The Assertion Consumer Service (ACS) URL, where the IdP sends the SAML response after authentication. Example: `"http://localhost:8585/api/v1/saml/acs"` SPX509 Certificate (spX509Certificate): --------------------------------------- The public certificate used by the Service Provider to verify the IdP's SAML response. Example: `""` (empty string means no certificate provided, needs to be set with actual certificate) Strict Mode (strictMode): ------------------------- Whether to enforce strict compliance with the SAML standard, ensuring the response is fully validated. Default: `false` Token Validity (tokenValidity): ------------------------------- The validity period of the SAML token in seconds. Default: `"3600"` (1 hour) Send Encrypted Name ID (sendEncryptedNameId): --------------------------------------------- Whether to send the NameID in an encrypted format in the SAML response. Default: `false` Send Signed Auth Request (sendSignedAuthRequest): ------------------------------------------------- Whether to sign the authentication request sent to the IdP. Default: `false` Sign SP Metadata (signSpMetadata): ---------------------------------- Whether to sign the Service Provider's metadata when exchanging SAML metadata with the IdP. Default: `false` Want Messages Signed (wantMessagesSigned): ------------------------------------------ Whether the Service Provider expects SAML messages to be signed. Default: `false` Want Assertions Signed (wantAssertionsSigned): ---------------------------------------------- Whether the Service Provider expects SAML assertions to be signed. Default: `false` Want Assertion Encrypted (wantAssertionEncrypted): -------------------------------------------------- Whether to encrypt the SAML assertion before sending it to the Service Provider. Default: `false` Want Name ID Encrypted (wantNameIdEncrypted): --------------------------------------------- Whether to encrypt the NameID element in the SAML response. Default: `false` Key Store File Path (keyStoreFilePath): --------------------------------------- The file path to the keystore file containing certificates and private keys used for signing and encryption. Example: `""` (empty string means no keystore file provided) KeyStore Alias (keyStoreAlias): ------------------------------- The alias used to refer to the key inside the keystore file. Example: `""` (empty string means no alias provided) KeyStore Password (keyStorePassword): ------------------------------------- The password used to access the keystore file. Example: `""` (empty string means no password provided) Class Name (className): ----------------------- Specifies the class that handles the authorization logic. Default: `"org.openmetadata.service.security.DefaultAuthorizer"` Container Request Filter (containerRequestFilter): -------------------------------------------------- Specifies the request filter used to process authentication, especially for handling JWT tokens. Default: `"org.openmetadata.service.security.JwtFilter"` Initial Admins (initialAdmins): ------------------------------- A list of users who will be granted administrative privileges during the initial setup. Example: `["suresh"]` Principal Domain (principalDomain): ----------------------------------- The domain that is associated with user accounts. Default: `"open-metadata.org"` Authority (authority): ---------------------- The base URL of the OIDC authority. Example: Replace `{IssuerUrl}` with the URL of your custom OIDC provider. Client ID (clientId): --------------------- The client ID for the application registered with the custom OIDC provider. Replace `{client id}` with the actual client ID. Host (host): ------------ The hostname of the LDAP server. Defaults to `localhost`. Port (port): ------------ The port number to connect to the LDAP server. Defaults to `10636`. DN Admin Principal (dnAdminPrincipal): -------------------------------------- The distinguished name (DN) of the admin user used for lookup operations in LDAP. Defaults to `"cn=admin,dc=example,dc=com"`. DN Admin Password (dnAdminPassword): ------------------------------------ The password for the admin user. Defaults to `"secret"`. Userbase DN (userBaseDN): ------------------------- The base DN for user lookup in LDAP. Defaults to `"ou=people,dc=example,dc=com"`. Mail Attribute Name (mailAttributeName): ---------------------------------------- The attribute name in LDAP that stores user email addresses. Defaults to `email`. Maximum Pool Size (maxPoolSize) (Optional): ------------------------------------------- Defines the maximum number of connections in the LDAP connection pool. Defaults to `3`. SSL Enabled (sslEnabled): ------------------------- Indicates if SSL is enabled for connecting to the LDAP server. Defaults to `true`. Custom Trust Manager Configuration (customTrustManagerConfig): -------------------------------------------------------------- * ### TrustStore FilePath (trustStoreFilePath): Path to the custom trust store file. Default is empty. * ### TrustStore File Password (trustStoreFilePassword): Password for the trust store file. Default is empty. * ### TrustStore File Format (trustStoreFileFormat): Format of the trust store file. Default is empty. * ### Verify Host Name (verifyHostname): If hostname verification is enabled. Default is empty. * ### Examine Validity Dates (examineValidityDates): Whether to check validity dates for certificates. Default is empty. Host Name Configuration (hostNameConfig): ----------------------------------------- * ### Allow Wild Cards (allowWildCards): Allows wildcard certificates in hostnames. Default is empty. * ### Acceptable Host Names (acceptableHostNames): A list of acceptable hostnames. Default is an empty list. JVM Default Configurations (jvmDefaultConfig): ---------------------------------------------- * ### Verify Host Name (verifyHostname): Enables hostname verification using JVM defaults. Default is empty. Trust All Configurations (trustAllConfig): ------------------------------------------ * ### Examine Validity Dates (examineValidityDates): Checks the validity dates of certificates when using `TrustAll` mode. Defaults to `true`. Enforce Principal Domain (enforcePrincipalDomain): -------------------------------------------------- Whether to enforce user principal matching with the defined principal domain Enable Secure Socket Connection (enableSecureSocketConnection): --------------------------------------------------------------- If true, enables secure connections (SSL/TLS) Use Roles From Provider (useRolesFromProvider): ----------------------------------------------- Whether to derive roles from the authentication provider Initial Admins (initialAdmins): ------------------------------- List of initial admin users for the system JWT Principal Claims (jwtPrincipalClaims): ------------------------------------------ JWT claims used to identify the principal (user) JWT Principal Claims Mapping (jwtPrincipalClaimsMapping): --------------------------------------------------------- Mapping of JWT claims to application-specific claims Enable Self Signup (enableSelfSignup): -------------------------------------- Allows users to sign up themselves if not already registered Preferred JWT Algorithms (preferredJwsAlgorithm): ------------------------------------------------- Preferred algorithm for JWT signature validation Allowed Email Registration Domains (allowedEmailRegistrationDomains): --------------------------------------------------------------------- Specifies allowed domains for email registration --- # LDAP SSO Configuration | OpenMetadata We use cookies to improve site navigation, analyze site usage, and enhance your user experience. Click "Accept" to enable cookies or "Reject" to reject cookies. AcceptReject ![](https://static.scarf.sh/a.png?x-pxid=8472f2b3-d307-4935-a5ed-3f9d3982c15d) deployment No menu items for this category [Home](https://docs.open-metadata.org/latest) /[Deployment](https://docs.open-metadata.org/latest/deployment) /[Security](https://docs.open-metadata.org/latest/deployment/security) /[Ldap](https://docs.open-metadata.org/latest/deployment/security/ldap) /[Ldap Configuration](https://docs.open-metadata.org/latest/deployment/security/ldap/ldap-configuration) OpenMetadata Documentation LDAP SSO Configuration ====================== * [Troubleshooting](https://docs.open-metadata.org/latest/deployment/security/ldap/ldap-configuration#troubleshooting) LDAP (Lightweight Directory Access Protocol) authentication enables users to log in using their enterprise directory credentials from systems such as **Active Directory**, **OpenLDAP**, or other LDAP-compatible identity providers. This guide explains how to configure LDAP as an authentication source in OpenMetadata. ![LDAP SSO Configuration](https://docs.open-metadata.org/images/v1.11/deployment/security/ldap/ldap1.png) Configuration Fields -------------------- ### Provider Name * **Definition:** A human-readable name for this LDAP SSO configuration instance. * **Example:** `Company LDAP`, `Corporate Directory`, `Internal LDAP` * **Why it matters:** Used for display and logging purposes. * **Note:** This name does not impact authentication behavior. ### Client ID * **Definition:** Identifier for the LDAP authentication configuration. * **Example:** `ldap-client-123` * **Why it matters:** Helps track and manage LDAP configurations. * **Note:** Optional for LDAP; mainly used for tracking. ### Callback URL * **Definition:** Redirect URI where users land after LDAP authentication. * **Example:** `https://yourapp.company.com/callback` * **Why it matters:** Defines the return URL post-authentication. * **Note:** Typically your OpenMetadata application URL. ### Authority * **Definition:** Base URL for the LDAP authentication authority. * **Example:** `https://yourapp.company.com/auth/ldap` * **Why it matters:** Defines the endpoint OpenMetadata uses for LDAP requests. * **Note:** Required for routing LDAP authentication calls. ### Enable Self Signup * **Definition:** Allows users to automatically create accounts on their first LDAP login. * **Options:** Enabled | Disabled * **Example:** Enabled * **Why it matters:** Controls whether new LDAP users are auto-provisioned. * **Note:** Disable for stricter access control. ### Public Key URLs * **Definition:** URLs where public keys are published for token verification. * **Example:** `["https://yourapp.company.com/.well-known/jwks.json"]` * **Why it matters:** Used when LDAP integrates with token-based authentication (LDAP + JWT). * **Note:** Optional for pure LDAP configurations. ### Token Validation Algorithm * **Definition:** Algorithm used to validate JWT tokens if LDAP uses token-based authentication. * **Options:** RS256 | RS384 | RS512 * **Default:** RS256 * **Example:** RS256 ### LDAP Host * **Definition:** Hostname or IP address of your LDAP server. * **Example:** `ldap.company.com` or `192.168.1.10` * **Note:** Do not include `ldap://` or `ldaps://`. ### LDAP Port * **Definition:** Port for the LDAP server connection. * **Example:** `389` (LDAP), `636` (LDAPS) * **Note:** * Use **389** for standard LDAP * Use **636** for secure LDAP (LDAPS) ### Max Pool Size * **Definition:** Maximum number of concurrent connections in the LDAP pool. * **Default:** 3 * **Example:** 5 * **Why it matters:** Controls connection performance and resource usage. ### Full DN Required * **Definition:** Whether users must log in using their full Distinguished Name (DN). * **Default:** false * **Example:** false * **Note:** * `false`: Login with username only * `true`: Requires full DN (e.g., `cn=john,ou=users,dc=company,dc=com`) ### Admin Principal DN * **Definition:** Distinguished Name of the LDAP admin user used for searches. * **Example:** `cn=admin,ou=system,dc=company,dc=com` * **Why it matters:** Required to search for and authenticate users. * **Note:** The account must have read access to users and groups. ### Admin Password * **Definition:** Password for the LDAP admin user. * **Example:** `adminPassword123` * **Why it matters:** Required for binding to the LDAP directory. * **Note:** Store securely (e.g., in a secret manager). ### SSL Enabled * **Definition:** Enables secure LDAP (LDAPS). * **Default:** false * **Example:** true * **Why it matters:** Encrypts communication between OpenMetadata and LDAP. * **Note:** * true β†’ Use LDAPS (port 636) * false β†’ Use LDAP (port 389) ### User Base DN * **Definition:** Base DN under which user accounts are stored. * **Example:** `ou=users,dc=company,dc=com` * **Why it matters:** Defines where to search for user entries. ### Group Base DN * **Definition:** Base DN where group objects reside. * **Example:** `ou=groups,dc=company,dc=com` * **Why it matters:** Required for role-based authorization. * **Note:** Optional if not using group-based role mapping. ### Admin Role Name * **Definition:** LDAP group that maps to OpenMetadata admin role. * **Example:** `OpenMetadata-Admins` * **Why it matters:** Members of this group get admin privileges. ### All Attribute Name * **Definition:** Attribute used to retrieve user attributes. * **Example:** `*` or `memberOf` * **Why it matters:** Defines which attributes to fetch from LDAP. ### Email Attribute Name * **Definition:** LDAP attribute containing user email. * **Example:** `mail` * **Note:** Common values include `mail`, `email`, or `userPrincipalName`. ### Username Attribute Name * **Definition:** LDAP attribute representing usernames. * **Example:** `uid` or `sAMAccountName` * **Note:** * Active Directory: `sAMAccountName` or `userPrincipalName` * OpenLDAP: `uid` or `cn` ### Group Attribute Name * **Definition:** Attribute that defines group membership. * **Example:** `memberOf` * **Why it matters:** Determines user’s LDAP group membership. ### Group Member Attribute Name * **Definition:** Attribute in group entries listing members. * **Example:** `member` or `uniqueMember` ### Auth Roles Mapping * **Definition:** JSON mapping of LDAP groups to OpenMetadata roles. * **Example:** `{"LDAP-Admins": "Admin", "LDAP-Users": "User"}` * **Why it matters:** Automatically assigns roles based on group membership. ### Auth Reassign Roles * **Definition:** Roles that are re-evaluated each time a user logs in. * **Example:** `["Admin", "DataConsumer"]` ### Trust Store Configuration * **Definition:** Configuration for SSL/TLS trust management. * **Why it matters:** Required for LDAPS with custom certificates. ### Trust Store Configuration Type * **Definition:** Type of SSL trust management. * **Options:** `TrustAll`, `JVMDefault`, `HostName`, `CustomTrustStore` * **Example:** `CustomTrustStore` * **Note:** * `TrustAll`: Accepts all certificates (unsafe) * `JVMDefault`: Uses system trust store * `CustomTrustStore`: Uses a custom certificate store ### Verify Hostname * **Definition:** Validates LDAP server certificate hostname. * **Default:** false * **Example:** true ### Examine Validity Dates * **Definition:** Checks SSL certificate validity period. * **Default:** false * **Example:** true ### Trust Store File Path * **Definition:** Path to truststore file containing trusted CA certificates. * **Example:** `/path/to/truststore.jks` ### Trust Store File Password * **Definition:** Password for accessing the truststore. * **Example:** `truststorePassword123` ### Trust Store File Format * **Definition:** Format of the truststore file. * **Example:** `JKS` or `PKCS12` ### Allow Wildcards * **Definition:** Accept wildcard certificates (e.g., `*.company.com`). * **Default:** false * **Example:** true ### Acceptable Host Names * **Definition:** List of hostnames valid for SSL validation. * **Example:** `["ldap.company.com", "ldap-backup.company.com"]` ### JWT Principal Claims * **Definition:** Claims used to identify users in LDAP + JWT setups. * **Example:** `["preferred_username", "email", "sub"]` ### JWT Principal Claims Mapping * **Definition:** Maps LDAP attributes to OpenMetadata user fields. * **Example:** `["email:mail", "name:displayName", "firstName:givenName"]` ### Admin Principals * **Definition:** Users with administrative privileges. * **Example:** `["admin@company.com", "superuser@company.com"]` ### Principal Domain * **Definition:** Default domain for user principals. * **Example:** `company.com` ### Enforce Principal Domain * **Definition:** Restricts logins to specific domains. * **Default:** false * **Example:** true ### Enable Secure Socket Connection * **Definition:** Enables SSL/TLS for LDAP communication. * **Default:** false * **Example:** true * **Note:** Must be enabled for production deployments. Summary Table ------------- | **Field** | **Example / Default** | | --- | --- | | LDAP Host | ldap.company.com | | LDAP Port | 636 | | SSL Enabled | true | | User Base DN | ou=users,dc=company,dc=com | | Group Base DN | ou=groups,dc=company,dc=com | | Username Attribute | uid | | Email Attribute | mail | | Group Attribute | memberOf | | Admin Principal DN | cn=admin,ou=system,dc=company,dc=com | | Token Validation Algorithm | RS256 | | Full DN Required | false | | Trust Store Config Type | CustomTrustStore | | Verify Hostname | true | | Admin Principals | \["admin@company.com"\] | | Principal Domain | company.com | | Enforce Principal Domain | false | | SSL/TLS | true | ### Troubleshooting If users are automatically logged out and unable to log in again due to a bad authentication configuration, you can reset the security setup using the following command: After executing the command, **restart the server**. The authentication values from your YAML or Helm chart will then be reapplied on startup. The following tiles detail how to apply this configuration across Docker, Kubernetes, and Bare Metal deployments: [Docker Security\ \ Configure SSO for your Docker Deployment.](https://docs.open-metadata.org/latest/deployment/security/configuration/docker) [Bare Metal Security\ \ Configure SSO for your Bare Metal Deployment.](https://docs.open-metadata.org/latest/deployment/security/configuration/bare-metal) [Kubernetes Security\ \ Configure SSO for your Kubernetes Deployment.](https://docs.open-metadata.org/latest/deployment/security/configuration/kubernetes) --- # Azure SSO Configuration for Confidential Apps We use cookies to improve site navigation, analyze site usage, and enhance your user experience. Click "Accept" to enable cookies or "Reject" to reject cookies. AcceptReject ![](https://static.scarf.sh/a.png?x-pxid=8472f2b3-d307-4935-a5ed-3f9d3982c15d) deployment No menu items for this category [Home](https://docs.open-metadata.org/latest) /[Deployment](https://docs.open-metadata.org/latest/deployment) /[Security](https://docs.open-metadata.org/latest/deployment/security) /[Azure](https://docs.open-metadata.org/latest/deployment/security/azure) /[Confidential Client](https://docs.open-metadata.org/latest/deployment/security/azure/confidential-client) OpenMetadata Documentation Azure AD SSO Authentication – Confidential Client Configuration =============================================================== * [Troubleshooting](https://docs.open-metadata.org/latest/deployment/security/azure/confidential-client#troubleshooting) Overview -------- **Azure Active Directory (Azure AD) Single Sign-On (SSO)** allows users to log in securely using their **Microsoft 365 / Entra ID** accounts via **OAuth 2.0** and **OpenID Connect (OIDC)**. This guide covers the **Confidential Client** configuration, intended for **web applications and backend services** that can securely store secrets. Confidential Configuration Fields --------------------------------- ![Azure AD SSO Configuration - Confidential Client](https://docs.open-metadata.org/images/v1.11/deployment/security/azure/azure2.png) ### 1\. Client Type * **Definition:** Defines whether the application is public (no client secret) or confidential (requires client secret). * **Options:** `Public` | `Confidential` * **Example:** `Confidential` * **Why it matters:** Determines security level and OAuth flow. * **Note:** * Use `Confidential` for secure applications and backend services. * Azure typically uses `Confidential` type. ### 2\. Authority * **Definition:** Azure AD endpoint that issues tokens for your tenant. * **Example:** `https://login.microsoftonline.com/your-tenant-id` * **Why it matters:** Tells OpenMetadata which Azure AD tenant to authenticate against. * **Note:** * Replace `your-tenant-id` with your Azure AD tenant ID. * Use `common` for multi-tenant applications. ### 3\. Public Key URLs * **Definition:** List of URLs where Azure AD publishes public keys for token verification. * **Example:** `["https://login.microsoftonline.com/common/discovery/v2.0/keys"]` * **Why it matters:** Verifies JWT token signatures. * **Note:** Usually auto-discovered via the discovery URI. ### 4\. Token Validation Algorithm * **Definition:** Algorithm used to validate JWT token signatures. * **Options:** `RS256`, `RS384`, `RS512` * **Default & Example:** `RS256` * **Why it matters:** Must match Azure AD’s signing algorithm. * **Note:** Azure AD typically uses `RS256`. ### 5\. Client Type (OIDC IDP Type) * **Definition:** Defines whether the application is public (no client secret) or confidential (requires client secret). * **Options:** `Public` | `Confidential` * **Example:** `Confidential` * **Why it matters:** Determines security level and OAuth flow. * **Note:** * Use `Confidential` for secure applications and backend services. * Azure typically uses `Confidential` type. ### 6\. OIDC Client ID * **Definition:** Application (client) ID for OIDC authentication with Azure AD. * **Example:** `12345678-1234-1234-1234-123456789012` * **Why it matters:** Identifies the application to Azure AD. * **Note:** Same as the Application (client) ID in Azure AD app registration. ### 7\. OIDC Client Secret * **Definition:** Secret key for confidential client authentication. * **Example:** `abc123def456ghi789jkl012mno345pqr678st` * **Why it matters:** Required to securely authenticate with Azure AD. * **Note:** * Create under **Certificates & secrets** in Azure AD. * Store securely and rotate periodically. ### 8\. OIDC Request Scopes * **Definition:** Permissions requested from Azure AD. * **Default:** `openid email profile` * **Example:** `openid email profile User.Read` * **Why it matters:** Defines what user information OpenMetadata can access. * **Note:** `openid email profile` is usually sufficient. ### 9\. OIDC Discovery URI * **Definition:** Azure AD’s OpenID Connect metadata endpoint. * **Example:** `https://login.microsoftonline.com/your-tenant-id/v2.0/.well-known/openid-configuration` * **Why it matters:** Auto-discovers OIDC endpoints. * **Note:** Replace `your-tenant-id` with your Azure AD tenant ID. ### 10\. OIDC Use Nonce * **Definition:** Prevents replay attacks. * **Default & Example:** `true` * **Why it matters:** Enhances OIDC request security. ### 11\. OIDC Preferred JWS Algorithm * **Default & Example:** `RS256` * **Why it matters:** Must match Azure AD’s token signing algorithm. * **Note:** Rarely needs to be changed. ### 12\. OIDC Response Type * **Definition:** OAuth response type expected. * **Options:** `id_token` | `code` * **Default & Example:** `id_token` * **Why it matters:** Determines the OAuth flow (`code` is more secure). ### 13\. OIDC Disable PKCE * **Definition:** Disables Proof Key for Code Exchange. * **Default & Example:** `false` * **Why it matters:** PKCE provides additional security. * **Note:** Should typically remain enabled (`false`). ### 14\. OIDC Max Clock Skew * **Definition:** Allowed time difference (in seconds) between systems. * **Example:** `0` * **Why it matters:** Prevents token rejection due to system clock differences. ### 15\. OIDC Client Authentication Method * **Definition:** How client authenticates with Azure AD. * **Options:** `client_secret_basic`, `client_secret_post`, `client_secret_jwt`, `private_key_jwt` * **Default & Example:** `client_secret_basic` * **Why it matters:** Must align with Azure AD app registration. ### 16\. OIDC Token Validity * **Definition:** Duration in seconds that tokens remain valid. * **Default:** `0` (use Azure AD default) * **Example:** `3600` * **Why it matters:** Balances token lifetime vs. security. ### 17\. OIDC Tenant * **Definition:** Azure AD tenant identifier. * **Example:** `your-tenant-id` or `company.onmicrosoft.com` * **Why it matters:** Defines which tenant to authenticate against. * **Note:** Use `common` for multi-tenant apps. ### 18\. OIDC Server URL * **Definition:** Base URL of Azure AD authentication server. * **Example:** `https://login.microsoftonline.com` * **Why it matters:** Defines the Azure AD issuer URL. ### 19\. Callback URL * **Definition:** Redirect URI where Azure AD sends auth responses. * **Example:** `https://yourapp.company.com/callback` * **Why it matters:** Must match Azure AD registered redirect URI. * **Note:** * Add to Azure AD β†’ App registrations β†’ Authentication. * Always use HTTPS. ### 20\. OIDC Max Age * **Definition:** Max authentication age before re-login is required. * **Example:** `3600` * **Why it matters:** Controls how often users must re-authenticate. ### 21\. OIDC Prompt * **Definition:** Controls login experience. * **Options:** `none`, `login`, `consent`, `select_account` * **Example:** `select_account` * **Why it matters:** Defines how login prompts behave. * **Note:** * `login`: Always prompt * `consent`: Prompt for permissions * `select_account`: Show account picker ### 22\. OIDC Session Expiry * **Definition:** How long user sessions remain valid (in seconds). * **Default & Example:** `604800` (7 days) * **Why it matters:** Controls session timeout for confidential clients. ### 23\. JWT Principal Claims * **Definition:** JWT fields used to identify users. * **Example:** `["preferred_username", "email", "sub"]` * **Why it matters:** Determines how users are mapped in OpenMetadata. * **Note:** Common claims: `email`, `preferred_username`, `upn`, `sub` ### 24\. JWT Principal Claims Mapping * **Definition:** Maps JWT claims to OpenMetadata user fields. * **Example:** * **Why it matters:** Controls user profile mapping in OpenMetadata. * **Note:** Format: `"openmetadata_field:jwt_claim"` ### 25\. Admin Principals * **Definition:** Users granted admin access in OpenMetadata. * **Example:** `["admin@company.com", "superuser@company.com"]` * **Why it matters:** Grants full admin privileges. * **Note:** Match these to your JWT claims. ### 26\. Bot Principals * **Definition:** Service/bot accounts used for background operations. * **Example:** `["metadata-bot@company.com"]` ### 27\. Principal Domain * **Definition:** Default domain for user principals. * **Example:** `company.com` * **Why it matters:** Helps construct full user identifiers. ### 28\. Enforce Principal Domain * **Definition:** Enforces user domain restriction. * **Default:** `false` * **Example:** `true` * **Why it matters:** Restricts logins to specified domains only. ### 29\. Enable Secure Socket Connection * **Definition:** Enables SSL/TLS for all SSO communication. * **Default:** `false` * **Example:** `true` * **Why it matters:** Ensures secure authentication flow. * **Note:** Recommended in production. ### Summary | Field | Example / Default | | --- | --- | | Client Type | Confidential | | OIDC Client ID | 12345678-1234-1234-1234-123456789012 | | OIDC Client Secret | abc123def456... | | Callback URL | https://yourapp.company.com/callback | | Authority | https://login.microsoftonline.com/your-tenant-id | | Discovery URI | https://login.microsoftonline.com/{tenant}/v2.0/.well-known/openid-configuration | | JWT Claims | \["preferred\_username", "email", "sub"\] | | JWT Mapping | \["email:email", "name:displayName", "firstName:given\_name"\] | | Token Validation Algorithm | RS256 | | Request Scopes | openid email profile User.Read | | Session Expiry | 604800 | ### Troubleshooting If users are automatically logged out and unable to log in again due to a bad authentication configuration, you can reset the security setup using the following command: After executing the command, **restart the server**. The authentication values from your YAML or Helm chart will then be reapplied on startup. The following tiles detail how to apply this configuration across Docker, Kubernetes, and Bare Metal deployments: [Docker Security\ \ Configure SSO for your Docker Deployment.](https://docs.open-metadata.org/latest/deployment/security/configuration/docker) [Bare Metal Security\ \ Configure SSO for your Bare Metal Deployment.](https://docs.open-metadata.org/latest/deployment/security/configuration/bare-metal) [Kubernetes Security\ \ Configure SSO for your Kubernetes Deployment.](https://docs.open-metadata.org/latest/deployment/security/configuration/kubernetes) --- # SAML SSO We use cookies to improve site navigation, analyze site usage, and enhance your user experience. Click "Accept" to enable cookies or "Reject" to reject cookies. AcceptReject ![](https://static.scarf.sh/a.png?x-pxid=8472f2b3-d307-4935-a5ed-3f9d3982c15d) deployment No menu items for this category [Home](https://docs.open-metadata.org/latest) /[Deployment](https://docs.open-metadata.org/latest/deployment) /[Security](https://docs.open-metadata.org/latest/deployment/security) /[Saml](https://docs.open-metadata.org/latest/deployment/security/saml) OpenMetadata Documentation SAML SSO ======== Security requirements for your **production** environment: * **DELETE** the admin default account shipped by OM. * **UPDATE** the Private / Public keys used for the [JWT Tokens](https://docs.open-metadata.org/latest/deployment/security/enable-jwt-tokens) in case it is enabled. Configuring Identity Provider and Service Provider -------------------------------------------------- ### Identity Provide (IDP) Configuration * Every IDP will have the following information 1. EntityId/Authority -> Same as IDP Openmetadata has an Entity Id 2. SignOn Url -> Service Provider SignOn Url 3. X509 Certificate -> In case the SP expects (wantAuthnRequestSigned) then provide certificate for validating. 4. Authority Url -> We just need to update the domain `localhost`. 5. NameID: This is sent as part of request and is provided by the IDP. Every IDP provides this information, we can download the XML Metadata and configure the OM taking the values from the XML. ### Service Provider (SP) Configuration * Openmetadata is the service provider, we just update the `localhost` to the hosted URI. 1. EntityId/Authority -> Normally a Url providing info about the provider. 2. SignOn Url -> Url to be used for signing purpose. 3. X509 Certificate -> In case the SP expects a signed response from IDP, the IDP can be configured with Signing Certificate given by SP. 4. Private Key -> In case SP expects a encrypted response from the IDP , the IDP can be configured with SPs public key for encryption and the Private Key can be used for SP for decrypting. When configuring the Private Key for the Service Provider, ensure you use the actual key content enclosed within the `-----BEGIN PRIVATE KEY-----` and `-----END PRIVATE KEY-----` block. Avoid using the Base64-encoded format of the key, as this is not the expected value. To add a private key, you need to include it in the keystore and update the configuration details accordingly [here](https://github.com/open-metadata/OpenMetadata/blob/main/conf/openmetadata.yaml#L219) . SP Metadata XML is available at "http://localhost:8585/api/v1/saml/acs", `localhost` needs to be updated with the correct URI. ### Security Configuration Security Configuration controls the SP requirement for the Security related aspects. The SP can be configured to send signed or encrypted or both request , and in return can also expect signed or encrypted or both responses from the IDP. Setup JWT Configuration ----------------------- Jwt Configuration is mandatory for Saml SSO. * Follow the guide here for JWT Configuration [Enable JWT Token](https://docs.open-metadata.org/latest/deployment/security/enable-jwt-tokens) . Security requirements for your **production** environment: * **UPDATE** the Private / Public keys used for the [JWT Tokens](https://docs.open-metadata.org/latest/deployment/security/enable-jwt-tokens) the ones shipped with OM are for POC only. More specific details on different IDPs can be found below: [AWS Saml\ \ Configure AWS as IDP.](https://docs.open-metadata.org/latest/deployment/security/saml/aws) [Azure Saml\ \ Configure AWS as IDP.](https://docs.open-metadata.org/latest/deployment/security/saml/azure) Configure Ingestion ------------------- Once your server security is set, it's time to review the ingestion configuration. Our bots support JWT tokens to authenticate to the server when sending requests. Find more information on [**Enabling JWT Tokens**](https://docs.open-metadata.org/latest/deployment/security/enable-jwt-tokens) and [**JWT Troubleshooting**](https://docs.open-metadata.org/latest/deployment/security/jwt-troubleshooting) to ensure seamless authentication. --- # Bare Metal Deployment | Official Documentation We use cookies to improve site navigation, analyze site usage, and enhance your user experience. Click "Accept" to enable cookies or "Reject" to reject cookies. AcceptReject deployment No menu items for this category [Home](https://docs.open-metadata.org/latest) /[Deployment](https://docs.open-metadata.org/latest/deployment) /[Bare Metal](https://docs.open-metadata.org/latest/deployment/bare-metal) OpenMetadata Documentation Deploy on Bare Metal ==================== Requirements This guide assumes you have access to a command-line environment or shell such as bash, zsh, etc. or Linux or Mac OS X or PowerShell on Microsoft Windows. This guide also assumes that your command-line environment has access to the tar utility. Please review additional requirements listed in the subsections below. Java (version 21.0.0) --------------------- OpenMetadata is built using Java, DropWizard, and Jetty. Type the following command to verify that you have a supported version of the Java runtime installed. To install Java or upgrade to Java 21, see the instructions for your operating system at [How do I install Java?](https://java.com/en/download/help/download_options.html#mac) . MySQL (version 8.0.0 or higher) ------------------------------- To install MySQL see the instructions for your operating system (OS) at [Installing and Upgrading MySQL](https://dev.mysql.com/doc/mysql-installation-excerpt/8.0/en/installing.html) or visit one of the following OS-specific guides. * [Installing MySQL on Linux](https://dev.mysql.com/doc/mysql-installation-excerpt/8.0/en/linux-installation.html) * [Installing MySQL on Windows](https://dev.mysql.com/doc/mysql-installation-excerpt/8.0/en/windows-installation.html) * [Installing MySQL on MacOS](https://dev.mysql.com/doc/mysql-installation-excerpt/8.0/en/macos-installation.html) Make sure to configure required databases and users for OpenMetadata. You can refer a sample script [here](https://github.com/open-metadata/OpenMetadata/blob/main/docker/mysql/mysql-script.sql) . Postgres (version 12.0 or higher) --------------------------------- To install Postgres see the instructions for your operating system (OS) at [Postgres Download](https://www.postgresql.org/download/) Make sure to configure required databases and users for OpenMetadata. You can refer a sample script [here](https://github.com/open-metadata/OpenMetadata/blob/main/docker/postgresql/postgres-script.sql) . Elasticsearch (version 8.X) --------------------------- OpenMetadata supports ElasticSearch version up to 8.11.4. To install or upgrade Elasticsearch to a supported version please see the instructions for your operating system at [Installing ElasticSearch](https://www.elastic.co/guide/en/elasticsearch/reference/current/install-elasticsearch.html) . Please follow the instructions here to [install ElasticSearch](https://www.elastic.co/guide/en/elasticsearch/reference/7.13/setup.html) . If you are using AWS OpenSearch Service, OpenMetadata Supports AWS OpenSearch Service engine version up to 2.19. For more information on AWS OpenSearch Service, please visit the official docs [here](https://docs.aws.amazon.com/opensearch-service/latest/developerguide/what-is.html) . Airflow or other workflow schedulers ------------------------------------ OpenMetadata performs metadata ingestion using the Ingestion Framework. Learn more about how to deploy and manage the ingestion workflows [here](https://docs.open-metadata.org/latest/deployment/ingestion) . OpenMetadata versions have specific Airflow compatibility requirements to ensure seamless metadata ingestion. OpenMetadata 1.5 supports Airflow 2.9, 1.6.4 supports Airflow 2.9.3, 1.6.5 supports Airflow 2.10.5 and from 1.11.x we support 3.1.5. Ensure that your Airflow version aligns with your OpenMetadata deployment to maintain stability and functionality. Minimum Sizing Requirements --------------------------- * Our minimum specs recommendation for the OpenMetadata Deployment (one replica) is 2 vCPUs and 4 Gigs with 20 Gigs of volume size if using persistent volumes for logs. * For Elasticsearch, 2 vCPUs and 2 Gigs RAM (per instance) with 30 Gigs of Storage volume attached. * For the database, 2 vCPUs and 2 Gigs RAM (per instance) with 30 Gigs of Storage Volume Attached (dynamic expansion up to 100 Gigs). These settings apply as well when using managed instances, such as RDS or AWS OpenSearch. Procedure ========= 1\. Download the distribution ----------------------------- Visit the [releases page](https://github.com/open-metadata/OpenMetadata/releases/latest) and download the latest binary release. Release binaries follow the naming convention of `openmetadata-x.y.z.tar.gz`. Where `x`, `y`, and `z` represent the major, minor, and patch release numbers. 2\. Untar the release download ------------------------------ Once the tar file has downloaded, run the following command, updated if necessary for the version of OpenMetadata that you downloaded. 3\. Navigate to the directory created ------------------------------------- Review and update the `openmetadata.yaml` configurations to match your environment. Specifically, consider aspects such as the connection to the MySQL database or ElasticSearch. You can find more information about these configurations [here](https://docs.open-metadata.org/latest/deployment/configuration) . 4\. Prepare the OpenMetadata Database and Indexes ------------------------------------------------- The command below will generate all the necessary tables and indexes in ElasticSearch. Note that if there's any data in that database, this command will drop it! 5\. Start OpenMetadata ---------------------- We recommend configuring `serviced` to monitor the OpenMetadata command to restart in case of any failures. Run OpenMetadata with a load balancer ------------------------------------- You may put one or more OpenMetadata instances behind a load balancer for reverse proxying. To do this you will need to add one or more entries to the configuration file for your reverse proxy. ### Apache mod\_proxy To use the Apache mod\_proxy module as a reverse proxy for load balancing, update the VirtualHost tag in your Apache config file to resemble the following. ### Nginx To use OpenMetadata behind an Nginx reverse proxy, add an entry resembling the following the http context of your Nginx configuration file for each OpenMetadata instance. Run OpenMetadata with AWS Services or your hosted DB/ElasticSearch ------------------------------------------------------------------ If you are running OpenMetadata in AWS, it is recommended to use [Amazon RDS](https://docs.aws.amazon.com/rds/index.html) and [Amazon OpenSearch Service](https://docs.aws.amazon.com/opensearch-service/?id=docs_gateway) . We support * Amazon RDS (MySQL) engine version 8 or higher * Amazon OpenSearch (ElasticSearch) engine version up to 8.11.4 or Amazon OpenSearch engine version up to 2.19 * Amazon RDS (PostgreSQL) engine version between 12 or higher For Production Systems, we recommend Amazon RDS to be in Multiple Availability Zones. For Amazon OpenSearch (or ElasticSearch) Service, we recommend Multiple Availability Zones with minimum 3 Master Nodes. Once you have the RDS and OpenSearch Services Setup, you can update the environment variables below for OpenMetadata bare metal systems to connect with Database and ElasticSearch. Below are the environment variables for OpenMetadata Server ### Configure MySQL connection ### Configure Postgres Connection ### Configure ElasticSearch Connection ### Configure OpenSearch If you want to separate indexes for production and non-production environments, you can set the `clusterAlias` in the configuration file. ### Configure Ingestion When setting up environment file if your custom password includes any special characters then make sure to follow the steps [here](https://github.com/open-metadata/OpenMetadata/issues/12110#issuecomment-1611341650) . Troubleshooting --------------- ### Java Memory Heap Issue If your openmetadata application logs speaks about the below issue - This is due to the default JVM Heap Space configuration (1 GiB) being not enough for your workloads. In order to resolve this issue, head over to your openmetadata environment variables list and append the below environment variable The flag `Xmx` specifies the maximum memory allocation pool for a Java virtual machine (JVM), while `Xms` specifies the initial memory allocation pool. Restart the OpenMetadata Application using `./bin/openmetadata.sh start` which will start the service using a linux process. Enable Security --------------- Please follow our [Enable Security Guide](https://docs.open-metadata.org/latest/deployment/bare-metal/security) to configure security for your OpenMetadata installation. --- # Secrets Manager | OpenMetadata Deployment Integration We use cookies to improve site navigation, analyze site usage, and enhance your user experience. Click "Accept" to enable cookies or "Reject" to reject cookies. AcceptReject deployment No menu items for this category [Home](https://docs.open-metadata.org/latest) /[Deployment](https://docs.open-metadata.org/latest/deployment) /[Secrets Manager](https://docs.open-metadata.org/latest/deployment/secrets-manager) /[Supported Implementations](https://docs.open-metadata.org/latest/deployment/secrets-manager/supported-implementations) OpenMetadata Documentation Supported implementations ========================= This is our list of supported Secrets Manager implementations: [AWS Secrets Manager\ \ AWS Secrets Manager](https://docs.open-metadata.org/latest/deployment/secrets-manager/supported-implementations/aws-secrets-manager) [AWS Systems Manager Parameter Store\ \ AWS Systems Manager Parameter Store](https://docs.open-metadata.org/latest/deployment/secrets-manager/supported-implementations/aws-ssm-parameter-store) [Azure Key Vault\ \ Azure Key Vault](https://docs.open-metadata.org/latest/deployment/secrets-manager/supported-implementations/azure-key-vault) [GCP Secrets Manager\ \ GCP Secrets Manager](https://docs.open-metadata.org/latest/deployment/secrets-manager/supported-implementations/gcp-secret-manager) --- # Dashboard Services | Connect BI Tools with OpenMetadata We use cookies to improve site navigation, analyze site usage, and enhance your user experience. Click "Accept" to enable cookies or "Reject" to reject cookies. AcceptReject connectors No menu items for this category [Home](https://docs.open-metadata.org/latest) /[Connectors](https://docs.open-metadata.org/latest/connectors) /[Dashboard](https://docs.open-metadata.org/latest/connectors/dashboard) OpenMetadata Documentation Dashboard Services ================== This is the supported list of connectors for Dashboard Services: [![Domo](https://docs.open-metadata.org/_next/image?url=%2Fimages%2Fconnectors%2Fdomo.webp&w=64&q=75)\ \ Domo\ \ PROD\ \ Available In](https://docs.open-metadata.org/latest/connectors/dashboard/domo-dashboard) [![Grafana](https://docs.open-metadata.org/_next/image?url=%2Fimages%2Fconnectors%2Fgrafana.webp&w=64&q=75)\ \ Grafana\ \ BETA\ \ Available In](https://docs.open-metadata.org/latest/connectors/dashboard/grafana) [![Hex](https://docs.open-metadata.org/_next/image?url=%2Fimages%2Fconnectors%2Fhex.webp&w=64&q=75)\ \ Hex\ \ BETA\ \ Available In](https://docs.open-metadata.org/latest/connectors/dashboard/hex) [![Lightdash](https://docs.open-metadata.org/_next/image?url=%2Fimages%2Fconnectors%2Flightdash.webp&w=64&q=75)\ \ Lightdash\ \ PROD\ \ Available In](https://docs.open-metadata.org/latest/connectors/dashboard/lightdash) [![Looker](https://docs.open-metadata.org/_next/image?url=%2Fimages%2Fconnectors%2Flooker.webp&w=64&q=75)\ \ Looker\ \ PROD\ \ Available In](https://docs.open-metadata.org/latest/connectors/dashboard/looker) [![Metabase](https://docs.open-metadata.org/_next/image?url=%2Fimages%2Fconnectors%2Fmetabase.webp&w=64&q=75)\ \ Metabase\ \ PROD\ \ Available In](https://docs.open-metadata.org/latest/connectors/dashboard/metabase) [![MicroStrategy](https://docs.open-metadata.org/_next/image?url=%2Fimages%2Fconnectors%2Fmicrostrategy.webp&w=64&q=75)\ \ MicroStrategy\ \ PROD\ \ Available In](https://docs.open-metadata.org/latest/connectors/dashboard/microstrategy) [![Mode](https://docs.open-metadata.org/_next/image?url=%2Fimages%2Fconnectors%2Fmode.webp&w=64&q=75)\ \ Mode\ \ PROD\ \ Available In](https://docs.open-metadata.org/latest/connectors/dashboard/mode) [![PowerBI](https://docs.open-metadata.org/_next/image?url=%2Fimages%2Fconnectors%2Fpower-bi.webp&w=64&q=75)\ \ PowerBI\ \ PROD\ \ Available In](https://docs.open-metadata.org/latest/connectors/dashboard/powerbi) [![Qlik Cloud](https://docs.open-metadata.org/_next/image?url=%2Fimages%2Fconnectors%2Fqlikcloud.webp&w=64&q=75)\ \ Qlik Cloud\ \ PROD\ \ Available In](https://docs.open-metadata.org/latest/connectors/dashboard/qlikcloud) [![Qlik Sense](https://docs.open-metadata.org/_next/image?url=%2Fimages%2Fconnectors%2Fqlik-sense.webp&w=64&q=75)\ \ Qlik Sense\ \ PROD\ \ Available In](https://docs.open-metadata.org/latest/connectors/dashboard/qliksense) [![QuickSight](https://docs.open-metadata.org/_next/image?url=%2Fimages%2Fconnectors%2Fquicksight.webp&w=64&q=75)\ \ QuickSight\ \ PROD\ \ Available In](https://docs.open-metadata.org/latest/connectors/dashboard/quicksight) [![Redash](https://docs.open-metadata.org/_next/image?url=%2Fimages%2Fconnectors%2Fredash.webp&w=64&q=75)\ \ Redash\ \ PROD\ \ Available In](https://docs.open-metadata.org/latest/connectors/dashboard/redash) [![Sigma](https://docs.open-metadata.org/_next/image?url=%2Fimages%2Fconnectors%2Fsigma.webp&w=64&q=75)\ \ Sigma\ \ PROD\ \ Available In](https://docs.open-metadata.org/latest/connectors/dashboard/sigma) [![Superset](https://docs.open-metadata.org/_next/image?url=%2Fimages%2Fconnectors%2Fsuperset.webp&w=64&q=75)\ \ Superset\ \ PROD\ \ Available In](https://docs.open-metadata.org/latest/connectors/dashboard/superset) [![Tableau](https://docs.open-metadata.org/_next/image?url=%2Fimages%2Fconnectors%2Ftableau.webp&w=64&q=75)\ \ Tableau\ \ PROD\ \ Available In](https://docs.open-metadata.org/latest/connectors/dashboard/tableau) If you have a request for a new connector, don't hesitate to reach out in [Slack](https://slack.open-metadata.org/) or open a [feature request](https://github.com/open-metadata/OpenMetadata/issues/new/choose) in our GitHub repo. --- # Pipeline Services | OpenMetadata Data Pipeline Guide We use cookies to improve site navigation, analyze site usage, and enhance your user experience. Click "Accept" to enable cookies or "Reject" to reject cookies. AcceptReject connectors No menu items for this category [Home](https://docs.open-metadata.org/latest) /[Connectors](https://docs.open-metadata.org/latest/connectors) /[Pipeline](https://docs.open-metadata.org/latest/connectors/pipeline) OpenMetadata Documentation Pipeline Services ================= This is the supported list of connectors for Pipeline Services: [![Airbyte](https://docs.open-metadata.org/_next/image?url=%2Fimages%2Fconnectors%2Fairbyte.webp&w=64&q=75)\ \ Airbyte\ \ PROD\ \ Available In](https://docs.open-metadata.org/latest/connectors/pipeline/airbyte) [![Airflow](https://docs.open-metadata.org/_next/image?url=%2Fimages%2Fconnectors%2Fairflow.webp&w=64&q=75)\ \ Airflow\ \ PROD\ \ Available In](https://docs.open-metadata.org/latest/connectors/pipeline/airflow) [![Dagster](https://docs.open-metadata.org/_next/image?url=%2Fimages%2Fconnectors%2Fdagster.webp&w=64&q=75)\ \ Dagster\ \ PROD\ \ Available In](https://docs.open-metadata.org/latest/connectors/pipeline/dagster) [![Databricks Pipeline](https://docs.open-metadata.org/_next/image?url=%2Fimages%2Fconnectors%2Fdatabrick.webp&w=64&q=75)\ \ Databricks Pipeline\ \ PROD\ \ Available In](https://docs.open-metadata.org/latest/connectors/pipeline/databricks-pipeline) [![dbt Cloud](https://docs.open-metadata.org/_next/image?url=%2Fimages%2Fconnectors%2Fdbtcloud.webp&w=64&q=75)\ \ dbt Cloud\ \ PROD\ \ Available In](https://docs.open-metadata.org/latest/connectors/pipeline/dbtcloud) [![Domo](https://docs.open-metadata.org/_next/image?url=%2Fimages%2Fconnectors%2Fdomo.webp&w=64&q=75)\ \ Domo\ \ PROD\ \ Available In](https://docs.open-metadata.org/latest/connectors/pipeline/domo-pipeline) [![Fivetran](https://docs.open-metadata.org/_next/image?url=%2Fimages%2Fconnectors%2Ffivetran.webp&w=64&q=75)\ \ Fivetran\ \ PROD\ \ Available In](https://docs.open-metadata.org/latest/connectors/pipeline/fivetran) [![Flink](https://docs.open-metadata.org/_next/image?url=%2Fimages%2Fconnectors%2Fflink.webp&w=64&q=75)\ \ Flink\ \ PROD\ \ Available In](https://docs.open-metadata.org/latest/connectors/pipeline/flink) [![Glue](https://docs.open-metadata.org/_next/image?url=%2Fimages%2Fconnectors%2Fglue.webp&w=64&q=75)\ \ Glue\ \ PROD\ \ Available In](https://docs.open-metadata.org/latest/connectors/pipeline/glue-pipeline) [![KafkaConnect](https://docs.open-metadata.org/_next/image?url=%2Fimages%2Fconnectors%2Fkafka.webp&w=64&q=75)\ \ KafkaConnect\ \ PROD\ \ Available In](https://docs.open-metadata.org/latest/connectors/pipeline/kafkaconnect) [![NiFi](https://docs.open-metadata.org/_next/image?url=%2Fimages%2Fconnectors%2Fapachenifi.webp&w=64&q=75)\ \ NiFi\ \ PROD\ \ Available In](https://docs.open-metadata.org/latest/connectors/pipeline/nifi) [![OpenLineage](https://docs.open-metadata.org/_next/image?url=%2Fimages%2Fconnectors%2Fopenlineage.webp&w=64&q=75)\ \ OpenLineage\ \ BETA\ \ Available In](https://docs.open-metadata.org/latest/connectors/pipeline/openlineage) [![Spline](https://docs.open-metadata.org/_next/image?url=%2Fimages%2Fconnectors%2Fspline.webp&w=64&q=75)\ \ Spline\ \ PROD\ \ Available In](https://docs.open-metadata.org/latest/connectors/pipeline/spline) If you have a request for a new connector, don't hesitate to reach out in [Slack](https://slack.open-metadata.org/) or open a [feature request](https://github.com/open-metadata/OpenMetadata/issues/new/choose) in our GitHub repo. --- # Enable SSL with Nginx | OpenMetadata Security Setup We use cookies to improve site navigation, analyze site usage, and enhance your user experience. Click "Accept" to enable cookies or "Reject" to reject cookies. AcceptReject ![](https://static.scarf.sh/a.png?x-pxid=8472f2b3-d307-4935-a5ed-3f9d3982c15d) deployment No menu items for this category [Home](https://docs.open-metadata.org/latest) /[Deployment](https://docs.open-metadata.org/latest/deployment) /[Security](https://docs.open-metadata.org/latest/deployment/security) /[Enable Ssl](https://docs.open-metadata.org/latest/deployment/security/enable-ssl) /[Nginx](https://docs.open-metadata.org/latest/deployment/security/enable-ssl/nginx) OpenMetadata Documentation Enable SSL with Nginx ===================== Nginx can be used as a load balancer or an SSL termination point for OpenMetadata. In this section, we will look at how to use Nginx and Certbot to deploy SSL. The below instructions are for Ubuntu 20 and any other flavor of Linux please find similar instructions. Install Nginx ------------- Nginx can be installed to a completely different host where you are running OpenMetadata Server or on the same host. For simplicity, we will do this on the same host as the OpenMetadata server. Configure Nginx to redirect requests to OpenMetadata ---------------------------------------------------- For Nginx to serve this content, it’s necessary to create a server block with the correct directives. Instead of modifying the default configuration file directly, let’s make a new one at `/etc/nginx/sites-available/openmetadata`: And add the below content In the above configuration, please ensure that the `server_name` matches the domain where you are hosting the OpenMetadata server. Also, the `proxy_pass` configuration should point to the OpenMetadata server port. Then, link the configuration to `sites-enabled` and restart nginx: The above configuration will serve at port 80, so if you configured a domain like `sandbox.open-metadata.org` one can start accessing OpenMetadata server by just pointing the browser to [http://sandbox.open-metadata.org](http://sandbox.open-metadata.org/) . Enable SSL using Certbot ------------------------ Certbot, [https://certbot.eff.org/](https://certbot.eff.org/) , is a non-profit org that distributes the certified X509 certs and renews them as well. Obtaining an SSL Certificate ---------------------------- Certbot provides a variety of ways to obtain SSL certificates through plugins. The Nginx plugin will take care of reconfiguring Nginx and reloading the config whenever necessary. To use this plugin, type the following: Replace `sandbox.open-metadata.org` with your domain for OpenMetadata. If this is your first time running certbot, you will be prompted to enter an email address and agree to the terms of service. After doing so, certbot will communicate with the `Let's Encrypt` server, then run a challenge to verify that you control the domain you’re requesting a certificate for. If that’s successful, certbot will ask how you’d like to configure your HTTPS settings. Verifying Certbot Auto-Renewal ------------------------------ `Let's Encrypt`'s certificates are only valid for ninety days. This is to encourage users to automate their certificate renewal process. The certbot package we installed takes care of this for us by adding a `systemd` timer that will run twice a day and automatically renew any certificate that’s within thirty days of expiration. You can query the status of the timer with `systemctl`: to renew, you can run the following command Summary ------- In this tutorial, we walked through the setup of Nginx to serve the requests to OpenMetadata and used Certbot to enable SSL on Nginx. Do keep in mind that we secured the external connection to Nginx, and Nginx terminates the SSL connections, and the rest of the transport Nginx to the OpenMetadata server is on Plaintext. However, OpenMetadata server should be configured to listen to only localhost requests, i.e., It cannot be reached directly from outside traffic except for Nginx on that host. This makes it a secure SSL. --- # Run the ingestion from GCP Composer | Official Documentation We use cookies to improve site navigation, analyze site usage, and enhance your user experience. Click "Accept" to enable cookies or "Reject" to reject cookies. AcceptReject deployment No menu items for this category [Home](https://docs.open-metadata.org/latest) /[Deployment](https://docs.open-metadata.org/latest/deployment) /[Ingestion](https://docs.open-metadata.org/latest/deployment/ingestion) /[External](https://docs.open-metadata.org/latest/deployment/ingestion/external) /[Gcp Composer](https://docs.open-metadata.org/latest/deployment/ingestion/external/gcp-composer) OpenMetadata Documentation This page is about running the Ingestion Framework **externally**! There are mainly 2 ways of running the ingestion: 1. Internally, by managing the workflows from OpenMetadata. 2. Externally, by using any other tool capable of running Python code. If you are looking for how to manage the ingestion process from OpenMetadata, you can follow this [doc](https://docs.open-metadata.org/latest/deployment/ingestion/openmetadata) . Run the ingestion from GCP Composer =================================== Requirements ------------ This approach has been last tested against: * Composer version 2.5.4 * Airflow version 2.6.3 It also requires the ingestion package to be at least `openmetadata-ingestion==1.3.1.0`. Using the Python Operator ------------------------- The most comfortable way to run the metadata workflows from GCP Composer is directly via a `PythonOperator`. Note that it will require you to install the packages and plugins directly on the host. ### Install the Requirements In your environment you will need to install the following packages: * `openmetadata-ingestion[]==x.y.z`. * `sqlalchemy==1.4.27`: This is needed to align OpenMetadata version with the Composer internal requirements. Where `x.y.z` is the version of the OpenMetadata ingestion package. Note that the version needs to match the server version. If we are using the server at 1.1.0, then the ingestion package needs to also be 1.1.0. The plugin parameter is a list of the sources that we want to ingest. An example would look like this `openmetadata-ingestion[mysql,snowflake,s3]==1.1.0`. ### Prepare the DAG! Note that this DAG is a usual connector DAG, just using the Airflow service with the `Backend` connection. As an example of a DAG pushing data to OpenMetadata under Google SSO, we could have: Ingestion Workflow classes -------------------------- We have different classes for different types of workflows. The logic is always the same, but you will need to change your import path. The rest of the method calls will remain the same. For example, for the `Metadata` workflow we'll use: The classes for each workflow type are: * `Metadata`: `from metadata.workflow.metadata import MetadataWorkflow` * `Lineage`: `from metadata.workflow.metadata import MetadataWorkflow` (same as metadata) * `Usage`: `from metadata.workflow.usage import UsageWorkflow` * `dbt`: `from metadata.workflow.metadata import MetadataWorkflow` * `Profiler`: `from metadata.workflow.profiler import ProfilerWorkflow` * `Data Quality`: `from metadata.workflow.data_quality import TestSuiteWorkflow` * `Data Insights`: `from metadata.workflow.data_insight import DataInsightWorkflow` * `Elasticsearch Reindex`: `from metadata.workflow.metadata import MetadataWorkflow` (same as metadata) Using the Kubernetes Pod Operator --------------------------------- In this second approach we won't need to install absolutely anything to the GCP Composer environment. Instead, we will rely on the `KubernetesPodOperator` to use the underlying k8s cluster of Composer. Then, the code won't directly run using the hosts' environment, but rather inside a container that we created with only the `openmetadata-ingestion` package. **Note:** This approach only has the `openmetadata/ingestion-base` ready from version 0.12.1 or higher! ### Prepare the DAG! Some remarks on this example code: #### Kubernetes Pod Operator You can name the task as you want (`task_id` and `name`). The important points here are the `cmds`, this should not be changed, and the `env_vars`. The `main.py` script that gets shipped within the image will load the env vars as they are shown, so only modify the content of the config YAML, but not this dictionary. Note that the example uses the image `openmetadata/ingestion-base:0.13.2`. Update that accordingly for higher version once they are released. Also, the image version should be aligned with your OpenMetadata server version to avoid incompatibilities. You can find more information about the `KubernetesPodOperator` and how to tune its configurations [here](https://cloud.google.com/composer/docs/how-to/using/using-kubernetes-pod-operator) . Note that depending on the kind of workflow you will be deploying, the YAML configuration will need to updated following the official OpenMetadata docs, and the value of the `pipelineType` configuration will need to hold one of the following values: * `metadata` * `usage` * `lineage` * `profiler` * `TestSuite` Which are based on the `PipelineType` [JSON Schema definitions](https://github.com/open-metadata/OpenMetadata/blob/main/openmetadata-spec/src/main/resources/json/schema/entity/services/ingestionPipelines/ingestionPipeline.json#L14) --- # Data Quality Tab | OpenMetadata Quality Interface We use cookies to improve site navigation, analyze site usage, and enhance your user experience. Click "Accept" to enable cookies or "Reject" to reject cookies. AcceptReject ![](https://static.scarf.sh/a.png?x-pxid=6814879a-96c0-40bd-be69-ddfeb14694aa) how-to-guides No menu items for this category [Home](https://docs.open-metadata.org/latest) /[How To Guides](https://docs.open-metadata.org/latest/how-to-guides) /[Data Quality Observability](https://docs.open-metadata.org/latest/how-to-guides/data-quality-observability) /[Quality](https://docs.open-metadata.org/latest/how-to-guides/data-quality-observability/quality) /[Tab](https://docs.open-metadata.org/latest/how-to-guides/data-quality-observability/quality/tab) OpenMetadata Documentation Data Quality Tab ================ The Profiler & Data Quality tab is displayed only for Tables. It has three sub-tabs for **Table Profile, Column Profile, and Data Quality**. Data quality tests can be run on the sample data. We can add tests at the table and column level. The Data Quality tab displays the total number of tests that were run, and also the number of tests that were successful, aborted, or failed. The list of test cases displays the details of the table or column on which the test was run. ![Profiler & Data Quality](https://docs.open-metadata.org/images/v1.11/how-to-guides/quality/dq1.png) Profiler & Data Quality You can click on a Test Case to view further details. You can use a time filter on these reports. You can also edit these tests by clicking on the pencil icon next to each test. ![Details of a Test Case](https://docs.open-metadata.org/images/v1.11/how-to-guides/quality/dq2.png) Details of a Test Case [How to Write and Deploy No-Code Test Cases from the UI\ \ Verify your data quality with table and column level tests.](https://docs.open-metadata.org/latest/how-to-guides/data-quality-observability/quality/test) --- # OpenMetadata System Architecture | Developer Guide We use cookies to improve site navigation, analyze site usage, and enhance your user experience. Click "Accept" to enable cookies or "Reject" to reject cookies. AcceptReject developers No menu items for this category [Home](https://docs.open-metadata.org/latest) /[Developers](https://docs.open-metadata.org/latest/developers) /[Architecture](https://docs.open-metadata.org/latest/developers/architecture) OpenMetadata Documentation Architecture ============ OpenMetadata Unlock the value of data assets with an end-to-end metadata platform that includes data discovery, governance, data quality, observability, and people collaboration. OpenMetadata depends on following components to build a metadata platform: * JsonSchemas for defining Metadata Schemas * Dropwizard/Jetty for REST APIs * MySQL 8.x to store Metadata * ElasticSearch 7.x to index Metadata and power search ![OpenMetadata architecture](https://docs.open-metadata.org/images/v1.11/developers/architecture/architecture.png) To understand the OpenMetadata Architecture and how everything fits together please go through [Design page](https://docs.open-metadata.org/latest/main-concepts/high-level-design) . For Schema design and how our API works here is an example of ML [Model entity page](https://docs.open-metadata.org/latest/sdk/python/entities/ml-model) --- # Enable SSL at the OpenMetadata Server We use cookies to improve site navigation, analyze site usage, and enhance your user experience. Click "Accept" to enable cookies or "Reject" to reject cookies. AcceptReject ![](https://static.scarf.sh/a.png?x-pxid=8472f2b3-d307-4935-a5ed-3f9d3982c15d) deployment No menu items for this category [Home](https://docs.open-metadata.org/latest) /[Deployment](https://docs.open-metadata.org/latest/deployment) /[Security](https://docs.open-metadata.org/latest/deployment/security) /[Enable Ssl](https://docs.open-metadata.org/latest/deployment/security/enable-ssl) /[Openmetadata Server](https://docs.open-metadata.org/latest/deployment/security/enable-ssl/openmetadata-server) OpenMetadata Documentation Enable SSL at the OpenMetadata Server ===================================== The OpenMetadata Server is built using **Dropwizard** and **Jetty**. In this section, we will go through the steps involved in setting up SSL for Jetty. If you would like a simple way to set up SSL, please refer to the guide using [Nginx](https://docs.open-metadata.org/latest/deployment/security/enable-ssl/nginx) . However, this step can be treated as an additional layer of adding SSL to OpenMetadata. In cases where one would use Nginx as a load balancer or AWS LB, you can set up SSL at the OpenMetadata server level such that traffic from the load balancer to OpenMetadata is going through an encrypted channel. Create Self-Signed Certificate ------------------------------ A self-signed certificate should only be used for POC (demo) or `localhost` installation. For production scenarios, please reach out to your DevOps team to issue an X509 certificate which you can import into a Keystore. Run the below command to generate an X509 Certificate and import it into keystore: ![keystore](https://docs.open-metadata.org/images/v1.11/deployment/security/enable-ssl/openmetadata-server/keystore-1.png) For this example, we are configuring the password to be `test12`. Copy the generated `openmetadata.keystore.jks` to OpenMetadata installation path under the `conf` directory. ![keystore](https://docs.open-metadata.org/images/v1.11/deployment/security/enable-ssl/openmetadata-server/keystore-2.png) Configure openmetadata.yaml --------------------------- Add the below section to your `openmetadata.yaml` under the `conf` directory. Please add the password you set for the Keystore generated above in the config below. Access OpenMetadata server in the browser ----------------------------------------- These steps are not necessary if you used proper X509 certificated signed by trusted CA Authority. Since we used self-signed certificates, browsers such as Chrome or Brave will not allow you to visit [https://localhost:8585](https://localhost:8585/) . You'll get the following error page and there is no way to proceed. ![browser](https://docs.open-metadata.org/images/v1.11/deployment/security/enable-ssl/openmetadata-server/browser.png) However, the Safari browser allows you to visit if you click advanced and click proceed. To work around this issue, on OS X, you can import the certificate into the keychain and trust it so that browsers can trust and allow you to access OpenMetadata. ### Export X509 certificate from Keystore Run the below command to export the X509 cert. ### Import public cert into Keychain - OS X only Open the KeyChain app in OS X, drag and drop the `public.cert` file generated in the previous command into the Keychain: ![import](https://docs.open-metadata.org/images/v1.11/deployment/security/enable-ssl/openmetadata-server/import-1.png) Double-click on `localhost`: ![import](https://docs.open-metadata.org/images/v1.11/deployment/security/enable-ssl/openmetadata-server/import-2.png) Click on `Trust` to open and set `Always Trust`: ![import](https://docs.open-metadata.org/images/v1.11/deployment/security/enable-ssl/openmetadata-server/import-3.png) Once the above steps are finished, all the browsers will allow you to visit the OpenMetadata server using HTTPS. However, you'll still a warning in the address bar. All of these steps are not necessary with an X509 certificate issued by a trusted authority and one should always use that in production. --- # Data Quality as Code We use cookies to improve site navigation, analyze site usage, and enhance your user experience. Click "Accept" to enable cookies or "Reject" to reject cookies. AcceptReject ![](https://static.scarf.sh/a.png?x-pxid=6814879a-96c0-40bd-be69-ddfeb14694aa) how-to-guides No menu items for this category [Home](https://docs.open-metadata.org/latest) /[How To Guides](https://docs.open-metadata.org/latest/how-to-guides) /[Data Quality Observability](https://docs.open-metadata.org/latest/how-to-guides/data-quality-observability) /[Quality](https://docs.open-metadata.org/latest/how-to-guides/data-quality-observability/quality) /[Data Quality As Code](https://docs.open-metadata.org/latest/how-to-guides/data-quality-observability/quality/data-quality-as-code) OpenMetadata Documentation Data Quality as Code ==================== Data Quality as Code enables you to programmatically build, run, and manage data quality tests within your ETL workflows using the OpenMetadata Python SDK. This approach allows data engineers and developers to integrate data quality validation directly into their data pipelines, ensuring data quality is verified at every stage of the data lifecycle. Why Data Quality as Code? ------------------------- Traditional data quality testing often requires manual configuration through UIs or separate workflow systems. Data Quality as Code brings several advantages: * **Integration with ETL workflows**: Run data quality tests directly within your existing Python-based ETL pipelines * **Version control**: Manage test definitions alongside your code in version control systems * **Developer-friendly**: Use familiar Python syntax and IDE features for test development * **Programmatic control**: Dynamically generate tests based on data discovery or metadata * **Immediate feedback**: Validate data transformations before loading to destinations * **Shared responsibility**: Data stewards define tests in OpenMetadata UI, engineers execute them in code Key Features ------------ ### TestRunner API Execute data quality tests against tables cataloged in OpenMetadata: ### DataFrame Validation Validate pandas DataFrames before loading them to destinations: ### Multiple Test Definition Sources Define tests in three flexible ways: 1. **Inline code**: Define tests directly in your Python code 2. **From OpenMetadata**: Load test definitions configured in the OpenMetadata UI 3. **From YAML files**: Load test configurations from YAML workflow files ### Comprehensive Test Library Access all test cases supported by OpenMetadata, covering: * **Table tests**: Row counts, column counts, custom SQL queries, table diffs * **Column tests**: Null checks, uniqueness, regex patterns, value ranges, statistical metrics Use Cases --------- ### 1\. ETL Data Validation Validate data after extraction and transformation, before loading: ### 2\. Collaborative Quality Management Data stewards define tests in the UI, engineers run them in pipelines: ### 3\. Chunk-Based Validation Validate large datasets processed in chunks: Getting Started --------------- [Getting Started\ \ Install the SDK and configure authentication to get started.](https://docs.open-metadata.org/latest/how-to-guides/data-quality-observability/quality/data-quality-as-code/getting-started) [TestRunner - Table Testing\ \ Run data quality tests against tables in OpenMetadata.](https://docs.open-metadata.org/latest/how-to-guides/data-quality-observability/quality/data-quality-as-code/test-runner) [DataFrame Validation\ \ Validate pandas DataFrames before loading to destinations.](https://docs.open-metadata.org/latest/how-to-guides/data-quality-observability/quality/data-quality-as-code/dataframe-validation) [Test Definitions Reference\ \ Complete reference of all available test types and their parameters.](https://docs.open-metadata.org/latest/how-to-guides/data-quality-observability/quality/data-quality-as-code/test-definitions) [Advanced Usage\ \ Learn advanced patterns including YAML workflows, custom configurations, and result publishing.](https://docs.open-metadata.org/latest/how-to-guides/data-quality-observability/quality/data-quality-as-code/advanced-usage) [Run our tutorials with examples\ \ Learn by doing by running our Jupyter Notebook examples](https://github.com/open-metadata/OpenMetadata/tree/main/examples/python-sdk/data-quality/README.md) Requirements ------------ * Python 3.10 or higher * `openmetadata-ingestion` package version 1.11.0.0 or later * Access to an OpenMetadata instance (1.11.0 or later) * Valid JWT token for authentication Architecture ------------ Data Quality as Code integrates seamlessly with OpenMetadata's existing data quality infrastructure: 1. **Test Definitions**: Tests can be defined in code, loaded from OpenMetadata, or imported from YAML files 2. **Execution Engine**: Leverages OpenMetadata's proven test execution engine 3. **Result Publishing**: Test results can be published back to OpenMetadata for visualization and alerting 4. **Service Connections**: Automatically uses service connections configured in OpenMetadata Next Steps ---------- Ready to get started? Follow the [Getting Started guide](https://docs.open-metadata.org/latest/how-to-guides/data-quality-observability/quality/data-quality-as-code/getting-started) to install the SDK and run your first data quality test. --- # Explore the Lineage View | Official Documentation We use cookies to improve site navigation, analyze site usage, and enhance your user experience. Click "Accept" to enable cookies or "Reject" to reject cookies. AcceptReject ![](https://static.scarf.sh/a.png?x-pxid=6814879a-96c0-40bd-be69-ddfeb14694aa) how-to-guides No menu items for this category [Home](https://docs.open-metadata.org/latest) /[How To Guides](https://docs.open-metadata.org/latest/how-to-guides) /[Data Lineage](https://docs.open-metadata.org/latest/how-to-guides/data-lineage) /[Explore](https://docs.open-metadata.org/latest/how-to-guides/data-lineage/explore) OpenMetadata Documentation Explore the Lineage View ======================== OpenMetadata UI displays end-to-end lineage traceability for the table and column levels. OpenMetadata supports lineage for Database, Dashboard, and Pipelines. Just search for an data asset and expand the graph to unfold lineage. It’ll display the upstreams and downstreams edges for each node. The lineage details specify the SQL query, pipeline information, and column lineage. In the lineage view, in the example below, the table on the left is the parent or **Source** node. The table on the right is the **Target** node. You can also identify the target node by looking at the arrow attached to it. The arrow connecting the data assets or tables is the **Edge**. Clicking on an edge connecting a source and a destination will display all the edge information: the Source, Target, Description, and SQL Query. It displays the SQL query used to generate the view (The table is of the Type View). The SQL query provides information on how the target table was generated from the source table. ![Edge Information: Source and Target](https://docs.open-metadata.org/images/v1.11/how-to-guides/lineage/edge.png) Edge Information: Source and Target **Tip:** Metadata ingestion also brings in the View Lineage, if the database has views (Data assets of the Type View). You can set up the **Lineage Config** to display the required number of Upstream and Downstream Nodes, as well as the Nodes per layer. You can set up to **3** Upstream and Downstream Nodes. ![Lineage Config](https://docs.open-metadata.org/images/v1.11/how-to-guides/lineage/nodes.png) Lineage Config You can click on the data assets to view the data asset details. * Users can view the Source, Name of the Data Asset, Description, Owner (Team/User details), Tier, and Usage information for the data asset. * Based on the **type of data asset** (Table, Topic, Dashboard, Pipeline, ML Model, Container), the quick preview provides additional information. For example, for `tables`, the type of table, the number of queries, and columns are displayed. * The **data quality and profiler metrics** displays the details on the Tests Passed, Aborted, and Failed. * Users can view all the **tags** associated with the data asset. * The **Schema** provides the details on the column names, type of column, and column description. ![Quick Glance at the Data Asset from Lineage View](https://docs.open-metadata.org/images/v1.11/how-to-guides/lineage/lineage2.png) Quick Glance at the Data Asset from Lineage View Clicking on the tables will display the list of columns and column-level lineage. ![Column-Level Data Lineage in OpenMetadata](https://docs.open-metadata.org/images/v1.11/how-to-guides/lineage/lineage1.png) Column-Level Data Lineage in OpenMetadata In case of **Pipelines**, we first have the lineage ingested from the databases. Further, when setting up the pipeline ingestion, we specify the database service name. That way we display the lineage of the database tables connected via pipelines. If a lineage is created through a pipeline, the same is displayed in the Edge information. ![Database and Pipeline Lineage](https://docs.open-metadata.org/images/v1.11/how-to-guides/lineage/pipeline.png) Database and Pipeline Lineage Similarly for a **Dashboard**, we first have the lineage ingested from the databases. Further, when setting up the dashboard ingestion, the data models and charts are ingested. That way we display the lineage of the database tables connected using the dashboard data models. Lineage Layers -------------- Lineage view supports multiple exploration layers that provide deeper insights into the structure, flow, and quality of data across your ecosystem. These layers help users visualize lineage not just at the dataset level, but also across services, domains, and business-critical data products. ### Column Layer The **Column layer** enables detailed exploration of column-level lineage, allowing users to trace the flow and transformation of specific fields (e.g., customer\_id, first\_name) across tables and pipelines. This granularity helps in understanding data dependencies at the attribute level. ![Column Layer in Lineage](https://docs.open-metadata.org/images/v1.11/how-to-guides/lineage/column-layer.png) Column Layer in Lineage ### Observability Layer The **Observability layer** integrates data quality insights directly into lineage by displaying test outcomes such as passes, failures, and pending checks. This helps users identify potential issues and assess the trustworthiness of data as it moves through the pipeline. ![Observability Layer in Lineage](https://docs.open-metadata.org/images/v1.11/how-to-guides/lineage/observability-layer.png) Observability Layer in Lineage ### Service Layer The **Service layer** visualizes how data flows across different platforms and services like Hive, Redshift, Power BI, and Tableau. It connects ingestion, transformation, and consumption points, offering a system-level view of the end-to-end data journey. ![Service Layer in Lineage](https://docs.open-metadata.org/images/v1.11/how-to-guides/lineage/service-layer.png) Service Layer in Lineage ### Domain Layer The **Domain layer** organizes datasets and assets into business-relevant categories such as β€œEcommerce” or β€œCustomer Data.” This classification provides contextual clarity and supports governance by aligning technical assets with business functions. ![Domain Layer in Lineage](https://docs.open-metadata.org/images/v1.11/how-to-guides/lineage/domain-layer.png) Domain Layer in Lineage ### Data Product Layer The **Data Product layer** highlights curated outputs like _Customer Registry_ or _Superstore_, representing the final, value-delivering datasets within a domain. It enables teams to track the lineage of trusted, consumption-ready data products across the organization. ![Data Product Layer in Lineage](https://docs.open-metadata.org/images/v1.11/how-to-guides/lineage/data-product-layer.png) Data Product Layer in Lineage [How Column-Level Lineage Works\ \ Explore and edit the rich column-level lineage.](https://docs.open-metadata.org/latest/how-to-guides/data-lineage/column) --- # Domo Dashboard Troubleshooting Guide | OpenMetadata Support We use cookies to improve site navigation, analyze site usage, and enhance your user experience. Click "Accept" to enable cookies or "Reject" to reject cookies. AcceptReject connectors No menu items for this category [Home](https://docs.open-metadata.org/latest) /[Connectors](https://docs.open-metadata.org/latest/connectors) /[Dashboard](https://docs.open-metadata.org/latest/connectors/dashboard) /[Domo Dashboard](https://docs.open-metadata.org/latest/connectors/dashboard/domo-dashboard) /[Troubleshooting](https://docs.open-metadata.org/latest/connectors/dashboard/domo-dashboard/troubleshooting) OpenMetadata Documentation Troubleshooting =============== Workflow Deployment Error ------------------------- If there were any errors during the workflow deployment process, the Ingestion Pipeline Entity will still be created, but no workflow will be present in the Ingestion container. * You can then Edit the Ingestion Pipeline and **Deploy** it again. * From the Connection tab, you can also Edit the Service if needed. Connector Debug Troubleshooting ------------------------------- This section provides instructions to help resolve common issues encountered during connector setup and metadata ingestion in OpenMetadata. Below are some of the most frequently observed troubleshooting scenarios. How to Enable Debug Logging for Any Ingestion --------------------------------------------- To enable debug logging for any ingestion workflow in OpenMetadata: 1. **Navigate to Services** Go to **Settings > Services > Service Type** (e.g., Database) in the OpenMetadata UI. 2. **Select a Service** Choose the specific service for which you want to enable debug logging. ![Select a Service](https://docs.open-metadata.org/images/v1.11/connectors/debug/debug1.png) Select a Service 3. **Access Ingestion Tab** Go to the **Ingestion tab** and click the three-dot menu on the right-hand side of the ingestion type, and select Edit. ![Access Agents Tab](https://docs.open-metadata.org/images/v1.11/connectors/debug/debug2.png) Access Agents Tab 4. **Enable Debug Logging** In the configuration dialog, enable the **Debug Log** option and click **Next**. ![Enable Debug Logging](https://docs.open-metadata.org/images/v1.11/connectors/debug/debug3.png) Enable Debug Logging 5. **Schedule and Submit** Configure the schedule if needed and click **Submit** to apply the changes. ![Schedule and Submit](https://docs.open-metadata.org/images/v1.11/connectors/debug/debug4.png) Schedule and Submit Permission Issues ----------------- If you encounter permission-related errors during connector setup or metadata ingestion, ensure that all the prerequisites and access configurations specified for each connector are properly implemented. Refer to the connector-specific documentation to verify the required permissions. --- # Glue Pipeline Troubleshooting Guide | OpenMetadata Support We use cookies to improve site navigation, analyze site usage, and enhance your user experience. Click "Accept" to enable cookies or "Reject" to reject cookies. AcceptReject connectors No menu items for this category [Home](https://docs.open-metadata.org/latest) /[Connectors](https://docs.open-metadata.org/latest/connectors) /[Pipeline](https://docs.open-metadata.org/latest/connectors/pipeline) /[Glue Pipeline](https://docs.open-metadata.org/latest/connectors/pipeline/glue-pipeline) /[Troubleshooting](https://docs.open-metadata.org/latest/connectors/pipeline/glue-pipeline/troubleshooting) OpenMetadata Documentation Troubleshooting =============== Workflow Deployment Error ------------------------- If there were any errors during the workflow deployment process, the Ingestion Pipeline Entity will still be created, but no workflow will be present in the Ingestion container. * You can then Edit the Ingestion Pipeline and **Deploy** it again. * From the Connection tab, you can also Edit the Service if needed. Connector Debug Troubleshooting ------------------------------- This section provides instructions to help resolve common issues encountered during connector setup and metadata ingestion in OpenMetadata. Below are some of the most frequently observed troubleshooting scenarios. How to Enable Debug Logging for Any Ingestion --------------------------------------------- To enable debug logging for any ingestion workflow in OpenMetadata: 1. **Navigate to Services** Go to **Settings > Services > Service Type** (e.g., Database) in the OpenMetadata UI. 2. **Select a Service** Choose the specific service for which you want to enable debug logging. ![Select a Service](https://docs.open-metadata.org/images/v1.11/connectors/debug/debug1.png) Select a Service 3. **Access Ingestion Tab** Go to the **Ingestion tab** and click the three-dot menu on the right-hand side of the ingestion type, and select Edit. ![Access Agents Tab](https://docs.open-metadata.org/images/v1.11/connectors/debug/debug2.png) Access Agents Tab 4. **Enable Debug Logging** In the configuration dialog, enable the **Debug Log** option and click **Next**. ![Enable Debug Logging](https://docs.open-metadata.org/images/v1.11/connectors/debug/debug3.png) Enable Debug Logging 5. **Schedule and Submit** Configure the schedule if needed and click **Submit** to apply the changes. ![Schedule and Submit](https://docs.open-metadata.org/images/v1.11/connectors/debug/debug4.png) Schedule and Submit Permission Issues ----------------- If you encounter permission-related errors during connector setup or metadata ingestion, ensure that all the prerequisites and access configurations specified for each connector are properly implemented. Refer to the connector-specific documentation to verify the required permissions. --- # How to Write and Deploy No-Code Test Cases We use cookies to improve site navigation, analyze site usage, and enhance your user experience. Click "Accept" to enable cookies or "Reject" to reject cookies. AcceptReject ![](https://static.scarf.sh/a.png?x-pxid=6814879a-96c0-40bd-be69-ddfeb14694aa) how-to-guides No menu items for this category [Home](https://docs.open-metadata.org/latest) /[How To Guides](https://docs.open-metadata.org/latest/how-to-guides) /[Data Quality Observability](https://docs.open-metadata.org/latest/how-to-guides/data-quality-observability) /[Quality](https://docs.open-metadata.org/latest/how-to-guides/data-quality-observability/quality) /[Test](https://docs.open-metadata.org/latest/how-to-guides/data-quality-observability/quality/test) OpenMetadata Documentation How to Write and Deploy No-Code Test Cases ========================================== OpenMetadata supports data quality tests at the table and column level on all of the supported database connectors. OpenMetadata supports both business-oriented tests as well as data engineering tests. The data engineering tests are more on the technical side to ascertain a sanity check on the data. It ensures that your data meets the technical definition of the data assets, like the columns are not null, columns are unique, etc. There is no need to fill a YAML file or a JSON config file to set up data quality tests in OpenMetadata. You can simply select the options and add in the details right from the UI to set up test cases. To create a test in OpenMetadata: * Navigate to the table you would like to create a test for. Click on the **Data Observability** tab. * Click on **Add Test** to select a `Table` or `Column` level test. ![Write and Deploy No-Code Test Cases](https://docs.open-metadata.org/images/v1.11/how-to-guides/quality/test1.png) Write and Deploy No-Code Test Cases Table Level Test ---------------- To create a **Table Level Test** enter the following details: * **Name:** Add a name that best defines your test case. * **Test Type:** Based on the test type, you will have further fields to define your test. * **Description:** Describe the test case. Click on **Submit** to set up a test. OpenMetadata currently supports the following table level test types: 1. Table Column Count to be Between: Define the Min. and Max. 2. Table Column Count to Equal: Define a number. 3. Table Column Name to Exist: Define a column name. 4. Table Column Names to Match Set: Add comma separated column names to match. You can also verify if the column names are in order. 5. Custom SQL Query: Define a SQL expression. Select a strategy if it should apply for Rows or for Count. Define a threshold to determine if the test passes or fails. 6. Table Row Count to be Between: Define the Min. and Max. 7. Table Row Count to Equal: Define a number. 8. Table Row Inserted Count to be Between: Define the Min. and Max. row count. This test will work for columns whose values are of the type Timestamp, Date, and Date Time field. Specify the range type in terms of Hour, Day, Month, or Year. Define the interval based on the range type selected. 9. Compare 2 Tables for Differences: Compare 2 tables for differences. Allows a user to check for integrity. 10. Table Data to Be Fresh: Validate the freshness of a table's data (Collate). ![Configure a Table Level Test](https://docs.open-metadata.org/images/v1.11/how-to-guides/quality/test4.png) Configure a Table Level Test Column Level Test ----------------- To create a **Column Level Test** enter the following details: * **Column:** Select a column. On the right hand side, you can view some context about that column. * **Name:** Add a name that best defines your test case. * **Test Type:** Based on the test type, you will have further fields to define your test. * **Description:** Describe the test case. Click on **Submit** to set up a test. OpenMetadata currently supports the following column level test types: 1. Column Value Lengths to be Between: Define the Min. and Max. 2. Column Value Max. to be Between: Define the Min. and Max. 3. Column Value Mean to be Between: Define the Min. and Max. 4. Column Value Median to be Between: Define the Min. and Max. 5. Column Value Min. to be Between: Define the Min. and Max. 6. Column Values Missing Count: Define the number of missing values. You can also match all null and empty values as missing. You can also configure additional missing strings like N/A. 7. Column Values Sum to be Between: Define the Min. and Max. 8. Column Value Std Dev to be Between: Define the Min. and Max. 9. Column Values to be Between: Define the Min. and Max. 10. Column Values to be in Set: You can add an array of allowed values. 11. Column Values to be Not in Set: You can add an array of forbidden values. 12. Column Values to be Not Null 13. Column Values to be Unique 14. Column Values to Match Regex Pattern: Define the regular expression that the column entries should match. 15. Column Values to Not Match Regex: Define the regular expression that the column entries should not match. All column-level tests also support **dimensional validation**, which allows you to group test results by business dimension columns. Instead of getting a single pass/fail result for your entire column, you can see results segmented by categories like region, product type, customer segment, or any other dimension that matters to your business. For example, when testing email completeness, dimensional validation can show you that North America has 99% complete emails while Asia Pacific has only 88%, helping you quickly identify which business segments need attention. To create a dimensional test, select the **Dimension Level** option when adding a new test. For complete guidance on when to use or not dimensional validation, performance considerations, and best practices, see the [Dimensional Validation guide](https://docs.open-metadata.org/latest/how-to-guides/data-quality-observability/quality/dimensional-validation) . ![Configure a Column Level Test](https://docs.open-metadata.org/images/v1.11/how-to-guides/quality/test2.png) Configure a Column Level Test Once the test has been created, you can view the test suite. The test case will be displayed in the Data Quality tab. You can also edit the Display Name and Description for the test. ![Column Level Test Created](https://docs.open-metadata.org/images/v1.11/how-to-guides/quality/test3.png) Column Level Test Created A pipeline can be set up for the tests to run at a regular cadence. * Click on the `Pipeline` tab * Add a pipeline ![Set up a Pipeline](https://docs.open-metadata.org/images/v1.11/how-to-guides/quality/test5.png) Set up a Pipeline * Set up the scheduler for the desired frequency. The timezone is in UTC. * Click on **Submit**. ![Schedule the Pipeline](https://docs.open-metadata.org/images/v1.11/how-to-guides/quality/test6.png) Schedule the Pipeline The pipeline has been set up and will run at the scheduled time. ![Pipeline Scheduled](https://docs.open-metadata.org/images/v1.11/how-to-guides/quality/test7.png) Pipeline Scheduled The tests will be run and the results will be updated in the Data Quality tab. ![Data Quality Tests](https://docs.open-metadata.org/images/v1.11/how-to-guides/quality/test8.png) Data Quality Tests If a **test fails**, you can **Edit the Test Status** to New, Acknowledged, or Resolved status by clicking on the Status icon. ![Failed Test: Edit Status](https://docs.open-metadata.org/images/v1.11/how-to-guides/quality/test9.png) Failed Test: Edit Status * Select the Test Status ![Edit Test Status](https://docs.open-metadata.org/images/v1.11/how-to-guides/quality/test10.png) Edit Test Status * If you are marking the test status as **Resolved**, you must specify the **Reason** for the failure and add a **Comment**. The reasons for failure can be Duplicates, False Positive, Missing Data, Other, or Out of Bounds. * Click on **Submit**. ![Resolved Status: Reason](https://docs.open-metadata.org/images/v1.11/how-to-guides/quality/test11.png) Resolved Status: Reason Users can also set up [alerts](https://docs.open-metadata.org/latest/how-to-guides/data-quality-observability/observability/alerts) to be notified when a test fails. [How to Set Alerts for Test Case Fails\ \ Get notified when a data quality test fails.](https://docs.open-metadata.org/latest/how-to-guides/data-quality-observability/observability/alerts) --- # Run the ingestion from GitHub Actions We use cookies to improve site navigation, analyze site usage, and enhance your user experience. Click "Accept" to enable cookies or "Reject" to reject cookies. AcceptReject deployment No menu items for this category [Home](https://docs.open-metadata.org/latest) /[Deployment](https://docs.open-metadata.org/latest/deployment) /[Ingestion](https://docs.open-metadata.org/latest/deployment/ingestion) /[External](https://docs.open-metadata.org/latest/deployment/ingestion/external) /[Github Actions](https://docs.open-metadata.org/latest/deployment/ingestion/external/github-actions) OpenMetadata Documentation This page is about running the Ingestion Framework **externally**! There are mainly 2 ways of running the ingestion: 1. Internally, by managing the workflows from OpenMetadata. 2. Externally, by using any other tool capable of running Python code. If you are looking for how to manage the ingestion process from OpenMetadata, you can follow this [doc](https://docs.open-metadata.org/latest/deployment/ingestion/openmetadata) . Run the ingestion from GitHub Actions ===================================== You can find a fully working demo of this setup [here](https://github.com/open-metadata/openmetadata-demo/tree/main/ingestion-github-actions) . The process to run the ingestion from GitHub Actions is the same as running it from anywhere else. 1. Get the YAML configuration, 2. Prepare the Python Script 3. Schedule the Ingestion 1\. YAML Configuration ---------------------- For any connector and workflow, you can pick it up from its doc [page](https://docs.open-metadata.org/latest/connectors) . 2\. Prepare the Python Script ----------------------------- In the GitHub Action we will just be triggering a custom Python script. This script will: * Load the secrets from environment variables (we don't want any security risks!), * Prepare the Workflow class from the Ingestion Framework that contains all the logic on how to run the metadata ingestion, * Execute the workflow and log the results. * A simplified version of such script looks like follows: Note how we are securing the credentials using environment variables. You will need to create these env vars in your GitHub repository. Follow the GitHub [docs](https://docs.github.com/en/actions/security-guides/encrypted-secrets) for more information on how to create and use Secrets. In the end, we'll map these secrets to environment variables in the process, that we can pick up with `os.getenv`. 3\. Schedule the Ingestion -------------------------- Now that we have all the ingredients, we just need to build a simple GitHub Actions with the following steps: * Install Python * Prepare virtual environment with the openmetadata-ingestion package * Run the script! * It is as simple as this. Internally the function run we created will be sending the results to the OpenMetadata server, so there's nothing else we need to do here. A first version of the action could be: \[Optional\] - Getting Alerts in Slack -------------------------------------- A very interesting option that GitHub Actions provide is the ability to get alerts in Slack after our action fails. This can become specially useful if we want to be notified when our metadata ingestion is not working as expected. We can use the same setup as above with a couple of slight changes: We have: * Marked the `Run Ingestion` step with a specific `id` and with `continue-on-error: true`. If anything happens, we don't want the action to stop. * We added a step with `slackapi/slack-github-action@v1.23.0`. By passing a Slack Webhook link via a secret, we can send any payload to a * specific Slack channel. You can find more info on how to set up a Slack Webhook [here](https://api.slack.com/messaging/webhooks) . * If our `ingestion` step fails, we still want to mark the action as failed, so we are forcing the failure we skipped before. --- # Custom Tests | OpenMetadata Quality Testing Guide We use cookies to improve site navigation, analyze site usage, and enhance your user experience. Click "Accept" to enable cookies or "Reject" to reject cookies. AcceptReject ![](https://static.scarf.sh/a.png?x-pxid=6814879a-96c0-40bd-be69-ddfeb14694aa) how-to-guides No menu items for this category [Home](https://docs.open-metadata.org/latest) /[How To Guides](https://docs.open-metadata.org/latest/how-to-guides) /[Data Quality Observability](https://docs.open-metadata.org/latest/how-to-guides/data-quality-observability) /[Quality](https://docs.open-metadata.org/latest/how-to-guides/data-quality-observability/quality) /[Custom Tests](https://docs.open-metadata.org/latest/how-to-guides/data-quality-observability/quality/custom-tests) OpenMetadata Documentation Adding Custom Tests =================== While OpenMetadata provides out of the box tests, you may want to write your test results from your own custom quality test suite or define your own data quality tests to be ran inside OpenMetadata. This is very easy to do using the API and our Python SDK. ### Step 1: Creating a `TestDefinition` First, you'll need to create a Test Definition for your test. You can use the following endpoint `/api/v1/dataQuality/testDefinition` using a POST protocol to create your Test Definition. You will need to pass the following data in the body your request at minimum. Here is a complete CURL request Make sure to keep the `UUID` from the response as you will need it to create the Test Case. **Important:** If you want to have the test definition available through OpenMetadata UI, `testPlatforms` need to include `OpenMetadata`. It will also require extra work that we'll cover below in Step 5. If you are just looking to create new test definition executable through OpenMetadata UI, you can skip ahead to Step 5. ### Step 2: Creating a `TestSuite` You'll also need to create a Test Suite for your Test Case -- note that you can also use an existing one if you want to. You can use the following endpoint `/api/v1/dataQuality/testSuites/executable` using a POST protocol to create your Test Definition. You will need to pass the following data in the body your request at minimum. Here is a complete CURL request Make sure to keep the `UUID` from the response as you will need it to create the Test Case. ### Step 3: Creating a `TestCase` Once you have your Test Definition created you can create a Test Case -- which is a specification of your Test Definition. You can use the following endpoint `/api/v1/dataQuality/testCases` using a POST protocol to create your Test Case. You will need to pass the following data in the body your request at minimum. **Important:** for `entityLink` make sure to include the starting and ending `<>` Here is a complete CURL request Make sure to keep the `UUID` from the response as you will need it to create the Test Case. ### Step 4: Writing `TestCaseResults` (Optional - if not executing test case through OpenMetadata UI) Once you have your Test Case created you can write your results to it. You can use the following endpoint `/api/v1/dataQuality/testCases/{test FQN}/testCaseResult` using a PUT protocol to add Test Case Results. You will need to pass the following data in the body your request at minimum. Here is a complete CURL request You will now be able to see your test in the Test Suite or the table entity. ### Step 5: Making Custom Test Case Available Through OpenMetadata UI (Optional) OpenMetadata offers the flexibility to user to create custom test cases that will be executable through the user interface. To accomplish our goal, we'll be leveraging OpenMetadata namespace `data_quality` submodule . #### A. Create Your Namespace Package The first in creating your own executable test case is to create a package where you'll be writing the logic to process the tests. Your package should have a minimum the below structure To add table and column level test cases to SQLAlchemy sources you will place your test respectively in: * `metadata/data_quality/validations/table/sqlalchemy/.py` * `metadata/data_quality/validations/column/sqlalchemy/.py` `` should match the name of your test definition in Step 1. **Important:** You will need to add an `__init__.py` file in every folder and these `__init__.py` should have the below line #### B. Create your Test Class Once you have created the different, you can add the logic in your `.py` file. You will need to create a class named `Validator` that will inherit from `BaseTestValidator`. If you need to, you can also inherit from `SQAValidatorMixin` -- this will give you access to additional methods out of the box. Once completed, you will simply need to implement the `run_validation` class. This method should return a `TestCaseResult` object. You can find a full implementation [here](https://github.com/open-metadata/openmetadata-demo/tree/main/custom-om-test) where we create an entropy test. #### C. `pip` Install Your Package Once you have completed A) and B) you should only need to `pip install` your package in the environment where openmetadata python SDK is install. ![Create test case](https://docs.open-metadata.org/images/v1.11/features/ingestion/workflows/data-quality/custom-test-definition.png) Create test case ![Create test case](https://docs.open-metadata.org/images/v1.11/features/ingestion/workflows/data-quality/custom-test-result.png) Create test case --- # Configuring Okta Public Authentication | OpenMetadata SSO Setup Guide We use cookies to improve site navigation, analyze site usage, and enhance your user experience. Click "Accept" to enable cookies or "Reject" to reject cookies. AcceptReject ![](https://static.scarf.sh/a.png?x-pxid=8472f2b3-d307-4935-a5ed-3f9d3982c15d) deployment No menu items for this category [Home](https://docs.open-metadata.org/latest) /[Deployment](https://docs.open-metadata.org/latest/deployment) /[Security](https://docs.open-metadata.org/latest/deployment/security) /[Okta](https://docs.open-metadata.org/latest/deployment/security/okta) /[Public Client](https://docs.open-metadata.org/latest/deployment/security/okta/public-client) OpenMetadata Documentation Okta SSO Configuration (Public) =============================== * [Troubleshooting](https://docs.open-metadata.org/latest/deployment/security/okta/public-client#troubleshooting) Okta Single Sign-On (SSO) enables users to log in to OpenMetadata with their **Okta credentials** using **OAuth 2.0** and **OpenID Connect (OIDC)**. This guide explains how to configure the **Public Client** setup for Okta authentication in OpenMetadata. ![Okta SSO Configuration - Public Client](https://docs.open-metadata.org/images/v1.11/deployment/security/okta/okta1.png) Provider Name ------------- A human-readable name for this Okta SSO configuration instance. **Example:** `Okta SSO`, `Company Okta`, `Corporate Identity` **Why it matters:** Helps identify this SSO configuration in logs and the OpenMetadata user interface. **Note:** This is a display name and does not impact authentication functionality. Authentication Configuration (Public) ------------------------------------- ### Client Type Defines whether the application is **public** (no client secret) or **confidential** (requires client secret). **Options:** Public | Confidential **Example:** Public **Why it matters:** Determines the authentication flow and security level. **Note:** * Choose **Public** for frontend or browser-based applications. * Choose **Confidential** for backend services or web apps. * Okta typically uses **Confidential** type, but **Public** is suitable for deployments without a client secret. ### Callback URL Redirect URI where Okta sends authentication responses after successful login. **Example:** `https://yourapp.company.com/callback` **Why it matters:** This must exactly match the **Sign-in redirect URI** configured in your Okta application. **Note:** * Configure under **Okta β†’ Applications β†’ Your App β†’ General β†’ Sign-in Redirect URIs**. * Always use **HTTPS** for production environments. ### Enable Self Signup Allows users to automatically create OpenMetadata accounts on their first login through Okta. **Options:** Enabled | Disabled **Example:** Enabled **Why it matters:** Controls whether new users can self-register or require admin approval. **Note:** Disable for stricter user access management policies. ### Authority Specifies the Okta domain responsible for issuing authentication tokens. **Example:** `https://dev-123456.okta.com` or `https://company.okta.com` **Why it matters:** Informs OpenMetadata which Okta tenant to authenticate users against. **Note:** Use the complete Okta domain URL (including the `https://` prefix). ### Public Key URLs A list of URLs where Okta publishes its **public keys** used to verify JWT token signatures. **Example:** `["https://dev-123456.okta.com/oauth2/v1/keys"]` **Why it matters:** OpenMetadata uses these keys to validate incoming tokens. **Note:** These URLs are typically auto-discovered from the OIDC discovery URI and rarely require manual configuration. ### Token Validation Algorithm Defines the algorithm used to verify the JWT token signatures from Okta. **Options:** RS256 | RS384 | RS512 **Default:** RS256 **Example:** RS256 **Why it matters:** Ensures the tokens are validated using the correct signing algorithm. **Note:** Okta typically uses **RS256**. ### JWT Principal Claims Specifies which JWT claims identify the authenticated user. **Example:** `["preferred_username", "email", "sub"]` **Why it matters:** Determines how OpenMetadata identifies unique users during authentication. **Note:** Common Okta claims include `email`, `preferred_username`, `sub`, and `login`. ### JWT Principal Claims Mapping Maps JWT claims from Okta to OpenMetadata user attributes. **Example:** `["email:email", "name:name", "firstName:given_name"]` **Why it matters:** Controls how user data from Okta is represented in OpenMetadata profiles. **Note:** Use the format `"openmetadata_field:jwt_claim"` (e.g., `email:email`). ### Admin Principals Specifies a list of user principals who have **administrative privileges** in OpenMetadata. **Example:** `["admin@company.com", "superuser@company.com"]` **Why it matters:** Grants full platform access to designated users. **Note:** Ensure these match the `email` or `preferred_username` values from Okta tokens. ### Principal Domain Defines the **default domain** used when constructing user principal names. **Example:** `company.com` **Why it matters:** Helps form complete usernames if only local parts (before @) are provided. **Note:** Typically corresponds to your organization’s primary domain. ### Enforce Principal Domain Indicates whether all users must belong to the defined **Principal Domain**. **Default:** false **Example:** true **Why it matters:** Adds a layer of security by ensuring only users from approved domains can log in. **Note:** Useful for multi-tenant Okta setups where access should be limited to a specific organization. ### Enable Secure Socket Connection Determines whether to use **SSL/TLS** for secure communication between OpenMetadata and Okta. **Default:** false **Example:** true **Why it matters:** Ensures encrypted data exchange during authentication. **Note:** This setting should always be **enabled in production** environments. Summary ------- | **Field** | **Example / Default** | | --- | --- | | Client Type | Public | | Callback URL | https://yourapp.company.com/callback | | Authority | https://dev-123456.okta.com | | Public Key URLs | https://dev-123456.okta.com/oauth2/v1/keys | | Token Validation Algorithm | RS256 | | JWT Principal Claims | \["preferred\_username", "email", "sub"\] | | JWT Mapping | \["email:email", "name:name", "firstName:given\_name"\] | | Admin Principals | \["admin@company.com"\] | | Principal Domain | company.com | | Enforce Principal Domain | false | | SSL/TLS | true | ### Troubleshooting If users are automatically logged out and unable to log in again due to a bad authentication configuration, you can reset the security setup using the following command: After executing the command, **restart the server**. The authentication values from your YAML or Helm chart will then be reapplied on startup. The following tiles detail how to apply this configuration across Docker, Kubernetes, and Bare Metal deployments: [Docker Security\ \ Configure SSO for your Docker Deployment.](https://docs.open-metadata.org/latest/deployment/security/configuration/docker) [Bare Metal Security\ \ Configure SSO for your Bare Metal Deployment.](https://docs.open-metadata.org/latest/deployment/security/configuration/bare-metal) [Kubernetes Security\ \ Configure SSO for your Kubernetes Deployment.](https://docs.open-metadata.org/latest/deployment/security/configuration/kubernetes) --- # SSO for Docker | Official Documentation We use cookies to improve site navigation, analyze site usage, and enhance your user experience. Click "Accept" to enable cookies or "Reject" to reject cookies. AcceptReject ![](https://static.scarf.sh/a.png?x-pxid=8472f2b3-d307-4935-a5ed-3f9d3982c15d) deployment No menu items for this category [Home](https://docs.open-metadata.org/latest) /[Deployment](https://docs.open-metadata.org/latest/deployment) /[Security](https://docs.open-metadata.org/latest/deployment/security) /[Configuration](https://docs.open-metadata.org/latest/deployment/security/configuration) /[Docker](https://docs.open-metadata.org/latest/deployment/security/configuration/docker) OpenMetadata Documentation SSO for Docker ============== To enable security for the Docker deployment, follow the next steps: 1\. Create an .env file ----------------------- Create an `openmetadata_oidc.env` file and add the following contents as an example. Use the information generated when setting up the account. The configuration values provided below are examples. Update them as required to match your specific environment and authentication settings. 2\. Start Docker ---------------- Configure Ingestion ------------------- Once your server security is set, it's time to review the ingestion configuration. Our bots support JWT tokens to authenticate to the server when sending requests. Find more information on [**Enabling JWT Tokens**](https://docs.open-metadata.org/latest/deployment/security/enable-jwt-tokens) and [**JWT Troubleshooting**](https://docs.open-metadata.org/latest/deployment/security/jwt-troubleshooting) to ensure seamless authentication. --- # Run the ingestion from AWS MWAA | Official Documentation We use cookies to improve site navigation, analyze site usage, and enhance your user experience. Click "Accept" to enable cookies or "Reject" to reject cookies. AcceptReject deployment No menu items for this category [Home](https://docs.open-metadata.org/latest) /[Deployment](https://docs.open-metadata.org/latest/deployment) /[Ingestion](https://docs.open-metadata.org/latest/deployment/ingestion) /[External](https://docs.open-metadata.org/latest/deployment/ingestion/external) /[Mwaa](https://docs.open-metadata.org/latest/deployment/ingestion/external/mwaa) OpenMetadata Documentation This page is about running the Ingestion Framework **externally**! There are mainly 2 ways of running the ingestion: 1. Internally, by managing the workflows from OpenMetadata. 2. Externally, by using any other tool capable of running Python code. If you are looking for how to manage the ingestion process from OpenMetadata, you can follow this [doc](https://docs.open-metadata.org/latest/deployment/ingestion/openmetadata) . Run the ingestion from AWS MWAA =============================== When running ingestion workflows from MWAA we have three approaches: 1. Install the openmetadata-ingestion package as a requirement in the Airflow environment. We will then run the process using a `PythonOperator` 2. Configure an ECS cluster and run the ingestion as an `ECSOperator`. 3. Install a plugin and run the ingestion with the `PythonVirtualenvOperator`. We will now discuss pros and cons of each aspect and how to configure them. OpenMetadata does not support using Amazon MWAA (Managed Workflows for Apache Airflow) for internal ingestion. This limitation exists because MWAA does not allow the installation of the `openmetadata-ingestion-rest-apis` plugin, which is required to expose the necessary REST APIs for initiating workflows. Ingestion Workflows as a Python Operator ---------------------------------------- ### PROs * It is the simplest approach * We don’t need to spin up any further infrastructure ### CONs * We need to install the [openmetadata-ingestion](https://pypi.org/project/openmetadata-ingestion/) package in the MWAA environment * The installation can clash with existing libraries * Upgrading the OM version will require to repeat the installation process To install the package, we need to update the `requirements.txt` file from the MWAA environment to add the following line: Where `x.y.z` is the version of the OpenMetadata ingestion package. Note that the version needs to match the server version. If we are using the server at 1.3.1, then the ingestion package needs to also be 1.3.1. The plugin parameter is a list of the sources that we want to ingest. An example would look like this `openmetadata-ingestion[mysql,snowflake,s3]==1.3.1`. A DAG deployed using a Python Operator would then look like follows Where you can update the YAML configuration and workflow classes accordingly. accordingly. Further examples on how to run the ingestion can be found on the documentation (e.g., [Snowflake](https://docs.open-metadata.org/latest/connectors/database/snowflake) ). Ingestion Workflow classes -------------------------- We have different classes for different types of workflows. The logic is always the same, but you will need to change your import path. The rest of the method calls will remain the same. For example, for the `Metadata` workflow we'll use: The classes for each workflow type are: * `Metadata`: `from metadata.workflow.metadata import MetadataWorkflow` * `Lineage`: `from metadata.workflow.metadata import MetadataWorkflow` (same as metadata) * `Usage`: `from metadata.workflow.usage import UsageWorkflow` * `dbt`: `from metadata.workflow.metadata import MetadataWorkflow` * `Profiler`: `from metadata.workflow.profiler import ProfilerWorkflow` * `Data Quality`: `from metadata.workflow.data_quality import TestSuiteWorkflow` * `Data Insights`: `from metadata.workflow.data_insight import DataInsightWorkflow` * `Elasticsearch Reindex`: `from metadata.workflow.metadata import MetadataWorkflow` (same as metadata) Ingestion Workflows as an ECS Operator -------------------------------------- ### PROs * Completely isolated environment * Easy to update each version ### CONs * We need to set up an ECS cluster and the required policies in MWAA to connect to ECS and handle Log Groups. We will now describe the steps, following the official AWS documentation. ### 1\. Create an ECS Cluster & Task Definition * The cluster needs a task to run in `FARGATE` mode. * The required image is `docker.getcollate.io/openmetadata/ingestion-base:x.y.z` * The same logic as above applies. The `x.y.z` version needs to match the server version. For example, `docker.getcollate.io/openmetadata/ingestion-base:1.3.1` We have tested this process with a Task Memory of 512MB and Task CPU (unit) of 256. This can be tuned depending on the amount of metadata that needs to be ingested. When creating the Task Definition, take notes on the **log groups** assigned, as we will need them to prepare the MWAA Executor Role policies. For example, if in the JSON from the Task Definition we see: We'll need to use the `/ecs/openmetadata` below when configuring the policies. ### 2\. Task Definition ARN & Networking 1. From the AWS Console, copy your task definition ARN. It will look something like this `arn:aws:ecs:::task-definition/:`. 2. Get the network details on where the task should execute. We will be using a JSON like: If you want to extract MWAA metadata, add the **VPC**, **subnets** and **security groups** used when setting up MWAA. We need to be in the same network environment as MWAA to reach the underlying database. ### 3\. Update MWAA Executor Role policies * Identify your MWAA executor role. This can be obtained from the details view of your MWAA environment. * Add the following two policies to the role, the first with ECS permissions: And for the Log Group permissions Note how you need to replace the `region`, `account-id` and the `log group` names for your Airflow Environment and ECS. ### 4\. Prepare the DAG A DAG created using the ECS Operator will then look like this: Note that depending on the kind of workflow you will be deploying, the YAML configuration will need to updated following the official OpenMetadata docs, and the value of the `pipelineType` configuration will need to hold one of the following values: * `metadata` * `usage` * `lineage` * `profiler` * `TestSuite` Which are based on the `PipelineType` [JSON Schema definitions](https://github.com/open-metadata/OpenMetadata/blob/main/openmetadata-spec/src/main/resources/json/schema/entity/services/ingestionPipelines/ingestionPipeline.json#L14) Moreover, one of the imports will depend on the MWAA Airflow version you are using: * If using Airflow < 2.5: `from airflow.providers.amazon.aws.operators.ecs import ECSOperator` * If using Airflow > 2.5: `from airflow.providers.amazon.aws.operators.ecs import EcsRunTaskOperator` Make sure to update the `ecs_operator_task` task call accordingly. Ingestion Workflows as a Python Virtualenv Operator --------------------------------------------------- ### PROs * Installation does not clash with existing libraries * Simpler than ECS ### CONs * We need to install an additional plugin in MWAA * DAGs take longer to run due to needing to set up the virtualenv from scratch for each run. We need to update the `requirements.txt` file from the MWAA environment to add the following line: Then, we need to set up a custom plugin in MWAA. Create a file named virtual\_python\_plugin.py. Note that you may need to update the python version (eg, python3.7 -> python3.10) depending on what your MWAA environment is running. This is modified from the [AWS sample](https://docs.aws.amazon.com/mwaa/latest/userguide/samples-virtualenv.html) . Next, create the plugins.zip file and upload it according to [AWS docs](https://docs.aws.amazon.com/mwaa/latest/userguide/configuring-dag-import-plugins.html) . You will also need to [disable lazy plugin loading in MWAA](https://docs.aws.amazon.com/mwaa/latest/userguide/samples-virtualenv.html#samples-virtualenv-airflow-config) . A DAG deployed using the PythonVirtualenvOperator would then look like: Where you can update the YAML configuration and workflow classes accordingly. accordingly. Further examples on how to run the ingestion can be found on the documentation (e.g., [Snowflake](https://docs.open-metadata.org/latest/connectors/database/snowflake) ). You will also need to determine the OpenMetadata ingestion extras and Airflow providers you need. Note that the Openmetadata version needs to match the server version. If we are using the server at 0.12.2, then the ingestion package needs to also be 0.12.2. An example of the extras would look like this `openmetadata-ingestion[mysql,snowflake,s3]==0.12.2.2`. For Airflow providers, you will want to pull the provider versions from [the matching constraints file](https://raw.githubusercontent.com/apache/airflow/constraints-2.4.3/constraints-3.7.txt) . Since this example installs Airflow Providers v2.4.3 on Python 3.7, we use that constraints file. Also note that the ingestion workflow function must be entirely self-contained as it will run by itself in the virtualenv. Any imports it needs, including the configuration, must exist within the function itself. Ingestion Workflow classes -------------------------- We have different classes for different types of workflows. The logic is always the same, but you will need to change your import path. The rest of the method calls will remain the same. For example, for the `Metadata` workflow we'll use: The classes for each workflow type are: * `Metadata`: `from metadata.workflow.metadata import MetadataWorkflow` * `Lineage`: `from metadata.workflow.metadata import MetadataWorkflow` (same as metadata) * `Usage`: `from metadata.workflow.usage import UsageWorkflow` * `dbt`: `from metadata.workflow.metadata import MetadataWorkflow` * `Profiler`: `from metadata.workflow.profiler import ProfilerWorkflow` * `Data Quality`: `from metadata.workflow.data_quality import TestSuiteWorkflow` * `Data Insights`: `from metadata.workflow.data_insight import DataInsightWorkflow` * `Elasticsearch Reindex`: `from metadata.workflow.metadata import MetadataWorkflow` (same as metadata) --- # SSO for Kubernetes | Official Documentation We use cookies to improve site navigation, analyze site usage, and enhance your user experience. Click "Accept" to enable cookies or "Reject" to reject cookies. AcceptReject ![](https://static.scarf.sh/a.png?x-pxid=8472f2b3-d307-4935-a5ed-3f9d3982c15d) deployment No menu items for this category [Home](https://docs.open-metadata.org/latest) /[Deployment](https://docs.open-metadata.org/latest/deployment) /[Security](https://docs.open-metadata.org/latest/deployment/security) /[Configuration](https://docs.open-metadata.org/latest/deployment/security/configuration) /[Kubernetes](https://docs.open-metadata.org/latest/deployment/security/configuration/kubernetes) OpenMetadata Documentation SSO for Kubernetes ================== Check the Helm information [here](https://artifacthub.io/packages/search?repo=open-metadata) . Once the `Client Id` is generated, see the snippet below for an example of where to place the client id value and update the authorizer configurations in the `values.yaml`. The configuration values provided below are examples. Update them as required to match your specific environment and authentication settings. Configure Ingestion ------------------- Once your server security is set, it's time to review the ingestion configuration. Our bots support JWT tokens to authenticate to the server when sending requests. Find more information on [**Enabling JWT Tokens**](https://docs.open-metadata.org/latest/deployment/security/enable-jwt-tokens) and [**JWT Troubleshooting**](https://docs.open-metadata.org/latest/deployment/security/jwt-troubleshooting) to ensure seamless authentication. --- # Data Lineage | OpenMetadata Lineage How-To Guide We use cookies to improve site navigation, analyze site usage, and enhance your user experience. Click "Accept" to enable cookies or "Reject" to reject cookies. AcceptReject ![](https://static.scarf.sh/a.png?x-pxid=6814879a-96c0-40bd-be69-ddfeb14694aa) how-to-guides No menu items for this category [Home](https://docs.open-metadata.org/latest) /[How To Guides](https://docs.open-metadata.org/latest/how-to-guides) /[Data Lineage](https://docs.open-metadata.org/latest/how-to-guides/data-lineage) OpenMetadata Documentation Overview of Data Lineage ======================== OpenMetadata tracks data lineage, showing how data moves through the organization's systems. Users can visualize how data is transformed and where it is used, helping with data traceability and impact analysis. OpenMetadata supports lineage for Database, Dashboard, and Pipelines. ![Data Lineage in OpenMetadata](https://docs.open-metadata.org/images/v1.11/how-to-guides/lineage/lineage1.png) Data Lineage in OpenMetadata Watch the video on data lineage to understand the different options to automatically extract the lineage from your data warehouses such as Snowflake, dashboard service like metabase. Also learn about creating lineage programmatically with python SDK. [Lineage Workflow\ \ Configure a lineage workflow right from the UI.](https://docs.open-metadata.org/latest/how-to-guides/data-lineage/workflow) [Explore Lineage\ \ Explore the rich lineage view in OpenMetadata.](https://docs.open-metadata.org/latest/how-to-guides/data-lineage/explore) [Column-Level Lineage\ \ Explore and edit the rich column-level lineage.](https://docs.open-metadata.org/latest/how-to-guides/data-lineage/column) [Manual Lineage\ \ Edit the table and column level lineage manually.](https://docs.open-metadata.org/latest/how-to-guides/data-lineage/manual) --- # Auth0 SSO Configuration for Confidential Apps We use cookies to improve site navigation, analyze site usage, and enhance your user experience. Click "Accept" to enable cookies or "Reject" to reject cookies. AcceptReject ![](https://static.scarf.sh/a.png?x-pxid=8472f2b3-d307-4935-a5ed-3f9d3982c15d) deployment No menu items for this category [Home](https://docs.open-metadata.org/latest) /[Deployment](https://docs.open-metadata.org/latest/deployment) /[Security](https://docs.open-metadata.org/latest/deployment/security) /[Auth 0](https://docs.open-metadata.org/latest/deployment/security/auth0) /[Confidential Client](https://docs.open-metadata.org/latest/deployment/security/auth0/confidential-client) OpenMetadata Documentation Auth0 SSO Configuration (Confidential) ====================================== * [Troubleshooting](https://docs.open-metadata.org/latest/deployment/security/auth0/confidential-client#troubleshooting) Auth0 SSO enables users to log in to OpenMetadata using their Auth0 credentials via OAuthβ€―2.0 and OpenID Connect (OIDC). This configuration supports **Confidential Clients** using both Clientβ€―ID and Clientβ€―Secret for secure backend authentication. Confidential Configuration Fields --------------------------------- ![Auth0 SSO Configuration - Confidential Client](https://docs.open-metadata.org/images/v1.11/deployment/security/auth0/auth02.png) ### Provider Name * **Definition:** A human‑readable name for this Auth0 SSO configuration instance. * **Example:** `Auth0 SSO`, `Company Auth0`, `Custom Identity Provider` * **Why it matters:** Helps identify this specific SSO configuration in logs and interfaces. * **Note:** Display only; does not impact authentication logic. ### Client Type * **Definition:** Whether the application is Public (no secret) or Confidential (requires a secret). * **Options:** Public | Confidential * **Example:** Confidential * **Why it matters:** Determines security level and authentication flow. Confidential clients can securely store secrets. * **Note:** * Use **Public** for simple or frontend‑only apps * Use **Confidential** for backend services or web applications * Auth0 typically uses Confidential client type ### Enable Self Signup * **Definition:** Allows users to automatically create accounts on first login. * **Options:** Enabled | Disabled * **Example:** Enabled * **Why it matters:** Controls whether new users from Auth0 can join automatically or require approval. * **Note:** Disable for tighter control of access. ### Authority * **Definition:** The Auth0 domain endpoint that issues tokens for your tenant. * **Example:** `https://dev‑abc123.us.auth0.com/your‑auth0‑domain` * **Why it matters:** Tells OpenMetadata which Auth0 tenant to authenticate against. * **Note:** * Replace `your‑auth0‑domain` with your Auth0 tenant ID * For multi‑tenant use, you may use `common` ### Public Key URLs * **Definition:** List of URLs where Auth0 publishes public keys for token verification. * **Example:** `["https://dev‑abc123.us.auth0.com/common/discovery/v2.0/keys"]` * **Why it matters:** Used to verify JWT token signatures from Auth0. * **Note:** Usually auto‑discovered from the discovery URI; manual configuration rarely needed. ### Token Validation Algorithm * **Definition:** Algorithm used to validate JWT token signatures. * **Options:** RS256 | RS384 | RS512 * **Default:** RS256 * **Example:** RS256 * **Why it matters:** Must match the algorithm used by Auth0 to sign tokens. ### OIDC Client ID * **Definition:** Application (client) ID for OIDC authentication with Auth0. * **Example:** `abc123def456ghi789jkl012mno345pqr` * **Why it matters:** Identifies your application in Auth0 OIDC flows. * **Note:** Same ID shown in Auth0 app registration. ### OIDC Client Secret * **Definition:** Secret key for confidential client authentication with Auth0. * **Example:** `abc123def456ghi789jkl012mno345pqr678st` * **Why it matters:** Required for confidential clients to securely authenticate with Auth0. * **Note:** * Generate in **Auth0 β†’ Applications β†’ Certificates & Secrets** * Store securely and rotate regularly. * Only required for Confidential client type. ### OIDC Request Scopes * **Definition:** Permissions requested from Auth0 during authentication. * **Default:** `openid email profile` * **Example:** `openid email profile User.Read` * **Why it matters:** Determines what user information OpenMetadata can access. * **Note:** Usually `openid email profile` is sufficient. ### OIDC Discovery URI * **Definition:** Auth0’s OpenID Connect metadata endpoint. * **Example:** `https://dev‑abc123.us.auth0.com/your‑auth0‑domain/v2.0/.well‑known/openid‑configuration` * **Why it matters:** Allows OpenMetadata to automatically discover Auth0’s OIDC endpoints. * **Note:** Replace `your‑auth0‑domain` with your actual tenant ID. ### OIDC Use Nonce * **Definition:** Security feature to prevent replay attacks in OIDC flows. * **Default:** true * **Example:** true * **Why it matters:** Ensures each authentication request is unique. * **Note:** Should generally be enabled. ### OIDC Preferred JWS Algorithm * **Definition:** Algorithm used to verify JWT token signatures from Auth0. * **Default:** RS256 * **Example:** RS256 * **Why it matters:** Must match Auth0’s token signing algorithm. ### OIDC Response Type * **Definition:** Type of response expected during authentication. * **Default:** `id_token` * **Options:** `id_token` | `code` * **Example:** `id_token` * **Why it matters:** Determines OAuth flow type (implicit vs authorization code). ### OIDC Disable PKCE * **Definition:** Whether to disable Proof Key for Code Exchange (PKCE). * **Default:** false * **Example:** false * **Why it matters:** PKCE adds security to the authorization code flow. * **Note:** Should typically remain enabled (`false`) for secure flows. ### OIDC Max Clock Skew * **Definition:** Maximum allowed time difference between systems when validating tokens. * **Example:** 0 (seconds) * **Why it matters:** Prevents token validation failures due to minor time differences. ### OIDC Client Authentication Method * **Definition:** Method used to authenticate the client with Auth0. * **Default:** `client_secret_basic` * **Options:** `client_secret_basic` | `client_secret_post` | `client_secret_jwt` | `private_key_jwt` * **Example:** `client_secret_basic` * **Why it matters:** Must match the configuration in your Auth0 app. ### OIDC Token Validity * **Definition:** Duration (in seconds) for which issued tokens remain valid. * **Default:** 0 (use provider default) * **Example:** 3600 (1 hour) * **Why it matters:** Controls token lifetime and session duration. ### OIDC Tenant * **Definition:** Auth0 tenant identifier for multi‑tenant applications. * **Example:** `your‑auth0‑domain` or `company.onmicrosoft.com` * **Why it matters:** Specifies which Auth0 tenant to authenticate against. ### OIDC Server URL * **Definition:** Your OM server URL. * **Example:** `https://yourapp.company.com`. * **Why it matters:** specifies the URL at which OM is hosted. ### OIDC Callback URL * **Definition:** Redirect URI for OIDC authentication responses. * **Example:** `https://yourapp.company.com/callback` * **Why it matters:** Must match the redirect URI configured in Auth0. * **Note:** Must be registered in Auth0 app registration. ### OIDC Max Age * **Definition:** Maximum authentication age (in seconds) before re‑authentication is required. * **Example:** 3600 * **Why it matters:** Controls how often users must re‑authenticate. ### OIDC Prompt * **Definition:** Controls authentication prompts shown to users. * **Options:** `none` | `login` | `consent` | `select_account` * **Example:** `select_account` * **Why it matters:** Affects user experience during login. ### OIDC Session Expiry * **Definition:** How long user sessions remain valid (in seconds). * **Default:** 604800 (7 days) * **Example:** 604800 * **Why it matters:** Controls how often users need to sign in. ### JWT Principal Claims * **Definition:** JWT claims used to identify the user principal. * **Example:** `["preferred_username", "email", "sub"]` * **Why it matters:** Determines which claim from the token identifies the user. ### JWT Principal Claims Mapping * **Definition:** Maps JWT claims to OpenMetadata user attributes. * **Example:** `["email:email", "name:displayName", "firstName:given_name"]` * **Why it matters:** Controls how user information from Auth0 maps to OpenMetadata profiles. * **Note:** Format: `"openmetadata_field:jwt_claim"` ### Admin Principals * **Definition:** List of user principals who will have admin access. * **Example:** `["admin@company.com", "superuser@company.com"]` * **Why it matters:** These users will have full administrative privileges. * **Note:** Use email addresses or UPNs matching JWT principal claims. ### Principal Domain * **Definition:** Default domain for user principals. * **Example:** `company.com` * **Why it matters:** Used to build full user principal names when only username is provided. ### Enforce Principal Domain * **Definition:** Whether to enforce that all users belong to the principal domain. * **Default:** false * **Example:** true * **Why it matters:** Adds a layer of security by restricting access to a specific domain. ### Enable Secure Socket Connection * **Definition:** Whether to use SSL/TLS for secure connections. * **Default:** false * **Example:** true * **Why it matters:** Ensures encrypted communication for secure authentication flows. * **Note:** Should be enabled in production environments. Summary Table ------------- | **Field** | **Example / Default** | | --- | --- | | Provider Name | Auth0 SSO | | Client Type | Confidential | | Client ID | abc123def456ghi789jkl012mno345pqr | | Client Secret | (hidden) | | Callback URL | https://yourapp.company.com/callback | | Authority | https://dev‑abc123.us.auth0.com | | Public Key URLs | https://dev‑abc123.us.auth0.com/.well‑known/jwks.json | | Token Validation Algorithm | RS256 | | JWT Principal Claims | \["preferred\_username", "email", "sub"\] | | JWT Mapping | \["email:email", "name:displayName", "firstName:given\_name"\] | | Admin Principals | \["admin@company.com"\] | | Principal Domain | company.com | | Enforce Principal Domain | false | | SSL/TLS | true | ### Troubleshooting If users are automatically logged out and unable to log in again due to a bad authentication configuration, you can reset the security setup using the following command: After executing the command, **restart the server**. The authentication values from your YAML or Helm chart will then be reapplied on startup. The following tiles detail how to apply this configuration across Docker, Kubernetes, and Bare Metal deployments: [Docker Security\ \ Configure SSO for your Docker Deployment.](https://docs.open-metadata.org/latest/deployment/security/configuration/docker) [Bare Metal Security\ \ Configure SSO for your Bare Metal Deployment.](https://docs.open-metadata.org/latest/deployment/security/configuration/bare-metal) [Kubernetes Security\ \ Configure SSO for your Kubernetes Deployment.](https://docs.open-metadata.org/latest/deployment/security/configuration/kubernetes) --- # Run the Domo Pipeline Connector Externally We use cookies to improve site navigation, analyze site usage, and enhance your user experience. Click "Accept" to enable cookies or "Reject" to reject cookies. AcceptReject connectors No menu items for this category [Home](https://docs.open-metadata.org/latest) /[Connectors](https://docs.open-metadata.org/latest/connectors) /[Pipeline](https://docs.open-metadata.org/latest/connectors/pipeline) /[Domo Pipeline](https://docs.open-metadata.org/latest/connectors/pipeline/domo-pipeline) /[Yaml](https://docs.open-metadata.org/latest/connectors/pipeline/domo-pipeline/yaml) OpenMetadata Documentation ![Domo](https://docs.open-metadata.org/_next/image?url=%2Fimages%2Fconnectors%2Fdomo.webp&w=64&q=75) Domo ==== PROD Available In Feature List Pipelines Pipeline Status Lineage Usage Owners Tags In this section, we provide guides and references to use the Domo Pipeline connector. Configure and schedule Domo Pipeline metadata and profiler workflows from the OpenMetadata UI: * [Requirements](https://docs.open-metadata.org/latest/connectors/pipeline/domo-pipeline/yaml#requirements) * [Metadata Ingestion](https://docs.open-metadata.org/latest/connectors/pipeline/domo-pipeline/yaml#metadata-ingestion) How to Run the Connector Externally ----------------------------------- To run the Ingestion via the UI you'll need to use the OpenMetadata Ingestion Container, which comes shipped with custom Airflow plugins to handle the workflow deployment. If, instead, you want to manage your workflows externally on your preferred orchestrator, you can check the following docs to run the Ingestion Framework **anywhere**. [#### External Schedulers\ \ Get more information about running the Ingestion Framework Externally](https://docs.open-metadata.org/latest/deployment/ingestion) Requirements ------------ **Note:** For metadata ingestion, kindly make sure add at least `data` scopes to the clientId provided. Question related to scopes, click [here](https://developer.domo.com/portal/1845fc11bbe5d-api-authentication) . ### Python Requirements We have support for Python versions 3.9-3.11 To run the Domo Pipeline ingestion, you will need to install: Metadata Ingestion ------------------ All connectors are defined as JSON Schemas. [Here](https://github.com/open-metadata/OpenMetadata/blob/main/openmetadata-spec/src/main/resources/json/schema/entity/services/connections/pipeline/airbyteConnection.json) you can find the structure to create a connection to Airbyte. In order to create and run a Metadata Ingestion workflow, we will follow the steps to create a YAML configuration able to connect to the source, process the Entities if needed, and reach the OpenMetadata server. The workflow is modeled around the following [JSON Schema](https://github.com/open-metadata/OpenMetadata/blob/main/openmetadata-spec/src/main/resources/json/schema/metadataIngestion/workflow.json) ### 1\. Define the YAML Config This is a sample config for Domo-Pipeline: #### Source Configuration - Service Connection **Client ID**: Client ID to Connect to DOMO Pipeline. **Secret Token**: Secret Token to Connect DOMO Pipeline. **Access Token**: Access to Connect to DOMO Pipeline. **API Host**: API Host to Connect to DOMO Pipeline instance. **Instance Domain**: URL to connect to your Domo instance UI. For example `https://.domo.com`. #### Source Configuration - Source Config The `sourceConfig` is defined [here](https://github.com/open-metadata/OpenMetadata/blob/main/openmetadata-spec/src/main/resources/json/schema/metadataIngestion/pipelineServiceMetadataPipeline.json) : * **dbServiceNames**: Database Service Name for the creation of lineage, if the source supports it. * **includeTags**: Set the 'Include Tags' toggle to control whether to include tags as part of metadata ingestion. * **includeUnDeployedPipelines**: Set the 'Include UnDeployed Pipelines' toggle to control whether to include un-deployed pipelines as part of metadata ingestion. By default it is set to `true` * **markDeletedPipelines**: Set the Mark Deleted Pipelines toggle to flag pipelines as soft-deleted if they are not present anymore in the source system. * **pipelineFilterPattern** and **chartFilterPattern**: Note that the `pipelineFilterPattern` and `chartFilterPattern` both support regex as include or exclude. * **includeOwners**: Set the 'Include Owners' toggle to control whether to include owners to the ingested entity if the owner email matches with a user stored in the OM server as part of metadata ingestion. If the ingested entity already exists and has an owner, the owner will not be overwritten.It supports boolean values either `true` or `false`. * **overrideLineage**: Set the 'Override Lineage' toggle to control whether to override the existing lineage. It supports boolean values either `true` or `false`. * **overrideMetadata**: Set the 'Override Metadata' toggle to control whether to override the existing metadata in the OpenMetadata server with the metadata fetched from the source. If the toggle is set to true, the metadata fetched from the source will override the existing metadata in the OpenMetadata server. If the toggle is set to false, the metadata fetched from the source will not override the existing metadata in the OpenMetadata server. This is applicable for fields like description, tags, owner and displayName. It supports boolean values either `true` or `false`. #### Sink Configuration To send the metadata to OpenMetadata, it needs to be specified as `type: metadata-rest`. #### Workflow Configuration The main property here is the `openMetadataServerConfig`, where you can define the host and security provider of your OpenMetadata installation. **Logger Level** You can specify the `loggerLevel` depending on your needs. If you are trying to troubleshoot an ingestion, running with `DEBUG` will give you far more traces for identifying issues. **JWT Token** JWT tokens will allow your clients to authenticate against the OpenMetadata server. To enable JWT Tokens, you will get more details [here](https://docs.open-metadata.org/latest/deployment/security/enable-jwt-tokens) . You can refer to the JWT Troubleshooting section [link](https://docs.open-metadata.org/latest/deployment/security/jwt-troubleshooting) for any issues in your JWT configuration. **Store Service Connection** If set to `true` (default), we will store the sensitive information either encrypted via the Fernet Key in the database or externally, if you have configured any [Secrets Manager](https://docs.open-metadata.org/latest/deployment/secrets-manager) . If set to `false`, the service will be created, but the service connection information will only be used by the Ingestion Framework at runtime, and won't be sent to the OpenMetadata server. **Store Service Connection** If set to `true` (default), we will store the sensitive information either encrypted via the Fernet Key in the database or externally, if you have configured any [Secrets Manager](https://docs.open-metadata.org/latest/deployment/secrets-manager) . If set to `false`, the service will be created, but the service connection information will only be used by the Ingestion Framework at runtime, and won't be sent to the OpenMetadata server. **SSL Configuration** If you have added SSL to the [OpenMetadata server](https://docs.open-metadata.org/latest/deployment/security/enable-ssl) , then you will need to handle the certificates when running the ingestion too. You can either set `verifySSL` to `ignore`, or have it as `validate`, which will require you to set the `sslConfig.caCertificate` with a local path where your ingestion runs that points to the server certificate file. Find more information on how to troubleshoot SSL issues [here](https://docs.open-metadata.org/latest/deployment/security/enable-ssl/ssl-troubleshooting) . **ingestionPipelineFQN** Fully qualified name of ingestion pipeline, used to identify the current ingestion pipeline. filename.yamlCopy ### 2\. Run with the CLI First, we will need to save the YAML file. Afterward, and with all requirements installed, we can run: Note that from connector to connector, this recipe will always be the same. By updating the YAML configuration, you will be able to extract metadata from different sources. --- # Run the Glue Pipeline Connector Externally We use cookies to improve site navigation, analyze site usage, and enhance your user experience. Click "Accept" to enable cookies or "Reject" to reject cookies. AcceptReject connectors No menu items for this category [Home](https://docs.open-metadata.org/latest) /[Connectors](https://docs.open-metadata.org/latest/connectors) /[Pipeline](https://docs.open-metadata.org/latest/connectors/pipeline) /[Glue Pipeline](https://docs.open-metadata.org/latest/connectors/pipeline/glue-pipeline) /[Yaml](https://docs.open-metadata.org/latest/connectors/pipeline/glue-pipeline/yaml) OpenMetadata Documentation ![Glue](https://docs.open-metadata.org/_next/image?url=%2Fimages%2Fconnectors%2Fglue.webp&w=64&q=75) Glue ==== PROD Available In Feature List Pipelines Pipeline Status Usage Lineage Owners Tags In this section, we provide guides and references to use the Glue connector. Configure and schedule Glue metadata and profiler workflows from the OpenMetadata UI: * [Requirements](https://docs.open-metadata.org/latest/connectors/pipeline/glue-pipeline/yaml#requirements) * [Metadata Ingestion](https://docs.open-metadata.org/latest/connectors/pipeline/glue-pipeline/yaml#metadata-ingestion) How to Run the Connector Externally ----------------------------------- To run the Ingestion via the UI you'll need to use the OpenMetadata Ingestion Container, which comes shipped with custom Airflow plugins to handle the workflow deployment. If, instead, you want to manage your workflows externally on your preferred orchestrator, you can check the following docs to run the Ingestion Framework **anywhere**. [#### External Schedulers\ \ Get more information about running the Ingestion Framework Externally](https://docs.open-metadata.org/latest/deployment/ingestion) Requirements ------------ The Glue connector ingests metadata through AWS [Boto3](https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/glue.html) Client. We will ingest Workflows, its jobs and their run status. The user must have the following permissions for the ingestion to run successfully: * `glue:ListWorkflows` * `glue:GetWorkflow` * `glue:GetJobRuns` ### Python Requirements We have support for Python versions 3.9-3.11 To run the Glue ingestion, you will need to install: Metadata Ingestion ------------------ All connectors are defined as JSON Schemas. [Here](https://github.com/open-metadata/OpenMetadata/blob/main/openmetadata-spec/src/main/resources/json/schema/entity/services/connections/database/glueConnection.json) you can find the structure to create a connection to Glue. In order to create and run a Metadata Ingestion workflow, we will follow the steps to create a YAML configuration able to connect to the source, process the Entities if needed, and reach the OpenMetadata server. The workflow is modeled around the following [JSON Schema](https://github.com/open-metadata/OpenMetadata/blob/main/openmetadata-spec/src/main/resources/json/schema/metadataIngestion/workflow.json) ### 1\. Define the YAML Config This is a sample config for Glue: #### Source Configuration - Service Connection * **awsAccessKeyId** & **awsSecretAccessKey**: When you interact with AWS, you specify your AWS security credentials to verify who you are and whether you have permission to access the resources that you are requesting. AWS uses the security credentials to authenticate and authorize your requests ([docs](https://docs.aws.amazon.com/IAM/latest/UserGuide/security-creds.html) ). Access keys consist of two parts: An **access key ID** (for example, `AKIAIOSFODNN7EXAMPLE`), and a **secret access key** (for example, `wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY`). You must use both the access key ID and secret access key together to authenticate your requests. You can find further information on how to manage your access keys [here](https://docs.aws.amazon.com/IAM/latest/UserGuide/id_credentials_access-keys.html) . **awsSessionToken**: If you are using temporary credentials to access your services, you will need to inform the AWS Access Key ID and AWS Secrets Access Key. Also, these will include an AWS Session Token. **awsRegion**: Each AWS Region is a separate geographic area in which AWS clusters data centers ([docs](https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/Concepts.RegionsAndAvailabilityZones.html) ). As AWS can have instances in multiple regions, we need to know the region the service you want reach belongs to. Note that the AWS Region is the only required parameter when configuring a connection. When connecting to the services programmatically, there are different ways in which we can extract and use the rest of AWS configurations. You can find further information about configuring your credentials [here](https://boto3.amazonaws.com/v1/documentation/api/latest/guide/credentials.html#configuring-credentials) . **endPointURL**: To connect programmatically to an AWS service, you use an endpoint. An _endpoint_ is the URL of the entry point for an AWS web service. The AWS SDKs and the AWS Command Line Interface (AWS CLI) automatically use the default endpoint for each service in an AWS Region. But you can specify an alternate endpoint for your API requests. Find more information on [AWS service endpoints](https://docs.aws.amazon.com/general/latest/gr/rande.html) . **profileName**: A named profile is a collection of settings and credentials that you can apply to a AWS CLI command. When you specify a profile to run a command, the settings and credentials are used to run that command. Multiple named profiles can be stored in the config and credentials files. You can inform this field if you'd like to use a profile other than `default`. Find here more information about [Named profiles for the AWS CLI](https://docs.aws.amazon.com/cli/latest/userguide/cli-configure-profiles.html) . **assumeRoleArn**: Typically, you use `AssumeRole` within your account or for cross-account access. In this field you'll set the `ARN` (Amazon Resource Name) of the policy of the other account. A user who wants to access a role in a different account must also have permissions that are delegated from the account administrator. The administrator must attach a policy that allows the user to call `AssumeRole` for the `ARN` of the role in the other account. This is a required field if you'd like to `AssumeRole`. Find more information on [AssumeRole](https://docs.aws.amazon.com/STS/latest/APIReference/API_AssumeRole.html) . When using Assume Role authentication, ensure you provide the following details: * **AWS Region**: Specify the AWS region for your deployment. * **Assume Role ARN**: Provide the ARN of the role in your AWS account that OpenMetadata will assume. **assumeRoleSessionName**: An identifier for the assumed role session. Use the role session name to uniquely identify a session when the same role is assumed by different principals or for different reasons. By default, we'll use the name `OpenMetadataSession`. Find more information about the [Role Session Name](https://docs.aws.amazon.com/STS/latest/APIReference/API_AssumeRole.html#:~:text=An%20identifier%20for%20the%20assumed%20role%20session.) . **assumeRoleSourceIdentity**: The source identity specified by the principal that is calling the `AssumeRole` operation. You can use source identity information in AWS CloudTrail logs to determine who took actions with a role. Find more information about [Source Identity](https://docs.aws.amazon.com/STS/latest/APIReference/API_AssumeRole.html#:~:text=Required%3A%20No-,SourceIdentity,-The%20source%20identity) . #### Source Configuration - Source Config The `sourceConfig` is defined [here](https://github.com/open-metadata/OpenMetadata/blob/main/openmetadata-spec/src/main/resources/json/schema/metadataIngestion/pipelineServiceMetadataPipeline.json) : * **dbServiceNames**: Database Service Name for the creation of lineage, if the source supports it. * **includeTags**: Set the 'Include Tags' toggle to control whether to include tags as part of metadata ingestion. * **includeUnDeployedPipelines**: Set the 'Include UnDeployed Pipelines' toggle to control whether to include un-deployed pipelines as part of metadata ingestion. By default it is set to `true` * **markDeletedPipelines**: Set the Mark Deleted Pipelines toggle to flag pipelines as soft-deleted if they are not present anymore in the source system. * **pipelineFilterPattern** and **chartFilterPattern**: Note that the `pipelineFilterPattern` and `chartFilterPattern` both support regex as include or exclude. * **includeOwners**: Set the 'Include Owners' toggle to control whether to include owners to the ingested entity if the owner email matches with a user stored in the OM server as part of metadata ingestion. If the ingested entity already exists and has an owner, the owner will not be overwritten.It supports boolean values either `true` or `false`. * **overrideLineage**: Set the 'Override Lineage' toggle to control whether to override the existing lineage. It supports boolean values either `true` or `false`. * **overrideMetadata**: Set the 'Override Metadata' toggle to control whether to override the existing metadata in the OpenMetadata server with the metadata fetched from the source. If the toggle is set to true, the metadata fetched from the source will override the existing metadata in the OpenMetadata server. If the toggle is set to false, the metadata fetched from the source will not override the existing metadata in the OpenMetadata server. This is applicable for fields like description, tags, owner and displayName. It supports boolean values either `true` or `false`. #### Sink Configuration To send the metadata to OpenMetadata, it needs to be specified as `type: metadata-rest`. #### Workflow Configuration The main property here is the `openMetadataServerConfig`, where you can define the host and security provider of your OpenMetadata installation. **Logger Level** You can specify the `loggerLevel` depending on your needs. If you are trying to troubleshoot an ingestion, running with `DEBUG` will give you far more traces for identifying issues. **JWT Token** JWT tokens will allow your clients to authenticate against the OpenMetadata server. To enable JWT Tokens, you will get more details [here](https://docs.open-metadata.org/latest/deployment/security/enable-jwt-tokens) . You can refer to the JWT Troubleshooting section [link](https://docs.open-metadata.org/latest/deployment/security/jwt-troubleshooting) for any issues in your JWT configuration. **Store Service Connection** If set to `true` (default), we will store the sensitive information either encrypted via the Fernet Key in the database or externally, if you have configured any [Secrets Manager](https://docs.open-metadata.org/latest/deployment/secrets-manager) . If set to `false`, the service will be created, but the service connection information will only be used by the Ingestion Framework at runtime, and won't be sent to the OpenMetadata server. **Store Service Connection** If set to `true` (default), we will store the sensitive information either encrypted via the Fernet Key in the database or externally, if you have configured any [Secrets Manager](https://docs.open-metadata.org/latest/deployment/secrets-manager) . If set to `false`, the service will be created, but the service connection information will only be used by the Ingestion Framework at runtime, and won't be sent to the OpenMetadata server. **SSL Configuration** If you have added SSL to the [OpenMetadata server](https://docs.open-metadata.org/latest/deployment/security/enable-ssl) , then you will need to handle the certificates when running the ingestion too. You can either set `verifySSL` to `ignore`, or have it as `validate`, which will require you to set the `sslConfig.caCertificate` with a local path where your ingestion runs that points to the server certificate file. Find more information on how to troubleshoot SSL issues [here](https://docs.open-metadata.org/latest/deployment/security/enable-ssl/ssl-troubleshooting) . **ingestionPipelineFQN** Fully qualified name of ingestion pipeline, used to identify the current ingestion pipeline. filename.yamlCopy ### 2\. Run with the CLI First, we will need to save the YAML file. Afterward, and with all requirements installed, we can run: Note that from connector to connector, this recipe will always be the same. By updating the YAML configuration, you will be able to extract metadata from different sources. --- # Okta SSO Configuration Guide | Confidential Client Setup We use cookies to improve site navigation, analyze site usage, and enhance your user experience. Click "Accept" to enable cookies or "Reject" to reject cookies. AcceptReject ![](https://static.scarf.sh/a.png?x-pxid=8472f2b3-d307-4935-a5ed-3f9d3982c15d) deployment No menu items for this category [Home](https://docs.open-metadata.org/latest) /[Deployment](https://docs.open-metadata.org/latest/deployment) /[Security](https://docs.open-metadata.org/latest/deployment/security) /[Okta](https://docs.open-metadata.org/latest/deployment/security/okta) /[Confidential Client](https://docs.open-metadata.org/latest/deployment/security/okta/confidential-client) OpenMetadata Documentation Okta SSO Configuration (Confidential Client) ============================================ * [Troubleshooting](https://docs.open-metadata.org/latest/deployment/security/okta/confidential-client#troubleshooting) Okta Single Sign-On (SSO) enables users to log in to OpenMetadata with their Okta credentials using **OAuth 2.0** and **OpenID Connect (OIDC)** protocols. ![Okta SSO Configuration - Confidential Client](https://docs.open-metadata.org/images/v1.11/deployment/security/okta/okta2.png) Provider Name ------------- Defines a human-readable name to identify this Okta SSO configuration. * **Example:** `Okta SSO`, `Company Okta` * **Note:** This is for display purposes only and does not affect authentication functionality. Authentication Configuration (Confidential) ------------------------------------------- ### Enable Self Signup Allows users to create OpenMetadata accounts on first login. * **Options:** Enabled | Disabled * **Recommended:** Disabled (for tighter access control) ### Authority Your Okta domain URL used to issue tokens. * **Example:** `https://dev-123456.okta.com` or `https://company.okta.com` * **Note:** This must match your Okta domain exactly. ### Public Key URLs URLs where Okta publishes its public signing keys. * **Example:** `["https://dev-123456.okta.com/oauth2/v1/keys"]` * **Note:** Usually auto-discovered via discovery URI. ### Token Validation Algorithm Specifies the JWT algorithm to validate token signatures. * **Options:** RS256 | RS384 | RS512 * **Default:** RS256 ### Client Type Defines the application type: public (no secret) or confidential (requires client secret). * **Recommended:** Confidential (for backend services and web apps) ### OIDC Client ID The client ID from your Okta app registration. * **Example:** `0oabc123def456ghi789` ### OIDC Client Secret The client secret for authenticating your confidential client. * **Example:** `abc123def456ghi789jkl012mno345pqr678st` * **Note:** Only used for confidential clients. Rotate regularly. ### OIDC Request Scopes Permissions requested during authentication. * **Default:** `openid email profile` * **Optional:** Add `groups` for group-based authorization. ### OIDC Discovery URI URI to retrieve Okta’s OIDC metadata. * **Example:** `https://dev-123456.okta.com/.well-known/openid-configuration` ### OIDC Use Nonce Enables anti-replay protection. * **Default:** true ### OIDC Preferred JWS Algorithm Preferred JWT signing algorithm. * **Default:** RS256 ### OIDC Response Type Defines the OAuth flow type. * **Options:** `id_token` | `code` * **Recommended:** `code` (authorization code flow) ### OIDC Disable PKCE Disables PKCE (Proof Key for Code Exchange). * **Default:** false * **Note:** Should generally remain enabled for security. ### OIDC Max Clock Skew Allowed time difference (in seconds) between systems during token validation. * **Example:** `0` ### OIDC Client Authentication Method Specifies how the client authenticates with Okta. * **Options:** `client_secret_basic` | `client_secret_post` | `client_secret_jwt` | `private_key_jwt` * **Default:** `client_secret_basic` ### OIDC Token Validity How long tokens remain valid (in seconds). * **Default:** `0` (uses Okta’s default) * **Example:** `3600` (1 hour) ### OIDC Tenant Your Okta organization subdomain. * **Example:** `dev-123456`, `company` ### OIDC Server URL * **Definition:** Your OM server URL. * **Example:** `https://yourapp.company.com`. * **Why it matters:** specifies the URL at which OM is hosted. ### Callback URL Redirect URI for handling login responses. * **Example:** `https://yourapp.company.com/callback` * **Note:** Must match exactly in Okta β†’ Applications β†’ Sign-in redirect URIs ### OIDC Max Age Maximum time (in seconds) before forcing re-authentication. * **Example:** `3600` * **Optional:** Leave empty to use default behavior. ### OIDC Prompt Controls authentication behavior. * **Options:** `none` | `login` | `consent` | `select_account` * **Recommended:** `login` (forces credential prompt) ### OIDC Session Expiry Controls user session duration (in seconds). * **Default:** `604800` (7 days) ### JWT Principal Claims JWT fields used to identify the authenticated user. * **Example:** `["preferred_username", "email", "sub"]` ### JWT Principal Claims Mapping Maps JWT claims to OpenMetadata user profile fields. * **Example:** `["email:email", "name:name", "firstName:given_name"]` * **Note:** Format: `"openmetadata_field:jwt_claim"` ### Admin Principals List of users with full admin access. * **Example:** `["admin@company.com", "superuser@company.com"]` * **Note:** Must match one of the JWT claim values. ### Principal Domain Default domain for user identifiers. * **Example:** `company.com` ### Enforce Principal Domain Restricts access to users within the configured domain. * **Default:** false * **Example:** true ### Enable Secure Socket Connection Enforces secure (SSL/TLS) communication. * **Default:** false * **Recommended:** true for production environments Summary ------- | **Field** | **Example / Default** | | --- | --- | | Client Type | Confidential | | OIDC Client ID | 0oabc123def456ghi789 | | OIDC Client Secret | abc123def456ghi789jkl012mno345pqr678st | | Callback URL | https://yourapp.company.com/callback | | Authority | https://dev-123456.okta.com | | OIDC Discovery URI | https://dev-123456.okta.com/.well-known/openid-configuration | | Public Key URLs | https://dev-123456.okta.com/oauth2/v1/keys | | Token Validation Algorithm | RS256 | | OIDC Response Type | code | | OIDC Request Scopes | openid email profile groups | | OIDC Preferred JWS Algorithm | RS256 | | OIDC Use Nonce | true | | OIDC Disable PKCE | false | | OIDC Client Authentication Method | client\_secret\_basic | | OIDC Max Clock Skew | 0 | | OIDC Token Validity | 3600 | | OIDC Max Age | 3600 | | OIDC Prompt | login | | OIDC Session Expiry | 604800 | | OIDC Tenant | dev-123456 | | OIDC Server URL | https://dev-123456.okta.com | | JWT Principal Claims | \["preferred\_username", "email", "sub"\] | | JWT Mapping | \["email:email", "name:name", "firstName:given\_name"\] | | Admin Principals | \["admin@company.com"\] | | Principal Domain | company.com | | Enforce Principal Domain | false | | SSL/TLS | true | ### Troubleshooting If users are automatically logged out and unable to log in again due to a bad authentication configuration, you can reset the security setup using the following command: After executing the command, **restart the server**. The authentication values from your YAML or Helm chart will then be reapplied on startup. The following tiles detail how to apply this configuration across Docker, Kubernetes, and Bare Metal deployments: [Docker Security\ \ Configure SSO for your Docker Deployment.](https://docs.open-metadata.org/latest/deployment/security/configuration/docker) [Bare Metal Security\ \ Configure SSO for your Bare Metal Deployment.](https://docs.open-metadata.org/latest/deployment/security/configuration/bare-metal) [Kubernetes Security\ \ Configure SSO for your Kubernetes Deployment.](https://docs.open-metadata.org/latest/deployment/security/configuration/kubernetes) --- # Run the ingestion from your Airflow We use cookies to improve site navigation, analyze site usage, and enhance your user experience. Click "Accept" to enable cookies or "Reject" to reject cookies. AcceptReject deployment No menu items for this category [Home](https://docs.open-metadata.org/latest) /[Deployment](https://docs.open-metadata.org/latest/deployment) /[Ingestion](https://docs.open-metadata.org/latest/deployment/ingestion) /[External](https://docs.open-metadata.org/latest/deployment/ingestion/external) /[Airflow](https://docs.open-metadata.org/latest/deployment/ingestion/external/airflow) OpenMetadata Documentation This page is about running the Ingestion Framework **externally**! There are mainly 2 ways of running the ingestion: 1. Internally, by managing the workflows from OpenMetadata. 2. Externally, by using any other tool capable of running Python code. If you are looking for how to manage the ingestion process from OpenMetadata, you can follow this [doc](https://docs.open-metadata.org/latest/deployment/ingestion/openmetadata) . Run the ingestion from your Airflow =================================== OpenMetadata integrates with Airflow to orchestrate ingestion workflows. You can use Airflow to [extract metadata](https://docs.open-metadata.org/latest/connectors/pipeline/airflow) and \[deploy workflows\] (/deployment/ingestion/openmetadata) directly. This guide explains how to run ingestion workflows in Airflow using three different operators: 1. [Python Operator](https://docs.open-metadata.org/latest/deployment/ingestion/external/airflow#python-operator) 2. [Docker Operator](https://docs.open-metadata.org/latest/deployment/ingestion/external/airflow#docker-operator) 3. [Python Virtualenv Operator](https://docs.open-metadata.org/latest/deployment/ingestion/external/airflow#python-virtualenv-operator) Using the Python Operator ------------------------- ### Prerequisites Install the `openmetadata-ingestion` package in your Airflow environment. This approach works best if you have access to the Airflow host and can manage dependencies. #### Installation Command: \-Replace [](https://github.com/open-metadata/OpenMetadata/blob/main/ingestion/setup.py) with the sources to ingest, such as mysql, snowflake, or s3. \-Replace x.y.z with the OpenMetadata version matching your server (e.g., 1.6.1). ### Example ### Example DAG ### Key Notes * **Function Setup**: The `python_callable` argument in the `PythonOperator` executes the `metadata_ingestion_workflow` function, which instantiates the workflow and runs the ingestion process. * **Drawback**: This method requires pre-installed dependencies, which may not always be feasible. Consider using the **DockerOperator** or **PythonVirtualenvOperator** as alternatives. Using the Docker Operator ------------------------- For this operator, we can use the `openmetadata/ingestion-base` image. This is useful to prepare DAGs without any installation required on the environment, although it needs for the host to have access to the Docker commands. ### Prerequisites Ensure the Airflow host can run Docker commands. For Docker Compose setups, map the Docker socket as follows: ### Example ### Example DAG Make sure to tune out the DAG configurations (`schedule_interval`, `start_date`, etc.) as your use case requires. If you encounter issues such as missing task instances or Airflow failing to locate a deployed DAG (e.g., `Dag '' could not be found`), this may be due to a **timezone mismatch** in your Airflow configuration. To resolve this, set the following in your `airflow.cfg`: This ensures that Airflow uses the system timezone, which is particularly important when OpenMetadata and Airflow are running on the same server. ### Key Notes * **Image Version**: Ensure the Docker image version matches your OpenMetadata server version (e.g., `openmetadata/ingestion-base:0.13.2`). * **Pipeline Types**: Set the `pipelineType` to `metadata`, `usage`, `lineage`, `profiler`, or other supported values. * **No Installation Required**: The `DockerOperator` eliminates the need to install dependencies directly on the Airflow host. Another important point here is making sure that the Airflow will be able to run Docker commands to create the task. As our example was done with Airflow in Docker Compose, that meant setting `docker_url="unix://var/run/docker.sock"`. The final important elements here are: * `command="python main.py"`: This does not need to be modified, as we are shipping the `main.py` script in the image, used to trigger the workflow. * `environment={"config": config, "pipelineType": "metadata"}`: Again, in most cases you will just need to update the `config` string to point to the right connector. Other supported values of `pipelineType` are `usage`, `lineage`, `profiler`, `dataInsight`, `elasticSearchReindex`, `dbt`, `application` or `TestSuite`. Pass the required flag depending on the type of workflow you want to execute. Make sure that the YAML config reflects what ingredients are required for your Workflow. Using the Python Virtualenv Operator ------------------------------------ ### Prerequisites As stated in Airflow's [docs](https://airflow.apache.org/docs/apache-airflow/stable/howto/operator/python.html#pythonvirtualenvoperator) , install the `virtualenv` package on the Airflow host.If using a different Python version in the virtual environment (e.g., Python 3.9 while Airflow uses 3.7), install additional packages such as: ### Example DAG ### Key Notes **Function Rules**: * Use a `def` function (not part of a class). * All imports must occur inside the function. * Avoid referencing variables outside the function's scope. Ingestion Workflow classes -------------------------- We have different classes for different types of workflows. The logic is always the same, but you will need to change your import path. The rest of the method calls will remain the same. For example, for the `Metadata` workflow we'll use: The classes for each workflow type are: * `Metadata`: `from metadata.workflow.metadata import MetadataWorkflow` * `Lineage`: `from metadata.workflow.metadata import MetadataWorkflow` (same as metadata) * `Usage`: `from metadata.workflow.usage import UsageWorkflow` * `dbt`: `from metadata.workflow.metadata import MetadataWorkflow` * `Profiler`: `from metadata.workflow.profiler import ProfilerWorkflow` * `Data Quality`: `from metadata.workflow.data_quality import TestSuiteWorkflow` * `Data Insights`: `from metadata.workflow.data_insight import DataInsightWorkflow` * `Elasticsearch Reindex`: `from metadata.workflow.metadata import MetadataWorkflow` (same as metadata) --- # Tests - UI Config | OpenMetadata Quality Config Guide We use cookies to improve site navigation, analyze site usage, and enhance your user experience. Click "Accept" to enable cookies or "Reject" to reject cookies. AcceptReject ![](https://static.scarf.sh/a.png?x-pxid=6814879a-96c0-40bd-be69-ddfeb14694aa) how-to-guides No menu items for this category [Home](https://docs.open-metadata.org/latest) /[How To Guides](https://docs.open-metadata.org/latest/how-to-guides) /[Data Quality Observability](https://docs.open-metadata.org/latest/how-to-guides/data-quality-observability) /[Quality](https://docs.open-metadata.org/latest/how-to-guides/data-quality-observability/quality) /[Tests Ui](https://docs.open-metadata.org/latest/how-to-guides/data-quality-observability/quality/tests-ui) OpenMetadata Documentation Tests in the OpenMetadata UI ============================ Here you can see all the supported tests definitions and how to configure them in the UI. A **Test Definition** is a generic definition of a test. This Test Definition then gets specified in a Test Case. This Test Case is where the parameter(s) of a Test Definition are specified. In this section, you will learn what tests we currently support and how to configure them in the OpenMetadata UI. * [Table Tests](https://docs.open-metadata.org/latest/how-to-guides/data-quality-observability/quality/tests-ui#table-tests) * [Column Tests](https://docs.open-metadata.org/latest/how-to-guides/data-quality-observability/quality/tests-ui#column-tests) Table Tests ----------- Tests applied on top of a Table. Here is the list of all table tests: * [Table Row Count to Equal](https://docs.open-metadata.org/latest/how-to-guides/data-quality-observability/quality/tests-ui#table-row-count-to-equal) * [Table Row Count to be Between](https://docs.open-metadata.org/latest/how-to-guides/data-quality-observability/quality/tests-ui#table-row-count-to-be-between) * [Table Column Count to Equal](https://docs.open-metadata.org/latest/how-to-guides/data-quality-observability/quality/tests-ui#table-column-count-to-equal) * [Table Column Count to be Between](https://docs.open-metadata.org/latest/how-to-guides/data-quality-observability/quality/tests-ui#table-column-count-to-be-between) * [Table Column Name to Exist](https://docs.open-metadata.org/latest/how-to-guides/data-quality-observability/quality/tests-ui#table-column-name-to-exist) * [Table Column to Match Set](https://docs.open-metadata.org/latest/how-to-guides/data-quality-observability/quality/tests-ui#table-column-to-match-set) * [Table Custom SQL Test](https://docs.open-metadata.org/latest/how-to-guides/data-quality-observability/quality/tests-ui#table-custom-sql-test) * [Table Row Inserted Count To Be Between](https://docs.open-metadata.org/latest/how-to-guides/data-quality-observability/quality/tests-ui#table-row-inserted-count-to-be-between) * [Compare 2 Tables for Differences](https://docs.open-metadata.org/latest/how-to-guides/data-quality-observability/quality/tests-ui#compare-2-tables-for-differences) * [Table Data to Be Fresh \[Collate\]](https://docs.open-metadata.org/latest/how-to-guides/data-quality-observability/quality/tests-ui#table-data-to-be-fresh-collate) ### Table Row Count to Equal Validate that the total number of rows in a table exactly matches an expected value.\*\* #### When to Use * To monitor tables where row count is expected to remain fixed (e.g., dimension tables). * To catch over- or under-loading issues after ETL processes. * To verify baseline data volumes for test/staging/prod comparisons. #### Test Summary | Property | Description | | --- | --- | | **Expected Value** | The exact number of rows the table should contain. | #### Test Logic | Condition | Status | | --- | --- | | Actual row count = expected value | βœ… Success | | Actual row count β‰  expected value | ❌ Failed | ![](https://docs.open-metadata.org/images/v1.11/how-to-guides/quality/table-test/equal.gif) ### Table Row Count to be Between Ensure that the total number of rows in the table falls within an expected range. #### When to Use * To monitor for abnormal growth or shrinkage in table size. * To catch failed inserts, unintended truncations, or unexpected data surges. * To set alerts based on historical data volume expectations. #### Test Summary | Property | Description | | --- | --- | | **Min Value** | Minimum expected number of rows (`minValue`) | | **Max Value** | Maximum allowed number of rows (`maxValue`) | * At least one of these values is required to run the test. #### Test Logic | Condition | Status | | --- | --- | | Row count is between `minValue` and `maxValue` | βœ… Success | | Row count is outside the defined range | ❌ Failed | ![](https://docs.open-metadata.org/images/v1.11/how-to-guides/quality/table-test/between.gif) ### Table Column Count to Equal Validate that the table contains exactly the expected number of columns. #### When to Use * To detect unapproved schema changes (e.g., columns being added or dropped). * To enforce data contracts between teams or systems. * To ensure structural consistency across environments. #### Test Summary | Property | Description | | --- | --- | | **Expected Count** | Exact number of columns the table must have. | #### Test Logic | Condition | Status | | --- | --- | | Actual column count = expected count | βœ… Success | | Actual column count β‰  expected count | ❌ Failed | ![](https://docs.open-metadata.org/images/v1.11/how-to-guides/quality/table-test/column-equal.gif) ### Table Column Count to be Between Validate that the number of columns in a table falls within a defined range. #### When to Use * To detect schema drift or changes in table structure. * To ensure a table has a predictable number of columns across environments (e.g., staging vs. production). #### Test Summary | Property | Description | | --- | --- | | **Min Columns** | Minimum number of expected columns (`minColValue`) | | **Max Columns** | Maximum number of allowed columns (`maxColValue`) | #### Test Logic | Condition | Status | | --- | --- | | Actual column count is within the defined range | βœ… Success | | Actual column count is outside the defined range | ❌ Failed | ![](https://docs.open-metadata.org/images/v1.11/how-to-guides/quality/table-test/column-between.gif) ### Table Column Name to Exist Ensure that a specific column is present in the table schema. #### When to Use * To validate that required schema fields exist (e.g., `order_id`, `customer_id`). * To monitor schema changes that might break downstream processes. * To enforce critical column presence in governed datasets. #### Test Summary | Property | Description | | --- | --- | | **Column Name** | Name of the column that must exist in the table. | #### Test Logic | Condition | Status | | --- | --- | | `columnName` exists in the table schema | βœ… Success | | `columnName` is missing from the table | ❌ Failed | ![](https://docs.open-metadata.org/images/v1.11/how-to-guides/quality/table-test/exist.gif) ### Table Column to Match Set Validate that a table’s column names match a predefined set β€” with or without order sensitivity. #### When to Use * To ensure schema alignment across different environments or pipeline stages. * To detect unexpected column additions, deletions, or reordering. * To enforce table contracts where the exact structure is critical. #### Test Summary | Property | Description | | --- | --- | | **Column Names** | Comma-separated list of expected column names (e.g., `col1, col2, col3`) | | **Ordered** | Boolean flag (`true` or `false`) β€” whether the order of columns must match. | #### Test Logic | Ordered | Condition | Status | | --- | --- | --- | | `false` | All expected column names exist (any order) | βœ… Success | | `true` | Column names match and appear in the exact order | βœ… Success | | `false` | Some columns are missing or extra | ❌ Failed | | `true` | Columns are present but order is incorrect | ❌ Failed | ![](https://docs.open-metadata.org/images/v1.11/how-to-guides/quality/table-test/match-set.gif) ### Table Custom SQL Test Use this test to define your own validation logic using a custom SQL expression. #### When to Use * To implement logic beyond predefined test definitions. * To detect outliers, nulls, duplicates, or business-specific data anomalies. * When you need full flexibility using SQL syntax. #### Test Summary | Property | Description | | --- | --- | | **SQL Expression** | The SQL query used to evaluate the test. | | **Strategy** | Defines how to interpret the result. Options: `ROWS` _(default)_ or `COUNT`. | | **Threshold** | The maximum allowed rows or count before marking the test as failed. Default is `0`. | #### Test Logic | Strategy | Condition | Status | | --- | --- | --- | | ROWS | Number of returned rows ≀ `threshold` | βœ… Success | | ROWS | Number of returned rows > `threshold` | ❌ Failed | | COUNT | Count result ≀ `threshold` | βœ… Success | | COUNT | Count result > `threshold` | ❌ Failed | ![](https://docs.open-metadata.org/images/v1.11/how-to-guides/quality/table-test/custom-sql.gif) ### Table Row Inserted Count To Be Between Check that the number of rows inserted during a defined time window falls within an expected range.\*\* #### When to Use * To detect whether recent data ingestion volumes are within acceptable limits. * To monitor time-partitioned tables for daily/hourly/monthly data drops or spikes. * To validate pipeline freshness and completeness over time. #### Test Summary | Property | Description | | --- | --- | | **Min Row Count** | Minimum number of inserted rows expected in the given range. | | **Max Row Count** | Maximum number of inserted rows allowed in the given range. | | **Column Name** | Timestamp column used to filter the inserted rows. | | **Range Type** | Time granularity: `HOUR`, `DAY`, `MONTH`, or `YEAR`. | | **Range Interval** | Number of units (e.g., last `1 DAY`, `2 HOURS`, etc.). | #### Test Logic | Condition | Status | | --- | --- | | Row count within `min` and `max` for the interval | βœ… Success | | Row count outside of the expected range | ❌ Failed | The Table Row Inserted Count To Be Between cannot be executed against tables that have configured a partition in OpenMetadata. The logic of the test performed will be similar to executing a Table Row Count to be Between test against a table with a partition configured. ![](https://docs.open-metadata.org/images/v1.11/how-to-guides/quality/table-test/inserted-count.gif) ### Compare 2 Tables for Differences Use this test to verify data consistency between two tables, even across different platforms or services. #### When to Use * After data replication or migration (e.g., Snowflake β†’ Redshift). * To validate data integrity between source and target systems. #### Test Summary | Property | Description | | --- | --- | | **Key Columns** | Columns used as the row-matching key. Defaults to the table's primary key if not specified. | | **Columns to Compare** | Subset of columns used for comparison. If not provided, all columns will be compared. | | **Second Table** | Fully qualified name of the second table (e.g., `redshift_dbt.dev.dbt_jaffle.boolean_test`). | | **Threshold** | Maximum number of mismatched rows allowed. Default is `0` (strict equality). | | **Filter Condition** | _(Optional)_ A `WHERE` clause (e.g., `id != 999`) to limit rows involved in the comparison. | | **Case-Sensitive Columns** | Set to `true` if column name case must match exactly (default is `false`). | #### Test Logic | Condition | Status | | --- | --- | | Number of differing rows ≀ threshold | βœ… Success | | Number of differing rows > threshold | ❌ Failed | #### 🌐 Supported Data Sources * Snowflake * BigQuery * Athena * Redshift * Postgres * MySQL * MSSQL * Oracle * Trino * SAP Hana ![](https://docs.open-metadata.org/images/v1.11/how-to-guides/quality/table-test/differences.gif) ### Table Data to Be Fresh \[Collate\] Ensure that table data is being updated frequently enough to be considered fresh. #### When to Use * To monitor data pipelines for staleness or lag. * To detect delays in scheduled batch updates. * To ensure compliance with SLAs for near real-time data delivery. #### Test Summary | Property | Description | | --- | --- | | **Column** | The datetime column used to determine the last update. | | **Time Since Update** | Time threshold (in seconds) β€” maximum age of the most recent data entry. | #### Test Logic | Condition | Status | | --- | --- | | Last update time ≀ `timeSinceUpdate` | βœ… Success | | Last update time > `timeSinceUpdate` | ❌ Failed | ![](https://docs.open-metadata.org/images/v1.11/how-to-guides/quality/table-test/fresh.gif) Column Tests ------------ Tests applied on top of Column metrics. Here is the list of all column tests: * [Column Values to Be Unique](https://docs.open-metadata.org/latest/how-to-guides/data-quality-observability/quality/tests-ui#column-values-to-be-unique) * [Column Values to Be Not Null](https://docs.open-metadata.org/latest/how-to-guides/data-quality-observability/quality/tests-ui#column-values-to-be-not-null) * [Column Values to Match Regex](https://docs.open-metadata.org/latest/how-to-guides/data-quality-observability/quality/tests-ui#column-values-to-match-regex) * [Column Values to not Match Regex](https://docs.open-metadata.org/latest/how-to-guides/data-quality-observability/quality/tests-ui#column-values-to-not-match-regex) * [Column Values to Be in Set](https://docs.open-metadata.org/latest/how-to-guides/data-quality-observability/quality/tests-ui#column-values-to-be-in-set) * [Column Values to Be Not In Set](https://docs.open-metadata.org/latest/how-to-guides/data-quality-observability/quality/tests-ui#column-values-to-be-not-in-set) * [Column Values to Be Between](https://docs.open-metadata.org/latest/how-to-guides/data-quality-observability/quality/tests-ui#column-values-to-be-between) * [Column Values Missing Count to Be Equal](https://docs.open-metadata.org/latest/how-to-guides/data-quality-observability/quality/tests-ui#column-values-missing-count-to-be-equal) * [Column Values Lengths to Be Between](https://docs.open-metadata.org/latest/how-to-guides/data-quality-observability/quality/tests-ui#column-values-lengths-to-be-between) * [Column Value Max to Be Between](https://docs.open-metadata.org/latest/how-to-guides/data-quality-observability/quality/tests-ui#column-value-max-to-be-between) * [Column Value Min to Be Between](https://docs.open-metadata.org/latest/how-to-guides/data-quality-observability/quality/tests-ui#column-value-min-to-be-between) * [Column Value Mean to Be Between](https://docs.open-metadata.org/latest/how-to-guides/data-quality-observability/quality/tests-ui#column-value-mean-to-be-between) * [Column Value Median to Be Between](https://docs.open-metadata.org/latest/how-to-guides/data-quality-observability/quality/tests-ui#column-value-median-to-be-between) * [Column Values Sum to Be Between](https://docs.open-metadata.org/latest/how-to-guides/data-quality-observability/quality/tests-ui#column-values-sum-to-be-between) * [Column Values Standard Deviation to Be Between](https://docs.open-metadata.org/latest/how-to-guides/data-quality-observability/quality/tests-ui#column-values-standard-deviation-to-be-between) * [Column Values To Be At Expected Location](https://docs.open-metadata.org/latest/how-to-guides/data-quality-observability/quality/tests-ui#column-values-to-be-at-expected-location) ### Column Values to Be Unique Ensures each value in a column appears only once. #### Dimension `Uniqueness` #### When to Use * Primary keys or natural identifiers * Fields like email, username, or ID #### Behavior | Condition | Status | | --- | --- | | All values are unique | βœ… | | Any duplicate value found | ❌ | ![](https://docs.open-metadata.org/images/v1.11/how-to-guides/quality/column-test/unique.gif) ### Column Values to Be Not Null Ensures there are no NULL entries in the column. #### Dimension `Completeness` #### When to Use * Mandatory fields such as `email`, `amount`, `created_at` * Required keys or business-critical columns #### Behavior | Condition | Status | | --- | --- | | No NULLs present | βœ… | | Any NULL value present | ❌ | ![](https://docs.open-metadata.org/images/v1.11/how-to-guides/quality/column-test/not-null.gif) ### Column Values to Match Regex This test allows us to specify how many values in a column we expect that will match a certain regex expression. Please note that for certain databases we will fall back to SQL `LIKE` expression. The databases supporting regex pattern as of 0.13.2 are: * redshift * postgres * oracle * mysql * mariaDB * sqlite * clickhouse * snowflake Ensures all values match a specified regular expression pattern. #### Dimension `Validity` #### When to Use * Emails, zip codes, IDs, structured formats #### Behavior | Condition | Status | | --- | --- | | All values match regex | βœ… | | Any value does not match | ❌ | ![](https://docs.open-metadata.org/images/v1.11/how-to-guides/quality/column-test/match-regex.gif) ### Column Values to not Match Regex This test allows us to specify values in a column we expect that will not match a certain regex expression. If the test find values matching the `forbiddenRegex` the test will fail. Please note that for certain databases we will fall back to SQL `LIKE` expression. The databases supporting regex pattern as of 0.13.2 are: * redshift * postgres * oracle * mysql * mariaDB * sqlite * clickhouse * snowflake The other databases will fall back to the `LIKE` expression Ensures values do **not** match a restricted regex pattern. #### Dimension `Validity` #### When to Use * Prevent forbidden values, test strings, or patterns #### Behavior | Condition | Status | | --- | --- | | No value matches forbidden pattern | βœ… | | Any value matches the pattern | ❌ | ![](https://docs.open-metadata.org/images/v1.11/how-to-guides/quality/column-test/not-match-regex.gif) ### Column Values to Be in Set Ensures values are within a predefined whitelist. #### Dimension `Validity` #### When to Use * Enum values: `status`, `currency`, `country_code` #### Behavior | Condition | Status | | --- | --- | | All values in set (if `matchEnum = true`) | βœ… | | Any value not in set (if `matchEnum = true`) | ❌ | | Any value from set exists (if `matchEnum = false`) | βœ… | | No values from set found (if `matchEnum = false`) | ❌ | ![](https://docs.open-metadata.org/images/v1.11/how-to-guides/quality/column-test/column-values-in-set.gif) ### Column Values to Be Not In Set Ensures values are **not** in a specified blacklist. #### Dimension `Validity` #### When to Use * Block invalid values like `"NA"`, `"Unknown"`, `-1` #### Behavior | Condition | Status | | --- | --- | | No values from forbidden set | βœ… | | Any value from forbidden set found | ❌ | ![](https://docs.open-metadata.org/images/v1.11/how-to-guides/quality/column-test/column-values-not-in-set.gif) ### Column Values to Be Between Validates numeric values of a column are within a given range. #### Dimension `Accuracy` #### When to Use * Username length, field input length validation #### Behavior | Condition | Status | | --- | --- | | Length within `[min, max]` | βœ… | | Length < min or > max | ❌ | ![](https://docs.open-metadata.org/images/v1.11/how-to-guides/quality/column-test/to-be-between.gif) ### Column Values Missing Count to Be Equal Ensures total missing values (NULL + defined "missing" strings) match a target count. #### Dimension `Completeness` #### When to Use * Auditing known missing values * Accounting for `"NA"`, `"N/A"`, `"null"` #### Behavior | Condition | Status | | --- | --- | | Missing count = expected value | βœ… | | Missing count β‰  expected value | ❌ | ![](https://docs.open-metadata.org/images/v1.11/how-to-guides/quality/column-test/missing-count.gif) ### Column Values Lengths to Be Between Ensures that the length of each string value in the column is within a defined character range. #### Dimension `Accuracy` #### When to Use * To validate field length constraints like `name`, `address`, or `description` * To catch too-short or too-long values that may break UI or downstream logic #### Behavior | Condition | Status | | --- | --- | | All values have length within `[min, max]` | βœ… | | Any value length < min or > max | ❌ | ![](https://docs.open-metadata.org/images/v1.11/how-to-guides/quality/column-test/lengths-to-be-between.gif) ### Column Value Max to Be Between Validates the **maximum** value of a column lies within a range. #### Dimension `Accuracy` #### When to Use * Cap validation for `score`, `amount`, `age` #### Behavior | Condition | Status | | --- | --- | | Max value in range `[min, max]` | βœ… | | Max < min or Max > max | ❌ | ![](https://docs.open-metadata.org/images/v1.11/how-to-guides/quality/column-test/max.gif) ### Column Value Min to Be Between Validates the **minimum** value of a column lies within a range. #### Dimension `Accuracy` #### When to Use * Threshold validation for `discount`, `price`, etc. #### Behavior | Condition | Status | | --- | --- | | Min value in range `[min, max]` | βœ… | | Min < min or Min > max | ❌ | ![](https://docs.open-metadata.org/images/v1.11/how-to-guides/quality/column-test/min.gif) ### Column Value Mean to Be Between Validates that the **mean (average)** value is in the expected range. #### Dimension `Accuracy` #### When to Use * Check dataset drift or pipeline behavior #### Behavior | Condition | Status | | --- | --- | | Mean value in `[min, max]` | βœ… | | Mean < min or Mean > max | ❌ | ![](https://docs.open-metadata.org/images/v1.11/how-to-guides/quality/column-test/mean.gif) ### Column Value Median to Be Between Validates the **median** value is in the expected range. #### Dimension `Accuracy` #### When to Use * Median income, score, latency checks #### Behavior | Condition | Status | | --- | --- | | Median in range `[min, max]` | βœ… | | Median < min or Median > max | ❌ | ![](https://docs.open-metadata.org/images/v1.11/how-to-guides/quality/column-test/median.gif) ### Column Values Sum to Be Between Validates the total **sum** of values in a column is within a defined range. #### Dimension `Accuracy` #### When to Use * Revenue, units sold, total scores, etc. #### Behavior | Condition | Status | | --- | --- | | Sum in range `[min, max]` | βœ… | | Sum < min or Sum > max | ❌ | ![](https://docs.open-metadata.org/images/v1.11/how-to-guides/quality/column-test/sum.gif) ### Column Values Standard Deviation to Be Between Validates the **standard deviation** (spread) of values is acceptable. #### Dimension `Accuracy` #### When to Use * Monitoring variance in numeric datasets #### Behavior | Condition | Status | | --- | --- | | Std Dev in `[min, max]` | βœ… | | Std Dev < min or > max | ❌ | ![](https://docs.open-metadata.org/images/v1.11/how-to-guides/quality/column-test/standard-deviation.gif) ### Column Values To Be At Expected Location Validates latitude/longitude values are within a defined area. #### Dimension `Accuracy` #### When to Use * Verifying address coordinates * Mapping regional data #### Behavior | Condition | Status | | --- | --- | | Coordinates within buffer of expected location | βœ… | | Any record outside allowed radius | ❌ | ![](https://docs.open-metadata.org/images/v1.11/how-to-guides/quality/column-test/expected-location.gif) --- # Dimensional Validation | Data Quality Testing by Dimension We use cookies to improve site navigation, analyze site usage, and enhance your user experience. Click "Accept" to enable cookies or "Reject" to reject cookies. AcceptReject ![](https://static.scarf.sh/a.png?x-pxid=6814879a-96c0-40bd-be69-ddfeb14694aa) how-to-guides No menu items for this category [Home](https://docs.open-metadata.org/latest) /[How To Guides](https://docs.open-metadata.org/latest/how-to-guides) /[Data Quality Observability](https://docs.open-metadata.org/latest/how-to-guides/data-quality-observability) /[Quality](https://docs.open-metadata.org/latest/how-to-guides/data-quality-observability/quality) /[Dimensional Validation](https://docs.open-metadata.org/latest/how-to-guides/data-quality-observability/quality/dimensional-validation) OpenMetadata Documentation Dimensional Validation ====================== Dimensional validation allows you to run data quality tests grouped by business dimensions, helping you identify which segments of your data contain quality issues. Instead of getting a single pass/fail result for an entire column, you can see test results broken down by region, product category, customer type, or any other dimension that matters to your business. What is Dimensional Validation? ------------------------------- When you run a standard data quality test in OpenMetadata, you get a single result that tells you whether your entire column meets your quality criteria. For example, a "Column Values to be Not Null" test might tell you that 95% of your data is valid across 10 million rows. Dimensional validation enhances this by grouping your test results by a dimension column. Using the same example, instead of just knowing that 95% is valid overall, you can also see: * **North America region**: 99% valid (excellent) * **Europe region**: 97% valid (good) * **Asia Pacific region**: 88% valid (needs attention) * **Latin America region**: 92% valid (acceptable) This granular view helps you quickly identify problem areas in your data and take targeted action. You can focus your data quality improvements on the specific regions, products, or business segments that need the most attention. Dimensional validation works with all column-level tests in OpenMetadata. You can dimension your tests by any categorical column in your tableβ€”such as geographic regions, product types, customer segments, or time periods. The feature automatically computes metrics for each dimension and ranks them by impact score, so you always see the most critical issues first. Dimensional validation is only available for **column-level tests**. Table-level tests (such as row count checks or table-wide comparisons) do not support dimensional validation since they operate on the entire table rather than individual column values. Key Concepts ------------ Before setting up dimensional validation, it's helpful to understand these core concepts: ### Dimension Column The **dimension column** is the column you use to group your test results. This should be a categorical column with meaningful business valuesβ€”such as region, department, product\_category, or order\_status. When you run a dimensional test, OpenMetadata automatically groups your data by unique values in this column and computes test metrics separately for each group. For example, if you choose "region" as your dimension column, and your region column contains values like "North America", "Europe", and "Asia Pacific", you'll get separate test results for each of these regions. ### Dimension Group A **dimension group** is one unique value in your dimension column. In the region example above, "North America" is one dimension group, "Europe" is another, and so on. OpenMetadata shows you metrics for each dimension group, helping you understand how data quality varies across your business segments. ### Top Dimensions By default, OpenMetadata shows you the **top 10 dimensions** ranked by impact score. These are the dimension groups with the most significant data quality issues. The impact score is calculated based on both the number of failing rows and the proportion of data affected. This ensures that you see the dimensions that matter most, whether they have high failure counts or high failure rates. ### "Others" Group If your dimension column has more than 10 unique values, OpenMetadata automatically creates an **"Others"** group that combines all dimensions outside the top 10. This keeps your results focused on the most important issues while still providing complete coverage of your data. For example, if you have 50 product categories, you'll see: * The top 10 categories with the worst data quality * An "Others" group representing the remaining 40 categories combined The metrics for the "Others" group are accurately calculated across all included dimensions, giving you a true picture of data quality in your long-tail segments. ### Cardinality **Cardinality** refers to the number of unique values in your dimension column. A region column with 5 values (North America, Europe, Asia, etc.) has low cardinality. A customer\_id column with 1 million unique values has high cardinality. Cardinality is crucial for performance. Dimensional validation works best with low cardinality. High cardinality dimensions can cause significant performance overhead and longer execution times. **Recommended cardinality range**: 5-25 unique values for optimal performance. Avoid using high-cardinality columns like customer\_id, order\_id, or timestamps as dimension columns. When to Use Dimensional Validation ---------------------------------- Dimensional validation is most valuable when you need to understand **where** data quality issues are occurring, not just whether they exist. Here are common scenarios where dimensional validation provides the most value: ### Multi-Region or Multi-Location Data If your organization operates across multiple regions, countries, or locations, dimensional validation helps you identify location-specific data quality issues. You might discover that data from a specific warehouse has higher null rates, or that records from a particular country have formatting problems. **Example**: A retail company discovers that null values in the `customer_email` column occur 10x more frequently in their Latin America region compared to other regions, indicating a regional data collection issue. ### Product or Category Analysis When you manage multiple product lines, brands, or categories, dimensional validation reveals product-specific quality patterns. This is especially valuable for companies with diverse product catalogs where data quality requirements may vary by product type. **Example**: An e-commerce platform finds that electronics products have 95% complete descriptions, while clothing products only have 70% complete descriptions, allowing them to focus content improvement efforts. ### Customer Segment Quality For customer-facing data, dimensional validation by customer type, subscription tier, or user segment reveals whether quality issues affect specific customer groups. This helps you prioritize fixes that impact your most valuable customers. **Example**: A SaaS company discovers that free-tier users have 30% incomplete profile data, while enterprise customers have 99% complete profiles, validating their assumption that enterprise onboarding processes are more effective. ### When NOT to Use Dimensional Validation Dimensional validation adds computational overhead and is not appropriate in all situations: * **When you only need yes/no answers**: If you just need to know whether data meets basic quality standards, standard tests are more efficient * **High-cardinality columns**: Avoid using columns with >100 unique values as dimensionsβ€”performance will degrade significantly * **Very large tables without optimization**: For tables >500GB, use sampling or partitioning strategies * **When dimension doesn't provide business value**: Only add dimensional validation when the grouping provides actionable insights Quick Start Guide ----------------- This guide walks you through creating your first dimensional test from the OpenMetadata UI and will focus only on the specifics of the dimensional tests. ### Step 1: Navigate to Your Table 1. Go to the table where you want to create a dimensional test 2. Click on the **Data Observability** tab 3. Click the **Add Test** button in the upper right corner 4. Select **Dimension Level** ![Dimensional Level](https://docs.open-metadata.org/images/v1.11/how-to-guides/quality/dimensional-add-test.png) Navigate to Data Observability and add a Dimension Level test ### Step 2: Select Column and Test Type 1. **Choose your target column**: Select the column you want to validate (e.g., `customer_email`, `order_amount`, `product_description`) 2. **Select the dimension**: Select the dimension you are interested in (e.g, `region`, `status`) 3. **Select a test type**: Choose from any of the supported column-level tests (see Supported Test Types section) 4. **Configure test parameters**: Set thresholds, ranges, or patterns based on your chosen test type For example, to test email completeness: * Column: `customer_email` * Dimension: `region` * Test Type: **Column Values to be Not Null** * Name: `Email Completeness Check` Choose dimension columns with **low-to-medium cardinality** (5-100 unique values) for best performance. Columns like region, product\_type, or order\_status work well. Avoid high-cardinality columns like customer\_id or order\_id. ![Configure Test](https://docs.open-metadata.org/images/v1.11/how-to-guides/quality/dimensional-select-test.png) Choose your column, test type, and dimension ### Step 3: Finish configuring your test, pipeline and running it This is exactly the same as with non-dimensional tests. For detailed information on setting up tests, see [Adding Test Cases to an Entity](https://docs.open-metadata.org/latest/how-to-guides/data-quality-observability/quality/adding-test-cases) and [Adding Test Suites](https://docs.open-metadata.org/latest/how-to-guides/data-quality-observability/quality/adding-test-suites) ### Step 4: View Results Once your test runs, return to the **Data Observability** tab to view dimensional results: 1. Find your dimensional test in the list 2. Click on the test to expand results 3. View the **Dimensional Results** section showing metrics for each dimension group ![View Dimensional Results](https://docs.open-metadata.org/images/v1.11/how-to-guides/quality/dimensional-results.png) Review test results grouped by dimension Understanding Results --------------------- When you view dimensional test results, OpenMetadata provides detailed metrics for each dimension group to help you understand data quality patterns. ### Result Metrics For each dimension group, you'll see: **Dimension Value**: The specific value of the dimension column (e.g., "North America", "Electronics", "Premium Tier") **Total Count**: The total number of rows in this dimension group **Failed Count**: The number of rows that failed the test criteria in this dimension **Impact Score**: A calculated score (0.0 to 1.0) representing the severity of quality issues in this dimension, based on both the failure rate and absolute number of failures **Test-Specific Metrics**: Additional metrics vary by test type. For example: * Not Null tests show null counts * Mean tests show actual mean values * Unique tests show duplicate counts ### Impact Score Ranking Dimensional results are automatically sorted by **impact score** in descending order, ensuring the most critical issues appear first. The impact score algorithm balances: 1. **Failure rate**: What percentage of rows failed in this dimension? 2. **Absolute volume**: How many rows failed in total? This balanced approach ensures you see both: * High-failure-rate dimensions that might affect smaller data volumes * High-volume dimensions where even moderate failure rates represent many problematic rows For example, if you have: * **Region A**: 1,000 rows, 500 failures (50% failure rate, impact score could be: 0.95) * **Region B**: 100,000 rows, 20,000 failures (20% failure rate, impact score could be: 0.87) Region A appears first due to its higher failure rate, but Region B still ranks high due to the large absolute number of failures. ### The "Others" Group If your dimension column has more than 10 unique values, you'll see an **"Others"** group representing all dimensions outside the top 10: * Metrics for "Others" are accurately calculated across all included dimensions * If "Others" shows quality issues, consider investigating which specific values it contains * You may need to create separate focused tests for specific values in the "Others" group ### Historical Trend View Click on any dimension group to view its quality trend over time: * See how this dimension's quality has changed across test runs * Identify whether issues are improving or degrading * Correlate quality changes with business events or system changes ![Dimensional Quality Trends](https://docs.open-metadata.org/images/v1.11/how-to-guides/quality/dimensional-trends.png) Track quality trends for each dimension over time Best Practices -------------- Follow these best practices to get the most value from dimensional validation while maintaining good performance. ### Choose Meaningful Dimensions Select dimension columns that provide **actionable business insights**: βœ… **Good dimension choices**: * Geographic segments (region, country, state, store\_location) * Business categories (product\_type, department, customer\_segment) * Operational groups (warehouse\_id, processing\_center, supplier\_name) * Time periods (order\_month, fiscal\_quarter, year) ❌ **Poor dimension choices**: * High-cardinality IDs (customer\_id, order\_id, transaction\_id) * Free-text fields (comments, descriptions, addresses) * Unique identifiers or timestamps * Columns with mostly null values ### Optimize Cardinality If your desired dimension has high cardinality, consider: * **Grouping values**: Convert specific cities into regions, or group dates into months * **Creating derived columns**: Add a calculated column with grouped values (e.g., `customer_tier` derived from `customer_id`) ### Start Small, Then Scale When implementing dimensional validation: 1. **Start with one or two critical tests** on important tables 2. **Choose low-cardinality dimensions** initially 3. **Validate performance** before expanding to more tests 4. **Add sampling** if execution time exceeds acceptable limits 5. **Gradually expand** to additional tables and dimensions ### Combine with Sampling for Large Tables For large tables, enable **Profile Sample** to reduce data scan volumes For detailed sampling configuration, see [Profiler Workflow - Profile Sample](https://docs.open-metadata.org/latest/how-to-guides/data-quality-observability/profiler/workflow#profiler-options) . ### Use Partitioning For large tables with partitioning columns, enable **partitioning** to focus tests on meaningful data For detailed partitioning configuration, see [Profiler Workflow - Enable Partition](https://docs.open-metadata.org/latest/how-to-guides/data-quality-observability/profiler/workflow#profiler-options) . ### Monitor Test Execution Time Keep an eye on how long dimensional tests take to run: * **Baseline tests** should complete in seconds to minutes * **Dimensional tests** will take longer than non-dimensional tests. This varies greatly depending on the test type. You can view execution times in the Pipeline tab after each test run. ### Name Tests Clearly Use descriptive test names that include the dimension: * βœ… `Email Completeness by Region` * βœ… `Price Range Validation by Product Category` * βœ… `Order Amount Accuracy per Warehouse` * ❌ `Test 1` * ❌ `Column Check` Clear naming helps your team understand what each test validates and which dimensions it covers. ### Document Business Context In the test description field, explain: * **Why this dimension matters**: What business decisions depend on this segmentation? * **Expected patterns**: Are some dimensions expected to have different quality levels? * **Action owners**: Who should be notified if this dimension fails? Good documentation ensures your team can act on test failures effectively. Real-World Examples ------------------- Here are practical examples showing how organizations use dimensional validation to solve real data quality challenges. ### Example 1: E-Commerce Product Data Quality **Scenario**: An e-commerce company wants to ensure product descriptions are complete across all categories. **Setup**: * Table: `product_catalog` * Column: `product_description` * Test Type: Column Values to be Not Null * Dimension: `product_category` **Results**: * Electronics: 98% complete (excellent) * Clothing: 72% complete (needs attention) * Home & Garden: 89% complete (good) * Sports: 94% complete (good) **Action**: The content team discovers that clothing descriptions are frequently missing. They prioritize improving clothing product content and implement stricter submission requirements for that category. ### Example 2: Multi-Region Customer Email Validation **Scenario**: A global SaaS company needs to validate email formats across different regional databases. **Setup**: * Table: `customers` * Column: `email_address` * Test Type: Column Values to Match Regex Pattern * Pattern: `^[a-zA-Z0-9._%+-]+@[a-zA-Z0-9.-]+\.[a-zA-Z]{2,}$` * Dimension: `registration_region` **Results**: * North America: 99.8% valid * Europe: 99.5% valid * Asia Pacific: 96.2% valid (needs attention) * Latin America: 99.1% valid **Action**: Investigation reveals that the Asia Pacific region's customer portal has a validation bug allowing malformed emails. The engineering team fixes the bug and implements a data cleanup script. ### Example 3: Financial Transaction Amount Monitoring **Scenario**: A financial services company wants to ensure transaction amounts fall within expected ranges across different account types. **Setup**: * Table: `transactions` * Column: `transaction_amount` * Test Type: Column Values to be Between * Range: $0.01 to $10,000 * Dimension: `account_type` **Results**: * Checking: 99.9% valid * Savings: 99.8% valid * Business: 94.3% valid (needs attention) * Credit: 99.5% valid **Action**: The compliance team discovers that business accounts have more out-of-range transactions due to recent limit changes. They update account limits and implement additional monitoring for business accounts. Limitations ----------- Understanding current limitations helps you plan effective dimensional validation strategies. ### High Cardinality Performance Impact **Issue**: Dimensions with many unique values cause significant performance overhead. **Impact**: Tests may take 5-10x longer to execute compared to baseline tests. **Workaround**: * Use sampling to reduce scan volume (10-50% samples) * Create derived columns with grouped values ### "Others" Group Limitations **Issue**: The "Others" group combines all dimensions outside the top 10, which can hide specific problem areas. **Impact**: You may not see individual quality issues for dimensions ranked 11th or lower. **Workaround**: * Reduce total dimension cardinality so all important values appear in top 10 Troubleshooting --------------- ### My test returns "No results" **Possible causes**: * The test hasn't run yetβ€”check the Pipeline tab to schedule execution * Partitioning filtered out all dataβ€”verify partition configuration includes relevant dates * Sampling excluded all dimension groupsβ€”increase sample percentage **Solution**: Verify the test has executed at least once, and check partition/sample settings. ### Dimensional results show "NULL" as a dimension value **Expected behavior**: Null values in the dimension column are grouped together as a "NULL" dimension group. **If unexpected**: Review your dimension column data qualityβ€”you may need to clean up null values in the dimension column before using it for segmentation. ### Test execution takes too long **Cause**: High cardinality dimension or large table without sampling/partitioning. **Solutions**: 1. Enable sampling 2. Enable partitioning to test only recent data 3. Choose a different dimension with lower cardinality --- # Enable Security (Docker) | OpenMetadata Docker Security We use cookies to improve site navigation, analyze site usage, and enhance your user experience. Click "Accept" to enable cookies or "Reject" to reject cookies. AcceptReject deployment No menu items for this category [Home](https://docs.open-metadata.org/latest) /[Deployment](https://docs.open-metadata.org/latest/deployment) /[Docker](https://docs.open-metadata.org/latest/deployment/docker) /[Security](https://docs.open-metadata.org/latest/deployment/docker/security) OpenMetadata Documentation Docker Security =============== Follow the steps for setting up the SSO, and then check the specific `Docker` section of your chosen SSO. By default Basic Authentication will be enabled as authentication mechanism. [Basic Authentication\ \ Configure Basic Authentication to access the UI and APIs](https://docs.open-metadata.org/latest/deployment/security/basic-auth) [Ldap Authentication\ \ Configure Ldap Authentication to access the UI and APIs](https://docs.open-metadata.org/latest/deployment/security/ldap/docker) [Auth0 SSO\ \ Configure Auth0 SSO to access the UI and APIs](https://docs.open-metadata.org/latest/deployment/security/auth0/docker) [Azure SSO\ \ Configure Azure SSO to access the UI and APIs](https://docs.open-metadata.org/latest/deployment/security/azure) [Custom OIDC SSO\ \ Configure a Custom OIDC SSO to access the UI and APIs](https://docs.open-metadata.org/latest/deployment/security/custom-oidc) [Google SSO\ \ Configure Google SSO to access the UI and APIs](https://docs.open-metadata.org/latest/deployment/security/google) [Okta SSO\ \ Configure Okta SSO to access the UI and APIs](https://docs.open-metadata.org/latest/deployment/security/okta) --- # Data Quality Observability Guide | Official Documentation We use cookies to improve site navigation, analyze site usage, and enhance your user experience. Click "Accept" to enable cookies or "Reject" to reject cookies. AcceptReject ![](https://static.scarf.sh/a.png?x-pxid=6814879a-96c0-40bd-be69-ddfeb14694aa) how-to-guides No menu items for this category [Home](https://docs.open-metadata.org/latest) /[How To Guides](https://docs.open-metadata.org/latest/how-to-guides) /[Data Quality Observability](https://docs.open-metadata.org/latest/how-to-guides/data-quality-observability) OpenMetadata Documentation Data Quality and Observability ============================== OpenMetadata offers a simple and easy-to-use solution for quality and observability. With no code tests, observability metrics, incident management, and root cause analysis (Collate feature), you have a unified solution for discovery, governance, and observability. OpenMetadata ensures the health and performance of your data systems by providing comprehensive data observability features. These features offer insights into the state of test cases, helping to detect, diagnose, and resolve data issues quickly. By monitoring data flows and data quality in real-time, data teams can ensure that data remains reliable and trustworthy. OpenMetadata supports [observability alerts and notifications](https://docs.open-metadata.org/latest/how-to-guides/data-quality-observability/observability/alerts) to help you maintain the integrity and performance of your data systems. [Data Quality\ \ Deep dive into how to set up quality tests, alert and triage and resolve incidents!](https://docs.open-metadata.org/latest/how-to-guides/data-quality-observability/quality) [Data Profiler\ \ Deep dive into how to set up the profiler in OpenMetadata.](https://docs.open-metadata.org/latest/how-to-guides/data-quality-observability/profiler) [Alerts & Notifications\ \ Set up observability alerts in OpenMetadata.](https://docs.open-metadata.org/latest/how-to-guides/data-quality-observability/alerts-notifications) [Incident Manager\ \ Set up incident management in OpenMetadata.](https://docs.open-metadata.org/latest/how-to-guides/data-quality-observability/incident-manager) --- # Data Profiler | OpenMetadata Data Profiling Guide We use cookies to improve site navigation, analyze site usage, and enhance your user experience. Click "Accept" to enable cookies or "Reject" to reject cookies. AcceptReject ![](https://static.scarf.sh/a.png?x-pxid=6814879a-96c0-40bd-be69-ddfeb14694aa) how-to-guides No menu items for this category [Home](https://docs.open-metadata.org/latest) /[How To Guides](https://docs.open-metadata.org/latest/how-to-guides) /[Data Quality Observability](https://docs.open-metadata.org/latest/how-to-guides/data-quality-observability) /[Profiler](https://docs.open-metadata.org/latest/how-to-guides/data-quality-observability/profiler) OpenMetadata Documentation Overview of Data Profiler ========================= The profiler in OpenMetadata helps to understand the shape of your data and to quickly validate assumptions. The data profiler helps to capture table usage statistics over a period of time. This happens as part of profiler ingestion. Data profiles enable you to check for null values in non-null columns, for duplicates in a unique column, etc. You can gain a better understanding of column data distributions through the descriptive statistics provided. Watch the video to understand OpenMetadata’s native Data Profiler and Data Quality tests. [Profiler Tab\ \ Get a complete picture of the Table Profile and Column Profile details.](https://docs.open-metadata.org/latest/how-to-guides/data-quality-observability/profiler/tab) [Profiler Workflow\ \ Configure and run the Profiler Workflow to extract Profiler data.](https://docs.open-metadata.org/latest/how-to-guides/data-quality-observability/profiler/workflow) [Metrics\ \ Learn about the supported profiler metrics.](https://docs.open-metadata.org/latest/how-to-guides/data-quality-observability/profiler/metrics) [External Workflow\ \ Run a single workflow profiler for the entire source externally.](https://docs.open-metadata.org/latest/how-to-guides/data-quality-observability/profiler/external-workflow) --- # How to Manually Add or Edit Lineage We use cookies to improve site navigation, analyze site usage, and enhance your user experience. Click "Accept" to enable cookies or "Reject" to reject cookies. AcceptReject ![](https://static.scarf.sh/a.png?x-pxid=6814879a-96c0-40bd-be69-ddfeb14694aa) how-to-guides No menu items for this category [Home](https://docs.open-metadata.org/latest) /[How To Guides](https://docs.open-metadata.org/latest/how-to-guides) /[Data Lineage](https://docs.open-metadata.org/latest/how-to-guides/data-lineage) /[Manual](https://docs.open-metadata.org/latest/how-to-guides/data-lineage/manual) OpenMetadata Documentation How to Manually Add or Edit Lineage =================================== Edit lineage to provide a richer understanding of the provenance of data. The OpenMetadata no-code editor provides a drag and drop interface. Drop tables, topics, pipelines, dashboards, ML models, containers, and pipelines onto the lineage graph. You may add new edges or delete existing edges to better represent data lineage. OpenMetadata supports manual editing of both table and column level lineage. We can build the lineage by creating edges. You can connect the source of the lineage to the destination by connecting the nodes. Once you have ingested your database and dashboard services. * Start by picking one database service, and select a table. In the data asset details page, navigate to the Lineage Tab. * Click on the Edit option to enable the lineage editor. * Select the type of data asset (table, topic, dashboard, ML model, container, pipeline) to connect to as the destination. ![Data Asset: Lineage Tab](https://docs.open-metadata.org/images/v1.11/how-to-guides/lineage/l1.png) Data Asset: Lineage Tab * Search and select the relevant data asset. * Create an edge between these two data assets. ![Link the Table to the Dashboard to Add Lineage Manually](https://docs.open-metadata.org/images/v1.11/how-to-guides/lineage/l2.png) Link the Table to the Dashboard to Add Lineage Manually * You can also expand a table to view the available columns * Link the relevant columns together by connecting the column edges to trace column-level lineage. ![Column-Level Lineage](https://docs.open-metadata.org/images/v1.11/how-to-guides/lineage/l3.png) Column-Level Lineage Here's a quick video on manually adding lineage. Watch the recording of the Webinar on Lineage (13:30 to 15:50) --- # Try OpenMetadata in Docker We use cookies to improve site navigation, analyze site usage, and enhance your user experience. Click "Accept" to enable cookies or "Reject" to reject cookies. AcceptReject ![](https://static.scarf.sh/a.png?x-pxid=466d9050-5f23-44d9-9533-b0a7a1417622) quick-start No menu items for this category [Home](https://docs.open-metadata.org/latest) /[Quick Start](https://docs.open-metadata.org/latest/quick-start) /[Local Docker Deployment](https://docs.open-metadata.org/latest/quick-start/local-docker-deployment) OpenMetadata Documentation Local Docker Deployment ======================= This installation doc will help you start a OpenMetadata standalone instance on your local machine. If you'd rather see the steps in a guided tutorial, we've got you covered! Otherwise, feel free to read the content below πŸ‘‡ Requirements (OSX, Linux and Windows) ===================================== Please ensure your host system meets the requirements listed below. Then continue to the Procedure for installing OpenMetadata. OSX and Linux ------------- ### Docker (version 20.10.0 or greater) [Docker](https://docs.docker.com/get-started/overview/) is an open-source platform for developing, shipping, and running applications. It enables you to separate your applications from your infrastructure, so you can deliver software quickly using OS-level virtualization. It helps deliver software in packages called Containers. To check the version of Docker you have, use the following command. If you need to install Docker, please visit [Get Docker](https://docs.docker.com/get-docker/) . You must allocate at least `6 GiB` of memory and `4 vCPUs` to Docker in order to run OpenMetadata. To change the memory allocation for Docker, please visit `Preferences -> Resources -> Advanced` in your Docker Desktop. ### Docker Compose (version v2.1.1 or greater) The Docker `compose` package enables you to define and run multi-container Docker applications. The compose command integrates compose functions into the Docker platform, making them available from the Docker command-line interface ( CLI). The Python packages you will install in the procedure below use compose to deploy OpenMetadata. * **MacOS X**: Docker on MacOS X ships with compose already available in the Docker CLI. * **Linux**: To install compose on Linux systems, please visit the Docker CLI command documentation and follow the instructions. To verify that the docker compose command is installed and accessible on your system, run the following command. Upon running this command you should see output similar to the following. ### Install Docker Compose Version 2.0.0 on Linux Follow the instructions [here](https://docs.docker.com/compose/cli-command/#install-on-linux) to install docker compose version 2.0.0 1. Run the following command to download the current stable release of Docker Compose This command installs Compose V2 for the active user under $HOME directory. To install Docker Compose for all users on your system, replace `~/.docker/cli-plugins` with `/usr/local/lib/docker/cli-plugins`. 2. Apply executable permissions to the binary 3. Test your installation Windows ------- ### WSL2, Ubuntu 20.04, and Docker for Windows * Install [WSL2](https://ubuntu.com/wsl) * Install [Ubuntu 20.04](https://www.microsoft.com/en-us/p/ubuntu-2004-lts/9n6svws3rx71) * Install [Docker for Windows](https://www.docker.com/products/docker-desktop) * Once installed, please follow the steps [here](https://docs.docker.com/desktop/windows/wsl/) and complete all the pre-requisites for a seamless installation and deployment. * After completion of the pre-requisites, please install `python3-pip` and `python3-venv` on your Ubuntu system. * Command: `apt install python3-pip python3-venv` (Ensure that you have the privilege to install packages, if not, please use Super User.) Procedure --------- ### 1\. Create a directory for OpenMetadata Create a new directory for OpenMetadata and navigate into that directory. ### 2\. Download Docker Compose File from GitHub Releases Download the docker-compose.yml file from the release page [here](https://github.com/open-metadata/OpenMetadata/releases/latest) . The latest version is at the top of the page * Deploying with MySQL: Download `docker-compose.yml` file from the above link. * Deploying with PostgreSQL: Download `docker-compose-postgres.yml` file from the above link. You can use the curl or wget command as well to fetch the docker compose files from your terminal - ### 3\. Start the Docker Compose Services Run the below command to deploy the OpenMetadata For OpenMetadata with MySQL Database - For OpenMetadata with PostgreSQL Database - These commands will pull the docker images of Openmetadata for MySQL / PostgreSQL, OpenMetadata-Server, OpenMetadata-Ingestion and Elasticsearch. Upon running this command you should see output similar to the following. You can validate that all containers are up by running with command `docker ps`. In a few seconds, you should be able to access the OpenMetadata UI at [http://localhost:8585](http://localhost:8585/) By default, we ship Docker Compose with [host and docker named volume mapping](https://docs.docker.com/storage/) for MySQL, PostgreSQL, ElasticSearch and Ingestion Services with quickstart docker compose services. This will be available under `docker-volume` directory on host machine in the same path as docker compose files. Log in to OpenMetadata ---------------------- OpenMetadata provides a default admin account to login. You can access OpenMetadata at [http://localhost:8585](http://localhost:8585/) . Use the following credentials to log in to OpenMetadata. * Username: `admin@open-metadata.org` * Password: `admin` Once you log in, you can goto Settings -> Users to add another user and make them admin as well. Log in to Airflow ----------------- OpenMetadata ships with an Airflow container to run the ingestion workflows that have been deployed via the UI. In the Airflow, you will also see some sample DAGs that will ingest sample data and serve as an example. You can access Airflow at [http://localhost:8080](http://localhost:8080/) . Use the following credentials to log in to Airflow. * Username: `admin` * Password: `admin` ### Customizing Airflow Admin Credentials: When using Docker Compose, you can change the default Airflow admin credentials by setting the following environment variables: * Username: `AIRFLOW_ADMIN_USER` * Password: `AIRFLOW_ADMIN_PASSWORD` Airflow DAGs Showcased in Deployment ------------------------------------ You can explore the examples of Airflow DAGs used with OpenMetadata. Refer [here](https://github.com/open-metadata/OpenMetadata/tree/main/ingestion/examples/airflow/dags) for more information. ![DAG_Examples](https://docs.open-metadata.org/images/v1.11/quickstart/docker/DAG_Examples.png) Go on a tour and start discovering the power of metadata & collaboration ------------------------------------------------------------------------ ![tour](https://docs.open-metadata.org/images/v1.11/quickstart/tour.png) Start and Stop -------------- From the same directory mentioned in [step 1](https://docs.open-metadata.org/latest/quick-start/local-docker-deployment#1.-create-a-directory-for-openmetadata) , use the following commands to start and stop the Docker Compose services. To stop the services To start the services Start and stop are used to completely halt or restart the running services. When services are stopped, their containers are shut down but remain available for restarting without rebuilding. Importantly, any data stored in Docker volumes remains unaffected during this process. The volumes stay intact and accessible, preserving your application’s state and making it easy to restart the services without losing data. This makes it a reliable option for temporary shutdowns while maintaining continuity. Cleanup ------- To stop the Docker Compose services, run the following command from the same directory mentioned in [step 1](https://docs.open-metadata.org/latest/quick-start/local-docker-deployment#1.-create-a-directory-for-openmetadata) . Stop the services If you want to clean up the associated named volumes as well, use the following command Named volumes are used to persist data created by containers, ensuring that the data remains even after the containers are stopped or removed. These volumes are managed by Docker and stored independently from the containers. Using the `--volumes` flag with the `docker compose down` command will delete these volumes, permanently removing all stored data. Troubleshooting --------------- ### Compose is not a docker command If you are getting an error such as `"compose" is not a docker command`, you might need to revisit the installation steps above to make sure that Docker Compose is properly added to your system. ### Network openmetadata\_app\_net Error You might see something like: A common solution is to run `docker network prune`: So be careful if you want to keep up some (unused) networks from your laptop. ### Connect Host Services from Docker Container You can connect Docker containers to communicate with Host Operating System Services. Navigate to the [official docker documentation](https://docs.docker.com/desktop/networking/#i-want-to-connect-from-a-container-to-a-service-on-the-host) which will help achieve the same. Security -------- Please follow our [Enable Security Guide](https://docs.open-metadata.org/latest/deployment/docker/security) to configure security for your OpenMetadata installation. Next Steps ---------- 1. Refer the [How-to Guides](https://docs.open-metadata.org/latest/how-to-guides) for an overview of all the features in OpenMetadata. 2. Visit the [Connectors](https://docs.open-metadata.org/latest/connectors) documentation to see what services you can integrate with OpenMetadata. 3. Visit the [API](https://docs.open-metadata.org/swagger.html) documentation and explore the rich set of OpenMetadata APIs. ### Volume Permissions: Operation not permitted If you are running on Windows (WSL2) and see permissions errors when starting the databases (either MySQL or Postgres), e.g., You can try to update the `/etc/wsl.conf` file from the WSL2 machine to add: --- # SSO for Bare Metal | Official Documentation We use cookies to improve site navigation, analyze site usage, and enhance your user experience. Click "Accept" to enable cookies or "Reject" to reject cookies. AcceptReject ![](https://static.scarf.sh/a.png?x-pxid=8472f2b3-d307-4935-a5ed-3f9d3982c15d) deployment No menu items for this category [Home](https://docs.open-metadata.org/latest) /[Deployment](https://docs.open-metadata.org/latest/deployment) /[Security](https://docs.open-metadata.org/latest/deployment/security) /[Configuration](https://docs.open-metadata.org/latest/deployment/security/configuration) /[Bare Metal](https://docs.open-metadata.org/latest/deployment/security/configuration/bare-metal) OpenMetadata Documentation SSO for Bare Metal ================== Update conf/openmetadata.yaml ----------------------------- Once the `Client Id` is generated, add the `Client Id` in `openmetadata.yaml` file in `client_id` field. Update the `providerName` config to the name you want to display in the `Sign In` button in the UI. For example, with the following configuration with `providerName` set to `KeyCloak`, the users will see `Sign In with KeyCloak SSO` in the `Sign In` page of the OpenMetadata UI. The configuration values provided below are examples. Update them as required to match your specific environment and authentication settings. Then, * Update `authorizerConfiguration` to add login names of the admin users in `adminPrincipals` section as shown below. * Update the `principalDomain` to your company domain name. Configure Ingestion ------------------- Once your server security is set, it's time to review the ingestion configuration. Our bots support JWT tokens to authenticate to the server when sending requests. Find more information on [**Enabling JWT Tokens**](https://docs.open-metadata.org/latest/deployment/security/enable-jwt-tokens) and [**JWT Troubleshooting**](https://docs.open-metadata.org/latest/deployment/security/jwt-troubleshooting) to ensure seamless authentication. ### Troubleshooting * If you are seeing the below trace in the logs, you need to add the discovery URL To resolve the error regarding the discovery URL, you need to set the `AUTHENTICATION_DISCOVERY_URI` in your configuration. This URI is used to discover the OpenID Connect provider's configuration. --- # Domo-Pipeline Troubleshooting Guide | OpenMetadata Support We use cookies to improve site navigation, analyze site usage, and enhance your user experience. Click "Accept" to enable cookies or "Reject" to reject cookies. AcceptReject connectors No menu items for this category [Home](https://docs.open-metadata.org/latest) /[Connectors](https://docs.open-metadata.org/latest/connectors) /[Pipeline](https://docs.open-metadata.org/latest/connectors/pipeline) /[Domo Pipeline](https://docs.open-metadata.org/latest/connectors/pipeline/domo-pipeline) /[Troubleshooting](https://docs.open-metadata.org/latest/connectors/pipeline/domo-pipeline/troubleshooting) OpenMetadata Documentation Troubleshooting =============== Workflow Deployment Error ------------------------- If there were any errors during the workflow deployment process, the Ingestion Pipeline Entity will still be created, but no workflow will be present in the Ingestion container. * You can then Edit the Ingestion Pipeline and **Deploy** it again. * From the Connection tab, you can also Edit the Service if needed. Connector Debug Troubleshooting ------------------------------- This section provides instructions to help resolve common issues encountered during connector setup and metadata ingestion in OpenMetadata. Below are some of the most frequently observed troubleshooting scenarios. How to Enable Debug Logging for Any Ingestion --------------------------------------------- To enable debug logging for any ingestion workflow in OpenMetadata: 1. **Navigate to Services** Go to **Settings > Services > Service Type** (e.g., Database) in the OpenMetadata UI. 2. **Select a Service** Choose the specific service for which you want to enable debug logging. ![Select a Service](https://docs.open-metadata.org/images/v1.11/connectors/debug/debug1.png) Select a Service 3. **Access Ingestion Tab** Go to the **Ingestion tab** and click the three-dot menu on the right-hand side of the ingestion type, and select Edit. ![Access Agents Tab](https://docs.open-metadata.org/images/v1.11/connectors/debug/debug2.png) Access Agents Tab 4. **Enable Debug Logging** In the configuration dialog, enable the **Debug Log** option and click **Next**. ![Enable Debug Logging](https://docs.open-metadata.org/images/v1.11/connectors/debug/debug3.png) Enable Debug Logging 5. **Schedule and Submit** Configure the schedule if needed and click **Submit** to apply the changes. ![Schedule and Submit](https://docs.open-metadata.org/images/v1.11/connectors/debug/debug4.png) Schedule and Submit Permission Issues ----------------- If you encounter permission-related errors during connector setup or metadata ingestion, ensure that all the prerequisites and access configurations specified for each connector are properly implemented. Refer to the connector-specific documentation to verify the required permissions. --- # Tests - YAML Config | OpenMetadata Quality Config Guide We use cookies to improve site navigation, analyze site usage, and enhance your user experience. Click "Accept" to enable cookies or "Reject" to reject cookies. AcceptReject ![](https://static.scarf.sh/a.png?x-pxid=6814879a-96c0-40bd-be69-ddfeb14694aa) how-to-guides No menu items for this category [Home](https://docs.open-metadata.org/latest) /[How To Guides](https://docs.open-metadata.org/latest/how-to-guides) /[Data Quality Observability](https://docs.open-metadata.org/latest/how-to-guides/data-quality-observability) /[Quality](https://docs.open-metadata.org/latest/how-to-guides/data-quality-observability/quality) /[Tests Yaml](https://docs.open-metadata.org/latest/how-to-guides/data-quality-observability/quality/tests-yaml) OpenMetadata Documentation Tests in the YAML Config ======================== Here you can see all the supported tests definitions and how to configure them in the YAML config file. A **Test Definition** is a generic definition of a test. This Test Definition then gets specified in a Test Case. This Test Case is where the parameter(s) of a Test Definition are specified. In this section, you will learn what tests we currently support and how to configure them in the YAML/JSON config file. * [Table Tests](https://docs.open-metadata.org/latest/how-to-guides/data-quality-observability/quality/tests-yaml#table-tests) * [Column Tests](https://docs.open-metadata.org/latest/how-to-guides/data-quality-observability/quality/tests-yaml#column-tests) Table Tests ----------- Tests applied on top of a Table. Here is the list of all table tests: * [Table Row Count to Equal](https://docs.open-metadata.org/latest/how-to-guides/data-quality-observability/quality/tests-yaml#table-row-count-to-equal) * [Table Row Count to be Between](https://docs.open-metadata.org/latest/how-to-guides/data-quality-observability/quality/tests-yaml#table-row-count-to-be-between) * [Table Column Count to Equal](https://docs.open-metadata.org/latest/how-to-guides/data-quality-observability/quality/tests-yaml#table-column-count-to-equal) * [Table Column Count to be Between](https://docs.open-metadata.org/latest/how-to-guides/data-quality-observability/quality/tests-yaml#table-column-count-to-be-between) * [Table Column Name to Exist](https://docs.open-metadata.org/latest/how-to-guides/data-quality-observability/quality/tests-yaml#table-column-name-to-exist) * [Table Column to Match Set](https://docs.open-metadata.org/latest/how-to-guides/data-quality-observability/quality/tests-yaml#table-column-to-match-set) * [Table Custom SQL Test](https://docs.open-metadata.org/latest/how-to-guides/data-quality-observability/quality/tests-yaml#table-custom-sql-test) * [Table Row Inserted Count To Be Between](https://docs.open-metadata.org/latest/how-to-guides/data-quality-observability/quality/tests-yaml#table-row-inserted-count-to-be-between) * [Compare 2 Tables for Differences](https://docs.open-metadata.org/latest/how-to-guides/data-quality-observability/quality/tests-yaml#compare-2-tables-for-differences) * [Table Data to Be Fresh \[Collate\]](https://docs.open-metadata.org/latest/how-to-guides/data-quality-observability/quality/tests-yaml#table-data-to-be-fresh-collate) ### Table Row Count to Equal Validate the total row count in the table is equal to the given value. **Dimension**: Integrity **Properties**: * `value`: Expected number of rows. **Behavior** | Condition | Status | | --- | --- | | `value` **match** the number of rows in the table | Success βœ… | | `value` **does not match** the number of rows in the table | Failed ❌ | **YAML Config** **JSON Config** ### Table Row Count to be Between Validate the total row count is within a given range of values. **Dimension**: Integrity **Properties**: * `minValue`: Lower bound of the interval. If informed, the number of rows should be bigger than this number. * `maxValue`: Upper bound of the interval. If informed, the number of rows should be lower than this number. Any of those two need to be informed. **Behavior** | Condition | Status | | --- | --- | | The number of rows in the table **is between** `minValue` and `maxValue` | Success βœ… | | The number of rows in the table **is not between** `minValue` and `maxValue` | Failed ❌ | **YAML Config** **JSON Config** ### Table Column Count to Equal Validate that the number of columns in a table is equal to a given value. **Dimension**: Integrity **Properties** * `columnCount`: Expected number of columns. **Behavior** | Condition | Status | | --- | --- | | `columnCount` **matches** the number of column in the table | Success βœ… | | `columnCount` **does not matches** the number of column in the table | Failed ❌ | **YAML Config** **JSON Config** ### Table Column Count to be Between Validate the number of columns in a table is between the given value **Dimension**: Integrity **Properties** * `minColValue`: lower bound * `maxColValue`: upper bound **Behavior** | Condition | Status | | --- | --- | | The number of columns in the table **is between** `minColValue` and `maxColValue` | Success βœ… | | The number of columns in the table **is not between** `minColValue` and `maxColValue` | Failed ❌ | **YAML Config** **JSON Config** ### Table Column Name to Exist Validate a column name is present in the table **Dimension**: Integrity **Properties** * `columnName`: the name of the column to check for **Behavior** | Condition | Status | | --- | --- | | `columnName` **exists** in the set of column name for the table | Success βœ… | | `columnName` **does not exists** in the set of column name for the table | Failed ❌ | **YAML Config** **JSON Config** ### Table Column to Match Set Validate a list of table column name matches an expected set of columns **Dimension**: Integrity **Properties** * `columnNames`: comma separated string of column name * `ordered`: whether the test should check for column ordering. Default to False **Behavior** | Condition | Status | | --- | --- | | \[`ordered=False`\] `columnNames` **matches** the list of column names in the table **regardless of the order** | Success βœ… | | \[`ordered=True`\] `columnNames` **matches** the list of column names in the table **in the corresponding order** (e.g. `["a","b"] == ["a","b"]` | Success βœ… | | \[`ordered=FALSE`\] `columnNames` **does no match** the list of column names in the table **regardless of the order** | Failed ❌ | | \[`ordered=True`\] `columnNames` **does no match** the list of column names in the table **and/or the corresponding order** (e.g. `["a","b"] != ["b","a"]` | Failed ❌ | **YAML Config** **JSON Config** ### Table Custom SQL Test Write you own SQL test. When writing your query you can use 2 strategies: * `ROWS` (default): expects the query to be written as `SELECT , FROM WHERE `. **Note** if your query returns a large amount of rows it might cause an "Out Of Memory" error. In this case we recommend you to use the `COUNT` strategy. * `COUNT`: expects the query to be written as `SELECT COUNT() FROM WHERE `. **How to use the Threshold Parameter?** The threshold allows you to define a limit for which you test should pass or fail - by default this number is 0. For example if my custom SQL query test returns 10 rows (or a COUNT value of 10) and my threshold is 5 the test will fail. If I update my threshold to 11 on my next run my test will pass. * When configuring a **Table Custom SQL Test**, specify the table using the format `database.schema.table`. * Using only the table name may not work, as it depends on the SQL engine's requirements. **Properties** * `sqlExpression`: SQL expression * `strategy`: one of `ROWS` or `COUNT` * `threshold`: an integer defining the threshold above which the test should fail (default to 0 if not specified) **Behavior** | Condition | Status | | --- | --- | | `sqlExpression` returns **row <= threshold (default to 0)** | Success βœ… | | `sqlExpression` returns **row > threshold (default to 0)** | Failed ❌ | **Example** **YAML Config** **JSON Config** ### Table Row Inserted Count To Be Between Validate the number of rows inserted for the defined period is between the expected range The Table Row Inserted Count To Be Between cannot be executed against tables that have configured a partition in OpenMetadata. The logic of the test performed will be similar to executing a Table Row Count to be Between test against a table with a partition configured. **Dimension**: Integrity **Properties** * `Min Row Count`: Lower bound * `Max Row Count`: Upper bound * `Column Name`: The name of the column used to apply the range filter * `Range Type`: One of `HOUR`, `DAY`, `MONTH`, `YEAR` * `Interval`: The range interval (e.g. 1,2,3,4,5, etc) **Behavior** | Condition | Status | | --- | --- | | Number of rows **is between** `Min Row Count` and `Max Row Count` | Success βœ… | | Number of rows **is not between** `Min Row Count` and \`Max Row Count | Failed ❌ | **YAML Config** **JSON Config** ### Compare 2 Tables for Differences Compare 2 tables for differences. Allows a user to check for integrity. Supports comparing tables across different services. For example, you can compare a table in Snowflake with a table in Redshift. Supported connectors: * Snowflake * BigQuery * Athena * Redshift * Postgres * MySQL * MSSQL * Oracle * Trino * SAP Hana **Dimension**: Consistency **Properties** * `keyColumns`: The key column to use as the key for the comparison. Resolves to the primary key (if defined) if not set * `useColumns`: The columns against which the comparison will done. If not provided it will use all the columns * `table2`: The table against which the comparison will be done. Must be the fully qualified name as defined in OpenMetadata * `threshold`: The threshold of different rows above which the test should fail -- default to 0 * `where`: Any `where` clause to pass * `caseSensitiveColumns`: Whether the column comparison should be case sensitive or not. Default to `false`. **Behavior** | Condition | Status | | --- | --- | | Number of rows **is greater** than the threshold (default to 0) | Failed ❌ | | Number of rows **is less than or equal** to the threshold | Success βœ… | **YAML Config** **JSON Config** ### Table Data to Be Fresh \[Collate\] Validate the freshness of a table's data. **Dimension**: Accuracy **Properties** * `column`: the column that will be used to check the table freshness * `timeSinceUpdate`: (in seconds) The data is expected to be updated within this number of seconds. If the time since the last update is greater than this value, the test will fail. **Behavior** | Condition | Status | | --- | --- | | Time since update is greater than **timeSinceUpdate** | Failed ❌ | | Time since update is less than or equal to **timeSinceUpdate** | Success βœ… | **YAML Config** **JSON Config** Column Tests ------------ Tests applied on top of Column metrics. Here is the list of all column tests: * [Column Values to Be Unique](https://docs.open-metadata.org/latest/how-to-guides/data-quality-observability/quality/tests-yaml#column-values-to-be-unique) * [Column Values to Be Not Null](https://docs.open-metadata.org/latest/how-to-guides/data-quality-observability/quality/tests-yaml#column-values-to-be-not-null) * [Column Values to Match Regex](https://docs.open-metadata.org/latest/how-to-guides/data-quality-observability/quality/tests-yaml#column-values-to-match-regex) * [Column Values to not Match Regex](https://docs.open-metadata.org/latest/how-to-guides/data-quality-observability/quality/tests-yaml#column-values-to-not-match-regex) * [Column Values to Be in Set](https://docs.open-metadata.org/latest/how-to-guides/data-quality-observability/quality/tests-yaml#column-values-to-be-in-set) * [Column Values to Be Not In Set](https://docs.open-metadata.org/latest/how-to-guides/data-quality-observability/quality/tests-yaml#column-values-to-be-not-in-set) * [Column Values to Be Between](https://docs.open-metadata.org/latest/how-to-guides/data-quality-observability/quality/tests-yaml#column-values-to-be-between) * [Column Values Missing Count to Be Equal](https://docs.open-metadata.org/latest/how-to-guides/data-quality-observability/quality/tests-yaml#column-values-missing-count-to-be-equal) * [Column Values Lengths to Be Between](https://docs.open-metadata.org/latest/how-to-guides/data-quality-observability/quality/tests-yaml#column-values-lengths-to-be-between) * [Column Value Max to Be Between](https://docs.open-metadata.org/latest/how-to-guides/data-quality-observability/quality/tests-yaml#column-value-max-to-be-between) * [Column Value Min to Be Between](https://docs.open-metadata.org/latest/how-to-guides/data-quality-observability/quality/tests-yaml#column-value-min-to-be-between) * [Column Value Mean to Be Between](https://docs.open-metadata.org/latest/how-to-guides/data-quality-observability/quality/tests-yaml#column-value-mean-to-be-between) * [Column Value Median to Be Between](https://docs.open-metadata.org/latest/how-to-guides/data-quality-observability/quality/tests-yaml#column-value-median-to-be-between) * [Column Values Sum to Be Between](https://docs.open-metadata.org/latest/how-to-guides/data-quality-observability/quality/tests-yaml#column-values-sum-to-be-between) * [Column Values Standard Deviation to Be Between](https://docs.open-metadata.org/latest/how-to-guides/data-quality-observability/quality/tests-yaml#column-values-standard-deviation-to-be-between) * [Column Values To Be At Expected Location](https://docs.open-metadata.org/latest/how-to-guides/data-quality-observability/quality/tests-yaml#column-values-to-be-at-expected-location) ### Column Values to Be Unique Makes sure that there are no duplicate values in a given column. **Dimension**: Uniqueness **Behavior** | Condition | Status | | --- | --- | | column values are unique | Success βœ… | | column values are not unique | Failed ❌ | **Properties** * `columnValuesToBeUnique`: To be set as `true`. This is required for proper JSON parsing in the profiler module. **YAML Config** **JSON Config** ### Column Values to Be Not Null Validates that there are no null values in the column. **Dimension**: Completeness **Properties** * `columnValuesToBeNotNull`: To be set as `true`. This is required for proper JSON parsing in the profiler module. **Behavior** | Condition | Status | | --- | --- | | No `NULL` values are present in the column | Success βœ… | | 1 or more `NULL` values are present in the column | Failed ❌ | **YAML Config** **JSON Config** ### Column Values to Match Regex This test allows us to specify how many values in a column we expect that will match a certain regex expression. Please note that for certain databases we will fall back to SQL `LIKE` expression. The databases supporting regex pattern as of 0.13.2 are: * redshift * postgres * oracle * mysql * mariaDB * sqlite * clickhouse * snowflake The other databases will fall back to the `LIKE` expression **Dimension**: Validity **Properties** * `regex`: expression to match a regex pattern. E.g., `[a-zA-Z0-9]{5}`. **Behavior** | Condition | Status | | --- | --- | | All column values match `regex` | Success βœ… | | 1 or more column values do not match `regex` | Failed ❌ | **YAML Config** **JSON Config** ### Column Values to not Match Regex This test allows us to specify values in a column we expect that will not match a certain regex expression. If the test find values matching the `forbiddenRegex` the test will fail. Please note that for certain databases we will fall back to SQL `LIKE` expression. The databases supporting regex pattern as of 0.13.2 are: * redshift * postgres * oracle * mysql * mariaDB * sqlite * clickhouse * snowflake The other databases will fall back to the `LIKE` expression **Dimension**: Validity **Properties** * `regex`: expression to match a regex pattern. E.g., `[a-zA-Z0-9]{5}`. **Behavior** | Condition | Status | | --- | --- | | 0 column value match `regex` | Success βœ… | | 1 or more column values match `regex` | Failed ❌ | **YAML Config** **JSON Config** ### Column Values to Be in Set Validate values form a set are present in a column. **Dimension**: Validity **Properties** * `allowedValues`: List of allowed strings or numbers. **Behavior** | Condition | Status | | --- | --- | | `matchEnum` is `false` and 1 or more values from `allowedValues` is found in the column | Success βœ… | | `matchEnum` is `true` and all columns have a value from `allowedValues` | Success βœ… | | `matchEnum` is `false` 0 value from `allowedValues` is found in the column | Failed ❌ | | `matchEnum` is `true` and 1 or more columns does not have a vluae from `allowedValues` | Failed ❌ | **YAML Config** **JSON Config** **JSON Config** ### Column Values to Be Not In Set Validate that there are no values in a column in a set of forbidden values. **Dimension**: Validity **Properties** * `forbiddenValues`: List of forbidden strings or numbers. **Behavior** | Condition | Status | | --- | --- | | 0 value from `forbiddenValues` is found in the column | Success βœ… | | 1 or more values from `forbiddenValues` is found in the column | Failed ❌ | **YAML Config** **JSON Config** ### Column Values to Be Between Validate that the values of a column are within a given range. > Only supports numerical types. **Dimension**: Accuracy **Properties** * `minValue`: Lower bound of the interval. If informed, the column values should be bigger than this number. * `maxValue`: Upper bound of the interval. If informed, the column values should be lower than this number. Any of those two need to be informed. **Behavior** | Condition | Status | | --- | --- | | value is **between** `minValue` and `maxValue` | Success βœ… | | value is **greater** than `minValue` if only `minValue` is specified | Success βœ… | | value is **less** then `maxValue` if only `maxValue` is specified | Success βœ… | | value is **not between** `minValue` and `maxValue` | Failed ❌ | | value is **less** than `minValue` if only `minValue` is specified | Failed ❌ | | value is **greater** then `maxValue` if only `maxValue` is specified | Failed ❌ | **YAML Config** **JSON Config** ### Column Values Missing Count to Be Equal Validates that the number of missing values matches a given number. Missing values are the sum of nulls, plus the sum of values in a given list which we need to consider as missing data. A clear example of that would be `NA` or `N/A`. **Dimension**: Completeness **Properties** * `missingCountValue`: The number of missing values needs to be equal to this. This field is mandatory. * `missingValueMatch` (Optional): A list of strings to consider as missing values. **Behavior** | Condition | Status | | --- | --- | | Number of missing value is **equal** to `missingCountValue` | Success βœ… | | Number of missing value is **not equal** to `missingCountValue` | Failed ❌ | **YAML Config** **JSON Config** **JSON Config** ### Column Values Lengths to Be Between Validates that the lengths of the strings in a column are within a given range. > Only supports concatenable types. **Dimension**: Accuracy **Properties** * `minLength`: Lower bound of the interval. If informed, the string length should be bigger than this number. * `maxLength`: Upper bound of the interval. If informed, the string length should be lower than this number. Any of those two need to be informed. **Behavior** | Condition | Status | | --- | --- | | value length is **between** `minLength` and `maxLength` | Success βœ… | | value length is **greater** than `minLength` if only `minLength` is specified | Success βœ… | | value length is **less** then `maxLength` if only `maxLength` is specified | Success βœ… | | value length is **not between** `minLength` and `maxLength` | Failed ❌ | | value length is **less** than `minLength` if only `minLength` is specified | Failed ❌ | | value length is **greater** then `maxLength` if only `maxLength` is specified | Failed ❌ | **YAML Config** **JSON Config** ### Column Value Max to Be Between Validate the maximum value of a column is between a specific range > Only supports numerical types. **Dimension**: Accuracy **Properties** * `minValueForMaxInCol`: lower bound * `maxValueForMaxInCol`: upper bound **Behavior** | Condition | Status | | --- | --- | | column max value is **between** `minValueForMaxInCol` and `maxValueForMaxInCol` | Success βœ… | | column max value is **greater** than `minValueForMaxInCol` if only `minValueForMaxInCol` is specified | Success βœ… | | column max value is **less** then `maxValueForMaxInCol` if only `maxValueForMaxInCol` is specified | Success βœ… | | column max value is **not between** `minValueForMaxInCol` and `maxValueForMaxInCol` | Failed ❌ | | column max value is **less** than `minValueForMaxInCol` if only `minValueForMaxInCol` is specified | Failed ❌ | | column max value is **greater** then `maxValueForMaxInCol` if only `maxValueForMaxInCol` is specified | Failed ❌ | **YAML Config** **JSON Config** ### Column Value Min to Be Between Validate the minimum value of a column is between a specific range > Only supports numerical types. **Dimension**: Accuracy **Properties** * `minValueForMinInCol`: lower bound * `maxValueForMinInCol`: upper bound **Behavior** | Condition | Status | | --- | --- | | column min value is **between** `minValueForMinInCol` and `maxValueForMinInCol` | Success βœ… | | column min value is **greater** than `minValueForMinInCol` if only `minValueForMinInCol` is specified | Success βœ… | | column min value is **less** then `maxValueForMinInCol` if only `maxValueForMinInCol` is specified | Success βœ… | | column min value is **not between** `minValueForMinInCol` and `maxValueForMinInCol` | Failed ❌ | | column min value is **less** than `minValueForMinInCol` if only `minValueForMinInCol` is specified | Failed ❌ | | column min value is **greater** then `maxValueForMinInCol` if only `maxValueForMinInCol` is specified | Failed ❌ | **YAML Config** **JSON Config** ### Column Value Mean to Be Between Validate the mean of a column is between a specific range > Only supports numerical types. **Dimension**: Accuracy **Properties** * `minValueForMeanInCol`: lower bound * `maxValueForMeanInCol`: upper bound **Behavior** | Condition | Status | | --- | --- | | column mean value is **between** `minValueForMeanInCol` and `maxValueForMeanInCol` | Success βœ… | | column mean value is **greater** than `minValueForMeanInCol` if only `minValueForMeanInCol` is specified | Success βœ… | | column mean value is **less** then `maxValueForMeanInCol` if only `maxValueForMeanInCol` is specified | Success βœ… | | column mean value is **not between** `minValueForMeanInCol` and `maxValueForMeanInCol` | Failed ❌ | | column mean value is **less** than `minValueForMeanInCol` if only `minValueForMeanInCol` is specified | Failed ❌ | | column mean value is **greater** then `maxValueForMeanInCol` if only `maxValueForMeanInCol` is specified | Failed ❌ | **YAML Config** **JSON Config** ### Column Value Median to Be Between Validate the median of a column is between a specific range > Only supports numerical types. **Dimension**: Accuracy **Properties** * `minValueForMedianInCol`: lower bound * `maxValueForMedianInCol`: upper bound **Behavior** | Condition | Status | | --- | --- | | column median value is **between** `minValueForMedianInCol` and `maxValueForMedianInCol` | Success βœ… | | column median value is **greater** than `minValueForMedianInCol` if only `minValueForMedianInCol` is specified | Success βœ… | | column median value is **less** then `maxValueForMedianInCol` if only `maxValueForMedianInCol` is specified | Success βœ… | | column median value is **not between** `minValueForMedianInCol` and `maxValueForMedianInCol` | Failed ❌ | | column median value is **less** than `minValueForMedianInCol` if only `minValueForMedianInCol` is specified | Failed ❌ | | column median value is **greater** then `maxValueForMedianInCol` if only `maxValueForMedianInCol` is specified | Failed ❌ | **YAML Config** **JSON Config** ### Column Values Sum to Be Between Validate the sum of a column is between a specific range > Only supports numerical types. **Dimension**: Accuracy **Properties** * `minValueForColSum`: lower bound * `maxValueForColSum`: upper bound **Behavior** | Condition | Status | | --- | --- | | Sum of the column values is **between** `minValueForColSum` and `maxValueForColSum` | Success βœ… | | Sum of the column values is **greater** than `minValueForColSum` if only `minValueForColSum` is specified | Success βœ… | | Sum of the column values is **less** then `maxValueForColSum` if only `maxValueForColSum` is specified | Success βœ… | | Sum of the column values is **not between** `minValueForColSum` and `maxValueForColSum` | Failed ❌ | | Sum of the column values is **less** than `minValueForColSum` if only `minValueForColSum` is specified | Failed ❌ | | Sum of the column values is **greater** then `maxValueForColSum` if only `maxValueForColSum` is specified | Failed ❌ | **YAML Config** **JSON Config** ### Column Values Standard Deviation to Be Between Validate the standard deviation of a column is between a specific range > Only supports numerical types. **Dimension**: Accuracy **Properties** * `minValueForStdDevInCol`: lower bound * `minValueForStdDevInCol`: upper bound **Behavior** | Condition | Status | | --- | --- | | column values standard deviation is **between** `minValueForStdDevInCol` and `minValueForStdDevInCol` | Success βœ… | | column values standard deviation is **greater** than `minValueForStdDevInCol` if only `minValueForStdDevInCol` is specified | Success βœ… | | column values standard deviation is **less** then `minValueForStdDevInCol` if only `minValueForStdDevInCol` is specified | Success βœ… | | column values standard deviation is **not between** `minValueForStdDevInCol` and `minValueForStdDevInCol` | Failed ❌ | | column values standard deviation is **less** than `minValueForStdDevInCol` if only `minValueForStdDevInCol` is specified | Failed ❌ | | column values standard deviation is **greater** then `minValueForStdDevInCol` if only `minValueForStdDevInCol` is specified | Failed ❌ | **YAML Config** **JSON Config** ### Column Values To Be At Expected Location Validate the reference value for a column is a the expected geographic location > Data will be temporarily stored in memory while the test case is running to validate the location. Not data will be permanently stored. France is the only supported location at this time. To add any additional location please reach out to the team in our slack support channel **Dimension**: Accuracy **Properties** * `locationReferenceType`: the type of location reference `CITY` or `POSTAL_CODE` * `longitudeColumnName`: longitude column name * `latitudeColumnName`: latitude column name * `radius`: radius in meter from which the location can be from the expected lat/long -- acts as a buffer **Behavior** | Condition | Status | | --- | --- | | column values lat/long is **within** the polygon of the column reference (+/- radius) | Success βœ… | | column values lat/long is **outside** the polygon of the column reference (+/- radius) | Failed ❌ | **YAML Config** **JSON Config** --- # SAML AZURE SSO We use cookies to improve site navigation, analyze site usage, and enhance your user experience. Click "Accept" to enable cookies or "Reject" to reject cookies. AcceptReject ![](https://static.scarf.sh/a.png?x-pxid=8472f2b3-d307-4935-a5ed-3f9d3982c15d) deployment No menu items for this category [Home](https://docs.open-metadata.org/latest) /[Deployment](https://docs.open-metadata.org/latest/deployment) /[Security](https://docs.open-metadata.org/latest/deployment/security) /[Saml](https://docs.open-metadata.org/latest/deployment/security/saml) /[Azure](https://docs.open-metadata.org/latest/deployment/security/saml/azure) OpenMetadata Documentation SAML AZURE SSO ============== Follow the sections in this guide to set up Azure SSO using SAML. Security requirements for your **production** environment: * **DELETE** the admin default account shipped by OM. * **UPDATE** the Private / Public keys used for the [JWT Tokens](https://docs.open-metadata.org/latest/deployment/security/enable-jwt-tokens) in case it is enabled. Key Notes on SAML Configuration ------------------------------- 1. **Set `AUTHENTICATION_PROVIDER` to `saml` (lowercase):** Ensure the `AUTHENTICATION_PROVIDER` field in your environment variables is explicitly set to `saml` for SAML authentication to function correctly. Without this, SAML integration will not work. 2. **Routing to IDP:** Users will only be routed to the IDP upon sign-in if `AUTHENTICATION_PROVIDER` is set to `saml`. Create OpenMetadata application ------------------------------- ### Step 1: Configure a new Application in Microsoft Entra ID * Login to [Azure Portal](https://portal.azure.com/) as an administrator and search for Microsoft Entra ID. ![EnterpriseApplications](https://docs.open-metadata.org/images/v1.11/deployment/security/saml/azure/saml-azure-1.png) * Click on `Enterprise Applications` and then `+ New Application` . ![new-application](https://docs.open-metadata.org/images/v1.11/deployment/security/saml/azure/saml-azure-2.png) * After that a new window will appear with different applications, click on `Create your own application`. ![create-own-application](https://docs.open-metadata.org/images/v1.11/deployment/security/saml/azure/saml-azure-3.png) * Give your application a name and select `Integrate any other application you don't find in the gallery` and then click `Create`. ![name-application-create](https://docs.open-metadata.org/images/v1.11/deployment/security/saml/azure/saml-azure-4.png) * Once you have the application created, open the app from list , and then click on `Single Sign-On` and then `SAML`. ![saml-create-single-sign-On](https://docs.open-metadata.org/images/v1.11/deployment/security/saml/azure/saml-azure-5.png) * Edit `Basic SAML Configuration` and populate the values as shown below for `EntityId` and `Assertion Consumer Service Url`. These value should match the one configured with Openmetadata Server side for `samlConfiguration.sp.entityId` and `samlConfiguration.sp.acs` respectively. After this click `Save`. ![edit-basic-saml-configuration](https://docs.open-metadata.org/images/v1.11/deployment/security/saml/azure/saml-azure-6.png) * Click on `Attributes and Claims` and click on the `Required Claim (NameId)`. ![edit-claims](https://docs.open-metadata.org/images/v1.11/deployment/security/saml/azure/saml-azure-7.png) * You will see the values as below image, we need to set the value `Source Attribute` to a user mail value claim from the IDP. Click on `Edit` and then select the `Source Attribute` as `user.mail` or `user.userprincipalname` (in some cases this is also a mail) and then click `Save`. ![edit-claim-value](https://docs.open-metadata.org/images/v1.11/deployment/security/saml/azure/saml-azure-8.png) * To Confirm the claim value we can navigate to user page and check the value of the user. In my case as you can see User Princpal Name is a my mail which i want to use for Openmetadata , so for me `user.userprincipalname` would be correct claim. ![user-claim-value](https://docs.open-metadata.org/images/v1.11/deployment/security/saml/azure/saml-azure-9.png) Security requirements for your **production** environment: * You must always communicate via signed Request for both request from SP to IDP and response from IDP to SP. * To do so we need to add SP certificate to IDP , so that IDP can validate the signed Auth Request coming from SP. * Generate the certificate using below command and then upload the certificate to IDP. * Under `Single Sign-On` you will see SAML Certificates, click on `Verification Certificates`. ![verification-certificate](https://docs.open-metadata.org/images/v1.11/deployment/security/saml/azure/saml-azure-11.png) * You can then check the `Require Verification Certificates` and import the certification with .cer format we generated previously. ### Step 2: Setup `OpenMetadata Server` * Open the downloaded metadata xml file, and populate the following properties in `openmetadata.yml` * Populate the above config from [xml metadata](https://docs.open-metadata.org/latest/deployment/security/saml/xml_file) ![populate-metadata](https://docs.open-metadata.org/images/v1.11/deployment/security/saml/aws/saml-aws-8.png) * IDP Config `entityID` -> Populate it from Metadata XML Entity ID `HTTP-Redirect SSO Login URL` -> always select HTTP-Redirect Url for SSO Login Url `X509 Certificate` -> This is also available in the IDP XML. `NameIDFormat` -> from MetadataXML NameIDFormat `authorityUrl` -> set as {http}/{https}://{domain}:{port}/api/v1/saml/login * SP Config `entityId` -> -> set as {http}/{https}://{domain}:{port}/api/v1/saml/acs `acs` -> Assertion Consumer Url , set as {http}/{https}://{domain}:{port}/api/v1/saml/acs `spX509Certificate` -> set to your X509 Signing Key `callback` -> set as {http}/{https}://{domain}/api/v1/saml/callback * Security Parameters can be configured in case we want to have signed or encrypted or both assertions. In any case we decided to use above config for security then it is mandatory to provide keystore config, from where the system can load the signing certificate or Private Key for encryption. * For **production** environment , it is always suggested to keep these true ### Step 3: Setup JWT Configuration * Follow the guide here for JWT Configuration [Enable JWT Token](https://docs.open-metadata.org/latest/deployment/security/enable-jwt-tokens) . Security requirements for your **production** environment: * **UPDATE** the Private / Public keys used for the [JWT Tokens](https://docs.open-metadata.org/latest/deployment/security/enable-jwt-tokens) the ones shipped with OM are for POC only. ### Step 4: Start the server * Start the OpenMetadata server. With `AUTHENTICATION_PROVIDER` set to saml, you should be routed to the IDP upon sign-in. --- # GCP Secret Manager Parameter Store | Official Documentation We use cookies to improve site navigation, analyze site usage, and enhance your user experience. Click "Accept" to enable cookies or "Reject" to reject cookies. AcceptReject deployment No menu items for this category [Home](https://docs.open-metadata.org/latest) /[Deployment](https://docs.open-metadata.org/latest/deployment) /[Secrets Manager](https://docs.open-metadata.org/latest/deployment/secrets-manager) /[Supported Implementations](https://docs.open-metadata.org/latest/deployment/secrets-manager/supported-implementations) /[Gcp Secret Manager](https://docs.open-metadata.org/latest/deployment/secrets-manager/supported-implementations/gcp-secret-manager) OpenMetadata Documentation GCP Secret Manager ================== The setup steps covers the use of the managed version of the GCP Secret Manager as secrets manager but for the non-managed follow only the steps related to the Airflow server and CLI. Setup ----- ### 1\. Permissions needed These are the permissions required in the service account to enable the GCP Secret Manager in OpenMetadata. We recommend to use the role named `roles/secretmanager.secretAccessor` to grant necessary permissions. * resourcemanager.projects.get * resourcemanager.projects.list * secretmanager.versions.access ### 2\. Update configuration We have to set up the secret manager provider we want to use, that in our case is `gcp`, and the credentials for our GCP information. The changes to be done in `openmetadata.yaml` file of the OpenMetadata server are: And these are the changes required in `airflow.cfg` of our Airflow instance: As an alternative to editing the `airflow.cfg` file, we can also set the following environment variables: If no parameters are provided for the GCP account, it will use Application Default Credentials (ADC). ADC will look for credentials in: 1. Local development environment 2. Cloud Shell or other Google Cloud cloud-based development environments 3. Compute Engine or other Google Cloud services that support attaching a service account 4. Google Kubernetes Engine or GKE Enterprise 5. On-premises or another cloud provider More info in [Set up Application Default Credentials](https://cloud.google.com/docs/authentication/provide-credentials-adc) ### 3\. Migrate Secrets & restart both servers After updating the configuration files, we are ready to migrate the secrets and restart both services. In order to ensure that the current sensitive information is properly migrated to the Secrets Manager, you need to run the following command: Make sure you are running it with the same environment variables required by the server. If everything goes as planned, all the data would be displayed using the parameters names which starts with `/openmetadata/...` in your GCP Secret Manager console. The following image shows what it should look like: ![gcp-secret-manager-console](https://docs.open-metadata.org/images/v1.11/deployment/secrets-manager/supported-implementations/gcp-secret-manager/gcp-secret-manager-console.png) **Note:** If we want to change the starting path for our secrets names from `openmetadata` to a different one, we have to change the property `clusterName` in our `openmetadata.yaml`. Also, if you inform the `prefix` value, it will be added before the `clusterName`, i.e., `///` You can inform the `tags` as well as a list of strings `[key1:value1,key2:value2,...]`. These tags will be added to the resource created in GCP. Airflow ------- If you enabled the Secret Manager and you are using your own Airflow to run the ingestions, make sure to configure your YAML files as: and follow the same environment variables to set up the Airflow configuration: --- # Azure Key Vault | OpenMetadata Secrets Manager Guide We use cookies to improve site navigation, analyze site usage, and enhance your user experience. Click "Accept" to enable cookies or "Reject" to reject cookies. AcceptReject deployment No menu items for this category [Home](https://docs.open-metadata.org/latest) /[Deployment](https://docs.open-metadata.org/latest/deployment) /[Secrets Manager](https://docs.open-metadata.org/latest/deployment/secrets-manager) /[Supported Implementations](https://docs.open-metadata.org/latest/deployment/secrets-manager/supported-implementations) /[Azure Key Vault](https://docs.open-metadata.org/latest/deployment/secrets-manager/supported-implementations/azure-key-vault) OpenMetadata Documentation Azure Key Vault =============== The setup steps covers the use of the managed version of the Azure Key Vault as secrets manager but for the non-managed follow only the steps related to the Airflow server and CLI. Setup ----- ### 1\. Create Principal #### Service Principal 1. Go to `Microsoft Entra ID` and create an [App Registration](https://learn.microsoft.com/en-us/entra/identity-platform/quickstart-register-app) . 2. Inside the App Registration, go to `Certificates & Secrets` and create a `Client secret`. Note down the `Value`, it will be our `clientSecret` configuration. 3. From the App Registration overview page, note down the `Application (client) ID` and the `Directory (tenant) ID`. #### Managed Identity (recommended) 1. In your Azure subscription create [Managed Identity](https://learn.microsoft.com/en-us/entra/identity/managed-identities-azure-resources/overview) 2. Use this created identity - for AKS users this means you need to use [Pod Identity](https://learn.microsoft.com/en-us/azure/aks/use-azure-ad-pod-identity) or [Workload Identity (recommended)](https://learn.microsoft.com/en-us/azure/aks/workload-identity-overview?tabs=dotnet) . Note that the using Managed Identity require using [default Authentication Credential](https://learn.microsoft.com/en-us/python/api/overview/azure/identity-readme?view=azure-python#defaultazurecredential) . ### 2\. Add RBAC roles It if possible to use different Principals for OpenMetadata Server and the Ingestion. In that case the server needs higher privileges - `Key Vault Secrets Officer` - to be able to create/read/update secrets in the Vault. While the Airflow part only needs to read the secrets hence the role `Key Vault Secrets Officer`. #### Open Metadata server 1. In your Key Vault overview page, note down the `Vault URI`. 2. Go to `Access Control (IAM)` and click on `Add Role Assignment`. 3. Give the permission `Key Vault Secrets Officer` to your Principal. #### Airflow 1. In your Key Vault overview page, note down the `Vault URI`. 2. Go to `Access Control (IAM)` and click on `Add Role Assignment`. 3. Give the permission `Key Vault Secrets User` to your Principal. ### 3\. Update configuration We have to set up the secret manager provider we want to use, that in our case is `azure-kv`, and the credentials. The changes to be done in `openmetadata.yaml` file of the OpenMetadata server are: #### Default Azure Credential For Helm Values, you will need to add `PIPELINE_SERVICE_CLIENT_SECRETS_MANAGER_LOADER` as part of `extraEnvs`. This will look like below - #### Client Secret Credential For Helm Values, you will need to add `PIPELINE_SERVICE_CLIENT_SECRETS_MANAGER_LOADER` as part of `extraEnvs`. This will look like below - The changes to be done in `airflow.yaml` file of the Airflow are: Note that the **Key Vault Name** parameter is MANDATORY for the system to know where to store and retrieve the secrets. And these are the changes required in `airflow.cfg` of our Airflow instance: As an alternative to editing the `airflow.cfg` file, we can also set the following environment variables: If only the ``, parameter is provided, we will use Azure's [default Authentication Credential](https://learn.microsoft.com/en-us/python/api/overview/azure/identity-readme?view=azure-python#defaultazurecredential) . Also if you are using [Microsoft Entra Workload ID](https://learn.microsoft.com/en-us/azure/aks/workload-identity-overview) with [Service Account Token Volume Projection](https://kubernetes.io/docs/tasks/configure-pod-container/configure-service-account/#serviceaccount-token-volume-projection) then you need also to use projected service account instead one created by Airflow and OpenMetadata: airflow.yaml: openmetadata.yaml: ### 3\. Migrate Secrets & restart both servers After updating the configuration files, we are ready to migrate the secrets and restart both services. In order to ensure that the current sensitive information is properly migrated to the Secrets Manager, you need to run the following command: Make sure you are running it with the same environment variables required by the server. If everything goes as planned, all the data would be displayed using the parameters names which starts with `openmetadata-...` in your Key Vault console. **Note:** If we want to change the starting path for our secrets names from `openmetadata` to a different one, we have to change the property `clusterName` in our `openmetadata.yaml`. Also, if you inform the `prefix` value, it will be added before the `clusterName`, i.e., `--` You can inform the `tags` as well as a list of strings `[key1:value1,key2:value2,...]`. These tags will be added to the created secret. CLI --- After enabling the Secret Manager, we also have to make a slight change in our workflows YAML files. In the `workflowConfig` we have to add the secret manager configuration: Make sure to follow the steps [here](https://learn.microsoft.com/en-us/python/api/overview/azure/identity-readme?view=azure-python#defaultazurecredential) to allow the Python client to authenticate to Azure. Note that the `AZURE_KEY_VAULT_NAME` variable is **REQUIRED** to know against which Key Vault service to point to. You can specify as well the environment variables of your App Registration if you're running the ingestion outside of Azure: [docs](https://learn.microsoft.com/en-us/python/api/azure-identity/azure.identity.environmentcredential?view=azure-python) . Airflow ------- If you enabled the Secret Manager and you are using your own Airflow to run the ingestions, make sure to configure your YAML files as: and follow the same environment variables to set up the Airflow configuration: Note that the `AIRFLOW__OPENMETADATA_SECRETS_MANAGER__AZURE_KEY_VAULT_NAME` variable is **REQUIRED** to know against which Key Vault service to point to. --- # Server Configuration Reference | Official Documentation We use cookies to improve site navigation, analyze site usage, and enhance your user experience. Click "Accept" to enable cookies or "Reject" to reject cookies. AcceptReject deployment No menu items for this category [Home](https://docs.open-metadata.org/latest) /[Deployment](https://docs.open-metadata.org/latest/deployment) /[Configuration](https://docs.open-metadata.org/latest/deployment/configuration) OpenMetadata Documentation Server Configuration Reference ============================== This document describes OpenMetadata Server Configuration Server Port ----------- By default, the OpenMetadata server runs on port 8585. It uses Jetty Server. The above config can be changed to make it run on a different port. Once you have updated the port details in config restart the server. Database -------- OpenMetadata supports MySQL or Postgres as the database. The database configurations and connection strings must be as specified below. The section below refers to the database connection details to MySQL database. We recommend you create a MySQL user with a strong password and update this section accordingly. The section below refers to the database connection details to Postgres database. OpenMetadata uses stored generated columns which is supported in Postgres 12. Ensure that you have Postgres 12 or a later version. We recommend you create a Postgres user with a strong password and update this section accordingly. ElasticSearch ------------- ElasticSearch is one of the pre-requisites to run OpenMetadata. Default configuration expects a single instance of ElasticSearch running on the local machine. Please make sure you update it with your production elastic search. Event Handlers -------------- EventHandler configuration is optional. It will update the AuditLog in MySQL DB and also ElasticSearch indexes whenever any entity is updated either through UI or API interactions. We recommend you leave it there as it enhances the user experience. Healthcheck ----------- Healthcheck API provides an API endpoint to check the OpenMetadata server health. We recommend in production settings to use this API to monitor the health of your OpenMetadata instance. Please tune the above configuration according to your production needs. Security -------- Please follow our [Enable Security Guide](https://docs.open-metadata.org/latest/deployment/security) to configure security for your OpenMetadata installation. --- # Enable Security | OpenMetadata Deployment Security Guide We use cookies to improve site navigation, analyze site usage, and enhance your user experience. Click "Accept" to enable cookies or "Reject" to reject cookies. AcceptReject deployment No menu items for this category [Home](https://docs.open-metadata.org/latest) /[Deployment](https://docs.open-metadata.org/latest/deployment) /[Bare Metal](https://docs.open-metadata.org/latest/deployment/bare-metal) /[Security](https://docs.open-metadata.org/latest/deployment/bare-metal/security) OpenMetadata Documentation Bare Metal Security =================== Follow the steps for setting up the SSO, and then check the specific `Bare Metal` section of your chosen SSO. By default, Basic Authentication will be enabled as authentication mechanism. [Basic Authentication\ \ Configure Basic Authentication to access the UI and APIs](https://docs.open-metadata.org/latest/deployment/security/basic-auth) [Ldap Authentication\ \ Configure Ldap Authentication to access the UI and APIs](https://docs.open-metadata.org/latest/deployment/security/ldap) [Auth0 SSO\ \ Configure Auth0 SSO to access the UI and APIs](https://docs.open-metadata.org/latest/deployment/security/auth0) [Azure SSO\ \ Configure Azure SSO to access the UI and APIs](https://docs.open-metadata.org/latest/deployment/security/azure) [Custom OIDC SSO\ \ Configure a Custom OIDC SSO to access the UI and APIs](https://docs.open-metadata.org/latest/deployment/security/custom-oidc) [Google SSO\ \ Configure Google SSO to access the UI and APIs](https://docs.open-metadata.org/latest/deployment/security/google) [Okta SSO\ \ Configure Okta SSO to access the UI and APIs](https://docs.open-metadata.org/latest/deployment/security/okta) [Amazon Cognito SSO\ \ Configure Amazon Cognito SSO to access the UI and APIs](https://docs.open-metadata.org/latest/deployment/security/amazon-cognito) [OneLogin SSO\ \ Configure OneLogin SSO to access the UI and APIs](https://docs.open-metadata.org/latest/deployment/security/one-login) [Keycloak SSO\ \ Configure Keycloak SSO to access the UI and APIs](https://docs.open-metadata.org/latest/deployment/security/keycloak) --- # KafkaConnect | OpenMetadata Messaging Pipeline Connector We use cookies to improve site navigation, analyze site usage, and enhance your user experience. Click "Accept" to enable cookies or "Reject" to reject cookies. AcceptReject connectors No menu items for this category [Home](https://docs.open-metadata.org/latest) /[Connectors](https://docs.open-metadata.org/latest/connectors) /[Pipeline](https://docs.open-metadata.org/latest/connectors/pipeline) /[Kafkaconnect](https://docs.open-metadata.org/latest/connectors/pipeline/kafkaconnect) OpenMetadata Documentation ![KafkaConnect](https://docs.open-metadata.org/_next/image?url=%2Fimages%2Fconnectors%2Fkafka.webp&w=64&q=75) KafkaConnect ============ PROD Available In Feature List Pipelines Pipeline Status Lineage Usage Owners Tags In this section, we provide guides and references to use the KafkaConnect connector. Configure and schedule KafkaConnect metadata and profiler workflows from the OpenMetadata UI: * [Requirements](https://docs.open-metadata.org/latest/connectors/pipeline/kafkaconnect#requirements) * [KafkaConnect Versions](https://docs.open-metadata.org/latest/connectors/pipeline/kafkaconnect#kafkaconnect-versions) * [Metadata Ingestion](https://docs.open-metadata.org/latest/connectors/pipeline/kafkaconnect#metadata-ingestion) * [Service Name](https://docs.open-metadata.org/latest/connectors/pipeline/kafkaconnect#service-name) * [Connection Details](https://docs.open-metadata.org/latest/connectors/pipeline/kafkaconnect#connection-details) * [Metadata Ingestion Options](https://docs.open-metadata.org/latest/connectors/pipeline/kafkaconnect#metadata-ingestion-options) * [Troubleshooting](https://docs.open-metadata.org/latest/connectors/pipeline/glue-pipeline/troubleshooting) * [Workflow Deployment Error](https://docs.open-metadata.org/latest/connectors/pipeline/kafkaconnect#workflow-deployment-error) Ingestion Deployment -------------------- To run the Ingestion via the UI you'll need to use the OpenMetadata Ingestion Container, which comes shipped with custom Airflow plugins to handle the workflow deployment. If you want to install it manually in an already existing Airflow host, you can follow [this](https://docs.open-metadata.org/latest/deployment/ingestion/openmetadata) guide. If you don't want to use the OpenMetadata Ingestion container to configure the workflows via the UI, then you can check the following docs to run the Ingestion Framework in any orchestrator externally. [#### Run Connectors from the OpenMetadata UI\ \ Learn how to manage your deployment to run connectors from the UI](https://docs.open-metadata.org/latest/deployment/ingestion/openmetadata) [#### Run the Connector Externally\ \ Get the YAML to run the ingestion externally](https://docs.open-metadata.org/latest/connectors/pipeline/kafkaconnect/yaml) [#### External Schedulers\ \ Get more information about running the Ingestion Framework Externally](https://docs.open-metadata.org/latest/deployment/ingestion) Requirements ------------ ### KafkaConnect Versions OpenMetadata is integrated with kafkaconnect up to version [3.6.1](https://docs.kafkaconnect.io/getting-started) and will continue to work for future kafkaconnect versions. The ingestion framework uses [kafkaconnect python client](https://libraries.io/pypi/kafka-connect-py) to connect to the kafkaconnect instance and perform the API calls Metadata Ingestion ------------------ #### 1\. Visit the Services Page Click `Settings` in the side navigation bar and then `Services`. The first step is to ingest the metadata from your sources. To do that, you first need to create a Service connection first. This Service will be the bridge between OpenMetadata and your source system. Once a Service is created, it can be used to configure your ingestion workflows. ![Visit Services Page](https://docs.open-metadata.org/images/v1.11/connectors/visit-services-page.png) Select your Service Type and Add a New Service #### 2\. Create a New Service Click on _Add New Service_ to start the Service creation. ![Create a new Service](https://docs.open-metadata.org/images/v1.11/connectors/create-new-service.png) Add a new Service from the Services page #### 3\. Select the Service Type Select KafkaConnect as the Service type and click _Next_. ![Select Service](https://docs.open-metadata.org/images/v1.11/connectors/kafkaconnect/select-service.webp) Select your Service from the list #### 4\. Name and Describe your Service Provide a name and description for your Service. #### Service Name OpenMetadata uniquely identifies Services by their **Service Name**. Provide a name that distinguishes your deployment from other Services, including the other KafkaConnect Services that you might be ingesting metadata from. Note that when the name is set, it cannot be changed. ![Add New Service](https://docs.open-metadata.org/images/v1.11/connectors/kafkaconnect/add-new-service.webp) Provide a Name and description for your Service #### 5\. Configure the Service Connection In this step, we will configure the connection settings required for KafkaConnect. Please follow the instructions below to properly configure the Service to read from your sources. You will also find helper documentation on the right-hand side panel in the UI. ![Configure Service connection](https://docs.open-metadata.org/images/v1.11/connectors/kafkaconnect/service-connection.webp) Configure the Service connection by filling the form #### Connection Details * **Host and Port**: The hostname or IP address of the Kafka Connect worker with the REST API enabled eg.`https://localhost:8083` or `https://127.0.0.1:8083` or `https://` * **Kafka Connect Config**: OpenMetadata supports username/password. 1. Basic Authentication * Username: Username to connect to Kafka Connect. This user should be able to send request to the Kafka Connect API and access the [Rest API](https://docs.confluent.io/platform/current/connect/references/restapi.html) GET endpoints. * Password: Password to connect to Kafka Connect. * **verifySSL** : Whether SSL verification should be perform when authenticating. * **Kafka Service Name** : The Service Name of the Ingested [Kafka](https://docs.open-metadata.org/latest/connectors/messaging/kafka#4.-name-and-describe-your-service) instance associated with this KafkaConnect instance. #### 6\. Test the Connection Once the credentials have been added, click on _Test Connection_ and _Save_ the changes. ![Test Connection](https://docs.open-metadata.org/images/v1.11/connectors/test-connection.png) Test the connection and save the Service #### 7\. Configure Metadata Ingestion In this step we will configure the metadata ingestion pipeline, Please follow the instructions below ![Configure Metadata Ingestion](https://docs.open-metadata.org/images/v1.11/connectors/configure-metadata-ingestion-pipeline.png) Configure Metadata Ingestion Page #### Metadata Ingestion Options * **Name**: This field refers to the name of ingestion pipeline, you can customize the name or use the generated name. * **Pipeline Filter Pattern (Optional)**: Use to pipeline filter patterns to control whether or not to include pipeline as part of metadata ingestion. * **Include**: Explicitly include pipeline by adding a list of comma-separated regular expressions to the Include field. OpenMetadata will include all pipeline with names matching one or more of the supplied regular expressions. All other schemas will be excluded. * **Exclude**: Explicitly exclude pipeline by adding a list of comma-separated regular expressions to the Exclude field. OpenMetadata will exclude all pipeline with names matching one or more of the supplied regular expressions. All other schemas will be included. * **Include lineage (toggle)**: Set the Include lineage toggle to control whether to include lineage between pipelines and data sources as part of metadata ingestion. * **Enable Debug Log (toggle)**: Set the Enable Debug Log toggle to set the default log level to debug. * **Mark Deleted Pipelines (toggle)**: Set the Mark Deleted Pipelines toggle to flag pipelines as soft-deleted if they are not present anymore in the source system. #### 8\. Schedule the Ingestion and Deploy Scheduling can be set up at an hourly, daily, weekly, or manual cadence. The timezone is in UTC. Select a Start Date to schedule for ingestion. It is optional to add an End Date. Review your configuration settings. If they match what you intended, click Deploy to create the service and schedule metadata ingestion. If something doesn't look right, click the Back button to return to the appropriate step and change the settings as needed. After configuring the workflow, you can click on Deploy to create the pipeline. ![Schedule the Workflow](https://docs.open-metadata.org/images/v1.11/connectors/schedule.png) Schedule the Ingestion Pipeline and Deploy #### 9\. View the Ingestion Pipeline Once the workflow has been successfully deployed, you can view the Ingestion Pipeline running from the Service Page. ![View Ingestion Pipeline](https://docs.open-metadata.org/images/v1.11/connectors/view-ingestion-pipeline.png) View the Ingestion Pipeline from the Service Page Debezium CDC Support -------------------- The KafkaConnect connector provides **full support for Debezium CDC connectors** with intelligent column extraction and accurate lineage tracking. ### What We Provide When you ingest Debezium connectors, OpenMetadata automatically: 1. **Detects CDC Envelope Structures** - Identifies Debezium's CDC format with `op`, `before`, and `after` fields 2. **Extracts Real Table Columns** - Parses actual database columns from the CDC payload instead of CDC envelope metadata 3. **Creates Accurate Column-Level Lineage** - Maps lineage from source database tables β†’ Kafka topics β†’ target systems ### Recognized Configuration Parameters OpenMetadata recognizes the following Debezium configuration parameters for intelligent CDC detection: * `database.server.name` - Server identifier (Debezium V1) * `topic.prefix` - Topic prefix (Debezium V2) * `table.include.list` - Tables to capture (e.g., `mydb.customers,mydb.orders`) ![Kafkaconnect Lineage](https://docs.open-metadata.org/images/v1.11/connectors/kafkaconnect/lineage.webp) Supported Connectors -------------------- Currently, the following source and sink connectors for Kafka Connect are supported for lineage tracking: * [MySQL](https://docs.open-metadata.org/latest/connectors/database/mysql) * [PostgreSQL](https://docs.open-metadata.org/latest/connectors/database/postgres) * [MSSQL](https://docs.open-metadata.org/latest/connectors/database/mssql) * [MongoDB](https://docs.open-metadata.org/latest/connectors/database/mongodb) * [Amazon S3](https://docs.open-metadata.org/latest/connectors/storage/s3) For these connectors, lineage information can be obtained provided they are configured with a source or sink and the corresponding metadata ingestion is enabled. **Note:** All supported database connectors listed above work seamlessly with **Debezium CDC connectors** for enhanced column-level lineage tracking. When using Debezium, OpenMetadata automatically detects the CDC envelope structure and extracts actual table columns for accurate lineage mapping. ### Missing Lineage If lineage information is not displayed for a Kafka Connect service, follow these steps to diagnose the issue. 1. _Kafka Service Association_: Make sure the Kafka service that the data is being ingested from is associated with this Kafka Connect service. Additionally, verify that the correct name is passed on in the Kafka Service Name field during configuration. This field helps establish the lineage between the Kafka service and the Kafka Connect flow. 2. _Source and Sink Configuration_: Verify that the Kafka Connect connector associated with the service is configured with a source and/or sink database or storage system. Connectors without a defined source or sink cannot provide lineage data. 3. _Metadata Ingestion_: Ensure that metadata for both the source and sink database/storage systems is ingested and passed to the lineage system. This typically involves configuring the relevant connectors to capture and transmit this information. --- # How Column-Level Lineage Works | Official Documentation We use cookies to improve site navigation, analyze site usage, and enhance your user experience. Click "Accept" to enable cookies or "Reject" to reject cookies. AcceptReject ![](https://static.scarf.sh/a.png?x-pxid=6814879a-96c0-40bd-be69-ddfeb14694aa) how-to-guides No menu items for this category [Home](https://docs.open-metadata.org/latest) /[How To Guides](https://docs.open-metadata.org/latest/how-to-guides) /[Data Lineage](https://docs.open-metadata.org/latest/how-to-guides/data-lineage) /[Column](https://docs.open-metadata.org/latest/how-to-guides/data-lineage/column) OpenMetadata Documentation How Column-Level Lineage Works ============================== OpenMetadata supports rich column-level lineage for understanding the relationship between tables and to perform impact analysis. Users can manually edit both the table and column level lineage to capture any information that is not automatically surfaced. ![Column-Level Data Lineage in OpenMetadata](https://docs.open-metadata.org/images/v1.11/how-to-guides/lineage/lineage1.png) Column-Level Data Lineage in OpenMetadata **Quick Tip:** Drilldown to view all the available columns for a table when viewing column-level lineage. You can generate the column-level lineage automatically by running the **Lineage Agent**. ![Lineage Ingestion](https://docs.open-metadata.org/images/v1.11/how-to-guides/lineage/ingestion.png) Lineage Ingestion Manually Edit Column Level Lineage ---------------------------------- OpenMetadata supports manual editing of both table and column level lineage. You can edit the lineage for the individual columns by clicking on the edit option on the top right. User the anchor points on either side of the columns to create links and trace individual columns through their lineage. You can also add new tables that have columns you want to trace. Connect the relevant columns to the current lineage. ![Manually Edit Column Level Lineage](https://docs.open-metadata.org/images/v1.11/how-to-guides/lineage/column1.png) Manually Edit Column Level Lineage Watch the video on editing column-level lineage. [How to Manually Add or Edit Lineage\ \ Edit the table and column level lineage manually.](https://docs.open-metadata.org/latest/how-to-guides/data-lineage/manual) --- # SSO | OpenMetadata Security Integration We use cookies to improve site navigation, analyze site usage, and enhance your user experience. Click "Accept" to enable cookies or "Reject" to reject cookies. AcceptReject ![](https://static.scarf.sh/a.png?x-pxid=8472f2b3-d307-4935-a5ed-3f9d3982c15d) deployment No menu items for this category [Home](https://docs.open-metadata.org/latest) /[Deployment](https://docs.open-metadata.org/latest/deployment) /[Security](https://docs.open-metadata.org/latest/deployment/security) /[Configuration](https://docs.open-metadata.org/latest/deployment/security/configuration) OpenMetadata Documentation SSO === Follow the sections in this guide to set up SSO. Security requirements for your **production** environment: * **DELETE** the admin default account shipped by OM in case you had [Basic Authentication](https://docs.open-metadata.org/latest/deployment/security/basic-auth) enabled before configuring the authentication with SSO. * **UPDATE** the Private / Public keys used for the [JWT Tokens](https://docs.open-metadata.org/latest/deployment/security/enable-jwt-tokens) . The keys we provide by default are aimed only for quickstart and testing purposes. They should NEVER be used in a production installation. Create Server Credentials ------------------------- * Go to the console of your preferred SSO provider * Create an OIDC client application with implicit flow enabled to get a client ID. ### Create Client ID and Secret Key In a **Public** client configuration, only the **Client ID** is required. **Client Secret** should not be provided, as public clients cannot securely store sensitive credentials. * Navigate to your preferred OIDC provider console and create an OIDC client application. * Generate client ID and secret key in JSON format. After applying these steps, you can update the configuration of your deployment: [Docker Security\ \ Configure SSO for your Docker Deployment.](https://docs.open-metadata.org/latest/deployment/security/configuration/docker) [Bare Metal Security\ \ Configure SSO for your Bare Metal Deployment.](https://docs.open-metadata.org/latest/deployment/security/configuration/bare-metal) [Kubernetes Security\ \ Configure SSO for your Kubernetes Deployment.](https://docs.open-metadata.org/latest/deployment/security/configuration/kubernetes) Configure Ingestion ------------------- Once your server security is set, it's time to review the ingestion configuration. Our bots support JWT tokens to authenticate to the server when sending requests. Find more information on [**Enabling JWT Tokens**](https://docs.open-metadata.org/latest/deployment/security/enable-jwt-tokens) and [**JWT Troubleshooting**](https://docs.open-metadata.org/latest/deployment/security/jwt-troubleshooting) to ensure seamless authentication. --- # Metrics | OpenMetadata Profiler Metrics Guide We use cookies to improve site navigation, analyze site usage, and enhance your user experience. Click "Accept" to enable cookies or "Reject" to reject cookies. AcceptReject ![](https://static.scarf.sh/a.png?x-pxid=6814879a-96c0-40bd-be69-ddfeb14694aa) how-to-guides No menu items for this category [Home](https://docs.open-metadata.org/latest) /[How To Guides](https://docs.open-metadata.org/latest/how-to-guides) /[Data Quality Observability](https://docs.open-metadata.org/latest/how-to-guides/data-quality-observability) /[Profiler](https://docs.open-metadata.org/latest/how-to-guides/data-quality-observability/profiler) /[Metrics](https://docs.open-metadata.org/latest/how-to-guides/data-quality-observability/profiler/metrics) OpenMetadata Documentation Profiler Metrics ================ Here you can find information about the supported metrics for the different types. A Metric is a computation that we can run on top of a Table or Column to receive a value back. They are the primary **building block** of OpenMetadata's Profiler. * **Metrics** define the queries and computations generically. They do not aim at specific columns or database dialects. Instead, they are expressions built with SQLAlchemy that should run everywhere. * A **Profiler** is the binding between a set of metrics and the external world. The Profiler contains the Table and Session information and is in charge of executing the metrics. On this page, you will learn all the metrics that we currently support and their meaning. We will base all the namings on the definitions on the JSON Schemas. You can check the definition of the `columnProfile` [here](https://github.com/open-metadata/OpenMetadata/blob/main/openmetadata-spec/src/main/resources/json/schema/entity/data/table.json#L271) . On the other hand, the metrics are implemented [here](https://github.com/open-metadata/OpenMetadata/tree/main/ingestion/src/metadata/profiler/metrics) . We will base all the namings on the definitions on the JSON Schemas. Table Metrics ------------- Those are the metrics computed at the Table level. ### Row Count It computes the number of rows in the Table. ### Column Count Returns the number of columns in the Table. System Metrics -------------- System metrics provide information related to DML operations performed on the table. These metrics present a concise view of your data freshness. In a typical data processing flow tables are updated at a certain frequency. Table freshness will be monitored by confirming a set of operations has been performed against the table. To increase trust in your data assets, OpenMetadata will monitor the `INSERT`, `UPDATE` and `DELETE` operations performed against your table to showcase 2 metrics related to freshness (see below for more details). With this information, you are able to see when a specific operation was last perform and how many rows it affected. ![table profile freshness metrics](https://docs.open-metadata.org/images/v1.11/features/ingestion/workflows/profiler/profiler-freshness-metrics.png) table profile freshness metrics These metrics are available for **BigQuery**, **Redshift** and **Snowflake**. Other database engines are currently not supported so the computation of the system metrics will be skipped. ### DML Operations This metrics shows all the DML operations performed (`INSERT`, `UPDATE`, `DELETE`) against the table in a timeseries fashion. ### Rows Affected by the DML Operation This metrics shows the number of rows that were affected by a DML operation (`INSERT`, `UPDATE`, `DELETE`) over time. Column Metrics -------------- List of Metrics that we run for all the columns. > Note that for now we are not supporting complex types such as ARRAY or STRUCT. The implementation will come down the road. ### Values Count It is the total count of the values in the column. Ignores nulls. ### Values Percentage Percentage of values in this column vs. the Row Count. ### Duplicate Count Informs the number of rows that have duplicated values in a column. We compute it as `count(col) - count(distinct(col))`. ### Null Count The number of null values in a column. ### Null Proportion It shows the ratio of null values vs. the total number of values in a column. ### Unique Count The number of unique values in a column, those that appear only once. E.g., `[1, 2, 2, 3, 3, 4] => [1, 4] => count = 2`. ### Unique Proportion Unique Count / Values Count ### Distinct Count The number of different items in a column. E.g., `[1, 2, 2, 3, 3, 4] => [1, 2, 3, 4] => count = 4`. ### Distinct Proportion Distinct Count / Values Count ### Min Only for numerical values. Returns the minimum. ### Max Only for numerical values. Returns the maximum. ### Min Length Only for concatenable values. Returns the minimum length of the values in a column. ### Max Length Only for concatenable values. Returns the maximum length of the values in a column. ### Mean * Numerical values: returns the average of the values. * Concatenable values: returns the average length of the values. ### Median Only for numerical values. This is currently not supported in MySQL. ### Sum Only for numerical values. Returns the sum of all values in a column. ### Standard Deviation Only for numerical values. Returns the standard deviation. ### Histogram The histogram returns a dictionary of the different bins and the number of values found for that bin. It will be computed only if the Inter Quartile Range value is available ### First Quartile Only for numerical values. Middle number between the smallest value and the median ### Third Quartile Only for numerical values. Middle number between the median and the greatest value ### Inter Quartile Range Only for numerical values. Difference between the third quartile and the first quartile ### Nonparametric Skew Measure of skewness of the column distribution. Nonparametric skew is computed as follow $$ S = \\frac{\\mu-\\tilde{\\mu}}{\\sigma} $$ Where $$ \\mu = mean\\ \\tilde{\\mu} = median\\ \\sigma = standard deviation\\ $$ Grant Access to User for System Metrics --------------------------------------- OpenMetadata uses system tables to compute system metrics. You can find the required access as well as more details for your database engine below. ### Snowflake OpenMetadata uses the `QUERY_HISTORY_BY_WAREHOUSE` view of the `INFORMATION_SCHEMA` to collect metrics about DML operations. To collect information about the `RESULT_SCAN` command alongside the QUERY ID will be passed to the `RESULT_SCAN` function to get the number of rows affected by the operation. You need to make sure the user running the profiler workflow has access to this view and this function. OpenMetadata will look at the past 24-hours to fetch the operations that were performed against a table. **Important** For snowflake system, the system will parse the DDL query and attempt to match `database`, `schema`, and `table` name to entities in OpenMetadata. If the DDL query does not include all 3 elements we will not be able to ingest this metric. ### Redshift OpenMetadata uses `stl_insert`, `stl_delete`, `svv_table_info`, and `stl_querytext` to fetch DML operations as well as the number of rows affected by these operations. You need to make sure the user running the profiler workflow has access to these views and tables. OpenMetadata will look at the previous day to fetch the operations that were performed against a table. ### BigQuery Bigquery uses the `JOBS` table of the `INFORMATION_SCHEMA` to fetch DML operations as well as the number of rows affected by these operations. You will need to make sure your data location is properly set when creating your BigQuery service connection in OpenMetadata. OpenMetadata will look at the previous day to fetch the operations that were performed against a table filter on the `creation_time` partition field to limit the size of data scanned. Reach out! ---------- Is there any metric you'd like to see? Open an [issue](https://github.com/open-metadata/OpenMetadata/issues/new/choose) or reach out on [Slack](https://slack.open-metadata.org/) . --- # Incident Manager | OpenMetadata Data Quality Management We use cookies to improve site navigation, analyze site usage, and enhance your user experience. Click "Accept" to enable cookies or "Reject" to reject cookies. AcceptReject ![](https://static.scarf.sh/a.png?x-pxid=6814879a-96c0-40bd-be69-ddfeb14694aa) how-to-guides No menu items for this category [Home](https://docs.open-metadata.org/latest) /[How To Guides](https://docs.open-metadata.org/latest/how-to-guides) /[Data Quality Observability](https://docs.open-metadata.org/latest/how-to-guides/data-quality-observability) /[Incident Manager](https://docs.open-metadata.org/latest/how-to-guides/data-quality-observability/incident-manager) OpenMetadata Documentation Overview of the Incident Manager ================================ Using Incident Manager, managing data quality issues becomes streamlined and efficient. By centralizing the resolution process, assigning tasks, and logging root causes, your team can quickly address and resolve failures. The historical record of past incidents serves as a comprehensive guide, aiding your team in troubleshooting and resolving issues more effectively. All the necessary context is readily available, making it easier to maintain high data quality standards. Opening and Triaging Incidents ------------------------------ In v1.1.0, we introduced the ability for user to manage and triage incidents linked to failures. When a test case fails, it will automatically open a new incident and mark it as new. If enough information is available, OpenMetadata will automatically assign a severity to the incident; note that you can override this severity. It indicates that a new failure has happened. ![Test suite results table](https://docs.open-metadata.org/images/v1.11/features/ingestion/workflows/data-quality/resolution-workflow-new.png) Test suite results table The Incident Manager serves as a centralized hub to handle the resolution flow of failed Data Quality Tests. Once an incident has been open you will be able to triage and manage it. You can perform different actions at this stage: * **Acknowledge the Issue:** Recognize and confirm that there is a problem that needs attention. By marking with `ack` you can inform users that people are aware of the ongoing incident. * **Assign Responsibility:** Designate a specific person or team to address the errors. By marking with `assign` you can open a task for the assignee. * **Log the Root Cause:** Document the underlying cause of the failure for future reference and analysis. ![Test suite results table](https://docs.open-metadata.org/images/v1.11/features/ingestion/workflows/data-quality/resolution-workflow-ack-form.png) Test suite results table ![Test suite results table](https://docs.open-metadata.org/images/v1.11/features/ingestion/workflows/data-quality/resolution-workflow-ack.png) Test suite results table You can mark the incident as `resolved`. The user will be required to specify the reason and add a comment. This provides context regarding the incident and helps users further understand what might have gone wrong ![Test suite results table](https://docs.open-metadata.org/images/v1.11/features/ingestion/workflows/data-quality/resolution-workflow-resolved-form.png) Test suite results table ![Test suite results table](https://docs.open-metadata.org/images/v1.11/features/ingestion/workflows/data-quality/resolution-workflow-resolved.png) Test suite results table Using the Test Resolution Flow ------------------------------ The Test Resolution flow is a critical feature of the Incident Manager. Here’s how it works: 1. **Failure Notification:** When a Data Quality Test fails, the system generates a notification. 2. **Acknowledge the Failure:** The designated user acknowledges the issue within the Incident Manager. 3. **Assignment:** The issue is then assigned to a knowledgeable user or team responsible for resolving it. 4. **Status Updates:** The assigned user can update the status of the issue, keeping the organization informed about progress and any developments. 5. **Sharing Updates:** All impacted users receive updates, ensuring everyone stays informed about the resolution process. Incidents Context & History --------------------------- When clicking on an open incident you will different information: **Open Incident:** this section will show you open incidents with the timeline and any comments/collaboration that might have been happening. **Closed Incidents:** this section will show you incidents that have been resolved in the past with the timeline and any comments/collaboration that might have been happening and the resolution reason. ![Test suite results table](https://docs.open-metadata.org/images/v1.11/features/ingestion/workflows/data-quality/incident-management-page.png) Test suite results table Building a Troubleshooting Handbook ----------------------------------- One of the powerful features of the Incident Manager is its ability to store all past failures. This historical data becomes a valuable troubleshooting handbook for your team. Here's how you can leverage it: * **Explore Similar Scenarios:** Review previous incidents to understand how similar issues were resolved. * **Contextual Information:** Access all necessary context directly within OpenMetadata, including previous resolutions, root causes, and responsible teams. * **Continuous Improvement:** Use historical data to improve data quality tests and prevent future failures. Steps to Get Started -------------------- 1. **Access the Incident Manager:** Navigate to the Incident Manager within the OpenMetadata platform. 2. **Monitor Data Quality Tests:** Keep an eye on your data quality tests to quickly identify any failures. 3. **Acknowledge and Assign:** Acknowledge any issues promptly and assign them to the appropriate team members. 4. **Log and Learn:** Document the root cause of each failure and use the stored information to learn and improve. By following these steps, you'll ensure that your organization effectively manages data quality issues, maintains high standards, and continuously improves its data quality processes. [How to work with the Incident Manager\ \ Set up the Incident Manager workflow.](https://docs.open-metadata.org/latest/how-to-guides/data-quality-observability/incident-manager/workflow) [Root Cause Analysis (Collate)\ \ Understand the nature of the failure and take corrective actions.](https://docs.open-metadata.org/latest/how-to-guides/data-quality-observability/incident-manager/root-cause-analysis) --- # Adding test suites through the UI We use cookies to improve site navigation, analyze site usage, and enhance your user experience. Click "Accept" to enable cookies or "Reject" to reject cookies. AcceptReject ![](https://static.scarf.sh/a.png?x-pxid=6814879a-96c0-40bd-be69-ddfeb14694aa) how-to-guides No menu items for this category [Home](https://docs.open-metadata.org/latest) /[How To Guides](https://docs.open-metadata.org/latest/how-to-guides) /[Data Quality Observability](https://docs.open-metadata.org/latest/how-to-guides/data-quality-observability) /[Quality](https://docs.open-metadata.org/latest/how-to-guides/data-quality-observability/quality) /[Adding Test Suites](https://docs.open-metadata.org/latest/how-to-guides/data-quality-observability/quality/adding-test-suites) OpenMetadata Documentation Adding Test Suites Through the UI ================================= Test Suites are logical container allowing you to group related Test Cases together from different tables. This is a great way to group related test cases together and set a single alert for test case failure. **Note:** you will need to make sure you have the right permission in OpenMetadata to create a test. Step 1: Creating a Test Suite ----------------------------- From the vertical navigation bar, click on `Quality icon > Data Quality` and navigate to the `By Test Suites` tab. From there click on `Add Test Suite` button on the top right corner. ![Write your first test](https://docs.open-metadata.org/images/v1.11/features/ingestion/workflows/data-quality/profiler-tab-view.png) Write your first test On the next page, enter the name and description (optional) of your test suite. ![Create test suite](https://docs.open-metadata.org/images/v1.11/features/ingestion/workflows/data-quality/test-suite-page.png) Create test suite Step 2: Add Test Cases ---------------------- On the next page, you will be able to add existing test cases from different entity to your test suite. This allows you to group together test cases from different entities **Note:** Test Case name needs to be unique across the whole platform. A warning message will show if your Test Case name is not unique. ![Create test case](https://docs.open-metadata.org/images/v1.11/features/ingestion/workflows/data-quality/test-case-page.png) Create test case Data Quality ------------ ### Adding Data Quality Test Cases from yaml config When creating a JSON config for a test workflow the source configuration is very simple. The only sections you need to modify here are the `serviceName` (this name needs to be unique) and `entityFullyQualifiedName` (the entity for which we'll be executing tests against) keys. Once you have defined your source configuration you'll need to define te processor configuration. The processor type should be set to `"orm-test-runner"`. For accepted test definition names and parameter value names refer to the [tests page](https://docs.open-metadata.org/latest/how-to-guides/data-quality-observability/quality/tests-yaml) . Note that while you can define tests directly in this YAML configuration, running the workflow will execute ALL THE TESTS present in the table, regardless of what you are defining in the YAML. This makes it easy for any user to contribute tests via the UI, while maintaining the test execution external. You can keep your YAML config as simple as follows if the table already has tests. ### Key reference: * `forceUpdate`: if the test case exists (base on the test case name) for the entity, implements the strategy to follow when running the test (i.e. whether or not to update parameters) * `testCases`: list of test cases to add to the entity referenced. Note that we will execute all the tests present in the Table. * `name`: test case name * `testDefinitionName`: test definition * `columnName`: only applies to column test. The name of the column to run the test against * `parameterValues`: parameter values of the test The `sink` and `workflowConfig` will have the same settings as the ingestion and profiler workflow. ### Full `yaml` config example ### How to Run Tests To run the tests from the CLI execute the following command --- # Profiler Tab | OpenMetadata Data Profiling Interface We use cookies to improve site navigation, analyze site usage, and enhance your user experience. Click "Accept" to enable cookies or "Reject" to reject cookies. AcceptReject ![](https://static.scarf.sh/a.png?x-pxid=6814879a-96c0-40bd-be69-ddfeb14694aa) how-to-guides No menu items for this category [Home](https://docs.open-metadata.org/latest) /[How To Guides](https://docs.open-metadata.org/latest/how-to-guides) /[Data Quality Observability](https://docs.open-metadata.org/latest/how-to-guides/data-quality-observability) /[Profiler](https://docs.open-metadata.org/latest/how-to-guides/data-quality-observability/profiler) /[Tab](https://docs.open-metadata.org/latest/how-to-guides/data-quality-observability/profiler/tab) OpenMetadata Documentation Profiler Tab ============ The Data Observability tab is displayed only for Tables. It has three sub-tabs for **Table Profile, Column Profile, and Data Quality**. Table Profile Tab ----------------- The table profile helps to monitor and understand the table structure. It displays the number of **rows and columns** in the table. You can view these details over a timeframe to understand how the table has been evolving. It displays the **profile sample** either as an absolute number or as a percentage of data. You also get details on the **size of the data** as well as when the table was created. ![Table Profile](https://docs.open-metadata.org/images/v1.11/how-to-guides/quality/tp.png) Table Profile The Table Profile tab also displays timeseries graphs on **Data Volume, Table Updates, and Volume Change**. ### Data Volume The **Data Volume** chart gives an overview on how the data is evolving across a time period. ![Table Profile: Data Volume](https://docs.open-metadata.org/images/v1.11/how-to-guides/quality/dv.png) Table Profile: Data Volume ### Table Updates In **Table Updates** chart, users can view the changes that happened in the table in terms of data inserts, updates, and deletes. ![Table Profile: Table Updates](https://docs.open-metadata.org/images/v1.11/how-to-guides/quality/tu.png) Table Profile: Table Updates ### Volume Change In **Volume Change** chart, users can view the changes that happened in the table in terms of data volume for inserts, updates, and deletes. ![Table Profile: Volume Change](https://docs.open-metadata.org/images/v1.11/how-to-guides/quality/vc.png) Table Profile: Volume Change Column Profile Tab ------------------ The Column Profile tab provides a summary of table metrics similar to the Table Profile tab. It displays the number of **rows and columns** over a period of time. It displays the **profile sample** either as an absolute number or as a percentage of data. You also get details on the **size of the data** as well as when the table was created. The column profile helps to monitor and understand the column structure with a summary of metrics for every column. You can view the type of each column, the value count, null value %, distinct value %, unique %, the tests run as well as the test status. ![Column Profile of a Table](https://docs.open-metadata.org/images/v1.11/how-to-guides/quality/cp.png) Column Profile of a Table By clicking on any column, you can view more detailed reports about that column. ![Column Profile of a Column](https://docs.open-metadata.org/images/v1.11/how-to-guides/quality/cp1.png) Column Profile of a Column The Column Profile for a particular column also displays timeseries graphs on **Data Counts, Data Proportions, Data Range, Data Aggregate, Data Quartiles, and Data Distribution**. Based on the type of column you are viewing, you can verify the accuracy of the data values. ### Data Counts The data counts chart provides information on the **Distinct Count, Null Count, Unique Count, and Values Count**. ![Column Profile: Data Counts](https://docs.open-metadata.org/images/v1.11/how-to-guides/quality/dc.png) Column Profile: Data Counts ### Data Proportions The data proportions chart displays the **Distinct, Null, and Unique Proportions**. ![Column Profile: Data Proportions](https://docs.open-metadata.org/images/v1.11/how-to-guides/quality/dp.png) Column Profile: Data Proportions ### Data Range The length of the string that are stored in the database is profiled. The data range displays the Minimum, Maximum, and Mean values, which can be helpful for users who are doing an NLP or Text analysis. ![Column Profile: Data Range](https://docs.open-metadata.org/images/v1.11/how-to-guides/quality/dr.png) Column Profile: Data Range ### Data Aggregate ![Column Profile: Data Aggregate](https://docs.open-metadata.org/images/v1.11/how-to-guides/quality/da.png) Column Profile: Data Aggregate ### Data Quartiles This chart displays the First Quartile, Median, Inter Quartile Range, and the Third Quartile. ![Column Profile: Data Quartiles](https://docs.open-metadata.org/images/v1.11/how-to-guides/quality/dq.png) Column Profile: Data Quartiles ### Data Distribution The distribution of the character length inside the column is displayed to help you get a sense of the structure of your data. ![Column Profile: Data Distribution](https://docs.open-metadata.org/images/v1.11/how-to-guides/quality/dd.png) Column Profile: Data Distribution --- # Adding Test Cases to an Entity We use cookies to improve site navigation, analyze site usage, and enhance your user experience. Click "Accept" to enable cookies or "Reject" to reject cookies. AcceptReject ![](https://static.scarf.sh/a.png?x-pxid=6814879a-96c0-40bd-be69-ddfeb14694aa) how-to-guides No menu items for this category [Home](https://docs.open-metadata.org/latest) /[How To Guides](https://docs.open-metadata.org/latest/how-to-guides) /[Data Quality Observability](https://docs.open-metadata.org/latest/how-to-guides/data-quality-observability) /[Quality](https://docs.open-metadata.org/latest/how-to-guides/data-quality-observability/quality) /[Adding Test Cases](https://docs.open-metadata.org/latest/how-to-guides/data-quality-observability/quality/adding-test-cases) OpenMetadata Documentation Adding Test Cases to an Entity ============================== Tests cases are actual test that will be ran and executed against your entity. This is where you will define the execution time and logic of these tests **Note:** you will need to make sure you have the right permission in OpenMetadata to create a test. Step 1: Creating a Test Case ---------------------------- Navigate to the entity you want to add a test to (we currently support quality test only for database entity). Go to `Data Observability` tab. From there, click on the `Add Test` button in the upper right corner and select the type of test you want to implement ![Write your first test](https://docs.open-metadata.org/images/v1.11/features/ingestion/workflows/data-quality/add-test-case.png) Write your first test Step 2: Select the Test Definition ---------------------------------- Select the type of test you want to run and set the parameters (if any) for your test case. If you have selected a `column` test, you will need to select which column you want to execute your test against. Give it a name and then submit it. **Note:** if you have a profiler workflow running, you will be able to visualize some context around your column or table data. ![Write your first test](https://docs.open-metadata.org/images/v1.11/features/ingestion/workflows/data-quality/add-test-defintion.png) Write your first test Step 3: Set an Execution Schedule (Optional) -------------------------------------------- Starting in 1.6 it is possible to create multiple pipeline for your test cases. If you want to execute all of your test cases within the same pipeline you can simply toggle on the `Select All` on the ingestion configuration page. Otherwise you can select the specific test cases the pipeline will execute. The second options allows you to orchestrate pipelines at different times for different test cases. ![Create an ingestion pipeline](https://docs.open-metadata.org/images/v1.11/features/ingestion/workflows/data-quality/add-ingestion.png) Create an ingestion pipeline ![Schedule you test execution](https://docs.open-metadata.org/images/v1.11/features/ingestion/workflows/data-quality/ingestion-page.png) Schedule you test execution --- # External Profiler Workflow | Official Documentation We use cookies to improve site navigation, analyze site usage, and enhance your user experience. Click "Accept" to enable cookies or "Reject" to reject cookies. AcceptReject ![](https://static.scarf.sh/a.png?x-pxid=6814879a-96c0-40bd-be69-ddfeb14694aa) how-to-guides No menu items for this category [Home](https://docs.open-metadata.org/latest) /[How To Guides](https://docs.open-metadata.org/latest/how-to-guides) /[Data Quality Observability](https://docs.open-metadata.org/latest/how-to-guides/data-quality-observability) /[Profiler](https://docs.open-metadata.org/latest/how-to-guides/data-quality-observability/profiler) /[External Workflow](https://docs.open-metadata.org/latest/how-to-guides/data-quality-observability/profiler/external-workflow) OpenMetadata Documentation External Profiler Workflow ========================== Note that this requires OpenMetadata 1.2.1 or higher. Consider a use case where you have a large database source with multiple databases and schemas which are maintained by different teams within your organization. You have created multiple database services within OpenMetadata depending on your use case by applying various filters on this large source. Now, instead of running a profiler pipeline for each service, you want to run a **single workflow profiler for the entire source**, irrespective of the OpenMetadata service which an asset would belong to. This document will guide you on how to achieve this. Note that running a single profiler workflow is only supported if you run the workflow **externally**, not from OpenMetadata. How to Run the Connector Externally ----------------------------------- To run the Ingestion via the UI you'll need to use the OpenMetadata Ingestion Container, which comes shipped with custom Airflow plugins to handle the workflow deployment. If, instead, you want to manage your workflows externally on your preferred orchestrator, you can check the following docs to run the Ingestion Framework **anywhere**. [#### External Schedulers\ \ Get more information about running the Ingestion Framework Externally](https://docs.open-metadata.org/latest/deployment/ingestion) ### Requirements In order to run the external profiler with external sample data you will need to install the following packages: Where `` is the name of the connector that you want to run against. Each specific installation command will be shared on its documentation [page](https://docs.open-metadata.org/latest/connectors/database) . For example, to run against Athena, we need to install: * The `athena` plugin will bring all the requirements to connect to the Athena Service * The `datalake` plugin helps us connect to S3 to manage the sample data. * The `trino` plugin will only be needed temporarily 1\. Define the YAML Config -------------------------- You will need to prepare a yaml file for the data profiler depending on the database source. You can get details of how to define a yaml file for data profiler for each connector [here](https://docs.open-metadata.org/latest/connectors/database) . For example, consider if the data source was snowflake, then the yaml file would have looked like as follows. Note that we do **NOT pass the Service Name** in this yaml file, unlike your typical profiler workflow 2\. Run the Workflow -------------------- ### Run the Workflow with the CLI One option to running the workflow externally is by leveraging the `metadata` CLI. After saving the YAML config, we will run the command: ### Run the Workflow from Python using the SDK If you'd rather have a Python script taking care of the execution, you can use: --- # Alerts & Notifications | OpenMetadata Guide We use cookies to improve site navigation, analyze site usage, and enhance your user experience. Click "Accept" to enable cookies or "Reject" to reject cookies. AcceptReject ![](https://static.scarf.sh/a.png?x-pxid=6814879a-96c0-40bd-be69-ddfeb14694aa) how-to-guides No menu items for this category [Home](https://docs.open-metadata.org/latest) /[How To Guides](https://docs.open-metadata.org/latest/how-to-guides) /[Data Quality Observability](https://docs.open-metadata.org/latest/how-to-guides/data-quality-observability) /[Alerts Notifications](https://docs.open-metadata.org/latest/how-to-guides/data-quality-observability/alerts-notifications) OpenMetadata Documentation Alerts & Notifications ====================== OpenMetadata has been providing observability alerts right from the start to notify users of important data lifecycle events: schema modifications, ownership shifts, and tagging updates. Users can define fine-grained alerts and notifications. Starting from the 1.3 release, Data Observability alerts have been completely revamped, simplifying the process of monitoring data. Users can quickly create alerts for: * **Changes in the Metadata:** such as schema changes, * **Data Quality Failures:** to filter by Test Suite, * **Pipeline Status Failures:** when ingesting runs from your ETL systems, and * **Ingestion Pipeline Monitoring:** for OpenMetadata’s ingestion workflows Depending on your use cases, notifications can be sent to owners, admins, teams, or users, providing a more personalized and informed experience. Teams can configure their dedicated Slack, MS Teams, or Google Chat channels to receive notifications related to their data assets, streamlining communication and collaboration. With the alerts and notifications in OpenMetadata, users can send Announcements over email, Slack, or Teams. Alerts are sent to a user when they are mentioned in a task or an activity feed. Observability Alerts & Notifications ------------------------------------ OpenMetadata provides a unified **Event Subscription** framework that allows you to configure alerts for two main purposes: 1. **Data Observability:** Monitoring the health of data systems (e.g., Data Quality failures, Schema Drift, Pipeline failures). 2. **System & Governance Notifications:** Monitoring metadata changes and collaboration events (e.g., new Glossary terms, Tag updates, Ownership changes). While these alerts serve different use cases and are accessed from different menus within the platform, the configuration process follows a similar workflow. ### Required Permissions [Required Permissions\ \ Setting up alerts and notifications requires **Create** permission on the entity `EventSubscription`.](https://docs.open-metadata.org/latest/how-to-guides/admin-guide/roles-policies) ### Configuration Workflow Select your use case below to see the specific configuration steps: [#### Data Observability Alerts\ \ Monitor the health of your data systems with alerts for pipeline failures, data quality issues, and schema changes](https://docs.open-metadata.org/latest/how-to-guides/data-quality-observability/alerts-notifications/data-observability-alerts) [#### System & Governance Notifications\ \ Stay informed about metadata changes, governance events, and collaboration activities across your data catalog](https://docs.open-metadata.org/latest/how-to-guides/data-quality-observability/alerts-notifications/system-governance-notifications) ### Additional Details for the Configuration of External Destinations OpenMetadata supports multiple external destinations for delivering alerts outside the platform. Each destination type requires a specific setup process. [#### Email\ \ Send alerts directly to recipient email addresses using your OpenMetadata server's email configuration](https://docs.open-metadata.org/latest/how-to-guides/data-quality-observability/alerts-notifications/email-alerts-configuration) [#### Slack\ \ Deliver alerts to Slack channels using Incoming Webhooks](https://docs.open-metadata.org/latest/how-to-guides/data-quality-observability/alerts-notifications/slack-alerts-configuration) [#### Microsoft Teams\ \ Post alerts to Teams channels using Workflow Webhooks with Adaptive Cards](https://docs.open-metadata.org/latest/how-to-guides/data-quality-observability/alerts-notifications/microsoft-teams-alerts-configuration) [#### Google Chat\ \ Send alerts to Google Chat Spaces using Webhooks](https://docs.open-metadata.org/latest/how-to-guides/data-quality-observability/alerts-notifications/google-chat-alerts-configuration) [#### Generic Webhook\ \ Integrate with custom applications using generic webhooks](https://docs.open-metadata.org/latest/how-to-guides/data-quality-observability/alerts-notifications/generic-webhook-alerts-configuration) --- # Data Observability Alerts | OpenMetadata We use cookies to improve site navigation, analyze site usage, and enhance your user experience. Click "Accept" to enable cookies or "Reject" to reject cookies. AcceptReject ![](https://static.scarf.sh/a.png?x-pxid=6814879a-96c0-40bd-be69-ddfeb14694aa) how-to-guides No menu items for this category [Home](https://docs.open-metadata.org/latest) /[How To Guides](https://docs.open-metadata.org/latest/how-to-guides) /[Data Quality Observability](https://docs.open-metadata.org/latest/how-to-guides/data-quality-observability) /[Alerts Notifications](https://docs.open-metadata.org/latest/how-to-guides/data-quality-observability/alerts-notifications) /[Data Observability Alerts](https://docs.open-metadata.org/latest/how-to-guides/data-quality-observability/alerts-notifications/data-observability-alerts) OpenMetadata Documentation Data Observability Alerts ========================= Monitor the health of your data systems by setting up alerts for pipeline failures, data quality issues, and schema changes. #### Open the Alerts Page Navigate to the lateral menu **Observability > Alerts**. ![Navigate to Observability Alerts](https://docs.open-metadata.org/images/v1.11/how-to-guides/data-quality-observability/observability/alerts/data-observability-navigate-om.webp) Navigate to Observability Alerts #### Create a new Alert Click the **Add Alert** button. ![Add Alert Button](https://docs.open-metadata.org/images/v1.11/how-to-guides/data-quality-observability/observability/alerts/data-observability-add-alert.webp) Add Alert Button #### Name the Alert and Add Context Provide a unique, descriptive **Name** for your alert. You can also add an optional **Description** to provide further context and clarity regarding the alert's intent. ![Input fields for Alert Name and Description](https://docs.open-metadata.org/images/v1.11/how-to-guides/data-quality-observability/observability/alerts/data-observability-alert-name-description.webp) Define the Alert Name and Optional Description #### Select a Source Choose the operational entity you want to monitor: * **Container** - Monitors schema changes for the container asset * **Pipeline** - Monitors updates to pipeline assets that you have ingested * **Table** - Monitors schema changes and table metrics changes * **Test Case** - Triggers an alert for the specific test case selected * **Test Suite** - Triggers an alert for any test case event linked to the test suite. This is a great way to group alerts and reduce notification fatigue * **Topic** - Monitors schema changes for the topic asset ![Select the Source Resource for Data Observability](https://docs.open-metadata.org/images/v1.11/how-to-guides/data-quality-observability/observability/alerts/data-observability-source-om.webp) Select the Source Resource #### Configure Filters (Optional) Filters allow you to refine the scope of the alert to focus only on relevant changes, significantly improving the signal-to-noise ratio. You can narrow down events based on a variety of criteria, including: * **Entity Specific Name:** Filter by the defined specific name of the entity. * **Owner Name:** Filter events based on the designated owner of the asset. * **Domain:** Filter events based on the Data Domain the entity belongs to. * **Filter By Updater Is Bot:** Filter to include or exclude changes made by automated ingestion or system processes. Use the **Include** toggle to define the logic for the filter condition: * **Include (Toggle ON):** If the event meets the filter condition, the alert is **sent**. * **Exclude (Toggle OFF):** If the event meets the filter condition, the alert is **silenced** (not sent). If you do not set any filter, the alert will apply to **all** relevant events over the selected source entity type, which may lead to excessive notifications. ![Define Filters](https://docs.open-metadata.org/images/v1.11/how-to-guides/data-quality-observability/observability/alerts/data-observability-filter-overview.webp) Define Filters ![Filter Options](https://docs.open-metadata.org/images/v1.11/how-to-guides/data-quality-observability/observability/alerts/data-observability-filter-options.webp) Filtering options for TestSuite source Entity. #### Select Trigger Conditions (Optional) Define the specific conditions that will trigger the alert: * **Schema Changes** - Alert on added, deleted, or updated columns * **Test Case Status** - Trigger when tests are `Failed`, `Aborted`, or `Queued` * **Pipeline Status** - Alert when pipeline execution is `Failed` or `Pending` * **Metric Updates** - Notify when table metrics are updated You can select multiple trigger conditions to create comprehensive monitoring coverage. ![Select Trigger Conditions](https://docs.open-metadata.org/images/v1.11/how-to-guides/admin-guide/trigger.webp) Select Trigger Conditions #### Select Destinations Choose where to send your alerts. OpenMetadata supports both internal and external channels. **Internal Destinations:** * **Admins** - Notify all platform administrators * **Followers** - Notify users following the asset * **Owners** - Alert the asset owners * **Teams or Specific Users** - Target specific teams or individuals **External Destinations:** * **Email** * **Chat**: Slack, MS Teams, Google Chat * **Automation**: Generic Webhooks ![Internal Destinations](https://docs.open-metadata.org/images/v1.11/how-to-guides/admin-guide/internal.webp) Internal Destinations ![External Destinations](https://docs.open-metadata.org/images/v1.11/how-to-guides/admin-guide/external.webp) External Destinations --- # System & Governance Notifications | OpenMetadata We use cookies to improve site navigation, analyze site usage, and enhance your user experience. Click "Accept" to enable cookies or "Reject" to reject cookies. AcceptReject ![](https://static.scarf.sh/a.png?x-pxid=6814879a-96c0-40bd-be69-ddfeb14694aa) how-to-guides No menu items for this category [Home](https://docs.open-metadata.org/latest) /[How To Guides](https://docs.open-metadata.org/latest/how-to-guides) /[Data Quality Observability](https://docs.open-metadata.org/latest/how-to-guides/data-quality-observability) /[Alerts Notifications](https://docs.open-metadata.org/latest/how-to-guides/data-quality-observability/alerts-notifications) /[System Governance Notifications](https://docs.open-metadata.org/latest/how-to-guides/data-quality-observability/alerts-notifications/system-governance-notifications) OpenMetadata Documentation System & Governance Notifications ================================= Stay informed about metadata changes, governance events, and collaboration activities across your data catalog. #### Open the Notifications Page Navigate to **Settings > Notifications > System**. #### Create a new Alert Click the **Add Notification** button. ![Add Notification Button](https://docs.open-metadata.org/images/v1.11/how-to-guides/data-quality-observability/observability/alerts/system-notifications-add-alert.webp) Add Notification Button #### Name the Alert and Add Context Provide a unique, descriptive **Name** for your alert. You can also add an optional **Description** to provide further context and clarity regarding the alert's intent. ![Input fields for Alert Name and Description](https://docs.open-metadata.org/images/v1.11/how-to-guides/data-quality-observability/observability/alerts/system-notifications-alert-name-description.webp) Define the Alert Name and Optional Description #### Select the Metadata Entity to Monitor Choose the specific entity type whose changes you want to track. These sources include data assets (like **Table** or **Pipeline**) as well as governance and platform entities (like **Glossary**, **Tag**, **User**, or **Announcement**). Any relevant change to the selected entity will generate an event for this notification. ![Select the Source Resource for System Notifications](https://docs.open-metadata.org/images/v1.11/how-to-guides/data-quality-observability/observability/alerts/system-notifications-source.webp) Select the Source Resource #### Configure Filters (Optional) If you do not set any filter, the alert will apply to **all** events over the selected source entity type. Filters allow refining the scope of the alert to focus only on relevant changes. The available filter criteria depend on the source entity selected. Events can be narrowed down basing on a variety of criteria, including: * **Owner:** Filter events based on the designated owner of the asset. * **Entity FQN:** Filter by the Fully Qualified Name of the entity. * **Event Type:** Filter by the specific action that occurred (e.g., Created, Updated, Deleted). * **Updater Name:** Filter based on the user or service that executed the change. * **Domain:** Filter events based on the Data Domain the entity belongs to. * **Filter By Updater Is Bot:** Filter to include or exclude changes made by automated ingestion or system processes. Use the **Include** toggle to define the logic for the filter condition: * **Include (Toggle ON):** If the event meets the filter condition, the alert is **sent**. * **Exclude (Toggle OFF):** If the event meets the filter condition, the alert is **silenced** (not sent). You can add multiple filters to a single alert subscription. ![Define Filters](https://docs.open-metadata.org/images/v1.11/how-to-guides/data-quality-observability/observability/alerts/system-notifications-filter-0.webp) Define Filters ![Define Filters](https://docs.open-metadata.org/images/v1.11/how-to-guides/data-quality-observability/observability/alerts/system-notifications-filter-1.webp) Define Filters #### Select Destinations Choose where to send your alerts. OpenMetadata supports both internal and external channels. **Internal Destinations:** * **Admins** - Notify all platform administrators * **Followers** - Notify users following the asset * **Owners** - Alert the asset owners * **Teams or Specific Users** - Target specific teams or individuals **External Destinations:** * **Email** * **Chat**: Slack, MS Teams, Google Chat * **Automation**: Generic Webhooks ![Internal Destinations](https://docs.open-metadata.org/images/v1.11/how-to-guides/admin-guide/internal.webp) Internal Destinations ![External Destinations](https://docs.open-metadata.org/images/v1.11/how-to-guides/admin-guide/external.webp) External Destinations --- # How to work with the Incident Manager | Official Documentation We use cookies to improve site navigation, analyze site usage, and enhance your user experience. Click "Accept" to enable cookies or "Reject" to reject cookies. AcceptReject ![](https://static.scarf.sh/a.png?x-pxid=6814879a-96c0-40bd-be69-ddfeb14694aa) how-to-guides No menu items for this category [Home](https://docs.open-metadata.org/latest) /[How To Guides](https://docs.open-metadata.org/latest/how-to-guides) /[Data Quality Observability](https://docs.open-metadata.org/latest/how-to-guides/data-quality-observability) /[Incident Manager](https://docs.open-metadata.org/latest/how-to-guides/data-quality-observability/incident-manager) /[Workflow](https://docs.open-metadata.org/latest/how-to-guides/data-quality-observability/incident-manager/workflow) OpenMetadata Documentation How to Work with the Incident Manager Workflow ============================================== 1\. Incident Dashboard ---------------------- The Incident Dashboard is the central hub where all incidents are displayed. Users can filter incidents by various criteria to manage and prioritize them effectively. ### Filters Available: * **Assignee:** View incidents assigned to specific team members. * **Status:** Filter incidents based on their current status (e.g., New, ACK, Assigned, Resolved). * **Test Cases:** Filter incidents associated with specific test cases. * **Time:** Sort incidents by the time they were reported or last updated. ![Incident Manager Dashboard](https://docs.open-metadata.org/images/v1.11/how-to-guides/observability/incident-manager-1.png) Incident Manager Dashboard 2\. Incident Status Change -------------------------- Incident status can be updated to reflect the current stage of the incident resolution process. The owner of the incident has the ability to assign it to an appropriate assignee for further action. ### Steps to Change Incident Status: 1. Navigate to the Incident Dashboard. 2. Select the incident that needs a status update. 3. Choose the new status from the dropdown menu. 4. Assign the incident to the appropriate team member. 5. You can review the Test Case Details. ![Incident Test Case Details](https://docs.open-metadata.org/images/v1.11/how-to-guides/observability/incident-manager-2.png) Incident Test Case Details ![Incident Status Change](https://docs.open-metadata.org/images/v1.11/how-to-guides/observability/incident-manager-3.png) Incident Status Change 3\. Incident Resolution ----------------------- Once an incident has been resolved, it can be officially closed. Ensure to describe a Root Cause Analysis (RCA) in the comments to provide context and understanding of the resolution process. ### Steps to Resolve and Close an Incident: 1. Verify that all necessary steps to resolve the incident have been completed. 2. Describe the RCA in the resolution comments. 3. Change the status of the incident to 'Resolved'. 4. Confirm the closure to update the incident in the dashboard. ![Incident Resolution](https://docs.open-metadata.org/images/v1.11/how-to-guides/observability/incident-manager-4.png) Incident Resolution 4\. Incident Activities ----------------------- Each incident includes a detailed timeline where all relevant information is consolidated. This timeline provides a comprehensive view of the incident's lifecycle, including key events, RCA documentation, and closure updates. ### How to View Incident Activities: 1. Open the incident from the Incident Dashboard. 2. Navigate to the 'Incident' tab within the incident details. 3. Review the chronological events, RCA, and closure updates associated with the incident. ![Incident Activities](https://docs.open-metadata.org/images/v1.11/how-to-guides/observability/incident-manager-5.png) Incident Activities --- # Microsoft Teams Alerts Configuration | OpenMetadata We use cookies to improve site navigation, analyze site usage, and enhance your user experience. Click "Accept" to enable cookies or "Reject" to reject cookies. AcceptReject ![](https://static.scarf.sh/a.png?x-pxid=6814879a-96c0-40bd-be69-ddfeb14694aa) how-to-guides No menu items for this category [Home](https://docs.open-metadata.org/latest) /[How To Guides](https://docs.open-metadata.org/latest/how-to-guides) /[Data Quality Observability](https://docs.open-metadata.org/latest/how-to-guides/data-quality-observability) /[Alerts Notifications](https://docs.open-metadata.org/latest/how-to-guides/data-quality-observability/alerts-notifications) /[Microsoft Teams Alerts Configuration](https://docs.open-metadata.org/latest/how-to-guides/data-quality-observability/alerts-notifications/microsoft-teams-alerts-configuration) OpenMetadata Documentation Microsoft Teams Alerts Configuration ------------------------------------ OpenMetadata integrates with Microsoft Teams using **Workflow Webhooks** to deliver alert messages directly to your Teams channels. This allows you to receive real-time notifications within Microsoft Teams. ### Important Migration Notice **Microsoft Deprecation Update**: Microsoft has deprecated the legacy **Office 365 Connector** approach in favor of **Workflow Webhooks**. **If you are using OpenMetadata v1.11.0 or later:** * βœ… No code changes are required * βœ… OpenMetadata already sends **Adaptive Cards** format * ⚠️ **Action Required**: If you previously saved an **Office 365 Connector URL**, you must replace it with a new **Workflow Webhook URL** For detailed Microsoft guidance, see: [Create incoming webhooks with Workflows for Microsoft Teams](https://support.microsoft.com/en-gb/office/create-incoming-webhooks-with-workflows-for-microsoft-teams-8ae491c7-0394-4861-ba59-055e33f75498) ### Setting Up a Microsoft Teams Workflow Webhook An incoming webhook allows OpenMetadata to share alert content directly in your Teams channels and chats. Follow these steps using Microsoft's pre-configured webhook templates. #### Step 1: Access Workflows from Your Channel 1. In **Microsoft Teams**, navigate to the **channel** or **chat** where you want alerts to be posted 2. Click **More options** (...) next to the channel or chat name 3. Select **Workflows** #### Step 2: Select the Webhook Template Microsoft provides pre-configured webhook templates. Search and select the "**Send webhook alerts to a channel**" pre-configured Webhook template to continue. #### Step 3: Name and Configure the Webhook 1. Enter a **Name** for the workflow (e.g., "OpenMetadata Alerts") 2. Optionally, add a **Description** (e.g., "Receives data quality and governance alerts from OpenMetadata") 3. Review the authentication settings (the defaults are typically fine) 4. If you need to use a different account to post messages, click **Switch account** 5. Click **Next** #### Step 4: Select the Destination 1. The workflow will ask you to specify where to post messages 2. Select your **Team**, **Channel**, or **Chat** (these fields auto-populate if you started from a specific channel) 3. Verify the account authentication is correct 4. Click **Add workflow** to create the workflow #### Step 5: Copy Your Webhook URL 1. A confirmation dialog will appear displaying your **HTTP POST URL** 2. **Copy** the complete URL exactly as shown (do not modify or truncate it) 3. Store this URL securely - you'll paste it into OpenMetadata next #### Finding Your Webhook URL Later If you need to retrieve your webhook URL again: 1. Open the **Workflows** app in Microsoft Teams 2. Find and select your OpenMetadata workflow 3. Click **Edit** 4. Expand **When a Teams webhook request is received** 5. Copy the **HTTP POST URL** displayed there ### Configuring Microsoft Teams Webhooks in OpenMetadata Once you have your Workflow Webhook URL, follow these steps to configure it in OpenMetadata: #### Step 1: Access Alert Configuration 1. In OpenMetadata, navigate to **Alerts & Notifications** from the main menu 2. Select the type of alert you want to configure: * [**Data Observability Alerts**](https://docs.open-metadata.org/latest/how-to-guides/data-quality-observability/alerts-notifications/data-observability-alerts) (for data quality and pipeline monitoring) * [**System & Governance Notifications**](https://docs.open-metadata.org/latest/how-to-guides/data-quality-observability/alerts-notifications/system-governance-notifications) (for metadata and governance events) #### Step 2: Add Microsoft Teams as a Destination 1. Click **Add Destination** 2. Select **Microsoft Teams** from the available destination options 3. Paste your **Workflow Webhook HTTP POST URL** into the **MS Teams Destination (HTTP POST URL)** field 4. Optionally, enter a **Display Name** for this Teams destination (e.g., "Data Quality Alerts") #### Step 3: Test the Connection 1. Click **Test Connection** to verify the webhook is working 2. A test message will be sent to your Teams channel 3. If successful, you'll see a confirmation message and the test message will appear in your Teams channel as an Adaptive Card #### Step 4: Save and Enable 1. Click **Save** to store the configuration 2. The Microsoft Teams destination is now ready to receive alerts ### Best Practices * **Dedicated Channels**: Create separate Teams channels for different types of alerts (e.g., #data-quality-alerts, #pipeline-failures, #governance-events) * **Channel Members**: Ensure all relevant team members are added to the Teams channel * **Multiple Webhooks**: You can configure multiple Workflow Webhooks to send different alert types to different channels * **Workflow Testing**: Always test your Workflow webhook in a non-production channel first * **Notifications**: Enable notifications for the channel so team members don't miss important alerts ### Migration Guide: From Office 365 Connector to Workflow Webhook If you're currently using the **legacy Office 365 Connector URLs**: 1. **Create a new Workflow Webhook** following the setup steps above 2. **Copy the new Webhook URL** from the Workflow 3. **Update OpenMetadata** with the new Workflow Webhook URL 4. **Test the configuration** in a test channel first 5. **Delete the old Office 365 Connector** from Microsoft Teams once the new Workflow is working No code changes are required - only the webhook URL needs to be updated. ### Reference For more information about Microsoft Teams Workflow Webhooks, visit [Create incoming webhooks with Workflows for Microsoft Teams](https://support.microsoft.com/en-gb/office/create-incoming-webhooks-with-workflows-for-microsoft-teams-8ae491c7-0394-4861-ba59-055e33f75498) . --- # Root Cause Analysis | OpenMetadata Incident Management We use cookies to improve site navigation, analyze site usage, and enhance your user experience. Click "Accept" to enable cookies or "Reject" to reject cookies. AcceptReject ![](https://static.scarf.sh/a.png?x-pxid=6814879a-96c0-40bd-be69-ddfeb14694aa) how-to-guides No menu items for this category [Home](https://docs.open-metadata.org/latest) /[How To Guides](https://docs.open-metadata.org/latest/how-to-guides) /[Data Quality Observability](https://docs.open-metadata.org/latest/how-to-guides/data-quality-observability) /[Incident Manager](https://docs.open-metadata.org/latest/how-to-guides/data-quality-observability/incident-manager) /[Root Cause Analysis](https://docs.open-metadata.org/latest/how-to-guides/data-quality-observability/incident-manager/root-cause-analysis) OpenMetadata Documentation Root Cause Analysis =================== Failed Rows Sample ------------------ Some tests will produce a failed sample upon failure. This allows the platform users to understand the nature of the failure and take corrective actions. The failed sample will be a subset of the rows that failed the test. The sample will be collected when the option `computePassedFailedRowCount` is set. ### Supported Test Definitions * [Column Values to Be Not Null](https://docs.open-metadata.org/latest/how-to-guides/data-quality-observability/quality/tests-yaml#column-values-to-be-not-null) * [Column Values to Match Regex](https://docs.open-metadata.org/latest/how-to-guides/data-quality-observability/quality/tests-yaml#column-values-to-match-regex) * [Column Values to not Match Regex](https://docs.open-metadata.org/latest/how-to-guides/data-quality-observability/quality/tests-yaml#column-values-to-not-match-regex) * [Column Values to Be in Set](https://docs.open-metadata.org/latest/how-to-guides/data-quality-observability/quality/tests-yaml#column-values-to-be-in-set) * [Column Values to Be Not In Set](https://docs.open-metadata.org/latest/how-to-guides/data-quality-observability/quality/tests-yaml#column-values-to-be-not-in-set) * [Column Values to Be Between](https://docs.open-metadata.org/latest/how-to-guides/data-quality-observability/quality/tests-yaml#column-values-to-be-between) * [Column Values Lengths to Be Between](https://docs.open-metadata.org/latest/how-to-guides/data-quality-observability/quality/tests-yaml#column-values-lengths-to-be-between) * [Custom SQL](https://docs.open-metadata.org/latest/how-to-guides/data-quality-observability/quality/tests-yaml#table-custom-sql-test) ### Deleting Sample Rows If you wish to delete sample rows, you can do so by clicking on the three dots above the table of sample rows. This will open a window with the `Delete` option. Note that failed sample rows will automatically be deleted upon test success. ![set compute row count](https://docs.open-metadata.org/images/v1.11/features/ingestion/workflows/data-quality/sample-row-failure-deletion.png) ### Example ![set compute row count](https://docs.open-metadata.org/images/v1.11/features/ingestion/workflows/data-quality/set_compute_row_count.png) ![test definition](https://docs.open-metadata.org/images/v1.11/features/ingestion/workflows/data-quality/failed_rows_sample_1.png) ![failed rows sampls](https://docs.open-metadata.org/images/v1.11/features/ingestion/workflows/data-quality/failed_rows_sample_2.png) Inspection Query ---------------- Supported test will generate an inspection query upon failure. This query can be run on the source data to understand the nature of the failure and take corrective actions. This query can be added to the table and shared with other users. ![inspection query](https://docs.open-metadata.org/images/v1.11/features/ingestion/workflows/data-quality/inspection-query.png) --- # Email Alerts Configuration | OpenMetadata We use cookies to improve site navigation, analyze site usage, and enhance your user experience. Click "Accept" to enable cookies or "Reject" to reject cookies. AcceptReject ![](https://static.scarf.sh/a.png?x-pxid=6814879a-96c0-40bd-be69-ddfeb14694aa) how-to-guides No menu items for this category [Home](https://docs.open-metadata.org/latest) /[How To Guides](https://docs.open-metadata.org/latest/how-to-guides) /[Data Quality Observability](https://docs.open-metadata.org/latest/how-to-guides/data-quality-observability) /[Alerts Notifications](https://docs.open-metadata.org/latest/how-to-guides/data-quality-observability/alerts-notifications) /[Email Alerts Configuration](https://docs.open-metadata.org/latest/how-to-guides/data-quality-observability/alerts-notifications/email-alerts-configuration) OpenMetadata Documentation Email Alerts Configuration -------------------------- OpenMetadata can send emails for various events such as sign-up confirmations, password resets, and alert notifications. To enable email alerts, you must first configure the SMTP server in the OpenMetadata platform. ### Configuring Email SMTP Settings To configure email notifications in OpenMetadata, navigate to **Settings > Preferences > Email**. ![Email Configuration](https://docs.open-metadata.org/images/v1.11/how-to-guides/admin-guide/email.webp) Email Configuration UI in OpenMetadata ### Email Configuration Fields #### Username The username of your SMTP account used for authentication. #### Password The password associated with the SMTP account username. #### Sender Email The email address that will appear as the sender in emails. This can be the same as the username, but some services like Amazon SES may allow a different email address. #### Server Endpoint The endpoint of your SMTP server. Examples: * `smtp.gmail.com` (for Gmail) * `smtp.sendgrid.net` (for SendGrid) * Your organization's SMTP server address #### Server Port The port number of the SMTP server. The port depends on the transportation strategy: | Transportation Strategy | Port | Description | | --- | --- | --- | | SMTP | 25 | Standard unencrypted SMTP | | SMTPS | 465 | SMTP with implicit TLS encryption | | SMTP\_TLS | 587 | SMTP with explicit TLS encryption (recommended) | #### Transportation Strategy Select the appropriate transportation strategy based on your SMTP server configuration: * **SMTP**: For unencrypted connections (port 25) * **SMTPS**: For implicit TLS encryption (port 465) * **SMTP\_TLS**: For explicit TLS encryption (port 587) - Recommended #### Emailing Entity The name of the entity that appears in email subjects and content. By default, this is set to "OpenMetadata". For example, if you set this to "JohnDoe", emails will show "JohnDoe" instead of "OpenMetadata" in the subject lines and email body. #### Enable SMTP Server A toggle to enable or disable the SMTP configuration. Set to `true` to activate email notifications, or `false` to disable them. #### Support URL A support URL link that will be included in emails for users to reach out in case of issues. Default: `https://slack.open-metadata.org` You can update this to point to your internal support channels or documentation. --- # Google Chat Alerts Configuration | OpenMetadata We use cookies to improve site navigation, analyze site usage, and enhance your user experience. Click "Accept" to enable cookies or "Reject" to reject cookies. AcceptReject ![](https://static.scarf.sh/a.png?x-pxid=6814879a-96c0-40bd-be69-ddfeb14694aa) how-to-guides No menu items for this category [Home](https://docs.open-metadata.org/latest) /[How To Guides](https://docs.open-metadata.org/latest/how-to-guides) /[Data Quality Observability](https://docs.open-metadata.org/latest/how-to-guides/data-quality-observability) /[Alerts Notifications](https://docs.open-metadata.org/latest/how-to-guides/data-quality-observability/alerts-notifications) /[Google Chat Alerts Configuration](https://docs.open-metadata.org/latest/how-to-guides/data-quality-observability/alerts-notifications/google-chat-alerts-configuration) OpenMetadata Documentation Google Chat Alerts Configuration -------------------------------- OpenMetadata integrates with Google Chat using **Webhooks** to deliver alert messages directly to your Google Chat Spaces. This allows you to receive real-time notifications within your Google Chat workspace. ### Setting Up a Google Chat Webhook Follow these steps to create a Webhook in your Google Chat Space: #### Step 1: Access Google Chat 1. Open [Google Chat](https://chat.google.com/) 2. Sign in with your Google account 3. Navigate to the **Space** where you want to receive alerts (or create a new Space) #### Step 2: Create a New Webhook 1. Click on the **Space name** at the top of the chat window 2. From the dropdown menu, select **Space Settings** 3. Click **Apps & Integrations** in the left menu. 4. Click **Add Webhooks** #### Step 3: Configure the Webhook 1. Enter a **Name** for the webhook (e.g., "OpenMetadata Alerts") 2. Select the **Avatar** or icon for the webhook (optional) 3. Click **Create** #### Step 4: Copy Your Webhook URL 1. You'll see your new webhook listed with a **Webhook URL** 2. Copy the complete **Webhook URL** (it looks like `https://chat.googleapis.com/v1/spaces/XXXXXXXXXXXXXXX/messages?key=...&token=...`) 3. Keep this URL safe - you'll need it to configure OpenMetadata ### Configuring Google Chat Webhooks in OpenMetadata Once you have your Google Chat Webhook URL, follow these steps to configure it in OpenMetadata: #### Step 1: Access Alert Configuration 1. In OpenMetadata, navigate to **Alerts & Notifications** from the main menu 2. Select the type of alert you want to configure: * [**Data Observability Alerts**](https://docs.open-metadata.org/latest/how-to-guides/data-quality-observability/alerts-notifications/data-observability-alerts) (for data quality and pipeline monitoring) * [**System & Governance Notifications**](https://docs.open-metadata.org/latest/how-to-guides/data-quality-observability/alerts-notifications/system-governance-notifications) (for metadata and governance events) #### Step 2: Add Google Chat as a Destination 1. Click **Add Destination** 2. Select **Google Chat** from the available destination options 3. Paste your Google Chat Webhook URL into the **Google Chat Webhook URL** field #### Step 3: Test the Connection 1. Click **Test Connection** to verify the webhook is working 2. A test message will be sent to your Google Chat Space 3. If successful, you'll see a confirmation message and the test message in your Space #### Step 4: Save and Enable 1. Click **Save** to store the Alert configuration 2. The Google Chat destination is now ready to receive alerts ### Best Practices * **Dedicated Spaces**: Create separate Google Chat Spaces for different types of alerts (e.g., "Data Quality Alerts", "Pipeline Failures", "Governance Events") * **Space Members**: Ensure all relevant team members are added to the Google Chat Space * **Multiple Webhooks**: You can configure multiple Google Chat webhooks to send different alert types to different Spaces * **Notifications**: Enable notifications for the Space so team members don't miss important alerts * **Monitoring**: Regularly review alerts in your Google Chat Space to ensure they're being delivered ### Reference For more information about Google Chat Webhooks, visit the [Google Chat Webhooks Documentation](https://developers.google.com/workspace/chat/quickstart/webhooks) --- # Run the KafkaConnect Connector Externally We use cookies to improve site navigation, analyze site usage, and enhance your user experience. Click "Accept" to enable cookies or "Reject" to reject cookies. AcceptReject connectors No menu items for this category [Home](https://docs.open-metadata.org/latest) /[Connectors](https://docs.open-metadata.org/latest/connectors) /[Pipeline](https://docs.open-metadata.org/latest/connectors/pipeline) /[Kafkaconnect](https://docs.open-metadata.org/latest/connectors/pipeline/kafkaconnect) /[Yaml](https://docs.open-metadata.org/latest/connectors/pipeline/kafkaconnect/yaml) OpenMetadata Documentation ![KafkaConnect](https://docs.open-metadata.org/_next/image?url=%2Fimages%2Fconnectors%2Fkafka.webp&w=64&q=75) KafkaConnect ============ PROD Available In Feature List Pipelines Pipeline Status Tags Usage Owners Lineage In this section, we provide guides and references to use the KafkaConnect connector. Configure and schedule KafkaConnect metadata and profiler workflows from the OpenMetadata UI: * [Requirements](https://docs.open-metadata.org/latest/connectors/pipeline/kafkaconnect/yaml#requirements) * [Metadata Ingestion](https://docs.open-metadata.org/latest/connectors/pipeline/kafkaconnect/yaml#metadata-ingestion) How to Run the Connector Externally ----------------------------------- To run the Ingestion via the UI you'll need to use the OpenMetadata Ingestion Container, which comes shipped with custom Airflow plugins to handle the workflow deployment. If, instead, you want to manage your workflows externally on your preferred orchestrator, you can check the following docs to run the Ingestion Framework **anywhere**. [#### External Schedulers\ \ Get more information about running the Ingestion Framework Externally](https://docs.open-metadata.org/latest/deployment/ingestion) Requirements ------------ ### Python Requirements We have support for Python versions 3.9-3.11 To run the KafkaConnect ingestion, you will need to install: Metadata Ingestion ------------------ All connectors are defined as JSON Schemas. [Here](https://github.com/open-metadata/OpenMetadata/blob/main/openmetadata-spec/src/main/resources/json/schema/entity/services/connections/pipeline/kafkaConnectConnection.json) you can find the structure to create a connection to KafkaConnect. In order to create and run a Metadata Ingestion workflow, we will follow the steps to create a YAML configuration able to connect to the source, process the Entities if needed, and reach the OpenMetadata server. The workflow is modeled around the following [JSON Schema](https://github.com/open-metadata/OpenMetadata/blob/main/openmetadata-spec/src/main/resources/json/schema/metadataIngestion/workflow.json) ### 1\. Define the YAML Config This is a sample config for KafkaConnect: #### Source Configuration - Service Connection **hostPort**: The hostname or IP address of the Kafka Connect worker with the REST API enabled **verifySSL**: Whether SSL verification should be perform when authenticating. **Kafka Connect Config**: OpenMetadata supports username/password or no Authentication. _Basic Authentication_ \- Username: Username to connect to Kafka Connect. This user should be able to send request to the Kafka Connect API and access the [Rest API](https://docs.confluent.io/platform/current/connect/references/restapi.html) GET endpoints. \- Password: Password to connect to Kafka Connect. **messagingServiceName**: Name of the Kafka Messaging Service associated with this KafkaConnect Pipeline Service. e.g. local\_kafka. #### Source Configuration - Source Config The `sourceConfig` is defined [here](https://github.com/open-metadata/OpenMetadata/blob/main/openmetadata-spec/src/main/resources/json/schema/metadataIngestion/pipelineServiceMetadataPipeline.json) : * **dbServiceNames**: Database Service Name for the creation of lineage, if the source supports it. * **includeTags**: Set the 'Include Tags' toggle to control whether to include tags as part of metadata ingestion. * **includeUnDeployedPipelines**: Set the 'Include UnDeployed Pipelines' toggle to control whether to include un-deployed pipelines as part of metadata ingestion. By default it is set to `true` * **markDeletedPipelines**: Set the Mark Deleted Pipelines toggle to flag pipelines as soft-deleted if they are not present anymore in the source system. * **pipelineFilterPattern** and **chartFilterPattern**: Note that the `pipelineFilterPattern` and `chartFilterPattern` both support regex as include or exclude. * **includeOwners**: Set the 'Include Owners' toggle to control whether to include owners to the ingested entity if the owner email matches with a user stored in the OM server as part of metadata ingestion. If the ingested entity already exists and has an owner, the owner will not be overwritten.It supports boolean values either `true` or `false`. * **overrideLineage**: Set the 'Override Lineage' toggle to control whether to override the existing lineage. It supports boolean values either `true` or `false`. * **overrideMetadata**: Set the 'Override Metadata' toggle to control whether to override the existing metadata in the OpenMetadata server with the metadata fetched from the source. If the toggle is set to true, the metadata fetched from the source will override the existing metadata in the OpenMetadata server. If the toggle is set to false, the metadata fetched from the source will not override the existing metadata in the OpenMetadata server. This is applicable for fields like description, tags, owner and displayName. It supports boolean values either `true` or `false`. #### Sink Configuration To send the metadata to OpenMetadata, it needs to be specified as `type: metadata-rest`. #### Workflow Configuration The main property here is the `openMetadataServerConfig`, where you can define the host and security provider of your OpenMetadata installation. **Logger Level** You can specify the `loggerLevel` depending on your needs. If you are trying to troubleshoot an ingestion, running with `DEBUG` will give you far more traces for identifying issues. **JWT Token** JWT tokens will allow your clients to authenticate against the OpenMetadata server. To enable JWT Tokens, you will get more details [here](https://docs.open-metadata.org/latest/deployment/security/enable-jwt-tokens) . You can refer to the JWT Troubleshooting section [link](https://docs.open-metadata.org/latest/deployment/security/jwt-troubleshooting) for any issues in your JWT configuration. **Store Service Connection** If set to `true` (default), we will store the sensitive information either encrypted via the Fernet Key in the database or externally, if you have configured any [Secrets Manager](https://docs.open-metadata.org/latest/deployment/secrets-manager) . If set to `false`, the service will be created, but the service connection information will only be used by the Ingestion Framework at runtime, and won't be sent to the OpenMetadata server. **Store Service Connection** If set to `true` (default), we will store the sensitive information either encrypted via the Fernet Key in the database or externally, if you have configured any [Secrets Manager](https://docs.open-metadata.org/latest/deployment/secrets-manager) . If set to `false`, the service will be created, but the service connection information will only be used by the Ingestion Framework at runtime, and won't be sent to the OpenMetadata server. **SSL Configuration** If you have added SSL to the [OpenMetadata server](https://docs.open-metadata.org/latest/deployment/security/enable-ssl) , then you will need to handle the certificates when running the ingestion too. You can either set `verifySSL` to `ignore`, or have it as `validate`, which will require you to set the `sslConfig.caCertificate` with a local path where your ingestion runs that points to the server certificate file. Find more information on how to troubleshoot SSL issues [here](https://docs.open-metadata.org/latest/deployment/security/enable-ssl/ssl-troubleshooting) . **ingestionPipelineFQN** Fully qualified name of ingestion pipeline, used to identify the current ingestion pipeline. filename.yamlCopy ### 2\. Run with the CLI First, we will need to save the YAML file. Afterward, and with all requirements installed, we can run: Note that from connector to connector, this recipe will always be the same. By updating the YAML configuration, you will be able to extract metadata from different sources. Debezium CDC Support -------------------- The KafkaConnect connector provides **full support for Debezium CDC connectors** with intelligent column extraction and accurate lineage tracking. ### What We Provide When you ingest Debezium connectors, OpenMetadata automatically: 1. **Detects CDC Envelope Structures** - Identifies Debezium's CDC format with `op`, `before`, and `after` fields 2. **Extracts Real Table Columns** - Parses actual database columns from the CDC payload instead of CDC envelope metadata 3. **Creates Accurate Column-Level Lineage** - Maps lineage from source database tables β†’ Kafka topics β†’ target systems ### Recognized Configuration Parameters OpenMetadata recognizes the following Debezium configuration parameters for intelligent CDC detection: * `database.server.name` - Server identifier (Debezium V1) * `topic.prefix` - Topic prefix (Debezium V2) * `table.include.list` - Tables to capture (e.g., `mydb.customers,mydb.orders`) --- # Slack Alerts Configuration | OpenMetadata We use cookies to improve site navigation, analyze site usage, and enhance your user experience. Click "Accept" to enable cookies or "Reject" to reject cookies. AcceptReject ![](https://static.scarf.sh/a.png?x-pxid=6814879a-96c0-40bd-be69-ddfeb14694aa) how-to-guides No menu items for this category [Home](https://docs.open-metadata.org/latest) /[How To Guides](https://docs.open-metadata.org/latest/how-to-guides) /[Data Quality Observability](https://docs.open-metadata.org/latest/how-to-guides/data-quality-observability) /[Alerts Notifications](https://docs.open-metadata.org/latest/how-to-guides/data-quality-observability/alerts-notifications) /[Slack Alerts Configuration](https://docs.open-metadata.org/latest/how-to-guides/data-quality-observability/alerts-notifications/slack-alerts-configuration) OpenMetadata Documentation Slack Alerts Configuration -------------------------- OpenMetadata integrates with Slack using **Incoming Webhooks** to deliver alert messages directly to your Slack channels. This allows you to receive real-time notifications without leaving Slack. ### Setting Up a Slack Webhook Follow these steps to create an Incoming Webhook in your Slack workspace: #### Step 1: Access Slack API Applications 1. Go to [Slack API Applications](https://api.slack.com/apps) 2. Sign in with your Slack workspace credentials 3. Click **Create New App** or **Create an App** #### Step 2: Create a New App 1. Select **From scratch** 2. Enter an **App Name** (e.g., "OpenMetadata Alerts") 3. Select your **Slack Workspace** 4. Click **Create App** #### Step 3: Enable and Configure Incoming Webhooks 1. In the left sidebar, navigate to **Incoming Webhooks** 2. Toggle **Activate Incoming Webhooks** to **On** 3. Click **Add New Webhook to Workspace** 4. Select the **Slack channel** where you want alerts to be posted 5. Click **Allow** to authorize the app #### Step 4: Copy Your Webhook URL 1. You'll now see your new webhook listed under **Webhook URLs for Your Workspace** 2. Copy the complete **Webhook URL** 3. Keep this URL safe - you'll need it to configure OpenMetadata ### Configuring Slack Webhooks in OpenMetadata Once you have your Slack Webhook URL, follow these steps to configure it in OpenMetadata: #### Step 1: Access Alert Configuration 1. In OpenMetadata, navigate to **Alerts & Notifications** from the main menu 2. Select the type of alert you want to configure: * [**Data Observability Alerts**](https://docs.open-metadata.org/latest/how-to-guides/data-quality-observability/alerts-notifications/data-observability-alerts) (for data quality and pipeline monitoring) * [**System & Governance Notifications**](https://docs.open-metadata.org/latest/how-to-guides/data-quality-observability/alerts-notifications/system-governance-notifications) (for metadata and governance events) #### Step 2: Add Slack as a Destination 1. Click **Add Destination** 2. Select **Slack** from the available destination options 3. Paste your Slack Webhook URL into the **Slack Webhook URL** field 4. Optionally, enter a **Display Name** for this Slack destination (e.g., "Data Quality Alerts") #### Step 3: Test the Connection 1. Click **Test Connection** to verify the webhook is working 2. A test message will be sent to your Slack channel 3. If successful, you'll see a confirmation message #### Step 4: Save and Enable 1. Click **Save** to store the configuration 2. The Slack destination is now ready to receive alerts ### Best Practices * **Dedicated Channels**: Create separate Slack channels for different types of alerts (e.g., #data-quality-alerts, #pipeline-failures) * **Channel Permissions**: Ensure the Slack channel is accessible to all team members who need to receive alerts * **Multiple Webhooks**: You can configure multiple Slack webhooks to send different alert types to different channels * **Monitoring**: Regularly monitor your Slack channels to ensure alerts are being delivered ### Reference For more information about Slack Incoming Webhooks, visit the [Slack API Documentation](https://api.slack.com/messaging/webhooks) --- # Generic Webhook Alert Configuration | OpenMetadata We use cookies to improve site navigation, analyze site usage, and enhance your user experience. Click "Accept" to enable cookies or "Reject" to reject cookies. AcceptReject ![](https://static.scarf.sh/a.png?x-pxid=6814879a-96c0-40bd-be69-ddfeb14694aa) how-to-guides No menu items for this category [Home](https://docs.open-metadata.org/latest) /[How To Guides](https://docs.open-metadata.org/latest/how-to-guides) /[Data Quality Observability](https://docs.open-metadata.org/latest/how-to-guides/data-quality-observability) /[Alerts Notifications](https://docs.open-metadata.org/latest/how-to-guides/data-quality-observability/alerts-notifications) /[Generic Webhook Alerts Configuration](https://docs.open-metadata.org/latest/how-to-guides/data-quality-observability/alerts-notifications/generic-webhook-alerts-configuration) OpenMetadata Documentation Generic Webhook Alerts Configuration ------------------------------------ The **Generic Webhook** destination provides maximum flexibility for integrating OpenMetadata alerts with virtually any external system. It allows you to send alert notifications to custom applications, automation platforms, and internal services that can receive HTTP POST requests. ### Use Cases Generic webhooks are ideal for: * **Custom Applications**: Internal tools and applications that need to receive alerts * **Automation Platforms**: Services like Zapier, Make (formerly Integromat), and IFTTT * **Monitoring Systems**: Integration with existing monitoring infrastructure * **Internal Services**: Microservices, APIs, and serverless functions * **Workflow Automation**: Triggering automated workflows and processes ### Preparing Your Webhook Endpoint Before configuring the webhook in OpenMetadata, you need a webhook endpoint URL from your external system. This could be: * An automation platform like Zapier or Make * A custom application or API endpoint * An internal service that accepts HTTP POST requests * A serverless function (AWS Lambda, Google Cloud Functions, etc.) Your endpoint must: * Accept HTTP POST requests * Be accessible from your OpenMetadata instance ### Configuring Generic Webhooks in OpenMetadata Once you have your webhook endpoint URL, follow these steps to configure it in OpenMetadata: #### Step 1: Access Alert Configuration 1. In OpenMetadata, navigate to **Alerts & Notifications** from the main menu 2. Select the type of alert you want to configure: * [**Data Observability Alerts**](https://docs.open-metadata.org/latest/how-to-guides/data-quality-observability/alerts-notifications/data-observability-alerts) (for data quality and pipeline monitoring) * [**System & Governance Notifications**](https://docs.open-metadata.org/latest/how-to-guides/data-quality-observability/alerts-notifications/system-governance-notifications) (for metadata and governance events) #### Step 2: Add Generic Webhook as a Destination 1. Click **Add Destination** 2. Select **Generic Webhook** from the available destination options 3. Paste your **Endpoint URL** into the **Endpoint URL** field (must start with `https://`) #### Step 3: Test the Connection 1. Click **Test Connection** to verify the webhook is working 2. A test payload will be sent to your endpoint 3. If successful, you'll see a confirmation message 4. Check your external system to verify the test message was received #### Step 4: Save and Enable 1. Click **Save** to store the configuration 2. The webhook destination is now ready to receive alerts ### Best Practices * **Use HTTPS**: Always use HTTPS endpoints for security * **Error Handling**: Ensure your endpoint handles errors gracefully and logs failures * **Response Time**: Keep endpoint response times under 30 seconds * **Multiple Webhooks**: You can configure multiple webhook endpoints for different alert types * **Testing**: Always test your webhook connection before enabling in production * **Validation**: Validate incoming data in your endpoint before processing --- # OpenMetadata Documentation: Get Help Instantly We use cookies to improve site navigation, analyze site usage, and enhance your user experience. Click "Accept" to enable cookies or "Reject" to reject cookies. AcceptReject [Home](https://docs.open-metadata.org/latest) [Quickstart](https://docs.open-metadata.org/latest/quick-start) [Deployment](https://docs.open-metadata.org/latest/deployment) [Connectors](https://docs.open-metadata.org/latest/connectors) [How-to Guides](https://docs.open-metadata.org/latest/how-to-guides) [Releases](https://docs.open-metadata.org/latest/releases) [Main Concepts](https://docs.open-metadata.org/latest/main-concepts) [Developers](https://docs.open-metadata.org/latest/developers) [SDK](https://docs.open-metadata.org/latest/sdk) How-to Guides [Data Quality and Observability](https://docs.open-metadata.org/latest/how-to-guides/data-quality-observability) [Data Quality](https://docs.open-metadata.org/latest/how-to-guides/data-quality-observability/quality) [Data Quality Tab](https://docs.open-metadata.org/latest/how-to-guides/data-quality-observability/quality/tab) [How to Write and Deploy No-Code Test Cases](https://docs.open-metadata.org/latest/how-to-guides/data-quality-observability/quality/test) [Configure Data Quality](https://docs.open-metadata.org/latest/how-to-guides/data-quality-observability/quality/configure) [Adding Test Cases](https://docs.open-metadata.org/latest/how-to-guides/data-quality-observability/quality/adding-test-cases) [Adding Test Suites](https://docs.open-metadata.org/latest/how-to-guides/data-quality-observability/quality/adding-test-suites) [Test Cases From YAML Config](https://docs.open-metadata.org/latest/how-to-guides/data-quality-observability/quality/test-cases-from-yaml-config) [How to Visualize Test Results](https://docs.open-metadata.org/latest/how-to-guides/data-quality-observability/quality/visualize) [Tests - YAML Config](https://docs.open-metadata.org/latest/how-to-guides/data-quality-observability/quality/tests-yaml) [Tests - UI Config](https://docs.open-metadata.org/latest/how-to-guides/data-quality-observability/quality/tests-ui) [Dimensional Validation](https://docs.open-metadata.org/latest/how-to-guides/data-quality-observability/quality/dimensional-validation) [Data Quality as Code](https://docs.open-metadata.org/latest/how-to-guides/data-quality-observability/quality/data-quality-as-code) [Custom Tests](https://docs.open-metadata.org/latest/how-to-guides/data-quality-observability/quality/custom-tests) [Data Profiler](https://docs.open-metadata.org/latest/how-to-guides/data-quality-observability/profiler) [Alerts & Notifications](https://docs.open-metadata.org/latest/how-to-guides/data-quality-observability/alerts-notifications) [Incident Manager](https://docs.open-metadata.org/latest/how-to-guides/data-quality-observability/incident-manager) Page not found :( ----------------- [#### Quickstart\ \ Deploy OpenMetadata and connect to your sources in minutes!](https://docs.open-metadata.org/latest/quick-start) [#### SaaS\ \ Enjoy 100% of OpenMetadata with 0% of the hassle.](https://www.getcollate.io/) [#### Knowledge Base\ \ Check out some frequent questions and answers](https://github.com/open-metadata/OpenMetadata/discussions/categories/q-a) [#### Deployment\ \ Deploy in Bare Metal, Docker or Kubernetes.](https://docs.open-metadata.org/latest/deployment) [#### connectors\ \ Connect to database, dashboard, messaging, pipeline and ML services.](https://docs.open-metadata.org/latest/connectors) --- # OpenMetadata Documentation: Get Help Instantly We use cookies to improve site navigation, analyze site usage, and enhance your user experience. Click "Accept" to enable cookies or "Reject" to reject cookies. AcceptReject [Home](https://docs.open-metadata.org/latest) [Quickstart](https://docs.open-metadata.org/latest/quick-start) [Deployment](https://docs.open-metadata.org/latest/deployment) [Connectors](https://docs.open-metadata.org/latest/connectors) [How-to Guides](https://docs.open-metadata.org/latest/how-to-guides) [Releases](https://docs.open-metadata.org/latest/releases) [Main Concepts](https://docs.open-metadata.org/latest/main-concepts) [Developers](https://docs.open-metadata.org/latest/developers) [SDK](https://docs.open-metadata.org/latest/sdk) Deployment [Minimum Requirements](https://docs.open-metadata.org/latest/deployment/minimum-requirements) [Bare Metal Deployment](https://docs.open-metadata.org/latest/deployment/bare-metal) [Docker Deployment](https://docs.open-metadata.org/latest/deployment/docker) [Enable Security](https://docs.open-metadata.org/latest/deployment/docker/security) [Enable Subpath](https://docs.open-metadata.org/latest/deployment/docker/subpath) [Kubernetes Deployment](https://docs.open-metadata.org/latest/deployment/kubernetes) [Ingestion](https://docs.open-metadata.org/latest/deployment/ingestion) [Enable Security](https://docs.open-metadata.org/latest/deployment/security) [Basic Authentication](https://docs.open-metadata.org/latest/deployment/security/basic-auth) [Ldap Authentication](https://docs.open-metadata.org/latest/deployment/security/ldap) [Auth0 SSO](https://docs.open-metadata.org/latest/deployment/security/auth0) [Public](https://docs.open-metadata.org/latest/deployment/security/auth0/public-client) [Confidential](https://docs.open-metadata.org/latest/deployment/security/auth0/confidential-client) [Azure SSO](https://docs.open-metadata.org/latest/deployment/security/azure) [Custom OIDC SSO](https://docs.open-metadata.org/latest/deployment/security/custom-oidc) [SSO Configuration](https://docs.open-metadata.org/latest/deployment/security/configuration) [Docker](https://docs.open-metadata.org/latest/deployment/security/configuration/docker) [Bare Metal](https://docs.open-metadata.org/latest/deployment/security/configuration/bare-metal) [Kubernetes](https://docs.open-metadata.org/latest/deployment/security/configuration/kubernetes) [OIDC SSO](https://docs.open-metadata.org/latest/deployment/security/oidc) [Google SSO](https://docs.open-metadata.org/latest/deployment/security/google) [Okta SSO](https://docs.open-metadata.org/latest/deployment/security/okta) [Amazon Cognito SSO](https://docs.open-metadata.org/latest/deployment/security/amazon-cognito) [One Login SSO](https://docs.open-metadata.org/latest/deployment/security/one-login) [Keycloak SSO](https://docs.open-metadata.org/latest/deployment/security/keycloak) [Saml](https://docs.open-metadata.org/latest/deployment/security/saml) [Enable SSL](https://docs.open-metadata.org/latest/deployment/security/enable-ssl) [Enable JWT Tokens](https://docs.open-metadata.org/latest/deployment/security/enable-jwt-tokens) [Configuration Reference Parameters](https://docs.open-metadata.org/latest/deployment/security/configuration-parameters) [JWT Troubleshooting](https://docs.open-metadata.org/latest/deployment/security/jwt-troubleshooting) [Enable Secrets Manager](https://docs.open-metadata.org/latest/deployment/secrets-manager) [How to enable AWS RDS IAM Auth](https://docs.open-metadata.org/latest/deployment/rds-iam-auth) [How to enable Azure Auth](https://docs.open-metadata.org/latest/deployment/azure-auth) [Azure - Enable Passwordless Database Backend Connection](https://docs.open-metadata.org/latest/deployment/azure-passwordless-auth) [Production-Ready Requirements](https://docs.open-metadata.org/latest/deployment/requirements) [Server Configuration Reference](https://docs.open-metadata.org/latest/deployment/configuration) [Database Connection Pooling](https://docs.open-metadata.org/latest/deployment/database-connection-pooling) [Upgrade OpenMetadata](https://docs.open-metadata.org/latest/deployment/upgrade) [Backup & Restore Metadata](https://docs.open-metadata.org/latest/deployment/backup-restore-metadata) [Metrics](https://docs.open-metadata.org/latest/deployment/metrics) [OSS Security](https://docs.open-metadata.org/latest/deployment/oss-security) Page not found :( ----------------- [#### Quickstart\ \ Deploy OpenMetadata and connect to your sources in minutes!](https://docs.open-metadata.org/latest/quick-start) [#### SaaS\ \ Enjoy 100% of OpenMetadata with 0% of the hassle.](https://www.getcollate.io/) [#### Knowledge Base\ \ Check out some frequent questions and answers](https://github.com/open-metadata/OpenMetadata/discussions/categories/q-a) [#### Deployment\ \ Deploy in Bare Metal, Docker or Kubernetes.](https://docs.open-metadata.org/latest/deployment) [#### connectors\ \ Connect to database, dashboard, messaging, pipeline and ML services.](https://docs.open-metadata.org/latest/connectors) --- # Configuring OpenMetadata to Run Under a Subpath We use cookies to improve site navigation, analyze site usage, and enhance your user experience. Click "Accept" to enable cookies or "Reject" to reject cookies. AcceptReject / / --- # Minimum Requirements | Official Documentation We use cookies to improve site navigation, analyze site usage, and enhance your user experience. Click "Accept" to enable cookies or "Reject" to reject cookies. AcceptReject / / --- # Database Connection Pooling We use cookies to improve site navigation, analyze site usage, and enhance your user experience. Click "Accept" to enable cookies or "Reject" to reject cookies. AcceptReject / / --- # Production-Ready Requirements for OpenMetadata Deployment We use cookies to improve site navigation, analyze site usage, and enhance your user experience. Click "Accept" to enable cookies or "Reject" to reject cookies. AcceptReject / / --- # How to enable Azure Auth We use cookies to improve site navigation, analyze site usage, and enhance your user experience. Click "Accept" to enable cookies or "Reject" to reject cookies. AcceptReject / / --- # Azure - Enable Passwordless Database Backend Connection We use cookies to improve site navigation, analyze site usage, and enhance your user experience. Click "Accept" to enable cookies or "Reject" to reject cookies. AcceptReject / / --- # How to Visualize Test Results We use cookies to improve site navigation, analyze site usage, and enhance your user experience. Click "Accept" to enable cookies or "Reject" to reject cookies. AcceptReject ![](https://static.scarf.sh/a.png?x-pxid=6814879a-96c0-40bd-be69-ddfeb14694aa) / / --- # Metrics We use cookies to improve site navigation, analyze site usage, and enhance your user experience. Click "Accept" to enable cookies or "Reject" to reject cookies. AcceptReject / / --- # How to enable AWS RDS IAM Auth | Official Documentation We use cookies to improve site navigation, analyze site usage, and enhance your user experience. Click "Accept" to enable cookies or "Reject" to reject cookies. AcceptReject / / --- # OSS Security Best Practices We use cookies to improve site navigation, analyze site usage, and enhance your user experience. Click "Accept" to enable cookies or "Reject" to reject cookies. AcceptReject deployment No menu items for this category [Home](https://docs.open-metadata.org/latest) /[Deployment](https://docs.open-metadata.org/latest/deployment) /[Oss Security](https://docs.open-metadata.org/latest/deployment/oss-security) OpenMetadata Documentation OSS Security ============ Encryption of Connection Credentials ------------------------------------ OpenMetadata ensures that sensitive information, such as passwords and connection secrets, is securely stored. * **Encryption Algorithm**: OpenMetadata uses **Fernet encryption** to encrypt secrets and passwords before storing them in the database. * **Fernet Encryption Details**: * Uses **AES-128 in CBC mode** with a strong key-based approach. * **Not based on hashing or salting**, but rather an encryption/decryption method with a symmetric key. * **Secrets Manager Support**: * Users can **avoid storing credentials** in OpenMetadata by configuring an external **Secrets Manager**. * More details on setting up a Secrets Manager can be found here: πŸ”— [Secrets Manager Documentation](https://docs.open-metadata.org/latest/deployment/secrets-manager) Secure Connections to Data Sources ---------------------------------- OpenMetadata supports **encrypted connections** to various databases and services. * **SSL/TLS Support**: * OpenMetadata allows users to configure **SSL/TLS encryption** for secure data transmission. * Users can specify **SSL modes** and provide **CA certificates** for SSL validation. * **How to Enable SSL?** * Each connector supports different SSL configurations. * Follow the detailed guide for enabling SSL in OpenMetadata: πŸ”— [Enable SSL in OpenMetadata](https://docs.open-metadata.org/latest/deployment/security/enable-ssl) **Additional Security Measures** -------------------------------- * **Role-Based Access Control (RBAC)**: OpenMetadata allows administrators to define user roles and permissions. * **Authentication & Authorization**: OpenMetadata supports integration with OAuth, SAML, and LDAP for secure authentication. * **Data Access Control**: Users can restrict access to metadata based on policies and governance rules. * **Passwords and secrets are securely encrypted** using **Fernet encryption**. * **Connections to data sources can be encrypted** using **SSL/TLS**. * **Secrets Managers** can be used to manage credentials externally. --- # Adding Data Quality Test Cases from yaml config We use cookies to improve site navigation, analyze site usage, and enhance your user experience. Click "Accept" to enable cookies or "Reject" to reject cookies. AcceptReject ![](https://static.scarf.sh/a.png?x-pxid=6814879a-96c0-40bd-be69-ddfeb14694aa) [Home](https://docs.open-metadata.org/latest) [Quickstart](https://docs.open-metadata.org/latest/quick-start) [Deployment](https://docs.open-metadata.org/latest/deployment) [Connectors](https://docs.open-metadata.org/latest/connectors) [How-to Guides](https://docs.open-metadata.org/latest/how-to-guides) [Releases](https://docs.open-metadata.org/latest/releases) [Main Concepts](https://docs.open-metadata.org/latest/main-concepts) [Developers](https://docs.open-metadata.org/latest/developers) [SDK](https://docs.open-metadata.org/latest/sdk) How-to Guides [Data Quality and Observability](https://docs.open-metadata.org/latest/how-to-guides/data-quality-observability) [Data Quality](https://docs.open-metadata.org/latest/how-to-guides/data-quality-observability/quality) [Data Profiler](https://docs.open-metadata.org/latest/how-to-guides/data-quality-observability/profiler) [Alerts & Notifications](https://docs.open-metadata.org/latest/how-to-guides/data-quality-observability/alerts-notifications) [Incident Manager](https://docs.open-metadata.org/latest/how-to-guides/data-quality-observability/incident-manager) [Home](https://docs.open-metadata.org/latest) /[How To Guides](https://docs.open-metadata.org/latest/how-to-guides) /[Data Quality Observability](https://docs.open-metadata.org/latest/how-to-guides/data-quality-observability) /[Quality](https://docs.open-metadata.org/latest/how-to-guides/data-quality-observability/quality) /[Test Cases From Yaml Config](https://docs.open-metadata.org/latest/how-to-guides/data-quality-observability/quality/test-cases-from-yaml-config) OpenMetadata Documentation Data Quality ------------ ### Adding Data Quality Test Cases from yaml config When creating a JSON config for a test workflow the source configuration is very simple. source: type: TestSuite serviceName: sourceConfig: config: type: TestSuite entityFullyQualifiedName: The only sections you need to modify here are the `serviceName` (this name needs to be unique) and `entityFullyQualifiedName` (the entity for which we'll be executing tests against) keys. Once you have defined your source configuration you'll need to define te processor configuration. processor: type: "orm-test-runner" config: forceUpdate: testCases: - name: testDefinitionName: columnValueLengthsToBeBetween columnName: parameterValues: - name: minLength value: 10 - name: maxLength value: 25 - name: testDefinitionName: tableRowCountToEqual parameterValues: - name: value value: 10 The processor type should be set to `"orm-test-runner"`. For accepted test definition names and parameter value names refer to the [tests page](https://docs.open-metadata.org/latest/how-to-guides/data-quality-observability/quality/tests-yaml) . Note that while you can define tests directly in this YAML configuration, running the workflow will execute ALL THE TESTS present in the table, regardless of what you are defining in the YAML. This makes it easy for any user to contribute tests via the UI, while maintaining the test execution external. You can keep your YAML config as simple as follows if the table already has tests. processor: type: "orm-test-runner" config: {} ### Key reference: * `forceUpdate`: if the test case exists (base on the test case name) for the entity, implements the strategy to follow when running the test (i.e. whether or not to update parameters) * `testCases`: list of test cases to add to the entity referenced. Note that we will execute all the tests present in the Table. * `name`: test case name * `testDefinitionName`: test definition * `columnName`: only applies to column test. The name of the column to run the test against * `parameterValues`: parameter values of the test The `sink` and `workflowConfig` will have the same settings as the ingestion and profiler workflow. ### Full `yaml` config example source: type: TestSuite serviceName: MyAwesomeTestSuite sourceConfig: config: type: TestSuite entityFullyQualifiedName: MySQL.default.openmetadata_db.tag_usage # testCases: ["run_only_this_test_case"] # Optional, if not provided all tests will be executed processor: type: "orm-test-runner" config: forceUpdate: false testCases: - name: column_value_length_tagFQN testDefinitionName: columnValueLengthsToBeBetween columnName: tagFQN parameterValues: - name: minLength value: 10 - name: maxLength value: 25 - name: table_row_count_test testDefinitionName: tableRowCountToEqual parameterValues: - name: value value: 10 sink: type: metadata-rest config: {} workflowConfig: openMetadataServerConfig: hostPort: authProvider: ### How to Run Tests To run the tests from the CLI execute the following command metadata test -c /path/to/my/config.yaml ---