# Table of Contents - [Temporal Docs | Temporal Platform Documentation](#temporal-docs-temporal-platform-documentation) - [Unknown](#unknown) - [AI Cookbook | Temporal Platform Documentation](#ai-cookbook-temporal-platform-documentation) - [Search | Temporal Platform Documentation](#search-temporal-platform-documentation) - [Tags | Temporal Platform Documentation](#tags-temporal-platform-documentation) - [One doc tagged with "claim-check" | Temporal Platform Documentation](#one-doc-tagged-with-claim-check-temporal-platform-documentation) - [6 docs tagged with "agents" | Temporal Platform Documentation](#6-docs-tagged-with-agents-temporal-platform-documentation) - [One doc tagged with "claude" | Temporal Platform Documentation](#one-doc-tagged-with-claude-temporal-platform-documentation) - [One doc tagged with "litellm" | Temporal Platform Documentation](#one-doc-tagged-with-litellm-temporal-platform-documentation) - [One doc tagged with "mcp" | Temporal Platform Documentation](#one-doc-tagged-with-mcp-temporal-platform-documentation) - [4 docs tagged with "foundations" | Temporal Platform Documentation](#4-docs-tagged-with-foundations-temporal-platform-documentation) - [4 docs tagged with "openai" | Temporal Platform Documentation](#4-docs-tagged-with-openai-temporal-platform-documentation) - [One doc tagged with "s3" | Temporal Platform Documentation](#one-doc-tagged-with-s3-temporal-platform-documentation) - [One doc tagged with "toolcalling" | Temporal Platform Documentation](#one-doc-tagged-with-toolcalling-temporal-platform-documentation) - [One doc tagged with "provider-neutral" | Temporal Platform Documentation](#one-doc-tagged-with-provider-neutral-temporal-platform-documentation) - [One doc tagged with "workflows" | Temporal Platform Documentation](#one-doc-tagged-with-workflows-temporal-platform-documentation) - [Deep Research](#deep-research) - [Basic Agentic Loop with Claude and Tool Calling](#basic-agentic-loop-with-claude-and-tool-calling) - [Basic Agentic Loop with Tool Calling](#basic-agentic-loop-with-tool-calling) - [Hello World](#hello-world) - [Hello World with LiteLLM](#hello-world-with-litellm) - [Human-in-the-Loop AI Agent](#human-in-the-loop-ai-agent) - [Durable Agent with Tools - OpenAI Agents SDK](#durable-agent-with-tools-openai-agents-sdk) - [Durable MCP Weather Server](#durable-mcp-weather-server) - [Structured Outputs with Temporal and OpenAI](#structured-outputs-with-temporal-and-openai) - [One doc tagged with "Actions" | Temporal Platform Documentation](#one-doc-tagged-with-actions-temporal-platform-documentation) - [One doc tagged with "Accounts" | Temporal Platform Documentation](#one-doc-tagged-with-accounts-temporal-platform-documentation) - [Tool calling agent](#tool-calling-agent) - [One doc tagged with "AI Agents" | Temporal Platform Documentation](#one-doc-tagged-with-ai-agents-temporal-platform-documentation) - [One doc tagged with "AI SDK" | Temporal Platform Documentation](#one-doc-tagged-with-ai-sdk-temporal-platform-documentation) - [Retry Policy from HTTP Responses](#retry-policy-from-http-responses) - [12 docs tagged with "python" | Temporal Platform Documentation](#12-docs-tagged-with-python-temporal-platform-documentation) - [One doc tagged with "Braintrust" | Temporal Platform Documentation](#one-doc-tagged-with-braintrust-temporal-platform-documentation) - [One doc tagged with "Authentication" | Temporal Platform Documentation](#one-doc-tagged-with-authentication-temporal-platform-documentation) - [Claim Check Pattern with Temporal](#claim-check-pattern-with-temporal) - [2 docs tagged with "Billing" | Temporal Platform Documentation](#2-docs-tagged-with-billing-temporal-platform-documentation) - [3 docs tagged with "Agent Frameworks" | Temporal Platform Documentation](#3-docs-tagged-with-agent-frameworks-temporal-platform-documentation) - [3 docs tagged with "AI Frameworks" | Temporal Platform Documentation](#3-docs-tagged-with-ai-frameworks-temporal-platform-documentation) - [One doc tagged with "Context Propagation" | Temporal Platform Documentation](#one-doc-tagged-with-context-propagation-temporal-platform-documentation) - [3 docs tagged with "AWS" | Temporal Platform Documentation](#3-docs-tagged-with-aws-temporal-platform-documentation) - [2 docs tagged with "Capacity Modes" | Temporal Platform Documentation](#2-docs-tagged-with-capacity-modes-temporal-platform-documentation) - [One doc tagged with "Deployment" | Temporal Platform Documentation](#one-doc-tagged-with-deployment-temporal-platform-documentation) - [One doc tagged with "Deploy" | Temporal Platform Documentation](#one-doc-tagged-with-deploy-temporal-platform-documentation) - [4 docs tagged with "API Keys" | Temporal Platform Documentation](#4-docs-tagged-with-api-keys-temporal-platform-documentation) - [2 docs tagged with "Configuration" | Temporal Platform Documentation](#2-docs-tagged-with-configuration-temporal-platform-documentation) - [One doc tagged with "design-patterns" | Temporal Platform Documentation](#one-doc-tagged-with-design-patterns-temporal-platform-documentation) - [One doc tagged with "enable-nexus" | Temporal Platform Documentation](#one-doc-tagged-with-enable-nexus-temporal-platform-documentation) - [One doc tagged with "Error handling" | Temporal Platform Documentation](#one-doc-tagged-with-error-handling-temporal-platform-documentation) - [One doc tagged with "Development Server" | Temporal Platform Documentation](#one-doc-tagged-with-development-server-temporal-platform-documentation) - [One doc tagged with "Features" | Temporal Platform Documentation](#one-doc-tagged-with-features-temporal-platform-documentation) - [One doc tagged with "evaluate temporal" | Temporal Platform Documentation](#one-doc-tagged-with-evaluate-temporal-temporal-platform-documentation) - [2 docs tagged with "developer guide" | Temporal Platform Documentation](#2-docs-tagged-with-developer-guide-temporal-platform-documentation) - [2 docs tagged with "Environment Variables" | Temporal Platform Documentation](#2-docs-tagged-with-environment-variables-temporal-platform-documentation) - [One doc tagged with "Logging" | Temporal Platform Documentation](#one-doc-tagged-with-logging-temporal-platform-documentation) - [One doc tagged with "Kubernetes" | Temporal Platform Documentation](#one-doc-tagged-with-kubernetes-temporal-platform-documentation) - [6 docs tagged with "API" | Temporal Platform Documentation](#6-docs-tagged-with-api-temporal-platform-documentation) - [One doc tagged with "Migration" | Temporal Platform Documentation](#one-doc-tagged-with-migration-temporal-platform-documentation) - [3 docs tagged with "Export Workflow History" | Temporal Platform Documentation](#3-docs-tagged-with-export-workflow-history-temporal-platform-documentation) - [2 docs tagged with "Interceptors" | Temporal Platform Documentation](#2-docs-tagged-with-interceptors-temporal-platform-documentation) - [2 docs tagged with "Limits" | Temporal Platform Documentation](#2-docs-tagged-with-limits-temporal-platform-documentation) - [One doc tagged with "Parent Close Policy" | Temporal Platform Documentation](#one-doc-tagged-with-parent-close-policy-temporal-platform-documentation) - [One doc tagged with "Notification" | Temporal Platform Documentation](#one-doc-tagged-with-notification-temporal-platform-documentation) - [One doc tagged with "operations" | Temporal Platform Documentation](#one-doc-tagged-with-operations-temporal-platform-documentation) - [4 docs tagged with "Deprecated" | Temporal Platform Documentation](#4-docs-tagged-with-deprecated-temporal-platform-documentation) - [5 docs tagged with "Connectivity" | Temporal Platform Documentation](#5-docs-tagged-with-connectivity-temporal-platform-documentation) - [3 docs tagged with "GCP" | Temporal Platform Documentation](#3-docs-tagged-with-gcp-temporal-platform-documentation) - [3 docs tagged with "Integrations" | Temporal Platform Documentation](#3-docs-tagged-with-integrations-temporal-platform-documentation) - [2 docs tagged with "On-Demand Capacity" | Temporal Platform Documentation](#2-docs-tagged-with-on-demand-capacity-temporal-platform-documentation) - [3 docs tagged with "Logs" | Temporal Platform Documentation](#3-docs-tagged-with-logs-temporal-platform-documentation) - [One doc tagged with "Priority and Fairness" | Temporal Platform Documentation](#one-doc-tagged-with-priority-and-fairness-temporal-platform-documentation) - [8 docs tagged with "Activity" | Temporal Platform Documentation](#8-docs-tagged-with-activity-temporal-platform-documentation) - [2 docs tagged with "Performance" | Temporal Platform Documentation](#2-docs-tagged-with-performance-temporal-platform-documentation) - [One doc tagged with "Recovery Point Objective" | Temporal Platform Documentation](#one-doc-tagged-with-recovery-point-objective-temporal-platform-documentation) - [One doc tagged with "rate limiting" | Temporal Platform Documentation](#one-doc-tagged-with-rate-limiting-temporal-platform-documentation) - [One doc tagged with "Recovery Time Objective" | Temporal Platform Documentation](#one-doc-tagged-with-recovery-time-objective-temporal-platform-documentation) - [4 docs tagged with "High Availability" | Temporal Platform Documentation](#4-docs-tagged-with-high-availability-temporal-platform-documentation) - [One doc tagged with "Replay" | Temporal Platform Documentation](#one-doc-tagged-with-replay-temporal-platform-documentation) - [3 docs tagged with "Multitenancy" | Temporal Platform Documentation](#3-docs-tagged-with-multitenancy-temporal-platform-documentation) - [2 docs tagged with "Provisioned Capacity" | Temporal Platform Documentation](#2-docs-tagged-with-provisioned-capacity-temporal-platform-documentation) - [One doc tagged with "Spring Boot" | Temporal Platform Documentation](#one-doc-tagged-with-spring-boot-temporal-platform-documentation) - [8 docs tagged with "Codec Server" | Temporal Platform Documentation](#8-docs-tagged-with-codec-server-temporal-platform-documentation) - [7 docs tagged with "continue-as-new" | Temporal Platform Documentation](#7-docs-tagged-with-continue-as-new-temporal-platform-documentation) - [6 docs tagged with "Event History" | Temporal Platform Documentation](#6-docs-tagged-with-event-history-temporal-platform-documentation) - [4 docs tagged with "OpenMetrics" | Temporal Platform Documentation](#4-docs-tagged-with-openmetrics-temporal-platform-documentation) - [3 docs tagged with "Pricing" | Temporal Platform Documentation](#3-docs-tagged-with-pricing-temporal-platform-documentation) - [One doc tagged with "Tasks" | Temporal Platform Documentation](#one-doc-tagged-with-tasks-temporal-platform-documentation) - [One doc tagged with "Standalone Activities" | Temporal Platform Documentation](#one-doc-tagged-with-standalone-activities-temporal-platform-documentation) - [8 docs tagged with "Debugging" | Temporal Platform Documentation](#8-docs-tagged-with-debugging-temporal-platform-documentation) - [3 docs tagged with "Side-effects" | Temporal Platform Documentation](#3-docs-tagged-with-side-effects-temporal-platform-documentation) - [10 docs tagged with "Child Workflows" | Temporal Platform Documentation](#10-docs-tagged-with-child-workflows-temporal-platform-documentation) - [10 docs tagged with "Certificates" | Temporal Platform Documentation](#10-docs-tagged-with-certificates-temporal-platform-documentation) - [8 docs tagged with "Durable Timers" | Temporal Platform Documentation](#8-docs-tagged-with-durable-timers-temporal-platform-documentation) - [4 docs tagged with "Support" | Temporal Platform Documentation](#4-docs-tagged-with-support-temporal-platform-documentation) - [4 docs tagged with "Task Queues" | Temporal Platform Documentation](#4-docs-tagged-with-task-queues-temporal-platform-documentation) - [11 docs tagged with "Best Practices" | Temporal Platform Documentation](#11-docs-tagged-with-best-practices-temporal-platform-documentation) - [8 docs tagged with "Messages" | Temporal Platform Documentation](#8-docs-tagged-with-messages-temporal-platform-documentation) - [2 docs tagged with "Terraform" | Temporal Platform Documentation](#2-docs-tagged-with-terraform-temporal-platform-documentation) - [7 docs tagged with "Patching" | Temporal Platform Documentation](#7-docs-tagged-with-patching-temporal-platform-documentation) - [3 docs tagged with "Temporal Web UI" | Temporal Platform Documentation](#3-docs-tagged-with-temporal-web-ui-temporal-platform-documentation) - [One doc tagged with "use-cases" | Temporal Platform Documentation](#one-doc-tagged-with-use-cases-temporal-platform-documentation) - [2 docs tagged with "TOML" | Temporal Platform Documentation](#2-docs-tagged-with-toml-temporal-platform-documentation) - [2 docs tagged with "TRUs" | Temporal Platform Documentation](#2-docs-tagged-with-trus-temporal-platform-documentation) - [5 docs tagged with "Temporal" | Temporal Platform Documentation](#5-docs-tagged-with-temporal-temporal-platform-documentation) - [12 docs tagged with "Durable Execution" | Temporal Platform Documentation](#12-docs-tagged-with-durable-execution-temporal-platform-documentation) - [5 docs tagged with "Timeouts" | Temporal Platform Documentation](#5-docs-tagged-with-timeouts-temporal-platform-documentation) - [12 docs tagged with "getting started" | Temporal Platform Documentation](#12-docs-tagged-with-getting-started-temporal-platform-documentation) - [9 docs tagged with "Queries" | Temporal Platform Documentation](#9-docs-tagged-with-queries-temporal-platform-documentation) - [9 docs tagged with "Search Attributes" | Temporal Platform Documentation](#9-docs-tagged-with-search-attributes-temporal-platform-documentation) - [9 docs tagged with "Schedules" | Temporal Platform Documentation](#9-docs-tagged-with-schedules-temporal-platform-documentation) - [12 docs tagged with "Namespaces" | Temporal Platform Documentation](#12-docs-tagged-with-namespaces-temporal-platform-documentation) - [9 docs tagged with "Signals" | Temporal Platform Documentation](#9-docs-tagged-with-signals-temporal-platform-documentation) - [10 docs tagged with "Production" | Temporal Platform Documentation](#10-docs-tagged-with-production-temporal-platform-documentation) - [2 docs tagged with "Workflow" | Temporal Platform Documentation](#2-docs-tagged-with-workflow-temporal-platform-documentation) - [15 docs tagged with "Encryption" | Temporal Platform Documentation](#15-docs-tagged-with-encryption-temporal-platform-documentation) - [4 docs tagged with "User groups" | Temporal Platform Documentation](#4-docs-tagged-with-user-groups-temporal-platform-documentation) - [8 docs tagged with "Testing" | Temporal Platform Documentation](#8-docs-tagged-with-testing-temporal-platform-documentation) - [12 docs tagged with "setup" | Temporal Platform Documentation](#12-docs-tagged-with-setup-temporal-platform-documentation) - [7 docs tagged with "UI Enrichment" | Temporal Platform Documentation](#7-docs-tagged-with-ui-enrichment-temporal-platform-documentation) - [16 docs tagged with "Metrics" | Temporal Platform Documentation](#16-docs-tagged-with-metrics-temporal-platform-documentation) - [14 docs tagged with "Reference" | Temporal Platform Documentation](#14-docs-tagged-with-reference-temporal-platform-documentation) - [12 docs tagged with "Temporal CLI" | Temporal Platform Documentation](#12-docs-tagged-with-temporal-cli-temporal-platform-documentation) - [6 docs tagged with "Worker" | Temporal Platform Documentation](#6-docs-tagged-with-worker-temporal-platform-documentation) - [6 docs tagged with "Visibility" | Temporal Platform Documentation](#6-docs-tagged-with-visibility-temporal-platform-documentation) - [7 docs tagged with "Users" | Temporal Platform Documentation](#7-docs-tagged-with-users-temporal-platform-documentation) - [9 docs tagged with "Updates" | Temporal Platform Documentation](#9-docs-tagged-with-updates-temporal-platform-documentation) - [19 docs tagged with "Data Converters" | Temporal Platform Documentation](#19-docs-tagged-with-data-converters-temporal-platform-documentation) - [15 docs tagged with "tcld" | Temporal Platform Documentation](#15-docs-tagged-with-tcld-temporal-platform-documentation) - [14 docs tagged with "Temporal Client" | Temporal Platform Documentation](#14-docs-tagged-with-temporal-client-temporal-platform-documentation) - [Temporal CLI command options reference | Temporal Platform Documentation](#temporal-cli-command-options-reference-temporal-platform-documentation) - [10 docs tagged with "Versioning" | Temporal Platform Documentation](#10-docs-tagged-with-versioning-temporal-platform-documentation) - [18 docs tagged with "Self-hosting" | Temporal Platform Documentation](#18-docs-tagged-with-self-hosting-temporal-platform-documentation) - [gcpregions | Temporal Platform Documentation](#gcpregions-temporal-platform-documentation) - [private-service | Temporal Platform Documentation](#private-service-temporal-platform-documentation) - [awsregions | Temporal Platform Documentation](#awsregions-temporal-platform-documentation) - [24 docs tagged with "Observability" | Temporal Platform Documentation](#24-docs-tagged-with-observability-temporal-platform-documentation) - [tcld connectivity-rule command reference | Temporal Platform Documentation](#tcld-connectivity-rule-command-reference-temporal-platform-documentation) - [User management | Temporal Platform Documentation](#user-management-temporal-platform-documentation) - [Worker Basics - Python SDK | Temporal Platform Documentation](#worker-basics-python-sdk-temporal-platform-documentation) - [Worker Versioning (Legacy) - Go SDK | Temporal Platform Documentation](#worker-versioning-legacy-go-sdk-temporal-platform-documentation) - [Worker Versioning (Legacy) - Java SDK | Temporal Platform Documentation](#worker-versioning-legacy-java-sdk-temporal-platform-documentation) - [Worker Versioning (Legacy) - Typescript SDK | Temporal Platform Documentation](#worker-versioning-legacy-typescript-sdk-temporal-platform-documentation) - [Worker Versioning (Legacy) - Python SDK | Temporal Platform Documentation](#worker-versioning-legacy-python-sdk-temporal-platform-documentation) - [Temporal's production deployment features | Temporal Platform Documentation](#temporal-s-production-deployment-features-temporal-platform-documentation) - [Install the TypeScript SDK | Temporal Platform Documentation](#install-the-typescript-sdk-temporal-platform-documentation) - [Worker versioning (legacy) | Temporal Platform Documentation](#worker-versioning-legacy-temporal-platform-documentation) - [Develop with AI | Temporal Platform Documentation](#develop-with-ai-temporal-platform-documentation) - [tctl v1.17 cluster command reference | Temporal Platform Documentation](#tctl-v1-17-cluster-command-reference-temporal-platform-documentation) - [tctl v1.17 activity command reference | Temporal Platform Documentation](#tctl-v1-17-activity-command-reference-temporal-platform-documentation) - [tctl 1.17 schedule command reference | Temporal Platform Documentation](#tctl-1-17-schedule-command-reference-temporal-platform-documentation) - [tctl v1.17 batch command reference | Temporal Platform Documentation](#tctl-v1-17-batch-command-reference-temporal-platform-documentation) - [Quickstarts | Temporal Platform Documentation](#quickstarts-temporal-platform-documentation) - [tctl v1.17 admin command reference | Temporal Platform Documentation](#tctl-v1-17-admin-command-reference-temporal-platform-documentation) - [tctl v1.17 namespace command reference | Temporal Platform Documentation](#tctl-v1-17-namespace-command-reference-temporal-platform-documentation) - [tctl v1.17 data-converter command reference | Temporal Platform Documentation](#tctl-v1-17-data-converter-command-reference-temporal-platform-documentation) - [Temporal Platform production deployments | Temporal Platform Documentation](#temporal-platform-production-deployments-temporal-platform-documentation) - [Why Temporal? | Temporal Platform Documentation](#why-temporal-temporal-platform-documentation) - [Temporal Platform security | Temporal Platform Documentation](#temporal-platform-security-temporal-platform-documentation) - [Temporal Worker Controller | Temporal Platform Documentation](#temporal-worker-controller-temporal-platform-documentation) - [Troubleshoot the blob size limit error | Temporal Platform Documentation](#troubleshoot-the-blob-size-limit-error-temporal-platform-documentation) - [tctl v1.17 taskqueue command reference | Temporal Platform Documentation](#tctl-v1-17-taskqueue-command-reference-temporal-platform-documentation) - [Error handling and troubleshooting | Temporal Platform Documentation](#error-handling-and-troubleshooting-temporal-platform-documentation) - [Quick launch - Deploying your Workers on Amazon EKS | Temporal Platform Documentation](#quick-launch-deploying-your-workers-on-amazon-eks-temporal-platform-documentation) - [Troubleshoot the failed reaching server error | Temporal Platform Documentation](#troubleshoot-the-failed-reaching-server-error-temporal-platform-documentation) - [tctl v1.17 command reference | Temporal Platform Documentation](#tctl-v1-17-command-reference-temporal-platform-documentation) - [Temporal use cases and design patterns | Temporal Platform Documentation](#temporal-use-cases-and-design-patterns-temporal-platform-documentation) - [tctl v1.17 workflow command reference | Temporal Platform Documentation](#tctl-v1-17-workflow-command-reference-temporal-platform-documentation) - [Understanding Temporal | Temporal Platform Documentation](#understanding-temporal-temporal-platform-documentation) - [Performance bottlenecks troubleshooting guide | Temporal Platform Documentation](#performance-bottlenecks-troubleshooting-guide-temporal-platform-documentation) - [Worker Versioning | Temporal Platform Documentation](#worker-versioning-temporal-platform-documentation) - [Temporal Worker deployments | Temporal Platform Documentation](#temporal-worker-deployments-temporal-platform-documentation) - [Troubleshoot the deadline-exceeded error | Temporal Platform Documentation](#troubleshoot-the-deadline-exceeded-error-temporal-platform-documentation) - [Temporal Server options reference | Temporal Platform Documentation](#temporal-server-options-reference-temporal-platform-documentation) - [Evaluate Temporal | Temporal Platform Documentation](#evaluate-temporal-temporal-platform-documentation) - [Temporal Platform references | Temporal Platform Documentation](#temporal-platform-references-temporal-platform-documentation) - [Temporal Web UI configuration reference | Temporal Platform Documentation](#temporal-web-ui-configuration-reference-temporal-platform-documentation) - [Operations | Temporal Platform Documentation](#operations-temporal-platform-documentation) - [Temporal Web UI environment variables reference | Temporal Platform Documentation](#temporal-web-ui-environment-variables-reference-temporal-platform-documentation) - [Safely deploying changes to Workflow code | Temporal Platform Documentation](#safely-deploying-changes-to-workflow-code-temporal-platform-documentation) - [Environment configuration | Temporal Platform Documentation](#environment-configuration-temporal-platform-documentation) - [Temporal Cluster dynamic configuration reference | Temporal Platform Documentation](#temporal-cluster-dynamic-configuration-reference-temporal-platform-documentation) - [Parent Close Policy | Temporal Platform Documentation](#parent-close-policy-temporal-platform-documentation) - [Self-hosted Temporal Service guide | Temporal Platform Documentation](#self-hosted-temporal-service-guide-temporal-platform-documentation) - [Temporal Failures reference | Temporal Platform Documentation](#temporal-failures-reference-temporal-platform-documentation) - [OSS Temporal Service metrics reference | Temporal Platform Documentation](#oss-temporal-service-metrics-reference-temporal-platform-documentation) - [Deploying a Temporal Service | Temporal Platform Documentation](#deploying-a-temporal-service-temporal-platform-documentation) - [Upgrade the Temporal Server | Temporal Platform Documentation](#upgrade-the-temporal-server-temporal-platform-documentation) - [Interceptors | Temporal Platform Documentation](#interceptors-temporal-platform-documentation) - [Server frontend API reference | Temporal Platform Documentation](#server-frontend-api-reference-temporal-platform-documentation) - [Temporal Cluster configuration reference | Temporal Platform Documentation](#temporal-cluster-configuration-reference-temporal-platform-documentation) - [Low latency - Temporal feature | Temporal Platform Documentation](#low-latency-temporal-feature-temporal-platform-documentation) - [Temporal Commands reference | Temporal Platform Documentation](#temporal-commands-reference-temporal-platform-documentation) - [Temporal development and production features | Temporal Platform Documentation](#temporal-development-and-production-features-temporal-platform-documentation) - [Codec Server - Temporal Platform feature guide | Temporal Platform Documentation](#codec-server-temporal-platform-feature-guide-temporal-platform-documentation) - [Embedding Temporal server as a Go library | Temporal Platform Documentation](#embedding-temporal-server-as-a-go-library-temporal-platform-documentation) - [Task Queue Priority and Fairness | Temporal Platform Documentation](#task-queue-priority-and-fairness-temporal-platform-documentation) - [What is Temporal? | Temporal Platform Documentation](#what-is-temporal-temporal-platform-documentation) - [Self-hosted Multi-Cluster Replication | Temporal Platform Documentation](#self-hosted-multi-cluster-replication-temporal-platform-documentation) - [Self-hosted Temporal Service defaults | Temporal Platform Documentation](#self-hosted-temporal-service-defaults-temporal-platform-documentation) - [Managing Namespaces | Temporal Platform Documentation](#managing-namespaces-temporal-platform-documentation) - [Temporal SDK metrics reference | Temporal Platform Documentation](#temporal-sdk-metrics-reference-temporal-platform-documentation) - [Self-hosted Archival setup | Temporal Platform Documentation](#self-hosted-archival-setup-temporal-platform-documentation) - [List Filter | Temporal Platform Documentation](#list-filter-temporal-platform-documentation) - [Temporal product release stages guide | Temporal Platform Documentation](#temporal-product-release-stages-guide-temporal-platform-documentation) - [Multi-tenant application patterns | Temporal Platform Documentation](#multi-tenant-application-patterns-temporal-platform-documentation) - [Worker tuning quick reference | Temporal Platform Documentation](#worker-tuning-quick-reference-temporal-platform-documentation) - [Monitor Temporal Platform metrics | Temporal Platform Documentation](#monitor-temporal-platform-metrics-temporal-platform-documentation) - [What is a Temporal Retry Policy? | Temporal Platform Documentation](#what-is-a-temporal-retry-policy-temporal-platform-documentation) - [Sticky Execution | Temporal Platform Documentation](#sticky-execution-temporal-platform-documentation) - [Patching | Temporal Platform Documentation](#patching-temporal-platform-documentation) - [Temporal Workflow message passing - Signals, Queries, & Updates | Temporal Platform Documentation](#temporal-workflow-message-passing-signals-queries-updates-temporal-platform-documentation) - [Sending Signals, Queries, & Updates | Temporal Platform Documentation](#sending-signals-queries-updates-temporal-platform-documentation) - [Rails integration - Ruby SDK | Temporal Platform Documentation](#rails-integration-ruby-sdk-temporal-platform-documentation) - [Temporal Platform's production readiness checklist | Temporal Platform Documentation](#temporal-platform-s-production-readiness-checklist-temporal-platform-documentation) - [Temporal Workflow | Temporal Platform Documentation](#temporal-workflow-temporal-platform-documentation) - [Self-hosted Temporal Nexus | Temporal Platform Documentation](#self-hosted-temporal-nexus-temporal-platform-documentation) - [Plugins | Temporal Platform Documentation](#plugins-temporal-platform-documentation) - [Dual Visibility | Temporal Platform Documentation](#dual-visibility-temporal-platform-documentation) - [Global Namespace | Temporal Platform Documentation](#global-namespace-temporal-platform-documentation) - [Event History walkthrough with the .NET SDK | Temporal Platform Documentation](#event-history-walkthrough-with-the-net-sdk-temporal-platform-documentation) - [Event History walkthrough with the Go SDK | Temporal Platform Documentation](#event-history-walkthrough-with-the-go-sdk-temporal-platform-documentation) - [Event History walkthrough with the Python SDK | Temporal Platform Documentation](#event-history-walkthrough-with-the-python-sdk-temporal-platform-documentation) - [Event History walkthrough with the TypeScript SDK | Temporal Platform Documentation](#event-history-walkthrough-with-the-typescript-sdk-temporal-platform-documentation) - [Visibility | Temporal Platform Documentation](#visibility-temporal-platform-documentation) - [Multi-Cluster Replication | Temporal Platform Documentation](#multi-cluster-replication-temporal-platform-documentation) - [Run a development server | Temporal Platform Documentation](#run-a-development-server-temporal-platform-documentation) - [Search Attributes | Temporal Platform Documentation](#search-attributes-temporal-platform-documentation) - [Task Routing and Worker sessions | Temporal Platform Documentation](#task-routing-and-worker-sessions-temporal-platform-documentation) - [Worker performance | Temporal Platform Documentation](#worker-performance-temporal-platform-documentation) - [Context Propagation | Temporal Platform Documentation](#context-propagation-temporal-platform-documentation) - [Job Queue | Temporal Platform Documentation](#job-queue-temporal-platform-documentation) - [Task Queues and naming best practices | Temporal Platform Documentation](#task-queues-and-naming-best-practices-temporal-platform-documentation) - [Worker processes - Ruby SDK | Temporal Platform Documentation](#worker-processes-ruby-sdk-temporal-platform-documentation) - [Handling Signals, Queries, & Updates | Temporal Platform Documentation](#handling-signals-queries-updates-temporal-platform-documentation) - [Enriching the user interface - Ruby SDK | Temporal Platform Documentation](#enriching-the-user-interface-ruby-sdk-temporal-platform-documentation) - [Detecting application failures | Temporal Platform Documentation](#detecting-application-failures-temporal-platform-documentation) - [Extensibility | Temporal Platform Documentation](#extensibility-temporal-platform-documentation) - [Worker Shutdown Behavior | Temporal Platform Documentation](#worker-shutdown-behavior-temporal-platform-documentation) - [Set up your local with the Ruby SDK | Temporal Platform Documentation](#set-up-your-local-with-the-ruby-sdk-temporal-platform-documentation) - [Schedule | Temporal Platform Documentation](#schedule-temporal-platform-documentation) - [Temporal Server | Temporal Platform Documentation](#temporal-server-temporal-platform-documentation) - [Cloud automation - Temporal feature | Temporal Platform Documentation](#cloud-automation-temporal-feature-temporal-platform-documentation) - [Interrupt a Workflow - Cancellation and Termination | Temporal Platform Documentation](#interrupt-a-workflow-cancellation-and-termination-temporal-platform-documentation) - [Standalone Activity | Temporal Platform Documentation](#standalone-activity-temporal-platform-documentation) - [Security in Temporal Nexus | Temporal Platform Documentation](#security-in-temporal-nexus-temporal-platform-documentation) - [Nexus services | Temporal Platform Documentation](#nexus-services-temporal-platform-documentation) - [Detecting Workflow failures | Temporal Platform Documentation](#detecting-workflow-failures-temporal-platform-documentation) - [Detecting Activity failures | Temporal Platform Documentation](#detecting-activity-failures-temporal-platform-documentation) - [Debugging - Temporal feature | Temporal Platform Documentation](#debugging-temporal-feature-temporal-platform-documentation) - [Client - TypeScript SDK | Temporal Platform Documentation](#client-typescript-sdk-temporal-platform-documentation) - [AI integrations | Temporal Platform Documentation](#ai-integrations-temporal-platform-documentation) - [Best Practices - Ruby SDK | Temporal Platform Documentation](#best-practices-ruby-sdk-temporal-platform-documentation) - [Enriching the user interface - TypeScript SDK | Temporal Platform Documentation](#enriching-the-user-interface-typescript-sdk-temporal-platform-documentation) - [Dynamic handler | Temporal Platform Documentation](#dynamic-handler-temporal-platform-documentation) - [Entity pattern - TypeScript SDK | Temporal Platform Documentation](#entity-pattern-typescript-sdk-temporal-platform-documentation) - [Multi-tenancy - Temporal feature | Temporal Platform Documentation](#multi-tenancy-temporal-feature-temporal-platform-documentation) - [Schedules - Temporal feature | Temporal Platform Documentation](#schedules-temporal-feature-temporal-platform-documentation) - [Temporal Nexus - Temporal feature | Temporal Platform Documentation](#temporal-nexus-temporal-feature-temporal-platform-documentation) - [Manage Interceptors - TypeScript SDK | Temporal Platform Documentation](#manage-interceptors-typescript-sdk-temporal-platform-documentation) - [Set up your local with the Typescript SDK | Temporal Platform Documentation](#set-up-your-local-with-the-typescript-sdk-temporal-platform-documentation) - [Tasks | Temporal Platform Documentation](#tasks-temporal-platform-documentation) - [Workflow Execution limits | Temporal Platform Documentation](#workflow-execution-limits-temporal-platform-documentation) - [Temporal Platform security features | Temporal Platform Documentation](#temporal-platform-security-features-temporal-platform-documentation) - [Testing - Ruby SDK | Temporal Platform Documentation](#testing-ruby-sdk-temporal-platform-documentation) - [Data encryption - Temporal feature | Temporal Platform Documentation](#data-encryption-temporal-feature-temporal-platform-documentation) - [Archival | Temporal Platform Documentation](#archival-temporal-platform-documentation) - [Temporal Events reference | Temporal Platform Documentation](#temporal-events-reference-temporal-platform-documentation) - [Event History walkthrough with the Java SDK | Temporal Platform Documentation](#event-history-walkthrough-with-the-java-sdk-temporal-platform-documentation) - [Manage Namespaces - TypeScript SDK | Temporal Platform Documentation](#manage-namespaces-typescript-sdk-temporal-platform-documentation) - [Best Practices - TypeScript SDK | Temporal Platform Documentation](#best-practices-typescript-sdk-temporal-platform-documentation) - [Payload Converter | Temporal Platform Documentation](#payload-converter-temporal-platform-documentation) - [Nexus Registry | Temporal Platform Documentation](#nexus-registry-temporal-platform-documentation) - [Worker Versioning | Temporal Platform Documentation](#worker-versioning-temporal-platform-documentation) - [Errors | Temporal Platform Documentation](#errors-temporal-platform-documentation) - [Activities - Ruby SDK | Temporal Platform Documentation](#activities-ruby-sdk-temporal-platform-documentation) - [Local Activity | Temporal Platform Documentation](#local-activity-temporal-platform-documentation) - [Temporal Namespace | Temporal Platform Documentation](#temporal-namespace-temporal-platform-documentation) - [Observability - Temporal feature | Temporal Platform Documentation](#observability-temporal-feature-temporal-platform-documentation) - [AI SDK by Vercel integration | Temporal Platform Documentation](#ai-sdk-by-vercel-integration-temporal-platform-documentation) - [Error handling - Ruby SDK | Temporal Platform Documentation](#error-handling-ruby-sdk-temporal-platform-documentation) - [Workflows - TypeScript SDK | Temporal Platform Documentation](#workflows-typescript-sdk-temporal-platform-documentation) - [Persistence | Temporal Platform Documentation](#persistence-temporal-platform-documentation) - [Payload Codec | Temporal Platform Documentation](#payload-codec-temporal-platform-documentation) - [Workflow Id and Run Id | Temporal Platform Documentation](#workflow-id-and-run-id-temporal-platform-documentation) - [Workers - Ruby SDK | Temporal Platform Documentation](#workers-ruby-sdk-temporal-platform-documentation) - [Asynchronous Activity - TypeScript SDK | Temporal Platform Documentation](#asynchronous-activity-typescript-sdk-temporal-platform-documentation) - [Cloud Ops API | Temporal Platform Documentation](#cloud-ops-api-temporal-platform-documentation) - [Remote data encoding | Temporal Platform Documentation](#remote-data-encoding-temporal-platform-documentation) - [What is a Temporal Worker? | Temporal Platform Documentation](#what-is-a-temporal-worker-temporal-platform-documentation) - [Temporal Workflow Execution overview | Temporal Platform Documentation](#temporal-workflow-execution-overview-temporal-platform-documentation) - [Temporal Service | Temporal Platform Documentation](#temporal-service-temporal-platform-documentation) - [Events and Event History | Temporal Platform Documentation](#events-and-event-history-temporal-platform-documentation) - [External Storage | Temporal Platform Documentation](#external-storage-temporal-platform-documentation) - [Worker processes - TypeScript SDK | Temporal Platform Documentation](#worker-processes-typescript-sdk-temporal-platform-documentation) - [Benign exceptions - TypeScript SDK | Temporal Platform Documentation](#benign-exceptions-typescript-sdk-temporal-platform-documentation) - [Client - Ruby SDK | Temporal Platform Documentation](#client-ruby-sdk-temporal-platform-documentation) - [Integrations - Ruby SDK | Temporal Platform Documentation](#integrations-ruby-sdk-temporal-platform-documentation) - [Activity execution - TypeScript SDK | Temporal Platform Documentation](#activity-execution-typescript-sdk-temporal-platform-documentation) - [Error Handling - Temporal Nexus | Temporal Platform Documentation](#error-handling-temporal-nexus-temporal-platform-documentation) - [Execution Debugging - Temporal Nexus | Temporal Platform Documentation](#execution-debugging-temporal-nexus-temporal-platform-documentation) - [Activity basics - TypeScript SDK | Temporal Platform Documentation](#activity-basics-typescript-sdk-temporal-platform-documentation) - [Activity Timeouts - Ruby SDK | Temporal Platform Documentation](#activity-timeouts-ruby-sdk-temporal-platform-documentation) - [Debugging - TypeScript SDK | Temporal Platform Documentation](#debugging-typescript-sdk-temporal-platform-documentation) - [Converters and encryption - Ruby SDK | Temporal Platform Documentation](#converters-and-encryption-ruby-sdk-temporal-platform-documentation) - [About Temporal SDKs | Temporal Platform Documentation](#about-temporal-sdks-temporal-platform-documentation) - [Client - Ruby SDK | Temporal Platform Documentation](#client-ruby-sdk-temporal-platform-documentation) - [Core application - Temporal feature | Temporal Platform Documentation](#core-application-temporal-feature-temporal-platform-documentation) - [Workflow message passing - Temporal feature | Temporal Platform Documentation](#workflow-message-passing-temporal-feature-temporal-platform-documentation) - [Continue-As-New | Temporal Platform Documentation](#continue-as-new-temporal-platform-documentation) - [Feature guide - TypeScript SDK feature guide | Temporal Platform Documentation](#feature-guide-typescript-sdk-feature-guide-temporal-platform-documentation) - [Self-hosted Visibility feature setup | Temporal Platform Documentation](#self-hosted-visibility-feature-setup-temporal-platform-documentation) - [Dynamic Workflow - Ruby SDK | Temporal Platform Documentation](#dynamic-workflow-ruby-sdk-temporal-platform-documentation) - [Observability - Ruby SDK | Temporal Platform Documentation](#observability-ruby-sdk-temporal-platform-documentation) - [Temporal Visibility | Temporal Platform Documentation](#temporal-visibility-temporal-platform-documentation) - [Nexus Operations | Temporal Platform Documentation](#nexus-operations-temporal-platform-documentation) - [Temporal Client - Ruby SDK | Temporal Platform Documentation](#temporal-client-ruby-sdk-temporal-platform-documentation) - [Workflow futures - Ruby SDK | Temporal Platform Documentation](#workflow-futures-ruby-sdk-temporal-platform-documentation) - [Temporal Testing Suite - Temporal feature | Temporal Platform Documentation](#temporal-testing-suite-temporal-feature-temporal-platform-documentation) - [Nexus Patterns | Temporal Platform Documentation](#nexus-patterns-temporal-platform-documentation) - [Timers and Start Delays | Temporal Platform Documentation](#timers-and-start-delays-temporal-platform-documentation) - [Temporal Workflow Definition | Temporal Platform Documentation](#temporal-workflow-definition-temporal-platform-documentation) - [Workflow basics - Ruby SDK | Temporal Platform Documentation](#workflow-basics-ruby-sdk-temporal-platform-documentation) - [Child Workflows - Temporal feature | Temporal Platform Documentation](#child-workflows-temporal-feature-temporal-platform-documentation) - [Client - TypeScript SDK | Temporal Platform Documentation](#client-typescript-sdk-temporal-platform-documentation) - [Workers - TypeScript SDK | Temporal Platform Documentation](#workers-typescript-sdk-temporal-platform-documentation) - [Cancellation - Ruby SDK | Temporal Platform Documentation](#cancellation-ruby-sdk-temporal-platform-documentation) - [Nexus Endpoints | Temporal Platform Documentation](#nexus-endpoints-temporal-platform-documentation) - [Nexus Metrics | Temporal Platform Documentation](#nexus-metrics-temporal-platform-documentation) - [Event History | Temporal Platform Documentation](#event-history-temporal-platform-documentation) - [Timers - Ruby SDK | Temporal Platform Documentation](#timers-ruby-sdk-temporal-platform-documentation) - [Testing - TypeScript SDK | Temporal Platform Documentation](#testing-typescript-sdk-temporal-platform-documentation) --- # Temporal Docs | Temporal Platform Documentation [Skip to main content](https://docs.temporal.io/#__docusaurus_skipToContent_fallback) * [](https://docs.temporal.io/develop/go) * [](https://docs.temporal.io/develop/java) * [](https://docs.temporal.io/develop/php) * [](https://docs.temporal.io/develop/python) * [](https://docs.temporal.io/develop/ruby) * [](https://docs.temporal.io/develop/typescript) * [](https://docs.temporal.io/develop/dotnet) Build applications that never fail ================================== Temporal is an open-source platform for building reliable applications. Temporal delivers crash-proof execution by guaranteeing that applications resume exactly where they left off after crashes, network failures, or infrastructure outages, whether that happens seconds, days, or even years later. Temporal enables developers to focus on building features that drive the business while ensuring that mission-critical processes such as order fulfillment, customer onboarding, and payment processing never fail or disappear, regardless of what goes wrong. [Quickstart](https://docs.temporal.io/quickstarts) [![Lightning icon](https://docs.temporal.io/img/icons/bolt-dark-mode-24x24.svg)![Lightning icon](https://docs.temporal.io/img/icons/bolt-24x24.svg)\ \ ### Quickstart\ \ Setup your local and run a Hello World workflow.](https://docs.temporal.io/quickstarts) [![Code icon](https://docs.temporal.io/img/icons/code-dark-mode-24x24.svg)![Code icon](https://docs.temporal.io/img/icons/code-24x24.svg)\ \ ### Developer Guide\ \ Dive into everything you need to know about building Temporal workflows.](https://docs.temporal.io/develop) [![Rocket icon](https://docs.temporal.io/img/icons/rocket-dark-mode-24x24.svg)![Rocket icon](https://docs.temporal.io/img/icons/rocket-24x24.svg)\ \ ### Deploy your Workflows\ \ Deploy your Temporal Application to your environment. Self-Host the Temporal Service or use Temporal Cloud.](https://docs.temporal.io/production-deployment) [![Cloud icon](https://docs.temporal.io/img/icons/cloud-dark-mode-24x24.svg)![Cloud icon](https://docs.temporal.io/img/icons/cloud-24x24.svg)\ \ ### Get started for free with $1000 in credits\ \ Sign up for Temporal Cloud and let us host the Temporal Service for you.](https://temporal.io/cloud) ![Slack](https://docs.temporal.io/img/icons/slack-dark-mode-24x24.svg)![Slack](https://docs.temporal.io/img/icons/slack-24x24.svg) ### Slack Community Join us on [temporal.io/slack](https://temporal.io/slack) and say hi or ask us a question. ![Message](https://docs.temporal.io/img/icons/forum-dark-mode-24x24.svg)![Message](https://docs.temporal.io/img/icons/forum-24x24.svg) ### Developer Forum [Find out](https://community.temporal.io/) if your question has already been asked. ![Education](https://docs.temporal.io/img/icons/learn-dark-mode-24x24.svg)![Education](https://docs.temporal.io/img/icons/learn-24x24.svg) ### Learn it all [Master Temporal](https://learn.temporal.io/courses/) with our courses and tutorials. --- # Unknown TEMPORAL TERMS OF SERVICE PLEASE READ THESE TERMS AND CONDITIONS CAREFULLY BEFORE USING THE SERVICE OFFERED BY TEMPORAL TECHNOLOGIES, INC. (“TEMPORAL”). BY MUTUALLY EXECUTING ONE OR MORE ORDER FORMS WITH TEMPORAL WHICH REFERENCE THESE TERMS (EACH, AN “ORDER”) OR BY ACCESSING OR USING THE SERVICE IN ANY MANNER, YOU (“YOU” OR “CUSTOMER”) AGREE TO BE BOUND BY THESE TERMS (TOGETHER WITH ALL ORDER FORMS, THE “AGREEMENT”) TO THE EXCLUSION OF ALL OTHER TERMS. YOU REPRESENT AND WARRANT THAT YOU HAVE THE AUTHORITY TO ENTER INTO THIS AGREEMENT; IF YOU ARE ENTERING INTO THIS AGREEMENT ON BEHALF OF AN ORGANIZATION OR ENTITY, REFERENCES TO “CUSTOMER” AND “YOU” IN THIS AGREEMENT, EXCEPT THIS SENTENCE, REFER TO THAT ORGANIZATION OR ENTITY. IF YOU DO NOT AGREE TO ALL OF THE FOLLOWING, YOU MAY NOT USE OR ACCESS THE SERVICES IN ANY MANNER. IF THE TERMS OF THIS AGREEMENT ARE CONSIDERED AN OFFER, ACCEPTANCE IS EXPRESSLY LIMITED TO SUCH TERMS. 1.SCOPE OF SERVICE AND RESTRICTIONS 1.1 Access to and Scope of Service. Subject to Temporal’s receipt of the applicable Fees with respect to the service(s) specified in the corresponding Order (the “Service”), Temporal will use commercially reasonable efforts to make the Service available to Customer as set forth in this Agreement and the Order. Subject to Customer’s compliance with the terms and condiZons of the Agreement and the Order, Customer may access and use the Service according to the authorized use specified in the Order (the “Authorized Use”), solely during the authorized period specified therein (the “Authorized Period”). Any such use of the Service by Customer is authorized solely for Customer’s internal business, and is subject to Customer’s compliance with any addiZonal limitaZons and restricZons specified in the Order. 1.2 Trials and No-fee Access. If Customer is accessing or making use of the Service on a no-fee, trial, or evaluaZon basis (the “Limited Use”), Customer may use the Service during the Limited Use provided such use does not to exceed the Service levels specified on the Order with respect to Limited Use. Customer acknowledges and agrees that the Limited Use is provided on an “as-is” basis, and the Limited Use is provided without any indemnificaZon, support, warranZes, or representaZon of any kind. AddiZonally, Customer acknowledges and agrees that Temporal may terminate the Limited Use at any Zme. 1.3 RestricKons. Customer will use the Service only in accordance with all applicable laws, including, but not limited to, laws related to data (whether applicable within the United States, the European Union, or otherwise). Customer agrees not to (and will not allow any third party to): (i) remove or otherwise alter any proprietary noZces or labels from the Service or any porZon thereof; (ii) reverse engineer, decompile, disassemble, or otherwise a\_empt to discover the underlying structure, ideas, or algorithms of the Service or any so\`ware used to provide or make the Service available; (iii) rent, resell or otherwise allow any third party access to or use of the Service; or (iv) access or use the Service other than according to the Authorized Use and during the Authorized Period. 1.4 Ownership. Temporal retains all right, Ztle, and interest in and to the Service, and any so\`ware, products, works or other intellectual property created, used, provided or made available by Temporal under or in connecZon with the Service. Customer may from Zme to Zme provide suggesZons, comments or other feedback to Temporal with respect to the Service (“Feedback”). Feedback, even if designated as confidenZal by Customer, shall not create any confidenZality obligaZon for Temporal notwithstanding anything else. Customer shall, and hereby does, grant to Temporal a nonexclusive, worldwide, perpetual, irrevocable, transferable, sublicensable, royalty-free, fully paid-up license to use and exploit the Feedback for any purpose. Nothing in this Agreement will impair Temporal’s right to develop, acquire, license, market, promote or distribute products, so\`ware or technologies that perform the same or similar funcZons as, or otherwise compete with any products, so\`ware or technologies that Customer may develop, produce, market, or distribute. 1.5 SoQware. Subject to the terms and condiZons of this Agreement, including but not limited to receipt of all applicable Fees, to the extent Temporal makes any Temporal so\`ware available to Customer, Temporal hereby grants to Customer, and Customer hereby accepts from Temporal, a limited, non-exclusive, non-transferable, non-assignable and non-sublicenseable license to: run such so\`ware solely as necessary to make use of the Service. Customer agrees that, it shall not: (a) exceed the scope of the licenses granted in SecZon 1.5; (b) make copies of the so\`ware; (c) distribute, sublicense, assign, delegate, rent, lease, sell, Zme-share or otherwise transfer the benefits of, use under, or rights to, the license granted in SecZon 1.5; (d) reverse engineer, decompile, disassemble or otherwise a\_empt to learn the source code, structure or algorithms underlying the so\`ware, except to the extent required to be permi\_ed under applicable law; (e) modify, translate or create derivaZve works of the so\`ware; (f) remove any copyright, trademark, patent or other proprietary noZce that appears on the so\`ware or copies thereof; or (g) combine or distribute any of the so\`ware with any third party so\`ware that is licensed under terms that seek to require that any of the so\`ware (or any associated intellectual property rights) be provided in source code form (e.g., as “open source”), licensed to others to allow the creaZon or distribuZon of derivaZve works, or distributed without charge. 1.6 Customer Data and Personal Data. Customer is solely responsible for Customer Data including, but not limited to: compliance with all applicable laws and this Agreement; any third-party claims with respect to Customer Data; and backing up and maintaining Customer Data. Notwithstanding anything to the contrary, Customer acknowledges and agrees that Temporal may use Customer Data for the purposes of providing the Services and making any improvements thereto, , and generaZng Aggregated Data. Temporal may freely use and make available Aggregated Data for Temporal’s business purposes (including without limitaZon, for purposes of improving, tesZng, operaZng, promoZng and markeZng Temporal’s products and services). “Aggregated Data” means data submi\_ed to, collected by, or generated by Temporal in connecZon with Customer’s use of the Service, but only in aggregate, de-idenZfied form which is not linked specifically to Customer or any individual. “Customer Data” means any data, informaZon or other material provided, uploaded, or submi\_ed by Customer to the Service in the course of using the Service. Customer shall retain all right, Ztle and interest in and to the Customer Data, including all intellectual property rights therein. Customer, not Temporal, shall have sole responsibility for the accuracy, quality, integrity, legality, reliability, appropriateness, and intellectual property ownership or right to use of all Customer Data. Temporal is not responsible for unauthorized access to Customer Data or the unauthorized use of the Service unless such access is due to Temporal’s gross negligence or willful misconduct. Customer is responsible for the use of the Service by any person to whom Customer has given access to the Service, even if Customer did not authorize such use. Temporal may retain Customer Data for up to thirty (30) days following the terminaZon or expiraZon of the corresponding Order. Therea\`er, Customer agrees and acknowledges that Customer Data may be irretrievably deleted. 1.7 InformaKon Security. To the extent Temporal accesses Customer’s network in connecZon with the Services, Temporal access shall be consistent with the Temporal informaZon security policy (the “InfoSec Policy”) a\_ached or referenced to the corresponding Order. 1.8 UpKme. Subject to Customer’s payment of the corresponding fees, Temporal will use commercially reasonable efforts to make the Service available to Customer according to the upZme service level agreement specified in an Order (the “SLA”). 1.9 Service Suspension. Temporal may suspend Customer’s access to or use of the Service as follows: (a) immediately if Temporal reasonably believes Customer’s use of the Service may pose a security risk to or may adversely impact the Service; (b) immediately if Customer become insolvent, has ceased to operate in the ordinary course, made an assignment for the benefit of creditors, or becomes the subject of any bankruptcy, reorganizaZon, liquidaZon, dissoluZon or similar proceeding; (c) following thirty (30) days wri\_en noZce if Customer is in breach of this Agreement or any Order (and has not cured such breach, if curable, within the thirty (30) days of such noZce); or (d) Customer has failed to pay Temporal the Fees with respect to the Service. 2.FEES, ORDERS AND TAXES 2.1 Fees. Customer shall pay to Temporal the fees as set forth in each applicable Order(s) (collecZvely, the “Fees”). Customer acknowledges that it shall have no right to return the Service and that all Fees shall be non- refundable. All amounts payable to Temporal under this Agreement shall be paid in United States dollars and shall be due thirty (30) days from the date of invoice. Notwithstanding any other rights of Temporal, in the event of late payment by Customer, Temporal shall be enZtled to interest on the amount owing at a rate of 1% per month or the highest rate allowed by applicable law, whichever is less. If Temporal is required to iniZate legal acZon due to nonpayment of fees, Customer shall bear all costs resulZng from the collecZon of such fees. 2.2 Orders. Licensee may place Orders for addiZonal Services or to extend the term of the exisZng Service by specifying such order details in an Order form agreed to in wriZng by the parZes referencing the terms and condiZons of this Agreement. 2.3 Taxes. Any and all payments made by Temporal in accordance with this Agreement are exclusive of any taxes that might be assessed against Customer by any jurisdicZon. Customer shall pay or reimburse Temporal for all value-added, sales, use, property and similar taxes; all customs duZes, import fees, stamp duZes, license fees and similar charges; and all other mandatory payments to government agencies of whatever kind, except taxes imposed on the net or gross income of Temporal. All amounts payable to Temporal under this Agreement shall be without set-off and without deducZon of any taxes, levies, imposts, charges, withholdings and/or duZes of any nature which may be levied or imposed, including without limitaZon, value added tax, customs duty and withholding tax. 3.TERM AND TERMINATION 3.1 Term. The term of this Agreement shall commence on the EffecZve and unless terminated earlier according to this SecZon 3, will end on the last day of the term specified in a last Order (the “Term”). Each Order will renew automaZcally at the end of the applicable term unless either party provides to the other advance wri\_en noZce with respect to non-renewal at least ninety (90) days prior to the end of the then current term. 3.2 TerminaKon for Breach. This Agreement and the Orders hereunder may be terminated: (a) by either party if the other has materially breached this Agreement, within thirty (30) calendar days a\`er wri\_en noZce of such breach to the other party if the breach is remediable or immediately upon noZce if the breach is not remediable; or (b) by Temporal upon wri\_en noZce to Customer if Customer (i) has made or a\_empted to make any assignment for the benefit of its creditors or any composiZons with creditors, (ii) has any acZon or proceedings under any bankruptcy or insolvency laws taken by or against it which have not been dismissed within sixty (60) days. 3.3 Effect of TerminaZon. Upon any expiraZon or terminaZon of this Agreement, Customer shall (i) immediately cease use of the Service, and (ii) return all Temporal ConfidenZal InformaZon, and Temporal provided so\`ware, and other materials and informaZon provided by Temporal. Any terminaZon or expiraZon shall not relieve Customer of its obligaZon to pay all Fees accruing prior to terminaZon. If the Agreement is terminated by Temporal pursuant to SecZon 3.2 (a), Customer shall pay to Temporal all of the Fees for the enZre term set forth in the corresponding Order(s). 3.4 Survival. The following provisions will survive terminaZon of this Agreement: SecZons 1.5 (Ownership), 3.3 (Effect of TerminaZon), SecZon 3.4 (Survival), SecZon 4 (ConfidenZality), SecZon 5.1 (IndemnificaZon by Customer), SecZon 7 (LimitaZon of Liability), SecZon 8 (Miscellaneous). 4.CONFIDENTIALITY During the term of this Agreement, either party may provide the other party with confidenZal and/or proprietary materials and informaZon (“ConfidenKal InformaKon”). All materials and informaZon provided by the disclosing party and idenZfied at the Zme of disclosure as “ConfidenZal” or bearing a similar legend, and all other informaZon that the receiving party reasonably should have known was the ConfidenZal InformaZon of the disclosing party, shall be considered ConfidenZal InformaZon. This Agreement is ConfidenZal InformaZon, and all pricing terms are Temporal ConfidenZal InformaZon. The receiving party shall maintain the confidenZality of the ConfidenZal InformaZon and will not disclose such informaZon to any third party without the prior wri\_en consent of the disclosing party. The receiving party will only use the ConfidenZal InformaZon internally for the purposes contemplated hereunder. The obligaZons in this SecZon shall not apply to any informaZon that: (a) is made generally available to the public without breach of this Agreement, (b) is developed by the receiving party independently from and without reference to the ConfidenZal InformaZon, (c) is disclosed to the receiving party by a third party without restricZon, or (d) was in the receiving party’s lawful possession prior to the disclosure and was not obtained by the receiving party either directly or indirectly from the disclosing party. The receiving party may disclose ConfidenZal InformaZon as required by law or court order; provided that, the receiving party provides the disclosing with prompt wri\_en noZce thereof and uses the receiving party’s best efforts to limit disclosure. At any Zme, upon the disclosing party’s wri\_en request, the receiving party shall return to the disclosing party all disclosing party’s ConfidenZal InformaZon in its possession, including, without limitaZon, all copies and extracts thereof. 5.INDEMNIFICATION 5.1 IndemnificaKon by Customer. Customer will defend, indemnify, and hold Temporal, its affiliates, suppliers and licensors harmless and each of their respecZve officers, directors, employees and representaZves from and against any claims, damages, losses, liabiliZes, costs, and expenses (including reasonable a\_orneys’ fees) arising out of or relaZng to any third party claim with respect to: (a) Customer Data; (b) breach of this Agreement or violaZon of applicable law by Customer; or (c) alleged infringement or misappropriaZon of third-party’s intellectual property rights resulZng from Customer Data. 5.2 IndemnificaKon by Temporal. Temporal will defend, indemnify, and hold Customer harmless from and against any third-party claims, damages, losses, liabiliZes, costs, and expenses (including reasonable a\_orneys’ fees) arising from claims by a thirty party that Customer’s use of the Service directly infringes or misappropriates a third party’s United States (or Berne ConvenZon signatory country) intellectual property rights (an “Infringement Claim”). Notwithstanding any other provision in this Agreement, Temporal shall have no obligaZon to indemnify or reimburse Customer with respect to any Infringement Claim to the extent arising from: (a) the combinaZon of any Customer Data with the Service; (b) the combinaZon of any products or services, other than those provided by Temporal to Customer under this Agreement, with the Service; or (c) non-discreZonary designs or specificaZons provided to Temporal by Customer that caused such Infringement Claim. Customer agrees to reimburse Temporal for any and all damages, losses, costs and expenses incurred as a result of any of the foregoing acZons. 5.3 NoZce of Claim and Indemnity Procedure. In the event of a claim for which a party seeks indemnity or reimbursement under this SecZon 5 (each an “Indemnified Party”) and as condiZons of the indemnity, the Indemnified Party shall: (a) noZfy the indemnifying party in wriZng as soon as pracZcable, but in no event later than thirty (30) days a\`er receipt of such claim, together with such further informaZon as is necessary for the indemnifying party to evaluate such claim; and (b) the Indemnified Party allows the indemnifying party to assume full control of the defense of the claim, including retaining counsel of its own choosing. Upon the assumpZon by the indemnifying party of the defense of a claim with counsel of its choosing, the indemnifying party will not be liable for the fees and expenses of addiZonal counsel retained by any Indemnified Party. The Indemnified Party shall cooperate with the indemnifying party in the defense of any such claim. Notwithstanding the foregoing provisions, the indemnifying party shall have no obligaZon to indemnify or reimburse for any losses, damages, costs, disbursements, expenses, se\_lement liability of a claim or other sums paid by any Indemnified Party voluntarily, and without the indemnifying party’s prior wri\_en consent, to se\_le a claim. Subject to the maximum liability set forth in SecZon 7, the provisions of this SecZon 5 consZtute the enZre understanding of the parZes regarding each party’s respecZve liability under this SecZon 5, including but not limited to Infringement Claims (including related claims for breach of warranty) and each party’s sole obligaZon to indemnify and reimburse any Indemnified Party. 6.WARRANTY 6.1 Warranty. The Service, when used by Customer in accordance with the provisions of this Agreement and in compliance with the applicable specificaZons will perform, in all material respects, the funcZons described in the Order (the “SpecificaKon”), during the term in the corresponding Order. 6.2 Exclusive Remedies. Customer shall report to Temporal, pursuant to the noZce provision of this Agreement, any breach of the warranty set forth in this SecZon 6. In the event of a breach of warranty by Temporal under this Agreement, Customer’s sole and exclusive remedy, and Temporal’s enZre liability, shall be prompt correcZon of any material non-conformance in order to minimize any material adverse effect on Customer’s business. 6.3 Disclaimer of Warranty. Temporal does not represent or warrant that the operaZon of the Service (or any porZon thereof) will be uninterrupted or error free, or that the Service (or any porZon thereof) will operate in combinaZon with other hardware, so\`ware, systems or data not provided by Temporal, except as expressly specified in the applicable SpecificaZon. CUSTOMER ACKNOWLEDGES THAT, EXCEPT AS EXPRESSLY SET FORTH IN SECTION 6.1, TEMPORAL MAKES NO EXPRESS OR IMPLIED REPRESENTATIONS OR WARRANTIES OF ANY KIND WITH RESPECT TO THE SERVICE OR SERVICES, OR THEIR CONDITION. TEMPORAL IS FURNISHING THE WARRANTY SET FORTH IN SECTION 6.1 IN LIEU OF, AND TEMPORAL HEREBY EXPRESSLY EXCLUDES, ANY AND ALL OTHER EXPRESS OR IMPLIED REPRESENTATIONS OR WARRANTIES, WHETHER UNDER COMMON LAW, STATUTE OR OTHERWISE, INCLUDING WITHOUT LIMITATION ANY AND ALL WARRANTIES AS TO MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE, SATISFACTORY QUALITY OR NON-INFRINGEMENT OF THIRD-PARTY RIGHTS. 7.LIMITATIONS OF LIABILITY IN NO EVENT SHALL TEMPORAL BE LIABLE FOR ANY LOST DATA, LOST PROFITS, BUSINESS INTERRUPTION, REPLACEMENT SERVICE OR OTHER SPECIAL, INCIDENTAL, CONSEQUENTIAL, PUNITIVE OR INDIRECT DAMAGES, HOWEVER CAUSED AND REGARDLESS OF THEORY OF LIABILITY. TEMPORAL’S LIABILITY FOR ALL CLAIMS ARISING UNDER THIS AGREEMENT, WHETHER IN CONTRACT, TORT OR OTHERWISE, SHALL NOT EXCEED THE AMOUNT OF FEES PAID OR PAYABLE BY CUSTOMER UNDER THE APPLICABLE ORDER DURING THE TWELVE (12) MONTH PERIOD PRECEEDING THE CLAIM. 8.MISCELLANEOUS 8.1 Export Control. Customer hereby cerZfies that Customer will comply with all current US Export Control laws. Customer agrees to defend, indemnify and hold Temporal harmless from any liability for Customer’s violaZon of U.S. Export Control laws. 8.2 Compliance with Laws. Customer shall comply with all applicable laws and regulaZons in its use of any Service, including without limitaZon the unlawful gathering or collecZng, or assisZng in the gathering or collecZng of informaZon in violaZon of any privacy laws or regulaZons. Customer shall, at its own expense, defend, indemnify and hold harmless Temporal from and against any and all claims, losses, liabiliZes, damages, judgments, government or federal sancZons, costs and expenses (including a\_orneys’ fees) incurred by Temporal arising from any claim or asserZon by any third party of violaZon of privacy laws or regulaZons by Customer or any of its agents, officers, directors or employees. 8.3 Assignment. Neither party may transfer and assign its rights and obligaZons under this Agreement without the prior wri\_en consent of the other party. Notwithstanding the foregoing, Temporal may transfer and assign its rights under this Agreement without consent from the other party in connecZon with a change in control, acquisiZon or sale of all or substanZally all of its assets. 8.4 Force Majeure. Neither party shall be responsible for failure or delay in performance by events out of their reasonable control, including but not limited to, acts of God, Internet outage, terrorism, war, fires, earthquakes and other disasters (each a “Force Majeure”). Notwithstanding the foregoing: (i) Customer shall be liable for payment obligaZons for Service rendered; and (ii) if a Force Majeure conZnues for more than thirty (30) days, either party may to terminate this agreement by wri\_en noZce to the other party. 8.5 NoKce. All noZces between the parZes shall be in wriZng and shall be deemed to have been given if personally delivered or sent by registered or cerZfied mail (return receipt), or by recognized courier service. 8.6 Independent Contractor. Temporal is an independent Contractor and both parZes agree that no agency, partnership, joint venture, or employment is created as a result of this Agreement. Customer does not have any authority of any kind to bind Temporal. 8.7 Governing Law. This Agreement shall be governed exclusively by, and construed exclusively in accordance with, the laws of the United States and the State of California, without regard to its conflict of laws provisions. The federal courts of the United States in the Northern District of California and the state courts of the State of California shall have exclusive jurisdicZon to adjudicate any dispute arising out of or relaZng to this Agreement. Each party hereby consents to the jurisdicZon of such courts and waives any right it may otherwise have to challenge the appropriateness of such forums, whether on the basis of the doctrine of forum non conveniens or otherwise. The United NaZons ConvenZon on Contracts for the InternaZonal Sale of Goods shall not apply to this Agreement or any Purchase Order issued under this Agreement. 8.8 MarkeKng. Customer hereby grants Temporal the right to idenZfy Customer as a Temporal customer, and use Customer’s name, mark and logo on Temporal’s website and in Temporal’s markeZng materials in connecZon with the Customer’s use of the Service. 8.9 EnKre Agreement. This Agreement is the complete and exclusive statement of the mutual understanding of the parZes and supersedes and cancels all previous wri\_en and oral agreements, communicaZons, and other understandings relaZng to the subject ma\_er of this Agreement, and all waivers and modificaZons must be in a wriZng signed by both parZes, except as otherwise provided herein. Any term or provision of this Agreement held to be illegal or unenforceable shall be, to the fullest extent possible, interpreted so as to be construed as valid, but in any event the validity or enforceability of the remainder hereof shall not be affected. In the event of a conflict between this Agreement and the Order document, the terms of this Agreement shall control, other than terms expressly modified in any Order with respect to such Order. Date: March 2021 --- # AI Cookbook | Temporal Platform Documentation [Skip to main content](https://docs.temporal.io/ai-cookbook#__docusaurus_skipToContent_fallback) [Hello World\ -----------\ \ Simple example demonstrating how to call an LLM from Temporal using the OpenAI Python API library.\ \ * FOUNDATIONS\ * OPENAI\ * PYTHON\ \ Learn more](https://docs.temporal.io/ai-cookbook/hello-world-openai-responses-python) [Structured Outputs with Temporal and OpenAI\ -------------------------------------------\ \ Use Temporal and OpenAI Responses API to reliably request output conforming to a specific data structure.\ \ * FOUNDATIONS\ * OPENAI\ * PYTHON\ \ Learn more](https://docs.temporal.io/ai-cookbook/structured-output-openai-responses-python) [Hello World with LiteLLM\ ------------------------\ \ Integrate LiteLLM into a Temporal Workflow in Python.\ \ * PYTHON\ * LITELLM\ * PROVIDER-NEUTRAL\ \ Learn more](https://docs.temporal.io/ai-cookbook/hello-world-litellm-python) [Retry Policy from HTTP Responses\ --------------------------------\ \ Extract retry information from HTTP response headers and make it available to Temporal's retry mechanisms.\ \ * FOUNDATIONS\ * OPENAI\ * PYTHON\ \ Learn more](https://docs.temporal.io/ai-cookbook/http-retry-enhancement-python) [Basic Agentic Loop with Claude and Tool Calling\ -----------------------------------------------\ \ A basic agentic loop using Claude (Anthropic) with tool calling.\ \ * AGENTS\ * PYTHON\ * CLAUDE\ \ Learn more](https://docs.temporal.io/ai-cookbook/agentic-loop-tool-call-claude-python) [Basic Agentic Loop with Tool Calling\ ------------------------------------\ \ A basic agentic loop that invokes a dynamic set of tools.\ \ * AGENTS\ * PYTHON\ \ Learn more](https://docs.temporal.io/ai-cookbook/agentic-loop-tool-call-openai-python) [Durable MCP Weather Server\ --------------------------\ \ A durable MCP server that uses Temporal workflows for reliable execution of weather tools.\ \ * MCP\ * PYTHON\ * WORKFLOWS\ \ Learn more](https://docs.temporal.io/ai-cookbook/hello-world-durable-mcp-server) [Tool calling agent\ ------------------\ \ Build a simple, non-looping agent that gives agency to the LLM to choose tools, and then invokes chosen tools.\ \ * AGENTS\ * PYTHON\ \ Learn more](https://docs.temporal.io/ai-cookbook/tool-call-openai-python) [Human-in-the-Loop AI Agent\ --------------------------\ \ Support human in the loop (HITL) in agentic flows.\ \ * AGENTS\ * PYTHON\ \ Learn more](https://docs.temporal.io/ai-cookbook/human-in-the-loop-python) [Durable Agent with Tools - OpenAI Agents SDK\ --------------------------------------------\ \ Build a durable AI agent with OpenAI Agents SDK and Temporal that can intelligently choose tools to answer user questions\ \ * AGENTS\ * PYTHON\ * OPENAI\ \ Learn more](https://docs.temporal.io/ai-cookbook/openai-agents-sdk-python) [Claim Check Pattern with Temporal\ ---------------------------------\ \ Use the Claim Check pattern to handle large payloads to workflows and activities.\ \ * FOUNDATIONS\ * CLAIM-CHECK\ * PYTHON\ * S3\ \ Learn more](https://docs.temporal.io/ai-cookbook/claim-check-pattern-python) [Deep Research\ -------------\ \ Build a simple deep research system embodying the standard deep research architecture.\ \ * AGENTS\ * TOOLCALLING\ * PYTHON\ \ Learn more](https://docs.temporal.io/ai-cookbook/basic-openai-python) --- # Search | Temporal Platform Documentation [Skip to main content](https://docs.temporal.io/search#__docusaurus_skipToContent_fallback) Search results ============== --- # Tags | Temporal Platform Documentation [Skip to main content](https://docs.temporal.io/ai-cookbook/tags#__docusaurus_skipToContent_fallback) Tags ==== A[​](https://docs.temporal.io/ai-cookbook/tags#A "Direct link to A") --------------------------------------------------------------------- * [agents6](https://docs.temporal.io/ai-cookbook/tags/agents) * * * C[​](https://docs.temporal.io/ai-cookbook/tags#C "Direct link to C") --------------------------------------------------------------------- * [claim-check1](https://docs.temporal.io/ai-cookbook/tags/claim-check) * [claude1](https://docs.temporal.io/ai-cookbook/tags/claude) * * * F[​](https://docs.temporal.io/ai-cookbook/tags#F "Direct link to F") --------------------------------------------------------------------- * [foundations4](https://docs.temporal.io/ai-cookbook/tags/foundations) * * * L[​](https://docs.temporal.io/ai-cookbook/tags#L "Direct link to L") --------------------------------------------------------------------- * [litellm1](https://docs.temporal.io/ai-cookbook/tags/litellm) * * * M[​](https://docs.temporal.io/ai-cookbook/tags#M "Direct link to M") --------------------------------------------------------------------- * [mcp1](https://docs.temporal.io/ai-cookbook/tags/mcp) * * * O[​](https://docs.temporal.io/ai-cookbook/tags#O "Direct link to O") --------------------------------------------------------------------- * [openai4](https://docs.temporal.io/ai-cookbook/tags/openai) * * * P[​](https://docs.temporal.io/ai-cookbook/tags#P "Direct link to P") --------------------------------------------------------------------- * [provider-neutral1](https://docs.temporal.io/ai-cookbook/tags/provider-neutral) * [python12](https://docs.temporal.io/ai-cookbook/tags/python) * * * S[​](https://docs.temporal.io/ai-cookbook/tags#S "Direct link to S") --------------------------------------------------------------------- * [s31](https://docs.temporal.io/ai-cookbook/tags/s-3) * * * T[​](https://docs.temporal.io/ai-cookbook/tags#T "Direct link to T") --------------------------------------------------------------------- * [toolcalling1](https://docs.temporal.io/ai-cookbook/tags/toolcalling) * * * W[​](https://docs.temporal.io/ai-cookbook/tags#W "Direct link to W") --------------------------------------------------------------------- * [workflows1](https://docs.temporal.io/ai-cookbook/tags/workflows) * * * --- # One doc tagged with "claim-check" | Temporal Platform Documentation [Skip to main content](https://docs.temporal.io/ai-cookbook/tags/claim-check#__docusaurus_skipToContent_fallback) [Claim Check Pattern with Temporal\ ---------------------------------](https://docs.temporal.io/ai-cookbook/claim-check-pattern-python) Use the Claim Check pattern to handle large payloads to workflows and activities. --- # 6 docs tagged with "agents" | Temporal Platform Documentation [Skip to main content](https://docs.temporal.io/ai-cookbook/tags/agents#__docusaurus_skipToContent_fallback) [Basic Agentic Loop with Claude and Tool Calling\ -----------------------------------------------](https://docs.temporal.io/ai-cookbook/agentic-loop-tool-call-claude-python) A basic agentic loop using Claude (Anthropic) with tool calling. [Basic Agentic Loop with Tool Calling\ ------------------------------------](https://docs.temporal.io/ai-cookbook/agentic-loop-tool-call-openai-python) A basic agentic loop that invokes a dynamic set of tools. [Deep Research\ -------------](https://docs.temporal.io/ai-cookbook/basic-openai-python) Build a simple deep research system embodying the standard deep research architecture. [Durable Agent with Tools - OpenAI Agents SDK\ --------------------------------------------](https://docs.temporal.io/ai-cookbook/openai-agents-sdk-python) Build a durable AI agent with OpenAI Agents SDK and Temporal that can intelligently choose tools to answer user questions [Human-in-the-Loop AI Agent\ --------------------------](https://docs.temporal.io/ai-cookbook/human-in-the-loop-python) Support human in the loop (HITL) in agentic flows. [Tool calling agent\ ------------------](https://docs.temporal.io/ai-cookbook/tool-call-openai-python) Build a simple, non-looping agent that gives agency to the LLM to choose tools, and then invokes chosen tools. --- # One doc tagged with "claude" | Temporal Platform Documentation [Skip to main content](https://docs.temporal.io/ai-cookbook/tags/claude#__docusaurus_skipToContent_fallback) [Basic Agentic Loop with Claude and Tool Calling\ -----------------------------------------------](https://docs.temporal.io/ai-cookbook/agentic-loop-tool-call-claude-python) A basic agentic loop using Claude (Anthropic) with tool calling. --- # One doc tagged with "litellm" | Temporal Platform Documentation [Skip to main content](https://docs.temporal.io/ai-cookbook/tags/litellm#__docusaurus_skipToContent_fallback) [Hello World with LiteLLM\ ------------------------](https://docs.temporal.io/ai-cookbook/hello-world-litellm-python) Integrate LiteLLM into a Temporal Workflow in Python. --- # One doc tagged with "mcp" | Temporal Platform Documentation [Skip to main content](https://docs.temporal.io/ai-cookbook/tags/mcp#__docusaurus_skipToContent_fallback) [Durable MCP Weather Server\ --------------------------](https://docs.temporal.io/ai-cookbook/hello-world-durable-mcp-server) A durable MCP server that uses Temporal workflows for reliable execution of weather tools. --- # 4 docs tagged with "foundations" | Temporal Platform Documentation [Skip to main content](https://docs.temporal.io/ai-cookbook/tags/foundations#__docusaurus_skipToContent_fallback) [Claim Check Pattern with Temporal\ ---------------------------------](https://docs.temporal.io/ai-cookbook/claim-check-pattern-python) Use the Claim Check pattern to handle large payloads to workflows and activities. [Hello World\ -----------](https://docs.temporal.io/ai-cookbook/hello-world-openai-responses-python) Simple example demonstrating how to call an LLM from Temporal using the OpenAI Python API library. [Retry Policy from HTTP Responses\ --------------------------------](https://docs.temporal.io/ai-cookbook/http-retry-enhancement-python) Extract retry information from HTTP response headers and make it available to Temporal's retry mechanisms. [Structured Outputs with Temporal and OpenAI\ -------------------------------------------](https://docs.temporal.io/ai-cookbook/structured-output-openai-responses-python) Use Temporal and OpenAI Responses API to reliably request output conforming to a specific data structure. --- # 4 docs tagged with "openai" | Temporal Platform Documentation [Skip to main content](https://docs.temporal.io/ai-cookbook/tags/openai#__docusaurus_skipToContent_fallback) [Durable Agent with Tools - OpenAI Agents SDK\ --------------------------------------------](https://docs.temporal.io/ai-cookbook/openai-agents-sdk-python) Build a durable AI agent with OpenAI Agents SDK and Temporal that can intelligently choose tools to answer user questions [Hello World\ -----------](https://docs.temporal.io/ai-cookbook/hello-world-openai-responses-python) Simple example demonstrating how to call an LLM from Temporal using the OpenAI Python API library. [Retry Policy from HTTP Responses\ --------------------------------](https://docs.temporal.io/ai-cookbook/http-retry-enhancement-python) Extract retry information from HTTP response headers and make it available to Temporal's retry mechanisms. [Structured Outputs with Temporal and OpenAI\ -------------------------------------------](https://docs.temporal.io/ai-cookbook/structured-output-openai-responses-python) Use Temporal and OpenAI Responses API to reliably request output conforming to a specific data structure. --- # One doc tagged with "s3" | Temporal Platform Documentation [Skip to main content](https://docs.temporal.io/ai-cookbook/tags/s-3#__docusaurus_skipToContent_fallback) [Claim Check Pattern with Temporal\ ---------------------------------](https://docs.temporal.io/ai-cookbook/claim-check-pattern-python) Use the Claim Check pattern to handle large payloads to workflows and activities. --- # One doc tagged with "toolcalling" | Temporal Platform Documentation [Skip to main content](https://docs.temporal.io/ai-cookbook/tags/toolcalling#__docusaurus_skipToContent_fallback) [Deep Research\ -------------](https://docs.temporal.io/ai-cookbook/basic-openai-python) Build a simple deep research system embodying the standard deep research architecture. --- # One doc tagged with "provider-neutral" | Temporal Platform Documentation [Skip to main content](https://docs.temporal.io/ai-cookbook/tags/provider-neutral#__docusaurus_skipToContent_fallback) [Hello World with LiteLLM\ ------------------------](https://docs.temporal.io/ai-cookbook/hello-world-litellm-python) Integrate LiteLLM into a Temporal Workflow in Python. --- # One doc tagged with "workflows" | Temporal Platform Documentation [Skip to main content](https://docs.temporal.io/ai-cookbook/tags/workflows#__docusaurus_skipToContent_fallback) [Durable MCP Weather Server\ --------------------------](https://docs.temporal.io/ai-cookbook/hello-world-durable-mcp-server) A durable MCP server that uses Temporal workflows for reliable execution of weather tools. --- # Deep Research [Skip to main content](https://docs.temporal.io/ai-cookbook/basic-openai-python#__docusaurus_skipToContent_fallback) On this page Deep research systems combine multiple agents with information retrieval from the web or other sources to produce evidence-based reports on specific topics. Commercial implementations include [Anthropic Research](https://www.anthropic.com/engineering/multi-agent-research-system) , [OpenAI Deep Research](https://openai.com/index/introducing-deep-research/) , and [Google Gemini Deep Research](https://gemini.google/overview/deep-research/) . This recipe demonstrates a simple deep research system embodying the standard deep research architecture. Deep research spans the following four phases: * **Planning**. Task decomposition and research strategy formulation. This involves identifying separate aspects of the research problem that can be worked on independently. * **Question Development/Query Generation**. Designing queries for each of the research questions. * **Web Exploration/Information Retrieval**. Searching the web to retrieve documents relevant to the research question. Extracting and summarizing relevant information. * **Report Generation/Synthesis**. Synthesizing findings into comprehensive, well-cited reports. Deep research tasks can involve dozens of searches and process hundreds of documents. This creates many possible failure modes that durable execution helps protect against. This recipe uses OpenAI's Responses API, which includes a tool for web search. It also uses OpenAI's [Structured Outputs API](https://platform.openai.com/docs/guides/structured-outputs) , which asks the model to generate outputs corresponding to desired data structures. Create the data structures[​](https://docs.temporal.io/ai-cookbook/basic-openai-python#create-the-data-structures "Direct link to Create the data structures") --------------------------------------------------------------------------------------------------------------------------------------------------------------- We will use Python classes to ensure information passes between agents in a structured way. The Planning Agent creates a `ResearchPlan`, which includes a research question, a list of `ResearchAspects`, expected sources, a search strategy, and success criteria. ResearchAspects include an aspect name, a priority, and a description. class ResearchPlan(BaseModel): research_question: str key_aspects: List[ResearchAspect] expected_sources: List[str] search_strategy: str success_criteria: List[str] class ResearchAspect(BaseModel): aspect: str priority: int description: str The Query Generation Agent creates a `QueryPlan`, and generates a list of `SearchQueries`. class QueryPlan(BaseModel): queries: List[SearchQuery] class SearchQuery(BaseModel): query: str rationale: str expected_info_type: str priority: int The Web Search Agent creates a `SearchResult`, which includes a query, a list of sources, a key finding, a relevance score, and a list of citations. class SearchResult(BaseModel): query: str sources: List[str] key_findings: str relevance_score: float citations: List[str] Finally, the Report Synthesis Agent creates a `ResearchReport`, which includes an executive summary, a detailed analysis, a list of key findings, a confidence assessment, a list of citations, and a list of follow-up questions. class ResearchReport(BaseModel): executive_summary: str detailed_analysis: str key_findings: List[str] confidence_assessment: str citations: List[str] follow_up_questions: List[str] Create the Agents[​](https://docs.temporal.io/ai-cookbook/basic-openai-python#create-the-agents "Direct link to Create the Agents") ------------------------------------------------------------------------------------------------------------------------------------ The deep research system uses four specialized agents, each implemented as Temporal activities. In this implementation, each agent is implemented as a single call to the OpenAI Responses API. This is possible because we are using structured outputs, which guarantee the response will be in the correct format, eliminating the need for retries. The web search agent also requires only a single API call because OpenAI integrates the web search tool into the Responses API. These agents run in the Workflow and use the `invoke_model` activity to make OpenAI API calls. It is critical to set the `start_to_close_timeout` for these activities to a value that is long enough to complete the task. If it is too short, the activity will fail with a timeout error, causing a retry loop that never completes. Response times for reasoning models such as `GPT-5` can vary significantly depending on the nature of the request. Web search times also vary depending on the size and content of the documents located by the search. ### Research Planning Agent[​](https://docs.temporal.io/ai-cookbook/basic-openai-python#research-planning-agent "Direct link to Research Planning Agent") Analyzes research queries and creates comprehensive research strategies. Takes an unstructured question and decomposes it into specific research aspects with priorities, identifies expected source types, and defines success criteria. _File: agents/research\_planning.py_ from .models import ResearchPlanfrom .config import COMPLEX_REASONING_MODELfrom activities.invoke_model import invoke_model, InvokeModelRequestfrom temporalio import workflowfrom datetime import timedeltaRESEARCH_PLANNING_INSTRUCTIONS = """You are a research planning specialist who creates focused research strategies.CORE RESPONSIBILITIES:1. Decompose the user's question into 3-7 key research aspects2. Identify required sources and evidence types3. Design a practical search strategy4. Set clear success criteriaOUTPUT REQUIREMENTS:- research_question: Clarified version of the original query- key_aspects: Specific areas requiring investigation, each with: - aspect: The research area name - priority: 1-5 ranking (5 highest priority) - description: What needs to be investigated- expected_sources: Types of sources likely to contain relevant information- search_strategy: High-level approach for information gathering- success_criteria: Specific indicators of research completeness"""async def plan_research(query: str) -> ResearchPlan: result = await workflow.execute_activity( invoke_model, InvokeModelRequest( model=COMPLEX_REASONING_MODEL, instructions=RESEARCH_PLANNING_INSTRUCTIONS, input=f"Research query: {query}", response_format=ResearchPlan, ), start_to_close_timeout=timedelta(seconds=300), summary="Planning research", ) return result.response ### Query Generation Agent[​](https://docs.temporal.io/ai-cookbook/basic-openai-python#query-generation-agent "Direct link to Query Generation Agent") Converts research plans into optimized web search queries. Creates 3-5 diverse queries that target different information types (factual data, expert analysis, case studies, recent news) with varied search styles and temporal modifiers. _File: agents/research\_query\_generation.py_ from .models import QueryPlan, ResearchPlanfrom .config import EFFICIENT_PROCESSING_MODELfrom activities.invoke_model import invoke_model, InvokeModelRequestfrom temporalio import workflowfrom datetime import timedeltaQUERY_GENERATION_INSTRUCTIONS = """You are a search query specialist who crafts effective web searches.CORE RESPONSIBILITIES:1. Generate 3-5 diverse search queries based on the research plan2. Balance specificity with discoverability3. Target different information types (factual, analytical, recent, historical)APPROACH:- Vary query styles: direct questions, topic + keywords, source-specific searches- Include temporal modifiers when relevant (recent, 2024, historical)- Use domain-specific terminology appropriatelyOUTPUT REQUIREMENTS:- queries: Search queries, each with: - query: The actual search string - rationale: Why this query addresses research needs - expected_info_type: One of "factual_data", "expert_analysis", "case_studies", "recent_news" - priority: 1-5 (5 highest priority)"""async def generate_queries(research_plan: ResearchPlan) -> QueryPlan: # Prepare input with research plan context plan_context = f"""Research Question: {research_plan.research_question}Key Aspects to Research:{chr(10).join([f"- {aspect.aspect} (Priority: {aspect.priority}): {aspect.description}" for aspect in research_plan.key_aspects])}Expected Sources: {", ".join(research_plan.expected_sources)}Search Strategy: {research_plan.search_strategy}Success Criteria: {", ".join(research_plan.success_criteria)}""" result = await workflow.execute_activity( invoke_model, InvokeModelRequest( model=EFFICIENT_PROCESSING_MODEL, instructions=QUERY_GENERATION_INSTRUCTIONS, input=plan_context, response_format=QueryPlan, ), start_to_close_timeout=timedelta(seconds=300), summary="Generating search queries", ) return result.response ### Web Search Agent[​](https://docs.temporal.io/ai-cookbook/basic-openai-python#web-search-agent "Direct link to Web Search Agent") Executes searches using OpenAI's web search tool and analyzes results. Prioritizes authoritative sources, extracts key findings, assesses relevance, and provides proper citations with reliability assessments. _File: agents/research\_web\_search.py_ from .models import SearchResult, SearchQueryfrom .config import EFFICIENT_PROCESSING_MODELfrom activities.invoke_model import invoke_model, InvokeModelRequestfrom temporalio import workflowfrom datetime import timedeltaWEB_SEARCH_INSTRUCTIONS = """You are a web research specialist who finds and evaluates information from web sources.CORE RESPONSIBILITIES:1. Execute web searches using the web search tool2. Prioritize authoritative sources: academic, government, established research organizations, prominent news outlets, primary sources3. Extract key information relevant to the research question4. Provide proper citations and assess reliabilityAPPROACH:- Focus on information directly relevant to the research question- Extract specific facts, data points, and evidence- Note conflicting information and limitations- Flag questionable or unverified claimsOUTPUT REQUIREMENTS:- query: The search query that was executed- sources: URLs and source descriptions consulted- key_findings: Synthesized information relevant to research question (2-4 paragraphs)- relevance_score: 0.0-1.0 assessment of how well results address the query- citations: Formatted sources with URLs"""async def search_web(query: SearchQuery) -> SearchResult: search_input = f"""Search Query: {query.query}Query Rationale: {query.rationale}Expected Information Type: {query.expected_info_type}Priority Level: {query.priority}Please search for information using the provided query and analyze the results according to the instructions.""" result = await workflow.execute_activity( invoke_model, InvokeModelRequest( model=EFFICIENT_PROCESSING_MODEL, instructions=WEB_SEARCH_INSTRUCTIONS, input=search_input, response_format=SearchResult, tools=[{"type": "web_search"}], ), start_to_close_timeout=timedelta(seconds=300), summary="Searching web for information", ) return result.response ### Report Synthesis Agent[​](https://docs.temporal.io/ai-cookbook/basic-openai-python#report-synthesis-agent "Direct link to Report Synthesis Agent") Directs the agent to synthesize all research findings into comprehensive, well-cited reports. These should include structured narratives with executive summaries, detailed analysis, key findings, confidence assessments, and follow-up research questions. _File: agents/research\_report\_synthesis.py_ from typing import Listfrom temporalio import workflowfrom datetime import timedeltafrom .models import ResearchReport, ResearchPlan, SearchResultfrom .config import COMPLEX_REASONING_MODELfrom activities.invoke_model import invoke_model, InvokeModelRequestREPORT_SYNTHESIS_INSTRUCTIONS = """You are a research synthesis expert who creates comprehensive research reports.CORE RESPONSIBILITIES:1. Synthesize all research into a coherent narrative2. Structure information logically with evidence support3. Provide comprehensive citations4. Assess confidence levels and acknowledge limitations5. Generate follow-up questions for deeper researchREPORT STRUCTURE:1. **Executive Summary**: Core findings and conclusions (1-2 paragraphs)2. **Detailed Analysis**: Examination organized by themes with evidence3. **Key Findings**: Bullet-point list of important discoveries4. **Confidence Assessment**: Rate findings as High/Medium/Low/Uncertain5. **Citations**: Complete source list with URLs6. **Follow-up Questions**: Up to 5 areas for additional research, as warrantedAPPROACH:- Address contradictory findings transparently- Weight authoritative sources more heavily- Distinguish facts from expert opinions- Be explicit about information limitationsOUTPUT REQUIREMENTS:- executive_summary: 1-2 paragraph summary of core findings- detailed_analysis: Multi-paragraph analysis organized by themes- key_findings: Bullet-point discoveries- confidence_assessment: Assessment of finding reliability- citations: All sources referenced- follow_up_questions: 3-5 specific questions for further research"""async def generate_synthesis( original_query: str, research_plan: ResearchPlan, search_results: List[SearchResult]) -> ResearchReport: # Prepare comprehensive input with all research context synthesis_input = f"""ORIGINAL RESEARCH QUERY: {original_query}RESEARCH PLAN:Research Question: {research_plan.research_question}Key Aspects Investigated: { ", ".join([aspect.aspect for aspect in research_plan.key_aspects]) }Search Strategy Used: {research_plan.search_strategy}Success Criteria: {", ".join(research_plan.success_criteria)}SEARCH RESULTS TO SYNTHESIZE:{ chr(10).join( [ f"Query: {result.query}{chr(10)}Findings: {result.key_findings}{chr(10)}Relevance: {result.relevance_score}{chr(10)}Sources: {', '.join(result.sources)}{chr(10)}Citations: {', '.join(result.citations)}{chr(10)}" for result in search_results ] ) }Please synthesize all this information into a comprehensive research report following the specified structure and quality standards.""" result = await workflow.execute_activity( invoke_model, InvokeModelRequest( model=COMPLEX_REASONING_MODEL, instructions=REPORT_SYNTHESIS_INSTRUCTIONS, input=synthesis_input, response_format=ResearchReport, ), start_to_close_timeout=timedelta(seconds=300), summary="Generating research report synthesis", ) return result.response Create the Workflow[​](https://docs.temporal.io/ai-cookbook/basic-openai-python#create-the-workflow "Direct link to Create the Workflow") ------------------------------------------------------------------------------------------------------------------------------------------ The `DeepResearchWorkflow` orchestrates the four-phase research process with built-in resilience and error handling: First, planning and query generation agents are run sequentially. Then, the workflow executes searches concurrently. For robustness, the workflow continues with partial results if some searches fail. Finally, the report synthesis agent pulls together the findings into a comprehensive report. _File: workflows/deep\_research\_workflow.py_ from temporalio import workflowfrom temporalio.exceptions import ApplicationErrorimport asynciofrom typing import Listfrom agents.research_planning import plan_researchfrom agents.research_query_generation import generate_queriesfrom agents.research_web_search import search_webfrom agents.research_report_synthesis import generate_synthesisfrom agents.models import SearchResult@workflow.defnclass DeepResearchWorkflow: @workflow.run async def run(self, query: str) -> str: # Step 1: Research Planning research_plan = await plan_research(query) # Step 2: Query Generation query_plan = await generate_queries(research_plan) # Step 3: Web Search (parallel execution with resilience) search_results = await self._execute_searches(query_plan.queries) # Ensure we have at least one successful search result if not search_results: raise ApplicationError( "All web searches failed - cannot generate report", "NO_SEARCH_RESULTS", non_retryable=True, ) # Step 4: Report Synthesis final_report = await generate_synthesis(query, research_plan, search_results) # Format the final output formatted_report = self._format_final_report(query, final_report) return formatted_report async def _execute_searches(self, search_queries) -> List[SearchResult]: """Execute web searches in parallel with resilience to individual failures""" # Create individual search coroutines async def execute_single_search(search_query): try: return await search_web(search_query) except Exception as e: workflow.logger.exception( f"Search failed for query '{search_query.query}': {e}" ) return None # Execute all searches in parallel search_tasks = [execute_single_search(query) for query in search_queries] results = await asyncio.gather(*search_tasks) # Filter out None results return [result for result in results if result is not None] def _format_final_report(self, original_query, report) -> str: """Format the final report for display""" return f"""# Deep Research Report**Research Query:** {original_query}## Executive Summary{report.executive_summary}## Detailed Analysis{report.detailed_analysis}## Key Findings{chr(10).join([f"• {finding}" for finding in report.key_findings])}## Confidence Assessment{report.confidence_assessment}## Sources and Citations{chr(10).join([f"• {citation}" for citation in report.citations])}## Recommended Follow-up Questions{chr(10).join([f"• {question}" for question in report.follow_up_questions])}""" Running[​](https://docs.temporal.io/ai-cookbook/basic-openai-python#running "Direct link to Running") ------------------------------------------------------------------------------------------------------ Start the Temporal Dev Server: temporal server start-dev Run the worker: uv run python -m worker Start execution: uv run python -m start_workflow To start execution with a specific query: uv run python -m start_workflow "What is the latest news on the stock market?" --- # Basic Agentic Loop with Claude and Tool Calling [Skip to main content](https://docs.temporal.io/ai-cookbook/agentic-loop-tool-call-claude-python#__docusaurus_skipToContent_fallback) On this page This example implements an agentic loop using Claude (Anthropic) that has a set of tools available. If the agent determines that no tools are needed to satisfy a user request, it will return the response directly. If Claude determines a tool should be used, it will return with the name of the chosen tool and any needed parameters. The agent then invokes the appropriate tool. Tools are supplied to Claude's Messages API through the `tools` parameter. The `tools` parameter is in JSON format and includes a description of the function as well as descriptions of each of the arguments using Claude's `input_schema` format. Being external API calls, invoking Claude and invoking any functions/tools are done within a Temporal Activity. This recipe highlights the following key design decisions: * We use dynamic Activities to allow the agent to be loosely coupled from specific tools. This sample isolates the tools in the `tools` directory; changing the tools requires NO changes to the agent implementation. * Because there is an agentic loop, each Claude invocation is passed the accumulated _conversation history_ in a structured messages array with role alternation (user/assistant). * Claude can return multiple tool calls in a single response, and can mix text with tool calls in the same response. * A generic Activity for invoking Claude's Messages API; instructions and other parameters are passed into the Activity making it appropriate for use in a variety of different use cases. * Retries are handled by Temporal and not by the Anthropic client library. This is important because client retries can interfere with correct and durable error handling and recovery. Also see this foundational [recipe for basic tool calling](https://docs.temporal.io/ai-cookbook/tool-call-openai-python) . Application Components[​](https://docs.temporal.io/ai-cookbook/agentic-loop-tool-call-claude-python#application-components "Direct link to Application Components") -------------------------------------------------------------------------------------------------------------------------------------------------------------------- This example includes the following components: * The [Workflow](https://docs.temporal.io/ai-cookbook/agentic-loop-tool-call-claude-python#create-the-agent-agentic-loop) that contains the agentic loop and tool calling logic; this is the core of the agent implementation. * The Activities for [invoking Claude](https://docs.temporal.io/ai-cookbook/agentic-loop-tool-call-claude-python#create-the-activity-for-claude-invocations) and for [invoking tools](https://docs.temporal.io/ai-cookbook/agentic-loop-tool-call-claude-python#create-the-activity-for-the-tool-invocation) . * A [helper function](https://docs.temporal.io/ai-cookbook/agentic-loop-tool-call-claude-python#create-the-helper-function) that creates tool definitions in Claude's format. * Sample [tools](https://docs.temporal.io/ai-cookbook/agentic-loop-tool-call-claude-python#create-tool-definitions) . * The [Worker](https://docs.temporal.io/ai-cookbook/agentic-loop-tool-call-claude-python#create-the-worker) that manages the Workflow and the Activities. * An application that [initiates an interaction](https://docs.temporal.io/ai-cookbook/agentic-loop-tool-call-claude-python#initiate-an-interaction-with-the-agent) with the agent. Create the Agent (Agentic Loop)[​](https://docs.temporal.io/ai-cookbook/agentic-loop-tool-call-claude-python#create-the-agent-agentic-loop "Direct link to Create the Agent (Agentic Loop)") --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- ### Create the main agentic loop[​](https://docs.temporal.io/ai-cookbook/agentic-loop-tool-call-claude-python#create-the-main-agentic-loop "Direct link to Create the main agentic loop") The agent is implemented as a Temporal Workflow that implements an agentic loop. The loop will continue until the agent responds with no tool calls. Each time through the loop: * Claude is called with the accumulated conversation history that is made up of the initial user input and any previous assistant responses and tool outputs. * The Workflow checks if Claude returned any tool calls (content blocks with `type: "tool_use"`). * If tool calls are present, the assistant's complete response (including all content blocks) is appended to the messages array, then all tools are executed, and their results are added as a user message. * If no tool has been called, the text response is returned. _File: workflows/agent.py_ from temporalio import workflowfrom datetime import timedeltaimport jsonwith workflow.unsafe.imports_passed_through(): from tools import get_tools from helpers import tool_helpers from activities import claude_responses@workflow.defnclass AgentWorkflow: @workflow.run async def run(self, input: str) -> str: # Initialize messages list with user input messages = [{"role": "user", "content": input}] # The agentic loop while True: print(80 * "=") # Consult Claude result = await workflow.execute_activity( claude_responses.create, claude_responses.ClaudeResponsesRequest( model="claude-sonnet-4-20250514", system=tool_helpers.HELPFUL_AGENT_SYSTEM_INSTRUCTIONS, messages=messages, tools=get_tools(), max_tokens=4096, ), start_to_close_timeout=timedelta(seconds=30), ) # Claude returns content blocks - check if any are tool_use tool_use_blocks = [block for block in result.content if block.type == "tool_use"] if tool_use_blocks: # We have tool calls to handle # First, add the assistant's response to messages # Convert content blocks to dictionaries for serialization assistant_content = [] for block in result.content: if block.type == "text": assistant_content.append({"type": "text", "text": block.text}) elif block.type == "tool_use": assistant_content.append({ "type": "tool_use", "id": block.id, "name": block.name, "input": block.input }) messages.append({"role": "assistant", "content": assistant_content}) # Execute all tool calls and collect results tool_results = [] for block in tool_use_blocks: print(f"[Agent] Tool call: {block.name}({block.input})") # Execute the tool tool_result = await self._execute_tool(block.name, block.input) print(f"[Agent] Tool result: {tool_result}") # Add tool result in Claude's expected format tool_results.append({ "type": "tool_result", "tool_use_id": block.id, "content": str(tool_result) }) # Add tool results as a user message messages.append({"role": "user", "content": tool_results}) else: # No tool calls - extract the text response and return text_blocks = [block for block in result.content if block.type == "text"] if text_blocks: response_text = text_blocks[0].text print(f"[Agent] Final response: {response_text}") return response_text else: return "No text response from Claude" ### Create the tool execution handler[​](https://docs.temporal.io/ai-cookbook/agentic-loop-tool-call-claude-python#create-the-tool-execution-handler "Direct link to Create the tool execution handler") The tool execution handler is invoked by the main agentic loop when Claude has chosen tools. Because the Activity implementation is dynamic, the arguments are passed to the Activity as a dictionary. The Activity invocation is the same as any non-dynamic Activity invocation, passing the name of the Activity, the arguments, and any Activity configurations. _File: workflows/agent.py_ async def _execute_tool(self, tool_name: str, tool_input: dict) -> str: """ Execute a tool dynamically. Args: tool_name: Name of the tool to execute tool_input: Dictionary of input parameters """ # Execute dynamic Activity with the tool name and arguments result = await workflow.execute_activity( tool_name, tool_input, start_to_close_timeout=timedelta(seconds=30), ) return result Create the Activity for Claude invocations[​](https://docs.temporal.io/ai-cookbook/agentic-loop-tool-call-claude-python#create-the-activity-for-claude-invocations "Direct link to Create the Activity for Claude invocations") -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- We create a wrapper for the `create` method of the `AsyncAnthropic` client object. This is a generic Activity that invokes Claude's Messages API. We set `max_retries=0` when creating the `AsyncAnthropic` client. This moves the responsibility for retries from the Anthropic client to Temporal. This means that the Activity should interpret any errors coming from Claude's API call and return the appropriate error type so that the Workflow knows if it should retry the Activity or not. In this implementation, we allow for the model, system instructions, messages, lis6t of tools, and max\_tokens (required) to be passed in. _File: activities/claude\_responses.py_ from temporalio import activityfrom anthropic import AsyncAnthropicfrom anthropic.types import Messagefrom dataclasses import dataclassfrom typing import Any# Temporal best practice: Create a data structure to hold the request parameters.@dataclassclass ClaudeResponsesRequest: model: str system: str messages: list[dict[str, Any]] tools: list[dict[str, Any]] max_tokens: int = 4096@activity.defnasync def create(request: ClaudeResponsesRequest) -> Message: # We disable retry logic in Anthropic API client library so that Temporal can handle retries. # In a real setting, you would need to handle any errors coming back from the Anthropic API, # so that Temporal can appropriately retry in the manner that Anthropic API would. client = AsyncAnthropic(max_retries=0) try: resp = await client.messages.create( model=request.model, system=request.system, messages=request.messages, tools=request.tools, max_tokens=request.max_tokens, ) return resp finally: await client.close() Create the Activity for the tool invocation[​](https://docs.temporal.io/ai-cookbook/agentic-loop-tool-call-claude-python#create-the-activity-for-the-tool-invocation "Direct link to Create the Activity for the tool invocation") ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Implement a single tool invocation Activity, as a dynamic Activity (note the `@activity.defn(dynamic=True)` annotation) that acts as a broker to the right tool function. The name of the Activity is drawn from the `activity.info()` and the property bag of arguments from the Activity payload. The `handler` is the function that maps to the `tool_name` (see [Create Tool Definitions](https://docs.temporal.io/ai-cookbook/agentic-loop-tool-call-claude-python#create-tool-definitions) for more details) and that function is then called with the supplied arguments. _File: activities/tool\_invoker.py_ from temporalio import activityfrom typing import Sequencefrom temporalio.common import RawValueimport inspectfrom pydantic import BaseModel# We use dynamic activities to allow the agent to be defined independently of the tools it can call.@activity.defn(dynamic=True)async def dynamic_tool_activity(args: Sequence[RawValue]) -> dict: from tools import get_handler # the name of the tool to execute - this is passed in via the execute_activity call in the Workflow tool_name = activity.info().activity_type tool_args = activity.payload_converter().from_payload(args[0].payload, dict) activity.logger.info(f"Running dynamic tool '{tool_name}' with args: {tool_args}") handler = get_handler(tool_name) # in dynamic activity sig = inspect.signature(handler) params = list(sig.parameters.values()) if len(params) == 0: call_args = [] else: ann = params[0].annotation if isinstance(tool_args, dict) and isinstance(ann, type) and issubclass(ann, BaseModel): call_args = [ann(**tool_args)] # or ann.model_validate(tool_args) on Pydantic v2 else: call_args = [tool_args] if not inspect.iscoroutinefunction(handler): raise TypeError("Tool handler must be async (awaitable).") result = await handler(*call_args) # Optionally log or augment the result activity.logger.info(f"Tool '{tool_name}' result: {result}") return result Create the helper function[​](https://docs.temporal.io/ai-cookbook/agentic-loop-tool-call-claude-python#create-the-helper-function "Direct link to Create the helper function") -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- The `claude_tool_from_model` function accepts a tool name and description, as well as a Pydantic model for the parameters, and returns JSON that is in the format expected for tool definitions in Claude's Messages API. _File: helpers/tool\_helpers.py_ from pydantic import BaseModelfrom typing import Anyimport jsondef claude_tool_from_model(name: str, description: str, model: type[BaseModel] | None) -> dict[str, Any]: """ Convert a Pydantic model to Claude's tool format. Claude's tool format structure: { "name": "tool_name", "description": "Tool description", "input_schema": { "type": "object", "properties": {...}, "required": [...] } } """ if model is None: # For tools without parameters return { "name": name, "description": description, "input_schema": { "type": "object", "properties": {}, "required": [] } } # Get the JSON schema from the Pydantic model schema = model.model_json_schema() # Claude expects an input_schema field return { "name": name, "description": description, "input_schema": { "type": "object", "properties": schema.get("properties", {}), "required": schema.get("required", []) } } This file also holds the system instruction for the agent. HELPFUL_AGENT_SYSTEM_INSTRUCTIONS = """You are a helpful agent that can use tools to help the user.You will be given input from the user and a list of tools to use.You may or may not need to use the tools to satisfy the user ask.If no tools are needed, respond in haikus.""" Create tool definitions[​](https://docs.temporal.io/ai-cookbook/agentic-loop-tool-call-claude-python#create-tool-definitions "Direct link to Create tool definitions") ----------------------------------------------------------------------------------------------------------------------------------------------------------------------- Tools are defined in the `tools` directory and should be thought of as independent from the agent implementation; as described above, dynamic Activities are leveraged for this loose coupling. The `__init__.py` file holds tools for providing location (`get_location_info`), IP address (`get_ip_address`), and weather alerts (`get_weather_alerts`). * The `get_tools` method returns the set of tool definitions that will be passed to Claude. * The `get_handler` method captures the mapping from tool name to tool function. _File: tools/**init**.py_ from typing import Any, Awaitable, Callable# Location and weather related toolsfrom .get_location import get_location_info, get_ip_addressfrom .get_weather import get_weather_alertsfrom . import get_weatherfrom . import get_locationToolHandler = Callable[..., Awaitable[Any]]def get_handler(tool_name: str) -> ToolHandler: if tool_name == "get_location_info": return get_location_info if tool_name == "get_ip_address": return get_ip_address if tool_name == "get_weather_alerts": return get_weather_alerts raise ValueError(f"Unknown tool name: {tool_name}")def get_tools() -> list[dict[str, Any]]: return [ get_weather.WEATHER_ALERTS_TOOL_CLAUDE, get_location.GET_LOCATION_TOOL_CLAUDE, get_location.GET_IP_ADDRESS_TOOL_CLAUDE ] The tool descriptions and functions are defined in `tools/get_location.py`, `tools/get_weather.py` and `tools/random_stuff.py` files. Each of these files contains: * data structures for function arguments * tool definitions (in JSON form using Claude's `input_schema` format) * the function definitions. `tools/get_location.py` # get_location.pyfrom typing import Anyimport httpxfrom pydantic import BaseModel, Fieldfrom helpers import tool_helpers# For the location finder we use Pydantic to create a structure that encapsulates the input parameter # (an IP address). # This is used for both the location finding function and to craft the tool definitions that # are passed to Claude.class GetLocationRequest(BaseModel): ipaddress: str = Field(description="An IP address")# Build the tool definitions for ClaudeGET_LOCATION_TOOL_CLAUDE: dict[str, Any] = tool_helpers.claude_tool_from_model( "get_location_info", "Get the location information for an IP address. This includes the city, state, and country.", GetLocationRequest)GET_IP_ADDRESS_TOOL_CLAUDE: dict[str, Any] = tool_helpers.claude_tool_from_model( "get_ip_address", "Get the IP address of the current machine.", None)# The functionsasync def get_ip_address() -> str: async with httpx.AsyncClient() as client: response = await client.get("https://icanhazip.com") response.raise_for_status() return response.text.strip()async def get_location_info(req: GetLocationRequest) -> str: async with httpx.AsyncClient() as client: response = await client.get(f"http://ip-api.com/json/{req.ipaddress}") response.raise_for_status() result = response.json() return f"{result['city']}, {result['regionName']}, {result['country']}" Create the Worker[​](https://docs.temporal.io/ai-cookbook/agentic-loop-tool-call-claude-python#create-the-worker "Direct link to Create the Worker") ----------------------------------------------------------------------------------------------------------------------------------------------------- The worker is the process that dispatches work to the various parts of the agent implementation - the orchestrator and the Activities for Claude and tool invocations. _File: worker.py_ import asynciofrom temporalio.client import Clientfrom temporalio.worker import Workerfrom workflows.agent import AgentWorkflowfrom activities import claude_responses, tool_invokerfrom temporalio.contrib.pydantic import pydantic_data_converterfrom concurrent.futures import ThreadPoolExecutorasync def main(): client = await Client.connect( "localhost:7233", data_converter=pydantic_data_converter, ) worker = Worker( client, task_queue="tool-invoking-agent-claude-python-task-queue", workflows=[ AgentWorkflow, ], activities=[ claude_responses.create, tool_invoker.dynamic_tool_activity, ], activity_executor=ThreadPoolExecutor(max_workers=10), ) await worker.run()if __name__ == "__main__": asyncio.run(main()) Initiate an interaction with the agent[​](https://docs.temporal.io/ai-cookbook/agentic-loop-tool-call-claude-python#initiate-an-interaction-with-the-agent "Direct link to Initiate an interaction with the agent") -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- In order to interact with this simple AI agent, we create a Temporal client and execute a Workflow. _File: start\_workflow.py_ import asyncioimport sysimport uuidfrom temporalio.client import Clientfrom workflows.agent import AgentWorkflowfrom temporalio.contrib.pydantic import pydantic_data_converterasync def main(): client = await Client.connect( "localhost:7233", data_converter=pydantic_data_converter, ) query = sys.argv[1] if len(sys.argv) > 1 else "Tell me about recursion" # Submit the agent Workflow for execution result = await client.execute_workflow( AgentWorkflow.run, query, id=f"agentic-loop-claude-id-{uuid.uuid4()}", task_queue="tool-invoking-agent-claude-python-task-queue", ) print(f"Result: {result}")if __name__ == "__main__": asyncio.run(main()) Running the app[​](https://docs.temporal.io/ai-cookbook/agentic-loop-tool-call-claude-python#running-the-app "Direct link to Running the app") ----------------------------------------------------------------------------------------------------------------------------------------------- In the terminal where you run the agent Worker, set an Anthropic API key: export ANTHROPIC_API_KEY=sk-ant-... uv sync Start the agent Worker: uv run python -m worker Make request to the agent: uv run python -m start_workflow "are there any weather alerts for where I am?" Try a number of different user prompts: uv run python -m start_workflow "where am I?"uv run python -m start_workflow "what is my ip address?"uv run python -m start_workflow "tell me about recursion" --- # Basic Agentic Loop with Tool Calling [Skip to main content](https://docs.temporal.io/ai-cookbook/agentic-loop-tool-call-openai-python#__docusaurus_skipToContent_fallback) On this page This example implements a basic agentic loop that has a set of tools available. If the agent determines that no tools are needed to satisfy a user request, it will respond directly. If the LLM determines a tool should be used it will return with the name of the chosen tool and any needed parameters. The agent then invokes the appropriate tool. Tools are supplied to the [`responses` API](https://platform.openai.com/docs/api-reference/responses/create) through the [`tools` parameter](https://platform.openai.com/docs/api-reference/responses/create#responses-create-tools) . The `tools` parameter is in `json` format and includes a description of the function as well as descriptions of each of the arguments. caution The API used to generate the tools `json` is an internal function from the [Open AI API](https://github.com/openai/openai-python) and may therefore change in the future. There currently is no public API to generate the tool definition from a Pydantic model or a function signature. Being external API calls, invoking the LLM and invoking any functions/tools are done within a Temporal Activity. This recipe highlights the following key design decisions: * We use dynamic Activities to allow the agent to be loosely coupled from specific tools. This sample isolates the tools in the `tools` directory; changing the tools requires NO changes to the agent implementation. * Because there is an agentic loop, each LLM invocation is passed the accumulated _conversation history_, that includes the initial user input as well as LLM and tool calls. * A generic Activity for invoking an LLM API; that is, instructions and other `responses` arguments are passed into the Activity making it appropriate for use in a variety of different use cases. Similarly, the result from the responses API call is returned out of the Activity so that it is usable in a variety of different use cases. * Retries are handled by Temporal and not by the underlying libraries such as the OpenAI client. This is important because if you leave the client retries on they can interfere with correct and durable error handling and recovery. Also see this foundational [recipe for basic tool calling](https://docs.temporal.io/ai-cookbook/tool-call-openai-python) . Application Components[​](https://docs.temporal.io/ai-cookbook/agentic-loop-tool-call-openai-python#application-components "Direct link to Application Components") -------------------------------------------------------------------------------------------------------------------------------------------------------------------- This example includes the following components: * The [workflow](https://docs.temporal.io/ai-cookbook/agentic-loop-tool-call-openai-python#create-the-agent-agentic-loop) that contains the agentic loop and tool calling logic; this is the core of the agent implementation. * The activities for [invoking the LLM](https://docs.temporal.io/ai-cookbook/agentic-loop-tool-call-openai-python#create-the-activity-for-llm-invocations) and for [invoking tools](https://docs.temporal.io/ai-cookbook/agentic-loop-tool-call-openai-python#create-the-activity-for-the-tool-invocation) . * A [helper function](https://docs.temporal.io/ai-cookbook/agentic-loop-tool-call-openai-python#create-the-helper-function) that creates tool definitions of the appropriate form. * Sample [tools](https://docs.temporal.io/ai-cookbook/agentic-loop-tool-call-openai-python#create-tool-definitions) . * The [worker](https://docs.temporal.io/ai-cookbook/agentic-loop-tool-call-openai-python#create-the-worker) that manages the Workflow and the Activities. * An application that [initiates an interaction](https://docs.temporal.io/ai-cookbook/agentic-loop-tool-call-openai-python#initiate-an-interaction-with-the-agent) with the agent. Create the Agent (Agentic Loop)[​](https://docs.temporal.io/ai-cookbook/agentic-loop-tool-call-openai-python#create-the-agent-agentic-loop "Direct link to Create the Agent (Agentic Loop)") --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- ### Create the main agentic loop[​](https://docs.temporal.io/ai-cookbook/agentic-loop-tool-call-openai-python#create-the-main-agentic-loop "Direct link to Create the main agentic loop") The agent is implemented as a Temporal workflow that: * implements an agentic loop. The loop will continue until the agent responds with no tool calls. Each time through the loop: * the LLM is called with the accumulated conversation history that is made up of the initial user input and any previous LLM responses and tool outputs. * the invocation of the function, if the LLM has chosen one * if a function is called the function result is added to the conversation history * if no tool has been called, the LLM response is returned. This example demonstrates a most simple UX where the user provides single shot input. Note however that the agent is not single shot. _File: workflows/agent.py_ from temporalio import workflowfrom datetime import timedeltaimport jsonwith workflow.unsafe.imports_passed_through(): from tools import get_tools from helpers import tool_helpers from activities import openai_responses@workflow.defnclass AgentWorkflow: @workflow.run async def run(self, input: str) -> str: input_list = [{"type": "message", "role": "user", "content": input}] # The agentic loop while True: print(80 * "=") # consult the LLM result = await workflow.execute_activity( openai_responses.create, openai_responses.OpenAIResponsesRequest( model="gpt-4o-mini", instructions=tool_helpers.HELPFUL_AGENT_SYSTEM_INSTRUCTIONS, input=input_list, tools=get_tools(), ), start_to_close_timeout=timedelta(seconds=30), ) # For this simple example, we only have one item in the output list # Either the LLM will have chosen a single function call or it will # have chosen to respond with a message. item = result.output[0] # Now process the LLM output to either call a tool or respond with a message. # if the result is a tool call, call the tool if item.type == "function_call": result = await self._handle_function_call(item, result, input_list) # add the tool call result to the input list for context input_list.append({"type": "function_call_output", "call_id": item.call_id, "output": result}) # if the result is not a tool call we will just respond with a message else: print(f"No tools chosen, responding with a message: {result.output_text}") return result.output_text ### Create the function call handler[​](https://docs.temporal.io/ai-cookbook/agentic-loop-tool-call-openai-python#create-the-function-call-handler "Direct link to Create the function call handler") The function call handler is invoked by the main agentic loop when an LLM has chosen a tool. Because the activty implementation is dynamic, the arguments are passed to the Activity in a property bag; the `args` variable is appropriately set. Otherwise, the Activity invocation is the same as any non-dynamic Activity invocation passing the name of the Activity, the arguments and any Activity configurations. _File: workflows/agent.py_ async def _handle_function_call(self, item, result, input_list): # serialize the LLM output - the decision the LLM made to call a tool i = result.output[0] input_list += [ i.model_dump() if hasattr(i, "model_dump") else i ] # execute dynamic activity with the tool name chosen by the LLM # and the arguments crafted by the LLM args = json.loads(item.arguments) if isinstance(item.arguments, str) else item.arguments result = await workflow.execute_activity( item.name, args, start_to_close_timeout=timedelta(seconds=30), ) print(f"Made a tool call to {item.name}") return result Create the Activity for LLM invocations[​](https://docs.temporal.io/ai-cookbook/agentic-loop-tool-call-openai-python#create-the-activity-for-llm-invocations "Direct link to Create the Activity for LLM invocations") ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- We create a wrapper for the `create` method of the `AsyncOpenAI` client object. This is a generic Activity that invokes the OpenAI LLM. We set `max_retries=0` when creating the `AsyncOpenAI` client. This moves the responsibility for retries from the OpenAI client to Temporal. This means that the Activity should interpret any errors coming from the OpenAI API call and return the appropriate error type so that the workflow knows if it should retry the Activity or not. In this implementation, we allow for the model, instructions and input to be passed in, and also the list of tools. _File: activities/openai\_responses.py_ from temporalio import activityfrom openai import AsyncOpenAIfrom openai.types.responses import Responsefrom dataclasses import dataclassfrom typing import Any# Temporal best practice: Create a data structure to hold the request parameters.@dataclassclass OpenAIResponsesRequest: model: str instructions: str input: object tools: list[dict[str, Any]]@activity.defnasync def create(request: OpenAIResponsesRequest) -> Response: # We disable retry logic in OpenAI API client library so that Temporal can handle retries. # In a real setting, you would need to handle any errors coming back from the OpenAI API, # so that Temporal can appropriately retry in the manner that OpenAI API would. # See the `http_retry_enhancement_python` example for inspiration. client = AsyncOpenAI(max_retries=0) try: resp = await client.responses.create( model=request.model, instructions=request.instructions, input=request.input, tools=request.tools, timeout=30, ) return resp finally: await client.close() Create the Activity for the tool invocation[​](https://docs.temporal.io/ai-cookbook/agentic-loop-tool-call-openai-python#create-the-activity-for-the-tool-invocation "Direct link to Create the Activity for the tool invocation") ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Implement a single tool invocation Activity, as a dynamic Activity (note the `@activity.defn(dynamic=True)` annotation) that acts as a broker to the right tool function. The name of the Activity is drawn from the `activity.info()` and the property bag of arguments from the Activity payload. The `handler` is the function that maps to the `tool_name` (see [Create Tool Definitions](https://docs.temporal.io/ai-cookbook/agentic-loop-tool-call-openai-python#create-tool-definitions) for more details) and that function is then called with the supplied arguments. _File: activities/tool\_invoker.py_ from temporalio import activityfrom typing import Sequencefrom temporalio.common import RawValueimport inspectfrom pydantic import BaseModel# We use dynamic activities to allow the agent to be defined independently of the tools it can call.@activity.defn(dynamic=True)async def dynamic_tool_activity(args: Sequence[RawValue]) -> dict: from tools import get_handler # the name of the tool to execute - this is passed in via the execute_activity call in the workflow tool_name = activity.info().activity_type tool_args = activity.payload_converter().from_payload(args[0].payload, dict) activity.logger.info(f"Running dynamic tool '{tool_name}' with args: {tool_args}") handler = get_handler(tool_name) # in dynamic activity sig = inspect.signature(handler) params = list(sig.parameters.values()) if len(params) == 0: call_args = [] else: ann = params[0].annotation if isinstance(tool_args, dict) and isinstance(ann, type) and issubclass(ann, BaseModel): call_args = [ann(**tool_args)] # or ann.model_validate(tool_args) on Pydantic v2 else: call_args = [tool_args] if not inspect.iscoroutinefunction(handler): raise TypeError("Tool handler must be async (awaitable).") result = await handler(*call_args) # Optionally log or augment the result activity.logger.info(f"Tool '{tool_name}' result: {result}") return result Create the helper function[​](https://docs.temporal.io/ai-cookbook/agentic-loop-tool-call-openai-python#create-the-helper-function "Direct link to Create the helper function") -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- The `oai_responses_tool_from_model` function accepts a tool name and description, as well as a list of argument name/description pairs and returns json that is in the format expected for tool definitions in the OpenAI responses API. caution The API used to generate the tools json is an internal function from the [Open AI API](https://github.com/openai/openai-python) and may therefore change in the future. There currently is no public API to generate the tool definition from a Pydantic model or a function signature. _File: helpers/tool\_helpers.py_ from openai.lib._pydantic import to_strict_json_schema # private API; may change# there currently is no public API to generate the tool definition from a Pydantic model# or a function signature.from pydantic import BaseModeldef oai_responses_tool_from_model(name: str, description: str, model: type[BaseModel]): return { "type": "function", "name": name, "description": description, # OpenAI Responses strict tools require a JSON Schema object where # additionalProperties is explicitly false. For tools without # parameters, supply an empty object schema. "parameters": ( to_strict_json_schema(model) if model else {"type": "object", "properties": {}, "required": [], "additionalProperties": False} ), "strict": True, } This file also holds the system instruction for the agent. HELPFUL_AGENT_SYSTEM_INSTRUCTIONS = """You are a helpful agent that can use tools to help the user.You will be given a input from the user and a list of tools to use.You may or may not need to use the tools to satisfy the user ask.If no tools are needed, respond in haikus.""" Create tool definitions[​](https://docs.temporal.io/ai-cookbook/agentic-loop-tool-call-openai-python#create-tool-definitions "Direct link to Create tool definitions") ----------------------------------------------------------------------------------------------------------------------------------------------------------------------- Tools are defined in the `tools` directory and should be thought of as independent from the agent implementation; as described above, dynamic Activities are leveraged for this loose coupling. The `__init__.py` file holds two examples of tool sets, one providing location and weather tools, the other a simple random number generating tool; comment and uncomment sets you would like to include (or combine them by updating the `get_tools` and `get_handler` methods). * The `get_tools` method returns the set of tool definitions that will be passed to the LLM. * The `get_handler` method captures the mapping from tool name to tool function _File: tools/**init**.py_ # Uncomment and comment out the tools you want to usefrom typing import Any, Awaitable, Callable# Location and weather related toolsfrom .get_location import get_location_info, get_ip_addressfrom .get_weather import get_weather_alertsToolHandler = Callable[..., Awaitable[Any]]def get_handler(tool_name: str) -> ToolHandler: if tool_name == "get_location_info": return get_location_info if tool_name == "get_ip_address": return get_ip_address if tool_name == "get_weather_alerts": return get_weather_alerts raise ValueError(f"Unknown tool name: {tool_name}")def get_tools() -> list[dict[str, Any]]: return [get_weather.WEATHER_ALERTS_TOOL_OAI, get_location.GET_LOCATION_TOOL_OAI, get_location.GET_IP_ADDRESS_TOOL_OAI]# Random number tool# from .random_stuff import get_random_number, RANDOM_NUMBER_TOOL_OAI# def get_handler(tool_name: str) -> ToolHandler:# if tool_name == "get_random_number":# return get_random_number# raise ValueError(f"Unknown tool name: {tool_name}")# def get_tools() -> list[dict[str, Any]]:# return [RANDOM_NUMBER_TOOL_OAI] The tool descriptions and functions are defined in `tools/get_location.py`, `tools/get_weather.py` and `tools/random_stuff.py` files. Each of these files contains: * data structures for function arguments * tool definitions (in `json` form) * the function definitions. `tools/get_location.py` # get_location.pyfrom typing import Anyimport requestsfrom pydantic import BaseModel, Fieldfrom helpers import tool_helpers# For the location finder we use Pydantic to create a structure that encapsulates the input parameter # (an IP address). # This is used for both the location finding function and to craft the tool definitions that # are passed to the OpenAI Responses API.class GetLocationRequest(BaseModel): ipaddress: str = Field(description="An IP address")# Build the tool definitions for the OpenAI Responses API. GET_LOCATION_TOOL_OAI: dict[str, Any] = tool_helpers.oai_responses_tool_from_model( "get_location_info", "Get the location information for an IP address. This includes the city, state, and country.", GetLocationRequest)GET_IP_ADDRESS_TOOL_OAI: dict[str, Any] = tool_helpers.oai_responses_tool_from_model( "get_ip_address", "Get the IP address of the current machine.", None)# The functionsdef get_ip_address() -> str: response = requests.get("https://icanhazip.com") response.raise_for_status() return response.text.strip()def get_location_info(req: GetLocationRequest) -> str: response = requests.get(f"http://ip-api.com/json/{req.ipaddress}") response.raise_for_status() result = response.json() return f"{result['city']}, {result['regionName']}, {result['country']}" See files in github for more tool definitions. Create the Worker[​](https://docs.temporal.io/ai-cookbook/agentic-loop-tool-call-openai-python#create-the-worker "Direct link to Create the Worker") ----------------------------------------------------------------------------------------------------------------------------------------------------- The worker is the process that dispatches work to the various parts of the agent implementation - the orchestrator and the activities for the LLM and tool invocations. _File: worker.py_ import asynciofrom temporalio.client import Clientfrom temporalio.worker import Workerfrom workflows.agent import AgentWorkflowfrom activities import openai_responses, tool_invokerfrom temporalio.contrib.pydantic import pydantic_data_converterfrom concurrent.futures import ThreadPoolExecutorasync def main(): client = await Client.connect( "localhost:7233", data_converter=pydantic_data_converter, ) worker = Worker( client, task_queue="tool-invoking-agent-python-task-queue", workflows=[ AgentWorkflow, ], activities=[ openai_responses.create, tool_invoker.dynamic_tool_activity, ], activity_executor=ThreadPoolExecutor(max_workers=10), ) await worker.run()if __name__ == "__main__": asyncio.run(main()) Initiate an interaction with the agent[​](https://docs.temporal.io/ai-cookbook/agentic-loop-tool-call-openai-python#initiate-an-interaction-with-the-agent "Direct link to Initiate an interaction with the agent") -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- In order to interact with this simple AI agent, we create a Temporal client and execute a workflow. _File:start\_workflow.py_ import asyncioimport sysimport uuidfrom temporalio.client import Clientfrom workflows.agent import AgentWorkflowfrom temporalio.contrib.pydantic import pydantic_data_converterasync def main(): client = await Client.connect( "localhost:7233", data_converter=pydantic_data_converter, ) query = sys.argv[1] if len(sys.argv) > 1 else "Tell me about recursion" # Submit the the agent workflow for execution result = await client.execute_workflow( AgentWorkflow.run, query, id=f"agentic-loop-id-{uuid.uuid4()}", task_queue="tool-invoking-agent-python-task-queue", ) print(f"Result: {result}")if __name__ == "__main__": asyncio.run(main()) Running the app[​](https://docs.temporal.io/ai-cookbook/agentic-loop-tool-call-openai-python#running-the-app "Direct link to Running the app") ----------------------------------------------------------------------------------------------------------------------------------------------- In the terminal where you run the agent worker, set an OpenAI API key: export OPENAI_API_KEY=sk... uv sync Start the agent worker: uv run python -m worker Make request to the agent: uv run python -m start_workflow "are there any weather alerts for where I am?" Try a number of different user prompts: uv run python -m start_workflow "where am I?"uv run python -m start_workflow "what is my ip address?"uv run python -m start_workflow "can I please have a random number?" --- # Hello World [Skip to main content](https://docs.temporal.io/ai-cookbook/hello-world-openai-responses-python#__docusaurus_skipToContent_fallback) On this page This is a simple example showing how to call an LLM from Temporal using the [OpenAI Python API library](https://github.com/openai/openai-python) . Being an external API call, the LLM invocation happens in a Temporal Activity. This recipe highlights two key design decisions: * A generic Activity for invoking an LLM API. This Activity can be re-used with different arguments throughout your codebase. * Configuring the Temporal client with a `dataconverter` to allow serialization of Pydantic types. * Retries are handled by Temporal and not by the underlying libraries such as the OpenAI client. This is important because if you leave the client retries on they can interfere with correct and durable error handling and recovery. Create the Activity[​](https://docs.temporal.io/ai-cookbook/hello-world-openai-responses-python#create-the-activity "Direct link to Create the Activity") ---------------------------------------------------------------------------------------------------------------------------------------------------------- We create a wrapper for the `create` method of the `AsyncOpenAI` client object. This is a generic Activity that invokes the OpenAI LLM. We set `max_retries=0` when creating the `AsyncOpenAI` client. This moves the responsibility for retries from the OpenAI client to Temporal. In this implementation, we include only the `instructions` and `input` argument, but it could be extended to others. _File: activities/openai\_responses.py_ from temporalio import activityfrom openai import AsyncOpenAIfrom openai.types.responses import Responsefrom dataclasses import dataclass# Temporal best practice: Create a data structure to hold the request parameters.@dataclassclass OpenAIResponsesRequest: model: str instructions: str input: str@activity.defnasync def create(request: OpenAIResponsesRequest) -> Response: # Temporal best practice: Disable retry logic in OpenAI API client library. client = AsyncOpenAI(max_retries=0) resp = await client.responses.create( model=request.model, instructions=request.instructions, input=request.input, timeout=15, ) return resp Create the Workflow[​](https://docs.temporal.io/ai-cookbook/hello-world-openai-responses-python#create-the-workflow "Direct link to Create the Workflow") ---------------------------------------------------------------------------------------------------------------------------------------------------------- In this example, we take the user input and generate a response in haiku format, using the OpenAI Responses Activity. The Workflow returns `result.output_text` from the OpenAI `Response`. As per usual, the Activity retry configuration is set here in the Workflow. In this case, a retry policy is not specified so the default retry policy is used (exponential backoff with 1s initial interval, 2.0 backoff coefficient, max interval 100× initial, unlimited attempts, no non-retryable errors). _File: workflows/hello\_world\_workflow.py_ from temporalio import workflowfrom datetime import timedeltafrom activities import openai_responses@workflow.defnclass HelloWorld: @workflow.run async def run(self, input: str) -> str: system_instructions = "You only respond in haikus." result = await workflow.execute_activity( openai_responses.create, openai_responses.OpenAIResponsesRequest( model="gpt-4o-mini", instructions=system_instructions, input=input, ), start_to_close_timeout=timedelta(seconds=30), ) return result.output_text Create the Worker[​](https://docs.temporal.io/ai-cookbook/hello-world-openai-responses-python#create-the-worker "Direct link to Create the Worker") ---------------------------------------------------------------------------------------------------------------------------------------------------- Create the process for executing Activities and Workflows. We configure the Temporal client with `pydantic_data_converter` so Temporal can serialize/deserialize output of the OpenAI SDK. _File: worker.py_ import asynciofrom temporalio.client import Clientfrom temporalio.worker import Workerfrom workflows.hello_world_workflow import HelloWorldfrom activities import openai_responsesfrom temporalio.contrib.pydantic import pydantic_data_converterasync def main(): client = await Client.connect( "localhost:7233", data_converter=pydantic_data_converter, ) worker = Worker( client, task_queue="hello-world-python-task-queue", workflows=[ HelloWorld, ], activities=[ openai_responses.create, ], ) await worker.run()if __name__ == "__main__": asyncio.run(main()) Create the Workflow Starter[​](https://docs.temporal.io/ai-cookbook/hello-world-openai-responses-python#create-the-workflow-starter "Direct link to Create the Workflow Starter") ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- The starter script submits the workflow to Temporal for execution, then waits for the result and prints it out. It uses the `pydantic_data_converter` to match the Worker configuration. _File: start\_workflow.py_ import asynciofrom temporalio.client import Clientfrom workflows.hello_world_workflow import HelloWorldfrom temporalio.contrib.pydantic import pydantic_data_converterasync def main(): client = await Client.connect( "localhost:7233", data_converter=pydantic_data_converter, ) # Submit the Hello World workflow for execution result = await client.execute_workflow( HelloWorld.run, "Tell me about recursion in programming.", id="my-workflow-id", task_queue="hello-world-python-task-queue", ) print(f"Result: {result}")if __name__ == "__main__": asyncio.run(main()) Running[​](https://docs.temporal.io/ai-cookbook/hello-world-openai-responses-python#running "Direct link to Running") ---------------------------------------------------------------------------------------------------------------------- Start the Temporal Dev Server: temporal server start-dev Run the worker: uv run python -m worker Start execution: uv run python -m start_workflow --- # Hello World with LiteLLM [Skip to main content](https://docs.temporal.io/ai-cookbook/hello-world-litellm-python#__docusaurus_skipToContent_fallback) On this page [LiteLLM](https://github.com/BerriAI/litellm) is a library for calling LLMs from Python. It makes it easy to access, and switch between, many providers, including OpenAI, Anthropic, Google, and more. This recipe mirrors the [Basic Python recipe](https://docs.temporal.io/ai-cookbook/hello-world-openai-responses-python) , but swaps the OpenAI SDK for LiteLLM. The workflow still delegates LLM calls to an Activity, letting Temporal coordinate retries and durability, while LiteLLM forwards those calls to your configured provider. Key points: * A reusable Activity that wraps `litellm.acompletion` and keeps retries in Temporal. * The most common LiteLLM parameters are on `LiteLLMRequest` ensuring type checking and IDE completion. Others may be passed via the `extra_options` dictionary, which functions as `kwargs` for `litellm.acompletion`. * The Activity returns the full LiteLLM response for processing by the workflow. Create the Activity[​](https://docs.temporal.io/ai-cookbook/hello-world-litellm-python#create-the-activity "Direct link to Create the Activity") ------------------------------------------------------------------------------------------------------------------------------------------------- `activities/models.py` from dataclasses import dataclass, fieldfrom typing import Any, Dict, List, Optional, Type, Union@dataclassclass LiteLLMRequest: model: str messages: List[Dict[str, Any]] temperature: Optional[float] = None max_tokens: Optional[int] = None timeout: Optional[Union[float, int]] = None response_format: Optional[Union[dict, Type[Any]]] = None extra_options: Dict[str, Any] = field(default_factory=dict) def to_acompletion_kwargs(self) -> Dict[str, Any]: kwargs = { "model": self.model, "messages": self.messages, } optional_values = { "temperature": self.temperature, "max_tokens": self.max_tokens, "timeout": self.timeout, "response_format": self.response_format, } for key, value in optional_values.items(): if value is not None: kwargs[key] = value if self.extra_options: kwargs.update(self.extra_options) return kwargs `activities/litellm_completion.py` from typing import Any, Dictimport litellmfrom temporalio import activityfrom temporalio.exceptions import ApplicationErrorfrom activities.models import LiteLLMRequest@activity.defn(name="activities.litellm_completion.create")async def create(request: LiteLLMRequest) -> Dict[str, Any]: kwargs = request.to_acompletion_kwargs() kwargs["num_retries"] = 0 try: response = await litellm.acompletion(**kwargs) except ( litellm.AuthenticationError, litellm.BadRequestError, litellm.InvalidRequestError, litellm.UnsupportedParamsError, litellm.JSONSchemaValidationError, litellm.ContentPolicyViolationError, litellm.NotFoundError, ) as exc: raise ApplicationError( str(exc), type=exc.__class__.__name__, non_retryable=True, ) from exc except litellm.APIError: raise return response LiteLLM supports many providers. Configure credentials via environment variables (for example `OPENAI_API_KEY`) before running the Activity. For Google-hosted models (Vertex AI or Gemini), the sample relies on the `google-cloud-aiplatform` and `google-auth` dependencies included in `pyproject.toml`; set the usual Google application credentials (`GOOGLE_APPLICATION_CREDENTIALS`, `GOOGLE_CLOUD_PROJECT`, `VERTEXAI_LOCATION`, etc.) so LiteLLM can obtain an access token. Create the Workflow[​](https://docs.temporal.io/ai-cookbook/hello-world-litellm-python#create-the-workflow "Direct link to Create the Workflow") ------------------------------------------------------------------------------------------------------------------------------------------------- `workflows/hello_world_workflow.py` from datetime import timedeltafrom temporalio import workflowfrom activities.models import LiteLLMRequest@workflow.defnclass HelloWorld: @workflow.run async def run(self, input: str) -> str: messages = [ {"role": "system", "content": "You only respond in haikus."}, {"role": "user", "content": input}, ] response = await workflow.execute_activity( "activities.litellm_completion.create", LiteLLMRequest( # LiteLLM lets you keep the same code and swap models/providers. # model="gpt-4o-mini", model="gemini-2.5-flash-lite", messages=messages, ), start_to_close_timeout=timedelta(seconds=30), ) message = response["choices"][0]["message"]["content"] if isinstance(message, list): message = "".join( part.get("text", "") for part in message if isinstance(part, dict) ) return message Temporal manages Activity retries, so LiteLLM's retry helper is disabled via `num_retries=0`. Use the `extra_options` escape hatch on `LiteLLMRequest` if you need to surface additional LiteLLM parameters without editing the sample. Create the Worker[​](https://docs.temporal.io/ai-cookbook/hello-world-litellm-python#create-the-worker "Direct link to Create the Worker") ------------------------------------------------------------------------------------------------------------------------------------------- `worker.py` import asynciofrom temporalio.client import Clientfrom temporalio.worker import Workerfrom activities import litellm_completionfrom workflows.hello_world_workflow import HelloWorldfrom temporalio.contrib.pydantic import pydantic_data_converterasync def main(): client = await Client.connect( "localhost:7233", data_converter=pydantic_data_converter, ) worker = Worker( client, task_queue="hello-world-python-task-queue", workflows=[ HelloWorld, ], activities=[ litellm_completion.create, ], ) await worker.run()if __name__ == "__main__": asyncio.run(main()) Create the Workflow Starter[​](https://docs.temporal.io/ai-cookbook/hello-world-litellm-python#create-the-workflow-starter "Direct link to Create the Workflow Starter") ------------------------------------------------------------------------------------------------------------------------------------------------------------------------- `start_workflow.py` import asynciofrom temporalio.client import Clientfrom temporalio.contrib.pydantic import pydantic_data_converterfrom workflows.hello_world_workflow import HelloWorldasync def main(): client = await Client.connect( "localhost:7233", data_converter=pydantic_data_converter, ) result = await client.execute_workflow( HelloWorld.run, "Tell me about recursion in programming.", id="my-workflow-id", task_queue="hello-world-python-task-queue", ) print(f"Result: {result}")if __name__ == "__main__": asyncio.run(main()) Running[​](https://docs.temporal.io/ai-cookbook/hello-world-litellm-python#running "Direct link to Running") ------------------------------------------------------------------------------------------------------------- Start the Temporal Dev Server: temporal server start-dev Install dependencies uv sync Set the appropriate environment variables before launching the worker (for example `export OPENAI_API_KEY=...` or export `GEMINI_API_KEY=...`) so LiteLLM can reach your chosen provider. Run the worker: uv run python -m worker Start the workflow: uv run python -m start_workflow --- # Human-in-the-Loop AI Agent [Skip to main content](https://docs.temporal.io/ai-cookbook/human-in-the-loop-python#__docusaurus_skipToContent_fallback) On this page This example demonstrates how to build an AI agent that requires human approval; we use Temporal Signals to bring that user input into the agent. Overview[​](https://docs.temporal.io/ai-cookbook/human-in-the-loop-python#overview "Direct link to Overview") -------------------------------------------------------------------------------------------------------------- The workflow implements the agent flow: 1. Uses an LLM to analyze a user request and propose an action. 2. If the proposed action is deemed risky, pauses and waits for human approval via Temporal Signal 3. Executes the action if auto-approved (if not risky) or human approved, or cancels if rejected/timed out Key features: * **Resource efficient waiting**: Can wait for approval for hours, days or indefinitely; while waiting, the agent consumes no compute resources. * **Signal-based approval**: External systems send approval decisions via Temporal Signals * **Durable timers**: Time limits placed on human int he loop steps survive any execution distruptions. * **Complete audit trail**: All decisions are logged for compliance Prerequisites[​](https://docs.temporal.io/ai-cookbook/human-in-the-loop-python#prerequisites "Direct link to Prerequisites") ----------------------------------------------------------------------------------------------------------------------------- * Python 3.11+ * Temporal server running locally * OpenAI API key Setup[​](https://docs.temporal.io/ai-cookbook/human-in-the-loop-python#setup "Direct link to Setup") ----------------------------------------------------------------------------------------------------- 1. Install dependencies: uv sync 2. Set your OpenAI API key: export OPENAI_API_KEY='your-api-key-here' 3. Start Temporal Dev Server: temporal server start-dev Running[​](https://docs.temporal.io/ai-cookbook/human-in-the-loop-python#running "Direct link to Running") ----------------------------------------------------------------------------------------------------------- ### Start the Worker[​](https://docs.temporal.io/ai-cookbook/human-in-the-loop-python#start-the-worker "Direct link to Start the Worker") In one terminal: uv run python -m worker ### Start a Workflow[​](https://docs.temporal.io/ai-cookbook/human-in-the-loop-python#start-a-workflow "Direct link to Start a Workflow") In another terminal: uv run python -m start_workflow "Delete all test data from the production database" The workflow will start, analyze the request, and pause for approval. Watch the worker output for instructions. ### Send Approval Decision[​](https://docs.temporal.io/ai-cookbook/human-in-the-loop-python#send-approval-decision "Direct link to Send Approval Decision") The worker output will show the workflow ID and request ID. In another terminal, run the `send_approval` script to approve or reject: **To approve:** uv run python -m send_approval approve "Looks good" **To reject:** uv run python -m send_approval reject "Too risky" ### Testing Timeout[​](https://docs.temporal.io/ai-cookbook/human-in-the-loop-python#testing-timeout "Direct link to Testing Timeout") To test timeout behavior, simply don't send any approval signal. After 5 minutes (default), the workflow will automatically complete with a timeout result. Architecture[​](https://docs.temporal.io/ai-cookbook/human-in-the-loop-python#architecture "Direct link to Architecture") -------------------------------------------------------------------------------------------------------------------------- * **Models** (`models/models.py`): Data structures for workflow input, approval requests and decisions * **Activities**: * `openai_responses.py`: Generic LLM invocation activity * `execute_action.py`: Executes approved actions * The "execution" of approved actions in this sample simply logs messages. * In a realistic scenario, a set of tools will have been provided to the LLM and the result might be a recommended tool call. In this case, if approved, the agent would invoke the tool via an activity. See the [agentic loop with tool calling](https://docs.temporal.io/ai-cookbook/agentic-loop-tool-call-openai-python) for guidance on how to use dynamic activities, allowing the tools to be loosely coupled from the agent implementation. * `notify_approval_needed.py`: Notifies external systems of approval requests * In this sample the notification comes in the form of messages printed in the terminal running the worker. * In a realistic scenario, the notification activity may send emails, deliver messages to slack, etc. * **Workflow** (`workflows/human_in_the_loop_workflow.py`): Orchestrates the approval process * **Scripts**: * `worker.py`: Runs the Temporal worker * `start_workflow.py`: Starts workflow execution * `send_approval.py`: Helper script to send approval signals Key Patterns[​](https://docs.temporal.io/ai-cookbook/human-in-the-loop-python#key-patterns "Direct link to Key Patterns") -------------------------------------------------------------------------------------------------------------------------- We use a Temporal signal to inject information from the human into the waiting workflow. The signal is delivered from some UI (in this case the `send_approval.py` script) that uses a Temporal client to deliver the data. ![](https://docs.temporal.io/assets/images/human-in-the-loop-python-assets-temporal-signal-handling-45443be1e6412967d97091ddc500651c.png) Within the agent implementation there are three main elements to the solution. ### Local state within the workflow implementation[​](https://docs.temporal.io/ai-cookbook/human-in-the-loop-python#local-state-within-the-workflow-implementation "Direct link to Local state within the workflow implementation") This state will be written to via the signal handler and will be part of the condition that defines the wait point. @workflow.defnclass HumanInTheLoopWorkflow: def __init__(self): self.current_decision: Optional[ApprovalDecision] = None self.pending_request_id: Optional[str] = None ### Signal Handler[​](https://docs.temporal.io/ai-cookbook/human-in-the-loop-python#signal-handler "Direct link to Signal Handler") The workflow uses a signal handler to receive approval decisions asynchronously: @workflow.signalasync def approval_decision(self, decision: ApprovalDecision): if decision.request_id == self.pending_request_id: self.approval_decision = decision ... ### Waiting with Timeout[​](https://docs.temporal.io/ai-cookbook/human-in-the-loop-python#waiting-with-timeout "Direct link to Waiting with Timeout") The workflow waits for approval with a configurable timeout: await workflow.wait_condition( lambda: self.approval_decision is not None, timeout=timedelta(seconds=timeout_seconds),) Extensions[​](https://docs.temporal.io/ai-cookbook/human-in-the-loop-python#extensions "Direct link to Extensions") -------------------------------------------------------------------------------------------------------------------- This pattern can be extended to support: * Multiple approvers with voting * Escalation workflows * Conditional approval based on action risk * Integration with Slack, email, or custom UIs * Query handlers to check approval status --- # Durable Agent with Tools - OpenAI Agents SDK [Skip to main content](https://docs.temporal.io/ai-cookbook/openai-agents-sdk-python#__docusaurus_skipToContent_fallback) On this page In this example, we show you how to build a Durable Agent using the [OpenAI Agents SDK Integration for Temporal](https://github.com/temporalio/sdk-python/tree/main/temporalio/contrib/openai_agents) . The AI agent we build will have access to [tools](https://github.com/temporalio/sdk-python/tree/main/temporalio/contrib/openai_agents#tool-calling) (Temporal Activities) to answer user questions. The agent can determine which tools to use based on the user's input and execute them as needed. This recipe highlights key implementation patterns: * **Agent-based architecture**: Uses the OpenAI Agents SDK to create an intelligent agent that can reason about which tools to use and handles LLM invocation for you. * **Tool integration**: Temporal Activities can be seamlessly used as tools by the agent. The integration offers the **activity\_as\_tool** helper function, which: * Automatically generates OpenAI-compatible tool schemas from activity function signatures * Wraps activities as agent tools that can be provided directly to the Agent * Enables the agent to invoke Temporal Activities as tools leveraging Temporal's durable execution for tool calls * **Durable execution**: The agent's state and execution are managed by Temporal, providing reliability and observability * **Plugin configuration**: Uses the `OpenAIAgentsPlugin` to configure Temporal for OpenAI Agents SDK integration Create the Activity[​](https://docs.temporal.io/ai-cookbook/openai-agents-sdk-python#create-the-activity "Direct link to Create the Activity") ----------------------------------------------------------------------------------------------------------------------------------------------- We create activities that serve as tools for the agent. These activities can perform various tasks like getting weather information or performing calculations. _File: activities/tools.py_ from dataclasses import dataclassfrom temporalio import activityimport math# Temporal best practice: Create a data structure to hold the request parameters.@dataclassclass Weather: city: str temperature_range: str conditions: str@activity.defnasync def get_weather(city: str) -> Weather: """Get the weather for a given city.""" return Weather(city=city, temperature_range="14-20C", conditions="Sunny with wind.")@activity.defnasync def calculate_circle_area(radius: float) -> float: """Calculate the area of a circle given its radius.""" return math.pi * radius ** 2 Create the Workflow[​](https://docs.temporal.io/ai-cookbook/openai-agents-sdk-python#create-the-workflow "Direct link to Create the Workflow") ----------------------------------------------------------------------------------------------------------------------------------------------- The workflow creates an agent with specific instructions and tools. The agent can then process user input and decide which tools to use to answer questions. Since LLM invocation is an external API call, this typically would happen in a Temporal Activity. However, because of the Temporal Integration with OpenAI Agents SDK, this is being handled for us and we do not need to implement the Activity ourselves. _File: workflows/hello\_world\_workflow.py_ from temporalio import workflowfrom datetime import timedeltafrom agents import Agent, Runnerfrom temporalio.contrib import openai_agentsfrom activities.tools import get_weather, calculate_circle_area@workflow.defnclass HelloWorldAgent: @workflow.run async def run(self, prompt: str) -> str: agent = Agent( name="Hello World Agent", instructions="You are a helpful assistant that determines what tool to use based on the user's question.", # Tools for the agent to use that are defined as activities tools=[ openai_agents.workflow.activity_as_tool( get_weather, start_to_close_timeout=timedelta(seconds=10) ), openai_agents.workflow.activity_as_tool( calculate_circle_area, start_to_close_timeout=timedelta(seconds=10) ) ] ) result = await Runner.run(agent, prompt) return result.final_output Create the Worker[​](https://docs.temporal.io/ai-cookbook/openai-agents-sdk-python#create-the-worker "Direct link to Create the Worker") ----------------------------------------------------------------------------------------------------------------------------------------- Create the process for executing Activities and Workflows. We configure the Temporal client with the `OpenAIAgentsPlugin` to enable OpenAI Agents SDK integration. _File: worker.py_ import asynciofrom datetime import timedeltafrom temporalio.client import Clientfrom temporalio.worker import Workerfrom temporalio.contrib.openai_agents import OpenAIAgentsPlugin, ModelActivityParametersfrom workflows.hello_world_workflow import HelloWorldAgentfrom activities.tools import get_weather, calculate_circle_areaasync def worker_main(): # Use the plugin to configure Temporal for use with OpenAI Agents SDK client = await Client.connect( "localhost:7233", plugins=[ OpenAIAgentsPlugin( model_params=ModelActivityParameters( start_to_close_timeout=timedelta(seconds=30) ) ), ], ) worker = Worker( client, task_queue="hello-world-openai-agent-task-queue", workflows=[HelloWorldAgent], activities=[get_weather, calculate_circle_area], ) await worker.run()if __name__ == "__main__": asyncio.run(worker_main()) Create the Workflow Starter[​](https://docs.temporal.io/ai-cookbook/openai-agents-sdk-python#create-the-workflow-starter "Direct link to Create the Workflow Starter") ----------------------------------------------------------------------------------------------------------------------------------------------------------------------- The starter script submits the agent workflow to Temporal for execution, then waits for the result and prints it out. It uses the `OpenAIAgentsPlugin` to match the Worker configuration. _File: start\_workflow.py_ import asynciofrom temporalio.client import Clientfrom temporalio.common import WorkflowIDConflictPolicyfrom temporalio.contrib.openai_agents import OpenAIAgentsPluginfrom workflows.hello_world_workflow import HelloWorldAgentasync def main(): client = await Client.connect( "localhost:7233", # Use the plugin to configure Temporal for use with OpenAI Agents SDK plugins=[OpenAIAgentsPlugin()], ) # Start workflow print( 80 * "-" ) # Get user input user_input = input("Enter a question: ") # Submit the Hello World Agent workflow for execution result = await client.execute_workflow( HelloWorldAgent.run, user_input, id="my-workflow-id", task_queue="hello-world-openai-agent-task-queue", id_conflict_policy=WorkflowIDConflictPolicy.TERMINATE_IF_RUNNING, ) print(f"Result: {result}") # End of workflow print( 80 * "-" ) print("Workflow completed")if __name__ == "__main__": asyncio.run(main()) Running[​](https://docs.temporal.io/ai-cookbook/openai-agents-sdk-python#running "Direct link to Running") ----------------------------------------------------------------------------------------------------------- Start the Temporal Dev Server: temporal server start-dev Open a new terminal where you will run the agent worker. Set an OpenAI API key: export OPENAI_API_KEY=sk... Run the worker: uv run python -m worker Start execution: uv run python -m start_workflow Example Interactions[​](https://docs.temporal.io/ai-cookbook/openai-agents-sdk-python#example-interactions "Direct link to Example Interactions") -------------------------------------------------------------------------------------------------------------------------------------------------- Try asking the agent questions like: * "What's the weather in London?" * "Calculate the area of a circle with radius 5" * "What's the weather in Tokyo and calculate the area of a circle with radius 3" The agent will determine which tools to use and provide intelligent responses based on the available tools. Use the [OpenAI Traces dashboard](https://platform.openai.com/traces) to visualize and monitor your workflows and tool calling. --- # Durable MCP Weather Server [Skip to main content](https://docs.temporal.io/ai-cookbook/hello-world-durable-mcp-server#__docusaurus_skipToContent_fallback) On this page This example demonstrates how to build a durable MCP (Model Context Protocol) server using Temporal Workflows for Durable Execution. The server exposes weather tools that fetch alerts and forecasts from the National Weather Service API. MCP tools are "actions" that the MCP server can perform. Within a given MCP tool, there are often multiple steps (API calls, functions, etc.) that must happen in a certain order to complete an action. For example, the `get_forecast` tool performs the following steps: * Call the National Weather Service API to find which region corresponds to the given latitude and longitude coordinates * Call the National Weather Service API again to retrieve the forecast for that region * Format and return the response to the user In this one tool alone, we are taking several steps to complete a given action. We implement these steps in a Temporal Workflow, which ensures durability out-of-the-box. This means that whenever your MCP tool is called, it kicks off the Temporal Workflow, and every step (API call, function) is executed reliably and all the way to completion. We use [FastMCP](https://github.com/jlowin/fastmcp) to implement the MCP Server and create tools using the decorator `@mcp.tool`. note External API calls are made within Temporal Activities. This ensures that network requests are retried appropriately and failures are handled gracefully. This recipe highlights the following key design decisions: * **Separation of concerns**: MCP tools act as thin wrappers that start Temporal Workflows. All business logic lives in workflows, ensuring durability and reliability. * **Durable Execution**: By moving multi-step operations into Temporal Workflows, we guarantee that operations complete even in the face of failures, network issues, or process restarts. * **Activity-based external calls**: All external API calls (like NWS API requests) are made within Temporal Activities, which provides automatic retries and proper error handling. * **Retry policies**: Workflows use configurable retry policies to handle transient failures gracefully. Also see this foundational [recipe for basic tool calling](https://docs.temporal.io/ai-cookbook/tool-call-openai-python) using the same weather tools. Application Components[​](https://docs.temporal.io/ai-cookbook/hello-world-durable-mcp-server#application-components "Direct link to Application Components") -------------------------------------------------------------------------------------------------------------------------------------------------------------- This example includes the following components: * The [MCP server](https://docs.temporal.io/ai-cookbook/hello-world-durable-mcp-server#create-the-mcp-server) (mcp\_server.py) that exposes tools via FastMCP and starts Temporal workflows * The [Workflows](https://docs.temporal.io/ai-cookbook/hello-world-durable-mcp-server#create-the-workflows) (weather\_workflows.py) that orchestrate the multi-step weather operations * The [Activity](https://docs.temporal.io/ai-cookbook/hello-world-durable-mcp-server#create-the-activity) (weather\_activities.py) for making external API calls to the National Weather Service * The [Worker](https://docs.temporal.io/ai-cookbook/hello-world-durable-mcp-server#create-the-worker) (worker.py) (that manages the Workflows and Activities) * [Config for Claude Desktop](https://docs.temporal.io/ai-cookbook/hello-world-durable-mcp-server#configure-claude-desktop) (claude\_desktop\_config.json) for connecting the MCP server to Claude Desktop Create the MCP Server[​](https://docs.temporal.io/ai-cookbook/hello-world-durable-mcp-server#create-the-mcp-server "Direct link to Create the MCP Server") ----------------------------------------------------------------------------------------------------------------------------------------------------------- The MCP server is implemented using FastMCP and exposes tools via the `@mcp.tool` decorator. Each tool is a thin wrapper that starts a Temporal Workflow and waits for the result. This design ensures that all business logic lives in durable Workflows. _File: mcp\_servers/weather.py_ from temporalio.client import Clientfrom fastmcp import FastMCP# Initialize FastMCP servermcp = FastMCP("weather")# Temporal client setup (do this once, then reuse)temporal_client = Noneasync def get_temporal_client(): global temporal_client if not temporal_client: config = ClientConfig.load_client_connect_config() config.setdefault("target_host", "localhost:7233") temporal_client = await Client.connect(**config) return temporal_client@mcp.toolasync def get_alerts(state: str) -> str: """Get weather alerts for a US state. Args: state: Two-letter US state code (e.g. CA, NY) """ # The business logic has been moved into the Temporal Workflow, the MCP tool kicks off the Workflow client = await get_temporal_client() handle = await client.start_workflow( "GetAlerts", state, id=f"alerts-{state.lower()}", task_queue="weather-task-queue" ) return await handle.result()@mcp.toolasync def get_forecast(latitude: float, longitude: float) -> str: """Get weather forecast for a US location. Args: latitude: Latitude of the location (must be within the US) longitude: Longitude of the location (must be within the US) """ # The business logic has been moved into the Temporal Workflow, the MCP tool kicks off the Workflow client = await get_temporal_client() handle = await client.start_workflow( workflow="GetForecast", args=[latitude, longitude], id=f"forecast-{latitude}-{longitude}", task_queue="weather-task-queue", ) return await handle.result()if __name__ == "__main__": # Initialize and run the server mcp.run(transport='stdio') Create the Workflows[​](https://docs.temporal.io/ai-cookbook/hello-world-durable-mcp-server#create-the-workflows "Direct link to Create the Workflows") -------------------------------------------------------------------------------------------------------------------------------------------------------- The Workflows contain the business logic for fetching weather data. They orchestrate multiple steps, including API calls and data formatting. By implementing this logic in workflows, we ensure that operations complete reliably even if there are failures or interruptions. ### GetAlerts Workflow[​](https://docs.temporal.io/ai-cookbook/hello-world-durable-mcp-server#getalerts-workflow "Direct link to GetAlerts Workflow") The `GetAlerts` workflow fetches active weather alerts for a US state. _File: workflows/weather\_workflows.py_ from datetime import timedeltafrom temporalio import workflowfrom temporalio.common import RetryPolicyretry_policy = RetryPolicy( maximum_attempts=0, # Infinite retries initial_interval=timedelta(seconds=2), maximum_interval=timedelta(minutes=1), backoff_coefficient=1.0,)# ConstantsNWS_API_BASE = "https://api.weather.gov"# Import Activities, passing them through the sandboxwith workflow.unsafe.imports_passed_through(): from activities.weather_activities import make_nws_requestdef format_alert(feature: dict) -> str: """Format an alert feature into a readable string.""" props = feature["properties"] return f"""Event: {props.get('event', 'Unknown')}Area: {props.get('areaDesc', 'Unknown')}Severity: {props.get('severity', 'Unknown')}Description: {props.get('description', 'No description available')}Instructions: {props.get('instruction', 'No specific instructions provided')}"""@workflow.defnclass GetAlerts: @workflow.run async def get_alerts(self, state: str) -> str: """Get weather alerts for a US state. Args: state: Two-letter US state code (e.g. CA, NY) """ url = f"{NWS_API_BASE}/alerts/active/area/{state}" data = await workflow.execute_activity( make_nws_request, url, schedule_to_close_timeout=timedelta(seconds=40), retry_policy=retry_policy, ) if not data or "features" not in data: return "Unable to fetch alerts or no alerts found." if not data["features"]: return "No active alerts for this state." alerts = [format_alert(feature) for feature in data["features"]] return "\n---\n".join(alerts) ### GetForecast Workflow[​](https://docs.temporal.io/ai-cookbook/hello-world-durable-mcp-server#getforecast-workflow "Direct link to GetForecast Workflow") The `GetForecast` workflow demonstrates a multi-step operation: it first fetches the forecast grid endpoint for a location, then uses that information to fetch the detailed forecast. _File: workflows/weather\_workflows.py_ @workflow.defnclass GetForecast: @workflow.run async def get_forecast(self, latitude: float, longitude: float) -> str: """Get weather forecast for a US location. Args: latitude: Latitude of the location (must be within the US) longitude: Longitude of the location (must be within the US) """ # First get the forecast grid endpoint points_url = f"{NWS_API_BASE}/points/{latitude},{longitude}" points_data = await workflow.execute_activity( make_nws_request, points_url, schedule_to_close_timeout=timedelta(seconds=40), retry_policy=retry_policy, ) if not points_data: return "Unable to fetch forecast data for this location." # Get the forecast URL from the points response forecast_url = points_data["properties"]["forecast"] forecast_data = await workflow.execute_activity( make_nws_request, forecast_url, schedule_to_close_timeout=timedelta(seconds=40), retry_policy=retry_policy, ) if not forecast_data: return "Unable to fetch detailed forecast." # Format the periods into a readable forecast periods = forecast_data["properties"]["periods"] forecasts = [] for period in periods[:5]: # Only show next 5 periods forecast = f""" {period['name']}: Temperature: {period['temperature']}°{period['temperatureUnit']} Wind: {period['windSpeed']} {period['windDirection']} Forecast: {period['detailedForecast']} """ forecasts.append(forecast) return "\n---\n".join(forecasts) Create the Activity[​](https://docs.temporal.io/ai-cookbook/hello-world-durable-mcp-server#create-the-activity "Direct link to Create the Activity") ----------------------------------------------------------------------------------------------------------------------------------------------------- We create an Activity for making HTTP requests to the National Weather Service API. All external API calls happen within Activities, which provides automatic retries and proper error handling through Temporal's retry mechanisms. _File: activities/weather\_activities.py_ from typing import Anyfrom temporalio import activityimport httpxUSER_AGENT = "weather-app/1.0"# External calls happen via Activities@activity.defnasync def make_nws_request(url: str) -> dict[str, Any] | None: """Make a request to the NWS API with proper error handling.""" headers = { "User-Agent": USER_AGENT, "Accept": "application/geo+json" } async with httpx.AsyncClient() as client: response = await client.get(url, headers=headers, timeout=5.0) response.raise_for_status() return response.json() Create the Worker[​](https://docs.temporal.io/ai-cookbook/hello-world-durable-mcp-server#create-the-worker "Direct link to Create the Worker") ----------------------------------------------------------------------------------------------------------------------------------------------- The Worker is the process that excutes Activities and Workflows. _File: worker.py_ import asynciofrom temporalio.client import Clientfrom temporalio.worker import Workerfrom workflows.weather_workflows import GetAlerts, GetForecastfrom activities.weather_activities import make_nws_requestasync def main(): config = ClientConfig.load_client_connect_config() config.setdefault("target_host", "localhost:7233") client = await Client.connect( **config, data_converter=pydantic_data_converter, ) # Register both Workflows and the Activity worker = Worker( client, task_queue="weather-task-queue", workflows=[GetAlerts, GetForecast], activities=[make_nws_request], ) print("Worker started. Listening for workflows...") await worker.run()# Start worker with both Workflows and Activitiesif __name__ == "__main__": asyncio.run(main()) Configure Claude Desktop[​](https://docs.temporal.io/ai-cookbook/hello-world-durable-mcp-server#configure-claude-desktop "Direct link to Configure Claude Desktop") -------------------------------------------------------------------------------------------------------------------------------------------------------------------- For this example, we are using Claude Desktop as the MCP Client. To use this MCP server with Claude Desktop, you need to configure it in your Claude Desktop configuration file. The config file tells Claude Desktop how to start the MCP server. _File: claude\_desktop\_config.json_ { "mcpServers": { "weather": { "command": "uv", "args": [ "--directory", "", "run", "mcp_servers/weather.py" ] } }} Replace `` with the absolute path to the `hello_world_durable_mcp_server` directory. Configuration[​](https://docs.temporal.io/ai-cookbook/hello-world-durable-mcp-server#configuration "Direct link to Configuration") ----------------------------------------------------------------------------------------------------------------------------------- This recipe uses Temporal's environment configuration system to connect to Temporal. By default, it connects to a local Temporal server. To use Temporal Cloud: 1. Set the `TEMPORAL_PROFILE` environment variable to use the cloud profile: export TEMPORAL_PROFILE=cloud 2. Configure the cloud profile using the Temporal CLI: temporal config set --profile cloud --prop address --value ""temporal config set --profile cloud --prop namespace --value ""temporal config set --profile cloud --prop api_key --value "" For TLS certificate authentication instead of API key, refer to the [Temporal environment configuration documentation](https://docs.temporal.io/develop/environment-configuration) for details. Running the MCP Server[​](https://docs.temporal.io/ai-cookbook/hello-world-durable-mcp-server#running-the-mcp-server "Direct link to Running the MCP Server") -------------------------------------------------------------------------------------------------------------------------------------------------------------- 1. Install dependencies: uv sync 2. Start a Temporal server: # Using Temporal CLItemporal server start-dev 3. Start the worker in one terminal: uv run python worker.py 4. Configure Claude Desktop by adding the configuration from `claude_desktop_config.json` to your Claude Desktop config file (typically located at `~/Library/Application Support/Claude/claude_desktop_config.json` on macOS). 5. Restart Claude Desktop to load the MCP server. Once configured, you should see the tool appear under the slider icon underneath the Claude Desktop chat input box. You can now ask Claude something like `What is the weather like in San Francisco, CA?`. Claude Desktop will understand that it needs to use the `get_forecast` tool in the Weather MCP server that you just configured. note The National Weather Service API only supports US locations. Asking about weather in non-US locations (e.g., "What is the weather in London?") will result in a 404 error from the API. After tool execution, Claude Desktop will send the result over to the LLM (with other context) for human formating, and then returns that result to the user. You can see these and other MCP-related actions in the `mcp_server.log`. --- # Structured Outputs with Temporal and OpenAI [Skip to main content](https://docs.temporal.io/ai-cookbook/structured-output-openai-responses-python#__docusaurus_skipToContent_fallback) On this page The OpenAI Responses API provides the [Structured Outputs API](https://platform.openai.com/docs/guides/structured-outputs) allowing you to request responses conforming to a specific data structure. In this example, we use structured outputs in a business data cleaning scenario. Structured outputs are also commonly used for tool calling. OpenAI usually returns the correct type. However, this is not always the case due to the non-deterministic nature of LLMs. When OpenAI returns an incorrect type, Temporal automatically retries the LLM call Activity. Invoke Model Activity[​](https://docs.temporal.io/ai-cookbook/structured-output-openai-responses-python#invoke-model-activity "Direct link to Invoke Model Activity") ---------------------------------------------------------------------------------------------------------------------------------------------------------------------- We create a model-calling Activity that uses the `responses.parse` method of the OpenAI client. Key challenges are related to serialization: 1. In `InvokeModelRequest` the `response_format` field is a class reference. We provide custom Pydantic serialization and deserialization logic. 2. In `InvokeModelResponse` the `response_model` must be deserialized to the correct type. We serialize the type in one field and the model, represented as a dictionary, in another. from temporalio import activityfrom openai import AsyncOpenAIfrom typing import Optional, List, cast, Any, TypeVar, Genericfrom typing_extensions import Annotatedfrom pydantic import BaseModelfrom pydantic.functional_validators import BeforeValidatorfrom pydantic.functional_serializers import PlainSerializerimport importlibT = TypeVar("T", bound=BaseModel)def _coerce_class(v: Any) -> type[Any]: """Pydantic validator: convert string path to class during deserialization.""" if isinstance(v, str): mod_path, sep, qual = v.partition(":") if not sep: # support "package.module.Class" mod_path, _, qual = v.rpartition(".") module = importlib.import_module(mod_path) obj = module for attr in qual.split("."): obj = getattr(obj, attr) return cast(type[Any], obj) elif isinstance(v, type): return v else: raise ValueError(f"Cannot coerce {v} to class")def _dump_class(t: type[Any]) -> str: """Pydantic serializer: convert class to string path during serialization.""" return f"{t.__module__}:{t.__qualname__}"# Custom type that automatically handles class <-> string conversion in Pydantic serializationClassReference = Annotated[ type[T], BeforeValidator(_coerce_class), PlainSerializer(_dump_class, return_type=str),]class InvokeModelRequest(BaseModel, Generic[T]): model: str instructions: str input: str response_format: Optional[ClassReference[T]] = None tools: Optional[List[dict]] = Noneclass InvokeModelResponse(BaseModel, Generic[T]): # response_format records the type of the response model response_format: Optional[ClassReference[T]] = None response_model: Any @property def response(self) -> T: """Reconstruct the original response type if response_format was provided.""" if self.response_format: model_cls = self.response_format return model_cls.model_validate(self.response_model) return self.response_model@activity.defnasync def invoke_model(request: InvokeModelRequest[T]) -> InvokeModelResponse[T]: client = AsyncOpenAI(max_retries=0) kwargs = { "model": request.model, "instructions": request.instructions, "input": request.input, } if request.response_format: kwargs["text_format"] = request.response_format if request.tools: kwargs["tools"] = request.tools # Use responses API consistently resp = await client.responses.parse(**kwargs) if request.response_format: # Convert structured response to dict for managed serialization. # This allows us to reconstruct the original response type while maintaining type safety. parsed_model = cast(BaseModel, resp.output_parsed) return InvokeModelResponse( response_model=parsed_model.model_dump(), response_format=request.response_format, ) else: return InvokeModelResponse( response_model=resp.output_text, response_format=None ) Workflow[​](https://docs.temporal.io/ai-cookbook/structured-output-openai-responses-python#workflow "Direct link to Workflow") ------------------------------------------------------------------------------------------------------------------------------- We define the `Business` class as a Pydantic model. We use the Pydantic's `EmailStr` type for the email field. For the phone field, we use a custom validator to ensure the phone number is in E.164 format. The validators should check for obvious structural errors that LLMs will only get wrong sporadically. If the LLM produces invalid responses consistently, Activity retries will fail consistently. To mitigate the cost of such futile retries, we limit the number of retry attempts when using structured outputs. from pydantic import BaseModel, Field, field_validator, EmailStrfrom pydantic_core import PydanticCustomErrorimport refrom temporalio import workflowfrom activities import invoke_modelfrom activities.invoke_model import InvokeModelRequestfrom typing import List, Optionalfrom datetime import timedeltafrom temporalio.common import RetryPolicyclass Business(BaseModel): name: Optional[str] = Field( None, description="The business name", json_schema_extra={"example": "Acme Corporation"}, ) email: Optional[EmailStr] = Field( None, description="Primary business email address", json_schema_extra={"example": "info@acmecorp.com"}, ) phone: Optional[str] = Field( None, description="Primary business phone number in E.164 format", json_schema_extra={"example": "+12025550173"}, ) address: Optional[str] = Field( None, description="Business mailing address", json_schema_extra={ "example": "123 Business Park Dr, Suite 100, New York, NY 10001" }, ) website: Optional[str] = Field( None, description="Business website URL", json_schema_extra={"example": "https://www.acmecorp.com"}, ) industry: Optional[str] = Field( None, description="Business industry or sector", json_schema_extra={"example": "Technology"}, ) @field_validator("phone", mode="before") def validate_phone(cls, v): # Allow None values if v is None: return None if isinstance(v, str): v = v.strip() # Allow empty strings to be converted to None for optional fields if not v: return None # E.164 format: + followed by 1-9, then 9-15 more digits e164_pattern = r"^\+[1-9]\d{9,15}$" if not re.match(e164_pattern, v): raise PydanticCustomError( "phone_format", "Phone number must be in E.164 format (e.g., +12025550173)", {"invalid_phone": v}, ) return v @field_validator("name", mode="before") def validate_name(cls, v): # Allow None values if v is None: return None if isinstance(v, str): v = v.strip() # Convert empty strings to None (this is acceptable) if not v: return None return vclass BusinessList(BaseModel): businesses: List[Business]@workflow.defnclass CleanDataWorkflow: @workflow.run async def run(self, data: str) -> BusinessList: results = await workflow.execute_activity( invoke_model.invoke_model, InvokeModelRequest( model="gpt-4o", instructions=f"""Extract and clean business data with these specific rules:1. BUSINESS NAME: Extract the main business name, normalize capitalization (Title Case for proper nouns)2. EMAIL: - Extract only ONE primary email address - If multiple emails, choose the one marked as "primary" or the first valid one - Validate format (must have @ and valid domain with .) - Set to null if invalid (e.g., "bob@email", "NONE PROVIDED")3. PHONE: - Convert to E.164 format (+1 prefix for US numbers, add if not provided) - Convert letters to numbers where appropriate (e.g., "1-800-FLOWERS" → "+18003569377") - Set to null if cannot be converted to valid E.164 format - Examples: "(555) 123-4567" → "+15551234567", "555 234 5678 ext 349i" → null (invalid), "5551234567" → "+15551234567"4. ADDRESS: - Provide complete, standardized address - Set to null if vague/incomplete (e.g., "north end of main st", "unknown", "[PRIVATE]")5. WEBSITE: - Standardize to https:// format - Remove "www." prefix, add https:// if missing - Set to null if broken/invalid (e.g., "broken-link.com/404", "down for maintenance")6. INDUSTRY: - Use clear, professional industry categories - Normalize similar terms (e.g., "fix cars and trucks" → "Automotive Repair")Return null for any field that cannot be reliably extracted or validated.""", input=data, response_format=BusinessList, ), start_to_close_timeout=timedelta(seconds=300), retry_policy=RetryPolicy( maximum_attempts=3, ), summary="Clean data", ) return results.response Running[​](https://docs.temporal.io/ai-cookbook/structured-output-openai-responses-python#running "Direct link to Running") ---------------------------------------------------------------------------------------------------------------------------- Start the Temporal Dev Server: temporal server start-dev Run the worker: uv run python -m worker Start execution: uv run python -m start_workflow --- # One doc tagged with "Actions" | Temporal Platform Documentation [Skip to main content](https://docs.temporal.io/tags/actions#__docusaurus_skipToContent_fallback) [Temporal Cloud Actions\ ----------------------](https://docs.temporal.io/cloud/actions) Temporal Cloud offers flexible, predictable pricing for Workflows, Activities, Workers, and Storage. Pay for what you use with volume discounts and credit savings. --- # One doc tagged with "Accounts" | Temporal Platform Documentation [Skip to main content](https://docs.temporal.io/tags/accounts#__docusaurus_skipToContent_fallback) [tcld account command reference\ ------------------------------](https://docs.temporal.io/cloud/tcld/account) Manage Temporal Cloud accounts using tcld commands. Get account details, configure and manage metrics endpoints, and handle end-entity certificates efficiently with various commands. --- # Tool calling agent [Skip to main content](https://docs.temporal.io/ai-cookbook/tool-call-openai-python#__docusaurus_skipToContent_fallback) On this page In this example, we demonstrate how function calling (also known as tool calling) works with the [Open AI API](https://github.com/openai/openai-python) and Temporal. Tool calling allows the model to make decisions on which, if any, functions should be invoked. It also provides information to the LLM that will allow it to structure the response in such a way that the agent can easily invoke the functions. Tools are supplied to the [`responses` API](https://platform.openai.com/docs/api-reference/responses/create) through the [`tools` parameter](https://platform.openai.com/docs/api-reference/responses/create#responses-create-tools) . The `tools` parameter is in`json` and includes a description of the function as well as descriptions of each of the arguments. caution The API used to generate the tools json is an internal function from the [Open AI API](https://github.com/openai/openai-python) and may therefore change in the future. There currently is no public API to generate the tool definition from a Pydantic model or a function signature. Being external API calls, invoking the LLM and invoking the function are each done within a Temporal Activity. This example lays the foundation for the core agentic pattern where the LLM makes the decision on functions/tools to invoke, the agent calls the function/tool(s) and the response from such calls is sent back to the LLM for interpretation. ![](https://docs.temporal.io/assets/images/tool-call-openai-python-assets-tool-calling-flow-616ae35809d90e7f788fe8b3702204cf.png) This recipe highlights these key design decisions: * A generic Activity for invoking an LLM API; that is, instructions and other responses arguments are passed into the Activity making it appropriate for use in a variety of different use cases. Similarly, the result from the responses API call is returned out of the Activity so that it is usable in a variety of different use cases. * We have intentionally not implemented the agentic loop so as to focus on how tool details are made available to the LLM and how functions are invoked. We do take the tool output and have the LLM interpret it in a manner consistent with the AI agent pattern. * Retries are handled by Temporal and not by the underlying libraries such as the OpenAI client. This is important because if you leave the client retries on they can interfere with correct and durable error handling and recovery. Create the Activity for LLM invocations[​](https://docs.temporal.io/ai-cookbook/tool-call-openai-python#create-the-activity-for-llm-invocations "Direct link to Create the Activity for LLM invocations") ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- We create a wrapper for the `create` method of the `AsyncOpenAI` client object. This is a generic Activity that invokes the OpenAI LLM. We set `max_retries=0` when creating the `AsyncOpenAI` client. This moves the responsibility for retries from the OpenAI client to Temporal. In this implementation, we allow for the model, instructions and input to be passed in, and also the list of tools. `activities/openai_responses.py` from temporalio import activityfrom openai import AsyncOpenAIfrom openai.types.responses import Responsefrom dataclasses import dataclassfrom typing import Any# Temporal best practice: Create a data structure to hold the request parameters.@dataclassclass OpenAIResponsesRequest: model: str instructions: str input: object tools: list[dict[str, Any]]@activity.defnasync def create(request: OpenAIResponsesRequest) -> Response: # Temporal best practice: Disable retry logic in OpenAI API client library. client = AsyncOpenAI(max_retries=0) resp = await client.responses.create( model=request.model, instructions=request.instructions, input=request.input, tools=request.tools, timeout=30, ) return resp Create the Activity for the tool invocation[​](https://docs.temporal.io/ai-cookbook/tool-call-openai-python#create-the-activity-for-the-tool-invocation "Direct link to Create the Activity for the tool invocation") ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- We create a wrapper for invoking the [National Weather Service API](https://www.weather.gov/documentation/services-web-api) , specifically for the weather alerts endpoint. We follow the Temporal best practice of encapsulating all input parameters to the activity in data structure, even here where this is only one argument. The `WEATHER_ALERTS_TOOL_OAI` leverages a function defined in `helpers/tool_helpers.py` that calls the aforementioned internal OpenAI function, generating a dictionary that becomes the argument passed into the OpenAI responses API. `activities/get_weather_alerts.py` # weather_activities.pyfrom typing import Anyfrom temporalio import activityimport httpximport jsonfrom pydantic import BaseModelimport openaifrom helpers import tool_helpersfrom pydantic import Field# ConstantsNWS_API_BASE = "https://api.weather.gov"USER_AGENT = "weather-app/1.0"def _alerts_url(state: str) -> str: return f"{NWS_API_BASE}/alerts/active/area/{state}"# External calls happen via activities nowasync def _make_nws_request(url: str) -> dict[str, Any] | None: """Make a request to the NWS API with proper error handling.""" headers = { "User-Agent": USER_AGENT, "Accept": "application/geo+json" } async with httpx.AsyncClient() as client: response = await client.get(url, headers=headers, timeout=5.0) response.raise_for_status() return response.json()# Build the tool for the OpenAI Responses API. We use Pydantic to create a structure# that encapsulates the input parameters for both the weather alerts activity and the# tool definition that is passed to the OpenAI Responses API.class GetWeatherAlertsRequest(BaseModel): state: str = Field(description="Two-letter US state code (e.g. CA, NY)")WEATHER_ALERTS_TOOL_OAI: dict[str, Any] = tool_helpers.oai_responses_tool_from_model( "get_weather_alerts", "Get weather alerts for a US state.", GetWeatherAlertsRequest)@activity.defnasync def get_weather_alerts(weather_alerts_request: GetWeatherAlertsRequest) -> str: """Get weather alerts for a US state. Args: state: Two-letter US state code (e.g. CA, NY) """ data = await _make_nws_request(_alerts_url(weather_alerts_request.state)) return json.dumps(data) ### Create the helper function[​](https://docs.temporal.io/ai-cookbook/tool-call-openai-python#create-the-helper-function "Direct link to Create the helper function") The `oai_responses_tool_from_model` function accepts a tool name and description, as well as a list of argument name/description pairs and returns json that is in the format expected for tool definitions in the OpenAI responses API. caution The API used to generate the tools json is an interal function from the [Open AI API](https://github.com/openai/openai-python) and may therefore change in the future. There currently is no public API to generate the tool definition from a Pydantic model or a function signature. `helpers/tool_helpers.py` from openai.lib._pydantic import to_strict_json_schema # private API; may change# there currently is no public API to generate the tool definition from a Pydantic model# or a function signature.from pydantic import BaseModeldef oai_responses_tool_from_model(name: str, description: str, model: type[BaseModel]): return { "type": "function", "name": name, "description": description, "parameters": to_strict_json_schema(model), "strict": True, } Create the Agent[​](https://docs.temporal.io/ai-cookbook/tool-call-openai-python#create-the-agent "Direct link to Create the Agent") ------------------------------------------------------------------------------------------------------------------------------------- The agent is implemented as a Temporal workflow that orchestrates * the intial LLM call with the initial user input and guidance to the LLM that they should respond in haiku when the user input doesn't lead to a tool call, * the invocation of the function, if the LLM has chosen one * and if a function has been called, the result is appended to the context that is then sent back to the LLM for interpretation (the LLM is instructed to format the tool response). `workflows/get_weather_workflow.py` from temporalio import workflowfrom datetime import timedeltaimport jsonfrom activities import openai_responseswith workflow.unsafe.imports_passed_through(): from activities import get_weather_alerts@workflow.defnclass ToolCallingWorkflow: @workflow.run async def run(self, input: str) -> str: input_list = [ {"role": "user", "content": input} ] # We take the user input and pass it to the LLM with the system instructions # and the tool to use, if applicable. system_instructions = "if no tools seem to be needed, respond in haikus." result = await workflow.execute_activity( openai_responses.create, openai_responses.OpenAIResponsesRequest( model="gpt-4o-mini", instructions=system_instructions, input=input_list, tools=[get_weather_alerts.WEATHER_ALERTS_TOOL_OAI], ), start_to_close_timeout=timedelta(seconds=30), ) # For this simple example, we only have one item in the output list item = result.output[0] # if the result is a tool call, call the tool if item.type == "function_call": if item.name == "get_weather_alerts": # serialize the output, which is an OpenAI object input_list += [ i.model_dump() if hasattr(i, "model_dump") else i for i in result.output ] result = await workflow.execute_activity( get_weather_alerts.get_weather_alerts, get_weather_alerts.GetWeatherAlertsRequest(state=json.loads(item.arguments)["state"]), start_to_close_timeout=timedelta(seconds=30), ) # add the tool call result to the input list for context input_list.append({"type": "function_call_output", "call_id": item.call_id, "output": result}) result = await workflow.execute_activity( openai_responses.create, openai_responses.OpenAIResponsesRequest( model="gpt-4o-mini", instructions="return the tool call result in a readable format", input=input_list, tools=[] ), start_to_close_timeout=timedelta(seconds=30), ) result = result.output_text return result Create the Worker[​](https://docs.temporal.io/ai-cookbook/tool-call-openai-python#create-the-worker "Direct link to Create the Worker") ---------------------------------------------------------------------------------------------------------------------------------------- The worker is the process that dispatches work to the various parts of the agent implementation - the orchestrator and the activities for the LLM and tool invocations. _File: worker.py_ import asynciofrom temporalio.client import Clientfrom temporalio.worker import Workerfrom workflows.get_weather_workflow import ToolCallingWorkflowfrom activities import openai_responses, get_weather_alertsfrom temporalio.contrib.pydantic import pydantic_data_converterasync def main(): client = await Client.connect( "localhost:7233", data_converter=pydantic_data_converter, ) worker = Worker( client, task_queue="tool-calling-python-task-queue", workflows=[ ToolCallingWorkflow, ], activities=[ openai_responses.create, get_weather_alerts.get_weather_alerts, ], ) await worker.run()if __name__ == "__main__": asyncio.run(main()) Initiate an interaction with the agent[​](https://docs.temporal.io/ai-cookbook/tool-call-openai-python#initiate-an-interaction-with-the-agent "Direct link to Initiate an interaction with the agent") ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- In order to interact with this simple AI agent, we create a Temporal client and execute a workflow. `start_workflow.py` import asyncioimport sysfrom temporalio.client import Clientfrom workflows.get_weather_workflow import ToolCallingWorkflowfrom temporalio.contrib.pydantic import pydantic_data_converterasync def main(): client = await Client.connect( "localhost:7233", data_converter=pydantic_data_converter, ) query = sys.argv[1] if len(sys.argv) > 1 else "Hello, how are you?" # Submit the Tool Calling workflow for execution result = await client.execute_workflow( ToolCallingWorkflow.run, query, id="my-workflow-id", task_queue="tool-calling-python-task-queue", ) print(f"Result: {result}")if __name__ == "__main__": asyncio.run(main()) Running[​](https://docs.temporal.io/ai-cookbook/tool-call-openai-python#running "Direct link to Running") ---------------------------------------------------------------------------------------------------------- ### Start the Temporal Dev Server[​](https://docs.temporal.io/ai-cookbook/tool-call-openai-python#start-the-temporal-dev-server "Direct link to Start the Temporal Dev Server") temporal server start-dev ### Install dependencies[​](https://docs.temporal.io/ai-cookbook/tool-call-openai-python#install-dependencies "Direct link to Install dependencies") From this directory: uv sync ### Run the worker[​](https://docs.temporal.io/ai-cookbook/tool-call-openai-python#run-the-worker "Direct link to Run the worker") First set the `OPENAI_API_KEY` environment variable and then: uv run python -m worker ### Initiate an interaction with the agent[​](https://docs.temporal.io/ai-cookbook/tool-call-openai-python#initiate-an-interaction-with-the-agent-1 "Direct link to Initiate an interaction with the agent") This user input should not result in any tool call uv run python -m start_workflow "Tell me about recursion in programming." This user input should invoke the tool and respond with current weather alerts for California. uv run python -m start_workflow "Are there any weather alerts in California?" --- # One doc tagged with "AI Agents" | Temporal Platform Documentation [Skip to main content](https://docs.temporal.io/tags/ai-agents#__docusaurus_skipToContent_fallback) [Plugins guide\ -------------](https://docs.temporal.io/develop/plugins-guide) Best practices for creating plugins for AI Agents --- # One doc tagged with "AI SDK" | Temporal Platform Documentation [Skip to main content](https://docs.temporal.io/tags/ai-sdk#__docusaurus_skipToContent_fallback) [AI SDK by Vercel integration\ ----------------------------](https://docs.temporal.io/develop/typescript/integrations/ai-sdk) Implement AI applications in TypeScript using the Temporal TypeScript SDK and the AI SDK. --- # Retry Policy from HTTP Responses [Skip to main content](https://docs.temporal.io/ai-cookbook/http-retry-enhancement-python#__docusaurus_skipToContent_fallback) On this page This recipe extends the [Basic Example](https://docs.temporal.io/ai-cookbook/hello-world-openai-responses-python) to show how to extract retry information from HTTP response headers and make it available to Temporal's retry mechanisms. HTTP response codes and headers on API calls have implications for retry behavior. For example, an HTTP `404 Not Found` generally represents an application-level error that should not be retried. By contrast, a `500 Internal Server Error` is typically transient, so should be retried. Servers can also set the `Retry-After` header to tell the client when to retry. This recipe introduces a utility function that processes the HTTP response and populates a Temporal `ApplicationError` to provide inputs to the retry mechanism. Temporal combines this information with other configuration, such as timeouts and exponential backoff, to implement the complete retry policy. Generate Temporal ApplicationErrors from HTTP responses[​](https://docs.temporal.io/ai-cookbook/http-retry-enhancement-python#generate-temporal-applicationerrors-from-http-responses "Direct link to Generate Temporal ApplicationErrors from HTTP responses") ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- We introduce a utility function that takes an `httpx.Response` object and returns a Temporal `ApplicationError` with two key fields populated: `non-retryable` and `next_retry_delay`. The `non-retryable` is determined by categorizing the HTTP status codes. The `X-Should-Retry` HTTP response header, when present, overrides the status code. **Example Retryable Status Codes:** * **408 Request Timeout** → Retry because server is unresponsive, which can have many causes * **409 Conflict** → Retry when resource is temporarily locked or in use * **429 Too Many Requests** → Retry after rate limit cooldown (respect `Retry-After` header when available) * **500 Internal Server Error** → Retry for temporary server issues * **502 Bad Gateway** → Retry when upstream server is temporarily unavailable * **503 Service Unavailable** → Retry when service is temporarily overloaded * **504 Gateway Timeout** → Retry when upstream server times out **Example Non-Retryable Status Codes:** * **400 Bad Request** → Do not retry - fix request format/parameters * **401 Unauthorized** → Do not retry - provide valid authentication * **403 Forbidden** → Do not retry - insufficient permissions * **404 Not Found** → Do not retry - resource does not exist * **422 Unprocessable Entity** → Do not retry - fix request validation errors * **Other 4xx Client Errors** → Do not retry - client-side issues need fixing * **2xx Success** → Do not expect to see this - call succeeded * **3xx Redirects** → Do not expect to see this - typically handled by httpx (with `follow_redirects=True`) If the error is retryable and if the `Retry-After` header is present, we parse it to set the retry delay. This implementation duplicates logic present in the [OpenAI Python API Library](https://github.com/openai/openai-python) , where it is part of the code generated by [Stainless](https://www.stainless.com/) . Duplicating the logic makes sense because it is not accessible via the public library interface and because it applies to HTTP APIs in general, not just the OpenAI API. _File: util/http\_retry.py_ import email.utilsimport timefrom datetime import timedeltafrom temporalio.exceptions import ApplicationErrorfrom temporalio import workflowfrom typing import Optional, Tuplewith workflow.unsafe.imports_passed_through(): from httpx import Response, Headers# Adapted from the OpenAI Python client (https://github.com/openai/openai-python/blob/main/src/openai/_base_client.py)# which is generated by the Stainless SDK Generator.def _parse_retry_after_header(response_headers: Optional[Headers] = None) -> float | None: """Returns a float of the number of seconds (not milliseconds) to wait after retrying, or None if unspecified. About the Retry-After header: https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/Retry-After See also https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/Retry-After#syntax """ if response_headers is None: return None # First, try the non-standard `retry-after-ms` header for milliseconds, # which is more precise than integer-seconds `retry-after` try: retry_ms_header = response_headers.get("retry-after-ms", None) return float(retry_ms_header) / 1000 except (TypeError, ValueError): pass # Next, try parsing `retry-after` header as seconds (allowing nonstandard floats). retry_header = response_headers.get("retry-after") try: # note: the spec indicates that this should only ever be an integer # but if someone sends a float there's no reason for us to not respect it return float(retry_header) except (TypeError, ValueError): pass # Last, try parsing `retry-after` as a date. retry_date_tuple = email.utils.parsedate_tz(retry_header) if retry_date_tuple is None: return None retry_date = email.utils.mktime_tz(retry_date_tuple) return float(retry_date - time.time())def _should_retry(response: Response) -> Tuple[bool, str]: # Note: this is not a standard header should_retry_header = response.headers.get("x-should-retry") # If the server explicitly says whether or not to retry, obey. if should_retry_header == "true": return True, f"Server requested retry via x-should-retry=true header (HTTP {response.status_code})" if should_retry_header == "false": return False, f"Server prevented retry via x-should-retry=false header (HTTP {response.status_code})" # Retry on request timeouts. if response.status_code == 408: return True, f"HTTP request timeout ({response.status_code}), will retry with backoff" # Retry on lock timeouts. if response.status_code == 409: return True, f"HTTP conflict/lock timeout ({response.status_code}), will retry with backoff" # Retry on rate limits. if response.status_code == 429: return True, f"HTTP rate limit exceeded ({response.status_code}), will retry with backoff" # Retry internal errors. if response.status_code >= 500: return True, f"HTTP server error ({response.status_code}), will retry with backoff" return False, f"HTTP client error ({response.status_code}), not retrying - check your request"def http_response_to_application_error(response: Response) -> ApplicationError: """Transform HTTP response into Temporal ApplicationError for retry handling. This function implements generic HTTP retry logic based on status codes and headers. Args: response: The httpx.Response from a failed HTTP request Returns: ApplicationError: Always returns an ApplicationError configured for Temporal's retry system: - non_retryable: False for retryable errors, True for non-retryable - next_retry_delay: Server-provided delay hint (if valid) Note: Even when x-should-retry=true, this function returns an ApplicationError with non_retryable=False rather than raising an exception, for cleaner functional style. """ should_retry, retry_message = _should_retry(response) if should_retry: # Calculate the retry delay only when retrying retry_after = _parse_retry_after_header(response.headers) # Make sure that the retry delay is in a reasonable range if retry_after is not None and 0 < retry_after <= 60: retry_after = timedelta(seconds=retry_after) else: retry_after = None # Add delay info for rate limits if response.status_code == 429 and retry_after is not None: retry_message = f"HTTP rate limit exceeded (429) (server requested {retry_after.total_seconds():.1f}s delay), will retry with backoff" return ApplicationError( retry_message, non_retryable=False, next_retry_delay=retry_after, ) else: return ApplicationError( retry_message, non_retryable=True, next_retry_delay=None, ) Raise the exception from the Activity[​](https://docs.temporal.io/ai-cookbook/http-retry-enhancement-python#raise-the-exception-from-the-activity "Direct link to Raise the exception from the Activity") ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- When API calls fail, the OpenAI Client raises an `APIStatusError` exception which contains a `response` field, containing the underlying `httpx.Response` object. We use the `http_response_to_application_error` function defined above to translate this to a Temporal `ApplicationError`, which we re-throw to pass the retry information to Temporal. _File: activities/openai\_responses.py_ from temporalio import activityfrom openai import AsyncOpenAIfrom openai.types.responses import Responsefrom dataclasses import dataclassfrom util.http_retry import http_response_to_application_errorfrom openai import APIStatusError# Temporal best practice: Create a data structure to hold the request parameters.@dataclassclass OpenAIResponsesRequest: model: str instructions: str input: str@activity.defnasync def create(request: OpenAIResponsesRequest) -> Response: # Temporal best practice: Disable retry logic in OpenAI API client library. client = AsyncOpenAI(max_retries=0) try: resp = await client.responses.create( model=request.model, instructions=request.instructions, input=request.input, timeout=15, ) return resp except APIStatusError as e: raise http_response_to_application_error(e.response) from e Running[​](https://docs.temporal.io/ai-cookbook/http-retry-enhancement-python#running "Direct link to Running") ---------------------------------------------------------------------------------------------------------------- Start the Temporal Dev Server: temporal server start-dev Run the worker: uv run python -m worker Start execution: uv run python -m start_workflow --- # 12 docs tagged with "python" | Temporal Platform Documentation [Skip to main content](https://docs.temporal.io/ai-cookbook/tags/python#__docusaurus_skipToContent_fallback) [Basic Agentic Loop with Claude and Tool Calling\ -----------------------------------------------](https://docs.temporal.io/ai-cookbook/agentic-loop-tool-call-claude-python) A basic agentic loop using Claude (Anthropic) with tool calling. [Basic Agentic Loop with Tool Calling\ ------------------------------------](https://docs.temporal.io/ai-cookbook/agentic-loop-tool-call-openai-python) A basic agentic loop that invokes a dynamic set of tools. [Claim Check Pattern with Temporal\ ---------------------------------](https://docs.temporal.io/ai-cookbook/claim-check-pattern-python) Use the Claim Check pattern to handle large payloads to workflows and activities. [Deep Research\ -------------](https://docs.temporal.io/ai-cookbook/basic-openai-python) Build a simple deep research system embodying the standard deep research architecture. [Durable Agent with Tools - OpenAI Agents SDK\ --------------------------------------------](https://docs.temporal.io/ai-cookbook/openai-agents-sdk-python) Build a durable AI agent with OpenAI Agents SDK and Temporal that can intelligently choose tools to answer user questions [Durable MCP Weather Server\ --------------------------](https://docs.temporal.io/ai-cookbook/hello-world-durable-mcp-server) A durable MCP server that uses Temporal workflows for reliable execution of weather tools. [Hello World\ -----------](https://docs.temporal.io/ai-cookbook/hello-world-openai-responses-python) Simple example demonstrating how to call an LLM from Temporal using the OpenAI Python API library. [Hello World with LiteLLM\ ------------------------](https://docs.temporal.io/ai-cookbook/hello-world-litellm-python) Integrate LiteLLM into a Temporal Workflow in Python. [Human-in-the-Loop AI Agent\ --------------------------](https://docs.temporal.io/ai-cookbook/human-in-the-loop-python) Support human in the loop (HITL) in agentic flows. [Retry Policy from HTTP Responses\ --------------------------------](https://docs.temporal.io/ai-cookbook/http-retry-enhancement-python) Extract retry information from HTTP response headers and make it available to Temporal's retry mechanisms. [Structured Outputs with Temporal and OpenAI\ -------------------------------------------](https://docs.temporal.io/ai-cookbook/structured-output-openai-responses-python) Use Temporal and OpenAI Responses API to reliably request output conforming to a specific data structure. [Tool calling agent\ ------------------](https://docs.temporal.io/ai-cookbook/tool-call-openai-python) Build a simple, non-looping agent that gives agency to the LLM to choose tools, and then invokes chosen tools. --- # One doc tagged with "Braintrust" | Temporal Platform Documentation [Skip to main content](https://docs.temporal.io/tags/braintrust#__docusaurus_skipToContent_fallback) [Braintrust integration\ ----------------------](https://docs.temporal.io/develop/python/integrations/braintrust) Add LLM observability and prompt management to Python Workflows using the Temporal Python SDK and Braintrust. --- # One doc tagged with "Authentication" | Temporal Platform Documentation [Skip to main content](https://docs.temporal.io/tags/authentication#__docusaurus_skipToContent_fallback) [Get started with Temporal Cloud\ -------------------------------](https://docs.temporal.io/cloud/get-started/) Get started with Temporal Cloud. Sign up, verify your Account Owner or Global Admin role, set up authentication, create a Namespace, invite users, and connect your Temporal Client and Workers. --- # Claim Check Pattern with Temporal [Skip to main content](https://docs.temporal.io/ai-cookbook/claim-check-pattern-python#__docusaurus_skipToContent_fallback) On this page This recipe demonstrates how to use the Claim Check pattern to offload data from Temporal Server's Event History to external storage. This can be useful in conversational AI applications that include the full conversation history with each LLM call, creating large Event History that can exceed server size limits. This recipe includes: * A `PayloadCodec` ([docs](https://docs.temporal.io/payload-codec) ) that stores large payloads in S3 and replaces them with keys * A client [plugin](https://docs.temporal.io/develop/plugins-guide) that wires the codec into the Temporal data converter * A lightweight codec server for a better Web UI experience * An AI/RAG example workflow that demonstrates the pattern end-to-end How the Claim Check Pattern Works[​](https://docs.temporal.io/ai-cookbook/claim-check-pattern-python#how-the-claim-check-pattern-works "Direct link to How the Claim Check Pattern Works") ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Each Temporal Workflow has an associated Event History that is stored in Temporal Server and used to provide durable execution. When using the Claim Check pattern, we store the payload content of the Event in separate storage system, then store a reference to that storage in the Temporal Event History instead. The Claim Check Recipe implements a `PayloadCodec` that: 1. Encode: Replaces large payloads with unique keys and stores the original data in external storage (S3, Database, etc.) 2. Decode: Retrieves the original payload using the key when needed Workflows operate with small, lightweight keys while maintaining transparent access to full data through automatic encoding/decoding. Claim Check Codec Implementation[​](https://docs.temporal.io/ai-cookbook/claim-check-pattern-python#claim-check-codec-implementation "Direct link to Claim Check Codec Implementation") ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- The `ClaimCheckCodec` implements `PayloadCodec` and adds an inline threshold to keep small payloads inline. This avoids the latency costs of uploading/downloading the payload externally when it's not required. _File: codec/claim\_check.py_ import uuidimport loggingfrom typing import Iterable, Listimport aioboto3from botocore.exceptions import ClientErrorfrom temporalio.api.common.v1 import Payloadfrom temporalio.converter import PayloadCodeclogger = logging.getLogger(__name__)class ClaimCheckCodec(PayloadCodec): """PayloadCodec that implements the Claim Check pattern using S3 storage. This codec stores large payloads in S3 and replaces them with unique keys, allowing Temporal workflows to operate with lightweight references instead of large payload data. """ def __init__( self, bucket_name: str = "temporal-claim-check", endpoint_url: str = None, region_name: str = "us-east-1", max_inline_bytes: int = 20 * 1024, ): """Initialize the claim check codec with S3 connection details. Args: bucket_name: S3 bucket name for storing claim check data endpoint_url: S3 endpoint URL (for MinIO or other S3-compatible services) region_name: AWS region name max_inline_bytes: Payloads up to this size will be left inline """ self.bucket_name = bucket_name self.endpoint_url = endpoint_url self.region_name = region_name self.max_inline_bytes = max_inline_bytes self.session = aioboto3.Session() self._bucket_created = False async def _ensure_bucket_exists(self): """Ensure the S3 bucket exists, creating it if necessary.""" if self._bucket_created: return async with self.session.client( 's3', endpoint_url=self.endpoint_url, region_name=self.region_name ) as s3_client: try: await s3_client.head_bucket(Bucket=self.bucket_name) except ClientError as e: error_code = e.response['Error']['Code'] if error_code in ['404', 'NoSuchBucket']: try: await s3_client.create_bucket(Bucket=self.bucket_name) except ClientError as create_error: # Handle bucket already exists race condition if create_error.response['Error']['Code'] not in ['BucketAlreadyExists', 'BucketAlreadyOwnedByYou']: raise create_error elif error_code not in ['403', 'Forbidden']: raise e self._bucket_created = True async def encode(self, payloads: Iterable[Payload]) -> List[Payload]: """Replace large payloads with keys and store original data in S3. Args: payloads: Iterable of payloads to encode Returns: List of encoded payloads (keys for claim-checked payloads) """ await self._ensure_bucket_exists() out: List[Payload] = [] for payload in payloads: # Leave small payloads inline to improve debuggability and avoid unnecessary indirection data_size = len(payload.data or b"") if data_size <= self.max_inline_bytes: out.append(payload) continue encoded = await self.encode_payload(payload) out.append(encoded) return out async def decode(self, payloads: Iterable[Payload]) -> List[Payload]: """Retrieve original payloads from S3 using stored keys. Args: payloads: Iterable of payloads to decode Returns: List of decoded payloads (original data retrieved from S3) Raises: ValueError: If a claim check key is not found in S3 """ await self._ensure_bucket_exists() out: List[Payload] = [] for payload in payloads: if payload.metadata.get("temporal.io/claim-check-codec", b"").decode() != "v1": # Not a claim-checked payload, pass through unchanged out.append(payload) continue s3_key = payload.data.decode("utf-8") stored_data = await self.get_payload_from_s3(s3_key) if stored_data is None: raise ValueError(f"Claim check key not found in S3: {s3_key}") original_payload = Payload.FromString(stored_data) out.append(original_payload) return out async def encode_payload(self, payload: Payload) -> Payload: """Store payload in S3 and return a key-based payload. Args: payload: Original payload to store Returns: Payload containing only the S3 key """ await self._ensure_bucket_exists() key = str(uuid.uuid4()) serialized_data = payload.SerializeToString() # Store the original payload data in S3 async with self.session.client( 's3', endpoint_url=self.endpoint_url, region_name=self.region_name ) as s3_client: await s3_client.put_object( Bucket=self.bucket_name, Key=key, Body=serialized_data ) # Return a lightweight payload containing only the key return Payload( metadata={ "encoding": b"claim-checked", "temporal.io/claim-check-codec": b"v1", }, data=key.encode("utf-8"), ) async def get_payload_from_s3(self, s3_key: str) -> bytes: """Retrieve payload data from S3. Args: s3_key: S3 object key Returns: Raw payload data bytes, or None if not found """ try: async with self.session.client( 's3', endpoint_url=self.endpoint_url, region_name=self.region_name ) as s3_client: response = await s3_client.get_object( Bucket=self.bucket_name, Key=s3_key ) return await response['Body'].read() except ClientError as e: if e.response['Error']['Code'] == 'NoSuchKey': return None raise e ### Inline payload threshold[​](https://docs.temporal.io/ai-cookbook/claim-check-pattern-python#inline-payload-threshold "Direct link to Inline payload threshold") * Default: 20KB * Where configured: `ClaimCheckCodec(max_inline_bytes=20 * 1024)` in `codec/claim_check.py` * Change by passing a different `max_inline_bytes` when constructing `ClaimCheckCodec` Claim Check Plugin[​](https://docs.temporal.io/ai-cookbook/claim-check-pattern-python#claim-check-plugin "Direct link to Claim Check Plugin") ---------------------------------------------------------------------------------------------------------------------------------------------- The `ClaimCheckPlugin` integrates the codec with the Temporal client configuration. _File: codec/plugin.py_ import osfrom temporalio.plugin import SimplePluginfrom temporalio.converter import DataConverterfrom .claim_check import ClaimCheckCodecclass ClaimCheckPlugin(SimplePlugin): """Temporal plugin that integrates the Claim Check codec with client configuration.""" def __init__(self): """Initialize the plugin with S3 connection configuration.""" super().__init__( name="claim-check", data_converter=DataConverter( payload_codec=ClaimCheckCodec( bucket_name=os.getenv("S3_BUCKET_NAME", "temporal-claim-check"), endpoint_url=os.getenv("S3_ENDPOINT_URL"), region_name=os.getenv("AWS_REGION", "us-east-1"), ), ), ) Example: AI / RAG Workflow using Claim Check[​](https://docs.temporal.io/ai-cookbook/claim-check-pattern-python#example-ai--rag-workflow-using-claim-check "Direct link to Example: AI / RAG Workflow using Claim Check") -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- This example ingests a large text, performs lightweight lexical retrieval, and answers a question with an LLM. Large intermediates (chunks, scores) are kept out of Temporal payloads via the Claim Check codec. Only the small final answer is returned inline. ### Shared Models[​](https://docs.temporal.io/ai-cookbook/claim-check-pattern-python#shared-models "Direct link to Shared Models") _File: shared/models.py_ from dataclasses import dataclassfrom typing import List, Dict, Any@dataclassclass IngestRequest: document_bytes: bytes filename: str mime_type: str chunk_size: int = 1500 chunk_overlap: int = 200 embedding_model: str = "text-embedding-3-large"@dataclassclass IngestResult: chunk_texts: List[str] metadata: Dict[str, Any]@dataclassclass RagRequest: question: str top_k: int = 4 generation_model: str = "gpt-4o-mini"@dataclassclass RagAnswer: answer: str sources: List[Dict[str, Any]] ### Activities[​](https://docs.temporal.io/ai-cookbook/claim-check-pattern-python#activities "Direct link to Activities") _File: activities/ai\_claim\_check.py_ from typing import Listfrom temporalio import activityfrom shared.models import IngestRequest, IngestResult, RagRequest, RagAnswerdef _split_text(text: str, chunk_size: int, overlap: int) -> List[str]: chunks: List[str] = [] start = 0 n = len(text) while start < n: end = min(n, start + chunk_size) chunks.append(text[start:end]) if end >= n: break start = max(end - overlap, start + 1) return chunks@activity.defnasync def ingest_document(req: IngestRequest) -> IngestResult: # Convert bytes to text. For PDFs/audio/images, integrate proper extractors. if req.mime_type != "text/plain": raise ValueError(f"Unsupported MIME type: {req.mime_type}") text = req.document_bytes.decode("utf-8", errors="ignore") chunks = _split_text(text, req.chunk_size, req.chunk_overlap) return IngestResult( chunk_texts=chunks, metadata={ "filename": req.filename, "mime_type": req.mime_type, "chunk_count": len(chunks), }, )@activity.defnasync def rag_answer(req: RagRequest, ingest_result: IngestResult) -> RagAnswer: # Import heavy dependencies inside the function, not at module level # This prevents NumPy from being loaded into the workflow sandbox from openai import AsyncOpenAI from rank_bm25 import BM25Okapi client = AsyncOpenAI(max_retries=0) # Lexical retrieval using BM25 over chunk texts # Simple whitespace tokenization tokenized_corpus: List[List[str]] = [chunk.split() for chunk in ingest_result.chunk_texts] bm25 = BM25Okapi(tokenized_corpus) tokenized_query = req.question.split() scores = bm25.get_scores(tokenized_query) # Get top-k indices by score top_indices = sorted(range(len(scores)), key=lambda i: scores[i], reverse=True)[: max(1, req.top_k)] contexts = [ingest_result.chunk_texts[i] for i in top_indices] sources = [{"chunk_index": i, "score": float(scores[i])} for i in top_indices] prompt = ( "Use the provided context chunks to answer the question.\n\n" f"Question: {req.question}\n\n" "Context:\n" + "\n---\n".join(contexts) + "\n\nAnswer:" ) chat = await client.chat.completions.create( model=req.generation_model, messages=[{"role": "user", "content": prompt}], temperature=0.2, ) answer = chat.choices[0].message.content.strip() return RagAnswer(answer=answer, sources=sources) ### Workflow[​](https://docs.temporal.io/ai-cookbook/claim-check-pattern-python#workflow "Direct link to Workflow") _File: workflows/ai\_rag\_workflow.py_ from temporalio import workflowfrom datetime import timedeltafrom shared.models import IngestRequest, IngestResult, RagRequest, RagAnswerfrom activities.ai_claim_check import ingest_document, rag_answer@workflow.defnclass AiRagWorkflow: @workflow.run async def run(self, document_bytes: bytes, filename: str, mime_type: str, question: str) -> RagAnswer: ingest: IngestResult = await workflow.execute_activity( ingest_document, IngestRequest( document_bytes=document_bytes, filename=filename, mime_type=mime_type, ), start_to_close_timeout=timedelta(minutes=10), summary="Ingest and embed large document", ) answer: RagAnswer = await workflow.execute_activity( rag_answer, args=[ RagRequest(question=question), ingest, ], start_to_close_timeout=timedelta(minutes=5), summary="RAG answer using embedded chunks", ) return answer Running[​](https://docs.temporal.io/ai-cookbook/claim-check-pattern-python#running "Direct link to Running") ------------------------------------------------------------------------------------------------------------- ### Prerequisites[​](https://docs.temporal.io/ai-cookbook/claim-check-pattern-python#prerequisites "Direct link to Prerequisites") * MinIO server (for local testing) or AWS S3 access (for production) * Temporal dev server * Python 3.9+ ### Configuration[​](https://docs.temporal.io/ai-cookbook/claim-check-pattern-python#configuration "Direct link to Configuration") Set environment variables to configure S3 and OpenAI: # For MinIO (recommended for local testing)export S3_ENDPOINT_URL=http://localhost:9000export S3_BUCKET_NAME=temporal-claim-checkexport AWS_ACCESS_KEY_ID=minioadminexport AWS_SECRET_ACCESS_KEY=minioadminexport AWS_REGION=us-east-1# For production AWS S3# export S3_BUCKET_NAME=your-bucket-name# export AWS_REGION=us-east-1# you may use access keys or...# export AWS_ACCESS_KEY_ID=your-access-key# export AWS_SECRET_ACCESS_KEY=your-secret-key# ... sso# export AWS_profile=your-profile# aws sso login --profile your-profileexport OPENAI_API_KEY=your_key_here ### Option 1: MinIO (Recommended for Testing)[​](https://docs.temporal.io/ai-cookbook/claim-check-pattern-python#option-1-minio-recommended-for-testing "Direct link to Option 1: MinIO (Recommended for Testing)") 1. Start MinIO: docker run -d -p 9000:9000 -p 9001:9001 \ --name minio \ -e "MINIO_ROOT_USER=minioadmin" \ -e "MINIO_ROOT_PASSWORD=minioadmin" \ quay.io/minio/minio server /data --console-address ":9001" The bucket will be auto-created by the code. You can view stored objects in the MinIO web console at [http://localhost:9001](http://localhost:9001/) (credentials: minioadmin/minioadmin). 2. Start Temporal dev server: temporal server start-dev 3. Run the worker: uv run python -m worker 4. Start execution: uv run python -m start_workflow ### Option 2: AWS S3 (Production)[​](https://docs.temporal.io/ai-cookbook/claim-check-pattern-python#option-2-aws-s3-production "Direct link to Option 2: AWS S3 (Production)") 1. Create an S3 bucket in your AWS account 2. Configure AWS credentials (via AWS CLI, environment variables, IAM roles or sso) 3. Set the environment variables for your bucket 4. Follow steps 2-4 from Option 1 ### Toggle Claim Check (optional)[​](https://docs.temporal.io/ai-cookbook/claim-check-pattern-python#toggle-claim-check-optional "Direct link to Toggle Claim Check (optional)") To demonstrate payload size failures without claim check, you can disable it in your local wiring (e.g., omit the plugin/codec) and re-run. With claim check disabled, large payloads may exceed Temporal's default payload size limits and fail. Codec Server for Web UI[​](https://docs.temporal.io/ai-cookbook/claim-check-pattern-python#codec-server-for-web-ui "Direct link to Codec Server for Web UI") ------------------------------------------------------------------------------------------------------------------------------------------------------------- When claim check is enabled, the Web UI would otherwise show opaque keys. This codec server shows helpful text with a link to view the raw data on demand. _File: codec/codec\_server.py_ from functools import partialfrom typing import Awaitable, Callable, Iterable, Listimport jsonimport osfrom aiohttp import hdrs, webfrom google.protobuf import json_formatfrom temporalio.api.common.v1 import Payload, Payloadsfrom .claim_check import ClaimCheckCodecdef build_codec_server() -> web.Application: # Create codec with environment variable configuration (same as plugin) codec = ClaimCheckCodec( bucket_name=os.getenv("S3_BUCKET_NAME", "temporal-claim-check"), endpoint_url=os.getenv("S3_ENDPOINT_URL"), region_name=os.getenv("AWS_REGION", "us-east-1") ) # Configure Web UI endpoint temporal_web_url = os.getenv("TEMPORAL_WEB_URL", "http://localhost:8233") # Configure codec server endpoint for viewing raw data codec_server_url = os.getenv("CODEC_SERVER_URL", "http://localhost:8081") # CORS handler - needed because Temporal Web UI runs on a different port/domain # and the browser blocks cross-origin requests by default; CORS headers allow these requests async def cors_options(req: web.Request) -> web.Response: resp = web.Response() if req.headers.get(hdrs.ORIGIN) == temporal_web_url: resp.headers[hdrs.ACCESS_CONTROL_ALLOW_ORIGIN] = temporal_web_url resp.headers[hdrs.ACCESS_CONTROL_ALLOW_METHODS] = "POST" resp.headers[hdrs.ACCESS_CONTROL_ALLOW_HEADERS] = "content-type,x-namespace" return resp # Custom decode function that provides URLs to view raw data async def decode_with_urls(payloads: Iterable[Payload]) -> List[Payload]: """Decode claim check payloads and provide URLs to view the raw data.""" out: List[Payload] = [] for payload in payloads: if payload.metadata.get("temporal.io/claim-check-codec", b"").decode() != "v1": # Not a claim-checked payload, pass through unchanged out.append(payload) continue # Get the S3 key s3_key = payload.data.decode("utf-8") # Return simple text with link - no data reading link_text = f"Claim check data (key: {s3_key}) - View at: {codec_server_url}/view/{s3_key}" summary_payload = Payload( metadata={"encoding": b"json/plain"}, data=json.dumps({"text": link_text}).encode("utf-8") ) out.append(summary_payload) return out # Endpoint to view raw payload data async def view_raw_data(req: web.Request) -> web.Response: """View the raw payload data for a given S3 key.""" s3_key = req.match_info['key'] try: stored_data = await codec.get_payload_from_s3(s3_key) if stored_data is None: return web.Response( text=json.dumps({"error": f"Key not found: {s3_key}"}), content_type="application/json", status=404 ) # Parse and return the original payload original_payload = Payload.FromString(stored_data) # Try to decode as text, fall back to base64 for binary data try: data_text = original_payload.data.decode("utf-8") return web.Response( text=data_text, content_type="text/plain" ) except UnicodeDecodeError: import base64 data_b64 = base64.b64encode(original_payload.data).decode("utf-8") return web.Response( text=f"Binary data (base64):\n{data_b64}", content_type="text/plain" ) except Exception as e: return web.Response( text=json.dumps({"error": f"Failed to retrieve data: {str(e)}"}), content_type="application/json", status=500 ) # General purpose payloads-to-payloads async def apply( fn: Callable[[Iterable[Payload]], Awaitable[List[Payload]]], req: web.Request ) -> web.Response: # Read payloads as JSON assert req.content_type == "application/json" data = await req.read() payloads = json_format.Parse(data.decode("utf-8"), Payloads()) # Apply payloads = Payloads(payloads=await fn(payloads.payloads)) # Apply CORS and return JSON resp = await cors_options(req) resp.content_type = "application/json" resp.text = json_format.MessageToJson(payloads) return resp # Build app app = web.Application() app.add_routes( [ web.post("/encode", partial(apply, codec.encode)), web.post("/decode", partial(apply, decode_with_urls)), web.get("/view/{key}", view_raw_data), web.options("/decode", cors_options), ] ) return appif __name__ == "__main__": web.run_app(build_codec_server(), host="127.0.0.1", port=8081) ### Running the Codec Server[​](https://docs.temporal.io/ai-cookbook/claim-check-pattern-python#running-the-codec-server "Direct link to Running the Codec Server") uv run python -m codec.codec_server Then [configure the Web UI to use the codec server](https://docs.temporal.io/production-deployment/data-encryption#set-your-codec-server-endpoints-with-web-ui-and-cli) . ### What it shows[​](https://docs.temporal.io/ai-cookbook/claim-check-pattern-python#what-it-shows "Direct link to What it shows") Instead of raw keys: abc123-def4-5678-9abc-def012345678 You will see text like: "Claim check data (key: abc123-def4-5678-9abc-def012345678) - View at: http://localhost:8081/view/abc123-def4-5678-9abc-def012345678" ### Endpoints[​](https://docs.temporal.io/ai-cookbook/claim-check-pattern-python#endpoints "Direct link to Endpoints") * `POST /encode`: Encodes payloads using the claim check codec * `POST /decode`: Returns helpful text with S3 key and view URL (no data reads) * `GET /view/{key}`: Serves raw payload data for inspection * `OPTIONS /decode`: Handles CORS preflight requests The server also includes CORS handling for the local Web UI. --- # 2 docs tagged with "Billing" | Temporal Platform Documentation [Skip to main content](https://docs.temporal.io/tags/billing#__docusaurus_skipToContent_fallback) [Cloud Billing API\ -----------------](https://docs.temporal.io/cloud/billing-api) The Temporal Cloud Billing API provides namespace-level cost attribution through on-demand billing reports. [Notifications\ -------------](https://docs.temporal.io/cloud/notifications) Receive updates about Temporal Cloud including when certificates will expire, billing updates, and when a failover has completed. --- # 3 docs tagged with "Agent Frameworks" | Temporal Platform Documentation [Skip to main content](https://docs.temporal.io/tags/agent-frameworks#__docusaurus_skipToContent_fallback) [AI integrations\ ---------------](https://docs.temporal.io/develop/integrations) AI framework integrations available for Temporal SDKs. [AI integrations\ ---------------](https://docs.temporal.io/develop/python/integrations/) Integrations with other tools and services. [AI integrations\ ---------------](https://docs.temporal.io/develop/typescript/integrations/) Integrations with other tools and services. --- # 3 docs tagged with "AI Frameworks" | Temporal Platform Documentation [Skip to main content](https://docs.temporal.io/tags/ai-frameworks#__docusaurus_skipToContent_fallback) [AI integrations\ ---------------](https://docs.temporal.io/develop/integrations) AI framework integrations available for Temporal SDKs. [AI integrations\ ---------------](https://docs.temporal.io/develop/python/integrations/) Integrations with other tools and services. [AI integrations\ ---------------](https://docs.temporal.io/develop/typescript/integrations/) Integrations with other tools and services. --- # One doc tagged with "Context Propagation" | Temporal Platform Documentation [Skip to main content](https://docs.temporal.io/tags/context-propagation#__docusaurus_skipToContent_fallback) [Context Propagation - Go SDK\ ----------------------------](https://docs.temporal.io/develop/go/best-practices/context-propagation) How to propagate custom key-value data across Workflow, Activity, and Child Workflow boundaries using the Temporal Go SDK. --- # 3 docs tagged with "AWS" | Temporal Platform Documentation [Skip to main content](https://docs.temporal.io/tags/aws#__docusaurus_skipToContent_fallback) [Audit Logs - AWS Kinesis\ ------------------------](https://docs.temporal.io/cloud/audit-logs-aws) Audit Logs in Temporal Cloud provides forensic information, integrating with AWS Kinesis for secure data handling and supporting key Admin and API Key operations. This streamlines audit and compliance processes. [AWS PrivateLink connectivity\ ----------------------------](https://docs.temporal.io/cloud/connectivity/aws-connectivity) Connect to Temporal Cloud using AWS PrivateLink [Exporting Workflow Event History to AWS S3\ ------------------------------------------](https://docs.temporal.io/cloud/export/aws-export-s3) Export Workflow History to AWS S3 --- # 2 docs tagged with "Capacity Modes" | Temporal Platform Documentation [Skip to main content](https://docs.temporal.io/tags/capacity-modes#__docusaurus_skipToContent_fallback) [Capacity modes\ --------------](https://docs.temporal.io/cloud/capacity-modes) Control how limits are assigned to a Namespace with Capacity Modes [Managing Actions per Second (APS) limits in Temporal Cloud\ ----------------------------------------------------------](https://docs.temporal.io/best-practices/managing-aps-limits) Control how limits are assigned to a Namespace with Capacity Modes --- # One doc tagged with "Deployment" | Temporal Platform Documentation [Skip to main content](https://docs.temporal.io/tags/deployment#__docusaurus_skipToContent_fallback) [Safely deploying changes to Workflow code\ -----------------------------------------](https://docs.temporal.io/develop/safe-deployments) Safely deploy changes to existing Workflow code by validating first for determinism errors before deploying to production. --- # One doc tagged with "Deploy" | Temporal Platform Documentation [Skip to main content](https://docs.temporal.io/tags/deploy#__docusaurus_skipToContent_fallback) [Quick launch - Deploying your Workers on Amazon EKS\ ---------------------------------------------------](https://docs.temporal.io/production-deployment/worker-deployments/deploy-workers-to-aws-eks) Deploy a Temporal Worker on Amazon Elastic Kubernetes Service (EKS) using the Python SDK. --- # 4 docs tagged with "API Keys" | Temporal Platform Documentation [Skip to main content](https://docs.temporal.io/tags/api-keys#__docusaurus_skipToContent_fallback) [Cloud automation - Temporal feature\ -----------------------------------](https://docs.temporal.io/evaluate/development-production-features/cloud-automation) Explore how cloud automation simplifies cloud management and enhances security through APIs, Terraform, and CLI. [Manage API keys\ ---------------](https://docs.temporal.io/cloud/api-keys) Temporal Cloud supports secure programmatic access through API key authentication, ensuring user-level and RBAC-based authorization. [Manage service accounts\ -----------------------](https://docs.temporal.io/cloud/service-accounts) Temporal Cloud introduces Service Accounts for machine authentication, enabling non-human identities to interact with Temporal Cloud. Manage Service Accounts via Cloud UI or CLI for secure, automated operations. [tcld apikey command reference\ -----------------------------](https://docs.temporal.io/cloud/tcld/apikey) Manage your API Keys in Temporal Cloud with tcld commands. Create, retrieve, list, delete, disable, and enable API Keys effortlessly using tcld apikey commands. --- # 2 docs tagged with "Configuration" | Temporal Platform Documentation [Skip to main content](https://docs.temporal.io/tags/configuration#__docusaurus_skipToContent_fallback) [Environment configuration\ -------------------------](https://docs.temporal.io/develop/environment-configuration) Configure Temporal Clients using environment variables and TOML configuration files [Environment configuration\ -------------------------](https://docs.temporal.io/references/client-environment-configuration) Reference for configuring Temporal Clients using environment variables and TOML configuration files. --- # One doc tagged with "design-patterns" | Temporal Platform Documentation [Skip to main content](https://docs.temporal.io/tags/design-patterns#__docusaurus_skipToContent_fallback) [Temporal use cases and design patterns\ --------------------------------------](https://docs.temporal.io/evaluate/use-cases-design-patterns) Discover various use cases and architectural design patterns for implementing Workflows with Temporal. --- # One doc tagged with "enable-nexus" | Temporal Platform Documentation [Skip to main content](https://docs.temporal.io/tags/enable-nexus#__docusaurus_skipToContent_fallback) [Self-hosted Temporal Nexus\ --------------------------](https://docs.temporal.io/production-deployment/self-hosted-guide/nexus) Use Nexus in your self-hosted Temporal Service. --- # One doc tagged with "Error handling" | Temporal Platform Documentation [Skip to main content](https://docs.temporal.io/tags/error-handling#__docusaurus_skipToContent_fallback) [Error handling - Go SDK\ -----------------------](https://docs.temporal.io/develop/go/best-practices/error-handling) How to handle errors using the Temporal Go SDK. --- # One doc tagged with "Development Server" | Temporal Platform Documentation [Skip to main content](https://docs.temporal.io/tags/development-server#__docusaurus_skipToContent_fallback) [Temporal CLI server command reference\ -------------------------------------](https://docs.temporal.io/cli/server) Manage your Temporal Server easily with CLI commands. Start a local server using \`temporal server start-dev\` and access the Web UI at http://localhost:8233. Customize with multiple options. --- # One doc tagged with "Features" | Temporal Platform Documentation [Skip to main content](https://docs.temporal.io/tags/features#__docusaurus_skipToContent_fallback) [Job Queue\ ---------](https://docs.temporal.io/evaluate/development-production-features/job-queue) Standalone Activities adds the ability to execute any Temporal Activity as a top-level primitive without the full overhead of a Workflow. --- # One doc tagged with "evaluate temporal" | Temporal Platform Documentation [Skip to main content](https://docs.temporal.io/tags/evaluate-temporal#__docusaurus_skipToContent_fallback) [Understanding Temporal\ ----------------------](https://docs.temporal.io/evaluate/understanding-temporal) This page provides a short overview of how Temporal works. --- # 2 docs tagged with "developer guide" | Temporal Platform Documentation [Skip to main content](https://docs.temporal.io/tags/developer-guide#__docusaurus_skipToContent_fallback) [Set up your local with the .NET SDK\ -----------------------------------](https://docs.temporal.io/develop/dotnet/set-up-your-local-dotnet) Configure your local development environment to get started developing with Temporal [Set up your local with the Typescript SDK\ -----------------------------------------](https://docs.temporal.io/develop/typescript/set-up-your-local-typescript) Configure your local development environment to get started developing with Temporal --- # 2 docs tagged with "Environment Variables" | Temporal Platform Documentation [Skip to main content](https://docs.temporal.io/tags/environment-variables#__docusaurus_skipToContent_fallback) [Environment configuration\ -------------------------](https://docs.temporal.io/develop/environment-configuration) Configure Temporal Clients using environment variables and TOML configuration files [Environment configuration\ -------------------------](https://docs.temporal.io/references/client-environment-configuration) Reference for configuring Temporal Clients using environment variables and TOML configuration files. --- # One doc tagged with "Logging" | Temporal Platform Documentation [Skip to main content](https://docs.temporal.io/tags/logging#__docusaurus_skipToContent_fallback) [Observability - Temporal feature\ --------------------------------](https://docs.temporal.io/evaluate/development-production-features/observability) Explore the observability and visibility features of Temporal, including Metrics, Tracing, Logging, and Visibility. --- # One doc tagged with "Kubernetes" | Temporal Platform Documentation [Skip to main content](https://docs.temporal.io/tags/kubernetes#__docusaurus_skipToContent_fallback) [Quick launch - Deploying your Workers on Amazon EKS\ ---------------------------------------------------](https://docs.temporal.io/production-deployment/worker-deployments/deploy-workers-to-aws-eks) Deploy a Temporal Worker on Amazon Elastic Kubernetes Service (EKS) using the Python SDK. --- # 6 docs tagged with "API" | Temporal Platform Documentation [Skip to main content](https://docs.temporal.io/tags/api#__docusaurus_skipToContent_fallback) [Account access\ --------------](https://docs.temporal.io/cloud/manage-access/) Manage access to your Temporal Cloud account [Cloud Billing API\ -----------------](https://docs.temporal.io/cloud/billing-api) The Temporal Cloud Billing API provides namespace-level cost attribution through on-demand billing reports. [Cloud Ops API\ -------------](https://docs.temporal.io/ops) The Temporal Cloud Operations API (Cloud Ops) allows programmatic management of Temporal Cloud control plane resources. [Get started with Temporal Cloud\ -------------------------------](https://docs.temporal.io/cloud/get-started/) Get started with Temporal Cloud. Sign up, verify your Account Owner or Global Admin role, set up authentication, create a Namespace, invite users, and connect your Temporal Client and Workers. [Manage API keys\ ---------------](https://docs.temporal.io/cloud/api-keys) Temporal Cloud supports secure programmatic access through API key authentication, ensuring user-level and RBAC-based authorization. [Server frontend API reference\ -----------------------------](https://docs.temporal.io/self-hosted-guide/server-frontend-api-reference) Easily interact with the Temporal Server via Client SDKs or CLI, or use the gRPC API for Workflow operations. Access code examples and API docs at api-docs.temporal.io. --- # One doc tagged with "Migration" | Temporal Platform Documentation [Skip to main content](https://docs.temporal.io/tags/migration#__docusaurus_skipToContent_fallback) [Migrate between regions\ -----------------------](https://docs.temporal.io/cloud/migrate/migrate-within-cloud) Use Temporal Cloud's High Availability features to migrate between regions. --- # 3 docs tagged with "Export Workflow History" | Temporal Platform Documentation [Skip to main content](https://docs.temporal.io/tags/export-workflow-history#__docusaurus_skipToContent_fallback) [Exporting Workflow Event History to AWS S3\ ------------------------------------------](https://docs.temporal.io/cloud/export/aws-export-s3) Export Workflow History to AWS S3 [Exporting Workflow Event History to GCS\ ---------------------------------------](https://docs.temporal.io/cloud/export/gcp-export-gcs) Export Workflow History to GCS [Workflow History Export\ -----------------------](https://docs.temporal.io/cloud/export) Workflow History Export in Temporal Cloud lets users export Closed Workflow Histories to an object storage for compliance and analytics. Configure via Cloud UI or tcld. --- # 2 docs tagged with "Interceptors" | Temporal Platform Documentation [Skip to main content](https://docs.temporal.io/tags/interceptors#__docusaurus_skipToContent_fallback) [Interceptors - Python SDK\ -------------------------](https://docs.temporal.io/develop/python/workers/interceptors) Implement Interceptors in the Temporal Python SDK to manage inbound and outbound SDK calls, enhance tracing, and add authorization to your Workflows and Activities. [Manage Interceptors - TypeScript SDK\ ------------------------------------](https://docs.temporal.io/develop/typescript/workers/interceptors) Implement Interceptors in TypeScript using the Temporal TypeScript SDK to manage inbound and outbound SDK calls, enhance tracing, and add authorization to your Workflows and Activities. --- # 2 docs tagged with "Limits" | Temporal Platform Documentation [Skip to main content](https://docs.temporal.io/tags/limits#__docusaurus_skipToContent_fallback) [Operations\ ----------](https://docs.temporal.io/references/operation-list) Temporal Cloud limits the number of operations per second per namespace to keep the service reliable. This page lists all those operations. [System limits - Temporal Cloud\ ------------------------------](https://docs.temporal.io/cloud/limits) Learn about Temporal Cloud limits, including accounts, namespaces, throughput, retention, task pollers, batch jobs, gRPC, search attributes, and more. --- # One doc tagged with "Parent Close Policy" | Temporal Platform Documentation [Skip to main content](https://docs.temporal.io/tags/parent-close-policy#__docusaurus_skipToContent_fallback) [Parent Close Policy\ -------------------](https://docs.temporal.io/parent-close-policy) Understand the Parent-Child Workflow relationship, including when to use Child Workflows and Parent Close Policies. --- # One doc tagged with "Notification" | Temporal Platform Documentation [Skip to main content](https://docs.temporal.io/tags/notification#__docusaurus_skipToContent_fallback) [Notifications\ -------------](https://docs.temporal.io/cloud/notifications) Receive updates about Temporal Cloud including when certificates will expire, billing updates, and when a failover has completed. --- # One doc tagged with "operations" | Temporal Platform Documentation [Skip to main content](https://docs.temporal.io/tags/operations#__docusaurus_skipToContent_fallback) [Operations\ ----------](https://docs.temporal.io/references/operation-list) Temporal Cloud limits the number of operations per second per namespace to keep the service reliable. This page lists all those operations. --- # 4 docs tagged with "Deprecated" | Temporal Platform Documentation [Skip to main content](https://docs.temporal.io/tags/deprecated#__docusaurus_skipToContent_fallback) [Worker Versioning (Legacy) - Go SDK\ -----------------------------------](https://docs.temporal.io/develop/go/worker-versioning-legacy) Learn the Go SDK's outdated Worker Versioning APIs. [Worker Versioning (Legacy) - Java SDK\ -------------------------------------](https://docs.temporal.io/develop/java/worker-versioning-legacy) Learn the Java SDK's outdated Worker Versioning APIs. [Worker Versioning (Legacy) - Python SDK\ ---------------------------------------](https://docs.temporal.io/develop/python/workflows/worker-versioning-legacy) Learn the Python SDK's outdated Worker Versioning APIs. [Worker Versioning (Legacy) - Typescript SDK\ -------------------------------------------](https://docs.temporal.io/develop/typescript/worker-versioning-legacy) Learn the Typescript SDK's outdated Worker Versioning APIs. --- # 5 docs tagged with "Connectivity" | Temporal Platform Documentation [Skip to main content](https://docs.temporal.io/tags/connectivity#__docusaurus_skipToContent_fallback) [AWS PrivateLink connectivity\ ----------------------------](https://docs.temporal.io/cloud/connectivity/aws-connectivity) Connect to Temporal Cloud using AWS PrivateLink [Connectivity\ ------------](https://docs.temporal.io/cloud/connectivity) Network connectivity details for using Temporal Cloud [Google Private Service Connect connectivity\ -------------------------------------------](https://docs.temporal.io/cloud/connectivity/gcp-connectivity) Connect to Temporal Cloud using Google Private Services Connect [tcld connectivity-rule command reference\ ----------------------------------------](https://docs.temporal.io/cloud/tcld/connectivity-rule) Connectivity rule operations [Temporal Cloud IP addresses\ ---------------------------](https://docs.temporal.io/cloud/connectivity/ip-addresses) Temporal Cloud IP addresses --- # 3 docs tagged with "GCP" | Temporal Platform Documentation [Skip to main content](https://docs.temporal.io/tags/gcp#__docusaurus_skipToContent_fallback) [Audit Logs - GCP Pub/Sub\ ------------------------](https://docs.temporal.io/cloud/audit-logs-gcp) Audit Logs in Temporal Cloud provides forensic information, integrating with GCP Pub/Sub for secure data handling and supporting key Admin and API Key operations. This streamlines audit and compliance processes. [Exporting Workflow Event History to GCS\ ---------------------------------------](https://docs.temporal.io/cloud/export/gcp-export-gcs) Export Workflow History to GCS [Google Private Service Connect connectivity\ -------------------------------------------](https://docs.temporal.io/cloud/connectivity/gcp-connectivity) Connect to Temporal Cloud using Google Private Services Connect --- # 3 docs tagged with "Integrations" | Temporal Platform Documentation [Skip to main content](https://docs.temporal.io/tags/integrations#__docusaurus_skipToContent_fallback) [AI integrations\ ---------------](https://docs.temporal.io/develop/integrations) AI framework integrations available for Temporal SDKs. [AI integrations\ ---------------](https://docs.temporal.io/develop/python/integrations/) Integrations with other tools and services. [AI integrations\ ---------------](https://docs.temporal.io/develop/typescript/integrations/) Integrations with other tools and services. --- # 2 docs tagged with "On-Demand Capacity" | Temporal Platform Documentation [Skip to main content](https://docs.temporal.io/tags/on-demand-capacity#__docusaurus_skipToContent_fallback) [Capacity modes\ --------------](https://docs.temporal.io/cloud/capacity-modes) Control how limits are assigned to a Namespace with Capacity Modes [Managing Actions per Second (APS) limits in Temporal Cloud\ ----------------------------------------------------------](https://docs.temporal.io/best-practices/managing-aps-limits) Control how limits are assigned to a Namespace with Capacity Modes --- # 3 docs tagged with "Logs" | Temporal Platform Documentation [Skip to main content](https://docs.temporal.io/tags/logs#__docusaurus_skipToContent_fallback) [Audit Logs\ ----------](https://docs.temporal.io/cloud/audit-logs) Audit Logs in Temporal Cloud provide forensic information, integrating with a data streaming service for secure data handling and supporting key Admin and API Key operations. This streamlines audit and compliance processes. [Audit Logs - AWS Kinesis\ ------------------------](https://docs.temporal.io/cloud/audit-logs-aws) Audit Logs in Temporal Cloud provides forensic information, integrating with AWS Kinesis for secure data handling and supporting key Admin and API Key operations. This streamlines audit and compliance processes. [Audit Logs - GCP Pub/Sub\ ------------------------](https://docs.temporal.io/cloud/audit-logs-gcp) Audit Logs in Temporal Cloud provides forensic information, integrating with GCP Pub/Sub for secure data handling and supporting key Admin and API Key operations. This streamlines audit and compliance processes. --- # One doc tagged with "Priority and Fairness" | Temporal Platform Documentation [Skip to main content](https://docs.temporal.io/tags/priority-and-fairness#__docusaurus_skipToContent_fallback) [Task Queue Priority and Fairness\ --------------------------------](https://docs.temporal.io/develop/task-queue-priority-fairness) How the Task Queue Priority and Fairness features can be used. --- # 8 docs tagged with "Activity" | Temporal Platform Documentation [Skip to main content](https://docs.temporal.io/tags/activity#__docusaurus_skipToContent_fallback) [Activity basics - TypeScript SDK\ --------------------------------](https://docs.temporal.io/develop/typescript/activities/basics) Shows how to create an Activity with the TypeScript SDK [Activity execution - .NET SDK\ -----------------------------](https://docs.temporal.io/develop/dotnet/activities/execution) Shows how to perform Activity execution with the .NET SDK [Activity execution - Go SDK\ ---------------------------](https://docs.temporal.io/develop/go/activities/execution) Shows how to perform Activity execution with the Go SDK [Activity execution - Java SDK\ -----------------------------](https://docs.temporal.io/develop/java/activities/execution) Shows how to perform Activity execution with the Java SDK [Activity execution - PHP SDK\ ----------------------------](https://docs.temporal.io/develop/php/activities/execution) Shows how to perform Activity execution with the PHP SDK [Activity execution - Python SDK\ -------------------------------](https://docs.temporal.io/develop/python/activities/execution) Shows how to perform Activity execution with the Python SDK [Activity execution - Ruby SDK\ -----------------------------](https://docs.temporal.io/develop/ruby/activities/execution) Shows how to perform Activity execution with the Ruby SDK [Activity execution - TypeScript SDK\ -----------------------------------](https://docs.temporal.io/develop/typescript/activities/execution) Shows how to perform Activity execution with the TypeScript SDK --- # 2 docs tagged with "Performance" | Temporal Platform Documentation [Skip to main content](https://docs.temporal.io/tags/performance#__docusaurus_skipToContent_fallback) [Worker performance\ ------------------](https://docs.temporal.io/develop/worker-performance) Optimize Temporal SDK performance by fine-tuning maxConcurrentWorkflowTaskExecutionSize, Worker Cache options, and Poll Success Rate. Ensure balanced Worker resources and monitor metrics for best results. [Worker tuning quick reference\ -----------------------------](https://docs.temporal.io/develop/worker-tuning-reference) A quick reference guide for Temporal Worker configuration defaults across SDKs, organized by resource type (compute, memory, IO) with key metrics for each. --- # One doc tagged with "Recovery Point Objective" | Temporal Platform Documentation [Skip to main content](https://docs.temporal.io/tags/recovery-point-objective#__docusaurus_skipToContent_fallback) [RPO and RTO\ -----------](https://docs.temporal.io/cloud/rpo-rto) Understand the Recovery Point Objective (RPO) and Recovery Time Objective (RTO) in Temporal Cloud. --- # One doc tagged with "rate limiting" | Temporal Platform Documentation [Skip to main content](https://docs.temporal.io/tags/rate-limiting#__docusaurus_skipToContent_fallback) [Operations\ ----------](https://docs.temporal.io/references/operation-list) Temporal Cloud limits the number of operations per second per namespace to keep the service reliable. This page lists all those operations. --- # One doc tagged with "Recovery Time Objective" | Temporal Platform Documentation [Skip to main content](https://docs.temporal.io/tags/recovery-time-objective#__docusaurus_skipToContent_fallback) [RPO and RTO\ -----------](https://docs.temporal.io/cloud/rpo-rto) Understand the Recovery Point Objective (RPO) and Recovery Time Objective (RTO) in Temporal Cloud. --- # 4 docs tagged with "High Availability" | Temporal Platform Documentation [Skip to main content](https://docs.temporal.io/tags/high-availability#__docusaurus_skipToContent_fallback) [Configure and trigger failovers\ -------------------------------](https://docs.temporal.io/cloud/high-availability/failovers) How automatic and manual failovers work with Temporal Cloud HA [High Availability\ -----------------](https://docs.temporal.io/cloud/high-availability) Temporal Cloud's Namespace with High Availability features offers automated failover, synchronized data, and replication for workloads requiring disaster-tolerant deployment and 99.99% uptime. [Migrate between regions\ -----------------------](https://docs.temporal.io/cloud/migrate/migrate-within-cloud) Use Temporal Cloud's High Availability features to migrate between regions. [Monitoring High Availability\ ----------------------------](https://docs.temporal.io/cloud/high-availability/monitoring) How to track the health and performance of your High Availability Namespaces. --- # One doc tagged with "Replay" | Temporal Platform Documentation [Skip to main content](https://docs.temporal.io/tags/replay#__docusaurus_skipToContent_fallback) [Safely deploying changes to Workflow code\ -----------------------------------------](https://docs.temporal.io/develop/safe-deployments) Safely deploy changes to existing Workflow code by validating first for determinism errors before deploying to production. --- # 3 docs tagged with "Multitenancy" | Temporal Platform Documentation [Skip to main content](https://docs.temporal.io/tags/multitenancy#__docusaurus_skipToContent_fallback) [Multi-tenancy - Temporal feature\ --------------------------------](https://docs.temporal.io/evaluate/development-production-features/multi-tenancy) Learn about Temporal's namespace isolation for multi-tenancy and how to build multi-tenant applications. [Multi-tenant application patterns\ ---------------------------------](https://docs.temporal.io/production-deployment/multi-tenant-patterns) Learn how to build multi-tenant applications using Temporal with task queue isolation patterns, worker design, and best practices. [Security model - Temporal Cloud\ -------------------------------](https://docs.temporal.io/cloud/security) Temporal Cloud provides robust security for applications, data, and its platform with features like mTLS, client-side encryption, PrivateLink, and SOC 2 Type 2 compliance. --- # 2 docs tagged with "Provisioned Capacity" | Temporal Platform Documentation [Skip to main content](https://docs.temporal.io/tags/provisioned-capacity#__docusaurus_skipToContent_fallback) [Capacity modes\ --------------](https://docs.temporal.io/cloud/capacity-modes) Control how limits are assigned to a Namespace with Capacity Modes [Managing Actions per Second (APS) limits in Temporal Cloud\ ----------------------------------------------------------](https://docs.temporal.io/best-practices/managing-aps-limits) Control how limits are assigned to a Namespace with Capacity Modes --- # One doc tagged with "Spring Boot" | Temporal Platform Documentation [Skip to main content](https://docs.temporal.io/tags/spring-boot#__docusaurus_skipToContent_fallback) [Spring Boot integration - Java SDK\ ----------------------------------](https://docs.temporal.io/develop/java/integrations/spring-boot-integration) Learn about the Temporal Spring Boot integration --- # 8 docs tagged with "Codec Server" | Temporal Platform Documentation [Skip to main content](https://docs.temporal.io/tags/codec-server#__docusaurus_skipToContent_fallback) [Codec Server\ ------------](https://docs.temporal.io/codec-server) A Codec Server is an HTTP server that provides remote encoding and decoding for Temporal Payloads. [Converters and encryption - .NET SDK\ ------------------------------------](https://docs.temporal.io/develop/dotnet/best-practices/converters-and-encryption) Use a custom Payload Codec and Converter in the .NET SDK to modify Temporal Data Conversion behavior, including examples for encryption and camel case conversion. [Converters and encryption - Java SDK\ ------------------------------------](https://docs.temporal.io/develop/java/best-practices/converters-and-encryption) Create and implement a Custom Payload Codec and Payload Converter in Java using the Temporal SDK for custom data encryption, compression, and type conversion. [Converters and encryption - Ruby SDK\ ------------------------------------](https://docs.temporal.io/develop/ruby/best-practices/converters-and-encryption) Use a custom Payload Codec and Converter in the Ruby SDK to modify Temporal Data Conversion behavior, including examples for encryption and formatting. [Converters and encryption - TypeScript SDK\ ------------------------------------------](https://docs.temporal.io/develop/typescript/converters-and-encryption) Create a custom Payload Converter in TypeScript with Temporal SDKs to handle non-JSON-serializable values, configure your Data Converter, and use protobufs and encryption seamlessly in your Workflows and Activities. [Data encryption - Temporal feature\ ----------------------------------](https://docs.temporal.io/evaluate/development-production-features/data-encryption) Implement data encryption in your Temporal Workflows to ensure the security and confidentiality of your data. [Payload encryption - Go SDK\ ---------------------------](https://docs.temporal.io/develop/go/data-handling/data-encryption) Encrypt data sent to and from the Temporal Service using a custom Payload Codec in the Go SDK. [Payload encryption - Python SDK\ -------------------------------](https://docs.temporal.io/develop/python/data-handling/data-encryption) Encrypt data sent to and from the Temporal Service using a custom Payload Codec in the Python SDK. --- # 7 docs tagged with "continue-as-new" | Temporal Platform Documentation [Skip to main content](https://docs.temporal.io/tags/continue-as-new#__docusaurus_skipToContent_fallback) [Continue-As-New - .NET SDK\ --------------------------](https://docs.temporal.io/develop/dotnet/workflows/continue-as-new) Use Temporal's Continue-As-New in .NET to manage large Event Histories by atomically creating new Workflow Executions with the same Workflow Id and fresh parameters. [Continue-As-New - Go SDK\ ------------------------](https://docs.temporal.io/develop/go/workflows/continue-as-new) Use Temporal's Continue-As-New in Go to manage large Event Histories by atomically creating new Workflow Executions with the same Workflow Id and fresh parameters. [Continue-As-New - Java SDK\ --------------------------](https://docs.temporal.io/develop/java/workflows/continue-as-new) Use Temporal's Continue-As-New in Java to manage large Event Histories by atomically creating new Workflow Executions with the same Workflow Id and fresh parameters. [Continue-As-New - PHP SDK\ -------------------------](https://docs.temporal.io/develop/php/workflows/continue-as-new) Use Temporal's Continue-As-New in PHP to manage large Event Histories by atomically creating new Workflow Executions with the same Workflow Id and fresh parameters. [Continue-As-New - Python SDK\ ----------------------------](https://docs.temporal.io/develop/python/workflows/continue-as-new) Use Temporal's Continue-As-New in Python to manage large Event Histories by atomically creating new Workflow Executions with the same Workflow Id and fresh parameters. [Continue-As-New - Ruby SDK\ --------------------------](https://docs.temporal.io/develop/ruby/workflows/continue-as-new) Use Continue-As-New with the Temporal Ruby SDK to manage Workflow Event Histories, ensuring optimal performance by starting new Executions seamlessly. [Continue-As-New - Typescript SDK\ --------------------------------](https://docs.temporal.io/develop/typescript/workflows/continue-as-new) Use Temporal's Continue-As-New in Typescript to manage large Event Histories by atomically creating new Workflow Executions with the same Workflow Id and fresh parameters. --- # 6 docs tagged with "Event History" | Temporal Platform Documentation [Skip to main content](https://docs.temporal.io/tags/event-history#__docusaurus_skipToContent_fallback) [Event History\ -------------](https://docs.temporal.io/encyclopedia/event-history/) Discover how Temporal uses the Event History to recreate a Workflow's state in the case of failure, such as a Worker crash, and how it uses replay to restore the Workflow's state to the point of failure. [Event History walkthrough with the .NET SDK\ -------------------------------------------](https://docs.temporal.io/encyclopedia/event-history/event-history-dotnet) Discover how Temporal uses the Event History to recreate a Workflow's state in the case of failure, such as a Worker crash, and how it uses replay to restore the Workflow's state to the point of failure. [Event History walkthrough with the Go SDK\ -----------------------------------------](https://docs.temporal.io/encyclopedia/event-history/event-history-go) Discover how Temporal uses the Event History to recreate a Workflow's state in the case of failure, such as a Worker crash, and how it uses replay to restore the Workflow's state to the point of failure. [Event History walkthrough with the Java SDK\ -------------------------------------------](https://docs.temporal.io/encyclopedia/event-history/event-history-java) Discover how Temporal uses the Event History to recreate a Workflow's state in the case of failure, such as a Worker crash, and how it uses replay to restore the Workflow's state to the point of failure. [Event History walkthrough with the Python SDK\ ---------------------------------------------](https://docs.temporal.io/encyclopedia/event-history/event-history-python) Discover how Temporal uses the Event History to recreate a Workflow's state in the case of failure, such as a Worker crash, and how it uses replay to restore the Workflow's state to the point of failure. [Event History walkthrough with the TypeScript SDK\ -------------------------------------------------](https://docs.temporal.io/encyclopedia/event-history/event-history-typescript) Discover how Temporal uses the Event History to recreate a Workflow's state in the case of failure, such as a Worker crash, and how it uses replay to restore the Workflow's state to the point of failure. --- # 4 docs tagged with "OpenMetrics" | Temporal Platform Documentation [Skip to main content](https://docs.temporal.io/tags/open-metrics#__docusaurus_skipToContent_fallback) [Metrics integrations\ --------------------](https://docs.temporal.io/cloud/metrics/openmetrics/metrics-integrations) Integrating with the Temporal Cloud OpenMetrics endpoint. [OpenMetrics API reference\ -------------------------](https://docs.temporal.io/cloud/metrics/openmetrics/api-reference) Detailed API documentation for the Temporal Cloud OpenMetrics endpoint. [OpenMetrics metrics reference\ -----------------------------](https://docs.temporal.io/cloud/metrics/openmetrics/metrics-reference) Detailed API documentation for the Temporal Cloud OpenMetrics endpoint. [OpenMetrics migration guide\ ---------------------------](https://docs.temporal.io/cloud/metrics/openmetrics/migration-guide) Migrate from the Prometheus query endpoint to the new OpenMetrics endpoint in Temporal Cloud. --- # 3 docs tagged with "Pricing" | Temporal Platform Documentation [Skip to main content](https://docs.temporal.io/tags/pricing#__docusaurus_skipToContent_fallback) [Pricing for Temporal Nexus\ --------------------------](https://docs.temporal.io/cloud/nexus/pricing) Learn about the pricing structure for using Nexus. [Temporal Cloud Actions\ ----------------------](https://docs.temporal.io/cloud/actions) Temporal Cloud offers flexible, predictable pricing for Workflows, Activities, Workers, and Storage. Pay for what you use with volume discounts and credit savings. [Temporal Cloud pricing\ ----------------------](https://docs.temporal.io/cloud/pricing) Temporal Cloud offers flexible, predictable pricing for Workflows, Activities, Workers, and Storage. Pay for what you use with volume discounts and credit savings. --- # One doc tagged with "Tasks" | Temporal Platform Documentation [Skip to main content](https://docs.temporal.io/tags/tasks#__docusaurus_skipToContent_fallback) [Tasks\ -----](https://docs.temporal.io/tasks) Learn about the types of Tasks in Temporal and their role in Workflow and Activity Executions. --- # One doc tagged with "Standalone Activities" | Temporal Platform Documentation [Skip to main content](https://docs.temporal.io/tags/standalone-activities#__docusaurus_skipToContent_fallback) [Job Queue\ ---------](https://docs.temporal.io/evaluate/development-production-features/job-queue) Standalone Activities adds the ability to execute any Temporal Activity as a top-level primitive without the full overhead of a Workflow. --- # 8 docs tagged with "Debugging" | Temporal Platform Documentation [Skip to main content](https://docs.temporal.io/tags/debugging#__docusaurus_skipToContent_fallback) [Debugging - .NET SDK\ --------------------](https://docs.temporal.io/develop/dotnet/best-practices/debugging) Debug Workflows in development and production environments using Temporal .NET SDK. Use logging, debugger, Web UI, CLI, replay, tracing, and more for efficient troubleshooting. [Debugging - Go SDK\ ------------------](https://docs.temporal.io/develop/go/best-practices/debugging) Use debugger tools and set TEMPORAL\_DEBUG to true for debugging Workflow Definitions with the Temporal Go SDK, and debug production Workflows via Web UI, CLI, or tracing. [Debugging - Java SDK\ --------------------](https://docs.temporal.io/develop/java/best-practices/debugging) Debug your Temporal Java Workflows using your favorite Java IDE's debugger. Set the TEMPORAL\_DEBUG environment variable to true during debugging to avoid deadlocks. Use Web UI, Temporal CLI, and logging for development and production. Optimize Worker performance with metrics and the Worker performance guide. [Debugging - PHP SDK\ -------------------](https://docs.temporal.io/develop/php/best-practices/debugging) Effectively debug your Workflow in both development and production environments using Web UI, Temporal CLI, and performance metrics for optimal Worker and Server performance. [Debugging - Python SDK\ ----------------------](https://docs.temporal.io/develop/python/best-practices/debugging) Debug Workflows in development and production environments using the Temporal Python SDK, Web UI, Temporal CLI, replay, tracing, logging, and performance metrics. [Debugging - Ruby SDK\ --------------------](https://docs.temporal.io/develop/ruby/best-practices/debugging) Debug Workflows in development and production environments using Temporal Ruby SDK. Use logging, debugger, CLI, replay, tracing, and more. [Debugging - Temporal feature\ ----------------------------](https://docs.temporal.io/evaluate/development-production-features/debugging) Discover Temporal's comprehensive debugging capabilities; tools and frameworks that facilitate Workflow and activity debugging across different programming languages with Temporal. [Debugging - TypeScript SDK\ --------------------------](https://docs.temporal.io/develop/typescript/best-practices/debugging) The Temporal TypeScript SDK Debugging guide provides tools and tips for debugging Workflows and Workers in development and production environments. Troubleshoot common issues using the Web UI, Temporal CLI, and more. --- # 3 docs tagged with "Side-effects" | Temporal Platform Documentation [Skip to main content](https://docs.temporal.io/tags/side-effects#__docusaurus_skipToContent_fallback) [Side Effects - Go SDK\ ---------------------](https://docs.temporal.io/develop/go/workflows/side-effects) Side Effects in Workflows execute non-deterministic code, storing results in the Workflow Event History to maintain determinism. Use Go's SideEffect function for integration. [Side Effects - Java SDK\ -----------------------](https://docs.temporal.io/develop/java/workflows/side-effects) Side Effects in a Workflow execute non-deterministic code like generating a UUID. The result is saved in Workflow Event History for consistent replays without re-execution. [Side Effects - PHP SDK\ ----------------------](https://docs.temporal.io/develop/php/workflows/side-effects) Use Side Effects in PHP to execute non-deterministic code like generating UUIDs or random numbers in a Workflow without compromising its determinism. --- # 10 docs tagged with "Child Workflows" | Temporal Platform Documentation [Skip to main content](https://docs.temporal.io/tags/child-workflows#__docusaurus_skipToContent_fallback) [Child Workflows\ ---------------](https://docs.temporal.io/child-workflows) A Child Workflow Execution in the Temporal platform is initiated from another Workflow within the same Namespace. [Child Workflows - .NET SDK\ --------------------------](https://docs.temporal.io/develop/dotnet/workflows/child-workflows) Start a Child Workflow Execution and set a Parent Close Policy using Temporal .NET SDK. Discover methods like ExecuteChildWorkflowAsync and manage Workflow behaviors. [Child Workflows - Go SDK\ ------------------------](https://docs.temporal.io/develop/go/workflows/child-workflows) Use the Go SDK to start a Child Workflow Execution and set a Parent Close Policy, including details on Workflow Options and future management. [Child Workflows - Java SDK\ --------------------------](https://docs.temporal.io/develop/java/workflows/child-workflows) Start a Child Workflow Execution and set a Parent Close Policy using the Java SDK. Manage Child Workflow Events and ensure successful execution. [Child Workflows - PHP SDK\ -------------------------](https://docs.temporal.io/develop/php/workflows/child-workflows) Start a Child Workflow Execution within a parent Workflow using Temporal in PHP. Configure ChildWorkflowOptions, handle Parent Close Policy, and implement asynchronous calls with promises. [Child Workflows - Python SDK\ ----------------------------](https://docs.temporal.io/develop/python/workflows/child-workflows) Start a Child Workflow Execution and set a Parent Close Policy using the Temporal Python SDK. Ensure proper progress logging and specify Parent Workflow behavior upon closure. [Child Workflows - Ruby SDK\ --------------------------](https://docs.temporal.io/develop/ruby/workflows/child-workflows) Start a Child Workflow Execution and set a Parent Close Policy using Temporal Ruby SDK. [Child Workflows - Temporal feature\ ----------------------------------](https://docs.temporal.io/evaluate/development-production-features/throughput-composability) Leverage Temporal Child Workflows for enhanced composability and efficiency. Partition steps, manage resources, invoke multiple services, and execute periodic logic seamlessly. [Child Workflows - TypeScript SDK\ --------------------------------](https://docs.temporal.io/develop/typescript/workflows/child-workflows) Start and manage Child Workflow Executions using Temporal's Child Workflow API, including setting Parent Close Policy, handling Events, and advanced Child Workflow options. [Parent Close Policy\ -------------------](https://docs.temporal.io/parent-close-policy) Understand the Parent-Child Workflow relationship, including when to use Child Workflows and Parent Close Policies. --- # 10 docs tagged with "Certificates" | Temporal Platform Documentation [Skip to main content](https://docs.temporal.io/tags/certificates#__docusaurus_skipToContent_fallback) [Account access\ --------------](https://docs.temporal.io/cloud/manage-access/) Manage access to your Temporal Cloud account [Authenticate with mTLS certificates\ -----------------------------------](https://docs.temporal.io/cloud/certificates) Temporal Cloud supports mTLS, which requires CA certificates for secure communication. Keep certificates updated to avoid disruptions in Workflow Execution. Manage and update certificates easily via the Temporal Cloud UI or tcld tool. [tcld generate-certificates command reference\ --------------------------------------------](https://docs.temporal.io/cloud/tcld/generate-certificates/) Generate certificate authority and end-entity TLS certificates for Temporal Cloud with tcld generate-certificates commands. Use modifiers for customization. [Temporal Client - .NET SDK\ --------------------------](https://docs.temporal.io/develop/dotnet/client/temporal-client) Create a Temporal Client, connect to Temporal Cloud, start a Workflow, and get Workflow results using the Temporal .NET SDK with detailed steps and code examples. [Temporal Client - Go SDK\ ------------------------](https://docs.temporal.io/develop/go/client/temporal-client) Connect to Temporal Service or Cloud, start Workflow Executions, manage Workflow options, and retrieve Workflow results using the Go SDK. Follow detailed steps and code examples to effectively use Temporal’s capabilities. [Temporal Client - Java SDK\ --------------------------](https://docs.temporal.io/develop/java/client/temporal-client) This guide introduces Temporal Clients, explaining their role and configuration in Java to connect to various Temporal Services, including starting Workflow Executions and customizing Workflow options. [Temporal Client - PHP SDK\ -------------------------](https://docs.temporal.io/develop/php/client/temporal-client) Connect a Temporal Client to a Temporal Service and start Workflow Executions. This guide covers communication, including sending signals and queries. [Temporal Client - Python SDK\ ----------------------------](https://docs.temporal.io/develop/python/client/temporal-client) Discover how to connect and use Temporal Clients with Python. Link your Client to Temporal Service, Temporal Cloud, start Workflow Executions, set Task Queues, Workflow Ids, and get Workflow results. [Temporal Client - Ruby SDK\ --------------------------](https://docs.temporal.io/develop/ruby/client/temporal-client) Create a Temporal Client, connect to Temporal Cloud, start a Workflow, and get Workflow results using the Temporal Ruby SDK. [Temporal Client - Typescript SDK\ --------------------------------](https://docs.temporal.io/develop/typescript/client/temporal-client) The Temporal Client SDK enables seamless communication with the Temporal Service, allowing applications to start Workflow Executions, send Signals, and query Workflows efficiently. --- # 8 docs tagged with "Durable Timers" | Temporal Platform Documentation [Skip to main content](https://docs.temporal.io/tags/durable-timers#__docusaurus_skipToContent_fallback) [Selectors - Go SDK\ ------------------](https://docs.temporal.io/develop/go/workflows/selectors) Use Temporal’s Go SDK Selectors with Futures, Timers, and Channels. Ensure deterministic Workflow execution and handle multiple parallel tasks efficiently. [Timers - .NET SDK\ -----------------](https://docs.temporal.io/develop/dotnet/workflows/timers) Set a Durable Timer using the Temporal .NET SDK. Pause Workflow execution for days or months. Timers are persisted and highly resource-efficient using Workflow.DelayAsync. [Timers - Go SDK\ ---------------](https://docs.temporal.io/develop/go/workflows/timers) Set Durable Timers in a Workflow using the sleep() or NewTimer() functions in Go with Temporal. Timers persist through Worker and Temporal Service downtime. [Timers - Java SDK\ -----------------](https://docs.temporal.io/develop/java/workflows/timers) A Workflow sets a durable Timer for delayed execution. Even if the Worker or Temporal Service is down, the Timer resumes once back up. Efficient and scalable. [Timers - PHP SDK\ ----------------](https://docs.temporal.io/develop/php/workflows/timers) A Timer in a Workflow sets a durable pause for a fixed time. Even after downtimes, your Workflow resumes execution. Lightweight and scalable, millions of Timers can run on a single Worker. [Timers - Python SDK\ -------------------](https://docs.temporal.io/develop/python/workflows/timers) Set durable Timers with Temporal Workflows using sleep() or timer(), ensuring code execution resumes after downtime. Sleep for months using resource-light operations in Python. [Timers - Ruby SDK\ -----------------](https://docs.temporal.io/develop/ruby/workflows/timers) Set a Durable Timer using the Temporal Ruby SDK. Pause Workflow execution for days or months. Timers are persisted and efficient. [Timers - TypeScript SDK\ -----------------------](https://docs.temporal.io/develop/typescript/workflows/timers) A Workflow sets durable Timers for fixed periods using sleep() or timer(). Timers are persisted, ensuring execution continues after downtime, using minimal resources. --- # 4 docs tagged with "Support" | Temporal Platform Documentation [Skip to main content](https://docs.temporal.io/tags/support#__docusaurus_skipToContent_fallback) [Service Level Agreement (SLA) - Temporal Cloud\ ----------------------------------------------](https://docs.temporal.io/cloud/sla) Temporal Cloud offers two availability levels; 99.99% uptime for standard and High Availability feature deployments, with SLAs guaranteeing 99.9% and 99.99% against service errors, respectively. [Services, support, and training - Temporal Cloud\ ------------------------------------------------](https://docs.temporal.io/cloud/support) Temporal Cloud offers support, services, and training for seamless onboarding, efficient app design, and scaling. Services include technical onboarding, design/code reviews, pre-production optimization, and load tests. [Temporal Cloud Actions\ ----------------------](https://docs.temporal.io/cloud/actions) Temporal Cloud offers flexible, predictable pricing for Workflows, Activities, Workers, and Storage. Pay for what you use with volume discounts and credit savings. [Temporal Cloud pricing\ ----------------------](https://docs.temporal.io/cloud/pricing) Temporal Cloud offers flexible, predictable pricing for Workflows, Activities, Workers, and Storage. Pay for what you use with volume discounts and credit savings. --- # 4 docs tagged with "Task Queues" | Temporal Platform Documentation [Skip to main content](https://docs.temporal.io/tags/task-queues#__docusaurus_skipToContent_fallback) [Task Queue Priority and Fairness\ --------------------------------](https://docs.temporal.io/develop/task-queue-priority-fairness) How the Task Queue Priority and Fairness features can be used. [Task Queues\ -----------](https://docs.temporal.io/task-queue) Explore the role of Worker Processes in polling Task Queues and executing Tasks. [Task Queues and naming best practices\ -------------------------------------](https://docs.temporal.io/task-queue/naming) A mismatch in Task Queue names creates separate queues, preventing the Worker from receiving tasks and stalling Workflow Execution. [Task Routing and Worker sessions\ --------------------------------](https://docs.temporal.io/task-routing) Learn about routing Tasks and Worker Sessions. --- # 11 docs tagged with "Best Practices" | Temporal Platform Documentation [Skip to main content](https://docs.temporal.io/tags/best-practices#__docusaurus_skipToContent_fallback) [Best practices\ --------------](https://docs.temporal.io/best-practices/) Comprehensive best practices for working with Temporal, covering namespace management, security, and operational excellence. [Knowledge Hub\ -------------](https://docs.temporal.io/best-practices/knowledge-hub) Best practices for building and maintaining an internal Temporal knowledge hub that accelerates developer onboarding, reduces platform team support load, and establishes consistent standards across your organization. [Managing Actions per Second (APS) limits in Temporal Cloud\ ----------------------------------------------------------](https://docs.temporal.io/best-practices/managing-aps-limits) Control how limits are assigned to a Namespace with Capacity Modes [Managing Temporal Cloud access control\ --------------------------------------](https://docs.temporal.io/best-practices/cloud-access-control) Best practices for managing access control, permissions, and user management in Temporal Cloud. [Multi-tenant application patterns\ ---------------------------------](https://docs.temporal.io/production-deployment/multi-tenant-patterns) Learn how to build multi-tenant applications using Temporal with task queue isolation patterns, worker design, and best practices. [Namespace best practices\ ------------------------](https://docs.temporal.io/best-practices/managing-namespace) Best practices for organizing and managing Temporal Namespaces, including naming conventions, organizational patterns, and production safeguards. [Plugins guide\ -------------](https://docs.temporal.io/develop/plugins-guide) Best practices for creating plugins for AI Agents [Pre-production testing\ ----------------------](https://docs.temporal.io/best-practices/pre-production-testing) Experience-driven testing practices for teams running Temporal applications, covering failure injection, load testing, and operational validation. [Security controls for Temporal Cloud\ ------------------------------------](https://docs.temporal.io/best-practices/security-controls) Best practices for implementing and managing security controls in Temporal Cloud environments. [Worker deployment and performance\ ---------------------------------](https://docs.temporal.io/best-practices/worker) Best practices for deploying and optimizing Temporal Workers for performance and reliability. [Workflow cost optimization\ --------------------------](https://docs.temporal.io/best-practices/cost-optimization) Strategies for optimizing costs associated with workloads running on Temporal Cloud while maintaining workflow reliability and observability. --- # 8 docs tagged with "Messages" | Temporal Platform Documentation [Skip to main content](https://docs.temporal.io/tags/messages#__docusaurus_skipToContent_fallback) [Handling Signals, Queries, & Updates\ ------------------------------------](https://docs.temporal.io/handling-messages) Signals, Queries, and Updates facilitate interactions with Workflow Executions. [Sending Signals, Queries, & Updates\ -----------------------------------](https://docs.temporal.io/sending-messages) Signals, Queries, and Updates facilitate interactions with Workflow Executions. [Temporal Workflow message passing - Signals, Queries, & Updates\ ---------------------------------------------------------------](https://docs.temporal.io/encyclopedia/workflow-message-passing/) Signals, Queries, and Updates facilitate interactions with Workflow Executions. [Workflow message passing - Go SDK\ ---------------------------------](https://docs.temporal.io/develop/go/workflows/message-passing) Develop with Queries, Signals, and Updates with the Temporal Go SDK. [Workflow message passing - Java SDK\ -----------------------------------](https://docs.temporal.io/develop/java/workflows/message-passing) Develop with Queries, Signals, and Updates with the Temporal Java SDK. [Workflow message passing - PHP SDK\ ----------------------------------](https://docs.temporal.io/develop/php/workflows/message-passing) Develop with Signals, Queries, and Updates in Temporal Workflows. Define, handle, and send Signals or Queries, and validate updates from a Temporal Client. [Workflow message passing - Python SDK\ -------------------------------------](https://docs.temporal.io/develop/python/workflows/message-passing) Develop with Queries, Signals, and Updates with the Temporal Python SDK. [Workflow message passing - TypeScript SDK\ -----------------------------------------](https://docs.temporal.io/develop/typescript/workflows/message-passing) Develop with Queries, Signals, and Updates with the Temporal Typescript SDK. --- # 2 docs tagged with "Terraform" | Temporal Platform Documentation [Skip to main content](https://docs.temporal.io/tags/terraform#__docusaurus_skipToContent_fallback) [Cloud automation - Temporal feature\ -----------------------------------](https://docs.temporal.io/evaluate/development-production-features/cloud-automation) Explore how cloud automation simplifies cloud management and enhances security through APIs, Terraform, and CLI. [Temporal Cloud Terraform provider\ ---------------------------------](https://docs.temporal.io/cloud/terraform-provider) Automate resource management on Temporal Cloud with the Terraform Temporal provider. Manage Namespaces and users with Terraform's infrastructure-as-code. --- # 7 docs tagged with "Patching" | Temporal Platform Documentation [Skip to main content](https://docs.temporal.io/tags/patching#__docusaurus_skipToContent_fallback) [Versioning - .NET SDK\ ---------------------](https://docs.temporal.io/develop/dotnet/workflows/versioning) Use the .NET SDK Patching API to safely deploy new code versions, handle deprecated patches, and manage Workflow activities using Temporal for long-running tasks. [Versioning - Go SDK\ -------------------](https://docs.temporal.io/develop/go/workflows/versioning) Temporal's Go SDK ensures Workflow determinism through Patching APIs and Worker Versioning. Update Workflow code without causing non-deterministic issues, understand versioning best practices, and use dynamic configuration parameters for seamless updating of long-running Workflows. [Versioning - Java SDK\ ---------------------](https://docs.temporal.io/develop/java/workflows/versioning) The Temporal Platform ensures deterministic Workflow code, offering versioning features in the Java SDK with Workflow Patching APIs and Worker Build Ids for efficient updates. [Versioning - PHP SDK feature guide\ ----------------------------------](https://docs.temporal.io/develop/php/workflows/versioning) Ensure deterministic Temporal Workflow execution and deploy updates with the PHP SDK's patching and Worker Versioning APIs. [Versioning - Python SDK\ -----------------------](https://docs.temporal.io/develop/python/workflows/versioning) Ensure deterministic Temporal Workflow execution and safely deploy updates using the Python SDK's patching and Worker Versioning APIs, for scalable long-running Workflows. [Versioning - Ruby SDK\ ---------------------](https://docs.temporal.io/develop/ruby/workflows/versioning) Use the Ruby SDK Patching API to safely deploy new code versions, handle deprecated patches, and manage Workflow activities using Temporal for long-running tasks. [Versioning - TypeScript SDK\ ---------------------------](https://docs.temporal.io/develop/typescript/workflows/versioning) Temporal TypeScript SDK ensures deterministic Workflow code with versioning features like Workflow Patching APIs, Worker Build IDs, and Workflow migration strategies. --- # 3 docs tagged with "Temporal Web UI" | Temporal Platform Documentation [Skip to main content](https://docs.temporal.io/tags/temporal-web-ui#__docusaurus_skipToContent_fallback) [Temporal Web UI\ ---------------](https://docs.temporal.io/web-ui) The Temporal Web UI offers comprehensive Workflow management, debugging tools, and metadata access. [Temporal Web UI configuration reference\ ---------------------------------------](https://docs.temporal.io/references/web-ui-configuration) Manage your Temporal Server efficiently with development.yaml. Set parameters for Auth, TLS, ports, and more. [Temporal Web UI environment variables reference\ -----------------------------------------------](https://docs.temporal.io/references/web-ui-environment-variables) Dynamically configure Temporal Web UI with environment variables in Docker for settings like TEMPORAL\_ADDRESS, authentication, TLS, OpenAPI, and more. --- # One doc tagged with "use-cases" | Temporal Platform Documentation [Skip to main content](https://docs.temporal.io/tags/use-cases#__docusaurus_skipToContent_fallback) [Temporal use cases and design patterns\ --------------------------------------](https://docs.temporal.io/evaluate/use-cases-design-patterns) Discover various use cases and architectural design patterns for implementing Workflows with Temporal. --- # 2 docs tagged with "TOML" | Temporal Platform Documentation [Skip to main content](https://docs.temporal.io/tags/toml#__docusaurus_skipToContent_fallback) [Environment configuration\ -------------------------](https://docs.temporal.io/develop/environment-configuration) Configure Temporal Clients using environment variables and TOML configuration files [Environment configuration\ -------------------------](https://docs.temporal.io/references/client-environment-configuration) Reference for configuring Temporal Clients using environment variables and TOML configuration files. --- # 2 docs tagged with "TRUs" | Temporal Platform Documentation [Skip to main content](https://docs.temporal.io/tags/tr-us#__docusaurus_skipToContent_fallback) [Capacity modes\ --------------](https://docs.temporal.io/cloud/capacity-modes) Control how limits are assigned to a Namespace with Capacity Modes [Managing Actions per Second (APS) limits in Temporal Cloud\ ----------------------------------------------------------](https://docs.temporal.io/best-practices/managing-aps-limits) Control how limits are assigned to a Namespace with Capacity Modes --- # 5 docs tagged with "Temporal" | Temporal Platform Documentation [Skip to main content](https://docs.temporal.io/tags/temporal#__docusaurus_skipToContent_fallback) [Evaluate Temporal\ -----------------](https://docs.temporal.io/evaluate/) Temporal enhances distributed application development with clear code structure, fault-tolerance, and execution guarantees, trusted by thousands for mission-critical workloads. [Temporal product release stages guide\ -------------------------------------](https://docs.temporal.io/evaluate/development-production-features/release-stages) Discover Temporal's Product Release Stages Guide for detailed criteria on Pre-release, Public Preview, and General Availability. Make informed decisions on feature adoption! [Understanding Temporal\ ----------------------](https://docs.temporal.io/evaluate/understanding-temporal) This page provides a short overview of how Temporal works. [What is Temporal?\ -----------------](https://docs.temporal.io/temporal) Temporal is a scalable platform that ensures the Durable Execution of application code, allowing reliable and resilient Workflow Executions even in the face of failures like network outages or server crashes. [Why Temporal?\ -------------](https://docs.temporal.io/evaluate/why-temporal) Temporal enhances Workflow reliability, productivity, and state visibility for developers by offering Durable Execution, simplified code structures, and robust monitoring tools. --- # 12 docs tagged with "Durable Execution" | Temporal Platform Documentation [Skip to main content](https://docs.temporal.io/tags/durable-execution#__docusaurus_skipToContent_fallback) [About Temporal SDKs\ -------------------](https://docs.temporal.io/encyclopedia/temporal-sdks) Temporal SDKs are open-source tools enabling scalable and reliable application development. They feature APIs for Workflow and Activity execution, automatic retries, and resilience mechanisms, making it easier to build fault-tolerant applications. [Activity Definition\ -------------------](https://docs.temporal.io/activity-definition) Learn about defining Temporal Activities, including Activity Types, parameters, and implementation details. [Activity Execution\ ------------------](https://docs.temporal.io/activity-execution) Understand how Activity Executions work in Temporal, including retries, timeouts, and failure handling. [Local Activity\ --------------](https://docs.temporal.io/local-activity) Learn about Local Activities in Temporal, their benefits, execution model, and when to use them. [Standalone Activity\ -------------------](https://docs.temporal.io/standalone-activity) Learn about Standalone Activities in Temporal, their benefits, execution model, and when to use them. [Temporal Platform's production readiness checklist\ --------------------------------------------------](https://docs.temporal.io/self-hosted-guide/production-checklist) Optimize your Temporal Service for production with scaling, metrics, load testing, and effective workflow versioning techniques. Ensure robust performance and future-proof your workflows. [Temporal Worker Controller\ --------------------------](https://docs.temporal.io/production-deployment/worker-deployments/kubernetes-controller) Use Temporal's provided Kubernetes controller to programmatically scale your Worker deployments. [Temporal Worker deployments\ ---------------------------](https://docs.temporal.io/production-deployment/worker-deployments) Programmatically scale and deploy your Temporal Workers into common production environments using our provided tooling and best practices. [What is a Temporal Activity?\ ----------------------------](https://docs.temporal.io/activities) Understand Temporal Activities, including Activity Definitions, Types, Executions, idempotency, cancellations, Local Activities, and Standalone Activities. [What is Temporal?\ -----------------](https://docs.temporal.io/temporal) Temporal is a scalable platform that ensures the Durable Execution of application code, allowing reliable and resilient Workflow Executions even in the face of failures like network outages or server crashes. [Why Temporal?\ -------------](https://docs.temporal.io/evaluate/why-temporal) Temporal enhances Workflow reliability, productivity, and state visibility for developers by offering Durable Execution, simplified code structures, and robust monitoring tools. [Worker Versioning\ -----------------](https://docs.temporal.io/production-deployment/worker-deployments/worker-versioning) Use Worker Versioning to pin Workflow revisions to individual Worker Deployment Versions, avoiding the need for patching to support multiple code paths. --- # 5 docs tagged with "Timeouts" | Temporal Platform Documentation [Skip to main content](https://docs.temporal.io/tags/timeouts#__docusaurus_skipToContent_fallback) [Detecting Activity failures\ ---------------------------](https://docs.temporal.io/encyclopedia/detecting-activity-failures) Understand Activity Execution timeouts in Temporal; Schedule-To-Start, Start-To-Close, Schedule-To-Close, and Activity Heartbeats, for effective Workflow management. [Detecting application failures\ ------------------------------](https://docs.temporal.io/encyclopedia/detecting-application-failures) In Temporal, timeouts detect and mitigate Workflow and Activity failures with automatic retries using configurable timeout settings and customizable RetryPolicies. [Detecting Workflow failures\ ---------------------------](https://docs.temporal.io/encyclopedia/detecting-workflow-failures) Learn about Workflow Execution Timeout, Workflow Run Timeout, and Workflow Task Timeout in Temporal. Maximize Workflow efficiency and manage durations effectively. [Failure detection - Temporal feature\ ------------------------------------](https://docs.temporal.io/evaluate/development-production-features/failure-detection) Explore Temporal's robust timeout and Retry Policy features for Workflows and Activities. Start with our tutorials or dive deep with our SDK guides and Encyclopedia resources. [Temporal Failures reference\ ---------------------------](https://docs.temporal.io/references/failures) A Failure in Temporal represents different types of errors in the system, categorized and managed uniquely within SDKs and protobuf messages, impacting Workflow and Activity operations. --- # 12 docs tagged with "getting started" | Temporal Platform Documentation [Skip to main content](https://docs.temporal.io/tags/getting-started#__docusaurus_skipToContent_fallback) [Nexus .NET Quickstart\ ---------------------](https://docs.temporal.io/develop/dotnet/nexus/quickstart) Build a Nexus Service that wraps an existing Temporal Workflow using the .NET SDK [Nexus Go Quickstart\ -------------------](https://docs.temporal.io/develop/go/nexus/quickstart) Build a Nexus Service that wraps an existing Temporal Workflow using the Go SDK [Nexus Java Quickstart\ ---------------------](https://docs.temporal.io/develop/java/nexus/quickstart) Build a Nexus Service that wraps an existing Temporal Workflow using the Java SDK [Nexus Python Quickstart\ -----------------------](https://docs.temporal.io/develop/python/nexus/quickstart) Build a Nexus Service that wraps an existing Temporal Workflow using the Python SDK [Nexus TypeScript Quickstart\ ---------------------------](https://docs.temporal.io/develop/typescript/nexus/quickstart) Build a Nexus Service that wraps an existing Temporal Workflow using the TypeScript SDK [Set up your local development with the PHP SDK\ ----------------------------------------------](https://docs.temporal.io/develop/php/set-up-your-local-php) Configure your local development environment to get started developing with Temporal [Set up your local with the .NET SDK\ -----------------------------------](https://docs.temporal.io/develop/dotnet/set-up-your-local-dotnet) Configure your local development environment to get started developing with Temporal [Set up your local with the Go SDK\ ---------------------------------](https://docs.temporal.io/develop/go/set-up-your-local-go) Configure your local development environment to get started developing with Temporal [Set up your local with the Java SDK\ -----------------------------------](https://docs.temporal.io/develop/java/set-up-your-local-java) Configure your local development environment to get started developing with Temporal [Set up your local with the Python SDK\ -------------------------------------](https://docs.temporal.io/develop/python/set-up-your-local-python) Configure your local development environment to get started developing with Temporal [Set up your local with the Ruby SDK\ -----------------------------------](https://docs.temporal.io/develop/ruby/set-up-local-ruby) Configure your local development environment to get started developing with Temporal [Set up your local with the Typescript SDK\ -----------------------------------------](https://docs.temporal.io/develop/typescript/set-up-your-local-typescript) Configure your local development environment to get started developing with Temporal --- # 9 docs tagged with "Queries" | Temporal Platform Documentation [Skip to main content](https://docs.temporal.io/tags/queries#__docusaurus_skipToContent_fallback) [Handling Signals, Queries, & Updates\ ------------------------------------](https://docs.temporal.io/handling-messages) Signals, Queries, and Updates facilitate interactions with Workflow Executions. [Sending Signals, Queries, & Updates\ -----------------------------------](https://docs.temporal.io/sending-messages) Signals, Queries, and Updates facilitate interactions with Workflow Executions. [Temporal Workflow message passing - Signals, Queries, & Updates\ ---------------------------------------------------------------](https://docs.temporal.io/encyclopedia/workflow-message-passing/) Signals, Queries, and Updates facilitate interactions with Workflow Executions. [Workflow message passing - Go SDK\ ---------------------------------](https://docs.temporal.io/develop/go/workflows/message-passing) Develop with Queries, Signals, and Updates with the Temporal Go SDK. [Workflow message passing - Java SDK\ -----------------------------------](https://docs.temporal.io/develop/java/workflows/message-passing) Develop with Queries, Signals, and Updates with the Temporal Java SDK. [Workflow message passing - PHP SDK\ ----------------------------------](https://docs.temporal.io/develop/php/workflows/message-passing) Develop with Signals, Queries, and Updates in Temporal Workflows. Define, handle, and send Signals or Queries, and validate updates from a Temporal Client. [Workflow message passing - Python SDK\ -------------------------------------](https://docs.temporal.io/develop/python/workflows/message-passing) Develop with Queries, Signals, and Updates with the Temporal Python SDK. [Workflow message passing - Temporal feature\ -------------------------------------------](https://docs.temporal.io/evaluate/development-production-features/workflow-message-passing) Enhance your Workflows with Signals and Queries, allowing dynamic responses to external events and real-time state access for comprehensive monitoring and tracking. [Workflow message passing - TypeScript SDK\ -----------------------------------------](https://docs.temporal.io/develop/typescript/workflows/message-passing) Develop with Queries, Signals, and Updates with the Temporal Typescript SDK. --- # 9 docs tagged with "Search Attributes" | Temporal Platform Documentation [Skip to main content](https://docs.temporal.io/tags/search-attributes#__docusaurus_skipToContent_fallback) [Observability - .NET SDK\ ------------------------](https://docs.temporal.io/develop/dotnet/platform/observability) Explore Temporal SDK observability features for Metrics, Tracing, Logging, and Visibility. Track Workflow Executions, set up Prometheus endpoints, customize metrics, configure tracing, and more. [Observability - Go SDK\ ----------------------](https://docs.temporal.io/develop/go/platform/observability) Monitor your Temporal Application state using Metrics, Tracing, Logging, and Visibility features. Emit metrics, configure tracing, customize logging, and use Search Attributes with the Temporal Go SDK for enhanced Workflow Execution insights. [Observability - Java SDK\ ------------------------](https://docs.temporal.io/develop/java/platform/observability) Explore the observability features of Temporal, including Metrics, Tracing, Logging, and Visibility. Emit Metrics with the Java SDK, set up Tracing, and use Search Attributes. [Observability - PHP SDK\ -----------------------](https://docs.temporal.io/develop/php/platform/observability) Explore the Temporal Developer’s guide on observability to learn about Visibility APIs and Search Attributes, helping you manage Workflow Executions efficiently. [Observability - Python SDK\ --------------------------](https://docs.temporal.io/develop/python/platform/observability) Discover how to monitor your Temporal Application using metrics, tracing, logging, and visibility APIs. Emit metrics, set up tracing, log from Workflows, and use custom Search Attributes. [Observability - Ruby SDK\ ------------------------](https://docs.temporal.io/develop/ruby/platform/observability) Explore Temporal SDK observability features for Metrics, Tracing, Logging, and Visibility using the Ruby SDK. [Observability - TypeScript SDK\ ------------------------------](https://docs.temporal.io/develop/typescript/platform/observability) Enhance the observability of your Temporal Application with metrics, tracing, logging, and visibility features. View Workflow state, set up OpenTelemetry, and customize logging for seamless monitoring and insights. [Search Attributes\ -----------------](https://docs.temporal.io/search-attribute) This guide on Temporal Search Attributes explains how to set up, configure, and use default and custom Search Attributes in Temporal Server versions. Learn about supported types, limits, and how to use them to enhance Workflow filtering and querying. [Temporal Visibility\ -------------------](https://docs.temporal.io/visibility) Temporal Visibility enables operators to view, filter, and search Workflow Executions using List Filters and Search Attributes. Learn about supported databases, Dual Visibility, and custom Search Attributes. --- # 9 docs tagged with "Schedules" | Temporal Platform Documentation [Skip to main content](https://docs.temporal.io/tags/schedules#__docusaurus_skipToContent_fallback) [Schedules - .NET SDK\ --------------------](https://docs.temporal.io/develop/dotnet/workflows/schedules) Manage and optimize Scheduled Workflows using the Temporal .NET SDK; Schedule, Create, Backfill, Update, Delete, Describe, List, Pause, Trigger, and use Start Delay options. [Schedules - Go SDK\ ------------------](https://docs.temporal.io/develop/go/workflows/schedules) Schedule Workflows, start them with delays or as Temporal Cron Jobs using the Go SDK. Master scheduling, backfilling, pausing, deleting, and updating Workflows. [Schedules - Java SDK\ --------------------](https://docs.temporal.io/develop/java/workflows/schedules) Schedule, Backfill, Delete, Describe, List, Pause, Trigger, Update, and set Cron and Start Delays for Workflow Executions in Java using Temporal's ScheduleClient. [Schedules - PHP SDK\ -------------------](https://docs.temporal.io/develop/php/workflows/schedules) Use Workflow Start Delay and Temporal Cron Jobs in PHP. Delay Workflow execution or set up recurring tasks with a Cron Schedule using Temporal Client. [Schedules - Python SDK\ ----------------------](https://docs.temporal.io/develop/python/workflows/schedules) Schedule, Create, Backfill, Delete, Describe, List, Pause, Trigger, and Update a Scheduled Workflow, along with Temporal Cron Jobs and Start Delay options. [Schedules - Ruby SDK\ --------------------](https://docs.temporal.io/develop/ruby/workflows/schedules) Manage and optimize Scheduled Workflows using the Temporal Ruby SDK; Schedule, Create, Backfill, Update, Delete, Describe, List, Pause, Trigger, and more. [Schedules - Temporal feature\ ----------------------------](https://docs.temporal.io/evaluate/development-production-features/schedules) Learn the benefits of scheduling Temporal Workflows and explore best practices to ensure timely and efficient execution of your business processes. [Schedules - TypeScript SDK\ --------------------------](https://docs.temporal.io/develop/typescript/workflows/schedules) Schedule automated tasks effortlessly with Temporal. Create, backfill, delete, describe, list, pause, trigger, and update Schedules. Control your Workflow execution with Temporal Cron Jobs and ensure timely, automated business processes. Automate repetitive tasks and reduce manual intervention now! [Temporal CLI schedule command reference\ ---------------------------------------](https://docs.temporal.io/cli/schedule) Temporal's Schedule commands allow users to create, update, and manage Workflow Executions seamlessly for automation, supporting commands for creation, backfill, deletion, and more. --- # 12 docs tagged with "Namespaces" | Temporal Platform Documentation [Skip to main content](https://docs.temporal.io/tags/namespaces#__docusaurus_skipToContent_fallback) [Global Namespace\ ----------------](https://docs.temporal.io/global-namespace) This guide covers everything about Global Namespaces within the Temporal Platform. [Manage Namespaces - TypeScript SDK\ ----------------------------------](https://docs.temporal.io/develop/typescript/client/namespaces) Efficiently create and manage Namespaces on Temporal using CLI or SDK APIs. Isolate Workflow Executions, control access with custom Authorizers, and manage via Temporal Cloud UI or CLI. [Manage users\ ------------](https://docs.temporal.io/cloud/users) Manage user invitations, account-level roles, and Namespace-level permissions in Temporal Cloud. Invite users, update roles, and delete users seamlessly using the Temporal Web UI, tcld, or the Cloud Ops API. [Managing Namespaces\ -------------------](https://docs.temporal.io/self-hosted-guide/namespaces) How to create and manage Namespaces in open source Temporal, including registration, configuration, and security. [Namespace best practices\ ------------------------](https://docs.temporal.io/best-practices/managing-namespace) Best practices for organizing and managing Temporal Namespaces, including naming conventions, organizational patterns, and production safeguards. [Namespaces\ ----------](https://docs.temporal.io/cloud/namespaces) A Namespace is a unit of isolation within Temporal Cloud, providing security boundaries, Workflow management, unique identifiers, and gRPC endpoints in Temporal Cloud. [Namespaces - Go SDK\ -------------------](https://docs.temporal.io/develop/go/client/namespaces) Register and manage Namespaces in Temporal using CLI or SDK APIs. Isolate Workflow Executions, match development lifecycles, and secure Namespace workflows. [Namespaces - Java SDK\ ---------------------](https://docs.temporal.io/develop/java/client/namespaces) Register, update, deprecate, and delete Namespaces using Temporal CLI or SDK APIs. Manage Workflow Executions with isolated Namespaces to match your needs. [tcld namespace command reference\ --------------------------------](https://docs.temporal.io/cloud/tcld/namespace/) Unlock the full potential of Temporal Cloud with tcld namespace commands. Efficiently manage Namespace operations, including add-region, create, delete, failover, get, list, and export. [Temporal Namespace\ ------------------](https://docs.temporal.io/namespaces) A Namespace is a unit of isolation within the Temporal Platform that provides resource separation, Workflow ID uniqueness, and configuration boundaries. [Troubleshoot the failed reaching server error\ ---------------------------------------------](https://docs.temporal.io/troubleshooting/last-connection-error) Troubleshoot server connection errors often caused by expired TLS certificates. Verify, renew, and update server configurations to resolve temporal client request issues effectively. [User management\ ---------------](https://docs.temporal.io/cloud/users-invite) Learn how to manage user invitations for Temporal Cloud --- # 9 docs tagged with "Signals" | Temporal Platform Documentation [Skip to main content](https://docs.temporal.io/tags/signals#__docusaurus_skipToContent_fallback) [Handling Signals, Queries, & Updates\ ------------------------------------](https://docs.temporal.io/handling-messages) Signals, Queries, and Updates facilitate interactions with Workflow Executions. [Sending Signals, Queries, & Updates\ -----------------------------------](https://docs.temporal.io/sending-messages) Signals, Queries, and Updates facilitate interactions with Workflow Executions. [Temporal Workflow message passing - Signals, Queries, & Updates\ ---------------------------------------------------------------](https://docs.temporal.io/encyclopedia/workflow-message-passing/) Signals, Queries, and Updates facilitate interactions with Workflow Executions. [Workflow message passing - Go SDK\ ---------------------------------](https://docs.temporal.io/develop/go/workflows/message-passing) Develop with Queries, Signals, and Updates with the Temporal Go SDK. [Workflow message passing - Java SDK\ -----------------------------------](https://docs.temporal.io/develop/java/workflows/message-passing) Develop with Queries, Signals, and Updates with the Temporal Java SDK. [Workflow message passing - PHP SDK\ ----------------------------------](https://docs.temporal.io/develop/php/workflows/message-passing) Develop with Signals, Queries, and Updates in Temporal Workflows. Define, handle, and send Signals or Queries, and validate updates from a Temporal Client. [Workflow message passing - Python SDK\ -------------------------------------](https://docs.temporal.io/develop/python/workflows/message-passing) Develop with Queries, Signals, and Updates with the Temporal Python SDK. [Workflow message passing - Temporal feature\ -------------------------------------------](https://docs.temporal.io/evaluate/development-production-features/workflow-message-passing) Enhance your Workflows with Signals and Queries, allowing dynamic responses to external events and real-time state access for comprehensive monitoring and tracking. [Workflow message passing - TypeScript SDK\ -----------------------------------------](https://docs.temporal.io/develop/typescript/workflows/message-passing) Develop with Queries, Signals, and Updates with the Temporal Typescript SDK. --- # 10 docs tagged with "Production" | Temporal Platform Documentation [Skip to main content](https://docs.temporal.io/tags/production#__docusaurus_skipToContent_fallback) [Automated migration\ -------------------](https://docs.temporal.io/cloud/migrate/automated) Automated migration is designed to provide a zero-downtime, secure means of migrating to Temporal Cloud. This guide outlines the current process for transitioning workflows from a self-hosted setup to one hosted within Temporal Cloud. [AWS PrivateLink connectivity\ ----------------------------](https://docs.temporal.io/cloud/connectivity/aws-connectivity) Connect to Temporal Cloud using AWS PrivateLink [Configure and trigger failovers\ -------------------------------](https://docs.temporal.io/cloud/high-availability/failovers) How automatic and manual failovers work with Temporal Cloud HA [Connectivity\ ------------](https://docs.temporal.io/cloud/connectivity) Network connectivity details for using Temporal Cloud [Google Private Service Connect connectivity\ -------------------------------------------](https://docs.temporal.io/cloud/connectivity/gcp-connectivity) Connect to Temporal Cloud using Google Private Services Connect [High Availability\ -----------------](https://docs.temporal.io/cloud/high-availability) Temporal Cloud's Namespace with High Availability features offers automated failover, synchronized data, and replication for workloads requiring disaster-tolerant deployment and 99.99% uptime. [Manual migration\ ----------------](https://docs.temporal.io/cloud/migrate/manual) Migrating to Temporal Cloud from self-hosted Temporal Service varies by Workflow requirements. This guide covers changing Client code, Workflow migration strategies, and necessary code adjustments. [Migrate between regions\ -----------------------](https://docs.temporal.io/cloud/migrate/migrate-within-cloud) Use Temporal Cloud's High Availability features to migrate between regions. [Monitoring High Availability\ ----------------------------](https://docs.temporal.io/cloud/high-availability/monitoring) How to track the health and performance of your High Availability Namespaces. [Temporal Cloud IP addresses\ ---------------------------](https://docs.temporal.io/cloud/connectivity/ip-addresses) Temporal Cloud IP addresses --- # 2 docs tagged with "Workflow" | Temporal Platform Documentation [Skip to main content](https://docs.temporal.io/tags/workflow#__docusaurus_skipToContent_fallback) [Cancellation scopes - TypeScript SDK\ ------------------------------------](https://docs.temporal.io/develop/typescript/workflows/cancellation-scopes) Shows cancellation scopes with the TypeScript SDK [Workflow basics - TypeScript SDK\ --------------------------------](https://docs.temporal.io/develop/typescript/workflows/basics) Shows how to create a Workflow with the TypeScript SDK --- # 15 docs tagged with "Encryption" | Temporal Platform Documentation [Skip to main content](https://docs.temporal.io/tags/encryption#__docusaurus_skipToContent_fallback) [Codec Server\ ------------](https://docs.temporal.io/codec-server) A Codec Server is an HTTP server that provides remote encoding and decoding for Temporal Payloads. [Converters and encryption - .NET SDK\ ------------------------------------](https://docs.temporal.io/develop/dotnet/best-practices/converters-and-encryption) Use a custom Payload Codec and Converter in the .NET SDK to modify Temporal Data Conversion behavior, including examples for encryption and camel case conversion. [Converters and encryption - Java SDK\ ------------------------------------](https://docs.temporal.io/develop/java/best-practices/converters-and-encryption) Create and implement a Custom Payload Codec and Payload Converter in Java using the Temporal SDK for custom data encryption, compression, and type conversion. [Converters and encryption - Ruby SDK\ ------------------------------------](https://docs.temporal.io/develop/ruby/best-practices/converters-and-encryption) Use a custom Payload Codec and Converter in the Ruby SDK to modify Temporal Data Conversion behavior, including examples for encryption and formatting. [Converters and encryption - TypeScript SDK\ ------------------------------------------](https://docs.temporal.io/develop/typescript/converters-and-encryption) Create a custom Payload Converter in TypeScript with Temporal SDKs to handle non-JSON-serializable values, configure your Data Converter, and use protobufs and encryption seamlessly in your Workflows and Activities. [Data encryption - Temporal feature\ ----------------------------------](https://docs.temporal.io/evaluate/development-production-features/data-encryption) Implement data encryption in your Temporal Workflows to ensure the security and confidentiality of your data. [Default and Custom Data Converters\ ----------------------------------](https://docs.temporal.io/default-custom-data-converters) Learn about the default Data Converter in Temporal SDKs and how to implement a custom Data Converter for custom serialization and encoding needs. [Failure Converter\ -----------------](https://docs.temporal.io/failure-converter) A Failure Converter transforms error messages and call stacks into encoded formats to enhance security and observability. [How does Temporal handle application data?\ ------------------------------------------](https://docs.temporal.io/dataconversion) This guide explores Data Converters in the Temporal Platform, detailing how they handle serialization and encoding for Workflow inputs and outputs, ensuring data stays secure and manageable. [Key management\ --------------](https://docs.temporal.io/key-management) Learn about key management practices for securing encryption keys in Temporal applications. [Payload Codec\ -------------](https://docs.temporal.io/payload-codec) A Payload Codec performs bytes-to-bytes transformations on Temporal Payloads, often for compression and encryption. [Payload Converter\ -----------------](https://docs.temporal.io/payload-converter) A Payload Converter serializes and deserializes values to and from bytes for use in the Temporal SDK. [Payload encryption - Go SDK\ ---------------------------](https://docs.temporal.io/develop/go/data-handling/data-encryption) Encrypt data sent to and from the Temporal Service using a custom Payload Codec in the Go SDK. [Payload encryption - Python SDK\ -------------------------------](https://docs.temporal.io/develop/python/data-handling/data-encryption) Encrypt data sent to and from the Temporal Service using a custom Payload Codec in the Python SDK. [Remote data encoding\ --------------------](https://docs.temporal.io/remote-data-encoding) Use remote encoding to transform data for the Temporal CLI and Web UI. --- # 4 docs tagged with "User groups" | Temporal Platform Documentation [Skip to main content](https://docs.temporal.io/tags/user-groups#__docusaurus_skipToContent_fallback) [Account access\ --------------](https://docs.temporal.io/cloud/manage-access/) Manage access to your Temporal Cloud account [Manage user groups\ ------------------](https://docs.temporal.io/cloud/user-groups) Learn how to manage user groups, members, and roles [SCIM user management\ --------------------](https://docs.temporal.io/cloud/scim) Link your IdP with your Temporal Cloud account to securely automate user and group management. [tcld user group command reference\ ---------------------------------](https://docs.temporal.io/cloud/tcld/user-group/) The tcld user-group commands manage user groups in Temporal Cloud. --- # 8 docs tagged with "Testing" | Temporal Platform Documentation [Skip to main content](https://docs.temporal.io/tags/testing#__docusaurus_skipToContent_fallback) [Temporal Testing Suite - Temporal feature\ -----------------------------------------](https://docs.temporal.io/evaluate/development-production-features/testing-suite) Explore Temporal's comprehensive testing suite; Frameworks that facilitate Workflow and integration testing across different programming languages with Temporal. [Testing - .NET SDK\ ------------------](https://docs.temporal.io/develop/dotnet/best-practices/testing-suite) The .NET test-suite guide covers Workflow and integration testing for Temporal. It includes end-to-end, integration, and unit testing, emphasizing the use of the test server to optimize test execution. [Testing - Go SDK\ ----------------](https://docs.temporal.io/develop/go/best-practices/testing-suite) The Testing section of the Temporal Application development guide details frameworks for Workflow and integration testing. Create end-to-end, integration, unit tests, and more for Workflows and Activities. Each test runs in an isolated environment, ensuring accurate and reliable testing. Discover how to mock and override Activities, test [Testing - Java SDK\ ------------------](https://docs.temporal.io/develop/java/best-practices/testing-suite) The Testing section of the Temporal Application development guide covers frameworks for Workflow and integration testing, including end-to-end, integration, and unit tests. Unit tests can be set up and run using the Temporal Java SDK's TestWorkflowEnvironment and TestWorkflowExtension classes for automated testing, allowing developers to test Workflows [Testing - PHP SDK\ -----------------](https://docs.temporal.io/develop/php/best-practices/testing-suite) The Temporal Application Testing section explains frameworks for Workflow and integration testing, including end-to-end, integration, unit tests, and how to mock Activities in a PHP environment. [Testing - Python SDK\ --------------------](https://docs.temporal.io/develop/python/best-practices/testing-suite) The Temporal Application Testing guide covers Frameworks facilitating Workflow and integration testing, including end-to-end, integration, and unit tests. Use mocked Activities, skip time in tests, and replay Workflow Executions. [Testing - Ruby SDK\ ------------------](https://docs.temporal.io/develop/ruby/best-practices/testing-suite) The Ruby test-suite guide covers Workflow and integration testing for Temporal. It includes end-to-end, integration, and unit testing, emphasizing the use of the test server to optimize test execution. [Testing - TypeScript SDK\ ------------------------](https://docs.temporal.io/develop/typescript/best-practices/testing-suite) The Testing section of the Temporal Application development guide covers frameworks for Workflow and integration testing, including end-to-end, integration, unit testing, and time-skipping functionalities. --- # 12 docs tagged with "setup" | Temporal Platform Documentation [Skip to main content](https://docs.temporal.io/tags/setup#__docusaurus_skipToContent_fallback) [Nexus .NET Quickstart\ ---------------------](https://docs.temporal.io/develop/dotnet/nexus/quickstart) Build a Nexus Service that wraps an existing Temporal Workflow using the .NET SDK [Nexus Go Quickstart\ -------------------](https://docs.temporal.io/develop/go/nexus/quickstart) Build a Nexus Service that wraps an existing Temporal Workflow using the Go SDK [Nexus Java Quickstart\ ---------------------](https://docs.temporal.io/develop/java/nexus/quickstart) Build a Nexus Service that wraps an existing Temporal Workflow using the Java SDK [Nexus Python Quickstart\ -----------------------](https://docs.temporal.io/develop/python/nexus/quickstart) Build a Nexus Service that wraps an existing Temporal Workflow using the Python SDK [Nexus TypeScript Quickstart\ ---------------------------](https://docs.temporal.io/develop/typescript/nexus/quickstart) Build a Nexus Service that wraps an existing Temporal Workflow using the TypeScript SDK [Set up your local development with the PHP SDK\ ----------------------------------------------](https://docs.temporal.io/develop/php/set-up-your-local-php) Configure your local development environment to get started developing with Temporal [Set up your local with the .NET SDK\ -----------------------------------](https://docs.temporal.io/develop/dotnet/set-up-your-local-dotnet) Configure your local development environment to get started developing with Temporal [Set up your local with the Go SDK\ ---------------------------------](https://docs.temporal.io/develop/go/set-up-your-local-go) Configure your local development environment to get started developing with Temporal [Set up your local with the Java SDK\ -----------------------------------](https://docs.temporal.io/develop/java/set-up-your-local-java) Configure your local development environment to get started developing with Temporal [Set up your local with the Python SDK\ -------------------------------------](https://docs.temporal.io/develop/python/set-up-your-local-python) Configure your local development environment to get started developing with Temporal [Set up your local with the Ruby SDK\ -----------------------------------](https://docs.temporal.io/develop/ruby/set-up-local-ruby) Configure your local development environment to get started developing with Temporal [Set up your local with the Typescript SDK\ -----------------------------------------](https://docs.temporal.io/develop/typescript/set-up-your-local-typescript) Configure your local development environment to get started developing with Temporal --- # 7 docs tagged with "UI Enrichment" | Temporal Platform Documentation [Skip to main content](https://docs.temporal.io/tags/ui-enrichment#__docusaurus_skipToContent_fallback) [Enriching the user interface - .NET SDK\ ---------------------------------------](https://docs.temporal.io/develop/dotnet/platform/enriching-ui) Add contextual information to Workflows and events in the Temporal UI using the .NET SDK. [Enriching the user interface - Go SDK\ -------------------------------------](https://docs.temporal.io/develop/go/platform/enriching-ui) Add contextual information to workflows and events in the Temporal UI using the Go SDK. [Enriching the user interface - Java SDK\ ---------------------------------------](https://docs.temporal.io/develop/java/platform/enriching-ui) Add contextual information to workflows and events in the Temporal UI using the Java SDK. [Enriching the user interface - PHP SDK\ --------------------------------------](https://docs.temporal.io/develop/php/platform/enriching-ui) Add contextual information to workflows and events in the Temporal UI using the PHP SDK. [Enriching the user interface - Python SDK\ -----------------------------------------](https://docs.temporal.io/develop/python/platform/enriching-ui) Add contextual information to workflows and events in the Temporal UI using the Python SDK. [Enriching the user interface - Ruby SDK\ ---------------------------------------](https://docs.temporal.io/develop/ruby/platform/enriching-ui) Add contextual information to workflows and events in the Temporal UI using the Ruby SDK. [Enriching the user interface - TypeScript SDK\ ---------------------------------------------](https://docs.temporal.io/develop/typescript/platform/enriching-ui) Add contextual information to workflows and events in the Temporal UI using the TypeScript SDK. --- # 16 docs tagged with "Metrics" | Temporal Platform Documentation [Skip to main content](https://docs.temporal.io/tags/metrics#__docusaurus_skipToContent_fallback) [General observability setup with metrics\ ----------------------------------------](https://docs.temporal.io/cloud/metrics/general-setup) Learn how to configure a metrics endpoint in Temporal Cloud using the UI or tcld CLI, assign certificates, and integrate with observability tools like Grafana. [Metrics integrations\ --------------------](https://docs.temporal.io/cloud/metrics/openmetrics/metrics-integrations) Integrating with the Temporal Cloud OpenMetrics endpoint. [Monitor SDK metrics with Prometheus and Grafana\ -----------------------------------------------](https://docs.temporal.io/cloud/metrics/sdk-metrics-setup) Set up Temporal SDK metrics with Prometheus and Grafana for monitoring Workers and Client performance. [Monitor worker health\ ---------------------](https://docs.temporal.io/cloud/worker-health) Detect and configure for Task backlogs, greedy Worker resources, misconfigured Workers, and Sticky cache settings. Optimize alert systems and get actionable insights on metrics like Schedule-To-Start latency, Sync Match Rate, and Poll Success Rate for improved application health. [Observability - Temporal feature\ --------------------------------](https://docs.temporal.io/evaluate/development-production-features/observability) Explore the observability and visibility features of Temporal, including Metrics, Tracing, Logging, and Visibility. [OpenMetrics API reference\ -------------------------](https://docs.temporal.io/cloud/metrics/openmetrics/api-reference) Detailed API documentation for the Temporal Cloud OpenMetrics endpoint. [OpenMetrics metrics reference\ -----------------------------](https://docs.temporal.io/cloud/metrics/openmetrics/metrics-reference) Detailed API documentation for the Temporal Cloud OpenMetrics endpoint. [OpenMetrics migration guide\ ---------------------------](https://docs.temporal.io/cloud/metrics/openmetrics/migration-guide) Migrate from the Prometheus query endpoint to the new OpenMetrics endpoint in Temporal Cloud. [OSS Temporal Service metrics reference\ --------------------------------------](https://docs.temporal.io/references/cluster-metrics) A Temporal Service emits metrics helping operators monitor performance and set alerts. Metrics like service requests, latencies, and errors are tracked. Use metric\_defs.go for more. [Performance bottlenecks troubleshooting guide\ ---------------------------------------------](https://docs.temporal.io/troubleshooting/performance-bottlenecks) Diagnose and resolve performance bottlenecks using Temporal SDK metrics [Prometheus Grafana setup\ ------------------------](https://docs.temporal.io/cloud/metrics/prometheus-grafana) Set up Grafana with Temporal Cloud observability to monitor performance and troubleshoot errors using the Prometheus HTTP API endpoint. [PromQL Metrics\ --------------](https://docs.temporal.io/cloud/metrics/promql) Get detailed insights into your Temporal Cloud Namespace metrics using your own observability tool. Access data with a CA certificate and retain raw metrics for seven days. [Temporal Cloud Metrics\ ----------------------](https://docs.temporal.io/cloud/metrics/) Monitor Temporal Cloud workloads with Cloud metrics and SDK metrics. [Temporal Cloud metrics reference\ --------------------------------](https://docs.temporal.io/cloud/metrics/reference) Explore Temporal Cloud metrics to query with PromQL or scrape via OpenMetrics, supporting rate and latency calculations. [Temporal Cloud OpenMetrics\ --------------------------](https://docs.temporal.io/cloud/metrics/openmetrics/) Export metrics from Temporal Cloud using the OpenMetrics standard and third party integrations. [Temporal SDK metrics reference\ ------------------------------](https://docs.temporal.io/references/sdk-metrics) Temporal SDKs emit metrics covering Client usage and Worker Processes. Metrics can be tuned to improve Worker performance and are prefixed with temporal\_ before export. --- # 14 docs tagged with "Reference" | Temporal Platform Documentation [Skip to main content](https://docs.temporal.io/tags/reference#__docusaurus_skipToContent_fallback) [Errors\ ------](https://docs.temporal.io/references/errors) This reference outlines possible Workflow Task errors, causes, and resolution steps in Temporal. It covers various error scenarios such as attribute failures, size limits, and system resource issues. [Glossary\ --------](https://docs.temporal.io/glossary) The following terms have specific definitions within the context of the Temporal Platform. [OSS Temporal Service metrics reference\ --------------------------------------](https://docs.temporal.io/references/cluster-metrics) A Temporal Service emits metrics helping operators monitor performance and set alerts. Metrics like service requests, latencies, and errors are tracked. Use metric\_defs.go for more. [Temporal Cluster configuration reference\ ----------------------------------------](https://docs.temporal.io/references/configuration) Configure a Temporal Cluster using the development.yaml file to set global parameters, metrics, security, persistence, and service roles, ensuring a streamlined setup and management process. [Temporal Cluster dynamic configuration reference\ ------------------------------------------------](https://docs.temporal.io/references/dynamic-configuration) Temporal Cluster offers dynamic configuration keys that you can update on the fly to optimize performance without service interruption. Customize these settings to meet specific Workflow, Activity, Namespace, or Task Queue requirements, and test thoroughly before deploying to production. For more details, visit the Temporal GitHub repository. [Temporal Commands reference\ ---------------------------](https://docs.temporal.io/references/commands) Discover the range of Commands Workers can issue to the Temporal Service after Workflow Task Execution, from Completing Workflow Execution to Start Timer and Signal External Workflow Execution. [Temporal Events reference\ -------------------------](https://docs.temporal.io/references/events) Events in a Temporal Service respond to external occurrences and Workflow Commands. Workflow Execution Event History includes WorkflowExecutionStarted, WorkflowExecutionCompleted, WorkflowExecutionFailed, and many more. [Temporal Failures reference\ ---------------------------](https://docs.temporal.io/references/failures) A Failure in Temporal represents different types of errors in the system, categorized and managed uniquely within SDKs and protobuf messages, impacting Workflow and Activity operations. [Temporal Platform references\ ----------------------------](https://docs.temporal.io/references/) Explore comprehensive references for SDK Metrics, Commands, Events, Web UI environment variables, Temporal Service and Web UI configurations, and API guides for Go, Java, Python, TypeScript, .NET, and PHP. [Temporal SDK metrics reference\ ------------------------------](https://docs.temporal.io/references/sdk-metrics) Temporal SDKs emit metrics covering Client usage and Worker Processes. Metrics can be tuned to improve Worker performance and are prefixed with temporal\_ before export. [Temporal Server options reference\ ---------------------------------](https://docs.temporal.io/references/server-options) Run the Temporal Server as a Go application by incorporating the package go.temporal.io/server/temporal. Customize server options like Config, Authorization, and TLS. [Temporal Web UI configuration reference\ ---------------------------------------](https://docs.temporal.io/references/web-ui-configuration) Manage your Temporal Server efficiently with development.yaml. Set parameters for Auth, TLS, ports, and more. [Temporal Web UI environment variables reference\ -----------------------------------------------](https://docs.temporal.io/references/web-ui-environment-variables) Dynamically configure Temporal Web UI with environment variables in Docker for settings like TEMPORAL\_ADDRESS, authentication, TLS, OpenAPI, and more. [Worker tuning quick reference\ -----------------------------](https://docs.temporal.io/develop/worker-tuning-reference) A quick reference guide for Temporal Worker configuration defaults across SDKs, organized by resource type (compute, memory, IO) with key metrics for each. --- # 12 docs tagged with "Temporal CLI" | Temporal Platform Documentation [Skip to main content](https://docs.temporal.io/tags/temporal-cli#__docusaurus_skipToContent_fallback) [Temporal CLI activity command reference\ ---------------------------------------](https://docs.temporal.io/cli/activity) Learn how to use Temporal Activity commands to perform operations on Activity Executions. [Temporal CLI batch command reference\ ------------------------------------](https://docs.temporal.io/cli/batch) Use Temporal CLI to manage multiple Workflow Executions with Batch Jobs that can Cancel, Signal, or Terminate Workflows. Filter and monitor Batch Jobs effectively. [Temporal CLI command options reference\ --------------------------------------](https://docs.temporal.io/cli/cmd-options) Discover how to manage Temporal Workflows, from Activity Execution to Workflow Ids, using clusters, cron schedules, dynamic configurations, and logging. Perfect for developers. [Temporal CLI command reference\ ------------------------------](https://docs.temporal.io/cli) The Temporal CLI offers terminal access to Temporal Services for managing, monitoring, and debugging Workflows and Activities, including Namespace and Task Queue management, with embedded development support. [Temporal CLI config command reference\ -------------------------------------](https://docs.temporal.io/cli/config) Temporal CLI 'config' commands allow the getting, setting, deleting, and listing of configuration properties for connecting to Temporal. [Temporal CLI env command reference\ ----------------------------------](https://docs.temporal.io/cli/env) Temporal CLI 'env' commands allow the configuration, setting, deleting, and listing of environmental properties, making it easy to manage Temporal Server instances. [Temporal CLI operator command reference\ ---------------------------------------](https://docs.temporal.io/cli/operator) Operator commands in Temporal allow actions on Namespaces, Search Attributes, Clusters and Nexus Endpoints using specific subcommands. Execute with "temporal operator \[command\] \[subcommand\] \[options\]". [Temporal CLI schedule command reference\ ---------------------------------------](https://docs.temporal.io/cli/schedule) Temporal's Schedule commands allow users to create, update, and manage Workflow Executions seamlessly for automation, supporting commands for creation, backfill, deletion, and more. [Temporal CLI server command reference\ -------------------------------------](https://docs.temporal.io/cli/server) Manage your Temporal Server easily with CLI commands. Start a local server using \`temporal server start-dev\` and access the Web UI at http://localhost:8233. Customize with multiple options. [Temporal CLI task-queue command reference\ -----------------------------------------](https://docs.temporal.io/cli/task-queue) Temporal Task Queue commands facilitate operations like describing poller info, displaying partitions, fetching compatible Build IDs, and determining Build ID reachability for effective Workflow and Activity management. [Temporal CLI worker command reference\ -------------------------------------](https://docs.temporal.io/cli/worker) Learn how to read or modify state associated with a Worker, such as Worker Deployments. [Temporal CLI workflow command reference\ ---------------------------------------](https://docs.temporal.io/cli/workflow) Temporal Workflow commands enable operations on Workflow Executions, such as cancel, count, delete, describe, execute, list, update-options, query, reset, reset-batch, show, signal, stack, start, terminate, trace, and update, enhancing efficiency and control. --- # 6 docs tagged with "Worker" | Temporal Platform Documentation [Skip to main content](https://docs.temporal.io/tags/worker#__docusaurus_skipToContent_fallback) [Run Worker processes - Go SDK\ -----------------------------](https://docs.temporal.io/develop/go/workers/run-worker-process) Shows how to run Worker processes with the Go SDK [Run Worker processes - PHP SDK\ ------------------------------](https://docs.temporal.io/develop/php/workers/run-worker-process) Shows how to run Worker processes with the PHP SDK [Worker processes - .NET SDK\ ---------------------------](https://docs.temporal.io/develop/dotnet/workers/run-worker-process) Shows how to run Worker processes with the .NET SDK [Worker processes - Python SDK\ -----------------------------](https://docs.temporal.io/develop/python/workers/run-worker-process) Shows how to run Worker processes with the Python SDK [Worker processes - Ruby SDK\ ---------------------------](https://docs.temporal.io/develop/ruby/workers/run-worker-process) Shows how to run Worker processes with the Ruby SDK [Worker processes - TypeScript SDK\ ---------------------------------](https://docs.temporal.io/develop/typescript/workers/run-worker-process) Shows how to run Worker processes with the TypeScript SDK --- # 6 docs tagged with "Visibility" | Temporal Platform Documentation [Skip to main content](https://docs.temporal.io/tags/visibility#__docusaurus_skipToContent_fallback) [Dual Visibility\ ---------------](https://docs.temporal.io/dual-visibility) This guide on Temporal Dual Visibility explains how to set up, configure, and use Dual Visibility in Temporal Server versions. Learn about configuring primary and secondary Visibility stores, migrating databases, and ensuring a smooth transition for Visibility data. [List Filter\ -----------](https://docs.temporal.io/list-filter) This guide on Temporal List Filters explains how to set up, configure, and use the List Filter API in Temporal Server versions. Filter and retrieve Workflow Executions, apply supported operators, and optimize queries for efficiency. [Observability - Temporal feature\ --------------------------------](https://docs.temporal.io/evaluate/development-production-features/observability) Explore the observability and visibility features of Temporal, including Metrics, Tracing, Logging, and Visibility. [Search Attributes\ -----------------](https://docs.temporal.io/search-attribute) This guide on Temporal Search Attributes explains how to set up, configure, and use default and custom Search Attributes in Temporal Server versions. Learn about supported types, limits, and how to use them to enhance Workflow filtering and querying. [Self-hosted Visibility feature setup\ ------------------------------------](https://docs.temporal.io/self-hosted-guide/visibility) A Visibility store is essential for your Temporal Service, supporting features like batch operations and Search Attribute-based filtering for Workflow Executions. For current self-hosted deployments, use advanced Visibility on PostgreSQL 12+, MySQL 8.0.17+, SQLite 3.31.0+, Elasticsearch, or OpenSearch. [Temporal Visibility\ -------------------](https://docs.temporal.io/visibility) Temporal Visibility enables operators to view, filter, and search Workflow Executions using List Filters and Search Attributes. Learn about supported databases, Dual Visibility, and custom Search Attributes. --- # 7 docs tagged with "Users" | Temporal Platform Documentation [Skip to main content](https://docs.temporal.io/tags/users#__docusaurus_skipToContent_fallback) [Account access\ --------------](https://docs.temporal.io/cloud/manage-access/) Manage access to your Temporal Cloud account [Manage user groups\ ------------------](https://docs.temporal.io/cloud/user-groups) Learn how to manage user groups, members, and roles [Manage users\ ------------](https://docs.temporal.io/cloud/users) Manage user invitations, account-level roles, and Namespace-level permissions in Temporal Cloud. Invite users, update roles, and delete users seamlessly using the Temporal Web UI, tcld, or the Cloud Ops API. [SAML authentication\ -------------------](https://docs.temporal.io/cloud/saml) Integrate SAML 2.0 with your Temporal Cloud account for secure user authentication. Connect via Microsoft Entra ID or Okta and ensure seamless SSO. Charges apply. [SCIM user management\ --------------------](https://docs.temporal.io/cloud/scim) Link your IdP with your Temporal Cloud account to securely automate user and group management. [tcld user command reference\ ---------------------------](https://docs.temporal.io/cloud/tcld/user/) Manage users easily in Temporal Cloud with tcld commands; delete, get info, invite, list, resend invites, set account roles, and set namespace permissions seamlessly. [User management\ ---------------](https://docs.temporal.io/cloud/users-invite) Learn how to manage user invitations for Temporal Cloud --- # 9 docs tagged with "Updates" | Temporal Platform Documentation [Skip to main content](https://docs.temporal.io/tags/updates#__docusaurus_skipToContent_fallback) [Handling Signals, Queries, & Updates\ ------------------------------------](https://docs.temporal.io/handling-messages) Signals, Queries, and Updates facilitate interactions with Workflow Executions. [Sending Signals, Queries, & Updates\ -----------------------------------](https://docs.temporal.io/sending-messages) Signals, Queries, and Updates facilitate interactions with Workflow Executions. [Temporal Workflow message passing - Signals, Queries, & Updates\ ---------------------------------------------------------------](https://docs.temporal.io/encyclopedia/workflow-message-passing/) Signals, Queries, and Updates facilitate interactions with Workflow Executions. [Workflow message passing - Go SDK\ ---------------------------------](https://docs.temporal.io/develop/go/workflows/message-passing) Develop with Queries, Signals, and Updates with the Temporal Go SDK. [Workflow message passing - Java SDK\ -----------------------------------](https://docs.temporal.io/develop/java/workflows/message-passing) Develop with Queries, Signals, and Updates with the Temporal Java SDK. [Workflow message passing - PHP SDK\ ----------------------------------](https://docs.temporal.io/develop/php/workflows/message-passing) Develop with Signals, Queries, and Updates in Temporal Workflows. Define, handle, and send Signals or Queries, and validate updates from a Temporal Client. [Workflow message passing - Python SDK\ -------------------------------------](https://docs.temporal.io/develop/python/workflows/message-passing) Develop with Queries, Signals, and Updates with the Temporal Python SDK. [Workflow message passing - Temporal feature\ -------------------------------------------](https://docs.temporal.io/evaluate/development-production-features/workflow-message-passing) Enhance your Workflows with Signals and Queries, allowing dynamic responses to external events and real-time state access for comprehensive monitoring and tracking. [Workflow message passing - TypeScript SDK\ -----------------------------------------](https://docs.temporal.io/develop/typescript/workflows/message-passing) Develop with Queries, Signals, and Updates with the Temporal Typescript SDK. --- # 19 docs tagged with "Data Converters" | Temporal Platform Documentation [Skip to main content](https://docs.temporal.io/tags/data-converters#__docusaurus_skipToContent_fallback) [Codec Server\ ------------](https://docs.temporal.io/codec-server) A Codec Server is an HTTP server that provides remote encoding and decoding for Temporal Payloads. [Converters and encryption - .NET SDK\ ------------------------------------](https://docs.temporal.io/develop/dotnet/best-practices/converters-and-encryption) Use a custom Payload Codec and Converter in the .NET SDK to modify Temporal Data Conversion behavior, including examples for encryption and camel case conversion. [Converters and encryption - Java SDK\ ------------------------------------](https://docs.temporal.io/develop/java/best-practices/converters-and-encryption) Create and implement a Custom Payload Codec and Payload Converter in Java using the Temporal SDK for custom data encryption, compression, and type conversion. [Converters and encryption - Ruby SDK\ ------------------------------------](https://docs.temporal.io/develop/ruby/best-practices/converters-and-encryption) Use a custom Payload Codec and Converter in the Ruby SDK to modify Temporal Data Conversion behavior, including examples for encryption and formatting. [Converters and encryption - TypeScript SDK\ ------------------------------------------](https://docs.temporal.io/develop/typescript/converters-and-encryption) Create a custom Payload Converter in TypeScript with Temporal SDKs to handle non-JSON-serializable values, configure your Data Converter, and use protobufs and encryption seamlessly in your Workflows and Activities. [Data handling - Go SDK\ ----------------------](https://docs.temporal.io/develop/go/data-handling) Learn how Temporal handles data through the Data Converter, including payload conversion, encryption, and large payload storage. [Data handling - Python SDK\ --------------------------](https://docs.temporal.io/develop/python/data-handling) Learn how Temporal handles data through the Data Converter, including payload conversion, encryption, and large payload storage. [Default and Custom Data Converters\ ----------------------------------](https://docs.temporal.io/default-custom-data-converters) Learn about the default Data Converter in Temporal SDKs and how to implement a custom Data Converter for custom serialization and encoding needs. [External Storage\ ----------------](https://docs.temporal.io/external-storage) External Storage offloads large payloads to an external store like S3, keeping only a small reference in the event history. [External Storage - Go SDK\ -------------------------](https://docs.temporal.io/develop/go/data-handling/external-storage) Offload large payloads to external storage using the claim check pattern in the Go SDK. [External Storage - Python SDK\ -----------------------------](https://docs.temporal.io/develop/python/data-handling/external-storage) Offload large payloads to external storage using the claim check pattern in the Python SDK. [Failure Converter\ -----------------](https://docs.temporal.io/failure-converter) A Failure Converter transforms error messages and call stacks into encoded formats to enhance security and observability. [How does Temporal handle application data?\ ------------------------------------------](https://docs.temporal.io/dataconversion) This guide explores Data Converters in the Temporal Platform, detailing how they handle serialization and encoding for Workflow inputs and outputs, ensuring data stays secure and manageable. [Key management\ --------------](https://docs.temporal.io/key-management) Learn about key management practices for securing encryption keys in Temporal applications. [Payload Codec\ -------------](https://docs.temporal.io/payload-codec) A Payload Codec performs bytes-to-bytes transformations on Temporal Payloads, often for compression and encryption. [Payload conversion - Go SDK\ ---------------------------](https://docs.temporal.io/develop/go/data-handling/data-conversion) Customize how Temporal serializes application objects using Payload Converters in the Go SDK, including composite data converters and custom type examples. [Payload conversion - Python SDK\ -------------------------------](https://docs.temporal.io/develop/python/data-handling/data-conversion) Customize how Temporal serializes application objects using Payload Converters in the Python SDK, including Pydantic and custom type examples. [Payload Converter\ -----------------](https://docs.temporal.io/payload-converter) A Payload Converter serializes and deserializes values to and from bytes for use in the Temporal SDK. [Remote data encoding\ --------------------](https://docs.temporal.io/remote-data-encoding) Use remote encoding to transform data for the Temporal CLI and Web UI. --- # 15 docs tagged with "tcld" | Temporal Platform Documentation [Skip to main content](https://docs.temporal.io/tags/tcld#__docusaurus_skipToContent_fallback) [Cloud automation - Temporal feature\ -----------------------------------](https://docs.temporal.io/evaluate/development-production-features/cloud-automation) Explore how cloud automation simplifies cloud management and enhances security through APIs, Terraform, and CLI. [tcld account command reference\ ------------------------------](https://docs.temporal.io/cloud/tcld/account) Manage Temporal Cloud accounts using tcld commands. Get account details, configure and manage metrics endpoints, and handle end-entity certificates efficiently with various commands. [tcld apikey command reference\ -----------------------------](https://docs.temporal.io/cloud/tcld/apikey) Manage your API Keys in Temporal Cloud with tcld commands. Create, retrieve, list, delete, disable, and enable API Keys effortlessly using tcld apikey commands. [tcld command reference\ ----------------------](https://docs.temporal.io/cloud/tcld) The Temporal Cloud CLI (tcld) is a command-line tool for interacting with Temporal Cloud, offering commands for account management, login, namespace, and more. Install via Homebrew or build from source. [tcld connectivity-rule command reference\ ----------------------------------------](https://docs.temporal.io/cloud/tcld/connectivity-rule) Connectivity rule operations [tcld feature command reference\ ------------------------------](https://docs.temporal.io/cloud/tcld/feature) Manage features in Temporal Cloud with the tcld feature commands. Use tcld feature get for details and tcld feature toggle to enable or disable specific features. [tcld generate-certificates command reference\ --------------------------------------------](https://docs.temporal.io/cloud/tcld/generate-certificates/) Generate certificate authority and end-entity TLS certificates for Temporal Cloud with tcld generate-certificates commands. Use modifiers for customization. [tcld login command reference\ ----------------------------](https://docs.temporal.io/cloud/tcld/login) Log in to your Temporal Cloud account with the tcld login command. Simply follow browser instructions. [tcld logout command reference\ -----------------------------](https://docs.temporal.io/cloud/tcld/logout/) The tcld logout command logs a user out of Temporal Cloud. Use the --disable-pop-up modifier to disable the browser pop-up. [tcld namespace command reference\ --------------------------------](https://docs.temporal.io/cloud/tcld/namespace/) Unlock the full potential of Temporal Cloud with tcld namespace commands. Efficiently manage Namespace operations, including add-region, create, delete, failover, get, list, and export. [tcld nexus command reference\ ----------------------------](https://docs.temporal.io/cloud/tcld/nexus) Manage Nexus resources in Temporal Cloud [tcld request command reference\ ------------------------------](https://docs.temporal.io/cloud/tcld/request/) Manage asynchronous requests in Temporal Cloud using tcld request commands. Use "get" to check request status with modifiers for Namespace or request ID for tailored control. [tcld user command reference\ ---------------------------](https://docs.temporal.io/cloud/tcld/user/) Manage users easily in Temporal Cloud with tcld commands; delete, get info, invite, list, resend invites, set account roles, and set namespace permissions seamlessly. [tcld user group command reference\ ---------------------------------](https://docs.temporal.io/cloud/tcld/user-group/) The tcld user-group commands manage user groups in Temporal Cloud. [tcld version command reference\ ------------------------------](https://docs.temporal.io/cloud/tcld/version/) The \`tcld version\` command retrieves version information about tcld. --- # 14 docs tagged with "Temporal Client" | Temporal Platform Documentation [Skip to main content](https://docs.temporal.io/tags/temporal-client#__docusaurus_skipToContent_fallback) [Environment configuration\ -------------------------](https://docs.temporal.io/develop/environment-configuration) Configure Temporal Clients using environment variables and TOML configuration files [Environment configuration\ -------------------------](https://docs.temporal.io/references/client-environment-configuration) Reference for configuring Temporal Clients using environment variables and TOML configuration files. [Run a development server\ ------------------------](https://docs.temporal.io/develop/run-a-development-server) Shows how to run a development Temporal Service [Standalone Activities - .NET SDK\ --------------------------------](https://docs.temporal.io/develop/dotnet/activities/standalone-activities) Execute Activities independently without a Workflow using the Temporal .NET SDK. [Standalone Activities - Go SDK\ ------------------------------](https://docs.temporal.io/develop/go/activities/standalone-activities) Execute Activities independently without a Workflow using the Temporal Go SDK. [Standalone Activities - Python SDK\ ----------------------------------](https://docs.temporal.io/develop/python/activities/standalone-activities) Execute Activities independently without a Workflow using the Temporal Python SDK. [Temporal Client - .NET SDK\ --------------------------](https://docs.temporal.io/develop/dotnet/client/temporal-client) Create a Temporal Client, connect to Temporal Cloud, start a Workflow, and get Workflow results using the Temporal .NET SDK with detailed steps and code examples. [Temporal Client - Go SDK\ ------------------------](https://docs.temporal.io/develop/go/client/temporal-client) Connect to Temporal Service or Cloud, start Workflow Executions, manage Workflow options, and retrieve Workflow results using the Go SDK. Follow detailed steps and code examples to effectively use Temporal’s capabilities. [Temporal Client - Java SDK\ --------------------------](https://docs.temporal.io/develop/java/client/temporal-client) This guide introduces Temporal Clients, explaining their role and configuration in Java to connect to various Temporal Services, including starting Workflow Executions and customizing Workflow options. [Temporal Client - PHP SDK\ -------------------------](https://docs.temporal.io/develop/php/client/temporal-client) Connect a Temporal Client to a Temporal Service and start Workflow Executions. This guide covers communication, including sending signals and queries. [Temporal Client - Python SDK\ ----------------------------](https://docs.temporal.io/develop/python/client/temporal-client) Discover how to connect and use Temporal Clients with Python. Link your Client to Temporal Service, Temporal Cloud, start Workflow Executions, set Task Queues, Workflow Ids, and get Workflow results. [Temporal Client - Ruby SDK\ --------------------------](https://docs.temporal.io/develop/ruby/client/temporal-client) Create a Temporal Client, connect to Temporal Cloud, start a Workflow, and get Workflow results using the Temporal Ruby SDK. [Temporal Client - Typescript SDK\ --------------------------------](https://docs.temporal.io/develop/typescript/client/temporal-client) The Temporal Client SDK enables seamless communication with the Temporal Service, allowing applications to start Workflow Executions, send Signals, and query Workflows efficiently. [Temporal Nexus - Go SDK feature guide\ -------------------------------------](https://docs.temporal.io/develop/go/nexus/feature-guide) Use Temporal Nexus within the Go SDK to connect durable executions within and across Namespaces using a Nexus Endpoint, a Nexus Service contract, and Nexus Operations. --- # Temporal CLI command options reference | Temporal Platform Documentation [Skip to main content](https://docs.temporal.io/cli/cmd-options#__docusaurus_skipToContent_fallback) On this page active-cluster[​](https://docs.temporal.io/cli/cmd-options#active-cluster "Direct link to active-cluster") ----------------------------------------------------------------------------------------------------------- Active cluster name. activity-id[​](https://docs.temporal.io/cli/cmd-options#activity-id "Direct link to activity-id") -------------------------------------------------------------------------------------------------- Identifies the Activity Execution. activity-jitter[​](https://docs.temporal.io/cli/cmd-options#activity-jitter "Direct link to activity-jitter") -------------------------------------------------------------------------------------------------------------- If set, the Activity will start at a random time within the specified jitter duration. activity-type[​](https://docs.temporal.io/cli/cmd-options#activity-type "Direct link to activity-type") -------------------------------------------------------------------------------------------------------- Command is applied to all running activities of this type. address[​](https://docs.temporal.io/cli/cmd-options#address "Direct link to address") -------------------------------------------------------------------------------------- The host and port (formatted as host:port) for the Temporal Frontend Service. api-key[​](https://docs.temporal.io/cli/cmd-options#api-key "Direct link to api-key") -------------------------------------------------------------------------------------- API key for request. archived[​](https://docs.temporal.io/cli/cmd-options#archived "Direct link to archived") ----------------------------------------------------------------------------------------- List archived Workflow Executions. note Caution: `--archived` is experimental. build-id[​](https://docs.temporal.io/cli/cmd-options#build-id "Direct link to build-id") ----------------------------------------------------------------------------------------- Identifies the build to retrieve reachability information for. Can be specified multiple times. calendar[​](https://docs.temporal.io/cli/cmd-options#calendar "Direct link to calendar") ----------------------------------------------------------------------------------------- Calendar specification in JSON `({"dayOfWeek":"Fri","hour":"17","minute":"5"})` or as a Cron string `("30 2 \* \* 5" or "@daily")`. catchup-window[​](https://docs.temporal.io/cli/cmd-options#catchup-window "Direct link to catchup-window") ----------------------------------------------------------------------------------------------------------- Maximum allowed catch-up time if server is down. cluster[​](https://docs.temporal.io/cli/cmd-options#cluster "Direct link to cluster") -------------------------------------------------------------------------------------- Cluster name. codec-auth[​](https://docs.temporal.io/cli/cmd-options#codec-auth "Direct link to codec-auth") ----------------------------------------------------------------------------------------------- Sets the authorization header on requests to the Codec Server. codec-endpoint[​](https://docs.temporal.io/cli/cmd-options#codec-endpoint "Direct link to codec-endpoint") ----------------------------------------------------------------------------------------------------------- Endpoint for a remote Codec Server. color[​](https://docs.temporal.io/cli/cmd-options#color "Direct link to color") -------------------------------------------------------------------------------- When to use color: auto, always, never. (default: auto) command-timeout[​](https://docs.temporal.io/cli/cmd-options#command-timeout "Direct link to command-timeout") -------------------------------------------------------------------------------------------------------------- The command execution timeout. 0s means no timeout. concurrency[​](https://docs.temporal.io/cli/cmd-options#concurrency "Direct link to concurrency") -------------------------------------------------------------------------------------------------- Request concurrency. cron[​](https://docs.temporal.io/cli/cmd-options#cron "Direct link to cron") ----------------------------------------------------------------------------- The Cron schedule can be formatted like the following: ┌───────────── minute (0 - 59)│ ┌───────────── hour (0 - 23)│ │ ┌───────────── day of the month (1 - 31)│ │ │ ┌───────────── month (1 - 12)│ │ │ │ ┌───────────── day of the week (0 - 6) (Sunday to Saturday)│ │ │ │ │* * * * * data[​](https://docs.temporal.io/cli/cmd-options#data "Direct link to data") ----------------------------------------------------------------------------- Namespace data in a key=value format. Values must be in JSON format. db-filename[​](https://docs.temporal.io/cli/cmd-options#db-filename "Direct link to db-filename") -------------------------------------------------------------------------------------------------- File in which to persist Temporal state. By default, Workflows are lost when the process dies. depth[​](https://docs.temporal.io/cli/cmd-options#depth "Direct link to depth") -------------------------------------------------------------------------------- The number of Child Workflows to fetch and expand. Use `-1` to fetch Child Workflows at any depth. description[​](https://docs.temporal.io/cli/cmd-options#description "Direct link to description") -------------------------------------------------------------------------------------------------- Namespace description or Nexus Endpoint description. You may use Markdown formatting in the Nexus Endpoint description. description-file[​](https://docs.temporal.io/cli/cmd-options#description-file "Direct link to description-file") ----------------------------------------------------------------------------------------------------------------- Path to the Nexus Endpoint description file. The contents of the description file may use Markdown formatting. detail[​](https://docs.temporal.io/cli/cmd-options#detail "Direct link to detail") ----------------------------------------------------------------------------------- A provided reason for failing an Activity. dry-run[​](https://docs.temporal.io/cli/cmd-options#dry-run "Direct link to dry-run") -------------------------------------------------------------------------------------- Simulate reset without resetting any Workflow Executions. dynamic-config-value[​](https://docs.temporal.io/cli/cmd-options#dynamic-config-value "Direct link to dynamic-config-value") ----------------------------------------------------------------------------------------------------------------------------- Dynamic config value, formatted as `KEY=JSON_VALUE`. String values require quotation marks. email[​](https://docs.temporal.io/cli/cmd-options#email "Direct link to email") -------------------------------------------------------------------------------- Owner email. enable-connection[​](https://docs.temporal.io/cli/cmd-options#enable-connection "Direct link to enable-connection") -------------------------------------------------------------------------------------------------------------------- Enable cross-cluster connection. end-time[​](https://docs.temporal.io/cli/cmd-options#end-time "Direct link to end-time") ----------------------------------------------------------------------------------------- Backfill end time. env[​](https://docs.temporal.io/cli/cmd-options#env "Direct link to env") -------------------------------------------------------------------------- Name of the environment to read environment variables from. env-file[​](https://docs.temporal.io/cli/cmd-options#env-file "Direct link to env-file") ----------------------------------------------------------------------------------------- Path to environment settings file. Defaults to `$HOME/.config/temporalio/temporal.yaml`. event-id[​](https://docs.temporal.io/cli/cmd-options#event-id "Direct link to event-id") ----------------------------------------------------------------------------------------- The Event Id for any Event after WorkflowTaskStarted you want to reset to (exclusive). It can be WorkflowTaskCompleted, WorkflowTaskFailed or others. exclude-file[​](https://docs.temporal.io/cli/cmd-options#exclude-file "Direct link to exclude-file") ----------------------------------------------------------------------------------------------------- Input file that specifies Workflow Executions to exclude from resetting. execution-timeout[​](https://docs.temporal.io/cli/cmd-options#execution-timeout "Direct link to execution-timeout") -------------------------------------------------------------------------------------------------------------------- Timeout (in seconds) for a [Workflow Execution](https://docs.temporal.io/workflow-execution) , including retries and `ContinueAsNew` tasks. existing-compatible-build-id[​](https://docs.temporal.io/cli/cmd-options#existing-compatible-build-id "Direct link to existing-compatible-build-id") ----------------------------------------------------------------------------------------------------------------------------------------------------- A Build Id that already exists in the version sets known by the Task Queue. New Build Ids are stored in the version set containing this Id, making them compatible with the versions in that set. fields[​](https://docs.temporal.io/cli/cmd-options#fields "Direct link to fields") ----------------------------------------------------------------------------------- Customize fields to print. Set to 'long' to automatically print more of main fields. first-execution-run-id[​](https://docs.temporal.io/cli/cmd-options#first-execution-run-id "Direct link to first-execution-run-id") ----------------------------------------------------------------------------------------------------------------------------------- Run update on the last execution in the chain that started with this Run Id. fold[​](https://docs.temporal.io/cli/cmd-options#fold "Direct link to fold") ----------------------------------------------------------------------------- Statuses for which Child Workflows will be folded in (this will reduce the number of information fetched and displayed). Case-insensitive and ignored if `--no-fold` is supplied. follow[​](https://docs.temporal.io/cli/cmd-options#follow "Direct link to follow") ----------------------------------------------------------------------------------- Follow the progress of a Workflow Execution. frontend-address[​](https://docs.temporal.io/cli/cmd-options#frontend-address "Direct link to frontend-address") ----------------------------------------------------------------------------------------------------------------- Frontend address of the remote Cluster. global[​](https://docs.temporal.io/cli/cmd-options#global "Direct link to global") ----------------------------------------------------------------------------------- Flag to indicate whether a Namespace is a Global Namespace. grpc-meta[​](https://docs.temporal.io/cli/cmd-options#grpc-meta "Direct link to grpc-meta") -------------------------------------------------------------------------------------------- Contains gRPC metadata to send with requests (format: `key=value`). Values must be in a valid JSON format. headless[​](https://docs.temporal.io/cli/cmd-options#headless "Direct link to headless") ----------------------------------------------------------------------------------------- Disable the Web UI. heartbeat-timeout[​](https://docs.temporal.io/cli/cmd-options#heartbeat-timeout "Direct link to heartbeat-timeout") -------------------------------------------------------------------------------------------------------------------- Maximum permitted time between successful Worker Heartbeats. history-archival-state[​](https://docs.temporal.io/cli/cmd-options#history-archival-state "Direct link to history-archival-state") ----------------------------------------------------------------------------------------------------------------------------------- Sets the history archival state. Valid values are "disabled" and "enabled". history-uri[​](https://docs.temporal.io/cli/cmd-options#history-uri "Direct link to history-uri") -------------------------------------------------------------------------------------------------- Optionally specify history archival URI (cannot be changed after first time archival is enabled). id-reuse-policy[​](https://docs.temporal.io/cli/cmd-options#id-reuse-policy "Direct link to id-reuse-policy") -------------------------------------------------------------------------------------------------------------- Allows the same Workflow Id to be used in a new Workflow Execution. Options: AllowDuplicate, AllowDuplicateFailedOnly, RejectDuplicate, TerminateIfRunning. identity[​](https://docs.temporal.io/cli/cmd-options#identity "Direct link to identity") ----------------------------------------------------------------------------------------- Specify operator's identity. input[​](https://docs.temporal.io/cli/cmd-options#input "Direct link to input") -------------------------------------------------------------------------------- Use the `--input` command option to include data in the command. This command option accepts a valid JSON string. If the entity that the command is acting on accepts multiple parameters, pass "null" for null values within the JSON string. The following is an example of starting a Workflow with the `--input` command option. This Workflow expects a single string as a parameter: temporal workflow start --input '"+1 555-555-5555"' input-file[​](https://docs.temporal.io/cli/cmd-options#input-file "Direct link to input-file") ----------------------------------------------------------------------------------------------- Passes optional input for the Workflow from a JSON file. If there are multiple JSON files, concatenate them and separate by space or newline. Input from the command line will overwrite file input. input-parallelism[​](https://docs.temporal.io/cli/cmd-options#input-parallelism "Direct link to input-parallelism") -------------------------------------------------------------------------------------------------------------------- Number of goroutines to run in parallel. Each goroutine processes one line for every second. input-separator[​](https://docs.temporal.io/cli/cmd-options#input-separator "Direct link to input-separator") -------------------------------------------------------------------------------------------------------------- Separator for the input file. The default is a tab (`\t`). interval[​](https://docs.temporal.io/cli/cmd-options#interval "Direct link to interval") ----------------------------------------------------------------------------------------- Interval duration, such as 90m, or 90m/13m to include phase offset. ip[​](https://docs.temporal.io/cli/cmd-options#ip "Direct link to ip") ----------------------------------------------------------------------- IPv4 address to bind the frontend service to. (default: 127.0.0.1) jitter[​](https://docs.temporal.io/cli/cmd-options#jitter "Direct link to jitter") ----------------------------------------------------------------------------------- Jitter duration. job-id[​](https://docs.temporal.io/cli/cmd-options#job-id "Direct link to job-id") ----------------------------------------------------------------------------------- Batch Job Id. keep-paused[​](https://docs.temporal.io/cli/cmd-options#keep-paused "Direct link to keep-paused") -------------------------------------------------------------------------------------------------- If this flag is provided and Activity was paused, it will stay paused after reset. limit[​](https://docs.temporal.io/cli/cmd-options#limit "Direct link to limit") -------------------------------------------------------------------------------- Number of items to print on a page. log-format[​](https://docs.temporal.io/cli/cmd-options#log-format "Direct link to log-format") ----------------------------------------------------------------------------------------------- Set the log formatting. Options: \["json", "pretty"\]. log-level[​](https://docs.temporal.io/cli/cmd-options#log-level "Direct link to log-level") -------------------------------------------------------------------------------------------- Set the log level. Options: \["debug" "info" "warn" "error" "fatal"\]. match-all[​](https://docs.temporal.io/cli/cmd-options#match-all "Direct link to match-all") -------------------------------------------------------------------------------------------- If set, all currently running activities will be unpaused. max-field-length[​](https://docs.temporal.io/cli/cmd-options#max-field-length "Direct link to max-field-length") ----------------------------------------------------------------------------------------------------------------- Maximum length for each attribute field. max-sets[​](https://docs.temporal.io/cli/cmd-options#max-sets "Direct link to max-sets") ----------------------------------------------------------------------------------------- Limits how many compatible sets will be returned. Specify 1 to return only the current default major version set. 0 returns all sets. memo[​](https://docs.temporal.io/cli/cmd-options#memo "Direct link to memo") ----------------------------------------------------------------------------- Set a memo on a schedule (format: key=value). Use valid JSON formats for value. memo-file[​](https://docs.temporal.io/cli/cmd-options#memo-file "Direct link to memo-file") -------------------------------------------------------------------------------------------- Set a memo from a file. Each line should follow the format key=value. Use valid JSON formats for value. metrics-port[​](https://docs.temporal.io/cli/cmd-options#metrics-port "Direct link to metrics-port") ----------------------------------------------------------------------------------------------------- Port for `/metrics`. Enabled by default with a randomly assigned port. name[​](https://docs.temporal.io/cli/cmd-options#name "Direct link to name") ----------------------------------------------------------------------------- Frontend address of the remote Cluster or the Endpoint name. namespace[​](https://docs.temporal.io/cli/cmd-options#namespace "Direct link to namespace") -------------------------------------------------------------------------------------------- Identifies a Namespace in the Temporal Workflow. namespace-id[​](https://docs.temporal.io/cli/cmd-options#namespace-id "Direct link to namespace-id") ----------------------------------------------------------------------------------------------------- Namespace Id. no-fold[​](https://docs.temporal.io/cli/cmd-options#no-fold "Direct link to no-fold") -------------------------------------------------------------------------------------- Disable folding. All Child Workflows within the set depth will be fetched and displayed. no-json-shorthand-payloads[​](https://docs.temporal.io/cli/cmd-options#no-json-shorthand-payloads "Direct link to no-json-shorthand-payloads") ----------------------------------------------------------------------------------------------------------------------------------------------- Raw payload output, even if the JSON option was used. no-pager[​](https://docs.temporal.io/cli/cmd-options#no-pager "Direct link to no-pager") ----------------------------------------------------------------------------------------- Disables the interactive pager. non-deterministic[​](https://docs.temporal.io/cli/cmd-options#non-deterministic "Direct link to non-deterministic") -------------------------------------------------------------------------------------------------------------------- Reset Workflow Execution only if its last Event is `WorkflowTaskFailed` with a nondeterminism error. notes[​](https://docs.temporal.io/cli/cmd-options#notes "Direct link to notes") -------------------------------------------------------------------------------- Initial value of notes field. output[​](https://docs.temporal.io/cli/cmd-options#output "Direct link to output") ----------------------------------------------------------------------------------- Format of output. Options: table, json, card. overlap-policy[​](https://docs.temporal.io/cli/cmd-options#overlap-policy "Direct link to overlap-policy") ----------------------------------------------------------------------------------------------------------- Overlap policy. Options: Skip, BufferOne, BufferAll, CancelOther, TerminateOther, AllowAll. pager[​](https://docs.temporal.io/cli/cmd-options#pager "Direct link to pager") -------------------------------------------------------------------------------- Sets the pager for the Temporal CLI to use. Options are less, more, and favoritePager. pause[​](https://docs.temporal.io/cli/cmd-options#pause "Direct link to pause") -------------------------------------------------------------------------------- Pauses the Schedule. pause-on-failure[​](https://docs.temporal.io/cli/cmd-options#pause-on-failure "Direct link to pause-on-failure") ----------------------------------------------------------------------------------------------------------------- Pause schedule after any Workflow failure. port[​](https://docs.temporal.io/cli/cmd-options#port "Direct link to port") ----------------------------------------------------------------------------- Port for the frontend gRPC service. promote-global[​](https://docs.temporal.io/cli/cmd-options#promote-global "Direct link to promote-global") ----------------------------------------------------------------------------------------------------------- Promote local Namespace to Global Namespace. query[​](https://docs.temporal.io/cli/cmd-options#query "Direct link to query") -------------------------------------------------------------------------------- Provides a SQL-like Query of Search Attributes to return Workflow Executions to reset. For more information, refer to the [`temporal workflow list`](https://docs.temporal.io/cli/workflow#list) command. raw[​](https://docs.temporal.io/cli/cmd-options#raw "Direct link to raw") -------------------------------------------------------------------------- Print raw data in a JSON format. For scripting, we recommend using this option instead of `-o json`. reachability-type[​](https://docs.temporal.io/cli/cmd-options#reachability-type "Direct link to reachability-type") -------------------------------------------------------------------------------------------------------------------- Specify how you'd like to filter the reachability of Build IDs. The following are valid choices: * `open`: reachable by one or more open Workflows. * `closed`: reachable by one or more closed Workflows. * `existing`: reachable by either open or closed Workflows. Build IDs that are reachable by new Workflows are always reported. reapply-type[​](https://docs.temporal.io/cli/cmd-options#reapply-type "Direct link to reapply-type") ----------------------------------------------------------------------------------------------------- Event types to reapply after the reset point. Options: Signal, None. reason[​](https://docs.temporal.io/cli/cmd-options#reason "Direct link to reason") ----------------------------------------------------------------------------------- Reason for the operation. reject-condition[​](https://docs.temporal.io/cli/cmd-options#reject-condition "Direct link to reject-condition") ----------------------------------------------------------------------------------------------------------------- Optional flag for rejecting Queries based on Workflow state. Valid values are "not\_open" and "not\_completed\_cleanly". remaining-actions[​](https://docs.temporal.io/cli/cmd-options#remaining-actions "Direct link to remaining-actions") -------------------------------------------------------------------------------------------------------------------- Total number of actions allowed. reset-attempts[​](https://docs.temporal.io/cli/cmd-options#reset-attempts "Direct link to reset-attempts") ----------------------------------------------------------------------------------------------------------- Providing this flag will reset the number of attempts. reset-heartbeat[​](https://docs.temporal.io/cli/cmd-options#reset-heartbeat "Direct link to reset-heartbeat") -------------------------------------------------------------------------------------------------------------- Providing this flag will reset the heartbeat details. reset-points[​](https://docs.temporal.io/cli/cmd-options#reset-points "Direct link to reset-points") ----------------------------------------------------------------------------------------------------- Only show Workflow Events that are eligible for reset. result[​](https://docs.temporal.io/cli/cmd-options#result "Direct link to result") ----------------------------------------------------------------------------------- Set the result value of Activity completion. retention[​](https://docs.temporal.io/cli/cmd-options#retention "Direct link to retention") -------------------------------------------------------------------------------------------- Workflow Execution retention. retry-backoff-coefficient[​](https://docs.temporal.io/cli/cmd-options#retry-backoff-coefficient "Direct link to retry-backoff-coefficient") -------------------------------------------------------------------------------------------------------------------------------------------- Coefficient used to calculate the next retry interval. The next retry interval is previous interval multiplied by the coefficient. Must be 1 or larger. retry-initial-interval[​](https://docs.temporal.io/cli/cmd-options#retry-initial-interval "Direct link to retry-initial-interval") ----------------------------------------------------------------------------------------------------------------------------------- Interval of the first retry. If retryBackoffCoefficient is 1.0 then it is used for all retries. retry-maximum-attempts[​](https://docs.temporal.io/cli/cmd-options#retry-maximum-attempts "Direct link to retry-maximum-attempts") ----------------------------------------------------------------------------------------------------------------------------------- Maximum number of attempts. When exceeded the retries stop even if not expired yet. 1 disables retries. 0 means unlimited (up to the timeouts). retry-maximum-interval[​](https://docs.temporal.io/cli/cmd-options#retry-maximum-interval "Direct link to retry-maximum-interval") ----------------------------------------------------------------------------------------------------------------------------------- Maximum interval between retries. Exponential backoff leads to interval increase. This value is the cap of the increase. Default is 100x of the initial interval. run-id[​](https://docs.temporal.io/cli/cmd-options#run-id "Direct link to run-id") ----------------------------------------------------------------------------------- Identifies the current Workflow Run. run-timeout[​](https://docs.temporal.io/cli/cmd-options#run-timeout "Direct link to run-timeout") -------------------------------------------------------------------------------------------------- Timeout (in seconds) of a single Workflow run. schedule-id[​](https://docs.temporal.io/cli/cmd-options#schedule-id "Direct link to schedule-id") -------------------------------------------------------------------------------------------------- Schedule Id. schedule-to-close-timeout[​](https://docs.temporal.io/cli/cmd-options#schedule-to-close-timeout "Direct link to schedule-to-close-timeout") -------------------------------------------------------------------------------------------------------------------------------------------- Indicates how long the caller is willing to wait for an Activity completion. Limits how long retries will be attempted. schedule-to-start-timeout[​](https://docs.temporal.io/cli/cmd-options#schedule-to-start-timeout "Direct link to schedule-to-start-timeout") -------------------------------------------------------------------------------------------------------------------------------------------- Limits time an Activity Task can stay in a task queue before a Worker picks it up. This timeout is always non retryable, as all a retry would achieve is to put it back into the same queue. Defaults to `schedule_to_close_timeout` or workflow execution timeout if not specified. search-attribute[​](https://docs.temporal.io/cli/cmd-options#search-attribute "Direct link to search-attribute") ----------------------------------------------------------------------------------------------------------------- Set Search Attribute on a Schedule (formatted as `key=value`). Use valid JSON formats for value. set-as-default[​](https://docs.temporal.io/cli/cmd-options#set-as-default "Direct link to set-as-default") ----------------------------------------------------------------------------------------------------------- When set, establishes the compatible set being targeted as the default for the Task Queue. If a different set is the current default, the targeted set replaces it. skip-base-is-not-current[​](https://docs.temporal.io/cli/cmd-options#skip-base-is-not-current "Direct link to skip-base-is-not-current") ----------------------------------------------------------------------------------------------------------------------------------------- Skip a Workflow Execution if the base Workflow Run is not the current Workflow Run. skip-current-open[​](https://docs.temporal.io/cli/cmd-options#skip-current-open "Direct link to skip-current-open") -------------------------------------------------------------------------------------------------------------------- Skip a Workflow Execution if the current Run is open for the same Workflow Id as the base Run. sqlite-pragma[​](https://docs.temporal.io/cli/cmd-options#sqlite-pragma "Direct link to sqlite-pragma") -------------------------------------------------------------------------------------------------------- Specify sqlite pragma statements in pragma=value format. Pragma options: \["journal\_mode" "synchronous"\]. start-delay[​](https://docs.temporal.io/cli/cmd-options#start-delay "Direct link to start-delay") -------------------------------------------------------------------------------------------------- Specify a delay before the workflow starts. start-time[​](https://docs.temporal.io/cli/cmd-options#start-time "Direct link to start-time") ----------------------------------------------------------------------------------------------- Backfill start time. start-to-close-timeout[​](https://docs.temporal.io/cli/cmd-options#start-to-close-timeout "Direct link to start-to-close-timeout") ----------------------------------------------------------------------------------------------------------------------------------- Maximum time an Activity is allowed to execute after being picked up by a Worker. This Timeout is always retryable. target-namespace[​](https://docs.temporal.io/cli/cmd-options#target-namespace "Direct link to target-namespace") ----------------------------------------------------------------------------------------------------------------- Namespace in which a handler Worker will poll for Nexus tasks. target-task-queue[​](https://docs.temporal.io/cli/cmd-options#target-task-queue "Direct link to target-task-queue") -------------------------------------------------------------------------------------------------------------------- Task Queue in which a handler Worker will poll for Nexus tasks. target-url[​](https://docs.temporal.io/cli/cmd-options#target-url "Direct link to target-url") ----------------------------------------------------------------------------------------------- An external Nexus Endpoint where Nexus requests are forwarded to. May be used as an alternative to `--target-namespace` and `--target-task-queue`. note Caution: `--target-url` is experimental. task-queue[​](https://docs.temporal.io/cli/cmd-options#task-queue "Direct link to task-queue") ----------------------------------------------------------------------------------------------- Task Queue. task-queue-type[​](https://docs.temporal.io/cli/cmd-options#task-queue-type "Direct link to task-queue-type") -------------------------------------------------------------------------------------------------------------- Task Queue type, which can be either Workflow or Activity. The default type is Workflow. task-timeout[​](https://docs.temporal.io/cli/cmd-options#task-timeout "Direct link to task-timeout") ----------------------------------------------------------------------------------------------------- Start-to-close timeout for a Workflow Task (in seconds). time-format[​](https://docs.temporal.io/cli/cmd-options#time-format "Direct link to time-format") -------------------------------------------------------------------------------------------------- Format time as: relative, iso, raw. time-zone[​](https://docs.temporal.io/cli/cmd-options#time-zone "Direct link to time-zone") -------------------------------------------------------------------------------------------- Time zone (IANA name). tls[​](https://docs.temporal.io/cli/cmd-options#tls "Direct link to tls") -------------------------------------------------------------------------- Enable TLS encryption without additional options such as mTLS or client certificates. tls-ca-data[​](https://docs.temporal.io/cli/cmd-options#tls-ca-data "Direct link to tls-ca-data") -------------------------------------------------------------------------------------------------- Data for server CA certificate. Can't be used with --tls-ca-path. tls-ca-path[​](https://docs.temporal.io/cli/cmd-options#tls-ca-path "Direct link to tls-ca-path") -------------------------------------------------------------------------------------------------- Path to server CA certificate. tls-cert-data[​](https://docs.temporal.io/cli/cmd-options#tls-cert-data "Direct link to tls-cert-data") -------------------------------------------------------------------------------------------------------- Data for x509 certificate. Can't be used with --tls-cert-path. tls-cert-path[​](https://docs.temporal.io/cli/cmd-options#tls-cert-path "Direct link to tls-cert-path") -------------------------------------------------------------------------------------------------------- Path to x509 certificate. tls-disable-host-verification[​](https://docs.temporal.io/cli/cmd-options#tls-disable-host-verification "Direct link to tls-disable-host-verification") -------------------------------------------------------------------------------------------------------------------------------------------------------- Disables TLS host name verification. tls-key-data[​](https://docs.temporal.io/cli/cmd-options#tls-key-data "Direct link to tls-key-data") ----------------------------------------------------------------------------------------------------- Private certificate key data. Can't be used with --tls-key-path. tls-key-path[​](https://docs.temporal.io/cli/cmd-options#tls-key-path "Direct link to tls-key-path") ----------------------------------------------------------------------------------------------------- Path to private certificate key. tls-server-name[​](https://docs.temporal.io/cli/cmd-options#tls-server-name "Direct link to tls-server-name") -------------------------------------------------------------------------------------------------------------- Overrides the target TLS server name. type[​](https://docs.temporal.io/cli/cmd-options#type "Direct link to type") ----------------------------------------------------------------------------- Search attribute type. Options: Text, Keyword, Int, Double, Bool, Datetime, KeywordList. ui-asset-path[​](https://docs.temporal.io/cli/cmd-options#ui-asset-path "Direct link to ui-asset-path") -------------------------------------------------------------------------------------------------------- UI Custom Assets path. ui-codec-endpoint[​](https://docs.temporal.io/cli/cmd-options#ui-codec-endpoint "Direct link to ui-codec-endpoint") -------------------------------------------------------------------------------------------------------------------- UI Remote data converter HTTP endpoint. ui-ip[​](https://docs.temporal.io/cli/cmd-options#ui-ip "Direct link to ui-ip") -------------------------------------------------------------------------------- IPv4 address to bind the Web UI to. ui-port[​](https://docs.temporal.io/cli/cmd-options#ui-port "Direct link to ui-port") -------------------------------------------------------------------------------------- Port for the Web UI. Default: `--port` + 1000 (for example, 4000). unpause[​](https://docs.temporal.io/cli/cmd-options#unpause "Direct link to unpause") -------------------------------------------------------------------------------------- Unpauses the Schedule. unset-description[​](https://docs.temporal.io/cli/cmd-options#unset-description "Direct link to unset-description") -------------------------------------------------------------------------------------------------------------------- Unset the description. verbose[​](https://docs.temporal.io/cli/cmd-options#verbose "Direct link to verbose") -------------------------------------------------------------------------------------- Print applied Namespace changes. visibility-archival-state[​](https://docs.temporal.io/cli/cmd-options#visibility-archival-state "Direct link to visibility-archival-state") -------------------------------------------------------------------------------------------------------------------------------------------- Visibility Archival state. Valid values: "disabled", "enabled". visibility-uri[​](https://docs.temporal.io/cli/cmd-options#visibility-uri "Direct link to visibility-uri") ----------------------------------------------------------------------------------------------------------- Specify URI for Visibility Archival. This cannot be changed after Archival is enabled. workflow-id[​](https://docs.temporal.io/cli/cmd-options#workflow-id "Direct link to workflow-id") -------------------------------------------------------------------------------------------------- Workflow Id. workflow-type[​](https://docs.temporal.io/cli/cmd-options#workflow-type "Direct link to workflow-type") -------------------------------------------------------------------------------------------------------- Workflow type name. yes[​](https://docs.temporal.io/cli/cmd-options#yes "Direct link to yes") -------------------------------------------------------------------------- Confirm all prompts. * [active-cluster](https://docs.temporal.io/cli/cmd-options#active-cluster) * [activity-id](https://docs.temporal.io/cli/cmd-options#activity-id) * [activity-jitter](https://docs.temporal.io/cli/cmd-options#activity-jitter) * [activity-type](https://docs.temporal.io/cli/cmd-options#activity-type) * [address](https://docs.temporal.io/cli/cmd-options#address) * [api-key](https://docs.temporal.io/cli/cmd-options#api-key) * [archived](https://docs.temporal.io/cli/cmd-options#archived) * [build-id](https://docs.temporal.io/cli/cmd-options#build-id) * [calendar](https://docs.temporal.io/cli/cmd-options#calendar) * [catchup-window](https://docs.temporal.io/cli/cmd-options#catchup-window) * [cluster](https://docs.temporal.io/cli/cmd-options#cluster) * [codec-auth](https://docs.temporal.io/cli/cmd-options#codec-auth) * [codec-endpoint](https://docs.temporal.io/cli/cmd-options#codec-endpoint) * [color](https://docs.temporal.io/cli/cmd-options#color) * [command-timeout](https://docs.temporal.io/cli/cmd-options#command-timeout) * [concurrency](https://docs.temporal.io/cli/cmd-options#concurrency) * [cron](https://docs.temporal.io/cli/cmd-options#cron) * [data](https://docs.temporal.io/cli/cmd-options#data) * [db-filename](https://docs.temporal.io/cli/cmd-options#db-filename) * [depth](https://docs.temporal.io/cli/cmd-options#depth) * [description](https://docs.temporal.io/cli/cmd-options#description) * [description-file](https://docs.temporal.io/cli/cmd-options#description-file) * [detail](https://docs.temporal.io/cli/cmd-options#detail) * [dry-run](https://docs.temporal.io/cli/cmd-options#dry-run) * [dynamic-config-value](https://docs.temporal.io/cli/cmd-options#dynamic-config-value) * [email](https://docs.temporal.io/cli/cmd-options#email) * [enable-connection](https://docs.temporal.io/cli/cmd-options#enable-connection) * [end-time](https://docs.temporal.io/cli/cmd-options#end-time) * [env](https://docs.temporal.io/cli/cmd-options#env) * [env-file](https://docs.temporal.io/cli/cmd-options#env-file) * [event-id](https://docs.temporal.io/cli/cmd-options#event-id) * [exclude-file](https://docs.temporal.io/cli/cmd-options#exclude-file) * [execution-timeout](https://docs.temporal.io/cli/cmd-options#execution-timeout) * [existing-compatible-build-id](https://docs.temporal.io/cli/cmd-options#existing-compatible-build-id) * [fields](https://docs.temporal.io/cli/cmd-options#fields) * [first-execution-run-id](https://docs.temporal.io/cli/cmd-options#first-execution-run-id) * [fold](https://docs.temporal.io/cli/cmd-options#fold) * [follow](https://docs.temporal.io/cli/cmd-options#follow) * [frontend-address](https://docs.temporal.io/cli/cmd-options#frontend-address) * [global](https://docs.temporal.io/cli/cmd-options#global) * [grpc-meta](https://docs.temporal.io/cli/cmd-options#grpc-meta) * [headless](https://docs.temporal.io/cli/cmd-options#headless) * [heartbeat-timeout](https://docs.temporal.io/cli/cmd-options#heartbeat-timeout) * [history-archival-state](https://docs.temporal.io/cli/cmd-options#history-archival-state) * [history-uri](https://docs.temporal.io/cli/cmd-options#history-uri) * [id-reuse-policy](https://docs.temporal.io/cli/cmd-options#id-reuse-policy) * [identity](https://docs.temporal.io/cli/cmd-options#identity) * [input](https://docs.temporal.io/cli/cmd-options#input) * [input-file](https://docs.temporal.io/cli/cmd-options#input-file) * [input-parallelism](https://docs.temporal.io/cli/cmd-options#input-parallelism) * [input-separator](https://docs.temporal.io/cli/cmd-options#input-separator) * [interval](https://docs.temporal.io/cli/cmd-options#interval) * [ip](https://docs.temporal.io/cli/cmd-options#ip) * [jitter](https://docs.temporal.io/cli/cmd-options#jitter) * [job-id](https://docs.temporal.io/cli/cmd-options#job-id) * [keep-paused](https://docs.temporal.io/cli/cmd-options#keep-paused) * [limit](https://docs.temporal.io/cli/cmd-options#limit) * [log-format](https://docs.temporal.io/cli/cmd-options#log-format) * [log-level](https://docs.temporal.io/cli/cmd-options#log-level) * [match-all](https://docs.temporal.io/cli/cmd-options#match-all) * [max-field-length](https://docs.temporal.io/cli/cmd-options#max-field-length) * [max-sets](https://docs.temporal.io/cli/cmd-options#max-sets) * [memo](https://docs.temporal.io/cli/cmd-options#memo) * [memo-file](https://docs.temporal.io/cli/cmd-options#memo-file) * [metrics-port](https://docs.temporal.io/cli/cmd-options#metrics-port) * [name](https://docs.temporal.io/cli/cmd-options#name) * [namespace](https://docs.temporal.io/cli/cmd-options#namespace) * [namespace-id](https://docs.temporal.io/cli/cmd-options#namespace-id) * [no-fold](https://docs.temporal.io/cli/cmd-options#no-fold) * [no-json-shorthand-payloads](https://docs.temporal.io/cli/cmd-options#no-json-shorthand-payloads) * [no-pager](https://docs.temporal.io/cli/cmd-options#no-pager) * [non-deterministic](https://docs.temporal.io/cli/cmd-options#non-deterministic) * [notes](https://docs.temporal.io/cli/cmd-options#notes) * [output](https://docs.temporal.io/cli/cmd-options#output) * [overlap-policy](https://docs.temporal.io/cli/cmd-options#overlap-policy) * [pager](https://docs.temporal.io/cli/cmd-options#pager) * [pause](https://docs.temporal.io/cli/cmd-options#pause) * [pause-on-failure](https://docs.temporal.io/cli/cmd-options#pause-on-failure) * [port](https://docs.temporal.io/cli/cmd-options#port) * [promote-global](https://docs.temporal.io/cli/cmd-options#promote-global) * [query](https://docs.temporal.io/cli/cmd-options#query) * [raw](https://docs.temporal.io/cli/cmd-options#raw) * [reachability-type](https://docs.temporal.io/cli/cmd-options#reachability-type) * [reapply-type](https://docs.temporal.io/cli/cmd-options#reapply-type) * [reason](https://docs.temporal.io/cli/cmd-options#reason) * [reject-condition](https://docs.temporal.io/cli/cmd-options#reject-condition) * [remaining-actions](https://docs.temporal.io/cli/cmd-options#remaining-actions) * [reset-attempts](https://docs.temporal.io/cli/cmd-options#reset-attempts) * [reset-heartbeat](https://docs.temporal.io/cli/cmd-options#reset-heartbeat) * [reset-points](https://docs.temporal.io/cli/cmd-options#reset-points) * [result](https://docs.temporal.io/cli/cmd-options#result) * [retention](https://docs.temporal.io/cli/cmd-options#retention) * [retry-backoff-coefficient](https://docs.temporal.io/cli/cmd-options#retry-backoff-coefficient) * [retry-initial-interval](https://docs.temporal.io/cli/cmd-options#retry-initial-interval) * [retry-maximum-attempts](https://docs.temporal.io/cli/cmd-options#retry-maximum-attempts) * [retry-maximum-interval](https://docs.temporal.io/cli/cmd-options#retry-maximum-interval) * [run-id](https://docs.temporal.io/cli/cmd-options#run-id) * [run-timeout](https://docs.temporal.io/cli/cmd-options#run-timeout) * [schedule-id](https://docs.temporal.io/cli/cmd-options#schedule-id) * [schedule-to-close-timeout](https://docs.temporal.io/cli/cmd-options#schedule-to-close-timeout) * [schedule-to-start-timeout](https://docs.temporal.io/cli/cmd-options#schedule-to-start-timeout) * [search-attribute](https://docs.temporal.io/cli/cmd-options#search-attribute) * [set-as-default](https://docs.temporal.io/cli/cmd-options#set-as-default) * [skip-base-is-not-current](https://docs.temporal.io/cli/cmd-options#skip-base-is-not-current) * [skip-current-open](https://docs.temporal.io/cli/cmd-options#skip-current-open) * [sqlite-pragma](https://docs.temporal.io/cli/cmd-options#sqlite-pragma) * [start-delay](https://docs.temporal.io/cli/cmd-options#start-delay) * [start-time](https://docs.temporal.io/cli/cmd-options#start-time) * [start-to-close-timeout](https://docs.temporal.io/cli/cmd-options#start-to-close-timeout) * [target-namespace](https://docs.temporal.io/cli/cmd-options#target-namespace) * [target-task-queue](https://docs.temporal.io/cli/cmd-options#target-task-queue) * [target-url](https://docs.temporal.io/cli/cmd-options#target-url) * [task-queue](https://docs.temporal.io/cli/cmd-options#task-queue) * [task-queue-type](https://docs.temporal.io/cli/cmd-options#task-queue-type) * [task-timeout](https://docs.temporal.io/cli/cmd-options#task-timeout) * [time-format](https://docs.temporal.io/cli/cmd-options#time-format) * [time-zone](https://docs.temporal.io/cli/cmd-options#time-zone) * [tls](https://docs.temporal.io/cli/cmd-options#tls) * [tls-ca-data](https://docs.temporal.io/cli/cmd-options#tls-ca-data) * [tls-ca-path](https://docs.temporal.io/cli/cmd-options#tls-ca-path) * [tls-cert-data](https://docs.temporal.io/cli/cmd-options#tls-cert-data) * [tls-cert-path](https://docs.temporal.io/cli/cmd-options#tls-cert-path) * [tls-disable-host-verification](https://docs.temporal.io/cli/cmd-options#tls-disable-host-verification) * [tls-key-data](https://docs.temporal.io/cli/cmd-options#tls-key-data) * [tls-key-path](https://docs.temporal.io/cli/cmd-options#tls-key-path) * [tls-server-name](https://docs.temporal.io/cli/cmd-options#tls-server-name) * [type](https://docs.temporal.io/cli/cmd-options#type) * [ui-asset-path](https://docs.temporal.io/cli/cmd-options#ui-asset-path) * [ui-codec-endpoint](https://docs.temporal.io/cli/cmd-options#ui-codec-endpoint) * [ui-ip](https://docs.temporal.io/cli/cmd-options#ui-ip) * [ui-port](https://docs.temporal.io/cli/cmd-options#ui-port) * [unpause](https://docs.temporal.io/cli/cmd-options#unpause) * [unset-description](https://docs.temporal.io/cli/cmd-options#unset-description) * [verbose](https://docs.temporal.io/cli/cmd-options#verbose) * [visibility-archival-state](https://docs.temporal.io/cli/cmd-options#visibility-archival-state) * [visibility-uri](https://docs.temporal.io/cli/cmd-options#visibility-uri) * [workflow-id](https://docs.temporal.io/cli/cmd-options#workflow-id) * [workflow-type](https://docs.temporal.io/cli/cmd-options#workflow-type) * [yes](https://docs.temporal.io/cli/cmd-options#yes) --- # 10 docs tagged with "Versioning" | Temporal Platform Documentation [Skip to main content](https://docs.temporal.io/tags/versioning#__docusaurus_skipToContent_fallback) [Patching\ --------](https://docs.temporal.io/patching) This Encyclopedia page provides an in-depth explanation of what happens during Workflow Patching [Versioning - .NET SDK\ ---------------------](https://docs.temporal.io/develop/dotnet/workflows/versioning) Use the .NET SDK Patching API to safely deploy new code versions, handle deprecated patches, and manage Workflow activities using Temporal for long-running tasks. [Versioning - Go SDK\ -------------------](https://docs.temporal.io/develop/go/workflows/versioning) Temporal's Go SDK ensures Workflow determinism through Patching APIs and Worker Versioning. Update Workflow code without causing non-deterministic issues, understand versioning best practices, and use dynamic configuration parameters for seamless updating of long-running Workflows. [Versioning - Java SDK\ ---------------------](https://docs.temporal.io/develop/java/workflows/versioning) The Temporal Platform ensures deterministic Workflow code, offering versioning features in the Java SDK with Workflow Patching APIs and Worker Build Ids for efficient updates. [Versioning - PHP SDK feature guide\ ----------------------------------](https://docs.temporal.io/develop/php/workflows/versioning) Ensure deterministic Temporal Workflow execution and deploy updates with the PHP SDK's patching and Worker Versioning APIs. [Versioning - Python SDK\ -----------------------](https://docs.temporal.io/develop/python/workflows/versioning) Ensure deterministic Temporal Workflow execution and safely deploy updates using the Python SDK's patching and Worker Versioning APIs, for scalable long-running Workflows. [Versioning - Ruby SDK\ ---------------------](https://docs.temporal.io/develop/ruby/workflows/versioning) Use the Ruby SDK Patching API to safely deploy new code versions, handle deprecated patches, and manage Workflow activities using Temporal for long-running tasks. [Versioning - TypeScript SDK\ ---------------------------](https://docs.temporal.io/develop/typescript/workflows/versioning) Temporal TypeScript SDK ensures deterministic Workflow code with versioning features like Workflow Patching APIs, Worker Build IDs, and Workflow migration strategies. [Worker Versioning\ -----------------](https://docs.temporal.io/worker-versioning) Learn the concepts behind Worker Versioning and the details behind how it works. [Worker versioning (legacy)\ --------------------------](https://docs.temporal.io/encyclopedia/worker-versioning-legacy) Remember how to use the now-deprecated pre-release version of Worker Versioning --- # 18 docs tagged with "Self-hosting" | Temporal Platform Documentation [Skip to main content](https://docs.temporal.io/tags/self-hosting#__docusaurus_skipToContent_fallback) [Automated migration\ -------------------](https://docs.temporal.io/cloud/migrate/automated) Automated migration is designed to provide a zero-downtime, secure means of migrating to Temporal Cloud. This guide outlines the current process for transitioning workflows from a self-hosted setup to one hosted within Temporal Cloud. [Codec Server - Temporal Platform feature guide\ ----------------------------------------------](https://docs.temporal.io/production-deployment/data-encryption) Encrypt data in Temporal Server to secure Workflow, Activity, and Worker information. Use custom Payload Codecs for encryption/decryption, set up Codec Servers for remote decoding, and ensure secure access. [Deploying a Temporal Service\ ----------------------------](https://docs.temporal.io/self-hosted-guide/deployment) Deploy a Temporal Service using Docker, Kubernetes, or from scratch. Requires a database such as Apache Cassandra, MySQL, or PostgreSQL. Customize setup for your infrastructure and tooling. [Embedding Temporal server as a Go library\ -----------------------------------------](https://docs.temporal.io/self-hosted-guide/embedded-server) Run Temporal server as an embedded Go library for testing and development. Learn how to use temporal.NewServer() to run Temporal server in-process. [Managing Namespaces\ -------------------](https://docs.temporal.io/self-hosted-guide/namespaces) How to create and manage Namespaces in open source Temporal, including registration, configuration, and security. [Manual migration\ ----------------](https://docs.temporal.io/cloud/migrate/manual) Migrating to Temporal Cloud from self-hosted Temporal Service varies by Workflow requirements. This guide covers changing Client code, Workflow migration strategies, and necessary code adjustments. [Monitor Temporal Platform metrics\ ---------------------------------](https://docs.temporal.io/self-hosted-guide/monitoring) Monitor and health check a self-hosted Temporal Platform using Prometheus, StatsD, and M3 to track Temporal Service, Client, and Worker metrics for performance and issue troubleshooting. [Self-hosted Archival setup\ --------------------------](https://docs.temporal.io/self-hosted-guide/archival) Configure Archival to store closed Workflow Event Histories and Visibility records in blob storage. [Self-hosted Multi-Cluster Replication\ -------------------------------------](https://docs.temporal.io/self-hosted-guide/multi-cluster-replication) Multi-Cluster Replication in Temporal ensures asynchronous replication of Workflow Executions from active to passive Clusters for backup and state reconstruction, enabling seamless failovers. [Self-hosted Temporal Nexus\ --------------------------](https://docs.temporal.io/production-deployment/self-hosted-guide/nexus) Use Nexus in your self-hosted Temporal Service. [Self-hosted Temporal Service defaults\ -------------------------------------](https://docs.temporal.io/self-hosted-guide/defaults) Explore the Temporal Platform defaults, limits, and configurations, including Workflow Execution, Activity, Worker, and Payload size constraints. Learn about error and warning thresholds. [Self-hosted Temporal Service guide\ ----------------------------------](https://docs.temporal.io/self-hosted-guide) Discover how to self-host the open-source Temporal Service to orchestrate durable applications, or consider using Temporal Cloud for ease of use. Start with our tutorials and dev guide. [Self-hosted Visibility feature setup\ ------------------------------------](https://docs.temporal.io/self-hosted-guide/visibility) A Visibility store is essential for your Temporal Service, supporting features like batch operations and Search Attribute-based filtering for Workflow Executions. For current self-hosted deployments, use advanced Visibility on PostgreSQL 12+, MySQL 8.0.17+, SQLite 3.31.0+, Elasticsearch, or OpenSearch. [Server frontend API reference\ -----------------------------](https://docs.temporal.io/self-hosted-guide/server-frontend-api-reference) Easily interact with the Temporal Server via Client SDKs or CLI, or use the gRPC API for Workflow operations. Access code examples and API docs at api-docs.temporal.io. [Temporal Platform production deployments\ ----------------------------------------](https://docs.temporal.io/production-deployment) Elevate your durable application to production with ease by deploying your code on your infrastructure, using Temporal Cloud for a fully-managed service or self-hosting it. [Temporal Platform security features\ -----------------------------------](https://docs.temporal.io/self-hosted-guide/security) Discover comprehensive security features of the Temporal Platform, including secure network communication with TLS and mTLS, robust authentication, customizable authorization, and single sign-on integration to protect your data and operations. [Temporal Platform's production readiness checklist\ --------------------------------------------------](https://docs.temporal.io/self-hosted-guide/production-checklist) Optimize your Temporal Service for production with scaling, metrics, load testing, and effective workflow versioning techniques. Ensure robust performance and future-proof your workflows. [Upgrade the Temporal Server\ ---------------------------](https://docs.temporal.io/self-hosted-guide/upgrade-server) Upgrade your Temporal Server effectively. Follow our step-by-step guide for seamless upgrades, ensuring data compatibility and schema alignment. --- # gcpregions | Temporal Platform Documentation [Skip to main content](https://docs.temporal.io/cloud/references/regions/gcpregions#__docusaurus_skipToContent_fallback) On this page ### North America - Iowa (`us-central1`)[​](https://docs.temporal.io/cloud/references/regions/gcpregions#north-america---iowa-us-central1 "Direct link to north-america---iowa-us-central1") * **Cloud API Code**: `gcp-us-central1` * **Regional Endpoint**: `gcp-us-central1.region.tmprl.cloud` * **Private Service Connect Service Attachment URI**: `projects/prod-d9ch6v2ybver8d2a8fyf7qru9/regions/us-central1/serviceAttachments/pl-5xzng` * **Same Region Replication**: Not Available * **Multi-Region Replication**: * `gcp-us-west1` * `gcp-us-east4` * **Multi-Cloud Replication**: * `aws-ca-central-1` * `aws-us-east-1` * `aws-us-east-2` * `aws-us-west-2` ### North America - Oregon (`us-west1`)[​](https://docs.temporal.io/cloud/references/regions/gcpregions#north-america---oregon-us-west1 "Direct link to north-america---oregon-us-west1") * **Cloud API Code**: `gcp-us-west1` * **Regional Endpoint**: `gcp-us-west1.region.tmprl.cloud` * **Private Service Connect Service Attachment URI**: `projects/prod-rbe76zxxzydz4cbdz2xt5b59q/regions/us-west1/serviceAttachments/pl-94w0x` * **Same Region Replication**: Not Available * **Multi-Region Replication**: * `gcp-us-central1` * `gcp-us-east4` * **Multi-Cloud Replication**: * `aws-ca-central-1` * `aws-us-east-1` * `aws-us-east-2` * `aws-us-west-2` ### North America - Northern Virginia (`us-east4`)[​](https://docs.temporal.io/cloud/references/regions/gcpregions#north-america---northern-virginia-us-east4 "Direct link to north-america---northern-virginia-us-east4") * **Cloud API Code**: `gcp-us-east4` * **Regional Endpoint**: `gcp-us-east4.region.tmprl.cloud` * **Private Service Connect Service Attachment URI**: `projects/prod-y399cvr9c2b43es2w3q3e4gvw/regions/us-east4/serviceAttachments/pl-8awsy` * **Same Region Replication**: Not Available * **Multi-Region Replication**: * `gcp-us-central1` * `gcp-us-west1` * **Multi-Cloud Replication**: * `aws-ca-central-1` * `aws-us-east-1` * `aws-us-east-2` * `aws-us-west-2` ### Europe - Frankfurt (`europe-west3`)[​](https://docs.temporal.io/cloud/references/regions/gcpregions#europe---frankfurt-europe-west3 "Direct link to europe---frankfurt-europe-west3") * **Cloud API Code**: `gcp-europe-west3` * **Regional Endpoint**: `gcp-europe-west3.region.tmprl.cloud` * **Private Service Connect Service Attachment URI**: `projects/prod-kwy7d4faxp6qgrgd9x94du36g/regions/europe-west3/serviceAttachments/pl-acgsh` * **Same Region Replication**: Not Available * **Multi-Region Replication**: * None * **Multi-Cloud Replication**: * `aws-eu-central-1` * `aws-eu-west-1` * `aws-eu-west-2` ### Asia Pacific - Mumbai (`asia-south1`)[​](https://docs.temporal.io/cloud/references/regions/gcpregions#asia-pacific---mumbai-asia-south1 "Direct link to asia-pacific---mumbai-asia-south1") * **Cloud API Code**: `gcp-asia-south1` * **Regional Endpoint**: `gcp-asia-south1.region.tmprl.cloud` * **Private Service Connect Service Attachment URI**: `projects/prod-d5spc2sfeshws33bg33vwdef7/regions/asia-south1/serviceAttachments/pl-7w7tw` * **Same Region Replication**: Not Available * **Multi-Region Replication**: * None * **Multi-Cloud Replication**: * `aws-ap-northeast-1` * `aws-ap-northeast-2` * `aws-ap-south-1` * `aws-ap-south-2` * `aws-ap-southeast-1` * `aws-ap-southeast-2` * [North America - Iowa (`us-central1`)](https://docs.temporal.io/cloud/references/regions/gcpregions#north-america---iowa-us-central1) * [North America - Oregon (`us-west1`)](https://docs.temporal.io/cloud/references/regions/gcpregions#north-america---oregon-us-west1) * [North America - Northern Virginia (`us-east4`)](https://docs.temporal.io/cloud/references/regions/gcpregions#north-america---northern-virginia-us-east4) * [Europe - Frankfurt (`europe-west3`)](https://docs.temporal.io/cloud/references/regions/gcpregions#europe---frankfurt-europe-west3) * [Asia Pacific - Mumbai (`asia-south1`)](https://docs.temporal.io/cloud/references/regions/gcpregions#asia-pacific---mumbai-asia-south1) --- # private-service | Temporal Platform Documentation [Skip to main content](https://docs.temporal.io/cloud/references/regions/private-service#__docusaurus_skipToContent_fallback) | Region | Private Service Connect Service Name | | --- | --- | | `asia-south1` | `projects/prod-d5spc2sfeshws33bg33vwdef7/regions/asia-south1/serviceAttachments/pl-7w7tw` | | `us-central1` | `projects/prod-d9ch6v2ybver8d2a8fyf7qru9/regions/us-central1/serviceAttachments/pl-5xzn` | | `us-west1` | `projects/prod-rbe76zxxzydz4cbdz2xt5b59q/regions/us-west1/serviceAttachments/pl-94w0x` | --- # awsregions | Temporal Platform Documentation [Skip to main content](https://docs.temporal.io/cloud/references/regions/awsregions#__docusaurus_skipToContent_fallback) On this page ### Asia Pacific - Tokyo (`ap-northeast-1`)[​](https://docs.temporal.io/cloud/references/regions/awsregions#asia-pacific---tokyo-ap-northeast-1 "Direct link to asia-pacific---tokyo-ap-northeast-1") * **Cloud API Code**: `aws-ap-northeast-1` * **Regional Endpoint**: `aws-ap-northeast-1.region.tmprl.cloud` * **PrivateLink Endpoint Service**: `com.amazonaws.vpce.ap-northeast-1.vpce-svc-08f34c33f9fb8a48a` * **Same Region Replication**: Not Available * **Multi-Region Replication**: * `aws-ap-northeast-2` * `aws-ap-south-1` * `aws-ap-south-2` * `aws-ap-southeast-1` * `aws-ap-southeast-2` * **Multi-Cloud Replication**: * `gcp-asia-south1` ### Asia Pacific - Seoul (`ap-northeast-2`)[​](https://docs.temporal.io/cloud/references/regions/awsregions#asia-pacific---seoul-ap-northeast-2 "Direct link to asia-pacific---seoul-ap-northeast-2") * **Cloud API Code**: `aws-ap-northeast-2` * **Regional Endpoint**: `aws-ap-northeast-2.region.tmprl.cloud` * **PrivateLink Endpoint Service**: `com.amazonaws.vpce.ap-northeast-2.vpce-svc-08c4d5445a5aad308` * **Same Region Replication**: Not Available * **Multi-Region Replication**: * `aws-ap-northeast-1` * `aws-ap-south-1` * `aws-ap-south-2` * `aws-ap-southeast-1` * `aws-ap-southeast-2` * **Multi-Cloud Replication**: * `gcp-asia-south1` ### Asia Pacific - Mumbai (`ap-south-1`)[​](https://docs.temporal.io/cloud/references/regions/awsregions#asia-pacific---mumbai-ap-south-1 "Direct link to asia-pacific---mumbai-ap-south-1") * **Cloud API Code**: `aws-ap-south-1` * **Regional Endpoint**: `aws-ap-south-1.region.tmprl.cloud` * **PrivateLink Endpoint Service**: `com.amazonaws.vpce.ap-south-1.vpce-svc-0ad4f8ed56db15662` * **Same Region Replication**: Not Available * **Multi-Region Replication**: * `aws-ap-northeast-1` * `aws-ap-northeast-2` * `aws-ap-south-2` * `aws-ap-southeast-1` * `aws-ap-southeast-2` * **Multi-Cloud Replication**: * `gcp-asia-south1` ### Asia Pacific - Hyderabad (`ap-south-2`)[​](https://docs.temporal.io/cloud/references/regions/awsregions#asia-pacific---hyderabad-ap-south-2 "Direct link to asia-pacific---hyderabad-ap-south-2") * **Cloud API Code**: `aws-ap-south-2` * **Regional Endpoint**: `aws-ap-south-2.region.tmprl.cloud` * **PrivateLink Endpoint Service**: `com.amazonaws.vpce.ap-south-2.vpce-svc-08bcf602b646c69c1` * **Same Region Replication**: Not Available * **Multi-Region Replication**: * `aws-ap-northeast-1` * `aws-ap-northeast-2` * `aws-ap-south-1` * `aws-ap-southeast-1` * `aws-ap-southeast-2` * **Multi-Cloud Replication**: * `gcp-asia-south1` ### Asia Pacific - Singapore (`ap-southeast-1`)[​](https://docs.temporal.io/cloud/references/regions/awsregions#asia-pacific---singapore-ap-southeast-1 "Direct link to asia-pacific---singapore-ap-southeast-1") * **Cloud API Code**: `aws-ap-southeast-1` * **Regional Endpoint**: `aws-ap-southeast-1.region.tmprl.cloud` * **PrivateLink Endpoint Service**: `com.amazonaws.vpce.ap-southeast-1.vpce-svc-05c24096fa89b0ccd` * **Same Region Replication**: Not Available * **Multi-Region Replication**: * `aws-ap-northeast-1` * `aws-ap-northeast-2` * `aws-ap-south-1` * `aws-ap-south-2` * `aws-ap-southeast-2` * **Multi-Cloud Replication**: * `gcp-asia-south1` ### Asia Pacific - Sydney (`ap-southeast-2`)[​](https://docs.temporal.io/cloud/references/regions/awsregions#asia-pacific---sydney-ap-southeast-2 "Direct link to asia-pacific---sydney-ap-southeast-2") * **Cloud API Code**: `aws-ap-southeast-2` * **Regional Endpoint**: `aws-ap-southeast-2.region.tmprl.cloud` * **PrivateLink Endpoint Service**: `com.amazonaws.vpce.ap-southeast-2.vpce-svc-0634f9628e3c15b08` * **Same Region Replication**: Not Available * **Multi-Region Replication**: * `aws-ap-northeast-1` * `aws-ap-northeast-2` * `aws-ap-south-1` * `aws-ap-south-2` * `aws-ap-southeast-1` * **Multi-Cloud Replication**: * `gcp-asia-south1` ### Europe - Frankfurt (`eu-central-1`)[​](https://docs.temporal.io/cloud/references/regions/awsregions#europe---frankfurt-eu-central-1 "Direct link to europe---frankfurt-eu-central-1") * **Cloud API Code**: `aws-eu-central-1` * **Regional Endpoint**: `aws-eu-central-1.region.tmprl.cloud` * **PrivateLink Endpoint Service**: `com.amazonaws.vpce.eu-central-1.vpce-svc-073a419b36663a0f3` * **Same Region Replication**: Not Available * **Multi-Region Replication**: * `aws-eu-west-1` * `aws-eu-west-2` * **Multi-Cloud Replication**: * `gcp-europe-west3` ### Europe - Ireland (`eu-west-1`)[​](https://docs.temporal.io/cloud/references/regions/awsregions#europe---ireland-eu-west-1 "Direct link to europe---ireland-eu-west-1") * **Cloud API Code**: `aws-eu-west-1` * **Regional Endpoint**: `aws-eu-west-1.region.tmprl.cloud` * **PrivateLink Endpoint Service**: `com.amazonaws.vpce.eu-west-1.vpce-svc-04388e89f3479b739` * **Same Region Replication**: Not Available * **Multi-Region Replication**: * `aws-eu-central-1` * `aws-eu-west-2` * **Multi-Cloud Replication**: * `gcp-europe-west3` ### Europe - London (`eu-west-2`)[​](https://docs.temporal.io/cloud/references/regions/awsregions#europe---london-eu-west-2 "Direct link to europe---london-eu-west-2") * **Cloud API Code**: `aws-eu-west-2` * **Regional Endpoint**: `aws-eu-west-2.region.tmprl.cloud` * **PrivateLink Endpoint Service**: `com.amazonaws.vpce.eu-west-2.vpce-svc-0ac7f9f07e7fb5695` * **Same Region Replication**: Not Available * **Multi-Region Replication**: * `aws-eu-central-1` * `aws-eu-west-1` * **Multi-Cloud Replication**: * `gcp-europe-west3` ### North America - Central Canada (`ca-central-1`)[​](https://docs.temporal.io/cloud/references/regions/awsregions#north-america---central-canada-ca-central-1 "Direct link to north-america---central-canada-ca-central-1") * **Cloud API Code**: `aws-ca-central-1` * **PrivateLink Endpoint Service**: `com.amazonaws.vpce.ca-central-1.vpce-svc-080a781925d0b1d9d` * **Regional Endpoint**: `aws-ca-central-1.region.tmprl.cloud` * **Same Region Replication**: Not Available * **Multi-Region Replication**: * `aws-us-east-1` * `aws-us-east-2` * `aws-us-west-2` * **Multi-Cloud Replication**: * `gcp-us-central1` * `gcp-us-west1` * `gcp-us-east4` ### North America - Northern Virginia (`us-east-1`)[​](https://docs.temporal.io/cloud/references/regions/awsregions#north-america---northern-virginia-us-east-1 "Direct link to north-america---northern-virginia-us-east-1") * **Cloud API Code**: `aws-us-east-1` * **Regional Endpoint**: `aws-us-east-1.region.tmprl.cloud` * **PrivateLink Endpoint Service**: `com.amazonaws.vpce.us-east-1.vpce-svc-0822256b6575ea37f` * **Same Region Replication**: Available * **Multi-Region Replication**: * `aws-ca-central-1` * `aws-us-east-2` * `aws-us-west-2` * **Multi-Cloud Replication**: * `gcp-us-central1` * `gcp-us-west1` * `gcp-us-east4` ### North America - Ohio (`us-east-2`)[​](https://docs.temporal.io/cloud/references/regions/awsregions#north-america---ohio-us-east-2 "Direct link to north-america---ohio-us-east-2") * **Cloud API Code**: `aws-us-east-2` * **Regional Endpoint**: `aws-us-east-2.region.tmprl.cloud` * **PrivateLink Endpoint Service**: `com.amazonaws.vpce.us-east-2.vpce-svc-01b8dccfc6660d9d4` * **Same Region Replication**: Not Available * **Multi-Region Replication**: * `aws-ca-central-1` * `aws-us-east-1` * `aws-us-west-2` * **Multi-Cloud Replication**: * `gcp-us-central1` * `gcp-us-west1` * `gcp-us-east4` ### North America - Oregon (`us-west-2`)[​](https://docs.temporal.io/cloud/references/regions/awsregions#north-america---oregon-us-west-2 "Direct link to north-america---oregon-us-west-2") * **Cloud API Code**: `aws-us-west-2` * **Regional Endpoint**: `aws-us-west-2.region.tmprl.cloud` * **PrivateLink Endpoint Service**: `com.amazonaws.vpce.us-west-2.vpce-svc-0f44b3d7302816b94` * **Same Region Replication**: Available * **Multi-Region Replication**: * `aws-ca-central-1` * `aws-us-east-1` * `aws-us-east-2` * **Multi-Cloud Replication**: * `gcp-us-central1` * `gcp-us-west1` * `gcp-us-east4` ### South America - São Paulo (`sa-east-1`)[​](https://docs.temporal.io/cloud/references/regions/awsregions#south-america---s%C3%A3o-paulo-sa-east-1 "Direct link to south-america---são-paulo-sa-east-1") * **Cloud API Code**: `aws-sa-east-1` * **Regional Endpoint**: `aws-sa-east-1.region.tmprl.cloud` * **PrivateLink Endpoint Service**: `com.amazonaws.vpce.sa-east-1.vpce-svc-0ca67a102f3ce525a` * **Same Region Replication**: Not Available * **Multi-Region Replication**: * None * **Multi-Cloud Replication**: * None * [Asia Pacific - Tokyo (`ap-northeast-1`)](https://docs.temporal.io/cloud/references/regions/awsregions#asia-pacific---tokyo-ap-northeast-1) * [Asia Pacific - Seoul (`ap-northeast-2`)](https://docs.temporal.io/cloud/references/regions/awsregions#asia-pacific---seoul-ap-northeast-2) * [Asia Pacific - Mumbai (`ap-south-1`)](https://docs.temporal.io/cloud/references/regions/awsregions#asia-pacific---mumbai-ap-south-1) * [Asia Pacific - Hyderabad (`ap-south-2`)](https://docs.temporal.io/cloud/references/regions/awsregions#asia-pacific---hyderabad-ap-south-2) * [Asia Pacific - Singapore (`ap-southeast-1`)](https://docs.temporal.io/cloud/references/regions/awsregions#asia-pacific---singapore-ap-southeast-1) * [Asia Pacific - Sydney (`ap-southeast-2`)](https://docs.temporal.io/cloud/references/regions/awsregions#asia-pacific---sydney-ap-southeast-2) * [Europe - Frankfurt (`eu-central-1`)](https://docs.temporal.io/cloud/references/regions/awsregions#europe---frankfurt-eu-central-1) * [Europe - Ireland (`eu-west-1`)](https://docs.temporal.io/cloud/references/regions/awsregions#europe---ireland-eu-west-1) * [Europe - London (`eu-west-2`)](https://docs.temporal.io/cloud/references/regions/awsregions#europe---london-eu-west-2) * [North America - Central Canada (`ca-central-1`)](https://docs.temporal.io/cloud/references/regions/awsregions#north-america---central-canada-ca-central-1) * [North America - Northern Virginia (`us-east-1`)](https://docs.temporal.io/cloud/references/regions/awsregions#north-america---northern-virginia-us-east-1) * [North America - Ohio (`us-east-2`)](https://docs.temporal.io/cloud/references/regions/awsregions#north-america---ohio-us-east-2) * [North America - Oregon (`us-west-2`)](https://docs.temporal.io/cloud/references/regions/awsregions#north-america---oregon-us-west-2) * [South America - São Paulo (`sa-east-1`)](https://docs.temporal.io/cloud/references/regions/awsregions#south-america---s%C3%A3o-paulo-sa-east-1) --- # 24 docs tagged with "Observability" | Temporal Platform Documentation [Skip to main content](https://docs.temporal.io/tags/observability#__docusaurus_skipToContent_fallback) [Context Propagation\ -------------------](https://docs.temporal.io/encyclopedia/context-propagation) Pass custom key-value data across Workflow, Activity, and Child Workflow boundaries using Temporal headers. [General observability setup with metrics\ ----------------------------------------](https://docs.temporal.io/cloud/metrics/general-setup) Learn how to configure a metrics endpoint in Temporal Cloud using the UI or tcld CLI, assign certificates, and integrate with observability tools like Grafana. [Metrics integrations\ --------------------](https://docs.temporal.io/cloud/metrics/openmetrics/metrics-integrations) Integrating with the Temporal Cloud OpenMetrics endpoint. [Monitor SDK metrics with Prometheus and Grafana\ -----------------------------------------------](https://docs.temporal.io/cloud/metrics/sdk-metrics-setup) Set up Temporal SDK metrics with Prometheus and Grafana for monitoring Workers and Client performance. [Monitor Temporal Cloud\ ----------------------](https://docs.temporal.io/cloud/service-health) Use Temporal Cloud metrics to monitor your production deployment Temporal Cloud. [Monitor Temporal Platform metrics\ ---------------------------------](https://docs.temporal.io/self-hosted-guide/monitoring) Monitor and health check a self-hosted Temporal Platform using Prometheus, StatsD, and M3 to track Temporal Service, Client, and Worker metrics for performance and issue troubleshooting. [Monitor worker health\ ---------------------](https://docs.temporal.io/cloud/worker-health) Detect and configure for Task backlogs, greedy Worker resources, misconfigured Workers, and Sticky cache settings. Optimize alert systems and get actionable insights on metrics like Schedule-To-Start latency, Sync Match Rate, and Poll Success Rate for improved application health. [Observability - .NET SDK\ ------------------------](https://docs.temporal.io/develop/dotnet/platform/observability) Explore Temporal SDK observability features for Metrics, Tracing, Logging, and Visibility. Track Workflow Executions, set up Prometheus endpoints, customize metrics, configure tracing, and more. [Observability - Go SDK\ ----------------------](https://docs.temporal.io/develop/go/platform/observability) Monitor your Temporal Application state using Metrics, Tracing, Logging, and Visibility features. Emit metrics, configure tracing, customize logging, and use Search Attributes with the Temporal Go SDK for enhanced Workflow Execution insights. [Observability - Java SDK\ ------------------------](https://docs.temporal.io/develop/java/platform/observability) Explore the observability features of Temporal, including Metrics, Tracing, Logging, and Visibility. Emit Metrics with the Java SDK, set up Tracing, and use Search Attributes. [Observability - PHP SDK\ -----------------------](https://docs.temporal.io/develop/php/platform/observability) Explore the Temporal Developer’s guide on observability to learn about Visibility APIs and Search Attributes, helping you manage Workflow Executions efficiently. [Observability - Python SDK\ --------------------------](https://docs.temporal.io/develop/python/platform/observability) Discover how to monitor your Temporal Application using metrics, tracing, logging, and visibility APIs. Emit metrics, set up tracing, log from Workflows, and use custom Search Attributes. [Observability - Ruby SDK\ ------------------------](https://docs.temporal.io/develop/ruby/platform/observability) Explore Temporal SDK observability features for Metrics, Tracing, Logging, and Visibility using the Ruby SDK. [Observability - Temporal feature\ --------------------------------](https://docs.temporal.io/evaluate/development-production-features/observability) Explore the observability and visibility features of Temporal, including Metrics, Tracing, Logging, and Visibility. [Observability - TypeScript SDK\ ------------------------------](https://docs.temporal.io/develop/typescript/platform/observability) Enhance the observability of your Temporal Application with metrics, tracing, logging, and visibility features. View Workflow state, set up OpenTelemetry, and customize logging for seamless monitoring and insights. [OpenMetrics API reference\ -------------------------](https://docs.temporal.io/cloud/metrics/openmetrics/api-reference) Detailed API documentation for the Temporal Cloud OpenMetrics endpoint. [OpenMetrics metrics reference\ -----------------------------](https://docs.temporal.io/cloud/metrics/openmetrics/metrics-reference) Detailed API documentation for the Temporal Cloud OpenMetrics endpoint. [OpenMetrics migration guide\ ---------------------------](https://docs.temporal.io/cloud/metrics/openmetrics/migration-guide) Migrate from the Prometheus query endpoint to the new OpenMetrics endpoint in Temporal Cloud. [Performance bottlenecks troubleshooting guide\ ---------------------------------------------](https://docs.temporal.io/troubleshooting/performance-bottlenecks) Diagnose and resolve performance bottlenecks using Temporal SDK metrics [Prometheus Grafana setup\ ------------------------](https://docs.temporal.io/cloud/metrics/prometheus-grafana) Set up Grafana with Temporal Cloud observability to monitor performance and troubleshoot errors using the Prometheus HTTP API endpoint. [PromQL Metrics\ --------------](https://docs.temporal.io/cloud/metrics/promql) Get detailed insights into your Temporal Cloud Namespace metrics using your own observability tool. Access data with a CA certificate and retain raw metrics for seven days. [Temporal Cloud Metrics\ ----------------------](https://docs.temporal.io/cloud/metrics/) Monitor Temporal Cloud workloads with Cloud metrics and SDK metrics. [Temporal Cloud metrics reference\ --------------------------------](https://docs.temporal.io/cloud/metrics/reference) Explore Temporal Cloud metrics to query with PromQL or scrape via OpenMetrics, supporting rate and latency calculations. [Temporal Cloud OpenMetrics\ --------------------------](https://docs.temporal.io/cloud/metrics/openmetrics/) Export metrics from Temporal Cloud using the OpenMetrics standard and third party integrations. --- # tcld connectivity-rule command reference | Temporal Platform Documentation [Skip to main content](https://docs.temporal.io/cloud/tcld/connectivity-rule#__docusaurus_skipToContent_fallback) On this page The `tcld connectivity-rule` commands manage [connectivity rules](https://docs.temporal.io/cloud/connectivity#connectivity-rules) in Temporal Cloud. Alias: `cr` * [tcld connectivity-rule create](https://docs.temporal.io/cloud/tcld/connectivity-rule#create) * [tcld connectivity-rule delete](https://docs.temporal.io/cloud/tcld/connectivity-rule#delete) * [tcld connectivity-rule get](https://docs.temporal.io/cloud/tcld/connectivity-rule#get) * [tcld connectivity-rule list](https://docs.temporal.io/cloud/tcld/connectivity-rule#list) create[​](https://docs.temporal.io/cloud/tcld/connectivity-rule#create "Direct link to create") ------------------------------------------------------------------------------------------------ The `tcld connectivity-rule create` command creates a connectivity rule. Alias: `c` #### \--connection-id[​](https://docs.temporal.io/cloud/tcld/connectivity-rule#--connection-id "Direct link to --connection-id") The connection ID of the private connection. Alias: `ci` #### \--connectivity-type[​](https://docs.temporal.io/cloud/tcld/connectivity-rule#--connectivity-type "Direct link to --connectivity-type") The type of connectivity, currently only support 'private' and 'public'. Alias: `ct` #### \--gcp-project-id[​](https://docs.temporal.io/cloud/tcld/connectivity-rule#--gcp-project-id "Direct link to --gcp-project-id") The GCP project ID of the connection, required if the cloud provider is 'gcp'. Alias: `gpi` #### \--region[​](https://docs.temporal.io/cloud/tcld/connectivity-rule#--region "Direct link to --region") The region of the connection. Alias: `r` delete[​](https://docs.temporal.io/cloud/tcld/connectivity-rule#delete "Direct link to delete") ------------------------------------------------------------------------------------------------ The `tcld connectivity-rule delete` command deletes a connectivity rule. Alias: `d` #### \--connectivity-rule-id[​](https://docs.temporal.io/cloud/tcld/connectivity-rule#--connectivity-rule-id "Direct link to --connectivity-rule-id") The connectivity rule ID. Alias: `id` get[​](https://docs.temporal.io/cloud/tcld/connectivity-rule#get "Direct link to get") --------------------------------------------------------------------------------------- The `tcld connectivity-rule get` command gets a connectivity rule. Alias: `g` #### \--connectivity-rule-id[​](https://docs.temporal.io/cloud/tcld/connectivity-rule#--connectivity-rule-id-1 "Direct link to --connectivity-rule-id") The connectivity rule ID. Alias: `id` list[​](https://docs.temporal.io/cloud/tcld/connectivity-rule#list "Direct link to list") ------------------------------------------------------------------------------------------ The `tcld connectivity-rule list` command lists connectivity rules. Alias: `l` #### \--namespace[​](https://docs.temporal.io/cloud/tcld/connectivity-rule#--namespace "Direct link to --namespace") The namespace hosted on temporal cloud. Alias: `n` * [create](https://docs.temporal.io/cloud/tcld/connectivity-rule#create) * [\--connection-id](https://docs.temporal.io/cloud/tcld/connectivity-rule#--connection-id) * [\--connectivity-type](https://docs.temporal.io/cloud/tcld/connectivity-rule#--connectivity-type) * [\--gcp-project-id](https://docs.temporal.io/cloud/tcld/connectivity-rule#--gcp-project-id) * [\--region](https://docs.temporal.io/cloud/tcld/connectivity-rule#--region) * [delete](https://docs.temporal.io/cloud/tcld/connectivity-rule#delete) * [\--connectivity-rule-id](https://docs.temporal.io/cloud/tcld/connectivity-rule#--connectivity-rule-id) * [get](https://docs.temporal.io/cloud/tcld/connectivity-rule#get) * [\--connectivity-rule-id](https://docs.temporal.io/cloud/tcld/connectivity-rule#--connectivity-rule-id-1) * [list](https://docs.temporal.io/cloud/tcld/connectivity-rule#list) * [\--namespace](https://docs.temporal.io/cloud/tcld/connectivity-rule#--namespace) --- # User management | Temporal Platform Documentation [Skip to main content](https://docs.temporal.io/cloud/users-invite#__docusaurus_skipToContent_fallback) * Web UI * tcld * Cloud Ops API To invite users using the Temporal Cloud UI: 1. In Temporal Web UI, select **Settings** in the left portion of the window. 2. On the **Settings** page, select **Create Users** in the upper-right portion of the window. 3. On the **Create Users** page in the **Email Addresses** box, type or paste one or more email addresses. 4. In **Account-Level Role**, select a [Role](https://docs.temporal.io/cloud/manage-access/roles-and-permissions#account-level-roles) . The Role applies to all users whose email addresses appear in **Email Addresses**. 5. If the account has any Namespaces, they are listed under **Grant access to Namespaces**. To add a permission, select the checkbox next to a Namespace, and then select a [permission](https://docs.temporal.io/cloud/manage-access/roles-and-permissions#namespace-level-permissions) . Repeat as needed. 6. When all permissions are assigned, select **Send Invite**. Use the [`tcld user invite`](https://docs.temporal.io/cloud/tcld/user/#invite) command. Specify the user's email, an account-level role, and optionally one or more Namespace permissions. Available account roles: `admin` | `developer` | `read`. Available Namespace permissions: `Admin` | `Write` | `Read`. tcld user invite \ --user-email \ --account-role \ --namespace-permission = You can invite multiple users and assign multiple Namespace permissions in a single command: tcld user invite \ --user-email user1@example.com \ --user-email user2@example.com \ --account-role developer \ --namespace-permission ns1=Admin \ --namespace-permission ns2=Write Use the [CreateUser](https://saas-api.tmprl.cloud/docs/httpapi.html#tag/users) endpoint to invite a user. POST /cloud/users The request body includes a `spec` with the following fields: * `spec.email` — The email address of the user to invite. * `spec.access.account_access.role` — The account-level role to assign. * `spec.access.namespace_accesses` — A map of Namespace names to permissions. Available roles: `ROLE_ADMIN` | `ROLE_DEVELOPER` | `ROLE_READ` | `ROLE_OWNER` | `ROLE_FINANCE_ADMIN`. Available Namespace permissions: `PERMISSION_ADMIN` | `PERMISSION_WRITE` | `PERMISSION_READ`. The new users receive an email with a link to accept the invitation and complete their setup. The new user must use this link to sign up to be added to your account unless the account has a SAML configuration. If your account has a SAML configuration, the new user can sign in using their existing SAML credentials and be included in the account automatically. caution The new user must use the same authentication method they originally signed up with to sign in to Temporal Cloud. If they used single sign-on (SSO), they must use the same SSO provider to sign in to Temporal Cloud. If they used email and password authentication, they must use the same email and password to sign in to Temporal Cloud, and cannot use SSO, even if the underlying email address is the same. --- # Worker Basics - Python SDK | Temporal Platform Documentation [Skip to main content](https://docs.temporal.io/develop/python/workers/basics#__docusaurus_skipToContent_fallback) --- # Worker Versioning (Legacy) - Go SDK | Temporal Platform Documentation [Skip to main content](https://docs.temporal.io/develop/go/worker-versioning-legacy#__docusaurus_skipToContent_fallback) On this page How to use Worker Versioning in Go (Deprecated)[​](https://docs.temporal.io/develop/go/worker-versioning-legacy#worker-versioning "Direct link to How to use Worker Versioning in Go (Deprecated)") ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- caution This section is for a deprecated Worker Versioning API. Please redirect your attention to [Worker Versioning](https://docs.temporal.io/production-deployment/worker-deployments/worker-versioning) . See the [Pre-release README](https://github.com/temporalio/temporal/blob/main/docs/worker-versioning.md) for more information. A Build ID corresponds to a deployment. If you don't already have one, we recommend a hash of the code--such as a Git SHA--combined with a human-readable timestamp. To use Worker Versioning, you need to pass a Build ID to your Go Worker and opt in to Worker Versioning. ### Assign a Build ID to your Worker and opt in to Worker Versioning[​](https://docs.temporal.io/develop/go/worker-versioning-legacy#assign-a-build-id-to-your-worker-and-opt-in-to-worker-versioning "Direct link to Assign a Build ID to your Worker and opt in to Worker Versioning") You should understand assignment rules before completing this step. See the [Worker Versioning Pre-release README](https://github.com/temporalio/temporal/blob/main/docs/worker-versioning.md) for more information. To enable Worker Versioning for your Worker, assign the Build ID--perhaps from an environment variable--and turn it on. // ...workerOptions := worker.Options{ BuildID: buildID, UseBuildIDForVersioning: true,// ...}w := worker.New(c, "your_task_queue_name", workerOptions)// ... danger Importantly, when you start this Worker, it won't receive any tasks until you set up assignment rules. ### Specify versions for Activities, Child Workflows, and Continue-as-New Workflows[​](https://docs.temporal.io/develop/go/worker-versioning-legacy#specify-versions-for-activities-child-workflows-and-continue-as-new-workflows "Direct link to Specify versions for Activities, Child Workflows, and Continue-as-New Workflows") By default, Activities, Child Workflows, and Continue-as-New Workflows are run on the build of the Workflow that created them if they are also configured to run on the same Task Queue. When configured to run on a separate Task Queue, they will default to using the current assignment rules. If you want to override this behavior, you can specify your intent via the `VersioningIntent` field on the appropriate options struct. For example, if you want an Activity to use the latest assignment rules rather than inheriting from its parent: // ...ao := workflow.ActivityOptions{ VersioningIntent: VersioningIntentUseAssignmentRules, // ...other options}activityCtx := workflow.WithActivityOptions(ctx, ao)var yourActivityResult YourActivityResultTypeerr := workflow.ExecuteActivity(ctx, YourActivityDefinition, yourActivityParam).Get(ctx, &yourActivityResult)// ... #### Specifying versions for Continue-As-New[​](https://docs.temporal.io/develop/go/worker-versioning-legacy#specifying-versions-for-continue-as-new "Direct link to Specifying versions for Continue-As-New") When using the Continue-As-New feature, use the `WithWorkflowVersioningIntent` context modifier: ctx = workflow.WithWorkflowVersioningIntent(ctx, temporal.VersioningIntentUseAssignmentRules)err := workflow.NewContinueAsNewError(ctx, "WorkflowName") ### Tell the Task Queue about your Worker's Build ID (Deprecated)[​](https://docs.temporal.io/develop/go/worker-versioning-legacy#tell-the-task-queue-about-your-workers-build-id-deprecated "Direct link to Tell the Task Queue about your Worker's Build ID (Deprecated)") caution This section is for a deprecated Worker Versioning API. Please redirect your attention to [Worker Versioning](https://docs.temporal.io/production-deployment/worker-deployments/worker-versioning) . Now you can use the SDK (or the Temporal CLI) to tell the Task Queue about your Worker's Build ID. You might want to do this as part of your CI deployment process. // ...err := client.UpdateWorkerBuildIdCompatibility(ctx, &client.UpdateWorkerBuildIdCompatibilityOptions{ TaskQueue: "your_task_queue_name", Operation: &client.BuildIDOpAddNewIDInNewDefaultSet{ BuildID: "deadbeef", },}) This code adds the `deadbeef` Build ID to the Task Queue as the sole version in a new version set, which becomes the default for the queue. New Workflows execute on Workers with this Build ID, and existing ones will continue to process by appropriately compatible Workers. If, instead, you want to add the Build ID to an existing compatible set, you can do this: // ...err := client.UpdateWorkerBuildIdCompatibility(ctx, &client.UpdateWorkerBuildIdCompatibilityOptions{ TaskQueue: "your_task_queue_name", Operation: &client.BuildIDOpAddNewCompatibleVersion{ BuildID: "deadbeef", ExistingCompatibleBuildId: "some-existing-build-id", },}) This code adds `deadbeef` to the existing compatible set containing `some-existing-build-id` and marks it as the new default Build ID for that set. You can also promote an existing Build ID in a set to be the default for that set: // ...err := client.UpdateWorkerBuildIdCompatibility(ctx, &client.UpdateWorkerBuildIdCompatibilityOptions{ TaskQueue: "your_task_queue_name", Operation: &client.BuildIDPromoteIDWithinSet{ BuildID: "some-existing-build-id", },}) * [How to use Worker Versioning in Go (Deprecated)](https://docs.temporal.io/develop/go/worker-versioning-legacy#worker-versioning) --- # Worker Versioning (Legacy) - Java SDK | Temporal Platform Documentation [Skip to main content](https://docs.temporal.io/develop/java/worker-versioning-legacy#__docusaurus_skipToContent_fallback) On this page How to use Worker Versioning in Java (Deprecated)[​](https://docs.temporal.io/develop/java/worker-versioning-legacy#worker-versioning "Direct link to How to use Worker Versioning in Java (Deprecated)") ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- caution This section is for a deprecated Worker Versioning API. Please redirect your attention to [Worker Versioning](https://docs.temporal.io/production-deployment/worker-deployments/worker-versioning) . See the [Pre-release README](https://github.com/temporalio/temporal/blob/main/docs/worker-versioning.md) for more information. A Build ID corresponds to a deployment. If you don't already have one, we recommend a hash of the code--such as a Git SHA--combined with a human-readable timestamp. To use Worker Versioning, you need to pass a Build ID to your Java Worker and opt in to Worker Versioning. ### Assign a Build ID to your Worker and opt in to Worker Versioning[​](https://docs.temporal.io/develop/java/worker-versioning-legacy#assign-a-build-id-to-your-worker-and-opt-in-to-worker-versioning "Direct link to Assign a Build ID to your Worker and opt in to Worker Versioning") You should understand assignment rules before completing this step. See the [Worker Versioning Pre-release README](https://github.com/temporalio/temporal/blob/main/docs/worker-versioning.md) for more information. To enable Worker Versioning for your worker, assign the Build ID--perhaps from an environment variable--and turn it on. // ...WorkerOptions workerOptions = WorkerOptions.newBuilder() .setBuildId(buildId) .setUseBuildIdForVersioning(true) // ... .build();Worker w = workerFactory.newWorker("your_task_queue_name", workerOptions);// ... danger Importantly, when you start this Worker, it won't receive any tasks until you set up assignment rules. ### Specify versions for Activities, Child Workflows, and Continue-as-New[​](https://docs.temporal.io/develop/java/worker-versioning-legacy#specify-versions-for-activities-child-workflows-and-continue-as-new "Direct link to Specify versions for Activities, Child Workflows, and Continue-as-New") caution Java support for this feature is under construction! By default, Activities, Child Workflows, and Continue-as-New Workflows are run on the build of the workflow that created them if they are also configured to run on the same Task Queue. When configured to run on a separate Task Queue, they will default to using the current assignment rules. If you want to override this behavior, you can specify your intent via the `setVersioningIntent` method on the `ActivityOptions`, `ChildWorkflowOptions`, or `ContinueAsNewOptions` objects. For example, if you want an Activity to use the latest assignment rules rather than inheriting from its parent: // ...private final MyActivity activity = Workflow.newActivityStub( MyActivity.class, ActivityOptions.newBuilder() .setScheduleToCloseTimeout(Duration.ofSeconds(10)) .setVersioningIntent(VersioningIntent.VERSIONING_INTENT_USE_ASSIGNMENT_RULES) // ...other options .build() );// ... ### Tell the Task Queue about your Worker's Build ID (Deprecated)[​](https://docs.temporal.io/develop/java/worker-versioning-legacy#tell-the-task-queue-about-your-workers-build-id-deprecated "Direct link to Tell the Task Queue about your Worker's Build ID (Deprecated)") caution This section is for a deprecated Worker Versioning API. Please redirect your attention to [Worker Versioning](https://docs.temporal.io/production-deployment/worker-deployments/worker-versioning) . Now you can use the SDK (or the Temporal CLI) to tell the Task Queue about your Worker's Build ID. You might want to do this as part of your CI deployment process. // ...workflowClient.updateWorkerBuildIdCompatability( "your_task_queue_name", BuildIdOperation.newIdInNewDefaultSet("deadbeef")); This code adds the `deadbeef` Build ID to the Task Queue as the sole version in a new version set, which becomes the default for the queue. New Workflows execute on Workers with this Build ID, and existing ones will continue to process by appropriately compatible Workers. If, instead, you want to add the Build ID to an existing compatible set, you can do this: // ...workflowClient.updateWorkerBuildIdCompatability( "your_task_queue_name", BuildIdOperation.newCompatibleVersion("deadbeef", "some-existing-build-id")); This code adds `deadbeef` to the existing compatible set containing `some-existing-build-id` and marks it as the new default Build ID for that set. You can also promote an existing Build ID in a set to be the default for that set: // ...workflowClient.updateWorkerBuildIdCompatability( "your_task_queue_name", BuildIdOperation.promoteBuildIdWithinSet("deadbeef")); You can also promote an entire set to become the default set for the queue. New Workflows will start using that set's default. // ...workflowClient.updateWorkerBuildIdCompatability( "your_task_queue_name", BuildIdOperation.promoteSetByBuildId("deadbeef")); * [How to use Worker Versioning in Java (Deprecated)](https://docs.temporal.io/develop/java/worker-versioning-legacy#worker-versioning) --- # Worker Versioning (Legacy) - Typescript SDK | Temporal Platform Documentation [Skip to main content](https://docs.temporal.io/develop/typescript/worker-versioning-legacy#__docusaurus_skipToContent_fallback) On this page How to use Worker Versioning in TypeScript (Deprecated)[​](https://docs.temporal.io/develop/typescript/worker-versioning-legacy#worker-versioning "Direct link to How to use Worker Versioning in TypeScript (Deprecated)") ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- caution This section is for a deprecated Worker Versioning API. Please redirect your attention to [Worker Versioning](https://docs.temporal.io/production-deployment/worker-deployments/worker-versioning) . See the [Pre-release README](https://github.com/temporalio/temporal/blob/main/docs/worker-versioning.md) for more information. A Build ID corresponds to a deployment. If you don't already have one, we recommend a hash of the code--such as a Git SHA--combined with a human-readable timestamp. To use Worker Versioning, you need to pass a Build ID to your Typescript Worker and opt in to Worker Versioning. ### Assign a Build ID to your Worker and opt in to Worker Versioning[​](https://docs.temporal.io/develop/typescript/worker-versioning-legacy#assign-a-build-id-to-your-worker-and-opt-in-to-worker-versioning "Direct link to Assign a Build ID to your Worker and opt in to Worker Versioning") You should understand assignment rules before completing this step. See the [Worker Versioning Pre-release README](https://github.com/temporalio/temporal/blob/main/docs/worker-versioning.md) for more information. To enable Worker Versioning for your Worker, assign the Build ID--perhaps from an environment variable--and turn it on. // ...const worker = await Worker.create({ taskQueue: 'your_task_queue_name', buildId: buildId, useVersioning: true, // ...});// ... danger Importantly, when you start this Worker, it won't receive any tasks until you set up assignment rules. ### Specify versions for Activities, Child Workflows, and Continue-as-New Workflows[​](https://docs.temporal.io/develop/typescript/worker-versioning-legacy#specify-versions-for-activities-child-workflows-and-continue-as-new-workflows "Direct link to Specify versions for Activities, Child Workflows, and Continue-as-New Workflows") caution This section is for a deprecated Worker Versioning API. Please redirect your attention to [Worker Versioning](https://docs.temporal.io/production-deployment/worker-deployments/worker-versioning) . By default, Activities, Child Workflows, and Continue-as-New Workflows are run on the build of the Workflow that created them if they are also configured to run on the same Task Queue. When configured to run on a separate task queue, they will default to using the current assignment rules. If you want to override this behavior, you can specify your intent via the `versioningIntent` field available on the options object for each of these commands. For example, if you want an Activity to use the latest assignment rules rather than inheriting from its parent: // ...const { echo } = proxyActivities({ startToCloseTimeout: '20s', versioningIntent: 'USE_ASSIGNMENT_RULES',});// ... ### Tell the Task Queue about your Worker's Build ID (Deprecated)[​](https://docs.temporal.io/develop/typescript/worker-versioning-legacy#tell-the-task-queue-about-your-workers-build-id-deprecated "Direct link to Tell the Task Queue about your Worker's Build ID (Deprecated)") caution This section is for a deprecated Worker Versioning API. Please redirect your attention to [Worker Versioning](https://docs.temporal.io/production-deployment/worker-deployments/worker-versioning) . Now you can use the SDK (or the Temporal CLI) to tell the Task Queue about your Worker's Build ID. You might want to do this as part of your CI deployment process. // ...await client.taskQueue.updateBuildIdCompatibility('your_task_queue_name', { operation: 'addNewIdInNewDefaultSet', buildId: 'deadbeef',}); This code adds the `deadbeef` Build ID to the Task Queue as the sole version in a new version set, which becomes the default for the queue. New Workflows execute on Workers with this Build ID, and existing ones will continue to process by appropriately compatible Workers. If, instead, you want to add the Build ID to an existing compatible set, you can do this: // ...await client.taskQueue.updateBuildIdCompatibility('your_task_queue_name', { operation: 'addNewCompatibleVersion', buildId: 'deadbeef', existingCompatibleBuildId: 'some-existing-build-id',}); This code adds `deadbeef` to the existing compatible set containing `some-existing-build-id` and marks it as the new default Build ID for that set. You can promote an existing Build ID in a set to be the default for that set: // ...await client.taskQueue.updateBuildIdCompatibility('your_task_queue_name', { operation: 'promoteBuildIdWithinSet', buildId: 'deadbeef',}); You can promote an entire set to become the default set for the queue. New Workflows will start using that set's default build. // ...await client.taskQueue.updateBuildIdCompatibility('your_task_queue_name', { operation: 'promoteSetByBuildId', buildId: 'deadbeef',}); You can merge two sets into one, preserving the primary set's default Build ID as the default for the merged set. // ...await client.taskQueue.updateBuildIdCompatibility('your_task_queue_name', { operation: 'mergeSets', primaryBuildId: 'deadbeef', secondaryBuildId: 'some-existing-build-id',}); * [How to use Worker Versioning in TypeScript (Deprecated)](https://docs.temporal.io/develop/typescript/worker-versioning-legacy#worker-versioning) --- # Worker Versioning (Legacy) - Python SDK | Temporal Platform Documentation [Skip to main content](https://docs.temporal.io/develop/python/workflows/worker-versioning-legacy#__docusaurus_skipToContent_fallback) On this page (Deprecated) How to use Worker Versioning in Python[​](https://docs.temporal.io/develop/python/workflows/worker-versioning-legacy#worker-versioning "Direct link to (Deprecated) How to use Worker Versioning in Python") -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- caution This section is for a deprecated Worker Versioning API. Please redirect your attention to [Worker Versioning](https://docs.temporal.io/production-deployment/worker-deployments/worker-versioning) . See the [Pre-release README](https://github.com/temporalio/temporal/blob/main/docs/worker-versioning.md) for more information. A Build ID corresponds to a deployment. If you don't already have one, we recommend a hash of the code--such as a Git SHA--combined with a human-readable timestamp. To use Worker Versioning, you need to pass a Build ID to your Python Worker and opt in to Worker Versioning. ### Assign a Build ID to your Worker and opt in to Worker Versioning[​](https://docs.temporal.io/develop/python/workflows/worker-versioning-legacy#assign-a-build-id-to-your-worker-and-opt-in-to-worker-versioning "Direct link to Assign a Build ID to your Worker and opt in to Worker Versioning") You should understand assignment rules before completing this step. See the [Worker Versioning Pre-release README](https://github.com/temporalio/temporal/blob/main/docs/worker-versioning.md) for more information. To enable Worker Versioning for your Worker, assign the Build ID--perhaps from an environment variable--and turn it on. # ...worker = Worker( task_queue="your_task_queue_name", build_id=build_id, use_worker_versioning=True, # ... register workflows & activities, etc)# ... danger Importantly, when you start this Worker, it won't receive any tasks until you set up assignment rules. ### Specify versions for Activities, Child Workflows, and Continue-as-New Workflows[​](https://docs.temporal.io/develop/python/workflows/worker-versioning-legacy#specify-versions-for-activities-child-workflows-and-continue-as-new-workflows "Direct link to Specify versions for Activities, Child Workflows, and Continue-as-New Workflows") caution This section is for a deprecated Worker Versioning API. Please redirect your attention to [Worker Versioning](https://docs.temporal.io/production-deployment/worker-deployments/worker-versioning) . By default, Activities, Child Workflows, and Continue-as-New Workflows are run on the build of the workflow that created them if they are also configured to run on the same Task Queue. When configured to run on a separate Task Queue, they will default to using the current assignment rules. If you want to override this behavior, you can specify your intent via the `versioning_intent` argument available on the methods you use to invoke these commands. For example, if you want an Activity to use the latest assignment rules rather than inheriting from its parent: # ...await workflow.execute_activity( say_hello, "hi", versioning_intent=VersioningIntent.USE_ASSIGNMENT_RULES, start_to_close_timeout=timedelta(seconds=5),)# ... ### Tell the Task Queue about your Worker's Build ID (Deprecated)[​](https://docs.temporal.io/develop/python/workflows/worker-versioning-legacy#tell-the-task-queue-about-your-workers-build-id-deprecated "Direct link to Tell the Task Queue about your Worker's Build ID (Deprecated)") caution This section is for a deprecated Worker Versioning API. Please redirect your attention to [Worker Versioning](https://docs.temporal.io/production-deployment/worker-deployments/worker-versioning) . Now you can use the SDK (or the Temporal CLI) to tell the Task Queue about your Worker's Build ID. You might want to do this as part of your CI deployment process. # ...await client.update_worker_build_id_compatibility( "your_task_queue_name", BuildIdOpAddNewDefault("deadbeef")) This code adds the `deadbeef` Build ID to the Task Queue as the sole version in a new version set, which becomes the default for the queue. New Workflows execute on Workers with this Build ID, and existing ones will continue to process by appropriately compatible Workers. If, instead, you want to add the Build ID to an existing compatible set, you can do this: # ...await client.update_worker_build_id_compatibility( "your_task_queue_name", BuildIdOpAddNewCompatible("deadbeef", "some-existing-build-id")) This code adds `deadbeef` to the existing compatible set containing `some-existing-build-id` and marks it as the new default Build ID for that set. You can also promote an existing Build ID in a set to be the default for that set: # ...await client.update_worker_build_id_compatibility( "your_task_queue_name", BuildIdOpPromoteBuildIdWithinSet("deadbeef")) You can also promote an entire set to become the default set for the queue. New Workflows will start using that set's default build. # ...await client.update_worker_build_id_compatibility( "your_task_queue_name", BuildIdOpPromoteSetByBuildId("deadbeef")) * [(Deprecated) How to use Worker Versioning in Python](https://docs.temporal.io/develop/python/workflows/worker-versioning-legacy#worker-versioning) --- # Temporal's production deployment features | Temporal Platform Documentation [Skip to main content](https://docs.temporal.io/evaluate/development-production-features/cloud-vs-self-hosted-features#__docusaurus_skipToContent_fallback) Transform your Temporal applications into production-ready systems by deploying your application code, Workflows, Activities, and Workers for operational use. When your application is ready to start serving production traffic, we offer two Temporal Service options: * **[Choose Temporal Cloud for your Temporal Service](https://docs.temporal.io/cloud) ** Let us handle the Temporal Service operations so you can focus on your applications. * **[Self-host a Temporal Service](https://docs.temporal.io/self-hosted-guide) ** Deploy your own production level Temporal Service to orchestrate your durable applications. | Feature | Temporal Cloud | Self-hosted | | --- | --- | --- | | **Multi-tenant** | ✅ Up to 100 Namespaces | ✅ Unlimited Namespaces | | **High availability and failover** | ✅ [Namespaces with High Availability features](https://docs.temporal.io/cloud/high-availability) | ✅ Global Namespaces & Multi-Cluster Replication | | **Application state persistence** | ✅ 30-90 day Retention | ✅ Unlimited | | **Long term state retention** | ✅ Workflow History Export | ✅ Archival | | **Community support** | ✅ Slack, Forum | ✅ Slack, Forum | | **Paid support** | ✅ Prioritized responses | ✖️ | --- # Install the TypeScript SDK | Temporal Platform Documentation [Skip to main content](https://docs.temporal.io/develop/typescript/install-typescript-sdk#__docusaurus_skipToContent_fallback) On this page How to install a Temporal SDK[​](https://docs.temporal.io/develop/typescript/install-typescript-sdk#install-a-temporal-sdk "Direct link to How to install a Temporal SDK") --------------------------------------------------------------------------------------------------------------------------------------------------------------------------- A [Temporal SDK](https://docs.temporal.io/encyclopedia/temporal-sdks) provides a framework for [Temporal Application](https://docs.temporal.io/temporal#temporal-application) development. An SDK provides you with the following: * A [Temporal Client](https://docs.temporal.io/encyclopedia/temporal-sdks#temporal-client) to communicate with a [Temporal Service](https://docs.temporal.io/temporal-service) . * APIs to develop [Workflows](https://docs.temporal.io/workflows) . * APIs to create and manage [Worker Processes](https://docs.temporal.io/workers#worker) . * APIs to author [Activities](https://docs.temporal.io/activity-definition) . [![NPM](https://img.shields.io/npm/v/temporalio.svg?style=for-the-badge)](https://www.npmjs.com/search?q=author%3Atemporal-sdk-team) This project requires Node.js 18 or later. **Create a project** npx @temporalio/create@latest ./your-app **Add to an existing project** npm install @temporalio/client @temporalio/worker @temporalio/workflow @temporalio/activity @temporalio/common note The TypeScript SDK is designed with TypeScript-first developer experience in mind, but it works equally well with JavaScript. ### How to find the TypeScript SDK API reference[​](https://docs.temporal.io/develop/typescript/install-typescript-sdk#api-reference "Direct link to How to find the TypeScript SDK API reference") The Temporal TypeScript SDK API reference is published to [typescript.temporal.io](https://typescript.temporal.io/) . ### Where are SDK-specific code examples?[​](https://docs.temporal.io/develop/typescript/install-typescript-sdk#code-samples "Direct link to Where are SDK-specific code examples?") You can find a complete list of executable code samples in [Temporal's GitHub repository](https://github.com/temporalio?q=samples-&type=all&language=&sort=) . Additionally, several of the [Tutorials](https://learn.temporal.io/) are backed by a fully executable template application. Use the [TypeScript samples library](https://github.com/temporalio/samples-typescript) stored on GitHub to demonstrate various capabilities of Temporal. **Where can I find video demos?** [Temporal TypeScript YouTube playlist](https://www.youtube.com/playlist?list=PLl9kRkvFJrlTavecydpk9r6cF7qBmQJvb) . ### How to import an ECMAScript module[​](https://docs.temporal.io/develop/typescript/install-typescript-sdk#ecmascript-modules "Direct link to How to import an ECMAScript module") The JavaScript ecosystem is quickly moving toward publishing ECMAScript modules (ESM) instead of CommonJS modules. For example, `node-fetch@3` is ESM, but `node-fetch@2` is CommonJS. For more information about importing a pure ESM dependency, see our [Fetch ESM](https://github.com/temporalio/samples-typescript/tree/main/fetch-esm) sample for the necessary configuration changes: * `package.json` must have include the `"type": "module"` attribute. * `tsconfig.json` should output in `esnext` format. * Imports must include the `.js` file extension. Linting and types in TypeScript[​](https://docs.temporal.io/develop/typescript/install-typescript-sdk#linting-and-types "Direct link to Linting and types in TypeScript") -------------------------------------------------------------------------------------------------------------------------------------------------------------------------- If you started your project with `@temporalio/create`, you already have our recommended TypeScript and ESLint configurations. If you incrementally added Temporal to an existing app, we do recommend setting up linting and types because they help catch bugs well before you ship them to production, and they improve your development feedback loop. Take a look at our recommended [.eslintrc](https://github.com/temporalio/samples-typescript/blob/main/.shared/.eslintrc.js) file and tweak to suit your needs. * [How to install a Temporal SDK](https://docs.temporal.io/develop/typescript/install-typescript-sdk#install-a-temporal-sdk) * [How to find the TypeScript SDK API reference](https://docs.temporal.io/develop/typescript/install-typescript-sdk#api-reference) * [Where are SDK-specific code examples?](https://docs.temporal.io/develop/typescript/install-typescript-sdk#code-samples) * [How to import an ECMAScript module](https://docs.temporal.io/develop/typescript/install-typescript-sdk#ecmascript-modules) * [Linting and types in TypeScript](https://docs.temporal.io/develop/typescript/install-typescript-sdk#linting-and-types) --- # Worker versioning (legacy) | Temporal Platform Documentation [Skip to main content](https://docs.temporal.io/encyclopedia/worker-versioning-legacy#__docusaurus_skipToContent_fallback) On this page Support, stability, and dependency info * This document refers to the 2023 draft of Worker Versioning, which was deprecated * It was not made available in Temporal Cloud * The 2024 draft was available in Cloud on an opt-in basis, and is documented in this [Pre-release README.md](https://github.com/temporalio/temporal/blob/main/docs/worker-versioning.md) . For newer revisions of this feature set, please see [Worker Versioning](https://docs.temporal.io/production-deployment/worker-deployments/worker-versioning) instead. Worker Versioning simplifies the process of deploying changes to [Workflow Definitions](https://docs.temporal.io/workflow-definition) . It does this by letting you define sets of versions that are compatible with each other, and then assigning a Build ID to the code that defines a Worker. The Temporal Server uses the Build ID to determine which versions of a Workflow Definition a Worker can process. We recommend that you read about Workflow Definitions before proceeding, because Workflow Versioning is largely concerned with helping to manage nondeterministic changes to those definitions. Worker Versioning helps manage nondeterministic changes by providing a convenient way to ensure that [Workers](https://docs.temporal.io/workers) with different Workflow and Activity Definitions operating on the same Task Queue don't attempt to process [Workflow Tasks](https://docs.temporal.io/tasks#workflow-task) and [Activity Tasks](https://docs.temporal.io/tasks#activity-task-execution) that they can't successfully process, according to sets of versions associated with that Task Queue that you've defined. Accomplish this goal by assigning a Build ID (a free-form string) to the code that defines a Worker, and specifying which Build IDs are compatible with each other by updating the version sets associated with the Task Queue, stored by the Temporal Server. ### When and why you should use Worker Versioning[​](https://docs.temporal.io/encyclopedia/worker-versioning-legacy#when-and-why-you-should-use-worker-versioning "Direct link to When and why you should use Worker Versioning") caution This section is for a deprecated Worker Versioning API. Please redirect your attention to [Worker Versioning](https://docs.temporal.io/production-deployment/worker-deployments/worker-versioning) . The main reason to use this feature is to deploy incompatible changes to short-lived [Workflows](https://docs.temporal.io/workflows) . On Task Queues using this feature, the Workflow starter doesn't have to know about the introduction of new versions. The new code in the newly deployed Workers executes new [Workflow Executions](https://docs.temporal.io/workflow-execution) , while only Workers with an appropriate version process old Workflow Executions. #### Decommission old Workers[​](https://docs.temporal.io/encyclopedia/worker-versioning-legacy#decommission-old-workers "Direct link to Decommission old Workers") You can decommission old Workers after you archive all open Workflows using their version. If you have no need to query closed Workflows, you can decommission them when no open Workflows remain at that version. For example, if you have a Workflow that completes within a day, a good strategy is to assign a new Build ID to every new Worker build and add it as the new overall default in the version sets. Because your Workflow completes in a day, you know that you won't need to keep older Workers running for more than a day after you deploy the new version (assuming availability). You can apply this technique to longer-lived Workflows too; however, you might need to run multiple Worker versions simultaneously while open Workflows complete. Version sets have a maximum size limit, which defaults to 100 Build IDs across all sets. Operations to add new Build IDs to the sets will fail if they exceed this limit. There is also a limit on the number of Version Sets, which defaults to 10. A version can only be garbage collected after a Workflow Execution is deleted. #### Deploy code changes to Workers[​](https://docs.temporal.io/encyclopedia/worker-versioning-legacy#deploy-code-changes-to-workers "Direct link to Deploy code changes to Workers") The feature also lets you implement compatible changes to or prevent a buggy code path from executing on currently open Workflows. You can achieve this by adding a new version to an existing set and defining it as _compatible_ with an existing version, which shouldn't execute any future Workflow Tasks. Because the new version processes existing [Event Histories](https://docs.temporal.io/workflow-execution/event#event-history) , it must adhere to the usual [deterministic constraints](https://docs.temporal.io/workflow-definition#deterministic-constraints) , and you might need to use one of the [versioning APIs](https://docs.temporal.io/workflow-definition#workflow-versioning) . Moreover, this feature lets you make incompatible changes to Activity Definitions in conjunction with incompatible changes to Workflow Definitions that use those Activities. This functionality works because any Activity that a Workflow schedules on the same Task Queue gets dispatched by default only to Workers compatible with the Workflow that scheduled it. If you want to change an Activity Definition's type signature while creating a new incompatible Build ID for a Worker, you can do so without worrying about the Activity failing to execute on some other Worker with an incompatible definition. The same principle applies to Child Workflows. For both Activities and Child Workflows, you can override the default behavior and run the Activity or Child Workflow on latest default version. tip Public-facing Workflows on a versioned Task Queue shouldn't change their signatures because doing so contradicts the purpose of Workflow-launching Clients remaining unaware of changes in the Workflow Definition. If you need to change a Workflow's signature, use a different Workflow Type or a completely new Task Queue. note If you schedule an Activity or a Child Workflow on _a different_ Task Queue from the one the Workflow runs on, the system doesn't assign a specific version. This means if the target queue is versioned, they run on the latest default, and if it's unversioned, they operate as they would have without this feature. **Continue-As-New and Worker Versioning** By default, a versioned Task Queue's Continue-as-New function starts the continued Workflow on the same compatible set as the original Workflow. If you continue-as-new onto a different Task Queue, the system doesn't assign any particular version. You also have the option to specify that the continued Workflow should start using the Task Queue's latest default version. ### How to use Worker Versioning[​](https://docs.temporal.io/encyclopedia/worker-versioning-legacy#how-to-use-worker-versioning "Direct link to How to use Worker Versioning") caution This section is for a deprecated Worker Versioning API. See [Worker Versioning](https://docs.temporal.io/production-deployment/worker-deployments/worker-versioning) . To use Worker Versioning, follow these steps: 1. Define Worker build-identifier version sets for the Task Queue. You can use either the Temporal CLI or your choice of SDK. 2. Enable the feature on your Worker by specifying a Build ID. #### Defining the version sets[​](https://docs.temporal.io/encyclopedia/worker-versioning-legacy#defining-the-version-sets "Direct link to Defining the version sets") Whether you use [Temporal CLI](https://docs.temporal.io/cli/) or an SDK, updating the version sets feels the same. You specify the Task Queue that you're targeting, the Build ID that you're adding (or promoting), whether it becomes the new default version, and any existing versions it should be considered compatible with. The rest of this section uses updates to one Task Queue's version sets as examples. By default, both Task Queues and Workers are in an unversioned state. [Unversioned Worker](https://docs.temporal.io/encyclopedia/worker-versioning-legacy#unversioned-workers) can poll unversioned Task Queues and receive tasks. To use this feature, both the Task Queue and the Worker must be associated with Build IDs. If you run a Worker using versioning against a Task Queue that has not been set up to use versioning (or is missing that Worker's Build ID), it won't get any tasks. Likewise, an unversioned Worker polling a Task Queue with versioning won't work either. Versions don't need to follow semver or any other semantic versioning scheme! The versions in the following examples look like semver versions for clarity, but they don't need to be. Versions can be any arbitrary string. First, add a version `1.0` to the Task Queue as the new default. Your version sets now look like this: | set 1 (default) | | --- | | 1.0 (default) | All new Workflows started on the Task Queue have their first tasks assigned to version `1.0`. Workers with their Build ID set to `1.0` receive these Tasks. If Workflows that don't have an assigned version are still running on the Task Queue, Workers without a version take those tasks. So ensure that such Workers are still operational if any Workflows were open when you added the first version. If you deployed any Workers with a _different_ version, those Workers receive no Tasks. Now, imagine you need to change the Workflow for some reason. Add `2.0` to the sets as the new default: | set 1 | set 2 (default) | | --- | --- | | 1.0 (default) | 2.0 (default) | All new Workflows started on the Task Queue have their first tasks assigned to version `2.0`. Existing `1.0` Workflows keep generating tasks targeting `1.0`. Each deployment of Workers receives their respective Tasks. This same concept carries forward for each new incompatible version. Maybe you have a bug in `2.0`, and you want to make sure all open `2.0` Workflows switch to some new code as fast as possible. So, you add `2.1` to the sets, marking it as compatible with `2.0`. Now your sets look like this: | set 1 | set 2 (default) | | --- | --- | | 1.0 (default) | 2.0 | | | 2.1 (default) | All new Workflow Tasks that are generated for Workflows whose last Workflow Task completion was on version `2.0` are now assigned to version `2.1`. Because you specified that `2.1` is compatible with `2.0`, Temporal Server assumes that Workers with this version can process the existing Event Histories successfully. Continue with your normal development cycle, adding a `3.0` version. Nothing new here: | set 1 | set 2 | set 3 (default) | | --- | --- | --- | | 1.0 (default) | 2.0 | 3.0 (default) | | | 2.1 (default) | | Now imagine that version `3.0` doesn't have an explicit bug, but something about the business logic is less than ideal. You are okay with existing `3.0` Workflows running to completion, but you want new Workflows to use the old `2.x` branch. This operation is supported by performing an update targeting `2.1` (or `2.0`) and setting its set as the current default, which results in these sets: | set 1 | set 3 | set 2 (default) | | --- | --- | --- | | 1.0 (default) | 3.0 (default) | 2.0 | | | | 2.1 (default) | Now new Workflows start on `2.1`. #### Permitted and forbidden operations on version sets[​](https://docs.temporal.io/encyclopedia/worker-versioning-legacy#permitted-and-forbidden-operations-on-version-sets "Direct link to Permitted and forbidden operations on version sets") A request to change the sets can do one of the following: * Add a version to the sets as the new default version in a new overall-default compatible set. * Add a version to an existing set that's compatible with an existing version. * Optionally making it the default for that set. * Optionally making that set the overall-default set. * Promote a version within an existing set to become the default for that set. * Promote a set to become the overall-default set. You can't explicitly delete versions. This helps you avoid the situation in which Workflows accidentally become stuck with no means of making progress because the version they're associated with no longer exists. However, sometimes you might want to do this intentionally. If you _want_ to make sure that all Workflows currently being processed by, say, `2.0` stop (even if you don't yet have a new version ready), you can add a new version `2.1` to the sets marked as compatible with `2.0`. New tasks will target `2.1`, but because you haven't deployed any `2.1` Workers, they won't make any progress. #### Set constraints[​](https://docs.temporal.io/encyclopedia/worker-versioning-legacy#set-constraints "Direct link to Set constraints") The sets have a maximum size limit, which defaults to 100 build IDs across all sets. This limit is configurable on Temporal Server via the `limit.versionBuildIdLimitPerQueue` dynamic config property. Operations to add new Build IDs to the sets fail if the limit would be exceeded. There is also a limit on the number of sets, which defaults to 10. This limit is configurable via the `limit.versionCompatibleSetLimitPerQueue` dynamic config property. In practice, these limits should rarely be a concern because a version is no longer needed after no open Workflows are using that version, and a background process will delete IDs and sets that are no longer needed. There is also a limit on the size of each Build ID or version string, which defaults to 255 characters. This limit is configurable on the server via the `limit.workerBuildIdSize` dynamic config property. ### Build ID reachability[​](https://docs.temporal.io/encyclopedia/worker-versioning-legacy#build-id-reachability "Direct link to Build ID reachability") caution This section is for a deprecated Worker Versioning API. See [Worker Versioning](https://docs.temporal.io/production-deployment/worker-deployments/worker-versioning) . Eventually, you'll want to know whether you can retire the old Worker versions. Temporal provides functionality to help you determine whether a version is still in use by open or closed Workflows. You can use the Temporal CLI to do this with the following command: temporal task-queue get-build-id-reachability The command determines, for each Task Queue, whether the Build ID in question is unreachable, only reachable by closed Workflows, or reachable by open and new Workflows. For example, this "2.0" Build ID is shown here by the Temporal CLI to be reachable by both new Workflows and some existing Workflows: temporal task-queue get-build-id-reachability --build-id "2.0" BuildId TaskQueue Reachability 2.0 build-id-versioning-dc0068f6-0426-428f-b0b2-703a7e409a97 [NewWorkflows ExistingWorkflows] For more information, see the [CLI documentation](https://docs.temporal.io/cli/) or help output. You can also use this API `GetWorkerTaskReachability` directly from within language SDKs. ### Unversioned Workers[​](https://docs.temporal.io/encyclopedia/worker-versioning-legacy#unversioned-workers "Direct link to Unversioned Workers") Unversioned Workers refer to Workers that have not opted into the Worker Versioning feature in their configuration. They receive tasks only from Task Queues that do not have any version sets defined on them, or that have open Workflows that began executing before versions were added to the queue. To migrate from an unversioned Task Queue, add a new default Build ID to the Task Queue. From there, deploy Workers with the same Build ID. Unversioned Workers will continue processing open Workflows, while Workers with the new Build ID will process new Workflow Executions. * [When and why you should use Worker Versioning](https://docs.temporal.io/encyclopedia/worker-versioning-legacy#when-and-why-you-should-use-worker-versioning) * [Decommission old Workers](https://docs.temporal.io/encyclopedia/worker-versioning-legacy#decommission-old-workers) * [Deploy code changes to Workers](https://docs.temporal.io/encyclopedia/worker-versioning-legacy#deploy-code-changes-to-workers) * [How to use Worker Versioning](https://docs.temporal.io/encyclopedia/worker-versioning-legacy#how-to-use-worker-versioning) * [Defining the version sets](https://docs.temporal.io/encyclopedia/worker-versioning-legacy#defining-the-version-sets) * [Permitted and forbidden operations on version sets](https://docs.temporal.io/encyclopedia/worker-versioning-legacy#permitted-and-forbidden-operations-on-version-sets) * [Set constraints](https://docs.temporal.io/encyclopedia/worker-versioning-legacy#set-constraints) * [Build ID reachability](https://docs.temporal.io/encyclopedia/worker-versioning-legacy#build-id-reachability) * [Unversioned Workers](https://docs.temporal.io/encyclopedia/worker-versioning-legacy#unversioned-workers) --- # Develop with AI | Temporal Platform Documentation [Skip to main content](https://docs.temporal.io/with-ai#__docusaurus_skipToContent_fallback) On this page Give your AI coding agent Temporal expertise with Skills and real-time documentation access with the Temporal Docs MCP Server. Skills[​](https://docs.temporal.io/with-ai#skills "Direct link to Skills") --------------------------------------------------------------------------- Skills give AI agents domain-specific Temporal expertise. They work with Claude Code, Codex, Cursor, and any agent that supports [Skills](https://agentskills.io/) . ### Temporal Developer Skill[​](https://docs.temporal.io/with-ai#temporal-developer-skill "Direct link to Temporal Developer Skill") The [Temporal Developer Skill](https://github.com/temporalio/skill-temporal-developer) gives your AI coding agent expert-level knowledge of Temporal's programming model — workflow determinism rules, activity patterns, retry policies, error handling, testing strategies, worker configuration, versioning, and common gotchas. * Claude Code Plugin * Cursor * npx * Manual 1. Add the Temporal skills marketplace to Claude Code: /plugin marketplace add temporalio/claude-temporal-plugin 2. Install the Temporal Developer Skill: /plugin install temporal-developer@temporal-marketplace Install the Temporal plugin from the [Cursor Marketplace](https://cursor.com/marketplace/temporal) , or run the following command in Cursor's agent chat: /add-plugin temporal This works with Claude Code, Codex, Cline, and other agents. Install the skill using the `skills` CLI: npx skills add https://github.com/temporalio/skill-temporal-developer Clone the skill repository into your Claude skills directory. Change the target directory if you are using agents other than Claude: git clone https://github.com/temporalio/skill-temporal-developer.git ~/.claude/skills/temporal-developer Restart your coding agent after installing. ### Temporal Cloud Skill[​](https://docs.temporal.io/with-ai#temporal-cloud-skill "Direct link to Temporal Cloud Skill") The [Temporal Cloud Skill](https://github.com/temporalio/skill-temporal-cloud) helps your AI coding agent troubleshoot Temporal Cloud connectivity, authentication, and configuration issues. * npx * Manual This works with Claude Code, Codex, Cline, and other agents. Install the skill using the `skills` CLI: npx skills add https://github.com/temporalio/skill-temporal-cloud Clone the skill repository into your Claude skills directory. Change the target directory if you are using agents other than Claude: git clone https://github.com/temporalio/skill-temporal-cloud.git ~/.claude/skills/temporal-cloud Restart your coding agent after installing. Temporal Docs MCP Server[​](https://docs.temporal.io/with-ai#temporal-docs-mcp-server "Direct link to Temporal Docs MCP Server") --------------------------------------------------------------------------------------------------------------------------------- Connect Temporal documentation directly to your AI assistant for accurate, up-to-date answers about Temporal. The Temporal docs MCP server gives AI tools real-time access to our documentation, so responses draw from current docs rather than training data. The server requires anonymous authentication using any Google account to enforce rate limits and prevent abuse. We cannot see nor do we collect any contact information from this. ### Claude Code[​](https://docs.temporal.io/with-ai#claude-code "Direct link to Claude Code") Add the Temporal docs MCP server globally so it's available in all your projects: 1. Register the MCP server with Claude Code: claude mcp add --scope user --transport http temporal-docs https://temporal.mcp.kapa.ai 2. Restart Claude Code and run `/mcp` to authenticate with your Google account. To add the server to a specific project only, omit the `--scope user` flag. This stores the configuration in the project's `.mcp.json` file: claude mcp add --transport http temporal-docs https://temporal.mcp.kapa.ai ### Claude Desktop[​](https://docs.temporal.io/with-ai#claude-desktop "Direct link to Claude Desktop") 1. Open Claude Desktop settings 2. Navigate to **Settings > Connectors** 3. Add a new MCP server with the URL: `https://temporal.mcp.kapa.ai` ### Other MCP-compatible tools[​](https://docs.temporal.io/with-ai#other-mcp-compatible-tools "Direct link to Other MCP-compatible tools") For any tool that supports the Model Context Protocol, use the following server URL: https://temporal.mcp.kapa.ai Configuration format varies by tool. Here's a generic JSON configuration: { "mcpServers": { "temporal-docs": { "transport": "http", "url": "https://temporal.mcp.kapa.ai" } }} * [Skills](https://docs.temporal.io/with-ai#skills) * [Temporal Developer Skill](https://docs.temporal.io/with-ai#temporal-developer-skill) * [Temporal Cloud Skill](https://docs.temporal.io/with-ai#temporal-cloud-skill) * [Temporal Docs MCP Server](https://docs.temporal.io/with-ai#temporal-docs-mcp-server) * [Claude Code](https://docs.temporal.io/with-ai#claude-code) * [Claude Desktop](https://docs.temporal.io/with-ai#claude-desktop) * [Other MCP-compatible tools](https://docs.temporal.io/with-ai#other-mcp-compatible-tools) --- # tctl v1.17 cluster command reference | Temporal Platform Documentation [Skip to main content](https://docs.temporal.io/tctl-v1/cluster#__docusaurus_skipToContent_fallback) On this page tctl is deprecated The tctl command line utility has been deprecated and is no longer actively supported. We recommend transitioning to [Temporal CLI](https://docs.temporal.io/cli) for continued use and access to new features. Thank you for being a valued part of the Temporal community. The `tctl cluster` command enables [Temporal Cluster](https://docs.temporal.io/temporal-service) operations. * [tctl cluster health](https://docs.temporal.io/tctl-v1/cluster#health) * [tctl cluster get-search-attributes](https://docs.temporal.io/tctl-v1/cluster#get-search-attributes) get-search-attributes[​](https://docs.temporal.io/tctl-v1/cluster#get-search-attributes "Direct link to get-search-attributes") -------------------------------------------------------------------------------------------------------------------------------- The `tctl cluster get-search-attributes` command lists all [Search Attributes](https://docs.temporal.io/search-attribute) that can be used in the `--query` modifier of the [`tctl workflow list`](https://docs.temporal.io/tctl-v1/workflow#list) command and the `--search_attr_key` and `--search_attr_value` modifiers of the [`tctl workflow run`](https://docs.temporal.io/tctl-v1/workflow#run) and [`tctl workflow start`](https://docs.temporal.io/tctl-v1/workflow#start) commands. **Example:** tctl cluster get-search-attributes The command has no modifiers. Example output: +-----------------------+----------+| NAME | TYPE |+-----------------------+----------+| BinaryChecksums | Keyword || CloseTime | Int || CustomBoolField | Bool || CustomDatetimeField | Datetime || CustomDoubleField | Double || CustomIntField | Int || CustomKeywordField | Keyword || CustomNamespace | Keyword || CustomStringField | String || ExecutionStatus | Int || ExecutionTime | Int || Operator | Keyword || RunId | Keyword || StartTime | Int || TaskQueue | Keyword || TemporalChangeVersion | Keyword || WorkflowId | Keyword || WorkflowType | Keyword |+-----------------------+----------+ The admin version of this command displays default and custom Search Attributes separately, and also shows the underlying Elasticsearch index schema and system Workflow status. health[​](https://docs.temporal.io/tctl-v1/cluster#health "Direct link to health") ----------------------------------------------------------------------------------- The `tctl cluster health` command checks the health of the [Frontend Service](https://docs.temporal.io/temporal-service/temporal-server#frontend-service) . `tctl cluster health` The command has no modifiers. * [get-search-attributes](https://docs.temporal.io/tctl-v1/cluster#get-search-attributes) * [health](https://docs.temporal.io/tctl-v1/cluster#health) --- # tctl v1.17 activity command reference | Temporal Platform Documentation [Skip to main content](https://docs.temporal.io/tctl-v1/activity#__docusaurus_skipToContent_fallback) On this page tctl is deprecated The tctl command line utility has been deprecated and is no longer actively supported. We recommend transitioning to [Temporal CLI](https://docs.temporal.io/cli) for continued use and access to new features. Thank you for being a valued part of the Temporal community. The `tctl activity` commands enable [Activity Execution](https://docs.temporal.io/activity-execution) operations. * [tctl activity complete](https://docs.temporal.io/tctl-v1/activity#complete) * [tctl activity fail](https://docs.temporal.io/tctl-v1/activity#fail) complete[​](https://docs.temporal.io/tctl-v1/activity#complete "Direct link to complete") ------------------------------------------------------------------------------------------ The `tctl activity complete` command completes an [Activity Execution](https://docs.temporal.io/activity-execution) . `tctl activity complete ` The following modifiers control the behavior of the command. ### \--workflow\_id[​](https://docs.temporal.io/tctl-v1/activity#--workflow_id "Direct link to --workflow_id") Specify the [Workflow Id](https://docs.temporal.io/workflow-execution/workflowid-runid#workflow-id) of an [Activity Execution](https://docs.temporal.io/activity-execution) to complete. Alias: `-w` **Example** tctl activity complete --workflow_id ### \--run\_id[​](https://docs.temporal.io/tctl-v1/activity#--run_id "Direct link to --run_id") Specify the [Run Id](https://docs.temporal.io/workflow-execution/workflowid-runid#run-id) of an [Activity Execution](https://docs.temporal.io/activity-execution) to complete. Alias: `-r` **Example** tctl activity complete --run_id ### \--activity\_id[​](https://docs.temporal.io/tctl-v1/activity#--activity_id "Direct link to --activity_id") Specify the [Activity Id](https://docs.temporal.io/activity-execution#activity-id) of an [Activity Execution](https://docs.temporal.io/activity-execution) to complete. **Example** tctl activity complete --activity_id ### \--result[​](https://docs.temporal.io/tctl-v1/activity#--result "Direct link to --result") Specify the result of an [Activity Execution](https://docs.temporal.io/activity-execution) when using tctl to complete the Activity Execution. **Example** tctl activity complete --result ### \--identity[​](https://docs.temporal.io/tctl-v1/activity#--identity "Direct link to --identity") Specify the identity of the operator when using tctl to complete an [Activity Execution](https://docs.temporal.io/activity-execution) . **Example** tctl activity complete --identity fail[​](https://docs.temporal.io/tctl-v1/activity#fail "Direct link to fail") ------------------------------------------------------------------------------ The `tctl activity fail` command fails an [Activity Execution](https://docs.temporal.io/activity-execution) . `tctl activity fail []` The following modifiers control the behavior of the command. ### \--workflow\_id[​](https://docs.temporal.io/tctl-v1/activity#--workflow_id-1 "Direct link to --workflow_id") Specify the [Workflow Id](https://docs.temporal.io/workflow-execution/workflowid-runid#workflow-id) of an [Activity Execution](https://docs.temporal.io/activity-execution) to fail. Alias: `-w` **Example** tctl activity fail --workflow_id ### \--run\_id[​](https://docs.temporal.io/tctl-v1/activity#--run_id-1 "Direct link to --run_id") Specify the [Run Id](https://docs.temporal.io/workflow-execution/workflowid-runid#run-id) of an [Activity Execution](https://docs.temporal.io/activity-execution) to fail. Alias: `-r` **Example** tctl activity fail --run_id ### \--activity\_id[​](https://docs.temporal.io/tctl-v1/activity#--activity_id-1 "Direct link to --activity_id") Specify the [Activity Id](https://docs.temporal.io/activity-execution#activity-id) of an [Activity Execution](https://docs.temporal.io/activity-execution) to fail. **Example** tctl activity fail --activity_id ### \--reason[​](https://docs.temporal.io/tctl-v1/activity#--reason "Direct link to --reason") Specify the reason for failing an [Activity Execution](https://docs.temporal.io/activity-execution) . **Example** tctl activity fail --reason ### \--detail[​](https://docs.temporal.io/tctl-v1/activity#--detail "Direct link to --detail") Specify details of the reason for failing an [Activity Execution](https://docs.temporal.io/activity-execution) . **Example** tctl activity fail --detail ### \--identity[​](https://docs.temporal.io/tctl-v1/activity#--identity-1 "Direct link to --identity") Specify the identity of the operator when using tctl to fail an [Activity Execution](https://docs.temporal.io/activity-execution) . **Example** tctl activity complete --identity * [complete](https://docs.temporal.io/tctl-v1/activity#complete) * [\--workflow\_id](https://docs.temporal.io/tctl-v1/activity#--workflow_id) * [\--run\_id](https://docs.temporal.io/tctl-v1/activity#--run_id) * [\--activity\_id](https://docs.temporal.io/tctl-v1/activity#--activity_id) * [\--result](https://docs.temporal.io/tctl-v1/activity#--result) * [\--identity](https://docs.temporal.io/tctl-v1/activity#--identity) * [fail](https://docs.temporal.io/tctl-v1/activity#fail) * [\--workflow\_id](https://docs.temporal.io/tctl-v1/activity#--workflow_id-1) * [\--run\_id](https://docs.temporal.io/tctl-v1/activity#--run_id-1) * [\--activity\_id](https://docs.temporal.io/tctl-v1/activity#--activity_id-1) * [\--reason](https://docs.temporal.io/tctl-v1/activity#--reason) * [\--detail](https://docs.temporal.io/tctl-v1/activity#--detail) * [\--identity](https://docs.temporal.io/tctl-v1/activity#--identity-1) --- # tctl 1.17 schedule command reference | Temporal Platform Documentation [Skip to main content](https://docs.temporal.io/tctl-v1/schedule#__docusaurus_skipToContent_fallback) On this page tctl is deprecated The tctl command line utility has been deprecated and is no longer actively supported. We recommend transitioning to [Temporal CLI](https://docs.temporal.io/cli) for continued use and access to new features. Thank you for being a valued part of the Temporal community. A [Schedule](https://docs.temporal.io/schedule) is an experimental feature available in `tctl 1.17` and `tctl next`. * [Backfill a Schedule using tctl](https://docs.temporal.io/tctl-v1/schedule#backfill) * [Create a Schedule using tctl](https://docs.temporal.io/tctl-v1/schedule#create) * [Delete a Schedule using tctl](https://docs.temporal.io/tctl-v1/schedule#delete) * [Describe a Schedule using tctl](https://docs.temporal.io/tctl-v1/schedule#describe) * [List Schedules using tctl](https://docs.temporal.io/tctl-v1/schedule#list) * [Toggle Pause on Schedule using tctl](https://docs.temporal.io/tctl-v1/schedule#toggle) * [Trigger an Action on a Schedule using tctl](https://docs.temporal.io/tctl-v1/schedule#trigger) * [Update a Schedule using tctl](https://docs.temporal.io/tctl-v1/schedule#update) backfill[​](https://docs.temporal.io/tctl-v1/schedule#backfill "Direct link to backfill") ------------------------------------------------------------------------------------------ Backfilling a Schedule means having it do now what it would have done over a specified time range (generally in the past, although it won't prevent you from giving a time range in the future). You might use this to fill in runs from a time period when the Schedule was paused due to an external condition that's now resolved, or a period before the Schedule was created. tctl schedule backfill --sid 'your-schedule-id' \ --overlap-policy 'BufferAll' \ --start-time '2022-05-01T00:00:00Z' \ --end-time '2022-05-31T23:59:59Z' Note that, similar to [tctl schedule trigger](https://docs.temporal.io/tctl-v1/schedule#trigger) immediately, you probably want to override the Overlap Policy. Specifying `AllowAll` runs all the backfilled Workflows at once; `BufferAll` runs them sequentially. The other policies don't make much sense in this context. create[​](https://docs.temporal.io/tctl-v1/schedule#create "Direct link to create") ------------------------------------------------------------------------------------ With tctl, create a Schedule like this: $ tctl config set version next # ensure you're using the new tctl$ tctl schedule create \ --schedule-id 'your-schedule-id' \ --interval '5h/15m' \ --calendar '{"dayOfWeek":"Fri","hour":"11","minute":"3"}' \ --overlap-policy 'BufferAll' \ --workflow-id 'your-workflow-id' \ --task-queue 'your-task-queue' \ --workflow-type 'YourWorkflowType' This Schedule takes action every 5 hours at 15 minutes past the hour and also at 11:03 on Fridays. It starts a Workflow `YourWorkflowType` on Task Queue `your-task-queue`, giving it a Workflow Id like `your-workflow-id-2022-06-17T11:03:00Z`. Workflows do not run in parallel. If they would otherwise overlap, they are buffered to run sequentially. You can also use traditional cron strings, including all features that are supported by `CronSchedule` today, such as `@weekly` and other shorthands, `@every`, and `CRON_TZ`. $ tctl schedule create \ --schedule-id 'your-schedule-id' \ --cron '3 11 * * Fri' \ --workflow-id 'your-workflow-id' \ --task-queue 'your-task-queue' \ --workflow-type 'YourWorkflowType' Temporal Workflow Schedule Cron strings follow this format: ┌───────────── minute (0 - 59)│ ┌───────────── hour (0 - 23)│ │ ┌───────────── day of the month (1 - 31)│ │ │ ┌───────────── month (1 - 12)│ │ │ │ ┌───────────── day of the week (0 - 6) (Sunday to Saturday)│ │ │ │ │* * * * * Any combination of `--calendar`, `--interval`, and `--cron` is supported and Actions will happen at any of the specified times. If you use both `--time-zone` and also `CRON_TZ`, they must agree. See `tctl schedule create --help` for the full set of available options. delete[​](https://docs.temporal.io/tctl-v1/schedule#delete "Direct link to delete") ------------------------------------------------------------------------------------ A Schedule can be deleted. Deleting a Schedule **does not** affect any Workflows started by the Schedule. Workflow Executions started by Schedules can be cancelled or terminated using the same methods as any others. However, Workflow Executions started by a Schedule can be identified by the Search Attributes added to them and can be targeted by a [batch](https://docs.temporal.io/tctl-v1/batch/) command for termination. $ tctl schedule delete --schedule-id 'your-schedule-id' describe[​](https://docs.temporal.io/tctl-v1/schedule#describe "Direct link to describe") ------------------------------------------------------------------------------------------ Display the current Schedule configuration as well as extra information about past, current, and future Runs. tctl schedule describe --schedule-id 'your-schedule-id' Because the Schedule Spec is converted to canonical representations, the output might not be in the same form as it was input. list[​](https://docs.temporal.io/tctl-v1/schedule#list "Direct link to list") ------------------------------------------------------------------------------ tctl schedule list Because the Schedule Spec is converted to canonical representations, the output might not be in the same form as it was input. toggle[​](https://docs.temporal.io/tctl-v1/schedule#toggle "Direct link to toggle") ------------------------------------------------------------------------------------ $ tctl schedule toggle --schedule-id 'your-schedule-id' --pause --reason "paused because the database is down"$ tctl schedule toggle --schedule-id 'your-schedule-id' --unpause --reason "the database is back up" trigger[​](https://docs.temporal.io/tctl-v1/schedule#trigger "Direct link to trigger") --------------------------------------------------------------------------------------- Starting a Workflow Run immediately with a Schedule, regardless of its configured Spec, is a common use case. $ tctl schedule trigger --schedule-id 'your-schedule-id' Note that the action that it takes is subject to the Overlap Policy of the Schedule by default: if the overlap policy is `Skip` and a Workflow is already running, the triggered Action to start the next Workflow Run is skipped! Likewise, if the overlap policy is `BufferAll`, the triggered run is buffered behind one or more runs. If you really want it to run right now, you can override the overlap policy for this request: $ tctl schedule trigger --schedule-id 'your-schedule-id' --overlap-policy 'AllowAll' update[​](https://docs.temporal.io/tctl-v1/schedule#update "Direct link to update") ------------------------------------------------------------------------------------ Any part of the Schedule configuration can be updated at any time. `tctl schedule update` takes the same options as `tctl schedule create` and replaces the entire configuration of the schedule with what's provided. This means if you want to change just one value, you have to provide everything else again. * [backfill](https://docs.temporal.io/tctl-v1/schedule#backfill) * [create](https://docs.temporal.io/tctl-v1/schedule#create) * [delete](https://docs.temporal.io/tctl-v1/schedule#delete) * [describe](https://docs.temporal.io/tctl-v1/schedule#describe) * [list](https://docs.temporal.io/tctl-v1/schedule#list) * [toggle](https://docs.temporal.io/tctl-v1/schedule#toggle) * [trigger](https://docs.temporal.io/tctl-v1/schedule#trigger) * [update](https://docs.temporal.io/tctl-v1/schedule#update) --- # tctl v1.17 batch command reference | Temporal Platform Documentation [Skip to main content](https://docs.temporal.io/tctl-v1/batch#__docusaurus_skipToContent_fallback) On this page tctl is deprecated The tctl command line utility has been deprecated and is no longer actively supported. We recommend transitioning to [Temporal CLI](https://docs.temporal.io/cli) for continued use and access to new features. Thank you for being a valued part of the Temporal community. **How to run a tctl batch command.** A `tctl batch` command enables you to affect multiple existing [Workflow Executions](https://docs.temporal.io/workflow-execution) with a single command. A batch job runs in the background and affects Workflow Executions one at a time. Use [tctl batch start](https://docs.temporal.io/tctl-v1/batch#start) to start a batch job. note `tctl-v1` can run `batch` and `batch-v2` commands. When starting a batch job, you must provide a [List Filter](https://docs.temporal.io/list-filter) and the type of batch job that should occur. Batch jobs run in the background and affect Workflow Executions one at a time. The List Filter identifies the set of Workflow Executions to be affected by the batch job. The `tctl batch start` command shows you how many Workflow Executions will be affected by the batch job and asks you to confirm before proceeding. The batch type determines what other parameters you must provide and what is being affected. There are three types of batch jobs: * Signal: Send a Signal to the set of Workflow Executions that the List Filter specifies. * Cancel: Cancel the set of Workflow Executions that the List Filter specifies. * Terminate: Terminate the set of Workflow Executions that the List Filter specifies. A successfully started batch job returns a Job ID. You can use this Job ID in the `tctl batch describe` command, which describes the progress of a specific batch job. You can also use the Job ID to terminate the batch job itself. Terminating a batch job does not roll back the operations already performed by the batch job. ### tctl batch commands[​](https://docs.temporal.io/tctl-v1/batch#tctl-batch-commands "Direct link to tctl batch commands") * [tctl batch describe](https://docs.temporal.io/tctl-v1/batch#describe) * [tctl batch list](https://docs.temporal.io/tctl-v1/batch#list) * [tctl batch start](https://docs.temporal.io/tctl-v1/batch#start) * [tctl batch terminate](https://docs.temporal.io/tctl-v1/batch#terminate) start[​](https://docs.temporal.io/tctl-v1/batch#start "Direct link to start") ------------------------------------------------------------------------------ The `tctl batch start` command starts a batch job. `tctl batch start --query ` The following modifiers control the behavior of the command. ### `--query`[​](https://docs.temporal.io/tctl-v1/batch#--query "Direct link to --query") _Required modifier_ Specify the [Workflow Executions](https://docs.temporal.io/workflow-execution) that this batch job should operate. The SQL-like query of [Search Attributes](https://docs.temporal.io/search-attribute) is the same as used by the `tctl workflow list --query` command. Alias: `-q` **Example** tctl batch start --query ### `--reason`[​](https://docs.temporal.io/tctl-v1/batch#--reason "Direct link to --reason") Specify a reason for running this batch job. **Example** tctl batch start --query --reason ### `--batch_type`[​](https://docs.temporal.io/tctl-v1/batch#--batch_type "Direct link to --batch_type") Specify the operation that this batch job performs. The supported operations are `signal`, `cancel`, and `terminate`. **Example** tctl batch start --query --batch_type ### `--signal_name`[​](https://docs.temporal.io/tctl-v1/batch#--signal_name "Direct link to --signal_name") Specify the name of a [Signal](https://docs.temporal.io/sending-messages#sending-signals) . This modifier is required when `--batch_type` is `signal`. **Example** tctl batch start --query --batch_type signal --signal_name ### `--input`[​](https://docs.temporal.io/tctl-v1/batch#--input "Direct link to --input") Pass input for the [Signal](https://docs.temporal.io/sending-messages#sending-signals) . Input must be in JSON format. Alias: `-i` **Example** tctl batch start --query --input ### `--rps`[​](https://docs.temporal.io/tctl-v1/batch#--rps "Direct link to --rps") Specify RPS of processing. The default value is 50. **Example** tctl batch start --query --rps ### `--yes`[​](https://docs.temporal.io/tctl-v1/batch#--yes "Direct link to --yes") Disable the confirmation prompt. Alias: `y` **Example** tctl batch start --query --yes list[​](https://docs.temporal.io/tctl-v1/batch#list "Direct link to list") --------------------------------------------------------------------------- The `tctl batch list` command lists all batch jobs. `tctl batch list ` note `tctl-v1` can run `batch` and `batch-v2` commands. The following modifier controls the behavior of the command. ### \--pagesize[​](https://docs.temporal.io/tctl-v1/batch#--pagesize "Direct link to --pagesize") Specify the maximum number of batch jobs to list on a page. The default value is 30. **Example** tctl batch list --pagesize describe[​](https://docs.temporal.io/tctl-v1/batch#describe "Direct link to describe") --------------------------------------------------------------------------------------- The `tctl batch describe` command describes the progress of a batch job. `tctl batch describe --job_id ` note `tctl` can run `batch` and `batch-v2` commands. The following modifier controls the behavior of the command. ### \--job\_id[​](https://docs.temporal.io/tctl-v1/batch#--job_id "Direct link to --job_id") _Required modifier_ Specify the job ID of a batch job. **Example** tctl batch describe --job_id terminate[​](https://docs.temporal.io/tctl-v1/batch#terminate "Direct link to terminate") ------------------------------------------------------------------------------------------ The `tctl batch terminate` command terminates a batch job. `tctl batch terminate --job_id ` note `tctl-v1` can run `batch` and `batch-v2` commands. The following modifiers control the behavior of the command. ### `--job_id`[​](https://docs.temporal.io/tctl-v1/batch#--job_id-1 "Direct link to --job_id-1") _Required modifier_ Specify the job ID of a batch job. **Example** tctl batch terminate --job_id ### `--reason`[​](https://docs.temporal.io/tctl-v1/batch#--reason-1 "Direct link to --reason-1") Specify a reason for terminating this batch job. **Example** tctl batch terminate --job_id --reason * [tctl batch commands](https://docs.temporal.io/tctl-v1/batch#tctl-batch-commands) * [start](https://docs.temporal.io/tctl-v1/batch#start) * [`--query`](https://docs.temporal.io/tctl-v1/batch#--query) * [`--reason`](https://docs.temporal.io/tctl-v1/batch#--reason) * [`--batch_type`](https://docs.temporal.io/tctl-v1/batch#--batch_type) * [`--signal_name`](https://docs.temporal.io/tctl-v1/batch#--signal_name) * [`--input`](https://docs.temporal.io/tctl-v1/batch#--input) * [`--rps`](https://docs.temporal.io/tctl-v1/batch#--rps) * [`--yes`](https://docs.temporal.io/tctl-v1/batch#--yes) * [list](https://docs.temporal.io/tctl-v1/batch#list) * [\--pagesize](https://docs.temporal.io/tctl-v1/batch#--pagesize) * [describe](https://docs.temporal.io/tctl-v1/batch#describe) * [\--job\_id](https://docs.temporal.io/tctl-v1/batch#--job_id) * [terminate](https://docs.temporal.io/tctl-v1/batch#terminate) * [`--job_id`](https://docs.temporal.io/tctl-v1/batch#--job_id-1) * [`--reason`](https://docs.temporal.io/tctl-v1/batch#--reason-1) --- # Quickstarts | Temporal Platform Documentation [Skip to main content](https://docs.temporal.io/quickstarts#__docusaurus_skipToContent_fallback) Choose your language to get started quickly. [### Go\ \ Install the Go SDK and run a Hello World Workflow in Go.](https://docs.temporal.io/develop/go/set-up-your-local-go) [### Java\ \ Install the Java SDK and run a Hello World Workflow in Java.](https://docs.temporal.io/develop/java/set-up-your-local-java) [### PHP\ \ Install the PHP SDK and run a Hello World Workflow in PHP.](https://docs.temporal.io/develop/php/set-up-your-local-php) [### Python\ \ Install the Python SDK and run a Hello World Workflow in Python.](https://docs.temporal.io/develop/python/set-up-your-local-python) [### Ruby\ \ Install the Ruby SDK and run a Hello World Workflow in Ruby.](https://docs.temporal.io/develop/ruby/set-up-local-ruby) [### TypeScript\ \ Install the TypeScript SDK and run a Hello World Workflow in TypeScript.](https://docs.temporal.io/develop/typescript/set-up-your-local-typescript) [### .NET\ \ Install the .NET SDK and run a Hello World Workflow in C#.](https://docs.temporal.io/develop/dotnet/set-up-your-local-dotnet) --- # tctl v1.17 admin command reference | Temporal Platform Documentation [Skip to main content](https://docs.temporal.io/tctl-v1/admin#__docusaurus_skipToContent_fallback) On this page tctl is deprecated The tctl command line utility has been deprecated and is no longer actively supported. We recommend transitioning to [Temporal CLI](https://docs.temporal.io/cli) for continued use and access to new features. Thank you for being a valued part of the Temporal community. A `tctl admin` command allows the user to run admin operations. Modifiers: #### \--help[​](https://docs.temporal.io/tctl-v1/admin#--help "Direct link to --help") `tctl admin [--help | -h]` cluster[​](https://docs.temporal.io/tctl-v1/admin#cluster "Direct link to cluster") ------------------------------------------------------------------------------------ The `tctl admin cluster` command runs the administrator-level operations on a given Cluster. `tctl admin cluster command [command modifiers] [arguments...]` * [add\_search\_attributes](https://docs.temporal.io/tctl-v1/admin#add_search_attributes) * [remove\_search\_attributes](https://docs.temporal.io/tctl-v1/admin#remove_search_attributes) * [get\_search\_attributes](https://docs.temporal.io/tctl-v1/admin#get_search_attributes) * [describe](https://docs.temporal.io/tctl-v1/admin#describe) * [list](https://docs.temporal.io/tctl-v1/admin#list) * [upsert\_remote\_cluster](https://docs.temporal.io/tctl-v1/admin#upsert_remote_cluster) * [remove\_remote\_cluster](https://docs.temporal.io/tctl-v1/admin#upsert_remote_cluster) ### add\_search\_attributes[​](https://docs.temporal.io/tctl-v1/admin#add_search_attributes "Direct link to add_search_attributes") The `tctl admin cluster add-search-attributes` command allows Search Attributes to be added to a Cluster. Custom Search Attributes can be used to make a Cluster more identifiable. note Due to Elasticsearch limitations, you can only add new custom Search Attributes. Existing Search Attributes cannot be renamed or removed from the Elasticsearch index. Use this command to add custom Search Attributes to your Temporal Cluster: tctl admin cluster add-search-attributes --name --type note If you are adding custom Search Attributes to a Cluster running from the `docker-compose-es.yml` file in the [temporalio/docker-compose](https://github.com/temporalio/docker-compose) repo, make sure to increase the Docker memory to more than 6 GB. #### \--skip\_schema\_update[​](https://docs.temporal.io/tctl-v1/admin#--skip_schema_update "Direct link to --skip_schema_update") Allows the user to skip the Elasticsearch index schema update. note This will only register in metadata. #### \--name[​](https://docs.temporal.io/tctl-v1/admin#--name "Direct link to --name") The name of the Search Attribute to add. Names can have multiple values. Search Attribute names are case sensitive. #### \--type[​](https://docs.temporal.io/tctl-v1/admin#--type "Direct link to --type") The type of Search Attribute to add. Multiple values can be added at once. Values: Text, Keyword, Int, Double, Bool, Datetime ### describe[​](https://docs.temporal.io/tctl-v1/admin#describe "Direct link to describe") The `tctl admin cluster describe` command provides information for the current Cluster. The following modifier changes the behavior of the command: #### \--cluster\_value[​](https://docs.temporal.io/tctl-v1/admin#--cluster_value "Direct link to --cluster_value") The name of the remote Cluster within the current Cluster. This modifier is optional, and can default to the return of current Cluster information. ### get\_search\_attributes[​](https://docs.temporal.io/tctl-v1/admin#get_search_attributes "Direct link to get_search_attributes") The `tctl admin cluster get_search_attributes` command retrieves existing Search Attributes for a given Cluster. The following modifier will change the behavior of the command: #### \--print\_json[​](https://docs.temporal.io/tctl-v1/admin#--print_json "Direct link to --print_json") Prints the existing search attributes in JSON format. ### list[​](https://docs.temporal.io/tctl-v1/admin#list "Direct link to list") The `tctl admin cluster list` command lists Cluster information on the given Cluster. Default: 100 The modifier below changes the behavior of the command: #### \--pagesize[​](https://docs.temporal.io/tctl-v1/admin#--pagesize "Direct link to --pagesize") The size of the page that the list is printed on. ### remove\_remote\_cluster[​](https://docs.temporal.io/tctl-v1/admin#remove_remote_cluster "Direct link to remove_remote_cluster") The `tctl admin cluster remove_remote_cluster` command removes remote Cluster information on the given Cluster. The modifier below changes the behavior of the operation: #### \--cluster[​](https://docs.temporal.io/tctl-v1/admin#--cluster "Direct link to --cluster") The name of the remote Cluster to remove. ### remove\_search\_attributes[​](https://docs.temporal.io/tctl-v1/admin#remove_search_attributes "Direct link to remove_search_attributes") > The Temporal tctl documentation covers version 1.17 of the Temporal CLI. The `tctl admin cluster remove-search-attributes` command removes custom Search Attribute metadata from a Cluster. This operation has no effect on Elasticsearch index schema. Use the following command to remove a [Search Attribute](https://docs.temporal.io/search-attribute) from a Cluster's metadata: tctl admin cluster remove-search-attributes --name Only custom Search Attributes can be removed from a Cluster's metadata. Default Search Attributes cannot be removed. Removing a Search Attribute removes it from the Cluster's metadata but does not remove it from the Elasticsearch index. This means that the Search Attribute can be added back later as the same type. After a Search Attribute has been added to the Elasticsearch index, it cannot be changed. The following modifier changes the behavior of the operation: #### \--name[​](https://docs.temporal.io/tctl-v1/admin#--name-1 "Direct link to --name") Name of the Search Attribute to remove. ### upsert\_remote\_cluster[​](https://docs.temporal.io/tctl-v1/admin#upsert_remote_cluster "Direct link to upsert_remote_cluster") The `tctl admin cluster upsert_remote_cluster` command adds or updates remote Cluster information in the current Cluster. #### \--frontend\_address[​](https://docs.temporal.io/tctl-v1/admin#--frontend_address "Direct link to --frontend_address") The remote Cluster frontend address. #### \--enable\_connection[​](https://docs.temporal.io/tctl-v1/admin#--enable_connection "Direct link to --enable_connection") Enables remote Cluster connection. db[​](https://docs.temporal.io/tctl-v1/admin#db "Direct link to db") --------------------------------------------------------------------- The `tctl admin db` command runs administrator-level operations on a given database. ### Usage[​](https://docs.temporal.io/tctl-v1/admin#usage "Direct link to Usage") `tctl admin db command [command modifiers] [arguments...]` ### Commands[​](https://docs.temporal.io/tctl-v1/admin#commands "Direct link to Commands") * [tctl admin db scan](https://docs.temporal.io/tctl-v1/admin#scan) * [tctl admin db clean](https://docs.temporal.io/tctl-v1/admin#clean) ### clean[​](https://docs.temporal.io/tctl-v1/admin#clean "Direct link to clean") The `tctl admin db clean` command cleans corrupted [Workflow Executions](https://docs.temporal.io/workflow-execution) from the targeted database. The modifiers below change the behavior of the command. #### \--db\_engine[​](https://docs.temporal.io/tctl-v1/admin#--db_engine "Direct link to --db_engine") Type of DB engine to use Default: `cassandra` Value: `cassandra` | `mysql` | `postgres` #### \--db\_address[​](https://docs.temporal.io/tctl-v1/admin#--db_address "Direct link to --db_address") Persistence address for the database. Default: 127.0.0.1 #### \--db\_port[​](https://docs.temporal.io/tctl-v1/admin#--db_port "Direct link to --db_port") Persistence port for the DB. Default: 9042 #### \--username[​](https://docs.temporal.io/tctl-v1/admin#--username "Direct link to --username") Database username. #### \--password[​](https://docs.temporal.io/tctl-v1/admin#--password "Direct link to --password") Database password. #### \--keyspace[​](https://docs.temporal.io/tctl-v1/admin#--keyspace "Direct link to --keyspace") Database keyspace Default: "temporal" #### \--input\_directory[​](https://docs.temporal.io/tctl-v1/admin#--input_directory "Direct link to --input_directory") The directory which contains the corrupted [Workflow Execution](https://docs.temporal.io/workflow-execution) files from running [`scan`](https://docs.temporal.io/tctl-v1/admin#scan) . #### \--lower\_shard\_bound[​](https://docs.temporal.io/tctl-v1/admin#--lower_shard_bound "Direct link to --lower_shard_bound") The minimum amount (inclusive) of corrupt shards to handle. Default: 0 #### \--upper\_shard\_bound[​](https://docs.temporal.io/tctl-v1/admin#--upper_shard_bound "Direct link to --upper_shard_bound") The maximum amount (exclusive) of corrupt shards to handle. Default: 16384 #### \--starting\_rps[​](https://docs.temporal.io/tctl-v1/admin#--starting_rps "Direct link to --starting_rps") starting rps of database queries. Default: 100 #### \--rps[​](https://docs.temporal.io/tctl-v1/admin#--rps "Direct link to --rps") Target rps of database queries. Default: 7000 #### \--concurrency[​](https://docs.temporal.io/tctl-v1/admin#--concurrency "Direct link to --concurrency") Number of threads to handle a scan. Default: 1000 #### \--report\_rate[​](https://docs.temporal.io/tctl-v1/admin#--report_rate "Direct link to --report_rate") The number of shards handled between each emittance of progress. Default: 10 note Enable `--tls` before using any of the following modifiers. #### \--tls\_cert\_path[​](https://docs.temporal.io/tctl-v1/admin#--tls_cert_path "Direct link to --tls_cert_path") Where the tls client cert is located. #### \--tls\_key\_path[​](https://docs.temporal.io/tctl-v1/admin#--tls_key_path "Direct link to --tls_key_path") Where the tls key is located. #### \--tls\_ca\_path[​](https://docs.temporal.io/tctl-v1/admin#--tls_ca_path "Direct link to --tls_ca_path") Where the tls ca is located. #### \--tls\_server\_name[​](https://docs.temporal.io/tctl-v1/admin#--tls_server_name "Direct link to --tls_server_name") The name of the Db tls server. #### \--tls\_disable\_host\_verification[​](https://docs.temporal.io/tctl-v1/admin#--tls_disable_host_verification "Direct link to --tls_disable_host_verification") Disables verification of the DB tls hostname and server cert. ### scan[​](https://docs.temporal.io/tctl-v1/admin#scan "Direct link to scan") The `tctl admin db scan` command scans concrete Workflow Executions in a given database, and detects corrupted ones. #### \--db\_engine[​](https://docs.temporal.io/tctl-v1/admin#--db_engine-1 "Direct link to --db_engine") Type of DB engine to use Default: `cassandra` Value: `cassandra` | `mysql` | `postgres` #### \--db\_address[​](https://docs.temporal.io/tctl-v1/admin#--db_address-1 "Direct link to --db_address") Persistence address for the DB. Default: 127.0.0.1 #### \--db\_port[​](https://docs.temporal.io/tctl-v1/admin#--db_port-1 "Direct link to --db_port") Persistence port for the DB. Default: 9042 #### \--username[​](https://docs.temporal.io/tctl-v1/admin#--username-1 "Direct link to --username") DB username. #### \--password[​](https://docs.temporal.io/tctl-v1/admin#--password-1 "Direct link to --password") DB password. #### \--keyspace[​](https://docs.temporal.io/tctl-v1/admin#--keyspace-1 "Direct link to --keyspace") DB keyspace Default: "temporal" #### \--lower\_shard\_bound value[​](https://docs.temporal.io/tctl-v1/admin#--lower_shard_bound-value "Direct link to --lower_shard_bound value") The minimum amount (inclusive) of corrupt shards to handle. Default: 0 #### \--upper\_shard\_bound[​](https://docs.temporal.io/tctl-v1/admin#--upper_shard_bound-1 "Direct link to --upper_shard_bound") The maximum amount (exclusive) of corrupt shards to handle. Default: 16384 #### \--starting\_rps[​](https://docs.temporal.io/tctl-v1/admin#--starting_rps-1 "Direct link to --starting_rps") starting rps of database queries. Default: 100 #### \--rps value[​](https://docs.temporal.io/tctl-v1/admin#--rps-value "Direct link to --rps value") Target rps of database queries. Default: 7000 #### \--pagesize[​](https://docs.temporal.io/tctl-v1/admin#--pagesize-1 "Direct link to --pagesize") The size of the page used to query database executions. Default: 500 #### \--concurrency[​](https://docs.temporal.io/tctl-v1/admin#--concurrency-1 "Direct link to --concurrency") Number of threads to handle a scan. Default: 1000 #### \--report\_rate[​](https://docs.temporal.io/tctl-v1/admin#--report_rate-1 "Direct link to --report_rate") The number of shards handled between each emittance of progress. Default: 10 #### \--tls[​](https://docs.temporal.io/tctl-v1/admin#--tls "Direct link to --tls") Enable TLS over the DB connection. note Enable `--tls` before using any of the following modifiers. #### \--tls\_cert\_path[​](https://docs.temporal.io/tctl-v1/admin#--tls_cert_path-1 "Direct link to --tls_cert_path") Where the tls client cert is located. #### \--tls\_key\_path[​](https://docs.temporal.io/tctl-v1/admin#--tls_key_path-1 "Direct link to --tls_key_path") Where the tls key is located. #### \--tls\_ca\_path[​](https://docs.temporal.io/tctl-v1/admin#--tls_ca_path-1 "Direct link to --tls_ca_path") Where the tls ca is located. #### \--tls\_server\_name[​](https://docs.temporal.io/tctl-v1/admin#--tls_server_name-1 "Direct link to --tls_server_name") The name of the Db tls server. #### \--tls\_disable\_host\_verification[​](https://docs.temporal.io/tctl-v1/admin#--tls_disable_host_verification-1 "Direct link to --tls_disable_host_verification") Disables verification of the DB tls hostname and server cert. decode[​](https://docs.temporal.io/tctl-v1/admin#decode "Direct link to decode") --------------------------------------------------------------------------------- The `tctl admin decode` command allows the user to decode payloads sent and received from executed Activities. `tctl admin decode command [command modifiers] [arguments...]` * [proto](https://docs.temporal.io/tctl-v1/admin#proto) * [base64](https://docs.temporal.io/tctl-v1/admin#base64) ### base64[​](https://docs.temporal.io/tctl-v1/admin#base64 "Direct link to base64") The `tctl admin decode base64` command decodes base64 Payloads. #### \--base64\_data[​](https://docs.temporal.io/tctl-v1/admin#--base64_data "Direct link to --base64_data") Decoded data in base64 format. #### \--base64\_file[​](https://docs.temporal.io/tctl-v1/admin#--base64_file "Direct link to --base64_file") Creates a file with data in base64 format. ### proto[​](https://docs.temporal.io/tctl-v1/admin#proto "Direct link to proto") The `tctl admin decode proto` command decodes the Payload to proto format. #### \--type[​](https://docs.temporal.io/tctl-v1/admin#--type-1 "Direct link to --type") The full name of the proto type to decode the Payload to. #### \--hex\_data[​](https://docs.temporal.io/tctl-v1/admin#--hex_data "Direct link to --hex_data") Decodes the data to hex format. #### \--hex\_file[​](https://docs.temporal.io/tctl-v1/admin#--hex_file "Direct link to --hex_file") Creates a file with the decoded hex data. #### \--binary\_file[​](https://docs.temporal.io/tctl-v1/admin#--binary_file "Direct link to --binary_file") Creates a file with the decoded binary data. dlq[​](https://docs.temporal.io/tctl-v1/admin#dlq "Direct link to dlq") ------------------------------------------------------------------------ The `tctl admin dlq` commands run admin operations on a given dead-letter queue (DLQ). `tctl admin dlq command [command modifiers] [arguments...]` * [tctl admin dlq read](https://docs.temporal.io/tctl-v1/admin#read) * [tctl admin dlq purge](https://docs.temporal.io/tctl-v1/admin#purge) * [tctl admin dlq merge](https://docs.temporal.io/tctl-v1/admin#merge) ### merge[​](https://docs.temporal.io/tctl-v1/admin#merge "Direct link to merge") The `tctl admin dlq merge` command allows dead-letter queue (DLQ) messages to be merged. The messages must have TaskIds with an equal or lesser value than the given TaskId. #### \--dlq\_type[​](https://docs.temporal.io/tctl-v1/admin#--dlq_type "Direct link to --dlq_type") The type of DLQ to manage. Options: namespace, history #### \--cluster[​](https://docs.temporal.io/tctl-v1/admin#--cluster-1 "Direct link to --cluster") Source cluster for the DLQ. #### \--shard\_id[​](https://docs.temporal.io/tctl-v1/admin#--shard_id "Direct link to --shard_id") ShardId provided for the command. #### \--last\_message\_id[​](https://docs.temporal.io/tctl-v1/admin#--last_message_id "Direct link to --last_message_id") Identifies the last read message. Default: 0 ### purge[​](https://docs.temporal.io/tctl-v1/admin#purge "Direct link to purge") The `tctl admin dlq purge` command deletes DLQ messages that have a Task Id equal to or less than the provided Task Id. #### \--dlq\_type[​](https://docs.temporal.io/tctl-v1/admin#--dlq_type-1 "Direct link to --dlq_type") The type of DLQ to manage. Options: namespace, history #### \--cluster[​](https://docs.temporal.io/tctl-v1/admin#--cluster-2 "Direct link to --cluster") Source cluster for the DLQ. #### \--shard\_id[​](https://docs.temporal.io/tctl-v1/admin#--shard_id-1 "Direct link to --shard_id") ShardId provided for the command. #### \--last\_message\_id[​](https://docs.temporal.io/tctl-v1/admin#--last_message_id-1 "Direct link to --last_message_id") Identifies the last read message. Default: 0 ### read[​](https://docs.temporal.io/tctl-v1/admin#read "Direct link to read") The `tctl admin dlq read` command reads out messages from the dead-letter queue (DLQ). * * * #### \--dlq\_type[​](https://docs.temporal.io/tctl-v1/admin#--dlq_type-2 "Direct link to --dlq_type") The type of DLQ to manage. Options: namespace, history #### \--cluster[​](https://docs.temporal.io/tctl-v1/admin#--cluster-3 "Direct link to --cluster") Source cluster for the DLQ. #### \--shard\_id[​](https://docs.temporal.io/tctl-v1/admin#--shard_id-2 "Direct link to --shard_id") ShardId provided for the command. #### \--max\_message\_count[​](https://docs.temporal.io/tctl-v1/admin#--max_message_count "Direct link to --max_message_count") The maximum number of messages to fetch. Default: 0 #### \--last\_message\_id[​](https://docs.temporal.io/tctl-v1/admin#--last_message_id-2 "Direct link to --last_message_id") Identifies the last read message. Default: 0 #### \--output\_filename[​](https://docs.temporal.io/tctl-v1/admin#--output_filename "Direct link to --output_filename") Provides a file to write output to. Output is written to stdout on default. history\_host[​](https://docs.temporal.io/tctl-v1/admin#history_host "Direct link to history_host") ---------------------------------------------------------------------------------------------------- The `tctl admin history_host` command runs an admin-level operation on the history host. Usage[​](https://docs.temporal.io/tctl-v1/admin#usage-1 "Direct link to Usage") -------------------------------------------------------------------------------- `tctl admin history_host command [command options] [arguments...]` Commands[​](https://docs.temporal.io/tctl-v1/admin#commands-1 "Direct link to Commands") ----------------------------------------------------------------------------------------- * [tctl admin history\_host describe](https://docs.temporal.io/tctl-v1/admin#describe) * [tctl admin history\_host get\_shardid](https://docs.temporal.io/tctl-v1/admin#get_shardid) ### describe[​](https://docs.temporal.io/tctl-v1/admin#describe-1 "Direct link to describe") The `tctl admin history_host describe` command describes the internal information of history host. The following modifiers change the behavior of the command. #### \--workflow\_id[​](https://docs.temporal.io/tctl-v1/admin#--workflow_id "Direct link to --workflow_id") Alias: `-w` The WorkflowId of the Workflow whose history host is to be described. #### \--history\_address[​](https://docs.temporal.io/tctl-v1/admin#--history_address "Direct link to --history_address") The history address of the history host. #### \--shard\_id[​](https://docs.temporal.io/tctl-v1/admin#--shard_id-3 "Direct link to --shard_id") The Id of the shard that belongs to the history host. #### \--print\_full[​](https://docs.temporal.io/tctl-v1/admin#--print_full "Direct link to --print_full") Print a full and detailed summary of the history host. ### get\_shardid[​](https://docs.temporal.io/tctl-v1/admin#get_shardid "Direct link to get_shardid") The `tctl admin history_host get_shardid` command gets the `shardId` for a given `namespaceId` and `workflowId`. The following modifiers change the behavior of this command. #### \--namespace\_id[​](https://docs.temporal.io/tctl-v1/admin#--namespace_id "Direct link to --namespace_id") The `namespaceId` of the history host where we're getting the `shardId`. #### \--workflow\_id[​](https://docs.temporal.io/tctl-v1/admin#--workflow_id-1 "Direct link to --workflow_id") Alias: `-w` The WorkflowId of the history host where we're getting the shardId. #### \--number\_of\_shards[​](https://docs.temporal.io/tctl-v1/admin#--number_of_shards "Direct link to --number_of_shards") The total amount of shards for the Temporal Cluster. Default: 0 membership[​](https://docs.temporal.io/tctl-v1/admin#membership "Direct link to membership") --------------------------------------------------------------------------------------------- The `tctl admin membership` command allows admin operations to be run on membership items. ### Usage[​](https://docs.temporal.io/tctl-v1/admin#usage-2 "Direct link to Usage") `tctl admin membership command [command modifiers] [arguments...]` ### Commands[​](https://docs.temporal.io/tctl-v1/admin#commands-2 "Direct link to Commands") * [list\_gossip](https://docs.temporal.io/tctl-v1/admin#list_gossip) * [list\_db](https://docs.temporal.io/tctl-v1/admin#list_db) ### list\_db[​](https://docs.temporal.io/tctl-v1/admin#list_db "Direct link to list_db") The `tctl admin membership list_db` command lists the Cluster items in a targeted membership. The following modifiers change the behavior of the command. #### \--heartbeated\_within[​](https://docs.temporal.io/tctl-v1/admin#--heartbeated_within "Direct link to --heartbeated_within") Filters the list by last Heartbeat time. #### \--role[​](https://docs.temporal.io/tctl-v1/admin#--role "Direct link to --role") Filters the results by membership role. Default: all Values: all, frontend, history, matching, worker ### list\_gossip[​](https://docs.temporal.io/tctl-v1/admin#list_gossip "Direct link to list_gossip") The `tctl admin membership list_gossip` command lists the ringpop membership items present on the targeted membership. The following modifier changes the behavior of the command: #### \--role value[​](https://docs.temporal.io/tctl-v1/admin#--role-value "Direct link to --role value") Filters the results by membership role Default: all Values: all, frontend, history, matching, worker shard[​](https://docs.temporal.io/tctl-v1/admin#shard "Direct link to shard") ------------------------------------------------------------------------------ The `tctl admin shard` commands enable admin-level operations on a specified shard. #### tctl admin shard commands[​](https://docs.temporal.io/tctl-v1/admin#tctl-admin-shard-commands "Direct link to tctl admin shard commands") * [describe](https://docs.temporal.io/tctl-v1/admin#describe) * [describe\_task](https://docs.temporal.io/tctl-v1/admin#describe_task) * [list\_tasks](https://docs.temporal.io/tctl-v1/admin#list_tasks) * [close\_shard](https://docs.temporal.io/tctl-v1/admin#close_shard) * [remove\_task](https://docs.temporal.io/tctl-v1/admin#remove_task) ### close\_shard[​](https://docs.temporal.io/tctl-v1/admin#close_shard "Direct link to close_shard") The `tctl admin shard close_shard` command closes a shard with an Id that corresponds to the value given in the command. `tctl admin shard close_shard [command options] [arguments...]` The modifier below will change the behavior and output of the command. #### \--shard\_id value[​](https://docs.temporal.io/tctl-v1/admin#--shard_id-value "Direct link to --shard_id value") ShardId managed by the Temporal Cluster. ### describe\_task[​](https://docs.temporal.io/tctl-v1/admin#describe_task "Direct link to describe_task") The `tctl admin shard describe_task` command describes a specified Task's Task Id, Task type, shard Id, and task visibility timestamp. The modifiers below control the output and behavior of the command. Enter all modifiers after the command as such: `tctl admin shard describe_task ` #### \--db\_engine[​](https://docs.temporal.io/tctl-v1/admin#--db_engine-2 "Direct link to --db_engine") The type of database (DB) engine for the shard to use. Default: "cassandra" Values: "cassandra", "mysql", "postgres" #### \--db\_address[​](https://docs.temporal.io/tctl-v1/admin#--db_address-2 "Direct link to --db_address") Persistence address for the database. Default: 127.0.0.1 #### \--db\_port[​](https://docs.temporal.io/tctl-v1/admin#--db_port-2 "Direct link to --db_port") Persistence port for the database. Default: 9042 #### \--username[​](https://docs.temporal.io/tctl-v1/admin#--username-2 "Direct link to --username") Username entered into the database. #### \--password[​](https://docs.temporal.io/tctl-v1/admin#--password-2 "Direct link to --password") Password entered into the database. #### \--keyspace[​](https://docs.temporal.io/tctl-v1/admin#--keyspace-2 "Direct link to --keyspace") Keyspace for the database. default: "temporal" #### \--tls[​](https://docs.temporal.io/tctl-v1/admin#--tls-1 "Direct link to --tls") Enables TLS over the database connection. #### \--tls\_cert\_path[​](https://docs.temporal.io/tctl-v1/admin#--tls_cert_path-2 "Direct link to --tls_cert_path") DB tls client cert path. Note: tls must be enabled #### \--tls\_server\_name[​](https://docs.temporal.io/tctl-v1/admin#--tls_server_name-2 "Direct link to --tls_server_name") DB tls server name Note: tls must be enabled #### \--tls\_disable\_host\_verification[​](https://docs.temporal.io/tctl-v1/admin#--tls_disable_host_verification-2 "Direct link to --tls_disable_host_verification") DB tls verify hostname and server cert Note: tls must be enabled #### \--shard\_id[​](https://docs.temporal.io/tctl-v1/admin#--shard_id-4 "Direct link to --shard_id") Identifies the specified shard. Default: 0 #### \--task\_id[​](https://docs.temporal.io/tctl-v1/admin#--task_id "Direct link to --task_id") Describes the task. Default: 0 #### \--task\_type[​](https://docs.temporal.io/tctl-v1/admin#--task_type "Direct link to --task_type") The kind of Task that is targeted within a shard. Default: transfer Values: transfer, timer, replication #### \--task\_timestamp[​](https://docs.temporal.io/tctl-v1/admin#--task_timestamp "Direct link to --task_timestamp") Task visibility timestamp in nanoseconds Default: 0 #### \--target\_cluster[​](https://docs.temporal.io/tctl-v1/admin#--target_cluster "Direct link to --target_cluster") Temporal cluster for the shard to use. Default: "active" ### describe[​](https://docs.temporal.io/tctl-v1/admin#describe-2 "Direct link to describe") The `tctl admin shard describe` command shows the Id for the specified shard. The modifier below controls the behavior of the command. #### \--shard\_id value[​](https://docs.temporal.io/tctl-v1/admin#--shard_id-value-1 "Direct link to --shard_id value") The Id of the shard to describe Default: 0 ### list\_tasks[​](https://docs.temporal.io/tctl-v1/admin#list_tasks "Direct link to list_tasks") The `tctl admin shard list_tasks` command will list the Tasks available for a given shard Id and Task type. The modifiers below affect the output and behavior of the command. #### \--more[​](https://docs.temporal.io/tctl-v1/admin#--more "Direct link to --more") Lists more pages of list tasks. The default setting is to list one page of 10 list tasks. #### \--pagesize value[​](https://docs.temporal.io/tctl-v1/admin#--pagesize-value "Direct link to --pagesize value") The size of the result page. Default: 10 #### \--target\_cluster value[​](https://docs.temporal.io/tctl-v1/admin#--target_cluster-value "Direct link to --target_cluster value") Temporal cluster to use. Default: "active" #### \--shard\_id value[​](https://docs.temporal.io/tctl-v1/admin#--shard_id-value-2 "Direct link to --shard_id value") The ID of the shard Default: 0 #### \--task\_type value[​](https://docs.temporal.io/tctl-v1/admin#--task_type-value "Direct link to --task_type value") The type of Task. Default: transfer Values: transfer, timer, replication, visibility #### \--min\_visibility\_ts value[​](https://docs.temporal.io/tctl-v1/admin#--min_visibility_ts-value "Direct link to --min_visibility_ts value") The minimum value that can be set as a Task Visibility timestamp. Supported formats include: * '2006-01-02T15:04:05+07:00' * Raw UnixNano * Time range (N-duration), where 0 < N < 1000000 and duration (full-notation/short-notation) can be: * second/s * minute/m * week/w * month/m * year/y #### \--max\_visibility\_ts value[​](https://docs.temporal.io/tctl-v1/admin#--max_visibility_ts-value "Direct link to --max_visibility_ts value") The maximum value that can be set as a Task Visibility timestamp. Supported formats: * '2006-01-02T15:04:05+07:00' * Raw UnixNano * Time range (N-duration), where 0 < N < 1000000 and duration (full-notation/short-notation) can be: * second/s * minute/m * week/w * month/m * year/y ### remove\_task[​](https://docs.temporal.io/tctl-v1/admin#remove_task "Direct link to remove_task") The `tctl admin shard remove_task` command removes a Task from the shard. `tctl admin shard remove_task [command options] [arguments...]` The Task removed must have values that matches what is given in the command line. The modifiers below change the behavior of the command. #### \--shard\_id value[​](https://docs.temporal.io/tctl-v1/admin#--shard_id-value-3 "Direct link to --shard_id value") The shardId for the Task to be removed. Default: 0 #### \--task\_id value[​](https://docs.temporal.io/tctl-v1/admin#--task_id-value "Direct link to --task_id value") The taskId for the Task to be removed. Default: 0 #### \--task\_type value[​](https://docs.temporal.io/tctl-v1/admin#--task_type-value-1 "Direct link to --task_type value") The type of Task to remove. Default: transfer Values: transfer, timer, replication #### \--task\_timestamp value[​](https://docs.temporal.io/tctl-v1/admin#--task_timestamp-value "Direct link to --task_timestamp value") The task visibility timestamp, given in nanoseconds. Default: 0 workflow[​](https://docs.temporal.io/tctl-v1/admin#workflow "Direct link to workflow") --------------------------------------------------------------------------------------- The `tctl admin workflow` commands enable administrator-level operations on Workflow Executions. `tctl admin workflow command [modifiers] [arguments...]` * [show](https://docs.temporal.io/tctl-v1/admin#show) * [describe](https://docs.temporal.io/tctl-v1/admin#describe) * [refresh\_tasks](https://docs.temporal.io/tctl-v1/admin#refresh_tasks) * [delete](https://docs.temporal.io/tctl-v1/admin#delete) ### delete[​](https://docs.temporal.io/tctl-v1/admin#delete "Direct link to delete") The `tctl admin workflow delete` command deletes the current [Workflow Execution](https://docs.temporal.io/workflow-execution) and the mutableState record. #### \--db\_engine value[​](https://docs.temporal.io/tctl-v1/admin#--db_engine-value "Direct link to --db_engine value") The type of database (DB) engine to use. Default: "cassandra" Values: "cassandra", "mysql", "postgres" #### \--db\_address value[​](https://docs.temporal.io/tctl-v1/admin#--db_address-value "Direct link to --db_address value") Persistence address for the database. Default: 127.0.0.1 #### \--db\_port value[​](https://docs.temporal.io/tctl-v1/admin#--db_port-value "Direct link to --db_port value") Persistence port for the database. Default: 9042 #### \--username value[​](https://docs.temporal.io/tctl-v1/admin#--username-value "Direct link to --username value") Username entered into the database. #### \--password value[​](https://docs.temporal.io/tctl-v1/admin#--password-value "Direct link to --password value") Password entered into the database. #### \--keyspace value[​](https://docs.temporal.io/tctl-v1/admin#--keyspace-value "Direct link to --keyspace value") Keyspace for the database. default: "temporal" #### \--url value[​](https://docs.temporal.io/tctl-v1/admin#--url-value "Direct link to --url value") URL of the Elasticsearch cluster. Default: "[http://127.0.0.1:9200](http://127.0.0.1:9200/) " #### \--es-username value[​](https://docs.temporal.io/tctl-v1/admin#--es-username-value "Direct link to --es-username value") Username for the Elasticsearch cluster. #### \--es-password value[​](https://docs.temporal.io/tctl-v1/admin#--es-password-value "Direct link to --es-password value") Password for the Elasticsearch cluster. #### \--version value[​](https://docs.temporal.io/tctl-v1/admin#--version-value "Direct link to --version value") The version of the Elasticsearch cluster for the Workflow. Default: v7 Values: v6, v7 #### \--index value[​](https://docs.temporal.io/tctl-v1/admin#--index-value "Direct link to --index value") Elasticsearch index name. #### \--workflow\_id value[​](https://docs.temporal.io/tctl-v1/admin#--workflow_id-value "Direct link to --workflow_id value") Alias: `-w` The Id of the current Workflow. #### \--run\_id value[​](https://docs.temporal.io/tctl-v1/admin#--run_id-value "Direct link to --run_id value") Alias: `-r` The Id of the current run. #### \--skip\_errors[​](https://docs.temporal.io/tctl-v1/admin#--skip_errors "Direct link to --skip_errors") Skip any errors that occur in the Workflow Execution. #### \--tls[​](https://docs.temporal.io/tctl-v1/admin#--tls-2 "Direct link to --tls") Enables TLS over the database connection. note TLS must be enabled to use the following modifiers. #### \--tls\_cert\_path value[​](https://docs.temporal.io/tctl-v1/admin#--tls_cert_path-value "Direct link to --tls_cert_path value") DB tls client cert path. Note: tls must be enabled #### \--tls\_key\_path value[​](https://docs.temporal.io/tctl-v1/admin#--tls_key_path-value "Direct link to --tls_key_path value") DB tls client key path Note: tls must be enabled #### \--tls\_ca\_path value[​](https://docs.temporal.io/tctl-v1/admin#--tls_ca_path-value "Direct link to --tls_ca_path value") DB tls client ca path Note: tls must be enabled #### \--tls\_server\_name value[​](https://docs.temporal.io/tctl-v1/admin#--tls_server_name-value "Direct link to --tls_server_name value") DB tls server name Note: tls must be enabled #### \--tls\_disable\_host\_verification[​](https://docs.temporal.io/tctl-v1/admin#--tls_disable_host_verification-3 "Direct link to --tls_disable_host_verification") DB tls verify hostname and server cert Note: tls must be enabled describe[​](https://docs.temporal.io/tctl-v1/admin#describe-3 "Direct link to describe") ----------------------------------------------------------------------------------------- The `tctl admin workflow describe` command describes internal information of the current [Workflow Execution](https://docs.temporal.io/workflow-execution) . #### \--workflow\_id value[​](https://docs.temporal.io/tctl-v1/admin#--workflow_id-value-1 "Direct link to --workflow_id value") Alias: `-w` The Id of the current Workflow. #### \--run\_id value[​](https://docs.temporal.io/tctl-v1/admin#--run_id-value-1 "Direct link to --run_id value") Alias: `-r` The Id of the current run. refresh\_tasks[​](https://docs.temporal.io/tctl-v1/admin#refresh_tasks "Direct link to refresh_tasks") ------------------------------------------------------------------------------------------------------- The `tctl admin workflow refresh_tasks` command updates all [Tasks](https://docs.temporal.io/tasks#task) in a [Workflow](https://docs.temporal.io/workflows) , provided that the command can fetch new information for Tasks. #### \--workflow\_id value[​](https://docs.temporal.io/tctl-v1/admin#--workflow_id-value-2 "Direct link to --workflow_id value") Alias: `-w` The Id of the current Workflow. #### \--run\_id value[​](https://docs.temporal.io/tctl-v1/admin#--run_id-value-2 "Direct link to --run_id value") Alias: `-r` The Id of the current run. show[​](https://docs.temporal.io/tctl-v1/admin#show "Direct link to show") --------------------------------------------------------------------------- The `tctl admin workflow show` command displays Workflow history from the database. #### \--workflow\_id value[​](https://docs.temporal.io/tctl-v1/admin#--workflow_id-value-3 "Direct link to --workflow_id value") Alias: `-w` The current Workflow. #### \--run\_id value[​](https://docs.temporal.io/tctl-v1/admin#--run_id-value-3 "Direct link to --run_id value") Alias: `-r` The current RunId. #### \--min\_event\_id value[​](https://docs.temporal.io/tctl-v1/admin#--min_event_id-value "Direct link to --min_event_id value") The minimum Event Id to include in the history. Default: 0 #### \--max\_event\_id value[​](https://docs.temporal.io/tctl-v1/admin#--max_event_id-value "Direct link to --max_event_id value") The maximum Event Id to include in the history. Default: 0 #### \--min\_event\_version value[​](https://docs.temporal.io/tctl-v1/admin#--min_event_version-value "Direct link to --min_event_version value") The start Event version to be included in the history. Default: 0 #### \--max\_event\_version value[​](https://docs.temporal.io/tctl-v1/admin#--max_event_version-value "Direct link to --max_event_version value") The end Event version to be included in the history. Default: 0 #### \--output\_filename value[​](https://docs.temporal.io/tctl-v1/admin#--output_filename-value "Direct link to --output_filename value") The file where the output is sent to. * [\--help](https://docs.temporal.io/tctl-v1/admin#--help) * [cluster](https://docs.temporal.io/tctl-v1/admin#cluster) * [add\_search\_attributes](https://docs.temporal.io/tctl-v1/admin#add_search_attributes) * [\--skip\_schema\_update](https://docs.temporal.io/tctl-v1/admin#--skip_schema_update) * [\--name](https://docs.temporal.io/tctl-v1/admin#--name) * [\--type](https://docs.temporal.io/tctl-v1/admin#--type) * [describe](https://docs.temporal.io/tctl-v1/admin#describe) * [\--cluster\_value](https://docs.temporal.io/tctl-v1/admin#--cluster_value) * [get\_search\_attributes](https://docs.temporal.io/tctl-v1/admin#get_search_attributes) * [\--print\_json](https://docs.temporal.io/tctl-v1/admin#--print_json) * [list](https://docs.temporal.io/tctl-v1/admin#list) * [\--pagesize](https://docs.temporal.io/tctl-v1/admin#--pagesize) * [remove\_remote\_cluster](https://docs.temporal.io/tctl-v1/admin#remove_remote_cluster) * [\--cluster](https://docs.temporal.io/tctl-v1/admin#--cluster) * [remove\_search\_attributes](https://docs.temporal.io/tctl-v1/admin#remove_search_attributes) * [\--name](https://docs.temporal.io/tctl-v1/admin#--name-1) * [upsert\_remote\_cluster](https://docs.temporal.io/tctl-v1/admin#upsert_remote_cluster) * [\--frontend\_address](https://docs.temporal.io/tctl-v1/admin#--frontend_address) * [\--enable\_connection](https://docs.temporal.io/tctl-v1/admin#--enable_connection) * [db](https://docs.temporal.io/tctl-v1/admin#db) * [Usage](https://docs.temporal.io/tctl-v1/admin#usage) * [Commands](https://docs.temporal.io/tctl-v1/admin#commands) * [clean](https://docs.temporal.io/tctl-v1/admin#clean) * [\--db\_engine](https://docs.temporal.io/tctl-v1/admin#--db_engine) * [\--db\_address](https://docs.temporal.io/tctl-v1/admin#--db_address) * [\--db\_port](https://docs.temporal.io/tctl-v1/admin#--db_port) * [\--username](https://docs.temporal.io/tctl-v1/admin#--username) * [\--password](https://docs.temporal.io/tctl-v1/admin#--password) * [\--keyspace](https://docs.temporal.io/tctl-v1/admin#--keyspace) * [\--input\_directory](https://docs.temporal.io/tctl-v1/admin#--input_directory) * [\--lower\_shard\_bound](https://docs.temporal.io/tctl-v1/admin#--lower_shard_bound) * [\--upper\_shard\_bound](https://docs.temporal.io/tctl-v1/admin#--upper_shard_bound) * [\--starting\_rps](https://docs.temporal.io/tctl-v1/admin#--starting_rps) * [\--rps](https://docs.temporal.io/tctl-v1/admin#--rps) * [\--concurrency](https://docs.temporal.io/tctl-v1/admin#--concurrency) * [\--report\_rate](https://docs.temporal.io/tctl-v1/admin#--report_rate) * [\--tls\_cert\_path](https://docs.temporal.io/tctl-v1/admin#--tls_cert_path) * [\--tls\_key\_path](https://docs.temporal.io/tctl-v1/admin#--tls_key_path) * [\--tls\_ca\_path](https://docs.temporal.io/tctl-v1/admin#--tls_ca_path) * [\--tls\_server\_name](https://docs.temporal.io/tctl-v1/admin#--tls_server_name) * [\--tls\_disable\_host\_verification](https://docs.temporal.io/tctl-v1/admin#--tls_disable_host_verification) * [scan](https://docs.temporal.io/tctl-v1/admin#scan) * [\--db\_engine](https://docs.temporal.io/tctl-v1/admin#--db_engine-1) * [\--db\_address](https://docs.temporal.io/tctl-v1/admin#--db_address-1) * [\--db\_port](https://docs.temporal.io/tctl-v1/admin#--db_port-1) * [\--username](https://docs.temporal.io/tctl-v1/admin#--username-1) * [\--password](https://docs.temporal.io/tctl-v1/admin#--password-1) * [\--keyspace](https://docs.temporal.io/tctl-v1/admin#--keyspace-1) * [\--lower\_shard\_bound value](https://docs.temporal.io/tctl-v1/admin#--lower_shard_bound-value) * [\--upper\_shard\_bound](https://docs.temporal.io/tctl-v1/admin#--upper_shard_bound-1) * [\--starting\_rps](https://docs.temporal.io/tctl-v1/admin#--starting_rps-1) * [\--rps value](https://docs.temporal.io/tctl-v1/admin#--rps-value) * [\--pagesize](https://docs.temporal.io/tctl-v1/admin#--pagesize-1) * [\--concurrency](https://docs.temporal.io/tctl-v1/admin#--concurrency-1) * [\--report\_rate](https://docs.temporal.io/tctl-v1/admin#--report_rate-1) * [\--tls](https://docs.temporal.io/tctl-v1/admin#--tls) * [\--tls\_cert\_path](https://docs.temporal.io/tctl-v1/admin#--tls_cert_path-1) * [\--tls\_key\_path](https://docs.temporal.io/tctl-v1/admin#--tls_key_path-1) * [\--tls\_ca\_path](https://docs.temporal.io/tctl-v1/admin#--tls_ca_path-1) * [\--tls\_server\_name](https://docs.temporal.io/tctl-v1/admin#--tls_server_name-1) * [\--tls\_disable\_host\_verification](https://docs.temporal.io/tctl-v1/admin#--tls_disable_host_verification-1) * [decode](https://docs.temporal.io/tctl-v1/admin#decode) * [base64](https://docs.temporal.io/tctl-v1/admin#base64) * [\--base64\_data](https://docs.temporal.io/tctl-v1/admin#--base64_data) * [\--base64\_file](https://docs.temporal.io/tctl-v1/admin#--base64_file) * [proto](https://docs.temporal.io/tctl-v1/admin#proto) * [\--type](https://docs.temporal.io/tctl-v1/admin#--type-1) * [\--hex\_data](https://docs.temporal.io/tctl-v1/admin#--hex_data) * [\--hex\_file](https://docs.temporal.io/tctl-v1/admin#--hex_file) * [\--binary\_file](https://docs.temporal.io/tctl-v1/admin#--binary_file) * [dlq](https://docs.temporal.io/tctl-v1/admin#dlq) * [merge](https://docs.temporal.io/tctl-v1/admin#merge) * [\--dlq\_type](https://docs.temporal.io/tctl-v1/admin#--dlq_type) * [\--cluster](https://docs.temporal.io/tctl-v1/admin#--cluster-1) * [\--shard\_id](https://docs.temporal.io/tctl-v1/admin#--shard_id) * [\--last\_message\_id](https://docs.temporal.io/tctl-v1/admin#--last_message_id) * [purge](https://docs.temporal.io/tctl-v1/admin#purge) * [\--dlq\_type](https://docs.temporal.io/tctl-v1/admin#--dlq_type-1) * [\--cluster](https://docs.temporal.io/tctl-v1/admin#--cluster-2) * [\--shard\_id](https://docs.temporal.io/tctl-v1/admin#--shard_id-1) * [\--last\_message\_id](https://docs.temporal.io/tctl-v1/admin#--last_message_id-1) * [read](https://docs.temporal.io/tctl-v1/admin#read) * [\--dlq\_type](https://docs.temporal.io/tctl-v1/admin#--dlq_type-2) * [\--cluster](https://docs.temporal.io/tctl-v1/admin#--cluster-3) * [\--shard\_id](https://docs.temporal.io/tctl-v1/admin#--shard_id-2) * [\--max\_message\_count](https://docs.temporal.io/tctl-v1/admin#--max_message_count) * [\--last\_message\_id](https://docs.temporal.io/tctl-v1/admin#--last_message_id-2) * [\--output\_filename](https://docs.temporal.io/tctl-v1/admin#--output_filename) * [history\_host](https://docs.temporal.io/tctl-v1/admin#history_host) * [Usage](https://docs.temporal.io/tctl-v1/admin#usage-1) * [Commands](https://docs.temporal.io/tctl-v1/admin#commands-1) * [describe](https://docs.temporal.io/tctl-v1/admin#describe-1) * [\--workflow\_id](https://docs.temporal.io/tctl-v1/admin#--workflow_id) * [\--history\_address](https://docs.temporal.io/tctl-v1/admin#--history_address) * [\--shard\_id](https://docs.temporal.io/tctl-v1/admin#--shard_id-3) * [\--print\_full](https://docs.temporal.io/tctl-v1/admin#--print_full) * [get\_shardid](https://docs.temporal.io/tctl-v1/admin#get_shardid) * [\--namespace\_id](https://docs.temporal.io/tctl-v1/admin#--namespace_id) * [\--workflow\_id](https://docs.temporal.io/tctl-v1/admin#--workflow_id-1) * [\--number\_of\_shards](https://docs.temporal.io/tctl-v1/admin#--number_of_shards) * [membership](https://docs.temporal.io/tctl-v1/admin#membership) * [Usage](https://docs.temporal.io/tctl-v1/admin#usage-2) * [Commands](https://docs.temporal.io/tctl-v1/admin#commands-2) * [list\_db](https://docs.temporal.io/tctl-v1/admin#list_db) * [\--heartbeated\_within](https://docs.temporal.io/tctl-v1/admin#--heartbeated_within) * [\--role](https://docs.temporal.io/tctl-v1/admin#--role) * [list\_gossip](https://docs.temporal.io/tctl-v1/admin#list_gossip) * [\--role value](https://docs.temporal.io/tctl-v1/admin#--role-value) * [shard](https://docs.temporal.io/tctl-v1/admin#shard) * [tctl admin shard commands](https://docs.temporal.io/tctl-v1/admin#tctl-admin-shard-commands) * [close\_shard](https://docs.temporal.io/tctl-v1/admin#close_shard) * [\--shard\_id value](https://docs.temporal.io/tctl-v1/admin#--shard_id-value) * [describe\_task](https://docs.temporal.io/tctl-v1/admin#describe_task) * [\--db\_engine](https://docs.temporal.io/tctl-v1/admin#--db_engine-2) * [\--db\_address](https://docs.temporal.io/tctl-v1/admin#--db_address-2) * [\--db\_port](https://docs.temporal.io/tctl-v1/admin#--db_port-2) * [\--username](https://docs.temporal.io/tctl-v1/admin#--username-2) * [\--password](https://docs.temporal.io/tctl-v1/admin#--password-2) * [\--keyspace](https://docs.temporal.io/tctl-v1/admin#--keyspace-2) * [\--tls](https://docs.temporal.io/tctl-v1/admin#--tls-1) * [\--tls\_cert\_path](https://docs.temporal.io/tctl-v1/admin#--tls_cert_path-2) * [\--tls\_server\_name](https://docs.temporal.io/tctl-v1/admin#--tls_server_name-2) * [\--tls\_disable\_host\_verification](https://docs.temporal.io/tctl-v1/admin#--tls_disable_host_verification-2) * [\--shard\_id](https://docs.temporal.io/tctl-v1/admin#--shard_id-4) * [\--task\_id](https://docs.temporal.io/tctl-v1/admin#--task_id) * [\--task\_type](https://docs.temporal.io/tctl-v1/admin#--task_type) * [\--task\_timestamp](https://docs.temporal.io/tctl-v1/admin#--task_timestamp) * [\--target\_cluster](https://docs.temporal.io/tctl-v1/admin#--target_cluster) * [describe](https://docs.temporal.io/tctl-v1/admin#describe-2) * [\--shard\_id value](https://docs.temporal.io/tctl-v1/admin#--shard_id-value-1) * [list\_tasks](https://docs.temporal.io/tctl-v1/admin#list_tasks) * [\--more](https://docs.temporal.io/tctl-v1/admin#--more) * [\--pagesize value](https://docs.temporal.io/tctl-v1/admin#--pagesize-value) * [\--target\_cluster value](https://docs.temporal.io/tctl-v1/admin#--target_cluster-value) * [\--shard\_id value](https://docs.temporal.io/tctl-v1/admin#--shard_id-value-2) * [\--task\_type value](https://docs.temporal.io/tctl-v1/admin#--task_type-value) * [\--min\_visibility\_ts value](https://docs.temporal.io/tctl-v1/admin#--min_visibility_ts-value) * [\--max\_visibility\_ts value](https://docs.temporal.io/tctl-v1/admin#--max_visibility_ts-value) * [remove\_task](https://docs.temporal.io/tctl-v1/admin#remove_task) * [\--shard\_id value](https://docs.temporal.io/tctl-v1/admin#--shard_id-value-3) * [\--task\_id value](https://docs.temporal.io/tctl-v1/admin#--task_id-value) * [\--task\_type value](https://docs.temporal.io/tctl-v1/admin#--task_type-value-1) * [\--task\_timestamp value](https://docs.temporal.io/tctl-v1/admin#--task_timestamp-value) * [workflow](https://docs.temporal.io/tctl-v1/admin#workflow) * [delete](https://docs.temporal.io/tctl-v1/admin#delete) * [\--db\_engine value](https://docs.temporal.io/tctl-v1/admin#--db_engine-value) * [\--db\_address value](https://docs.temporal.io/tctl-v1/admin#--db_address-value) * [\--db\_port value](https://docs.temporal.io/tctl-v1/admin#--db_port-value) * [\--username value](https://docs.temporal.io/tctl-v1/admin#--username-value) * [\--password value](https://docs.temporal.io/tctl-v1/admin#--password-value) * [\--keyspace value](https://docs.temporal.io/tctl-v1/admin#--keyspace-value) * [\--url value](https://docs.temporal.io/tctl-v1/admin#--url-value) * [\--es-username value](https://docs.temporal.io/tctl-v1/admin#--es-username-value) * [\--es-password value](https://docs.temporal.io/tctl-v1/admin#--es-password-value) * [\--version value](https://docs.temporal.io/tctl-v1/admin#--version-value) * [\--index value](https://docs.temporal.io/tctl-v1/admin#--index-value) * [\--workflow\_id value](https://docs.temporal.io/tctl-v1/admin#--workflow_id-value) * [\--run\_id value](https://docs.temporal.io/tctl-v1/admin#--run_id-value) * [\--skip\_errors](https://docs.temporal.io/tctl-v1/admin#--skip_errors) * [\--tls](https://docs.temporal.io/tctl-v1/admin#--tls-2) * [\--tls\_cert\_path value](https://docs.temporal.io/tctl-v1/admin#--tls_cert_path-value) * [\--tls\_key\_path value](https://docs.temporal.io/tctl-v1/admin#--tls_key_path-value) * [\--tls\_ca\_path value](https://docs.temporal.io/tctl-v1/admin#--tls_ca_path-value) * [\--tls\_server\_name value](https://docs.temporal.io/tctl-v1/admin#--tls_server_name-value) * [\--tls\_disable\_host\_verification](https://docs.temporal.io/tctl-v1/admin#--tls_disable_host_verification-3) * [describe](https://docs.temporal.io/tctl-v1/admin#describe-3) * [\--workflow\_id value](https://docs.temporal.io/tctl-v1/admin#--workflow_id-value-1) * [\--run\_id value](https://docs.temporal.io/tctl-v1/admin#--run_id-value-1) * [refresh\_tasks](https://docs.temporal.io/tctl-v1/admin#refresh_tasks) * [\--workflow\_id value](https://docs.temporal.io/tctl-v1/admin#--workflow_id-value-2) * [\--run\_id value](https://docs.temporal.io/tctl-v1/admin#--run_id-value-2) * [show](https://docs.temporal.io/tctl-v1/admin#show) * [\--workflow\_id value](https://docs.temporal.io/tctl-v1/admin#--workflow_id-value-3) * [\--run\_id value](https://docs.temporal.io/tctl-v1/admin#--run_id-value-3) * [\--min\_event\_id value](https://docs.temporal.io/tctl-v1/admin#--min_event_id-value) * [\--max\_event\_id value](https://docs.temporal.io/tctl-v1/admin#--max_event_id-value) * [\--min\_event\_version value](https://docs.temporal.io/tctl-v1/admin#--min_event_version-value) * [\--max\_event\_version value](https://docs.temporal.io/tctl-v1/admin#--max_event_version-value) * [\--output\_filename value](https://docs.temporal.io/tctl-v1/admin#--output_filename-value) --- # tctl v1.17 namespace command reference | Temporal Platform Documentation [Skip to main content](https://docs.temporal.io/tctl-v1/namespace#__docusaurus_skipToContent_fallback) On this page tctl is deprecated The tctl command line utility has been deprecated and is no longer actively supported. We recommend transitioning to [Temporal CLI](https://docs.temporal.io/cli) for continued use and access to new features. Thank you for being a valued part of the Temporal community. The `tctl namespace` commands enable [Namespace](https://docs.temporal.io/namespaces) operations. Alias: `n` * [tctl namespace describe](https://docs.temporal.io/tctl-v1/namespace#describe) * [tctl namespace list](https://docs.temporal.io/tctl-v1/namespace#list) * [tctl namespace register](https://docs.temporal.io/tctl-v1/namespace#register) * [tctl namespace update](https://docs.temporal.io/tctl-v1/namespace#update) describe[​](https://docs.temporal.io/tctl-v1/namespace#describe "Direct link to describe") ------------------------------------------------------------------------------------------- The `tctl namespace describe` command describes a [Namespace](https://docs.temporal.io/namespaces) . `tctl namespace describe` The following modifier controls the behavior of the command. ### \--namespace\_id[​](https://docs.temporal.io/tctl-v1/namespace#--namespace_id "Direct link to --namespace_id") Specify the ID of a Namespace to describe. This modifier is required unless the global `--namespace` modifier is specified (`tctl --namespace describe`). **Example** tctl namespace describe --namespace_id Example results for a [Global Namespace](https://docs.temporal.io/global-namespace) $ tctl --ns canary-namespace n descName: canary-namespaceDescription: testing namespaceOwnerEmail: dev@yourtech.ioNamespaceData:Status: REGISTEREDRetentionInDays: 7EmitMetrics: trueActiveClusterName: dc1Clusters: dc1, dc2 list[​](https://docs.temporal.io/tctl-v1/namespace#list "Direct link to list") ------------------------------------------------------------------------------- The `tctl namespace list` command lists all [Namespaces](https://docs.temporal.io/namespaces) . `tctl namespace list` The command has no modifiers. register[​](https://docs.temporal.io/tctl-v1/namespace#register "Direct link to register") ------------------------------------------------------------------------------------------- The `tctl namespace register` command registers a [Namespace](https://docs.temporal.io/namespaces) . `tctl namespace register` By default, Temporal uses a "default" Namespace. Create and register a new Namespace with the following command: tctl --namespace your-namespace namespace register# OR using short aliastctl --ns your-namespace n re The following modifiers control the behavior of the command. ### \--active\_cluster[​](https://docs.temporal.io/tctl-v1/namespace#--active_cluster "Direct link to --active_cluster") Specify the name of the active [Temporal Cluster](https://docs.temporal.io/temporal-service) when registering a [Namespace](https://docs.temporal.io/namespaces) . This value changes for Global Namespaces when a failover occurs. **Example** tctl namespace register --active_cluster ### \--clusters[​](https://docs.temporal.io/tctl-v1/namespace#--clusters "Direct link to --clusters") Specify a list of [Temporal Clusters](https://docs.temporal.io/temporal-service) when registering a [Namespace](https://docs.temporal.io/namespaces) . The list contains the names of Clusters (separated by spaces) to which the Namespace can fail over. Make sure to include to the currently active Cluster. This is a read-only setting and cannot be changed. This modifier is valid only when the `--global_namespace` modifier is set to true. **Example** tctl namespace register --clusters ### \--description[​](https://docs.temporal.io/tctl-v1/namespace#--description "Direct link to --description") Specify a description when registering a [Namespace](https://docs.temporal.io/namespaces) . **Example** tctl namespace register --description ### \--global\_namespace[​](https://docs.temporal.io/tctl-v1/namespace#--global_namespace "Direct link to --global_namespace") Specifies whether a [Namespace](https://docs.temporal.io/namespaces) is a [Global Namespace](https://docs.temporal.io/global-namespace) . When enabled, it controls the creation of replication tasks on updates allowing the state to be replicated across Clusters. This is a read-only setting and cannot be changed. **Example** tctl namespace register --global_namespace ### \--history\_archival\_state[​](https://docs.temporal.io/tctl-v1/namespace#--history_archival_state "Direct link to --history_archival_state") Set the state of [Archival](https://docs.temporal.io/temporal-service/archival) . Valid values are `disabled` and `enabled`. **Example** tctl namespace register --history_archival_state ### \--history\_uri[​](https://docs.temporal.io/tctl-v1/namespace#--history_uri "Direct link to --history_uri") Specify the URI for [Archival](https://docs.temporal.io/temporal-service/archival) . The URI cannot be changed after Archival is first enabled. **Example** tctl namespace register --history_uri ### \--namespace\_data[​](https://docs.temporal.io/tctl-v1/namespace#--namespace_data "Direct link to --namespace_data") Specify data for a [Namespace](https://docs.temporal.io/namespaces) in the form of key-value pairs (such as `k1:v1,k2:v2,k3:v3`). **Example** tctl namespace register --namespace_data ### \--owner\_email[​](https://docs.temporal.io/tctl-v1/namespace#--owner_email "Direct link to --owner_email") Specify the email address of the [Namespace](https://docs.temporal.io/namespaces) owner. **Example** tctl namespace register --owner_email ### \--retention[​](https://docs.temporal.io/tctl-v1/namespace#--retention "Direct link to --retention") Set the [Retention Period](https://docs.temporal.io/temporal-service/temporal-server#retention-period) for the [Namespace](https://docs.temporal.io/namespaces) . The Retention Period applies to Closed [Workflow Executions](https://docs.temporal.io/workflow-execution) . **Example** tctl namespace register --retention ### \--visibility\_archival\_state[​](https://docs.temporal.io/tctl-v1/namespace#--visibility_archival_state "Direct link to --visibility_archival_state") Set the visibility state for [Archival](https://docs.temporal.io/temporal-service/archival) . Valid values are `disabled` and `enabled`. **Example** tctl namespace register --visibility_archival_state ### \--visibility\_uri[​](https://docs.temporal.io/tctl-v1/namespace#--visibility_uri "Direct link to --visibility_uri") Specify the visibility URI for [Archival](https://docs.temporal.io/temporal-service/archival) . The URI cannot be changed after Archival is first enabled. **Example** tctl namespace register --visibility_uri update[​](https://docs.temporal.io/tctl-v1/namespace#update "Direct link to update") ------------------------------------------------------------------------------------- The `tctl namespace update` command updates a [Namespace](https://docs.temporal.io/namespaces) . `tctl namespace update` The following modifiers control the behavior of the command. ### \--active\_cluster[​](https://docs.temporal.io/tctl-v1/namespace#--active_cluster-1 "Direct link to --active_cluster") Specify the name of the active [Temporal Cluster](https://docs.temporal.io/temporal-service) when updating a [Namespace](https://docs.temporal.io/namespaces) . **Example** tctl namespace update --active_cluster ### \--add\_bad\_binary[​](https://docs.temporal.io/tctl-v1/namespace#--add_bad_binary "Direct link to --add_bad_binary") Add a binary checksum to use when resetting a [Workflow Execution](https://docs.temporal.io/workflow-execution) . Temporal will not dispatch any [Commands](https://docs.temporal.io/workflow-execution#command) to the given binary. See also [`--remove_bad_binary`](https://docs.temporal.io/tctl-v1/namespace#--remove_bad_binary) . **Example** tctl namespace update --add_bad_binary ### \--clusters[​](https://docs.temporal.io/tctl-v1/namespace#--clusters-1 "Direct link to --clusters") Specify a list of [Temporal Clusters](https://docs.temporal.io/temporal-service) when updating a [Namespace](https://docs.temporal.io/namespaces) . The list contains the names of Clusters (separated by spaces) to which the Namespace can fail over. This modifier is valid only when the `--global_namespace` modifier is set to true. **Example** tctl namespace update --clusters ### \--description[​](https://docs.temporal.io/tctl-v1/namespace#--description-1 "Direct link to --description") Specify a description when updating a [Namespace](https://docs.temporal.io/namespaces) . **Example** tctl namespace update --description ### \--history\_archival\_state[​](https://docs.temporal.io/tctl-v1/namespace#--history_archival_state-1 "Direct link to --history_archival_state") Set the state of [Archival](https://docs.temporal.io/temporal-service/archival) . Valid values are `disabled` and `enabled`. **Example** tctl namespace update --history_archival_state ### \--history\_uri[​](https://docs.temporal.io/tctl-v1/namespace#--history_uri-1 "Direct link to --history_uri") Specify the URI for [Archival](https://docs.temporal.io/temporal-service/archival) . The URI cannot be changed after Archival is first enabled. **Example** tctl namespace update --history_uri ### \--namespace\_data[​](https://docs.temporal.io/tctl-v1/namespace#--namespace_data-1 "Direct link to --namespace_data") Specify data for a [Namespace](https://docs.temporal.io/namespaces) in the form of key-value pairs (such as `k1:v1,k2:v2,k3:v3`). **Example** tctl namespace update --namespace_data ### \--owner\_email[​](https://docs.temporal.io/tctl-v1/namespace#--owner_email-1 "Direct link to --owner_email") Specify the email address of the [Namespace](https://docs.temporal.io/namespaces) owner. **Example** tctl namespace update --owner_email ### \--reason[​](https://docs.temporal.io/tctl-v1/namespace#--reason "Direct link to --reason") Specify a reason for updating a [Namespace](https://docs.temporal.io/namespaces) . **Example** tctl namespace update --reason ### \--remove\_bad\_binary[​](https://docs.temporal.io/tctl-v1/namespace#--remove_bad_binary "Direct link to --remove_bad_binary") Remove a binary checksum. See also [`--add_bad_binary`](https://docs.temporal.io/tctl-v1/namespace#--add_bad_binary) . **Example** tctl namespace update --remove_bad_binary ### \--retention[​](https://docs.temporal.io/tctl-v1/namespace#--retention-1 "Direct link to --retention") Specify the number of days to retain [Workflow Executions](https://docs.temporal.io/workflow-execution) . **Example** tctl namespace update --retention ### \--visibility\_archival\_state[​](https://docs.temporal.io/tctl-v1/namespace#--visibility_archival_state-1 "Direct link to --visibility_archival_state") Set the visibility state for [Archival](https://docs.temporal.io/temporal-service/archival) . Valid values are `disabled` and `enabled`. **Example** tctl namespace update --visibility_archival_state ### \--visibility\_uri[​](https://docs.temporal.io/tctl-v1/namespace#--visibility_uri-1 "Direct link to --visibility_uri") Specify the visibility URI for [Archival](https://docs.temporal.io/temporal-service/archival) . The URI cannot be changed after Archival is first enabled. **Example** tctl namespace update --visibility_uri * [describe](https://docs.temporal.io/tctl-v1/namespace#describe) * [\--namespace\_id](https://docs.temporal.io/tctl-v1/namespace#--namespace_id) * [list](https://docs.temporal.io/tctl-v1/namespace#list) * [register](https://docs.temporal.io/tctl-v1/namespace#register) * [\--active\_cluster](https://docs.temporal.io/tctl-v1/namespace#--active_cluster) * [\--clusters](https://docs.temporal.io/tctl-v1/namespace#--clusters) * [\--description](https://docs.temporal.io/tctl-v1/namespace#--description) * [\--global\_namespace](https://docs.temporal.io/tctl-v1/namespace#--global_namespace) * [\--history\_archival\_state](https://docs.temporal.io/tctl-v1/namespace#--history_archival_state) * [\--history\_uri](https://docs.temporal.io/tctl-v1/namespace#--history_uri) * [\--namespace\_data](https://docs.temporal.io/tctl-v1/namespace#--namespace_data) * [\--owner\_email](https://docs.temporal.io/tctl-v1/namespace#--owner_email) * [\--retention](https://docs.temporal.io/tctl-v1/namespace#--retention) * [\--visibility\_archival\_state](https://docs.temporal.io/tctl-v1/namespace#--visibility_archival_state) * [\--visibility\_uri](https://docs.temporal.io/tctl-v1/namespace#--visibility_uri) * [update](https://docs.temporal.io/tctl-v1/namespace#update) * [\--active\_cluster](https://docs.temporal.io/tctl-v1/namespace#--active_cluster-1) * [\--add\_bad\_binary](https://docs.temporal.io/tctl-v1/namespace#--add_bad_binary) * [\--clusters](https://docs.temporal.io/tctl-v1/namespace#--clusters-1) * [\--description](https://docs.temporal.io/tctl-v1/namespace#--description-1) * [\--history\_archival\_state](https://docs.temporal.io/tctl-v1/namespace#--history_archival_state-1) * [\--history\_uri](https://docs.temporal.io/tctl-v1/namespace#--history_uri-1) * [\--namespace\_data](https://docs.temporal.io/tctl-v1/namespace#--namespace_data-1) * [\--owner\_email](https://docs.temporal.io/tctl-v1/namespace#--owner_email-1) * [\--reason](https://docs.temporal.io/tctl-v1/namespace#--reason) * [\--remove\_bad\_binary](https://docs.temporal.io/tctl-v1/namespace#--remove_bad_binary) * [\--retention](https://docs.temporal.io/tctl-v1/namespace#--retention-1) * [\--visibility\_archival\_state](https://docs.temporal.io/tctl-v1/namespace#--visibility_archival_state-1) * [\--visibility\_uri](https://docs.temporal.io/tctl-v1/namespace#--visibility_uri-1) --- # tctl v1.17 data-converter command reference | Temporal Platform Documentation [Skip to main content](https://docs.temporal.io/tctl-v1/dataconverter#__docusaurus_skipToContent_fallback) On this page tctl is deprecated The tctl command line utility has been deprecated and is no longer actively supported. We recommend transitioning to [Temporal CLI](https://docs.temporal.io/cli) for continued use and access to new features. Thank you for being a valued part of the Temporal community. The `tctl dataconverter` command enables custom [Data Converter](https://docs.temporal.io/dataconversion) operations. * [tctl dataconverter web](https://docs.temporal.io/tctl-v1/dataconverter#web) web[​](https://docs.temporal.io/tctl-v1/dataconverter#web "Direct link to web") -------------------------------------------------------------------------------- The `tctl dataconverter web` command specifies the WebSocket URL of a custom [Data Converter](https://docs.temporal.io/dataconversion) to use with Temporal Web. `tctl dataconverter web --web_ui_url ` The following modifiers control the behavior of the command. ### \--port[​](https://docs.temporal.io/tctl-v1/dataconverter#--port "Direct link to --port") Specify a port for the WebSocket URL of a custom [Data Converter](https://docs.temporal.io/dataconversion) . The default value is 0. **Example** tctl dataconverter web --web_ui_url --port ### \--web\_ui\_url[​](https://docs.temporal.io/tctl-v1/dataconverter#--web_ui_url "Direct link to --web_ui_url") _Required modifier_ Specify the WebSocket URL of a custom [Data Converter](https://docs.temporal.io/dataconversion) . **Example** tctl dataconverter web --web_ui_url * [web](https://docs.temporal.io/tctl-v1/dataconverter#web) * [\--port](https://docs.temporal.io/tctl-v1/dataconverter#--port) * [\--web\_ui\_url](https://docs.temporal.io/tctl-v1/dataconverter#--web_ui_url) --- # Temporal Platform production deployments | Temporal Platform Documentation [Skip to main content](https://docs.temporal.io/production-deployment#__docusaurus_skipToContent_fallback) On this page To take your application to production, you'll need to deploy the following components: * Your application code, including your Workflows, Activities, and Workers, on your infrastructure using your existing build, test and deploy tools. * A production-ready Temporal Service to coordinate the execution of your Workflows and Activities. * You can use Temporal Cloud, a fully managed platform, or you can self-host the service. Do you need a production Temporal Service? If you're still developing and testing your application locally, you may not need a production Temporal Service. Use the [Temporal CLI development server](https://docs.temporal.io/cli/server#start-dev) — a single binary with no external dependencies: `temporal server start-dev` This starts a complete Temporal Service with Web UI on your local machine. We recommend this for local development regardless of whether you plan to use Temporal Cloud or self-host in production. See the [Temporal CLI server](https://docs.temporal.io/cli/server) page for configuration options. Use Temporal Cloud[​](https://docs.temporal.io/production-deployment#use-temporal-cloud "Direct link to Use Temporal Cloud") ----------------------------------------------------------------------------------------------------------------------------- You can let us handle the operations of running the Temporal Service, and focus on your application. Follow the [Temporal Cloud guide](https://docs.temporal.io/cloud) to get started. ![Connect your application instances to Temporal Cloud](https://docs.temporal.io/diagrams/basic-platform-topology-cloud.svg) Connect your application instances to Temporal Cloud Run a Self-hosted Temporal Service[​](https://docs.temporal.io/production-deployment#run-a-self-hosted-temporal-service "Direct link to Run a Self-hosted Temporal Service") ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Alternatively, you can run your own production level Temporal Service to orchestrate your durable applications. Follow the [Self-hosted guide](https://docs.temporal.io/self-hosted-guide) to get started. ![Connect your application instances to your self-hosted Temporal Service](https://docs.temporal.io/diagrams/basic-platform-topology-self-hosted.svg) Connect your application instances to your self-hosted Temporal Service Worker deployments[​](https://docs.temporal.io/production-deployment#worker-deployments "Direct link to Worker deployments") ----------------------------------------------------------------------------------------------------------------------------- Whether you're hosting with Temporal Cloud or on your own, you have control over where to run and scale your Workers. We provide guidance on [Worker Deployments](https://docs.temporal.io/production-deployment/worker-deployments) . * [Use Temporal Cloud](https://docs.temporal.io/production-deployment#use-temporal-cloud) * [Run a Self-hosted Temporal Service](https://docs.temporal.io/production-deployment#run-a-self-hosted-temporal-service) * [Worker deployments](https://docs.temporal.io/production-deployment#worker-deployments) --- # Why Temporal? | Temporal Platform Documentation [Skip to main content](https://docs.temporal.io/evaluate/why-temporal#__docusaurus_skipToContent_fallback) On this page Temporal solves many problems that developers face while building distributed applications. But most of them revolve around these three themes: * Reliable distributed applications * Productive development paradigms and code structure * Visible distributed application state See Temporal in action Watch the following video to see how Temporal ensures an order-fulfillment system can recover from various failures, from process crashes to unreachable APIs. Reliable execution[​](https://docs.temporal.io/evaluate/why-temporal#reliable-execution "Direct link to Reliable execution") ----------------------------------------------------------------------------------------------------------------------------- **How does Temporal make applications reliable?** Temporal makes it easier for developers to build and operate reliable, scalable applications without sacrificing productivity. The design of the system ensures that, once started, an application's main function executes to completion, whether that takes minutes, hours, days, weeks, or even years. Temporal calls this _Durable Execution._ Code structure[​](https://docs.temporal.io/evaluate/why-temporal#code-structure "Direct link to Code structure") ----------------------------------------------------------------------------------------------------------------- **How does Temporal simplify application code for software developers?** By shifting the burden of failure handling from the application to the platform, there is less code for application developers to write, test, and maintain. Temporal's programming model offers developers a way to express their business logic into coherent _Workflows_ that are much easier to develop than distributed code bases. Choose the SDK that best suits your preferred programming language and start writing your business logic. Integrate your favorite IDE, libraries, and tools into your development process. Temporal also supports polyglot and idiomatic programming - which enables developers to leverage the strengths of various programming languages and integrate Temporal into existing codebases. Developers achieve all of this without having to manage queues or complex state machines. State visibility[​](https://docs.temporal.io/evaluate/why-temporal#state-visibility "Direct link to State visibility") ----------------------------------------------------------------------------------------------------------------------- **How does Temporal make it easier to view the state of the application?** Temporal provides out-of-the-box tooling that enables developers to see the state of their applications whenever they need to. The Temporal CLI allows developers to manage, monitor, and debug Temporal applications effectively. The browser-based Web UI lets you quickly isolate, debug, and resolve production problems. * [Reliable execution](https://docs.temporal.io/evaluate/why-temporal#reliable-execution) * [Code structure](https://docs.temporal.io/evaluate/why-temporal#code-structure) * [State visibility](https://docs.temporal.io/evaluate/why-temporal#state-visibility) --- # Temporal Platform security | Temporal Platform Documentation [Skip to main content](https://docs.temporal.io/security#__docusaurus_skipToContent_fallback) Find security information for your Temporal deployment, whether you're using Temporal Cloud or self-hosting. [### Company Security\ \ Learn about Temporal Technologies' general security practices, compliance certifications, and organizational security measures.](https://trust.temporal.io/) [### Temporal Cloud Security\ \ Explore the security features of our SaaS offering, including mTLS, end-to-end encryption, and enterprise compliance.](https://docs.temporal.io/cloud/security) [### Self-Hosted Security\ \ Discover how to deploy and secure your own Temporal Platform infrastructure with production-ready best practices.](https://docs.temporal.io/self-hosted-guide/security) [### Temporal Cloud Security Whitepaper\ \ Learn how Temporal Cloud provides provable security by design - orchestrating encrypted workflows without ever accessing your sensitive data.](https://temporal.io/pages/cloud-security-white-paper) --- # Temporal Worker Controller | Temporal Platform Documentation [Skip to main content](https://docs.temporal.io/production-deployment/worker-deployments/kubernetes-controller#__docusaurus_skipToContent_fallback) On this page The [Temporal Worker Controller](https://github.com/temporalio/temporal-worker-controller) provides automation to enable rainbow deployments of your Workers by simplifying the tracking of which versions still have active Workflows, managing the lifecycle of versioned Worker deployments, and calling Temporal APIs to update the routing config of Temporal Worker Deployments. The Temporal Worker Controller makes it simple and safe to deploy Temporal Workers on Kubernetes. If you run versioned Workers on Kubernetes, the Worker Controller is the recommended way to manage rollouts and autoscaling together. ### Why adopt the Worker Controller?[​](https://docs.temporal.io/production-deployment/worker-deployments/kubernetes-controller#why-adopt-the-worker-controller "Direct link to Why adopt the Worker Controller?") The traditional approach to revising Temporal Workflows is to add branches using the [Versioning APIs](https://docs.temporal.io/workflow-definition#workflow-versioning) . Over time these checks can become a source of technical debt, as safely removing them from a codebase is a careful process that often involves querying all running Workflows. [Worker Versioning](https://docs.temporal.io/production-deployment/worker-deployments/worker-versioning) is a Temporal feature that allows you to pin Workflows to individual versions of your Workers, which are called Worker Deployment Versions. Using pinning, you will not need to add branching to your Workflows to avoid non-determinism errors. This allows you to bypass the other Versioning APIs. The Worker Controller gives you direct, programmatic control over your Worker deployments, and integrates with the [Temporal CLI](https://docs.temporal.io/production-deployment/worker-deployments/worker-versioning#rolling-out-changes-with-the-cli) . You do not need to use the Worker Controller to use Worker Versioning, but when used together, Worker Versioning and the Worker Controller can provide more graceful deployments and upgrades, and less need to manually tune your Workers. Note that in Temporal, **Worker Deployment** is sometimes referred to as **Deployment**, but since the Worker Controller makes significant references to Kubernetes Deployment resource, within this page we will stick to these terms: * [**Worker Deployment**](https://docs.temporal.io/worker-versioning#deployments) : A Worker Deployment is a logical service that groups similar Workers together for unified management. Each Deployment has a name (such as your service name) and supports versioning through a series of Worker Deployment Versions. * [**Worker Deployment Version**](https://docs.temporal.io/worker-versioning#deployment-versions) : A Worker Deployment Version represents an iteration of a Worker Deployment. Each Deployment Version consists of Workers that share the same code build and environment. When a Worker starts polling for Workflow and Activity Tasks, it reports its Deployment Version to the Temporal Server. * **Deployment**: A Kubernetes Deployment resource. A Deployment is "versioned" if it is running versioned Temporal workers/pollers. ### Features[​](https://docs.temporal.io/production-deployment/worker-deployments/kubernetes-controller#features "Direct link to Features") * Registration of new Temporal Worker Deployment Versions * Creation of versioned Deployment resources (that manage the Pods that run your Temporal pollers) * Deletion of resources associated with drained Worker Deployment Versions * `Manual`, `AllAtOnce`, and `Progressive` rollouts of new versions * Ability to specify a "gate" Workflow that must succeed on the new version before routing real traffic to that version * Autoscaling of versioned Deployments using Kubernetes Horizontal Pod Autoscaler (HPA) Refer to the [Temporal Worker Controller repo](https://github.com/temporalio/temporal-worker-controller/) for usage details. Autoscaling versioned Workers[​](https://docs.temporal.io/production-deployment/worker-deployments/kubernetes-controller#autoscaling-versioned-workers "Direct link to Autoscaling versioned Workers") ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- The Worker Controller can manage autoscaling for versioned Worker Deployments without forcing you to choose between safe rollout behavior and elastic capacity. Use the Worker Controller when you need all of the following: * [Worker Versioning](https://docs.temporal.io/production-deployment/worker-deployments/worker-versioning) for safe Workflow code changes * Kubernetes-native rollout automation * autoscaling that follows each active Worker Deployment Version separately Because the Worker Controller uses Kubernetes HPA, you can scale on any metric available to your HPA pipeline, including: * CPU and memory utilization * Task Queue backlog metrics exposed through your metrics pipeline * slot utilization and other Worker-specific metrics * custom metrics surfaced through Prometheus or another Kubernetes metrics adapter ### TemporalWorkerOwnedResource[​](https://docs.temporal.io/production-deployment/worker-deployments/kubernetes-controller#temporalworkerownedresource "Direct link to TemporalWorkerOwnedResource") To attach autoscaling or other Kubernetes resources to each Worker Deployment Version, use a `TemporalWorkerOwnedResource` (TWOR). A TWOR lets you define a resource template once and have the Worker Controller create a version-specific copy for each active Worker Deployment Version. This is useful for resources such as: * `HorizontalPodAutoscaler` * `PodDisruptionBudget` * other Kubernetes resources that should track the lifecycle of a versioned Deployment The Worker Controller manages these resources alongside the versioned Deployments it creates, so they are updated and cleaned up as versions roll forward and drain. ### Why use this instead of KEDA?[​](https://docs.temporal.io/production-deployment/worker-deployments/kubernetes-controller#why-use-this-instead-of-keda "Direct link to Why use this instead of KEDA?") If you are already using the Worker Controller for Worker Versioning, use the Worker Controller for autoscaling as well. This keeps rollout management and scaling attached to the same versioned Kubernetes Deployments. KEDA can still be a valid option for non-versioned or legacy worker deployments. However, for versioned Workers, the Worker Controller is the preferred path because it keeps autoscaling aligned with Worker Deployment Versions. Configuring Worker Lifecycles[​](https://docs.temporal.io/production-deployment/worker-deployments/kubernetes-controller#configuring-worker-lifecycles "Direct link to Configuring Worker Lifecycles") ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- To use the Temporal Worker Controller, tag your Workers following the guidance for using [Worker Versioning](https://docs.temporal.io/production-deployment/worker-deployments/worker-versioning) . Here is an example of a progressive rollout strategy gated on the success of the `HelloWorld` Workflow: rollout: strategy: Progressive steps: - rampPercentage: 1 pauseDuration: 30s - rampPercentage: 10 pauseDuration: 1m gate: workflowType: "HelloWorld" As you ship new deployment versions, the Worker Controller automatically detects them and gradually makes that version the new **Current Version** of the Worker deployment it is a part of. As older pinned Workflows finish executing and deprecated deployment versions become drained, the Worker Controller also frees up resources by sunsetting the `Deployment` resources polling those versions. When you use autoscaling with the Worker Controller, each active Worker Deployment Version can scale independently while it is serving traffic. This allows older versions to drain safely while newer versions scale based on live demand. Running the Temporal Worker Controller[​](https://docs.temporal.io/production-deployment/worker-deployments/kubernetes-controller#running-the-temporal-worker-controller "Direct link to Running the Temporal Worker Controller") ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- You can install the Temporal Worker Controller using our Helm chart: RELEASE=temporal-worker-controllerNAMESPACE=temporal-systemVERSION=1.0.0helm install $RELEASE oci://docker.io/temporalio/helm-charts/temporal-worker-controller \ --version $VERSION \ --namespace $NAMESPACE \ --create-namespace helm install temporal-worker-controller ./helm/temporal-worker-controller \ --namespace $NAMESPACE \ --create-namespace Refer to [GitHub](https://github.com/temporalio/temporal-worker-controller/tree/main/helm/temporal-worker-controller/templates) for other Worker Controller deployment templates. * [Why adopt the Worker Controller?](https://docs.temporal.io/production-deployment/worker-deployments/kubernetes-controller#why-adopt-the-worker-controller) * [Features](https://docs.temporal.io/production-deployment/worker-deployments/kubernetes-controller#features) * [Autoscaling versioned Workers](https://docs.temporal.io/production-deployment/worker-deployments/kubernetes-controller#autoscaling-versioned-workers) * [TemporalWorkerOwnedResource](https://docs.temporal.io/production-deployment/worker-deployments/kubernetes-controller#temporalworkerownedresource) * [Why use this instead of KEDA?](https://docs.temporal.io/production-deployment/worker-deployments/kubernetes-controller#why-use-this-instead-of-keda) * [Configuring Worker Lifecycles](https://docs.temporal.io/production-deployment/worker-deployments/kubernetes-controller#configuring-worker-lifecycles) * [Running the Temporal Worker Controller](https://docs.temporal.io/production-deployment/worker-deployments/kubernetes-controller#running-the-temporal-worker-controller) --- # Troubleshoot the blob size limit error | Temporal Platform Documentation [Skip to main content](https://docs.temporal.io/troubleshooting/blob-size-limit-error#__docusaurus_skipToContent_fallback) On this page The `BlobSizeLimitError` is an error that occurs when the size of a blob (payloads including Workflow context and each Workflow and Activity argument and return value) exceeds the set limit in Temporal. * The max payload for a single request is 2 MB. * The max size limit for any given [Event History](https://docs.temporal.io/workflow-execution/event#event-history) transaction is 4 MB. Why does this error occur?[​](https://docs.temporal.io/troubleshooting/blob-size-limit-error#why-does-this-error-occur "Direct link to Why does this error occur?") -------------------------------------------------------------------------------------------------------------------------------------------------------------------- This error occurs when the size of the blob exceeds the maximum size allowed by Temporal. This limit helps ensure that the Temporal Service prevents excessive resource usage and potential performance issues when handling large payloads. How do I resolve this error?[​](https://docs.temporal.io/troubleshooting/blob-size-limit-error#how-do-i-resolve-this-error "Direct link to How do I resolve this error?") -------------------------------------------------------------------------------------------------------------------------------------------------------------------------- To resolve this error, reduce the size of the blob so that it is within the 4 MB limit. There are multiple strategies you can use to avoid this error: 1. Use compression with a [custom payload codec](https://docs.temporal.io/payload-codec) for large payloads. * This addresses the immediate issue of the blob size limit; however, if blob sizes continue to grow this problem can arise again. 2. Break larger batches of commands into smaller batch sizes: * Workflow-level batching: 1. Modify the Workflow to process Activities or Child Workflows into smaller batches. 2. Iterate through each batch, waiting for completion before moving to the next. * Workflow Task-level batching: 1. Execute Activities in smaller batches within a single Workflow Task. 2. Introduce brief pauses or sleeps (for example, 1ms) between batches. 3. Consider offloading large payloads to an object store to reduce the risk of exceeding blob size limits: 1. Pass references to the stored payloads within the Workflow instead of the actual data. 2. Retrieve the payloads from the object store when needed during execution. Workflow termination due to oversized response[​](https://docs.temporal.io/troubleshooting/blob-size-limit-error#workflow-termination-due-to-oversized-response "Direct link to Workflow termination due to oversized response") --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- When a Workflow Task response exceeds the 4 MB gRPC message size limit, Temporal automatically terminates the Workflow Execution. This is a non-recoverable error. The Workflow can't progress if it generates a response that's too large, so retrying won't help. This typically happens when a Workflow schedules too many Activities, Child Workflows, or other commands in a single Workflow Task. The total size of all commands generated by the Workflow Task must fit within the 4 MB limit. If your Workflow was terminated for this reason, you'll see a `WorkflowExecutionTerminated` event in the Event History with the cause `WORKFLOW_TASK_FAILED_CAUSE_GRPC_MESSAGE_TOO_LARGE`. To prevent this, use the batching strategies described above to split work across multiple Workflow Tasks instead of scheduling everything at once. See the [gRPC Message Too Large error reference](https://docs.temporal.io/references/errors#grpc-message-too-large) for more details. * [Why does this error occur?](https://docs.temporal.io/troubleshooting/blob-size-limit-error#why-does-this-error-occur) * [How do I resolve this error?](https://docs.temporal.io/troubleshooting/blob-size-limit-error#how-do-i-resolve-this-error) * [Workflow termination due to oversized response](https://docs.temporal.io/troubleshooting/blob-size-limit-error#workflow-termination-due-to-oversized-response) --- # tctl v1.17 taskqueue command reference | Temporal Platform Documentation [Skip to main content](https://docs.temporal.io/tctl-v1/taskqueue#__docusaurus_skipToContent_fallback) On this page tctl is deprecated The tctl command line utility has been deprecated and is no longer actively supported. We recommend transitioning to [Temporal CLI](https://docs.temporal.io/cli) for continued use and access to new features. Thank you for being a valued part of the Temporal community. The `tctl taskqueue` command enables [Task Queue](https://docs.temporal.io/task-queue) operations. Alias: `t` * [tctl taskqueue describe](https://docs.temporal.io/tctl-v1/taskqueue#describe) * [tctl taskqueue list-partition](https://docs.temporal.io/tctl-v1/taskqueue#list-partition) describe[​](https://docs.temporal.io/tctl-v1/taskqueue#describe "Direct link to describe") ------------------------------------------------------------------------------------------- The `tctl taskqueue describe` command describes the poller information of a [Task Queue](https://docs.temporal.io/task-queue) . `tctl taskqueue describe ` The following modifiers control the behavior of the command. ### \--taskqueue[​](https://docs.temporal.io/tctl-v1/taskqueue#--taskqueue "Direct link to --taskqueue") _Required modifier_ Specify a [Task Queue](https://docs.temporal.io/task-queue) . Alias: `--t` **Example** tctl taskqueue describe --taskqueue ### \--taskqueuetype[​](https://docs.temporal.io/tctl-v1/taskqueue#--taskqueuetype "Direct link to --taskqueuetype") Specify the type of a [Task Queue](https://docs.temporal.io/task-queue) . The type can be `workflow` or `activity`. The default is `workflow`. **Example** tctl taskqueue describe --taskqueue --taskqueuetype list-partition[​](https://docs.temporal.io/tctl-v1/taskqueue#list-partition "Direct link to list-partition") ------------------------------------------------------------------------------------------------------------- The `tctl taskqueue list-partition` command lists the partitions of a [Task Queue](https://docs.temporal.io/task-queue) and the hostname for the partitions. `tctl taskqueue list-partition --taskqueue ` The following modifier controls the behavior of the command. ### \--taskqueue[​](https://docs.temporal.io/tctl-v1/taskqueue#--taskqueue-1 "Direct link to --taskqueue") _Required modifier_ Specify a [Task Queue](https://docs.temporal.io/task-queue) description. Alias: `--t` **Example** tctl taskqueue list-partition --taskqueue * [describe](https://docs.temporal.io/tctl-v1/taskqueue#describe) * [\--taskqueue](https://docs.temporal.io/tctl-v1/taskqueue#--taskqueue) * [\--taskqueuetype](https://docs.temporal.io/tctl-v1/taskqueue#--taskqueuetype) * [list-partition](https://docs.temporal.io/tctl-v1/taskqueue#list-partition) * [\--taskqueue](https://docs.temporal.io/tctl-v1/taskqueue#--taskqueue-1) --- # Error handling and troubleshooting | Temporal Platform Documentation [Skip to main content](https://docs.temporal.io/troubleshooting#__docusaurus_skipToContent_fallback) Even the most reliable systems can encounter issues. Our troubleshooting guides are designed to help you quickly identify and resolve potential errors, ensuring your Temporal applications run smoothly and efficiently. * [Troubleshoot the BlobSizeLimitError](https://docs.temporal.io/troubleshooting/blob-size-limit-error) : The `BlobSizeLimitError` happens when the size of a blob (payloads including Workflow context and each Workflow and Activity argument and return value) is too large. The maximum payload for a single request is 2 MB, and the maximum size for any Event History transaction is 4 MB. * [Troubleshoot the Deadline-Exceeded Error](https://docs.temporal.io/troubleshooting/deadline-exceeded-error) : The "Context: deadline exceeded" error occurs when requests to the Temporal Service by the Client or Worker cannot be completed. This can be due to network issues, timeouts, server overload, or Query errors. * [Troubleshoot the Failed Reaching Server Error](https://docs.temporal.io/troubleshooting/last-connection-error) : The message "Failed reaching server: last connection error" often happens due to an expired TLS certificate or during the Server startup process when Client requests reach the Server before roles are fully initialized. --- # Quick launch - Deploying your Workers on Amazon EKS | Temporal Platform Documentation [Skip to main content](https://docs.temporal.io/production-deployment/worker-deployments/deploy-workers-to-aws-eks#__docusaurus_skipToContent_fallback) On this page Temporal Workers running in [Kubernetes](https://kubernetes.io/) \-based deployments deliver scale, resilience, and flexible resource management. Amazon EKS (Elastic Kubernetes Service) offers one of the most popular choices for running Temporal Workers. It integrates smoothly with AWS services and supports auto-scaling and fault tolerance—key features for many Temporal users. Follow this guide to deploy and manage your Temporal Workers in EKS. This guide walks you through writing Temporal Worker code, containerizing and publishing the Worker to the Amazon Elastic Container Registry (ECR), and deploying the worker to Amazon EKS. The example on this page uses Temporal’s Python SDK and Temporal Cloud. For production Kubernetes deployments that use [Worker Versioning](https://docs.temporal.io/production-deployment/worker-deployments/worker-versioning) , use the [Temporal Worker Controller](https://docs.temporal.io/production-deployment/worker-deployments/kubernetes-controller) so deployment rollouts and autoscaling stay attached to each Worker Deployment Version. tip This guide applies to running Workers for both Temporal OSS and Temporal Cloud. However, there are some differences when working with Temporal OSS. For example, you'll need to use mTLS certificates instead of API keys. You must modify your Kubernetes deployments to handle and mount the TLS certificates for your use case. The specifics will vary depending on your deployment. Before you begin[​](https://docs.temporal.io/production-deployment/worker-deployments/deploy-workers-to-aws-eks#before-you-begin "Direct link to Before you begin") -------------------------------------------------------------------------------------------------------------------------------------------------------------------- To get started deploying your Workers to EKS, you’ll need: * Your Temporal Cloud account, including: * A Namespace using [API key authentication](https://docs.temporal.io/cloud/api-keys#namespace-authentication) * Your API Key for a [Service Account](https://docs.temporal.io/cloud/api-keys#generate-an-api-key-for-a-service-account) * An Amazon Web Services (AWS) account, including: * A deployed EKS cluster within your AWS Account * An installed version of the [`aws` CLI](https://aws.amazon.com/cli/) * [`docker`](https://www.docker.com/get-started/) * The [`kubectl`](https://kubernetes.io/docs/reference/kubectl/) command line tool, configured with your deployed EKS cluster Write your Worker code[​](https://docs.temporal.io/production-deployment/worker-deployments/deploy-workers-to-aws-eks#write-your-worker-code "Direct link to Write your Worker code") -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- In Temporal applications, business logic lives within your main Workflow code. Your Worker code runs separately, and is responsible for executing your Workflows and Activities. Make sure to configure your Worker to use environment variables so you can dynamically route your Worker to different Temporal Instances, Namespaces, and Task Queues on the fly: TEMPORAL_ADDRESS = os.environ.get("TEMPORAL_ADDRESS", "localhost:7233")TEMPORAL_NAMESPACE = os.environ.get("TEMPORAL_NAMESPACE", "default")TEMPORAL_TASK_QUEUE = os.environ.get("TEMPORAL_TASK_QUEUE", "test-task-queue")TEMPORAL_API_KEY = os.environ.get("TEMPORAL_API_KEY", "") After configuration, instantiate your Temporal client: client = await Client.connect( TEMPORAL_ADDRESS, namespace=TEMPORAL_NAMESPACE, rpc_metadata={"temporal-namespace": TEMPORAL_NAMESPACE}, api_key=TEMPORAL_API_KEY, tls=True) Here is a complete Python boilerplate that showcases how to instantiate a Client and pass it to the Worker before starting the Worker execution: import asyncioimport osfrom temporalio.worker import Workerfrom temporalio.client import Clientfrom workflows import your_workflowfrom activities import your_first_activity, your_second_activity, your_third_activityTEMPORAL_ADDRESS = os.environ.get("TEMPORAL_ADDRESS", "localhost:7233")TEMPORAL_NAMESPACE = os.environ.get("TEMPORAL_NAMESPACE", "default")TEMPORAL_TASK_QUEUE = os.environ.get("TEMPORAL_TASK_QUEUE", "test-task-queue")TEMPORAL_API_KEY = os.environ.get("TEMPORAL_API_KEY", "your-api-key")async def main(): client = await Client.connect( TEMPORAL_ADDRESS, namespace=TEMPORAL_NAMESPACE, rpc_metadata={"temporal-namespace": TEMPORAL_NAMESPACE}, api_key=TEMPORAL_API_KEY, tls=True ) print("Initializing worker...") # Run the worker worker = Worker( client, task_queue=TEMPORAL_TASK_QUEUE, workflows=[your_workflow], activities=[ your_first_activity, your_second_activity, your_third_activity ] ) print("Starting worker... Waiting for tasks.") await worker.run()if __name__ == "__main__": asyncio.run(main()) Containerize the Worker for Kubernetes[​](https://docs.temporal.io/production-deployment/worker-deployments/deploy-workers-to-aws-eks#containerize-the-worker-for-kubernetes "Direct link to Containerize the Worker for Kubernetes") -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- You need to containerize your Worker code to run it with Kubernetes. Here is a sample Python Dockerfile, complete with the Temporal Python SDK installed: # Use Python 3.11 slim image as baseFROM python:3.11-slim# Set working directoryWORKDIR /app# Install system dependenciesRUN apt-get update && apt-get install -y \ gcc \ && rm -rf /var/lib/apt/lists/*# Install the Temporal Python SDK dependencyRUN pip install --no-cache-dir temporalio# Copy application codeCOPY . .# Set Python to run in unbuffered modeENV PYTHONUNBUFFERED=1# Run the workerCMD ["python", "worker.py"] Build the Docker image and target the `linux/amd64` architecture: docker buildx build \ --platform linux/amd64 \ -t your-app . Publish the Worker Image to Amazon ECR[​](https://docs.temporal.io/production-deployment/worker-deployments/deploy-workers-to-aws-eks#publish-the-worker-image-to-amazon-ecr "Direct link to Publish the Worker Image to Amazon ECR") -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- After building the Docker image, you’re ready to publish it to Amazon ECR. Make sure that you’re authenticated with AWS, and that you’ve set your `AWS_REGION` and `AWS_ACCOUNT_ID` environment variables: export AWS_ACCOUNT_ID=export AWS_REGION= Create an ECR repository and authenticate ECR with the Docker container client: aws ecr create-repository \ --repository-name your-appaws ecr get-login-password --region $AWS_REGION | \ docker login --username AWS --password-stdin \ $AWS_ACCOUNT_ID.dkr.ecr.$AWS_REGION.amazonaws.com After authenticating Docker with ECR, tag your container and publish it: docker tag your-app $AWS_ACCOUNT_ID.dkr.ecr.$AWS_REGION.amazonaws.com/your-app:latestdocker push $AWS_ACCOUNT_ID.dkr.ecr.$AWS_REGION.amazonaws.com/your-app:latest Deploy the Workers to EKS[​](https://docs.temporal.io/production-deployment/worker-deployments/deploy-workers-to-aws-eks#deploy-the-workers-to-eks "Direct link to Deploy the Workers to EKS") ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- With your Worker containerized, you’re ready to deploy it to EKS. Create a namespace in your EKS cluster. You’ll use the namespace to run your Temporal Workers: kubectl create namespace your-namespace Create a `ConfigMap` to hold non-sensitive values that Kubernetes will inject into your Worker deployment. These enable dynamic routing for instances, Namespaces, and Task Queues. To set these values, build a config-map.yaml file like the following example: apiVersion: v1kind: ConfigMapmetadata: name: temporal-worker-config namespace: temporal-systemdata: TEMPORAL_HOST_URL: ““ TEMPORAL_NAMESPACE: “” TEMPORAL_TASK_QUEUE: “” Apply the `ConfigMap` to your namespace: kubectl apply -f config-map.yaml \ --namespace your-namespace For sensitive values, use Kubernetes Secrets. Create a secret to hold your Temporal API key: kubectl create secret generic temporal-secret \ --from-literal=TEMPORAL_API_KEY=$TEMPORAL_API_KEY \ --namespace your-namespace With your configuration in place, you can deploy the Worker. Create a deployment.yaml file configuring your Worker image, resources, and secret values. For common deployments, tune the resources you specify so they match your production workloads. Note that the spun-up container reads your Temporal API key from the Kubernetes secret you just created: apiVersion: apps/v1kind: Deploymentmetadata: name: your-app namespace: your-namespace labels: app: your-appspec: selector: matchLabels: app: your-app replicas: 1 template: metadata: labels: app: your-app spec: serviceAccountName: your-app containers: - name: your-app image: env: - name: TEMPORAL_ADDRESS valueFrom: configMapKeyRef: name: temporal-worker-config key: TEMPORAL_ADDRESS - name: TEMPORAL_NAMESPACE valueFrom: configMapKeyRef: name: temporal-worker-config key: TEMPORAL_NAMESPACE - name: TEMPORAL_TASK_QUEUE valueFrom: configMapKeyRef: name: temporal-worker-config key: TEMPORAL_TASK_QUEUE - name: TEMPORAL_API_KEY valueFrom: secretKeyRef: name: temporal-secret key: TEMPORAL_API_KEY resources: limits: cpu: "0.5" memory: "512Mi" requests: cpu: "0.2" memory: "256Mi" Apply the deployment.yaml file to the EKS cluster: kubectl apply -f deployment.yaml \ --namespace your-namespace Verify that the Workers are Connected[​](https://docs.temporal.io/production-deployment/worker-deployments/deploy-workers-to-aws-eks#verify-that-the-workers-are-connected "Direct link to Verify that the Workers are Connected") ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- After deploying your Workers to EKS, confirm that they have connected to Temporal Cloud. Retrieve the pod listing for the Kubernetes/EKS namespace that you created: kubectl get pods -n temporal-system After listing the pods, access the Worker logs to confirm you’re properly connected to Temporal Cloud: kubectl logs -n temporal-system You confirm connection when you see: Initializing worker...Starting worker... Waiting for tasks. You have now successfully deployed your Temporal Worker to EKS. * [Before you begin](https://docs.temporal.io/production-deployment/worker-deployments/deploy-workers-to-aws-eks#before-you-begin) * [Write your Worker code](https://docs.temporal.io/production-deployment/worker-deployments/deploy-workers-to-aws-eks#write-your-worker-code) * [Containerize the Worker for Kubernetes](https://docs.temporal.io/production-deployment/worker-deployments/deploy-workers-to-aws-eks#containerize-the-worker-for-kubernetes) * [Publish the Worker Image to Amazon ECR](https://docs.temporal.io/production-deployment/worker-deployments/deploy-workers-to-aws-eks#publish-the-worker-image-to-amazon-ecr) * [Deploy the Workers to EKS](https://docs.temporal.io/production-deployment/worker-deployments/deploy-workers-to-aws-eks#deploy-the-workers-to-eks) * [Verify that the Workers are Connected](https://docs.temporal.io/production-deployment/worker-deployments/deploy-workers-to-aws-eks#verify-that-the-workers-are-connected) --- # Troubleshoot the failed reaching server error | Temporal Platform Documentation [Skip to main content](https://docs.temporal.io/troubleshooting/last-connection-error#__docusaurus_skipToContent_fallback) On this page The message `Failed reaching server: last connection error` can often result from an expired TLS certificate or during the Server startup process, in which the Client requests reach the Server before the roles are fully initialized. This troubleshooting guide shows you how to do the following: * Verify the certification expiration date * Renew the certification * Update the server configuration ### Verify TLS certification expiration date[​](https://docs.temporal.io/troubleshooting/last-connection-error#verify-tls-certification-expiration-date "Direct link to Verify TLS certification expiration date") The first step in troubleshooting this error is to verify the expiration date of the TLS certification. Then you can renew the certification and update the server configuration. Choose one of the following methods to verify the expiration date of the TLS certification: **Verify the expiration date of the TLS certification** List the expiration date with the following command: tcld namespace accepted-client-ca list \ --namespace . | \ jq -r '.[0].notAfter' If the returned date is in the past, the certificate has expired. **Existing certificate management infrastructure** If you are using an existing certificate management infrastructure, use it to verify the TLS connection. For example, if you are using OpenSSL, run the following command: openssl s_client -connect -showcerts -cert ~/certs/path.pem -key .~/certs/path.key -tls1_2 **Self-signed certificate** If you are using a self-signed certificate, run the following Temporal CLI command: temporal namespace describe \ --namespace . \ --address \ --tls-cert-path \ --tls-key-path Your Namespace gRPC endpoint is available on the details page for your [Temporal Cloud Namespace](https://cloud.temporal.io/namespaces) . ### Renew TLS certification[​](https://docs.temporal.io/troubleshooting/last-connection-error#renew-tls-certification "Direct link to Renew TLS certification") If the certificate has expired or is about to expire, the next step is to renew it. You can do this by contacting the certificate authority (CA) that issued the certificate and requesting a renewal. **Existing certificate management infrastructure** If you are using an existing certificate management infrastructure, contact the administrator of the infrastructure to renew the certificate. **Self-signed certificate** If you are using a self-signed certificate or don't have an existing infrastructure, you can generate a new certificate using OpenSSL, [step CLI](https://github.com/smallstep/cli) , or similar tools. For information on generating a self-signed certificate, see [Control authorization](https://docs.temporal.io/cloud/certificates#control-authorization) . ### Update the CA certification in the server configuration[​](https://docs.temporal.io/troubleshooting/last-connection-error#update-the-ca-certification-in-the-server-configuration "Direct link to Update the CA certification in the server configuration") Update the new CA certificate in the Temporal Cloud server configuration. You can update certificates using any of the following methods: * [Update certificates using Temporal Cloud UI](https://docs.temporal.io/cloud/certificates#update-certificates-using-temporal-cloud-ui) * [Update certificates using tcld](https://docs.temporal.io/cloud/certificates#update-certificates-using-tcld) After you update the TLS certification in the server configuration, retry your connection. ### Set reminders[​](https://docs.temporal.io/troubleshooting/last-connection-error#set-reminders "Direct link to Set reminders") Don't let your certificates expire. Add reminders to your calendar to issue new CA certificates well before the expiration dates of the existing ones. ### Additional resources[​](https://docs.temporal.io/troubleshooting/last-connection-error#additional-resources "Direct link to Additional resources") The preceding steps should help you troubleshoot the `failed reaching server: last connection error` error caused by an expired TLS certificate. If this issue persists, verify that the Client you are using to connect to the server is using the correct TLS certification and that the Client requests reach the server after the roles are fully initialized. If you still need help, [create a support ticket](https://docs.temporal.io/cloud/support#support-ticket) . * [Verify TLS certification expiration date](https://docs.temporal.io/troubleshooting/last-connection-error#verify-tls-certification-expiration-date) * [Renew TLS certification](https://docs.temporal.io/troubleshooting/last-connection-error#renew-tls-certification) * [Update the CA certification in the server configuration](https://docs.temporal.io/troubleshooting/last-connection-error#update-the-ca-certification-in-the-server-configuration) * [Set reminders](https://docs.temporal.io/troubleshooting/last-connection-error#set-reminders) * [Additional resources](https://docs.temporal.io/troubleshooting/last-connection-error#additional-resources) --- # tctl v1.17 command reference | Temporal Platform Documentation [Skip to main content](https://docs.temporal.io/tctl-v1#__docusaurus_skipToContent_fallback) On this page tctl is deprecated The tctl command line utility has been deprecated and is no longer actively supported. We recommend transitioning to [Temporal CLI](https://docs.temporal.io/cli) for continued use and access to new features. Thank you for being a valued part of the Temporal community. note This documentation reflects tctl version 1.17. The Temporal CLI (tctl) is a command-line tool that you can use to interact with a Temporal Cluster. It can perform [Namespace](https://docs.temporal.io/namespaces) operations (such as register, update, and describe) and [Workflow](https://docs.temporal.io/workflows) operations (such as start Workflow, show Workflow History, and Signal Workflow). * [How to install tctl](https://docs.temporal.io/tctl-v1#install) * [Environment variables for tctl](https://docs.temporal.io/tctl-v1#environment-variables) tctl commands[​](https://docs.temporal.io/tctl-v1#tctl-commands "Direct link to tctl commands") ------------------------------------------------------------------------------------------------ * [tctl activity](https://docs.temporal.io/tctl-v1/activity/) * [tctl admin](https://docs.temporal.io/tctl-v1/admin/) * [tctl batch](https://docs.temporal.io/tctl-v1/batch/) * [tctl cluster](https://docs.temporal.io/tctl-v1/cluster/) * [tctl dataconverter](https://docs.temporal.io/tctl-v1/dataconverter/) * [tctl namespace](https://docs.temporal.io/tctl-v1/namespace/) * [tctl taskqueue](https://docs.temporal.io/tctl-v1/taskqueue/) * [tctl workflow](https://docs.temporal.io/tctl-v1/workflow/) How to install tctl[​](https://docs.temporal.io/tctl-v1#install "Direct link to How to install tctl") ------------------------------------------------------------------------------------------------------ > The Temporal tctl documentation covers version 1.17 of the Temporal CLI. You can install [tctl](https://docs.temporal.io/tctl-v1) in the following ways. * Install locally by using [Homebrew](https://brew.sh/) : `brew install tctl` * Run locally together with Temporal Server in [Docker Compose](https://github.com/temporalio/docker-compose) : `docker exec temporal-admin-tools tctl YOUR COMMANDS HERE` * To invoke [tctl](https://docs.temporal.io/tctl-v1) as though it is installed locally (such as `tctl namespace describe`), set an alias: `alias tctl="docker exec temporal-admin-tools tctl"` * Run the [temporal-admin-tools](https://hub.docker.com/r/temporalio/admin-tools) Docker image: * On Linux: `docker run --rm -it --entrypoint tctl --network host --env TEMPORAL_CLI_ADDRESS=localhost:7233 temporalio/admin-tools:1.14.0` * On macOS or Windows: `docker run --rm -it --entrypoint tctl --env TEMPORAL_CLI_ADDRESS=host.docker.internal:7233 temporalio/admin-tools:1.14.0` * If your Temporal Server is running on a remote host, change the value of `TEMPORAL_CLI_ADDRESS`. * To simplify command lines, create a `tctl` alias. * Install the latest version of the tctl in your `GOPATH`: `go install github.com/temporalio/tctl/cmd/tctl@latest` **Note:** To use [tctl](https://docs.temporal.io/tctl-v1) , you must have a Temporal Server running. To see help for [tctl](https://docs.temporal.io/tctl-v1) commands, enter the following commands. | Command | Description | | --- | --- | | `tctl -h` | Display help for top-level commands and global options | | `tctl namespace -h` | Display help for [Namespace](https://docs.temporal.io/namespaces)
operations | | `tctl workflow -h` | Display help for [Workflow](https://docs.temporal.io/workflows)
operations | | `tctl taskqueue -h` | Display help for [Task Queue](https://docs.temporal.io/task-queue)
operations | Global modifiers[​](https://docs.temporal.io/tctl-v1#global-modifiers "Direct link to Global modifiers") --------------------------------------------------------------------------------------------------------- You can supply the values for many of these modifiers by setting [environment variables](https://docs.temporal.io/tctl-v1#environment-variables) instead of including the modifiers in a tctl command. ### \--address[​](https://docs.temporal.io/tctl-v1#--address "Direct link to --address") Specify a host and port for the Frontend Service. The default is `127.0.0.1:7233`. ### \--auto\_confirm[​](https://docs.temporal.io/tctl-v1#--auto_confirm "Direct link to --auto_confirm") Automatically confirm all prompts. ### \--context\_timeout[​](https://docs.temporal.io/tctl-v1#--context_timeout "Direct link to --context_timeout") Specify a timeout for the context of an RPC call in seconds. The default value is 5. ### \--data\_converter\_plugin[​](https://docs.temporal.io/tctl-v1#--data_converter_plugin "Direct link to --data_converter_plugin") Specify the name of the executable for a custom Data Converter plugin. ### \--headers\_provider\_plugin[​](https://docs.temporal.io/tctl-v1#--headers_provider_plugin "Direct link to --headers_provider_plugin") Specify the name of the executable for a headers provider plugin. ### \--help[​](https://docs.temporal.io/tctl-v1#--help "Direct link to --help") Display help for tctl in the CLI. Alias: `-h` ### \--namespace[​](https://docs.temporal.io/tctl-v1#--namespace "Direct link to --namespace") Specify a Namespace. By using this modifier, you don't need to specify a `--namespace` modifier for a sub-command. The default Namespace is `default`. Alias: `--n` ### \--tls\_ca\_path[​](https://docs.temporal.io/tctl-v1#--tls_ca_path "Direct link to --tls_ca_path") Specify the path to a server Certificate Authority (CA) certificate file. ### \--tls\_cert\_path[​](https://docs.temporal.io/tctl-v1#--tls_cert_path "Direct link to --tls_cert_path") Specify the path to a public X.509 certificate file for mutual TLS authentication. If you use this modifier, you must also use the `--tls_key_path` modifier. ### \--tls\_disable\_host\_verification[​](https://docs.temporal.io/tctl-v1#--tls_disable_host_verification "Direct link to --tls_disable_host_verification") Disable verification of the server certificate (and thus host verification). ### \--tls\_key\_path[​](https://docs.temporal.io/tctl-v1#--tls_key_path "Direct link to --tls_key_path") Specify the path to a private key file for mutual TLS authentication. If you use this modifier, you must also use the `--tls_cert_path` modifier. ### \--tls\_server\_name[​](https://docs.temporal.io/tctl-v1#--tls_server_name "Direct link to --tls_server_name") Specify an override for the name of the target server that is used for TLS host verification. The name must be one of the DNS names listed in the server TLS certificate. Specifying this modifier also enables host verification. ### \--version[​](https://docs.temporal.io/tctl-v1#--version "Direct link to --version") Display the version of tctl in the CLI. ### \--codec\_endpoint[​](https://docs.temporal.io/tctl-v1#--codec_endpoint "Direct link to --codec_endpoint") The URL and port number for a Codec Server. Environment variables[​](https://docs.temporal.io/tctl-v1#environment-variables "Direct link to Environment variables") ------------------------------------------------------------------------------------------------------------------------ Setting environment variables for repeated parameters can shorten tctl commands. ### TEMPORAL\_CLI\_ADDRESS[​](https://docs.temporal.io/tctl-v1#temporal_cli_address "Direct link to TEMPORAL_CLI_ADDRESS") Specify a host and port for the Frontend Service. The default is `127.0.0.1:7233`. ### TEMPORAL\_CLI\_AUTHORIZATION\_TOKEN[​](https://docs.temporal.io/tctl-v1#temporal_cli_authorization_token "Direct link to TEMPORAL_CLI_AUTHORIZATION_TOKEN") Specify a token to be used by the HTTP Basic Authorization plugin. ### TEMPORAL\_CLI\_AUTH[​](https://docs.temporal.io/tctl-v1#temporal_cli_auth "Direct link to TEMPORAL_CLI_AUTH") Specify the authorization header to be set for a gRPC request. ### TEMPORAL\_CLI\_NAMESPACE[​](https://docs.temporal.io/tctl-v1#temporal_cli_namespace "Direct link to TEMPORAL_CLI_NAMESPACE") Specify a Namespace. By setting this variable, you don't need to specify a `--namespace` modifier in a tctl command. The default Namespace is `default`. ### TEMPORAL\_CLI\_PLUGIN\_DATA\_CONVERTER[​](https://docs.temporal.io/tctl-v1#temporal_cli_plugin_data_converter "Direct link to TEMPORAL_CLI_PLUGIN_DATA_CONVERTER") Specify the name of the executable for a custom Data Converter plugin. ### TEMPORAL\_CLI\_PLUGIN\_HEADERS\_PROVIDER[​](https://docs.temporal.io/tctl-v1#temporal_cli_plugin_headers_provider "Direct link to TEMPORAL_CLI_PLUGIN_HEADERS_PROVIDER") Specify the name of the executable for a headers provider plugin. ### TEMPORAL\_CLI\_TLS\_CA[​](https://docs.temporal.io/tctl-v1#temporal_cli_tls_ca "Direct link to TEMPORAL_CLI_TLS_CA") Specify the path to a server Certificate Authority (CA) certificate file. ### TEMPORAL\_CLI\_TLS\_CERT[​](https://docs.temporal.io/tctl-v1#temporal_cli_tls_cert "Direct link to TEMPORAL_CLI_TLS_CERT") Specify the path to a public X.509 certificate file for mutual TLS authentication. ### TEMPORAL\_CLI\_TLS\_DISABLE\_HOST\_VERIFICATION[​](https://docs.temporal.io/tctl-v1#temporal_cli_tls_disable_host_verification "Direct link to TEMPORAL_CLI_TLS_DISABLE_HOST_VERIFICATION") Set to disable verification of the server certificate (and thus host verification). ### TEMPORAL\_CLI\_TLS\_KEY[​](https://docs.temporal.io/tctl-v1#temporal_cli_tls_key "Direct link to TEMPORAL_CLI_TLS_KEY") Specify the path to a private key file for mutual TLS authentication. If you set this variable, you must also set the `TEMPORAL_CLI_TLS_CERT` variable. ### TEMPORAL\_CLI\_TLS\_SERVER\_NAME[​](https://docs.temporal.io/tctl-v1#temporal_cli_tls_server_name "Direct link to TEMPORAL_CLI_TLS_SERVER_NAME") Specify an override for the name of the target server that is used for TLS host verification. The name must be one of the DNS names listed in the server TLS certificate. Setting this variable also enables host verification. ### TEMPORAL\_CONTEXT\_TIMEOUT[​](https://docs.temporal.io/tctl-v1#temporal_context_timeout "Direct link to TEMPORAL_CONTEXT_TIMEOUT") Specify a timeout for the context of an RPC call in seconds. The default value is 5. * [tctl commands](https://docs.temporal.io/tctl-v1#tctl-commands) * [How to install tctl](https://docs.temporal.io/tctl-v1#install) * [Global modifiers](https://docs.temporal.io/tctl-v1#global-modifiers) * [\--address](https://docs.temporal.io/tctl-v1#--address) * [\--auto\_confirm](https://docs.temporal.io/tctl-v1#--auto_confirm) * [\--context\_timeout](https://docs.temporal.io/tctl-v1#--context_timeout) * [\--data\_converter\_plugin](https://docs.temporal.io/tctl-v1#--data_converter_plugin) * [\--headers\_provider\_plugin](https://docs.temporal.io/tctl-v1#--headers_provider_plugin) * [\--help](https://docs.temporal.io/tctl-v1#--help) * [\--namespace](https://docs.temporal.io/tctl-v1#--namespace) * [\--tls\_ca\_path](https://docs.temporal.io/tctl-v1#--tls_ca_path) * [\--tls\_cert\_path](https://docs.temporal.io/tctl-v1#--tls_cert_path) * [\--tls\_disable\_host\_verification](https://docs.temporal.io/tctl-v1#--tls_disable_host_verification) * [\--tls\_key\_path](https://docs.temporal.io/tctl-v1#--tls_key_path) * [\--tls\_server\_name](https://docs.temporal.io/tctl-v1#--tls_server_name) * [\--version](https://docs.temporal.io/tctl-v1#--version) * [\--codec\_endpoint](https://docs.temporal.io/tctl-v1#--codec_endpoint) * [Environment variables](https://docs.temporal.io/tctl-v1#environment-variables) * [TEMPORAL\_CLI\_ADDRESS](https://docs.temporal.io/tctl-v1#temporal_cli_address) * [TEMPORAL\_CLI\_AUTHORIZATION\_TOKEN](https://docs.temporal.io/tctl-v1#temporal_cli_authorization_token) * [TEMPORAL\_CLI\_AUTH](https://docs.temporal.io/tctl-v1#temporal_cli_auth) * [TEMPORAL\_CLI\_NAMESPACE](https://docs.temporal.io/tctl-v1#temporal_cli_namespace) * [TEMPORAL\_CLI\_PLUGIN\_DATA\_CONVERTER](https://docs.temporal.io/tctl-v1#temporal_cli_plugin_data_converter) * [TEMPORAL\_CLI\_PLUGIN\_HEADERS\_PROVIDER](https://docs.temporal.io/tctl-v1#temporal_cli_plugin_headers_provider) * [TEMPORAL\_CLI\_TLS\_CA](https://docs.temporal.io/tctl-v1#temporal_cli_tls_ca) * [TEMPORAL\_CLI\_TLS\_CERT](https://docs.temporal.io/tctl-v1#temporal_cli_tls_cert) * [TEMPORAL\_CLI\_TLS\_DISABLE\_HOST\_VERIFICATION](https://docs.temporal.io/tctl-v1#temporal_cli_tls_disable_host_verification) * [TEMPORAL\_CLI\_TLS\_KEY](https://docs.temporal.io/tctl-v1#temporal_cli_tls_key) * [TEMPORAL\_CLI\_TLS\_SERVER\_NAME](https://docs.temporal.io/tctl-v1#temporal_cli_tls_server_name) * [TEMPORAL\_CONTEXT\_TIMEOUT](https://docs.temporal.io/tctl-v1#temporal_context_timeout) --- # Temporal use cases and design patterns | Temporal Platform Documentation [Skip to main content](https://docs.temporal.io/evaluate/use-cases-design-patterns#__docusaurus_skipToContent_fallback) On this page This page provides an overview of how leading organizations leverage Temporal to solve real-world problems, general use cases, and architectural design patterns. Use Cases of Temporal in Production[​](https://docs.temporal.io/evaluate/use-cases-design-patterns#use-cases-of-temporal-in-production "Direct link to Use Cases of Temporal in Production") --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Here are some examples where Temporal is most impactful and running in production at large organizations today. For more examples, see our [Temporal Use Cases](https://temporal.io/in-use) page. ### Transactions[​](https://docs.temporal.io/evaluate/use-cases-design-patterns#transactions "Direct link to Transactions") Actions or activities involving two or more parties or things that reciprocally affect or influence each other. For example: * [Payment processing at Stripe](https://temporal.io/resources/on-demand/stripe) * [Money movement at Coinbase](https://temporal.io/in-use/coinbase) * [Content management at Box](https://temporal.io/resources/case-studies/box) ### Business processes[​](https://docs.temporal.io/evaluate/use-cases-design-patterns#business-processes "Direct link to Business processes") A sequence of tasks that find their end in the delivery of a service or product to a client. For example: * [Bookings at Turo](https://temporal.io/replay/videos/temporal-adoption-and-integration-at-turo) * [Orders/logistics at Maersk](https://temporal.io/replay/videos/building-a-time-machine-for-the-logistics-industry) * [Marketing Campaigns at AirBnb](https://medium.com/airbnb-engineering/journey-platform-a-low-code-tool-for-creating-interactive-user-workflows-9954f51fa3f8) * [Human-in-the-loop at Checkr](https://temporal.io/in-use/checkr) ### Entity lifecycle[​](https://docs.temporal.io/evaluate/use-cases-design-patterns#entity-lifecycle "Direct link to Entity lifecycle") Complex long-running processes that accumulate state over time. For example: * [Mortgage underwriting applications at ANZ](https://temporal.io/in-use/anz-story) * [Menu versioning at Yum! Brands](https://temporal.io/replay-2023/videos/synchronizing-concurrent-workflows) ### Operations[​](https://docs.temporal.io/evaluate/use-cases-design-patterns#operations "Direct link to Operations") An automated method for getting a repeatable, mundane task accomplished. For example: * [Infrastructure services at DataDog](https://www.youtube.com/watch?v=Hz7ZZzafBoE) * [Custom CI/CD at Netflix](https://temporal.io/replay-2023/videos/actor-workflows-reliably-orchestrating-thousands-of-flink-clusters-at) ### AI / ML and Data Engineering[​](https://docs.temporal.io/evaluate/use-cases-design-patterns#ai--ml-and-data-engineering "Direct link to AI / ML and Data Engineering") AI and ML developers face challenges in system orchestration, such as managing complex data pipelines and job coordination across GPU resources. Temporal's code-first approach helps build reliable services faster, making it popular among AI companies. For example: * [Orchestrating video processing at Descript](https://temporal.io/blog/ai-ml-and-data-engineering-workflows-with-temporal#descript) * [Automating data pipelines at Neosync](https://temporal.io/blog/ai-ml-and-data-engineering-workflows-with-temporal#neosync) ### AI Agents[​](https://docs.temporal.io/evaluate/use-cases-design-patterns#ai-agents "Direct link to AI Agents") AI Agents present new uses for Temporal, such as maintaining state over long periods and enabling seamless human intervention when needed. Temporal ensures Durable Execution of tools, LLMs, and conversations, letting you focus on business logic instead of handling failures. For example: * [Creating reliable, observable Agents at Lindy](https://temporal.io/resources/case-studies/lindy-reliability-observability-ai-agents-temporal-cloud) * [Long-running, durable Agents at Dust](https://temporal.io/blog/how-dust-builds-agentic-ai-temporal) * [Creating account summaries with Agents at ZoomInfo](https://temporal.io/resources/on-demand/account-summaries-gen-ai) General Use Cases[​](https://docs.temporal.io/evaluate/use-cases-design-patterns#general-use-cases "Direct link to General Use Cases") --------------------------------------------------------------------------------------------------------------------------------------- ### Human in the Loop[​](https://docs.temporal.io/evaluate/use-cases-design-patterns#human-in-the-loop "Direct link to Human in the Loop") "Human in the Loop" systems require human interaction for certain steps, such as customer onboarding, forms, or invoice approval. These are event-driven systems with humans generating events, and may be challenging to implement due to timing or unreliable connections between the human to the rest of the system. They can use schedules and timers to prompt for user input. **Example**: [Background checks example using the Go SDK](https://learn.temporal.io/examples/go/background-checks/) . **Code Sample**: [Candidate acceptance example prompting for a response](https://learn.temporal.io/examples/go/background-checks/candidate-acceptance) ### Polyglot Systems[​](https://docs.temporal.io/evaluate/use-cases-design-patterns#polyglot-systems "Direct link to Polyglot Systems") Modern development teams often work with different programming languages based on their expertise and project requirements. Temporal supports this through built-in multi-language capabilities, allowing teams to continue using their preferred languages while working together. The example below showcases how Workflow Executions, written in different languages, can send messages to each other. Go, Java, PHP, and TypeScript SDKs are represented in this sample. It also shows how to properly propagate errors, including how to do so across Workflows written in different languages. **Example**: [Polyglot example](https://github.com/temporalio/temporal-polyglot) . ### Long Running Tasks[​](https://docs.temporal.io/evaluate/use-cases-design-patterns#long-running-tasks "Direct link to Long Running Tasks") This use case is particularly relevant for scenarios like shopping cart Workflows in an eCommerce app, where you can handle long-running tasks efficiently without managing state in a separate database. It processes one message at a time, ensuring each message is processed only once. This approach addresses issues that can arise with long message processing times, which in other systems might cause consumer failover (typically with a default 5-minute message poll timeout) and potentially result in duplicate message processing by multiple consumers. Temporal's ability to handle extended task durations makes it well-suited for such scenarios. The [heartbeat](https://docs.temporal.io/encyclopedia/detecting-activity-failures#activity-heartbeat) feature allows you to know that an activity is still working, providing insight into the progress of long-running processes. **Example**: [eCommerce example](https://learn.temporal.io/tutorials/go/build-an-ecommerce-app/) . **Code Sample**: [Temporal eCommerce](https://github.com/temporalio/temporal-ecommerce) Design Patterns[​](https://docs.temporal.io/evaluate/use-cases-design-patterns#design-patterns "Direct link to Design Patterns") --------------------------------------------------------------------------------------------------------------------------------- ### Saga[​](https://docs.temporal.io/evaluate/use-cases-design-patterns#saga "Direct link to Saga") The Saga pattern is a design pattern used to manage and handle failures in complex Workflows by breaking down a transaction into a series of smaller, manageable sub-transactions. If a step in the Workflow fails, the Saga pattern compensates for this failure by executing specific actions to undo the previous steps. This ensures that even in the event of a failure, the system can revert to a consistent state. **Examples:** * [Build a trip booking application in Python](https://learn.temporal.io/tutorials/python/trip-booking-app/) . * [Saga Pattern with Temporal Whitepaper](https://pages.temporal.io/download-saga-pattern-made-easy) * [To choreograph or orchestrate your saga, that is the question](https://temporal.io/blog/to-choreograph-or-orchestrate-your-saga-that-is-the-question) * [Saga Webinar](https://pages.temporal.io/on-demand-webinar-what-is-a-saga.html) ### State Machine[​](https://docs.temporal.io/evaluate/use-cases-design-patterns#state-machine "Direct link to State Machine") A state machine is a software design pattern used to modify a system’s behavior in response to changes in its state. While state machines are widely used in software development, applying them to complex business processes can be a difficult undertaking. Temporal simplifies the complexity of state machines by providing a structured approach to workflow development, avoiding the intricate state management code required for state machines. **Example**: [State Machine Simplified Whitepaper](https://pages.temporal.io/download-state-machines-simplified.html) tip If you're interested in code to help get you started, check out our [Temporal Example Applications](https://learn.temporal.io/examples/) , [Getting Started Tutorials](https://learn.temporal.io/getting_started/) , or [Project-based Tutorials](https://learn.temporal.io/tutorials/) . * [Use Cases of Temporal in Production](https://docs.temporal.io/evaluate/use-cases-design-patterns#use-cases-of-temporal-in-production) * [Transactions](https://docs.temporal.io/evaluate/use-cases-design-patterns#transactions) * [Business processes](https://docs.temporal.io/evaluate/use-cases-design-patterns#business-processes) * [Entity lifecycle](https://docs.temporal.io/evaluate/use-cases-design-patterns#entity-lifecycle) * [Operations](https://docs.temporal.io/evaluate/use-cases-design-patterns#operations) * [AI / ML and Data Engineering](https://docs.temporal.io/evaluate/use-cases-design-patterns#ai--ml-and-data-engineering) * [AI Agents](https://docs.temporal.io/evaluate/use-cases-design-patterns#ai-agents) * [General Use Cases](https://docs.temporal.io/evaluate/use-cases-design-patterns#general-use-cases) * [Human in the Loop](https://docs.temporal.io/evaluate/use-cases-design-patterns#human-in-the-loop) * [Polyglot Systems](https://docs.temporal.io/evaluate/use-cases-design-patterns#polyglot-systems) * [Long Running Tasks](https://docs.temporal.io/evaluate/use-cases-design-patterns#long-running-tasks) * [Design Patterns](https://docs.temporal.io/evaluate/use-cases-design-patterns#design-patterns) * [Saga](https://docs.temporal.io/evaluate/use-cases-design-patterns#saga) * [State Machine](https://docs.temporal.io/evaluate/use-cases-design-patterns#state-machine) --- # tctl v1.17 workflow command reference | Temporal Platform Documentation [Skip to main content](https://docs.temporal.io/tctl-v1/workflow#__docusaurus_skipToContent_fallback) On this page tctl is deprecated The tctl command line utility has been deprecated and is no longer actively supported. We recommend transitioning to [Temporal CLI](https://docs.temporal.io/cli) for continued use and access to new features. Thank you for being a valued part of the Temporal community. The `tctl workflow` commands enable [Workflow Execution](https://docs.temporal.io/workflow-execution) operations. * [tctl workflow cancel](https://docs.temporal.io/tctl-v1/workflow#cancel) * [tctl workflow count](https://docs.temporal.io/tctl-v1/workflow#count) * [tctl workflow describe](https://docs.temporal.io/tctl-v1/workflow#describe) * [tctl workflow describeid](https://docs.temporal.io/tctl-v1/workflow#describeid) * [tctl workflow list](https://docs.temporal.io/tctl-v1/workflow#list) * [tctl workflow listall](https://docs.temporal.io/tctl-v1/workflow#listall) * [tctl workflow listarchived](https://docs.temporal.io/tctl-v1/workflow#listarchived) * [tctl workflow observe](https://docs.temporal.io/tctl-v1/workflow#observe) * [tctl workflow observeid](https://docs.temporal.io/tctl-v1/workflow#observeid) * [tctl workflow query](https://docs.temporal.io/tctl-v1/workflow#query) * [tctl workflow reset](https://docs.temporal.io/tctl-v1/workflow#reset) * [tctl workflow reset-batch](https://docs.temporal.io/tctl-v1/workflow#reset-batch) * [tctl workflow run](https://docs.temporal.io/tctl-v1/workflow#run) * [tctl workflow scan](https://docs.temporal.io/tctl-v1/workflow#scan) * [tctl workflow show](https://docs.temporal.io/tctl-v1/workflow#show) * [tctl workflow showid](https://docs.temporal.io/tctl-v1/workflow#showid) * [tctl workflow signal](https://docs.temporal.io/tctl-v1/workflow#signal) * [tctl workflow stack](https://docs.temporal.io/tctl-v1/workflow#stack) * [tctl workflow start](https://docs.temporal.io/tctl-v1/workflow#start) * [tctl workflow terminate](https://docs.temporal.io/tctl-v1/workflow#terminate) cancel[​](https://docs.temporal.io/tctl-v1/workflow#cancel "Direct link to cancel") ------------------------------------------------------------------------------------ The `tctl workflow cancel --query` command cancels a [Workflow Execution](https://docs.temporal.io/workflow-execution) . Canceling a running Workflow Execution records a `WorkflowExecutionCancelRequested` event in the History. A new [Workflow Task](https://docs.temporal.io/tasks#workflow-task) will be scheduled. After cancellation, the Workflow Execution can perform cleanup work. See also [`tctl workflow terminate --query`](https://docs.temporal.io/tctl-v1/workflow#terminate) . `tctl workflow cancel --query ` The following modifiers control the behavior of the command. ### \--workflow\_id[​](https://docs.temporal.io/tctl-v1/workflow#--workflow_id "Direct link to --workflow_id") Specify a [Workflow Id](https://docs.temporal.io/workflow-execution/workflowid-runid#workflow-id) . Alias: `-w` **Example** tctl workflow cancel --workflow_id ### \--run\_id[​](https://docs.temporal.io/tctl-v1/workflow#--run_id "Direct link to --run_id") Specify a [Run Id](https://docs.temporal.io/workflow-execution/workflowid-runid#run-id) . Alias: `-r` **Example** tctl workflow cancel --run_id count[​](https://docs.temporal.io/tctl-v1/workflow#count "Direct link to count") --------------------------------------------------------------------------------- The `tctl workflow count` command counts [Workflow Executions](https://docs.temporal.io/workflow-execution) . This command requires Elasticsearch to be enabled. `tctl workflow count ` The following modifier controls the behavior of the command. ### \--query[​](https://docs.temporal.io/tctl-v1/workflow#--query "Direct link to --query") _Required modifier_ Specify an SQL-like query of [Search Attributes](https://docs.temporal.io/search-attribute) . Alias: `-q` **Example** To count all open [Workflow Executions](https://docs.temporal.io/workflow-execution) : tctl workflow count --query 'ExecutionStatus="Running"' describe[​](https://docs.temporal.io/tctl-v1/workflow#describe "Direct link to describe") ------------------------------------------------------------------------------------------ The `tctl workflow describe` command shows information about a [Workflow Execution](https://docs.temporal.io/workflow-execution) . This information can be used to locate a failed Workflow Execution, for example. To find a Workflow with a given Run Id, refer to [`tctl workflow describeid`](https://docs.temporal.io/tctl-v1/workflow#describeid) . `tctl workflow describe ` The following modifiers control the behavior of the command. Always include required modifiers when executing this command. ### \--workflow\_id[​](https://docs.temporal.io/tctl-v1/workflow#--workflow_id-1 "Direct link to --workflow_id") **This is a required modifier.** Specify a [Workflow Id](https://docs.temporal.io/workflow-execution/workflowid-runid#workflow-id) . Alias: `-w` **Example** tctl workflow describe --workflow_id ### \--run\_id[​](https://docs.temporal.io/tctl-v1/workflow#--run_id-1 "Direct link to --run_id") Specify a [Run Id](https://docs.temporal.io/workflow-execution/workflowid-runid#run-id) . If a Run Id is not provided, the command will show the latest Workflow Execution of that Workflow Id. Alias: `-r` **Example** tctl workflow describe --run_id ### \--print\_raw[​](https://docs.temporal.io/tctl-v1/workflow#--print_raw "Direct link to --print_raw") Print properties exactly as they are stored. **Example** tctl workflow describe --print_raw ### \--reset\_points\_only[​](https://docs.temporal.io/tctl-v1/workflow#--reset_points_only "Direct link to --reset_points_only") Show only events that are eligible for reset. If successful, the command returns the Run Id of all deployments, and the times at which the Events were created. **Example** tctl workflow describe --reset_points_only describeid[​](https://docs.temporal.io/tctl-v1/workflow#describeid "Direct link to describeid") ------------------------------------------------------------------------------------------------ The `tctl workflow describeid` command shows information about a [Workflow Execution](https://docs.temporal.io/workflow-execution) for the specified [Workflow Id](https://docs.temporal.io/workflow-execution/workflowid-runid#workflow-id) and optional [Run Id](https://docs.temporal.io/workflow-execution/workflowid-runid#run-id) . `tctl workflow describeid ` This command is a shortcut for `tctl workflow describe --workflow_id --run_id `. The following modifiers control the behavior of the command. ### \--print\_raw[​](https://docs.temporal.io/tctl-v1/workflow#--print_raw-1 "Direct link to --print_raw") Print properties exactly as they are stored. **Example** tctl workflow describeid --print_raw ### \--reset\_points\_only[​](https://docs.temporal.io/tctl-v1/workflow#--reset_points_only-1 "Direct link to --reset_points_only") Show only events that are eligible for reset. **Example** tctl workflow describeid --reset_points_only list[​](https://docs.temporal.io/tctl-v1/workflow#list "Direct link to list") ------------------------------------------------------------------------------ The `tctl workflow list` command lists open or closed [Workflow Executions](https://docs.temporal.io/workflow-execution) . By default, this command lists a maximum of 10 closed Workflow Executions. * To set the size of a page, use the `--pagesize` option. * To list multiple pages, use the `--more` option. * To list open Workflow Executions, use the `--open` option. See also [`tctl workflow listall`](https://docs.temporal.io/tctl-v1/workflow#listall) , [`tctl workflow listarchived`](https://docs.temporal.io/tctl-v1/workflow#listarchived) , and [`tctl workflow scan`](https://docs.temporal.io/tctl-v1/workflow#scan) . `tctl workflow list ` The following modifiers control the behavior of the command. ### \--print\_raw\_time[​](https://docs.temporal.io/tctl-v1/workflow#--print_raw_time "Direct link to --print_raw_time") Print the raw timestamp. **Example** tctl workflow list --print_raw_time ### \--print\_datetime[​](https://docs.temporal.io/tctl-v1/workflow#--print_datetime "Direct link to --print_datetime") Print the timestamp. **Example** tctl workflow list --print_datetime ### \--print\_memo[​](https://docs.temporal.io/tctl-v1/workflow#--print_memo "Direct link to --print_memo") Print a memo. **Example** tctl workflow list --print_memo ### \--print\_search\_attr[​](https://docs.temporal.io/tctl-v1/workflow#--print_search_attr "Direct link to --print_search_attr") Print the [Search Attributes](https://docs.temporal.io/search-attribute) . **Example** tctl workflow list --print_search_attr ### \--print\_full[​](https://docs.temporal.io/tctl-v1/workflow#--print_full "Direct link to --print_full") Print the full message without table formatting. **Example** tctl workflow list --print_full ### \--print\_json[​](https://docs.temporal.io/tctl-v1/workflow#--print_json "Direct link to --print_json") Print the raw JSON objects. **Example** tctl workflow list --print_json ### \--open[​](https://docs.temporal.io/tctl-v1/workflow#--open "Direct link to --open") List open [Workflow Executions](https://docs.temporal.io/workflow-execution) . (By default, the `tctl workflow list` command lists closed Workflow Executions.) **Example** tctl workflow list --open ### \--earliest\_time[​](https://docs.temporal.io/tctl-v1/workflow#--earliest_time "Direct link to --earliest_time") Specify the earliest start time to list. Supported format are as follows: * `--T::<+|->:` * Raw Unix Epoch time (the number of milliseconds since 0000 UTC on January 1, 1970). * `` is a value between 0 and 1000000, and `` is one of the following: * `second` or `s` * `minute` or `m` * `hour` or `h` * `day` or `d` * `week` or `w` * `month` or `M` * `year` or `y` **Examples** To specify 3:04:05 PM India Standard Time on January 2, 2022: tctl workflow list --earliest-time '2022-01-02T15:04:05+05:30' To specify 15 minutes before the current time: tctl workflow list --earliest-time '15minute' ### \--latest\_time[​](https://docs.temporal.io/tctl-v1/workflow#--latest_time "Direct link to --latest_time") Specify the latest start time to list. Supported formats are as follows: * `--T::<+|->:` * Raw Unix Epoch time (the number of milliseconds since 0000 UTC on January 1, 1970). * `` is a value between 0 and 1000000, and `` is one of the following: * `second` or `s` * `minute` or `m` * `hour` or `h` * `day` or `d` * `week` or `w` * `month` or `M` * `year` or `y` **Examples** To specify 11:02:17 PM Pacific Daylight Time on April 13, 2022: tctl workflow list --latest_time '2022-04-13T23:02:17-07:00' To specify 10s before the current time: tctl workflow list --latest_time '10second' ### \--workflow\_id[​](https://docs.temporal.io/tctl-v1/workflow#--workflow_id-2 "Direct link to --workflow_id") Specify a [Workflow Id](https://docs.temporal.io/workflow-execution/workflowid-runid#workflow-id) . Alias: `-w` **Example** tctl workflow list --workflow_id ### \--workflow\_type[​](https://docs.temporal.io/tctl-v1/workflow#--workflow_type "Direct link to --workflow_type") Specify the name of a [Workflow Type](https://docs.temporal.io/workflow-definition#workflow-type) . **Example** tctl workflow list --workflow_type ### \--status[​](https://docs.temporal.io/tctl-v1/workflow#--status "Direct link to --status") Specify the status of a [Workflow Execution](https://docs.temporal.io/workflow-execution) . Supported values are as follows: * `completed` * `failed` * `canceled` * `terminated` * `continuedasnew` * `timedout` **Example** tctl workflow list --status ### \--query[​](https://docs.temporal.io/tctl-v1/workflow#--query-1 "Direct link to --query") **How to list and filter Workflow Executions with a [List Filter](https://docs.temporal.io/list-filter) using tctl.** The `--query` flag is supported only when [Advanced Visibility](https://docs.temporal.io/visibility#advanced-visibility) is configured with the Cluster. Using the `--query` option causes tctl to ignore all other filter options, including `open`, `earliest_time`, `latest_time`, `workflow_id`, and `workflow_type`. Alias: `-q` **Example** tctl workflow list --query "WorkflowId=" More examples: tctl workflow list \ --query "WorkflowType='main.SampleParentWorkflow' AND ExecutionStatus='Running'" tctl workflow list \ --query '(CustomKeywordField = "keyword1" and CustomIntField >= 5) or CustomKeywordField = "keyword2"' \ --print_search_attr tctl workflow list \ --query 'CustomKeywordField in ("keyword2", "keyword1") and CustomIntField >= 5 and CloseTime between "2018-06-07T16:16:36-08:00" and "2019-06-07T16:46:34-08:00" order by CustomDatetimeField desc' \ --print_search_attr tctl workflow list \ --query 'WorkflowType = "main.Workflow" and (WorkflowId = "1645a588-4772-4dab-b276-5f9db108b3a8" or RunId = "be66519b-5f09-40cd-b2e8-20e4106244dc")' tctl workflow list \ --query 'WorkflowType = "main.Workflow" StartTime > "2019-06-07T16:46:34-08:00" and ExecutionStatus = "Running"' ### \--more[​](https://docs.temporal.io/tctl-v1/workflow#--more "Direct link to --more") List more than one page. (By default, the `tctl workflow list` command lists one page of results.) **Example** tctl workflow list --more ### \--pagesize[​](https://docs.temporal.io/tctl-v1/workflow#--pagesize "Direct link to --pagesize") Specify the maximum number of [Workflow Executions](https://docs.temporal.io/workflow-execution) to list on a page. (By default, the `tctl workflow list` command lists 10 Workflow Executions per page.) **Example** tctl workflow list --pagesize listall[​](https://docs.temporal.io/tctl-v1/workflow#listall "Direct link to listall") --------------------------------------------------------------------------------------- The `tctl workflow listall` command lists all open or closed [Workflow Executions](https://docs.temporal.io/workflow-execution) . By default, this command lists all closed Workflow Executions. To list open Workflow Executions, use the `--open` option. See also [`tctl workflow list`](https://docs.temporal.io/tctl-v1/workflow#list) , [`tctl workflow listarchived`](https://docs.temporal.io/tctl-v1/workflow#listarchived) , and [`tctl workflow scan`](https://docs.temporal.io/tctl-v1/workflow#scan) . `tctl workflow listall ` The following modifiers control the behavior of the command. ###\`--print\_raw\_time Print the raw timestamp. **Example** tctl workflow listall --print_raw_time ### \--print\_datetime[​](https://docs.temporal.io/tctl-v1/workflow#--print_datetime-1 "Direct link to --print_datetime") Print the timestamp. **Example** tctl workflow listall --print_datetime ### \--print\_memo[​](https://docs.temporal.io/tctl-v1/workflow#--print_memo-1 "Direct link to --print_memo") Print a memo. **Example** tctl workflow listall --print_memo ### \--print\_search\_attr[​](https://docs.temporal.io/tctl-v1/workflow#--print_search_attr-1 "Direct link to --print_search_attr") Print the [Search Attributes](https://docs.temporal.io/search-attribute) . **Example** tctl workflow listall --print_search_attr ### `--print_full`[​](https://docs.temporal.io/tctl-v1/workflow#--print_full-1 "Direct link to --print_full-1") Print the full message without table formatting. **Example** tctl workflow listall --print_full ### \--print\_json[​](https://docs.temporal.io/tctl-v1/workflow#--print_json-1 "Direct link to --print_json") Print the raw JSON objects. **Example** tctl workflow listall --print_json ### \--open[​](https://docs.temporal.io/tctl-v1/workflow#--open-1 "Direct link to --open") List open [Workflow Executions](https://docs.temporal.io/workflow-execution) . (By default, the `tctl workflow listall` command lists closed Workflow Executions.) **Example** tctl workflow listall --open ### \--earliest\_time[​](https://docs.temporal.io/tctl-v1/workflow#--earliest_time-1 "Direct link to --earliest_time") Specify the earliest start time to list. Supported format are as follows: * `--T::<+|->:` * Raw Unix Epoch time (the number of milliseconds since 0000 UTC on January 1, 1970). * `` is a value between 0 and 1000000, and `` is one of the following: * `second` or `s` * `minute` or `m` * `hour` or `h` * `day` or `d` * `week` or `w` * `month` or `M` * `year` or `y` **Examples** To specify 3:04:05 PM India Standard Time on January 2, 2022: tctl workflow listall --earliest-time '2022-01-02T15:04:05+05:30' To specify 15 minutes before the current time: tctl workflow listall --earliest-time '15minute' ### \--latest\_time[​](https://docs.temporal.io/tctl-v1/workflow#--latest_time-1 "Direct link to --latest_time") Specify the latest start time to list. Supported formats are as follows: * `--T::<+|->:` * Raw Unix Epoch time (the number of milliseconds since 0000 UTC on January 1, 1970). * `` is a value between 0 and 1000000, and `` is one of the following: * `second` or `s` * `minute` or `m` * `hour` or `h` * `day` or `d` * `week` or `w` * `month` or `M` * `year` or `y` Alias: `--lt` **Examples** To specify 11:02:17 PM Pacific Daylight Time on April 13, 2022: tctl workflow listall --latest-time '2022-04-13T23:02:17-07:00' To specify 10 seconds before the current time: tctl workflow listall --latest-time '10second' ### \--workflow\_id[​](https://docs.temporal.io/tctl-v1/workflow#--workflow_id-3 "Direct link to --workflow_id") Specify a [Workflow Id](https://docs.temporal.io/workflow-execution/workflowid-runid#workflow-id) . Alias: `-w` **Example** tctl workflow listall --workflow_id ### \--workflow\_type[​](https://docs.temporal.io/tctl-v1/workflow#--workflow_type-1 "Direct link to --workflow_type") Specify the name of a [Workflow Type](https://docs.temporal.io/workflow-definition#workflow-type) . **Example** tctl workflow listall --workflow_type ### \--status[​](https://docs.temporal.io/tctl-v1/workflow#--status-1 "Direct link to --status") Specify the status of a [Workflow Execution](https://docs.temporal.io/workflow-execution) . Supported values are as follows: * `completed` * `failed` * `canceled` * `terminated` * `continuedasnew` * `timedout` **Example** tctl workflow listall --status ### \--query[​](https://docs.temporal.io/tctl-v1/workflow#--query-2 "Direct link to --query") Specify an SQL-like query of [Search Attributes](https://docs.temporal.io/search-attribute) . Using the `--query` option causes tctl to ignore all other filter options, including `open`, `earliest_time`, `latest_time`, `workflow_id`, and `workflow_type`. Alias: `-q` **Example** tctl workflow listall --query listarchived[​](https://docs.temporal.io/tctl-v1/workflow#listarchived "Direct link to listarchived") ------------------------------------------------------------------------------------------------------ The `tctl workflow listarchived` command lists archived [Workflow Executions](https://docs.temporal.io/workflow-execution) . By default, this command lists a maximum of 100 Workflow Executions. * To set the size of a page, use the `--pagesize` option. * To list all pages, use the `--all` option. See also [`tctl workflow list`](https://docs.temporal.io/tctl-v1/workflow#list) , [`tctl workflow listall`](https://docs.temporal.io/tctl-v1/workflow#listall) , and [`tctl workflow scan`](https://docs.temporal.io/tctl-v1/workflow#scan) . `tctl workflow listarchived ` The following modifiers control the behavior of the command. ### \--print\_raw\_time[​](https://docs.temporal.io/tctl-v1/workflow#--print_raw_time-1 "Direct link to --print_raw_time") Print the raw timestamp. **Example** tctl workflow listarchived --print_raw_time ### \--print\_datetime[​](https://docs.temporal.io/tctl-v1/workflow#--print_datetime-2 "Direct link to --print_datetime") Print the timestamp. **Example** tctl workflow listarchived --print_datetime ### \--print\_memo[​](https://docs.temporal.io/tctl-v1/workflow#--print_memo-2 "Direct link to --print_memo") Print a memo. **Example** tctl workflow listarchived --print_memo ### \--print\_search\_attr[​](https://docs.temporal.io/tctl-v1/workflow#--print_search_attr-2 "Direct link to --print_search_attr") Print the [Search Attributes](https://docs.temporal.io/search-attribute) . **Example** tctl workflow listarchived --print_search_attr ### \--print\_full[​](https://docs.temporal.io/tctl-v1/workflow#--print_full-2 "Direct link to --print_full") Print the full message without table formatting. **Example** tctl workflow listarchived --print_full ### \--print\_json[​](https://docs.temporal.io/tctl-v1/workflow#--print_json-2 "Direct link to --print_json") Print the raw JSON objects. **Example** tctl workflow listarchived --print_json ### \--query[​](https://docs.temporal.io/tctl-v1/workflow#--query-3 "Direct link to --query") Specify an SQL-like query of [Search Attributes](https://docs.temporal.io/search-attribute) . Consult the documentation of the visibility archiver that is used by your [Namespace](https://docs.temporal.io/namespaces) for detailed instructions. Alias: `-q` **Example** tctl workflow listarchived --query ### \--pagesize[​](https://docs.temporal.io/tctl-v1/workflow#--pagesize-1 "Direct link to --pagesize") Specify the maximum number of [Workflow Executions](https://docs.temporal.io/workflow-execution) to list on a page. (By default, the `tctl workflow listarchived` command lists 100 Workflow Executions per page.) **Example** tctl workflow listarchived --pagesize ### \--all[​](https://docs.temporal.io/tctl-v1/workflow#--all "Direct link to --all") List all pages. **Example** tctl workflow listarchived --all observe[​](https://docs.temporal.io/tctl-v1/workflow#observe "Direct link to observe") --------------------------------------------------------------------------------------- The `tctl workflow observe` command shows the progress of the [Event History](https://docs.temporal.io/workflow-execution/event#event-history) of a [Workflow Execution](https://docs.temporal.io/workflow-execution) . See also [`tctl workflow observeid`](https://docs.temporal.io/tctl-v1/workflow#observeid) . `tctl workflow observe ` Alias: `o` The following modifiers control the behavior of the command. ### \--workflow\_id[​](https://docs.temporal.io/tctl-v1/workflow#--workflow_id-4 "Direct link to --workflow_id") Specify a [Workflow Id](https://docs.temporal.io/workflow-execution/workflowid-runid#workflow-id) . Alias: `-w` **Example** tctl workflow observe --workflow_id ### \--run\_id[​](https://docs.temporal.io/tctl-v1/workflow#--run_id-2 "Direct link to --run_id") Specify a [Run Id](https://docs.temporal.io/workflow-execution/workflowid-runid#run-id) . Alias: `-r` **Example** tctl workflow observe --run_id ### \--show\_detail[​](https://docs.temporal.io/tctl-v1/workflow#--show_detail "Direct link to --show_detail") Show event details. **Example** tctl workflow observe --show_detail ### \--max\_field\_length[​](https://docs.temporal.io/tctl-v1/workflow#--max_field_length "Direct link to --max_field_length") Specify the maximum length for each attribute field. The default value is 0. **Example** tctl workflow observe --max_field_length observeid[​](https://docs.temporal.io/tctl-v1/workflow#observeid "Direct link to observeid") --------------------------------------------------------------------------------------------- The `tctl workflow observeid` command shows the progress of the [Event History](https://docs.temporal.io/workflow-execution/event#event-history) of a [Workflow Execution](https://docs.temporal.io/workflow-execution) for the specified [Workflow Id](https://docs.temporal.io/workflow-execution/workflowid-runid#workflow-id) and optional [Run Id](https://docs.temporal.io/workflow-execution/workflowid-runid#run-id) . `tctl workflow observeid [] ` This command is a shortcut for `tctl workflow observe --workflow_id [--run_id ]`. The following modifiers control the behavior of the command. ### \--show\_detail[​](https://docs.temporal.io/tctl-v1/workflow#--show_detail-1 "Direct link to --show_detail") Show event details. **Example** tctl workflow observeid --show_detail ### \--max\_field\_length[​](https://docs.temporal.io/tctl-v1/workflow#--max_field_length-1 "Direct link to --max_field_length") Specify the maximum length for each attribute field. The default value is 0. **Example** tctl workflow observeid --max_field_length query[​](https://docs.temporal.io/tctl-v1/workflow#query "Direct link to query") --------------------------------------------------------------------------------- Alias: `q` The `tctl workflow query` command sends a [Query](https://docs.temporal.io/sending-messages#sending-queries) to a [Workflow Execution](https://docs.temporal.io/workflow-execution) . Queries can be used to retrieve all or part of the Workflow state with given parameters. $ tctl workflow query --workflow_id "HelloQuery" --query_type "getCount"Query result as JSON:3 Queries can also be used on completed Workflows. Let's complete a Workflow by updating its greeting, and then query the now-finished Workflow. $ tctl workflow signal --workflow_id "HelloQuery" --name "updateGreeting" --input \"Bye\"Signal workflow succeeded.$ tctl workflow query --workflow_id "HelloQuery" --query_type "getCount"Query result as JSON:4 Queries are written as follows: `tctl workflow query --workflow_id [modifiers]` The following modifiers control the behavior of the command. Always include required modifiers when executing this command. ### \--workflow\_id[​](https://docs.temporal.io/tctl-v1/workflow#--workflow_id-5 "Direct link to --workflow_id") Specify a [Workflow Id](https://docs.temporal.io/workflow-execution/workflowid-runid#workflow-id) . **This modifier is required.** Alias: `-w` **Example** tctl workflow query --workflow_id ### \--run\_id[​](https://docs.temporal.io/tctl-v1/workflow#--run_id-3 "Direct link to --run_id") Specify a [Run Id](https://docs.temporal.io/workflow-execution/workflowid-runid#run-id) . Alias: `-r` **Example** tctl workflow query --run_id ### \--query\_type[​](https://docs.temporal.io/tctl-v1/workflow#--query_type "Direct link to --query_type") Specify the type of Query to run. **Example** tctl workflow query --query_type ### \--input[​](https://docs.temporal.io/tctl-v1/workflow#--input "Direct link to --input") Pass input for the Query. Input must be in JSON format. For multiple JSON objects, concatenate them and use spaces as separators. Alias: `-i` **Example** tctl workflow query --input ### \--input\_file[​](https://docs.temporal.io/tctl-v1/workflow#--input_file "Direct link to --input_file") Pass input for the Query from a JSON file. For multiple JSON objects, concatenate them and use spaces or newline characters as separators. Input from the command line overwrites input from the file. **Example** tctl workflow query --input_file ### \--query\_reject\_condition[​](https://docs.temporal.io/tctl-v1/workflow#--query_reject_condition "Direct link to --query_reject_condition") Reject Queries based on Workflow state. Valid values are `not_open` and `not_completed_cleanly`. **Example** tctl workflow query --query_reject_condition reset[​](https://docs.temporal.io/tctl-v1/workflow#reset "Direct link to reset") --------------------------------------------------------------------------------- The `tctl workflow reset` command resets a [Workflow Execution](https://docs.temporal.io/workflow-execution) by either [`eventId`](https://docs.temporal.io/tctl-v1/workflow#--event_id) or [`resetType`](https://docs.temporal.io/tctl-v1/workflow#--reset_type) . Resetting a Workflow allows the process to be resumed from a certain point without losing your parameters or Event History. To run multiple Reset operations at once, see [`tctl workflow reset-batch`](https://docs.temporal.io/tctl-v1/workflow#reset-batch) . `tctl workflow reset ` The following modifiers control the behavior of the command. ### \--workflow\_id[​](https://docs.temporal.io/tctl-v1/workflow#--workflow_id-6 "Direct link to --workflow_id") Specify a [Workflow Id](https://docs.temporal.io/workflow-execution/workflowid-runid#workflow-id) . Alias: `-w` **Example** tctl workflow reset --workflow_id ### \--run\_id[​](https://docs.temporal.io/tctl-v1/workflow#--run_id-4 "Direct link to --run_id") Specify a [Run Id](https://docs.temporal.io/workflow-execution/workflowid-runid#run-id) . Alias: `-r` **Example** tctl workflow reset --run_id ### \--event\_id[​](https://docs.temporal.io/tctl-v1/workflow#--event_id "Direct link to --event_id") Specify the `eventId` of any event after `WorkflowTaskStarted` to which you want to reset. Valid values are `WorkflowTaskCompleted`, `WorkflowTaskFailed`, and `WorkflowTaskTimeout`. **Example** tctl workflow reset --event_id ### \--reason[​](https://docs.temporal.io/tctl-v1/workflow#--reason "Direct link to --reason") Specify a reason for resetting the [Workflow Execution](https://docs.temporal.io/workflow-execution) . **Example** tctl workflow reset --reason ### \--reset\_type[​](https://docs.temporal.io/tctl-v1/workflow#--reset_type "Direct link to --reset_type") Specify the event type to which you want to reset. | Value | Description | | --- | --- | | `FirstWorkflowTask` | Reset to the beginning of the Event History. | | `LastWorkflowTask` | Reset to the end of the Event History. | | `LastContinuedAsNew` | Reset to the end of the Event History for the previous Run. | | `BadBinary` | Reset to the point where a bad binary was used. | **Example** tctl workflow reset --reset_type ### \--reset\_reapply\_type[​](https://docs.temporal.io/tctl-v1/workflow#--reset_reapply_type "Direct link to --reset_reapply_type") Specify the types of events to reapply after the reset point. Valid values are `All`, `Signal`, and `None`. The default is `All`. **Example** tctl workflow reset --reset_reapply_type ### \--reset\_bad\_binary\_checksum[​](https://docs.temporal.io/tctl-v1/workflow#--reset_bad_binary_checksum "Direct link to --reset_bad_binary_checksum") Specify the binary checksum when using `--reset_type BadBinary`. **Example** tctl workflow reset --reset_bad_binary_checksum reset-batch[​](https://docs.temporal.io/tctl-v1/workflow#reset-batch "Direct link to reset-batch") --------------------------------------------------------------------------------------------------- The `tctl workflow reset-batch` command resets a batch of [Workflow Executions](https://docs.temporal.io/workflow-execution) by [`resetType`](https://docs.temporal.io/tctl-v1/workflow#--reset_type) . Resetting a Workflow allows the process to be resumed from a certain point without losing your parameters or Event History. To reset individual Workflows, see [`tctl workflow reset`](https://docs.temporal.io/tctl-v1/workflow#reset) . `tctl workflow reset-batch ` The following modifiers control the behavior of the command. ### \--input\_file[​](https://docs.temporal.io/tctl-v1/workflow#--input_file-1 "Direct link to --input_file") Provide an input file that specifies [Workflow Execution](https://docs.temporal.io/workflow-execution) to reset. Each line contains one [Workflow Id](https://docs.temporal.io/workflow-execution/workflowid-runid#workflow-id) as the base Run and, optionally, a [Run Id](https://docs.temporal.io/workflow-execution/workflowid-runid#run-id) . If a Run Id is not specified, the current Run Id is used. **Example** tctl workflow reset-batch --input_file ### \--query[​](https://docs.temporal.io/tctl-v1/workflow#--query-4 "Direct link to --query") Specify an SQL-like query of [Search Attributes](https://docs.temporal.io/search-attribute) describing the [Workflow Executions](https://docs.temporal.io/workflow-execution) to reset. Alias: `-q` **Example** tctl workflow reset-batch --query ### \--exclude\_file[​](https://docs.temporal.io/tctl-v1/workflow#--exclude_file "Direct link to --exclude_file") Provide an input file that specifies [Workflow Executions](https://docs.temporal.io/workflow-execution) to exclude from resetting. Each line contains one [Workflow Id](https://docs.temporal.io/workflow-execution/workflowid-runid#workflow-id) . **Example** tctl workflow reset-batch --exclude_file ### \--input\_separator[​](https://docs.temporal.io/tctl-v1/workflow#--input_separator "Direct link to --input_separator") Specify the separator for the input file. The default is a tab (`\t`). **Example** tctl workflow reset-batch --input_separator ### \--reason[​](https://docs.temporal.io/tctl-v1/workflow#--reason-1 "Direct link to --reason") Specify a reason for resetting the [Workflow Executions](https://docs.temporal.io/workflow-execution) . **Example** tctl workflow reset-batch --reason ### \--input\_parallism[​](https://docs.temporal.io/tctl-v1/workflow#--input_parallism "Direct link to --input_parallism") Specify the number of goroutines to run in parallel. Each goroutine processes one line for every second. The default is 1. **Example** tctl workflow reset-batch --input_parallism ### \--skip\_current\_open[​](https://docs.temporal.io/tctl-v1/workflow#--skip_current_open "Direct link to --skip_current_open") Indicate that a [Workflow Execution](https://docs.temporal.io/workflow-execution) should be skipped if the current Run is open for the same [Workflow Id](https://docs.temporal.io/workflow-execution/workflowid-runid#workflow-id) as the base Run. **Example** tctl workflow reset-batch --skip_current_open ### \--skip\_base\_is\_not\_current[​](https://docs.temporal.io/tctl-v1/workflow#--skip_base_is_not_current "Direct link to --skip_base_is_not_current") Indicate that a [Workflow Execution](https://docs.temporal.io/workflow-execution) should be skipped if the base Run is not the current Run. **Example** tctl workflow reset-batch --skip_base_is_not_current ### \--only\_non\_deterministic[​](https://docs.temporal.io/tctl-v1/workflow#--only_non_deterministic "Direct link to --only_non_deterministic") Indicate that a [Workflow Execution](https://docs.temporal.io/workflow-execution) should be reset only if its last event is `WorkflowTaskFailed` with a nondeterminism error. **Example** tctl workflow reset-batch --only_non_deterministic ### \--dry\_run[​](https://docs.temporal.io/tctl-v1/workflow#--dry_run "Direct link to --dry_run") Simulate use of the `tctl workflow reset-batch` command without resetting any [Workflow Executions](https://docs.temporal.io/workflow-execution) . Output is logged to `stdout`. **Example** tctl workflow reset-batch --dry_run ### \--reset\_type[​](https://docs.temporal.io/tctl-v1/workflow#--reset_type-1 "Direct link to --reset_type") Specify the event type to which you want to reset. | Value | Description | | --- | --- | | `FirstWorkflowTask` | Reset to the beginning of the Event History. | | `LastWorkflowTask` | Reset to the end of the Event History. | | `LastContinuedAsNew` | Reset to the end of the Event History for the previous Run. | | `BadBinary` | Reset to the point where a bad binary was used. | **Example** tctl workflow reset-batch --reset_type ### \--reset\_bad\_binary\_checksum[​](https://docs.temporal.io/tctl-v1/workflow#--reset_bad_binary_checksum-1 "Direct link to --reset_bad_binary_checksum") Specify the binary checksum when using `--reset_type BadBinary`. **Example** tctl workflow reset-batch --reset_bad_binary_checksum run[​](https://docs.temporal.io/tctl-v1/workflow#run "Direct link to run") --------------------------------------------------------------------------- The `tctl workflow run` command starts a new [Workflow Execution](https://docs.temporal.io/workflow-execution) and can show the progress of a Workflow Execution. The command is entered in the following format: `tctl workflow run ` To run a Workflow, the user must specify the following: * Task queue name (`--taskqueue`) * Workflow Type (`--workflow_type`) tctl workflow run --taskqueue your-task-queue-name --workflow_type YourWorkflowDefinitionName Single quotes (`''`) are used to wrap input as JSON. This command doesn't finish until the Workflow completes. The following modifiers control the behavior of the command. ### \--taskqueue[​](https://docs.temporal.io/tctl-v1/workflow#--taskqueue "Direct link to --taskqueue") Specify a [Task Queue](https://docs.temporal.io/task-queue) . Alias: `--t` **Example** tctl workflow run --taskqueue ### \--workflow\_id[​](https://docs.temporal.io/tctl-v1/workflow#--workflow_id-7 "Direct link to --workflow_id") Specify a [Workflow Id](https://docs.temporal.io/workflow-execution/workflowid-runid#workflow-id) . Alias: `-w` **Example** tctl workflow run --workflow_id ### \--workflow\_type[​](https://docs.temporal.io/tctl-v1/workflow#--workflow_type-2 "Direct link to --workflow_type") Specify the name of a [Workflow Type](https://docs.temporal.io/workflow-definition#workflow-type) . **Example** tctl workflow run --workflow_type ### \--execution\_timeout[​](https://docs.temporal.io/tctl-v1/workflow#--execution_timeout "Direct link to --execution_timeout") Specify the [Start-To-Close Timeout](https://docs.temporal.io/encyclopedia/detecting-activity-failures#start-to-close-timeout) of the [Workflow Execution](https://docs.temporal.io/workflow-execution) in seconds. The default value is 0. **Example** tctl workflow run --execution_timeout ### \--workflow\_task\_timeout[​](https://docs.temporal.io/tctl-v1/workflow#--workflow_task_timeout "Direct link to --workflow_task_timeout") Specify the [Start-To-Close Timeout](https://docs.temporal.io/encyclopedia/detecting-activity-failures#start-to-close-timeout) of the [Workflow Task](https://docs.temporal.io/tasks#workflow-task) in seconds. The default value is 10. **Example** tctl workflow run --workflow_task_timeout ### \--cron[​](https://docs.temporal.io/tctl-v1/workflow#--cron "Direct link to --cron") Specify a [Cron Schedule](https://docs.temporal.io/cron-job#cron-schedules) . **Example** tctl workflow run --cron ### \--workflowidreusepolicy[​](https://docs.temporal.io/tctl-v1/workflow#--workflowidreusepolicy "Direct link to --workflowidreusepolicy") Specify a [Workflow Id Reuse Policy](https://docs.temporal.io/workflow-execution/workflowid-runid#workflow-id-reuse-policy) . Configure if the same [Workflow Id](https://docs.temporal.io/workflow-execution/workflowid-runid#workflow-id) is allowed for use in new [Workflow Execution](https://docs.temporal.io/workflow-execution) . There are three allowed values: * [AllowDuplicateFailedOnly](https://docs.temporal.io/workflow-execution/workflowid-runid#workflow-id-reuse-policy) * [AllowDuplicate](https://docs.temporal.io/workflow-execution/workflowid-runid#workflow-id-reuse-policy) * [RejectDuplicate](https://docs.temporal.io/workflow-execution/workflowid-runid#workflow-id-reuse-policy) **Examples** tctl workflow run --workflowidreusepolicy AllowDuplicatetctl workflow run --workflowidreusepolicy AllowDuplicateFailedOnlytctl workflow run --workflowidreusepolicy RejectDuplicate ### \--input[​](https://docs.temporal.io/tctl-v1/workflow#--input-1 "Direct link to --input") Pass input for the Workflow. Input must be in JSON format. For multiple JSON objects, pass each in a separate `--input` option. Use `null` for null values. Alias: `-i` **Example** tctl workflow run --input ### \--input\_file[​](https://docs.temporal.io/tctl-v1/workflow#--input_file-2 "Direct link to --input_file") Pass input for the Workflow from a JSON file. For multiple JSON objects, concatenate them and use spaces or newline characters as separators. Input from the command line overwrites input from the file. **Example** tctl workflow run --input_file ### \--memo\_key[​](https://docs.temporal.io/tctl-v1/workflow#--memo_key "Direct link to --memo_key") Pass a key for a memo. For multiple keys, concatenate them and use spaces as separators. **Example** tctl workflow run --memo_key ### \--memo[​](https://docs.temporal.io/tctl-v1/workflow#--memo "Direct link to --memo") Pass a memo. A memo is information in JSON format that can be shown when the Workflow is listed. For multiple memos, concatenate them and use spaces as separators. The order must match the order of keys in `--memo_key`. **Example** tctl workflow run --memo ### \--memo\_file[​](https://docs.temporal.io/tctl-v1/workflow#--memo_file "Direct link to --memo_file") Pass information for a memo from a JSON file. For multiple JSON objects, concatenate them and use spaces or newline characters as separators. The order must match the order of keys in `--memo_key`. **Example** tctl workflow run --memo_file ### \--search\_attr\_key[​](https://docs.temporal.io/tctl-v1/workflow#--search_attr_key "Direct link to --search_attr_key") Specify a [Search Attribute](https://docs.temporal.io/search-attribute) key. For multiple keys, concatenate them and use pipes (`|`) as separators. To list valid keys, use the `tctl cluster get-search-attributes` command. **Example** tctl workflow run --search_attr_key ### \--search\_attr\_value[​](https://docs.temporal.io/tctl-v1/workflow#--search_attr_value "Direct link to --search_attr_value") Specify a [Search Attribute](https://docs.temporal.io/search-attribute) value. For multiple values, concatenate them and use pipes (`|`) as separators. If a value is an array, use JSON format, such as `["a","b"]`, `[1,2]`, `["true","false"]`, or `["2022-06-07T17:16:34-08:00","2022-06-07T18:16:34-08:00"]`. To list valid keys and value types, use the `tctl cluster get-search-attributes` command. **Example** tctl workflow run --search_attr_value ### \--show\_detail[​](https://docs.temporal.io/tctl-v1/workflow#--show_detail-2 "Direct link to --show_detail") Get event details. **Example** tctl workflow run --show_detail ### \--max\_field\_length[​](https://docs.temporal.io/tctl-v1/workflow#--max_field_length-2 "Direct link to --max_field_length") Specify the maximum length for each attribute field. The default value is 0. **Example** tctl workflow run --max_field_length scan[​](https://docs.temporal.io/tctl-v1/workflow#scan "Direct link to scan") ------------------------------------------------------------------------------ The `tctl workflow scan` command lists [Workflow Executions](https://docs.temporal.io/workflow-execution) . It is faster than the `tctl workflow listall` command, but the results are not sorted. By default, this command lists a maximum of 2000 Workflow Executions. To set the size of a page, use the `--pagesize` option. See also [`tctl workflow list`](https://docs.temporal.io/tctl-v1/workflow#list) , [`tctl workflow listall`](https://docs.temporal.io/tctl-v1/workflow#listall) , and [`tctl workflow listarchived`](https://docs.temporal.io/tctl-v1/workflow#listarchived) . `tctl workflow scan ` The following modifiers control the behavior of the command. ### \--print\_raw\_time[​](https://docs.temporal.io/tctl-v1/workflow#--print_raw_time-2 "Direct link to --print_raw_time") Print the raw timestamp. **Example** tctl workflow scan --print_raw_time ### \--print\_datetime[​](https://docs.temporal.io/tctl-v1/workflow#--print_datetime-3 "Direct link to --print_datetime") Print the timestamp. **Example** tctl workflow scan --print_datetime ### \--print\_memo[​](https://docs.temporal.io/tctl-v1/workflow#--print_memo-3 "Direct link to --print_memo") Print a memo. **Example** tctl workflow scan --print_memo ### \--print\_search\_attr[​](https://docs.temporal.io/tctl-v1/workflow#--print_search_attr-3 "Direct link to --print_search_attr") Print the [Search Attributes](https://docs.temporal.io/search-attribute) . **Example** tctl workflow scan --print_search_attr ### \--print\_full[​](https://docs.temporal.io/tctl-v1/workflow#--print_full-3 "Direct link to --print_full") Print the full message without table formatting. **Example** tctl workflow scan --print_full ### \--print\_json[​](https://docs.temporal.io/tctl-v1/workflow#--print_json-3 "Direct link to --print_json") Print the raw JSON objects. **Example** tctl workflow scan --print_json ### \--pagesize[​](https://docs.temporal.io/tctl-v1/workflow#--pagesize-2 "Direct link to --pagesize") Specify the maximum number of [Workflow Execution](https://docs.temporal.io/workflow-execution) to list on a page. (By default, the `tctl workflow scan` command lists 2000 Workflow Executions per page.) **Example** tctl workflow scan --pagesize ### \--query[​](https://docs.temporal.io/tctl-v1/workflow#--query-5 "Direct link to --query") Specify an SQL-like query of [Search Attributes](https://docs.temporal.io/search-attribute) . Alias: `-q` **Example** tctl workflow scan --query show[​](https://docs.temporal.io/tctl-v1/workflow#show "Direct link to show") ------------------------------------------------------------------------------ The `tctl workflow show` command shows the [Event History](https://docs.temporal.io/workflow-execution/event#event-history) for the specified [Workflow Execution](https://docs.temporal.io/workflow-execution) . `tctl workflow show ` See also [`tctl workflow showid`](https://docs.temporal.io/tctl-v1/workflow#showid) . The following modifiers control the behavior of the command. ### \--workflow\_id[​](https://docs.temporal.io/tctl-v1/workflow#--workflow_id-8 "Direct link to --workflow_id") Show the History of a [Workflow Execution](https://docs.temporal.io/workflow-execution) by specifying a [Workflow Id](https://docs.temporal.io/workflow-execution/workflowid-runid#workflow-id) . Alias: `-w` **Example** tctl workflow show --workflow_id ### \--run\_id[​](https://docs.temporal.io/tctl-v1/workflow#--run_id-5 "Direct link to --run_id") Show the History of a [Workflow Execution](https://docs.temporal.io/workflow-execution) by specifying a [Run Id](https://docs.temporal.io/workflow-execution/workflowid-runid#run-id) . Alias: `-r` **Example** tctl workflow show --run_id ### \--print\_datetime[​](https://docs.temporal.io/tctl-v1/workflow#--print_datetime-4 "Direct link to --print_datetime") Print the timestamp. **Example** tctl workflow show --print_datetime ### \--print\_raw\_time[​](https://docs.temporal.io/tctl-v1/workflow#--print_raw_time-3 "Direct link to --print_raw_time") Print the raw timestamp. **Example** tctl workflow show --print_raw_time ### \--output\_filename[​](https://docs.temporal.io/tctl-v1/workflow#--output_filename "Direct link to --output_filename") Serialize an event to a file. **Example** tctl workflow show --output_filename ### \--print\_full[​](https://docs.temporal.io/tctl-v1/workflow#--print_full-4 "Direct link to --print_full") Print full event details. **Example** tctl workflow show --print_full ### \--print\_event\_version[​](https://docs.temporal.io/tctl-v1/workflow#--print_event_version "Direct link to --print_event_version") Print the event version. **Example** tctl workflow show --print_event_version ### \--event\_id[​](https://docs.temporal.io/tctl-v1/workflow#--event_id-1 "Direct link to --event_id") Print the details of a specified event. The default value is 0. **Example** tctl workflow show --event_id ### \--max\_field\_length[​](https://docs.temporal.io/tctl-v1/workflow#--max_field_length-3 "Direct link to --max_field_length") Specify the maximum length for each attribute field. The default value is 500. **Example** tctl workflow show --max_field_length ### \--reset\_points\_only[​](https://docs.temporal.io/tctl-v1/workflow#--reset_points_only-2 "Direct link to --reset_points_only") Show only events that are eligible for reset. **Example** tctl workflow show --reset_points_only showid[​](https://docs.temporal.io/tctl-v1/workflow#showid "Direct link to showid") ------------------------------------------------------------------------------------ The `tctl workflow showid` command shows the Workflow Execution Event History for the specified [Workflow Id](https://docs.temporal.io/workflow-execution/workflowid-runid#workflow-id) and optional [Run Id](https://docs.temporal.io/workflow-execution/workflowid-runid#run-id) . `tctl workflow showid [] ` This command is a shortcut for `tctl workflow show --workflow_id [--run_id ]`. Example: tctl workflow showid Example output: 1 WorkflowExecutionStarted {WorkflowType:{Name:HelloWorld}, ParentInitiatedEventId:0, TaskQueue:{Name:HelloWorldTaskQueue, Kind:Normal}, Input:[Temporal], WorkflowExecutionTimeout:1h0m0s, WorkflowRunTimeout:1h0m0s, WorkflowTaskTimeout:10s, Initiator:Unspecified, LastCompletionResult:[], OriginalExecutionRunId:f0c04163-833f-490b-99a9-ee48b6199213, Identity:tctl@z0mb1e, FirstExecutionRunId:f0c04163-833f-490b-99a9-ee48b6199213, Attempt:1, WorkflowExecutionExpirationTime:2020-10-13 21:41:06.349 +0000 UTC, FirstWorkflowTaskBackoff:0s}2 WorkflowTaskScheduled {TaskQueue:{Name:HelloWorldTaskQueue, Kind:Normal}, StartToCloseTimeout:10s, Attempt:1}3 WorkflowTaskStarted {ScheduledEventId:2, Identity:15079@z0mb1e, RequestId:731f7b41-5ae4-42e4-9695-ecd857d571f1}4 WorkflowTaskCompleted {ScheduledEventId:2, StartedEventId:3, Identity:15079@z0mb1e}5 WorkflowExecutionCompleted {Result:[], WorkflowTaskCompletedEventId:4} The following modifiers control the behavior of the command. ### \--print\_datetime[​](https://docs.temporal.io/tctl-v1/workflow#--print_datetime-5 "Direct link to --print_datetime") Print the timestamp. **Example** tctl workflow showid --print_datetime ### \--print\_raw\_time[​](https://docs.temporal.io/tctl-v1/workflow#--print_raw_time-4 "Direct link to --print_raw_time") Print the raw timestamp. **Example** tctl workflow showid --print_raw_time ### \--output\_filename[​](https://docs.temporal.io/tctl-v1/workflow#--output_filename-1 "Direct link to --output_filename") Serialize an event to a file. **Example** tctl workflow showid --output_filename ### \--print\_full[​](https://docs.temporal.io/tctl-v1/workflow#--print_full-5 "Direct link to --print_full") Print full event details. **Example** tctl workflow showid --print_full ### \--print\_event\_version[​](https://docs.temporal.io/tctl-v1/workflow#--print_event_version-1 "Direct link to --print_event_version") Print the event version. **Example** tctl workflow showid --print_event_version ### \--event\_id[​](https://docs.temporal.io/tctl-v1/workflow#--event_id-2 "Direct link to --event_id") Print the details of a specified event. The default value is 0. **Example** tctl workflow showid --event_id ### \--max\_field\_length[​](https://docs.temporal.io/tctl-v1/workflow#--max_field_length-4 "Direct link to --max_field_length") Specify the maximum length for each attribute field. The default value is 500. **Example** tctl workflow showid --max_field_length ### \--reset\_points\_only[​](https://docs.temporal.io/tctl-v1/workflow#--reset_points_only-3 "Direct link to --reset_points_only") Show only events that are eligible for reset. **Example** tctl workflow showid --reset_points_only signal[​](https://docs.temporal.io/tctl-v1/workflow#signal "Direct link to signal") ------------------------------------------------------------------------------------ The `tctl workflow signal` command [Signals](https://docs.temporal.io/sending-messages#sending-signals) a [Workflow Execution](https://docs.temporal.io/workflow-execution) . Workflows listen for Signals by their Signal name, and can be made to listen to one or more Signal names. Workflows can also listen for SQL queries. The Workflow below listens for instances of "HelloSignal": tctl workflow start --workflow_id "HelloSignal" --taskqueue HelloWorldTaskQueue --workflow_type HelloWorld --execution_timeout 3600 --input \"World\" The Worker would return this output upon receiving the Signal: 13:57:44.258 [workflow-method] INFO c.t.s.javaquickstart.GettingStarted - 1: Hello World! Signals can also be used to change variable values. tctl workflow signal --workflow_id "HelloSignal" --name "updateGreeting" --input \"Hi\" The output would change from the first Signal received. 13:57:44.258 [workflow-method] INFO c.t.s.javaquickstart.GettingStarted - 1: Hello World!13:58:22.352 [workflow-method] INFO c.t.s.javaquickstart.GettingStarted - 2: Hi World! When a Signal is sent, an await condition is made to block any Signals that contain the same input value. However, changing the greeting in our example unblocks it: tctl workflow signal --workflow_id "HelloSignal" --name "updateGreeting" --input \"Welcome\" Worker output: 13:57:44.258 [workflow-method] INFO c.t.s.javaquickstart.GettingStarted - 1: Hello World!13:58:22.352 [workflow-method] INFO c.t.s.javaquickstart.GettingStarted - 2: Hi World!13:59:29.097 [workflow-method] INFO c.t.s.javaquickstart.GettingStarted - 3: Welcome World! Sending Signals does not require a running Worker. tctl workflow signal --workflow_id "HelloSignal" --name "updateGreeting" --input \"Welcome\" CLI output: Signal workflow succeeded. The Signal request is queued inside the Temporal Server until the Worker is restarted. If the given Signal contains the same input as before, the queued Signal will be ignored. Complete the Workflow by sending a Signal with a "Bye" greeting: tctl workflow signal --workflow_id "HelloSignal" --name "updateGreeting" --input \"Bye\" Check that the Workflow Execution has been completed. tctl workflow showid HelloSignal Signals are written as follows: tctl workflow signal --workflow_id or tctl workflow signal --query The following modifiers control the behavior of the command. Make sure to include required modifiers in all command executions. ### \--workflow\_id[​](https://docs.temporal.io/tctl-v1/workflow#--workflow_id-9 "Direct link to --workflow_id") Specify a [Workflow Id](https://docs.temporal.io/workflow-execution/workflowid-runid#workflow-id) . **This modifier is required.** Alias: `-w` **Example** tctl workflow signal --workflow_id ### \--run\_id[​](https://docs.temporal.io/tctl-v1/workflow#--run_id-6 "Direct link to --run_id") Specify a [Run Id](https://docs.temporal.io/workflow-execution/workflowid-runid#run-id) . Alias: `-r` **Example** tctl workflow signal --run_id ### \--name[​](https://docs.temporal.io/tctl-v1/workflow#--name "Direct link to --name") Specify the name of a [Signal](https://docs.temporal.io/sending-messages#sending-signals) . **Example** tctl workflow signal --query --name ### \--input[​](https://docs.temporal.io/tctl-v1/workflow#--input-2 "Direct link to --input") Pass input for the [Signal](https://docs.temporal.io/sending-messages#sending-signals) . Input must be in JSON format. Alias: `-i` **Example** tctl workflow signal --query --input ### \--input\_file[​](https://docs.temporal.io/tctl-v1/workflow#--input_file-3 "Direct link to --input_file") Pass input for the [Signal](https://docs.temporal.io/sending-messages#sending-signals) from a JSON file. **Example** tctl workflow signal --query --input_file stack[​](https://docs.temporal.io/tctl-v1/workflow#stack "Direct link to stack") --------------------------------------------------------------------------------- The `tctl workflow stack` command queries [Workflow Execution](https://docs.temporal.io/workflow-execution) with `__stack_trace` as the query type. This command can be used to locate errors and blocks in a [Workflow Definition](https://docs.temporal.io/workflow-definition) . `tctl workflow stack ` The following modifiers control the behavior of the command. ### \--workflow\_id[​](https://docs.temporal.io/tctl-v1/workflow#--workflow_id-10 "Direct link to --workflow_id") **This is a required modifier.** Specify a [Workflow Id](https://docs.temporal.io/workflow-execution/workflowid-runid#workflow-id) . Alias: `-w` **Example** tctl workflow stack --workflow_id ### \--run\_id[​](https://docs.temporal.io/tctl-v1/workflow#--run_id-7 "Direct link to --run_id") Specify a [Run Id](https://docs.temporal.io/workflow-execution/workflowid-runid#run-id) . Alias: `-r` **Example** tctl workflow stack --run_id ### \--input[​](https://docs.temporal.io/tctl-v1/workflow#--input-3 "Direct link to --input") Pass input for the query. Input must be in JSON format. For multiple JSON objects, concatenate them and use spaces as separators. Alias: `-i` **Example** tctl workflow stack --input ### \--input\_file[​](https://docs.temporal.io/tctl-v1/workflow#--input_file-4 "Direct link to --input_file") Pass input for the query from a JSON file. For multiple JSON objects, concatenate them and use spaces or newline characters as separators. Input from the command line overwrites input from the file. **Example** tctl workflow stack --input_file ### \--query\_reject\_condition[​](https://docs.temporal.io/tctl-v1/workflow#--query_reject_condition-1 "Direct link to --query_reject_condition") Reject queries based on Workflow state. Valid values are `not_open` and `not_completed_cleanly`. **Example** tctl workflow stack --query_reject_condition start[​](https://docs.temporal.io/tctl-v1/workflow#start "Direct link to start") --------------------------------------------------------------------------------- The `tctl workflow start` command starts a new [Workflow Execution](https://docs.temporal.io/workflow-execution) . Unlike `run`, this command returns the Workflow Id and Run Id immediately after starting the Workflow. `tctl workflow start ` The following modifiers control the behavior of the command. Always include required modifiers when executing this command. ### \--taskqueue[​](https://docs.temporal.io/tctl-v1/workflow#--taskqueue-1 "Direct link to --taskqueue") Specify a [Task Queue](https://docs.temporal.io/task-queue) . Alias: `--t` **Example** tctl workflow start --taskqueue ### \--workflow\_id[​](https://docs.temporal.io/tctl-v1/workflow#--workflow_id-11 "Direct link to --workflow_id") **This is a required modifier.** Specify a [Workflow Id](https://docs.temporal.io/workflow-execution/workflowid-runid#workflow-id) . Alias: `-w` **Example** tctl workflow start --workflow_id If a Workflow is started without providing an Id, the Client generates one in the form of a UUID. Temporal recommends using a business id rather than the client-generated UUID. **Example** tctl workflow start --workflow_id "HelloTemporal1" --taskqueue HelloWorldTaskQueue --workflow_type HelloWorld --execution_timeout 3600 --input \"Temporal\" ### \--workflow\_type[​](https://docs.temporal.io/tctl-v1/workflow#--workflow_type-3 "Direct link to --workflow_type") Specify the name of a [Workflow Type](https://docs.temporal.io/workflow-definition#workflow-type) . **Example** tctl workflow start --workflow_type ### \--execution\_timeout[​](https://docs.temporal.io/tctl-v1/workflow#--execution_timeout-1 "Direct link to --execution_timeout") Specify the [Start-To-Close Timeout](https://docs.temporal.io/encyclopedia/detecting-activity-failures#start-to-close-timeout) of the [Workflow Execution](https://docs.temporal.io/workflow-execution) in seconds. The default value is 0. **Example** tctl workflow start --execution_timeout ### \--workflow\_task\_timeout[​](https://docs.temporal.io/tctl-v1/workflow#--workflow_task_timeout-1 "Direct link to --workflow_task_timeout") Specify the [Start-To-Close Timeout](https://docs.temporal.io/encyclopedia/detecting-activity-failures#start-to-close-timeout) of the [Workflow Task](https://docs.temporal.io/tasks#workflow-task) in seconds. The default value is 10. **Example** tctl workflow start --workflow_task_timeout ### \--cron[​](https://docs.temporal.io/tctl-v1/workflow#--cron-1 "Direct link to --cron") Specify a [Cron Schedule](https://docs.temporal.io/cron-job#cron-schedules) . **Example** tctl workflow start --cron ### \--workflowidreusepolicy[​](https://docs.temporal.io/tctl-v1/workflow#--workflowidreusepolicy-1 "Direct link to --workflowidreusepolicy") Specify a [Workflow Id Reuse Policy](https://docs.temporal.io/workflow-execution/workflowid-runid#workflow-id-reuse-policy) . Configure if the same [Workflow Id](https://docs.temporal.io/workflow-execution/workflowid-runid#workflow-id) is allowed for use in new [Workflow Execution](https://docs.temporal.io/workflow-execution) . There are three allowed values: * [AllowDuplicateFailedOnly](https://docs.temporal.io/workflow-execution/workflowid-runid#workflow-id-reuse-policy) * [AllowDuplicate](https://docs.temporal.io/workflow-execution/workflowid-runid#workflow-id-reuse-policy) * [RejectDuplicate](https://docs.temporal.io/workflow-execution/workflowid-runid#workflow-id-reuse-policy) **Examples** tctl workflow start --workflowidreusepolicy AllowDuplicatetctl workflow start --workflowidreusepolicy AllowDuplicateFailedOnlytctl workflow start --workflowidreusepolicy RejectDuplicate note Multiple Workflows with the same Id cannot be run at the same time ### \--input[​](https://docs.temporal.io/tctl-v1/workflow#--input-4 "Direct link to --input") Pass input for the Workflow. Input must be in JSON format. For multiple JSON objects, pass each in a separate `--input` option. Use `null` for null values. Alias: `-i` **Example** tctl workflow start --input ### \--input\_file[​](https://docs.temporal.io/tctl-v1/workflow#--input_file-5 "Direct link to --input_file") Pass input for the Workflow from a JSON file. For multiple JSON objects, concatenate them and use spaces or newline characters as separators. Input from the command line overwrites input from the file. **Example** tctl workflow start --input_file ### \--memo\_key[​](https://docs.temporal.io/tctl-v1/workflow#--memo_key-1 "Direct link to --memo_key") Pass a key for a memo. For multiple keys, concatenate them and use spaces as separators. **Example** tctl workflow start --memo_key ### \--memo[​](https://docs.temporal.io/tctl-v1/workflow#--memo-1 "Direct link to --memo") Pass information for a [memo](https://docs.temporal.io/workflow-execution#memo) from a JSON file. Memos are immutable key/value pairs that can be attached to a workflow run when starting the workflow. Memos are visible when listing workflows. For multiple memos, concatenate them and use spaces as separators. The order must match the order of keys in `--memo_key`. **Example** tctl workflow start \ -tq your-task-queue \ -wt your-workflow \ -et 60 \ -i '"temporal"' \ -memo_key '' \ -memo '' ### \--memo\_file[​](https://docs.temporal.io/tctl-v1/workflow#--memo_file-1 "Direct link to --memo_file") Pass information for a memo from a JSON file. For multiple JSON objects, concatenate them and use spaces or newline characters as separators. The order must match the order of keys in `--memo_key`. **Example** tctl workflow start --memo_file ### \--search\_attr\_key[​](https://docs.temporal.io/tctl-v1/workflow#--search_attr_key-1 "Direct link to --search_attr_key") Specify a [Search Attribute](https://docs.temporal.io/search-attribute) name. For multiple names, concatenate them and use pipes (`|`) as separators. To list valid Search Attributes, use the `tctl cluster get-search-attributes` command. **Example** tctl workflow start --search_attr_key ### \--search\_attr\_value[​](https://docs.temporal.io/tctl-v1/workflow#--search_attr_value-1 "Direct link to --search_attr_value") Specify a [Search Attribute](https://docs.temporal.io/search-attribute) value. For multiple values, concatenate them and use pipes (`|`) as separators. If a value is an array, use JSON format, such as `["a","b"]`, `[1,2]`, `["true","false"]`, or `["2022-06-07T17:16:34-08:00","2022-06-07T18:16:34-08:00"]`. To list valid Search Attributes and value types, use the `tctl cluster get-search-attributes` command. **Example** tctl workflow start --search_attr_value terminate[​](https://docs.temporal.io/tctl-v1/workflow#terminate "Direct link to terminate") --------------------------------------------------------------------------------------------- The `tctl workflow terminate` command terminates a [Workflow Execution](https://docs.temporal.io/workflow-execution) . Terminating a running Workflow Execution records a `WorkflowExecutionTerminated` event as the closing event in the History. No more [Workflow Task](https://docs.temporal.io/tasks#workflow-task) will be scheduled. See also [`tctl workflow cancel`](https://docs.temporal.io/tctl-v1/workflow#cancel) . `tctl workflow terminate --query ` The following modifiers control the behavior of the command. ### \--workflow\_id[​](https://docs.temporal.io/tctl-v1/workflow#--workflow_id-12 "Direct link to --workflow_id") _Required modifier_ Specify a [Workflow Id](https://docs.temporal.io/workflow-execution/workflowid-runid#workflow-id) . Alias: `-w` **Example** tctl workflow terminate --workflow_id ### \--run\_id[​](https://docs.temporal.io/tctl-v1/workflow#--run_id-8 "Direct link to --run_id") Specify a [Run Id](https://docs.temporal.io/workflow-execution/workflowid-runid#run-id) . If `run_id` is not specified, `tctl` terminates the last Workflow Execution for the specified `workflow_id`. Alias: `-r` **Example** tctl workflow terminate --run_id ### \--reason[​](https://docs.temporal.io/tctl-v1/workflow#--reason-2 "Direct link to --reason") Specify a reason for terminating the [Workflow Execution](https://docs.temporal.io/workflow-execution) . **Example** tctl workflow terminate --workflow_id --reason * [cancel](https://docs.temporal.io/tctl-v1/workflow#cancel) * [\--workflow\_id](https://docs.temporal.io/tctl-v1/workflow#--workflow_id) * [\--run\_id](https://docs.temporal.io/tctl-v1/workflow#--run_id) * [count](https://docs.temporal.io/tctl-v1/workflow#count) * [\--query](https://docs.temporal.io/tctl-v1/workflow#--query) * [describe](https://docs.temporal.io/tctl-v1/workflow#describe) * [\--workflow\_id](https://docs.temporal.io/tctl-v1/workflow#--workflow_id-1) * [\--run\_id](https://docs.temporal.io/tctl-v1/workflow#--run_id-1) * [\--print\_raw](https://docs.temporal.io/tctl-v1/workflow#--print_raw) * [\--reset\_points\_only](https://docs.temporal.io/tctl-v1/workflow#--reset_points_only) * [describeid](https://docs.temporal.io/tctl-v1/workflow#describeid) * [\--print\_raw](https://docs.temporal.io/tctl-v1/workflow#--print_raw-1) * [\--reset\_points\_only](https://docs.temporal.io/tctl-v1/workflow#--reset_points_only-1) * [list](https://docs.temporal.io/tctl-v1/workflow#list) * [\--print\_raw\_time](https://docs.temporal.io/tctl-v1/workflow#--print_raw_time) * [\--print\_datetime](https://docs.temporal.io/tctl-v1/workflow#--print_datetime) * [\--print\_memo](https://docs.temporal.io/tctl-v1/workflow#--print_memo) * [\--print\_search\_attr](https://docs.temporal.io/tctl-v1/workflow#--print_search_attr) * [\--print\_full](https://docs.temporal.io/tctl-v1/workflow#--print_full) * [\--print\_json](https://docs.temporal.io/tctl-v1/workflow#--print_json) * [\--open](https://docs.temporal.io/tctl-v1/workflow#--open) * [\--earliest\_time](https://docs.temporal.io/tctl-v1/workflow#--earliest_time) * [\--latest\_time](https://docs.temporal.io/tctl-v1/workflow#--latest_time) * [\--workflow\_id](https://docs.temporal.io/tctl-v1/workflow#--workflow_id-2) * [\--workflow\_type](https://docs.temporal.io/tctl-v1/workflow#--workflow_type) * [\--status](https://docs.temporal.io/tctl-v1/workflow#--status) * [\--query](https://docs.temporal.io/tctl-v1/workflow#--query-1) * [\--more](https://docs.temporal.io/tctl-v1/workflow#--more) * [\--pagesize](https://docs.temporal.io/tctl-v1/workflow#--pagesize) * [listall](https://docs.temporal.io/tctl-v1/workflow#listall) * [\--print\_datetime](https://docs.temporal.io/tctl-v1/workflow#--print_datetime-1) * [\--print\_memo](https://docs.temporal.io/tctl-v1/workflow#--print_memo-1) * [\--print\_search\_attr](https://docs.temporal.io/tctl-v1/workflow#--print_search_attr-1) * [`--print_full`](https://docs.temporal.io/tctl-v1/workflow#--print_full-1) * [\--print\_json](https://docs.temporal.io/tctl-v1/workflow#--print_json-1) * [\--open](https://docs.temporal.io/tctl-v1/workflow#--open-1) * [\--earliest\_time](https://docs.temporal.io/tctl-v1/workflow#--earliest_time-1) * [\--latest\_time](https://docs.temporal.io/tctl-v1/workflow#--latest_time-1) * [\--workflow\_id](https://docs.temporal.io/tctl-v1/workflow#--workflow_id-3) * [\--workflow\_type](https://docs.temporal.io/tctl-v1/workflow#--workflow_type-1) * [\--status](https://docs.temporal.io/tctl-v1/workflow#--status-1) * [\--query](https://docs.temporal.io/tctl-v1/workflow#--query-2) * [listarchived](https://docs.temporal.io/tctl-v1/workflow#listarchived) * [\--print\_raw\_time](https://docs.temporal.io/tctl-v1/workflow#--print_raw_time-1) * [\--print\_datetime](https://docs.temporal.io/tctl-v1/workflow#--print_datetime-2) * [\--print\_memo](https://docs.temporal.io/tctl-v1/workflow#--print_memo-2) * [\--print\_search\_attr](https://docs.temporal.io/tctl-v1/workflow#--print_search_attr-2) * [\--print\_full](https://docs.temporal.io/tctl-v1/workflow#--print_full-2) * [\--print\_json](https://docs.temporal.io/tctl-v1/workflow#--print_json-2) * [\--query](https://docs.temporal.io/tctl-v1/workflow#--query-3) * [\--pagesize](https://docs.temporal.io/tctl-v1/workflow#--pagesize-1) * [\--all](https://docs.temporal.io/tctl-v1/workflow#--all) * [observe](https://docs.temporal.io/tctl-v1/workflow#observe) * [\--workflow\_id](https://docs.temporal.io/tctl-v1/workflow#--workflow_id-4) * [\--run\_id](https://docs.temporal.io/tctl-v1/workflow#--run_id-2) * [\--show\_detail](https://docs.temporal.io/tctl-v1/workflow#--show_detail) * [\--max\_field\_length](https://docs.temporal.io/tctl-v1/workflow#--max_field_length) * [observeid](https://docs.temporal.io/tctl-v1/workflow#observeid) * [\--show\_detail](https://docs.temporal.io/tctl-v1/workflow#--show_detail-1) * [\--max\_field\_length](https://docs.temporal.io/tctl-v1/workflow#--max_field_length-1) * [query](https://docs.temporal.io/tctl-v1/workflow#query) * [\--workflow\_id](https://docs.temporal.io/tctl-v1/workflow#--workflow_id-5) * [\--run\_id](https://docs.temporal.io/tctl-v1/workflow#--run_id-3) * [\--query\_type](https://docs.temporal.io/tctl-v1/workflow#--query_type) * [\--input](https://docs.temporal.io/tctl-v1/workflow#--input) * [\--input\_file](https://docs.temporal.io/tctl-v1/workflow#--input_file) * [\--query\_reject\_condition](https://docs.temporal.io/tctl-v1/workflow#--query_reject_condition) * [reset](https://docs.temporal.io/tctl-v1/workflow#reset) * [\--workflow\_id](https://docs.temporal.io/tctl-v1/workflow#--workflow_id-6) * [\--run\_id](https://docs.temporal.io/tctl-v1/workflow#--run_id-4) * [\--event\_id](https://docs.temporal.io/tctl-v1/workflow#--event_id) * [\--reason](https://docs.temporal.io/tctl-v1/workflow#--reason) * [\--reset\_type](https://docs.temporal.io/tctl-v1/workflow#--reset_type) * [\--reset\_reapply\_type](https://docs.temporal.io/tctl-v1/workflow#--reset_reapply_type) * [\--reset\_bad\_binary\_checksum](https://docs.temporal.io/tctl-v1/workflow#--reset_bad_binary_checksum) * [reset-batch](https://docs.temporal.io/tctl-v1/workflow#reset-batch) * [\--input\_file](https://docs.temporal.io/tctl-v1/workflow#--input_file-1) * [\--query](https://docs.temporal.io/tctl-v1/workflow#--query-4) * [\--exclude\_file](https://docs.temporal.io/tctl-v1/workflow#--exclude_file) * [\--input\_separator](https://docs.temporal.io/tctl-v1/workflow#--input_separator) * [\--reason](https://docs.temporal.io/tctl-v1/workflow#--reason-1) * [\--input\_parallism](https://docs.temporal.io/tctl-v1/workflow#--input_parallism) * [\--skip\_current\_open](https://docs.temporal.io/tctl-v1/workflow#--skip_current_open) * [\--skip\_base\_is\_not\_current](https://docs.temporal.io/tctl-v1/workflow#--skip_base_is_not_current) * [\--only\_non\_deterministic](https://docs.temporal.io/tctl-v1/workflow#--only_non_deterministic) * [\--dry\_run](https://docs.temporal.io/tctl-v1/workflow#--dry_run) * [\--reset\_type](https://docs.temporal.io/tctl-v1/workflow#--reset_type-1) * [\--reset\_bad\_binary\_checksum](https://docs.temporal.io/tctl-v1/workflow#--reset_bad_binary_checksum-1) * [run](https://docs.temporal.io/tctl-v1/workflow#run) * [\--taskqueue](https://docs.temporal.io/tctl-v1/workflow#--taskqueue) * [\--workflow\_id](https://docs.temporal.io/tctl-v1/workflow#--workflow_id-7) * [\--workflow\_type](https://docs.temporal.io/tctl-v1/workflow#--workflow_type-2) * [\--execution\_timeout](https://docs.temporal.io/tctl-v1/workflow#--execution_timeout) * [\--workflow\_task\_timeout](https://docs.temporal.io/tctl-v1/workflow#--workflow_task_timeout) * [\--cron](https://docs.temporal.io/tctl-v1/workflow#--cron) * [\--workflowidreusepolicy](https://docs.temporal.io/tctl-v1/workflow#--workflowidreusepolicy) * [\--input](https://docs.temporal.io/tctl-v1/workflow#--input-1) * [\--input\_file](https://docs.temporal.io/tctl-v1/workflow#--input_file-2) * [\--memo\_key](https://docs.temporal.io/tctl-v1/workflow#--memo_key) * [\--memo](https://docs.temporal.io/tctl-v1/workflow#--memo) * [\--memo\_file](https://docs.temporal.io/tctl-v1/workflow#--memo_file) * [\--search\_attr\_key](https://docs.temporal.io/tctl-v1/workflow#--search_attr_key) * [\--search\_attr\_value](https://docs.temporal.io/tctl-v1/workflow#--search_attr_value) * [\--show\_detail](https://docs.temporal.io/tctl-v1/workflow#--show_detail-2) * [\--max\_field\_length](https://docs.temporal.io/tctl-v1/workflow#--max_field_length-2) * [scan](https://docs.temporal.io/tctl-v1/workflow#scan) * [\--print\_raw\_time](https://docs.temporal.io/tctl-v1/workflow#--print_raw_time-2) * [\--print\_datetime](https://docs.temporal.io/tctl-v1/workflow#--print_datetime-3) * [\--print\_memo](https://docs.temporal.io/tctl-v1/workflow#--print_memo-3) * [\--print\_search\_attr](https://docs.temporal.io/tctl-v1/workflow#--print_search_attr-3) * [\--print\_full](https://docs.temporal.io/tctl-v1/workflow#--print_full-3) * [\--print\_json](https://docs.temporal.io/tctl-v1/workflow#--print_json-3) * [\--pagesize](https://docs.temporal.io/tctl-v1/workflow#--pagesize-2) * [\--query](https://docs.temporal.io/tctl-v1/workflow#--query-5) * [show](https://docs.temporal.io/tctl-v1/workflow#show) * [\--workflow\_id](https://docs.temporal.io/tctl-v1/workflow#--workflow_id-8) * [\--run\_id](https://docs.temporal.io/tctl-v1/workflow#--run_id-5) * [\--print\_datetime](https://docs.temporal.io/tctl-v1/workflow#--print_datetime-4) * [\--print\_raw\_time](https://docs.temporal.io/tctl-v1/workflow#--print_raw_time-3) * [\--output\_filename](https://docs.temporal.io/tctl-v1/workflow#--output_filename) * [\--print\_full](https://docs.temporal.io/tctl-v1/workflow#--print_full-4) * [\--print\_event\_version](https://docs.temporal.io/tctl-v1/workflow#--print_event_version) * [\--event\_id](https://docs.temporal.io/tctl-v1/workflow#--event_id-1) * [\--max\_field\_length](https://docs.temporal.io/tctl-v1/workflow#--max_field_length-3) * [\--reset\_points\_only](https://docs.temporal.io/tctl-v1/workflow#--reset_points_only-2) * [showid](https://docs.temporal.io/tctl-v1/workflow#showid) * [\--print\_datetime](https://docs.temporal.io/tctl-v1/workflow#--print_datetime-5) * [\--print\_raw\_time](https://docs.temporal.io/tctl-v1/workflow#--print_raw_time-4) * [\--output\_filename](https://docs.temporal.io/tctl-v1/workflow#--output_filename-1) * [\--print\_full](https://docs.temporal.io/tctl-v1/workflow#--print_full-5) * [\--print\_event\_version](https://docs.temporal.io/tctl-v1/workflow#--print_event_version-1) * [\--event\_id](https://docs.temporal.io/tctl-v1/workflow#--event_id-2) * [\--max\_field\_length](https://docs.temporal.io/tctl-v1/workflow#--max_field_length-4) * [\--reset\_points\_only](https://docs.temporal.io/tctl-v1/workflow#--reset_points_only-3) * [signal](https://docs.temporal.io/tctl-v1/workflow#signal) * [\--workflow\_id](https://docs.temporal.io/tctl-v1/workflow#--workflow_id-9) * [\--run\_id](https://docs.temporal.io/tctl-v1/workflow#--run_id-6) * [\--name](https://docs.temporal.io/tctl-v1/workflow#--name) * [\--input](https://docs.temporal.io/tctl-v1/workflow#--input-2) * [\--input\_file](https://docs.temporal.io/tctl-v1/workflow#--input_file-3) * [stack](https://docs.temporal.io/tctl-v1/workflow#stack) * [\--workflow\_id](https://docs.temporal.io/tctl-v1/workflow#--workflow_id-10) * [\--run\_id](https://docs.temporal.io/tctl-v1/workflow#--run_id-7) * [\--input](https://docs.temporal.io/tctl-v1/workflow#--input-3) * [\--input\_file](https://docs.temporal.io/tctl-v1/workflow#--input_file-4) * [\--query\_reject\_condition](https://docs.temporal.io/tctl-v1/workflow#--query_reject_condition-1) * [start](https://docs.temporal.io/tctl-v1/workflow#start) * [\--taskqueue](https://docs.temporal.io/tctl-v1/workflow#--taskqueue-1) * [\--workflow\_id](https://docs.temporal.io/tctl-v1/workflow#--workflow_id-11) * [\--workflow\_type](https://docs.temporal.io/tctl-v1/workflow#--workflow_type-3) * [\--execution\_timeout](https://docs.temporal.io/tctl-v1/workflow#--execution_timeout-1) * [\--workflow\_task\_timeout](https://docs.temporal.io/tctl-v1/workflow#--workflow_task_timeout-1) * [\--cron](https://docs.temporal.io/tctl-v1/workflow#--cron-1) * [\--workflowidreusepolicy](https://docs.temporal.io/tctl-v1/workflow#--workflowidreusepolicy-1) * [\--input](https://docs.temporal.io/tctl-v1/workflow#--input-4) * [\--input\_file](https://docs.temporal.io/tctl-v1/workflow#--input_file-5) * [\--memo\_key](https://docs.temporal.io/tctl-v1/workflow#--memo_key-1) * [\--memo](https://docs.temporal.io/tctl-v1/workflow#--memo-1) * [\--memo\_file](https://docs.temporal.io/tctl-v1/workflow#--memo_file-1) * [\--search\_attr\_key](https://docs.temporal.io/tctl-v1/workflow#--search_attr_key-1) * [\--search\_attr\_value](https://docs.temporal.io/tctl-v1/workflow#--search_attr_value-1) * [terminate](https://docs.temporal.io/tctl-v1/workflow#terminate) * [\--workflow\_id](https://docs.temporal.io/tctl-v1/workflow#--workflow_id-12) * [\--run\_id](https://docs.temporal.io/tctl-v1/workflow#--run_id-8) * [\--reason](https://docs.temporal.io/tctl-v1/workflow#--reason-2) --- # Understanding Temporal | Temporal Platform Documentation [Skip to main content](https://docs.temporal.io/evaluate/understanding-temporal#__docusaurus_skipToContent_fallback) On this page Temporal offers an entirely new way to build scalable and reliable applications. Build Invincible Apps[​](https://docs.temporal.io/evaluate/understanding-temporal#build-invincible-apps "Direct link to Build Invincible Apps") ------------------------------------------------------------------------------------------------------------------------------------------------ In any complex system, failures are bound to happen. Software engineers spend a lot of time ensuring that what they build can withstand potential failures. Temporal makes your code execution reliable and durable by default. Normally, if a crash occurs then the state of your application's execution is lost. The application has no memory of what happened before the failure, requiring extensive error handling logic and complex recovery code to resume. The process is time-consuming and error-prone, making it difficult to ensure reliability. Temporal tracks the progress of your application. If something goes wrong, like a power outage, it guarantees that your application can pick up right where it left off — it’s like having the ultimate autosave. Offloading the responsibility of failure management from the application to the platform removes the need for extensive recovery coding, testing, and maintenance tasks. ### Durable Execution[​](https://docs.temporal.io/evaluate/understanding-temporal#durable-execution "Direct link to Durable Execution") Temporal is a Durable Execution Platform. Durable Execution ensures that your application behaves correctly despite adverse conditions by guaranteeing that it will run to completion. This shift simplifies the development process. If a failure or a crash happens, your business processes keep running seamlessly without interruptions. Developers shift their focus to business logic rather than infrastructure concerns and create applications that are inherently scalable and maintainable. Thousands of developers trust Temporal for use cases like order processing, customer onboarding, and payment handling because it enables them to build invincible applications that are resilient, durable, and _just work_. With Temporal, your applications keep running, no matter what happens. Temporal Application: The Building Blocks[​](https://docs.temporal.io/evaluate/understanding-temporal#temporal-application-the-building-blocks "Direct link to Temporal Application: The Building Blocks") ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- ### Workflow[​](https://docs.temporal.io/evaluate/understanding-temporal#workflow "Direct link to Workflow") Conceptually, a Workflow is a sequence of steps. You've likely encountered Workflows in your daily life, whether it's: * Using a mobile app to transfer money * Booking a vacation * Filing an expense report * Creating a new employee onboarding process * Deploying cloud infrastructure * Training an AI model A Temporal Workflow is your business logic, defined in code, outlining each step in your process. Temporal isn’t a no-code Workflow engine — it is **Workflows-as-Code**. Instead of dragging and dropping steps in a visual interface, you write your Workflows in code in your favorite programming language, code editor, and other tools. No-code engines eventually hit their limitations however, Temporal gives you full control and flexibility over your business processes. This allows you to build exactly what you need. ### Activities[​](https://docs.temporal.io/evaluate/understanding-temporal#activities "Direct link to Activities") Activities are the individual units of work in your Workflow. Activities are defined as either functions or methods, depending on the programming language. Activities often involve interacting with the outside world, such as sending emails, making network requests, writing to a database, or calling an API, which are prone to failure. You can call Activities directly from your Workflow code. If an Activity fails, Temporal automatically retries it based on your configuration. Since Activities often rely on external systems, transient issues can occur. These include temporary but critical problems like network failures, timeouts, or service outages. You have full control over how often and how many times these retries should happen for each Activity. ### SDK[​](https://docs.temporal.io/evaluate/understanding-temporal#sdk "Direct link to SDK") Developers create Temporal applications by writing code, just like you would to create any other software. A Temporal SDK (software development kit) is an open-source library that developers add to their application to use Temporal. It provides everything needed to build Workflows, Activities, and various other Temporal features in a specific programming language. Temporal offers seven SDKs: .NET, Go, Java, PHP, Python, Ruby, TypeScript. Since Temporal supports multiple programming languages, you can mix-and-match between languages for polyglot teams. You can easily add any Temporal SDK to your current projects without changing the tools you're already using to build and deploy. Temporal fits right into your existing tech stack. Temporal Service[​](https://docs.temporal.io/evaluate/understanding-temporal#temporal-service "Direct link to Temporal Service") --------------------------------------------------------------------------------------------------------------------------------- Temporal has two main parts: 1. Your application 2. The Temporal Service (a set of services and components) At the heart of Temporal architecture is the Temporal Service, which provides durability, scalability, and reliability for your application. Your application communicates with the Temporal Service and the Temporal Service oversees the execution of critical tasks such as making an API call, then records their completion. It maintains a detailed history of each event, which it reliably persists to a database. One of the biggest advantages of the Temporal Service is how it handles failures. The Temporal Service maintains a meticulous record of every step in your Workflows. By keeping a history of every step in your Workflow, it ensures that even if something goes wrong your Workflow can continue from the last successful point. The Temporal Service knows exactly where to resume without losing any work. This saves you from having to write complex error handling code or painstaking recovery mechanisms yourself. You can run the Temporal Service on your own infrastructure or use Temporal Cloud, a managed service that handles operational overhead and offers scalability and expert support. Workers[​](https://docs.temporal.io/evaluate/understanding-temporal#workers "Direct link to Workers") ------------------------------------------------------------------------------------------------------ The real strength of Temporal comes from the combination of your application and the Temporal Service. Whenever your application needs to perform a task, like sending a notification or processing a payment, the Temporal Service orchestrates what needs to be done. Workers, which are part of your application and provided by the Temporal SDK, then carry out the tasks defined in your Workflow. The Worker polls the Temporal Service to see if there are tasks available and the Temporal Service matches the task with the Worker. The Worker runs the Workflow code based on the details specified in the task. This collaboration is crucial for building reliable, scalable, and durable applications. You can run multiple Workers — often dozens, hundreds, or even thousands — to improve application performance and scalability. A common misconception is that the Temporal Service runs your code. In fact, the Worker runs your code and works with your data directly. Temporal applications are secure by design. Workflows and Activities are seamlessly deployed within your infrastructure, fully integrated into your application. Your data is also protected with your encryption libraries and keys. You maintain full control over the security of your application from end to end. Visibility[​](https://docs.temporal.io/evaluate/understanding-temporal#visibility "Direct link to Visibility") --------------------------------------------------------------------------------------------------------------- There are two tools provided by Temporal that allow you to see behind the scenes and interact with your Workflows. These are powerful for debugging uses and provide real-time monitoring of your applications. ### Temporal UI[​](https://docs.temporal.io/evaluate/understanding-temporal#temporal-ui "Direct link to Temporal UI") The Temporal UI is a browser-based user interface that allows you to see the progress of your application. Also known as the Web UI, it can also help you to quickly isolate, debug, and resolve production problems. ![Recent Workflows page](https://docs.temporal.io/img/webui/workflow-details-page-hiw.avif) Recent Workflows page ### Temporal CLI[​](https://docs.temporal.io/evaluate/understanding-temporal#temporal-cli "Direct link to Temporal CLI") The Temporal CLI is a command-line interface tool for managing, monitoring, and debugging Temporal Applications. Through your terminal, you can: * Start a Workflow * Trace the progress of a Workflow * Cancel or terminate a Workflow * And perform other operations The Temporal CLI provides developers with direct access to a Temporal Service for local development purposes. ### Event History[​](https://docs.temporal.io/evaluate/understanding-temporal#event-history "Direct link to Event History") With Temporal, your Workflows can seamlessly recover from crashes. This is made possible by the [Event History](https://docs.temporal.io/workflow-execution/event) , a complete and durable log of everything that has happened in the lifecycle of a Workflow Execution, as well as the ability of the Temporal Service to durably persist the Events during Replay. Temporal uses the Event History to record every step taken along the way. Each time your Workflow Definition makes an API call to execute an Activity or start a Timer for instance, it doesn’t perform the action directly. Instead, it sends a Command to the Temporal Service. A Command is a requested action issued by a Worker to the Temporal Service after a Workflow Task Execution completes. The Temporal Service will act on these Commands such as scheduling an Activity or scheduling a timer. These Commands are then mapped to Events which are persisted in case of failure. For example, if the Worker crashes, the Worker uses the Event History to replay the code and recreate the state of the Workflow Execution to what it was immediately before the crash. It then resumes progress from the point of failure as if the failure never occurred. For a deep dive into the Event History or Commands, visit the Temporal [Encyclopedia page](https://docs.temporal.io/encyclopedia/event-history) or enroll in one of [our courses](https://learn.temporal.io/courses/) . Reliable as Gravity[​](https://docs.temporal.io/evaluate/understanding-temporal#reliable-as-gravity "Direct link to Reliable as Gravity") ------------------------------------------------------------------------------------------------------------------------------------------ Temporal provides effortless durability, allowing applications to run for days, weeks, or even years without interruption even if the underlying infrastructure fails. This is what we call _Durable Execution_. Temporal also represents a paradigm shift in software development. It's not just about making existing patterns more reliable; it's about enabling entirely new approaches to building complex, distributed systems. Temporal simplifies state management and developers don't have to write tons of extra code to handle every possible thing that could go wrong. With built-in scalability, Temporal ensures that your application runs smoothly, no matter its size or complexity. tip Follow one of our tutorials to [Get Started](https://learn.temporal.io/getting_started/) learning how to use a Temporal SDK. Or, jump straight into an [Introduction to Temporal 101](https://learn.temporal.io/courses/temporal_101/) course. Looking for more? Explore Temporal's [Resource Library](https://temporal.io/resources) . * [Build Invincible Apps](https://docs.temporal.io/evaluate/understanding-temporal#build-invincible-apps) * [Durable Execution](https://docs.temporal.io/evaluate/understanding-temporal#durable-execution) * [Temporal Application: The Building Blocks](https://docs.temporal.io/evaluate/understanding-temporal#temporal-application-the-building-blocks) * [Workflow](https://docs.temporal.io/evaluate/understanding-temporal#workflow) * [Activities](https://docs.temporal.io/evaluate/understanding-temporal#activities) * [SDK](https://docs.temporal.io/evaluate/understanding-temporal#sdk) * [Temporal Service](https://docs.temporal.io/evaluate/understanding-temporal#temporal-service) * [Workers](https://docs.temporal.io/evaluate/understanding-temporal#workers) * [Visibility](https://docs.temporal.io/evaluate/understanding-temporal#visibility) * [Temporal UI](https://docs.temporal.io/evaluate/understanding-temporal#temporal-ui) * [Temporal CLI](https://docs.temporal.io/evaluate/understanding-temporal#temporal-cli) * [Event History](https://docs.temporal.io/evaluate/understanding-temporal#event-history) * [Reliable as Gravity](https://docs.temporal.io/evaluate/understanding-temporal#reliable-as-gravity) --- # Performance bottlenecks troubleshooting guide | Temporal Platform Documentation [Skip to main content](https://docs.temporal.io/troubleshooting/performance-bottlenecks#__docusaurus_skipToContent_fallback) On this page This guide outlines common performance bottlenecks in Temporal Workers and Clients. It covers key latency metrics and root causes of high values, and provides diagnostic steps and troubleshooting strategies. These metrics can help you optimize Temporal deployments and Workflow execution. To get the most out of this guide, you should be familiar with [Temporal architecture](https://docs.temporal.io/temporal) , [Workflows](https://docs.temporal.io/workflows) , [Activities](https://docs.temporal.io/activities) , and [Task Queues](https://docs.temporal.io/task-queue) . You should also know how to use key metrics like latency, counter, rate, CPU utilization, and memory usage. Task processing metrics[​](https://docs.temporal.io/troubleshooting/performance-bottlenecks#task-processing-metrics "Direct link to Task processing metrics") -------------------------------------------------------------------------------------------------------------------------------------------------------------- These metrics provide insights into various stages of the [Task](https://docs.temporal.io/tasks) lifecycle, from scheduling to completion. The following sections detail common metrics, their potential causes for high latency or resource depletion, and strategies for diagnosing and resolving performance issues. ### `temporal_workflow_task_schedule_to_start_latency` spike[​](https://docs.temporal.io/troubleshooting/performance-bottlenecks#temporal_workflow_task_schedule_to_start_latency-spike "Direct link to temporal_workflow_task_schedule_to_start_latency-spike") High [`temporal_workflow_task_schedule_to_start_latency`](https://docs.temporal.io/references/sdk-metrics#workflow_task_schedule_to_start_latency) (P95 higher than one second) can be caused by several factors. This metric represents the time between when a [Workflow Task](https://docs.temporal.io/tasks#workflow-task) is scheduled (enqueued) and when it is picked up by a Worker for processing. Here are some potential causes: * Insufficient Worker capacity: If there aren't enough Workers or if the Workers are overloaded, they may not be able to pick up Tasks quickly enough. This can lead to Tasks waiting longer in the queue ([Detect Task Backlog](https://docs.temporal.io/cloud/worker-health#detect-task-backlog) ). * Worker configuration issues: Improperly configured Workers, such as having too few pollers or Task slots, can lead to increased latency ([Detect Task Backlog](https://docs.temporal.io/cloud/worker-health#detect-task-backlog) ). * High Workflow lock latency: If many updates are made to a single execution, this can cause Workflow lock latency, which in turn affects the Schedule-to-start latency. Reduce the rate of Signals. * Network latency: Workers in a different region from the Temporal cluster, or large payload size, can introduce additional latency. To diagnose and address high `temporal_workflow_task_schedule_to_start_latency`, you should: 1. Check Worker CPU and memory usage. 2. Review Worker configuration (number of pollers, Task slots, etc.). 3. Look for any spikes in Workflow or Activity starts that might be overwhelming the system. 4. Ensure Workers are in the same region as the Temporal cluster if possible. ### `temporal_activity_schedule_to_start_latency` spike[​](https://docs.temporal.io/troubleshooting/performance-bottlenecks#temporal_activity_schedule_to_start_latency-spike "Direct link to temporal_activity_schedule_to_start_latency-spike") High [`temporal_activity_schedule_to_start_latency`](https://docs.temporal.io/references/sdk-metrics#activity_schedule_to_start_latency) (P95 higher than one second) can be caused by several factors. This metric represents the time between when an [Activity Task](https://docs.temporal.io/tasks#activity-task) is scheduled (enqueued) and when it is picked up by a Worker for processing. Here are some potential causes: * Insufficient Worker capacity: If there aren't enough Workers or if the Workers are overloaded, they may not be able to pick up Tasks quickly enough. This can lead to Tasks waiting longer in the queue ([Detect Task Backlog](https://docs.temporal.io/cloud/worker-health#detect-task-backlog) ). * Worker configuration issues: Improperly configured Workers, such as having too few pollers or Task slots, can lead to increased latency ([Detect Task Backlog](https://docs.temporal.io/cloud/worker-health#detect-task-backlog) ). * Task Queue configuration: Setting `TaskQueueActivitiesPerSecond` too low can limit the rate at which Activities are started, leading to increased Schedule-to-start latency. * Network latency: Workers in a different region from the Temporal cluster, or large payload size can introduce additional latency. To diagnose and address high `temporal_activity_schedule_to_start_latency`: 1. Check Worker CPU and memory usage. 2. Review Worker configuration (number of pollers, Task slots, etc.). 3. Look for any spikes in Workflow or Activity starts that might be overwhelming the system. 4. Ensure Workers are in the same region as the Temporal cluster if possible. ### `temporal_workflow_endtoend_latency` spike[​](https://docs.temporal.io/troubleshooting/performance-bottlenecks#temporal_workflow_endtoend_latency-spike "Direct link to temporal_workflow_endtoend_latency-spike") The [`temporal_workflow_endtoend_latency`](https://docs.temporal.io/references/sdk-metrics#workflow_endtoend_latency) metric represents the total Workflow Execution time from Schedule to the closure for a single Workflow Run. Normal ranges for this metric depend on the use case, but here are some potential causes for the unexpected spikes: * Complex Workflows: If the Workflows have many Activities or if the Activities take a long time to execute. * Workflow and Activity retries: If Workflows or Activities are configured to retry upon failure and they fail often, this can increase the end-to-end latency as the system will wait for the retry delay before reattempting the failed operation. * Worker capacity and configuration: If there aren't enough Workers or if the Workers are overloaded, they may not be able to pick up and process Tasks quickly enough. This can lead to Tasks waiting longer in the queue, thereby increasing the end-to-end latency ([Detect Task Backlog](https://docs.temporal.io/cloud/worker-health#detect-task-backlog) ). * External dependencies: If your Workflows or Activities depend on external systems or services (such as databases or APIs) and these systems are slow or unreliable, they can increase the end-to-end latency. * Network latency: Workers in a different region from the Temporal cluster can introduce additional latency. To diagnose and address high `temporal_workflow_endtoend_latency`: 1. Review your Workflow and Activity designs to ensure they are as efficient as possible. 2. Monitor your Workers to ensure they have sufficient capacity (CPU and memory) and are not overloaded. 3. Monitor your external dependencies to ensure they are performing well. 4. Ensure Workers are in the same region as the Temporal cluster if possible. ### High `temporal_workflow_task_execution_latency`[​](https://docs.temporal.io/troubleshooting/performance-bottlenecks#high-temporal_workflow_task_execution_latency "Direct link to high-temporal_workflow_task_execution_latency") The [`temporal_workflow_task_execution_latency`](https://docs.temporal.io/references/sdk-metrics#workflow_task_execution_latency) metric represents the time taken by a Worker to execute a Workflow Task. The Temporal SDK raises a “Deadlock detected during Workflow run” error or [TMPRL1101](https://github.com/temporalio/rules/blob/main/rules/TMPRL1101.md) when a Workflow Task takes more than one or two seconds to complete. Here are some potential causes: * CPU-intensive work: Performing CPU-intensive operations in your Workflow Task can lead to slow execution. * Slow local Activities: Workflow Task execution time includes the Local Activity execution time. * Slow Workflow replay: Workflow Task execution time includes the Workflow Replay time. Refer to `workflow_task_replay_latency` for more details. * Worker resource constraints: High CPU usage on Worker pods can lead to slower Workflow Task execution. Workers with insufficient CPU resources can cause delays. * Infinite loops or blocking calls: Workflow code with infinite loops or blocking external API calls can cause the Workflow Task to execute slowly or time out. * Slow data conversion: Your custom Data Converter is taking too long to encode/decode payloads, for example, when talking to a remote encryption service. To diagnose and address slow Workflow Task execution, you can: 1. Monitor Worker CPU and memory utilization. 2. Ensure that your Workers have adequate resources and are properly scaled for your workload. 3. Consider running your Workflow code in a profiler using a replayer to see where CPU cycles are spent. 4. Review your Workflow code for potential optimizations or to remove blocking operations. 5. Disable deadlock detection for Data Converter: It does not reduce Task execution latency but does remove the “Deadlock detected during Workflow run” or TMPRL1101 error. In Go, wrap it with `workflow.DataConverterWithoutDeadlockDetection`. In Java, surround your Data Converter code with `WorkflowUnsafe.deadlockDetectorOff`. ### High `workflow_task_replay_latency`[​](https://docs.temporal.io/troubleshooting/performance-bottlenecks#high-workflow_task_replay_latency "Direct link to high-workflow_task_replay_latency") Workflow Task replay is the process of reconstructing the Workflow's state by re-executing the Workflow code from the beginning, using the recorded Event History. This process ensures that the Workflow can continue from where it left off, even after interruptions or failures. [`workflow_task_replay_latency`](https://docs.temporal.io/references/sdk-metrics#workflow_task_replay_latency) is high if it exceeds a few milliseconds. Here are the main causes: * Large Event Histories: Workflows with long histories take more time to replay, as the Worker needs to process all events to reconstruct the Workflow state. * Data Converter performance: Slow Data Converters, especially those that perform encryption or interact with external services, can impact replay. * Large payloads: Activities or Signals with large payloads can slow down the replay process, especially if the Data Converter needs to process these payloads. * Complex Workflow logic: Workflow code with complex logic or computationally intensive operations, such as scheduling many concurrent child Workflows or Activities, can increase replay latency. * Frequent cache evictions: If workers often evict Workflow Executions from their cache (due to memory constraints or frequent restarts), it leads to more replays and higher latency. * Worker resource constraints: High CPU utilization or memory pressure on Worker nodes can slow down the replay. To diagnose and address slow Workflow Task replay, you can: 1. Monitor SDK Metrics: Keep a close eye on the `temporal_workflow_task_replay_latency` metric. This histogram metric measures the time it takes to replay a Workflow Task. 2. Analyze Workflow History Size: Check the number of events in your Workflow histories and consider using the "Continue-As-New" feature for long-running Workflows. 3. Optimize Data Converters: If you're using custom Data Converters, especially for encryption or complex serialization, look for opportunities to optimize their performance. 4. Review Payload Sizes: Large Activity or Signal payloads can slow down replay. Consider optimizing the size of data being passed in your Workflows. 5. Profile Workflow Code: Use a profiler to identify CPU-intensive parts of your Workflow code that might be slowing down replay. 6. Manage Worker Cache: Frequent cache evictions can lead to more replays. Tune your Worker's cache size and eviction policies. ### `temporal_activity_execution_latency` spike[​](https://docs.temporal.io/troubleshooting/performance-bottlenecks#temporal_activity_execution_latency-spike "Direct link to temporal_activity_execution_latency-spike") The [`temporal_activity_execution_latency`](https://docs.temporal.io/references/sdk-metrics#activity_execution_latency) metric measures the time from when a Worker starts processing an Activity Task until it reports to the service that the Task is complete or failed. There are several potential causes for high `temporal_activity_execution_latency`: * Activity implementation: The most common cause of high Activity Execution latency is the actual implementation of the Activity itself. If the Activity is performing time-consuming operations or making slow external API calls, it will take longer to execute. * External dependencies: If your Activity is constrained by an external resource or service that all Activities access, it could cause increased latency. * Worker resource constraints: Under-resourced Worker nodes or experiencing high CPU utilization can lead to slower Activity Execution. * Network latency: High latency between your Workers and external services or the Temporal service itself can contribute to increased Activity Execution time. To diagnose and address high Activity Execution latency: 1. Monitor the `activity_execution_latency` metric, which you can filter by Activity type and Activity Task queue. 2. Optimize your Activity implementation to reduce latency, especially with external services or database interactions. 3. Check your Worker CPU and memory utilization to make sure they have adequate resources. 4. Examine your Worker configuration, particularly `(Max)ConcurrentActivityExecutionSize` and `(Max)WorkerActivitiesPerSecond`, to ensure they are not limiting your activity execution. ### Depletion of `temporal_worker_task_slots_available` for `WorkflowWorker`[​](https://docs.temporal.io/troubleshooting/performance-bottlenecks#depletion-of-temporal_worker_task_slots_available-for-workflowworker "Direct link to depletion-of-temporal_worker_task_slots_available-for-workflowworker") The [`temporal_worker_task_slots_available{worker_type=”WorkflowWorker”}`](https://docs.temporal.io/references/sdk-metrics#worker_task_slots_available) metric indicates the number of available slots for executing Workflow Tasks on a Worker. This metric may go to zero for several reasons: * High Workflow Task Load: If there are more Tasks than the Worker can handle concurrently, the available slots will be depleted. This can happen if the rate of incoming Tasks is higher than the rate at which tasks are being completed. * Worker Configuration: The number of available slots is determined by the Worker configuration, specifically the `MaxConcurrentWorkflowTaskExecutionSize` setting. If these are set too low, the Worker may not have enough slots to handle the Task load. * High `temporal_workflow_task_execution_latency` and `workflow_task_replay_latency`. To prevent depletion of Workflow Task slots: 1. Monitor Worker CPU and Memory usage while increasing `(Max)ConcurrentWorkflowTaskExecutionSize` to add more execution slots. 2. Scale up Workers both vertically (increasing CPU and Memory) and horizontally (increasing Worker instances). ### Depletion of `temporal_worker_task_slots_available` for `ActivityWorker`[​](https://docs.temporal.io/troubleshooting/performance-bottlenecks#depletion-of-temporal_worker_task_slots_available-for-activityworker "Direct link to depletion-of-temporal_worker_task_slots_available-for-activityworker") The [`temporal_worker_task_slots_available{worker_type=”ActivityWorker”}`](https://docs.temporal.io/references/sdk-metrics#worker_task_slots_available) metric indicates the number of available slots for executing Activity Tasks on a Worker. This metric may go to zero for several reasons: * Blocked Activities and Zombie Activities: The most common cause is activities that are blocked or not returning on time. Zombie Activities are a subset of this category. They occur when an Activity times out (hits its `StartToClose` or `HeartbeatTimeout` timeout) and has stopped Heartbeating but continues to run, occupying some or all the slots as more retries occur. This can happen if: * The Activity code is blocking on a downstream service call or an infinite loop. * There's a mismatch between the Activity's `StartToClose` timeout and any client-side timeouts for external calls. * Resource Utilization: High CPU or memory usage on Workers can cause activities to block and not release slots. To prevent depletion of Activity Task slots: 1. Monitor Worker CPU and Memory usage while increasing `(Max)ConcurrentActivityExecutionSize` to add more execution slots. 2. Add client-side timeout to your downstream API client. 3. Review your Task code to ensure Tasks complete within a reasonable time measured by `temporal_activity_execution_latency`. Network requests[​](https://docs.temporal.io/troubleshooting/performance-bottlenecks#network-requests "Direct link to Network requests") ----------------------------------------------------------------------------------------------------------------------------------------- Network issues can impact Temporal clients and workers, leading to delays, failures, and overall system instability. This section focuses on metrics that reveal common network-related problems with your Temporal deployment, specifically related to network connectivity, latency, and request failures. These metrics can indicate where bottlenecks exist within the communication channels between Temporal clients (including Temporal Workers) and the Temporal server. ### High `temporal_long_request_failure`[​](https://docs.temporal.io/troubleshooting/performance-bottlenecks#high-temporal_long_request_failure "Direct link to high-temporal_long_request_failure") The [`temporal_long_request_failure`](https://docs.temporal.io/references/sdk-metrics#long_request_failure) metric counts the number of failed RPC long poll requests for `PollWorkflowTaskQueue`, `PollActivityTaskQueue`, and `GetWorkflowExecutionHistory` (when polling new events). High values of this metric can be caused by several factors: * Network Issues: Problems with the network connection between the Temporal Client and the Temporal Server, including firewalls and proxies, can cause long poll requests to fail. * Rate Limiting: If the rate of requests exceeds the configured limits on the Temporal Server or Temporal Cloud, additional requests may be rejected, increasing the `temporal_long_request_failure` count. This is often indicated by a `ResourceExhausted` status code. * Server Errors: If the Temporal Server is experiencing issues, it may fail to respond to long poll requests correctly, leading to an increase in `temporal_long_request_failure`. To diagnose the cause of high `temporal_long_request_failure`, you can: 1. Check the operation and the status or code tag of the `temporal_long_request_failure` metric to see the type of errors that are occurring. 2. If you receive a `ResourceExhausted` status code, review the rate limits configured on the Temporal Server or ask for help from Temporal Support for Temporal Cloud. 3. Check the network connection between the Temporal Client and the Temporal Server. ### High `temporal_request_failure_total`[​](https://docs.temporal.io/troubleshooting/performance-bottlenecks#high-temporal_request_failure_total "Direct link to high-temporal_request_failure_total") The [`temporal_request_failure_total`](https://docs.temporal.io/references/sdk-metrics#request_failure) metric counts the number of RPC requests made by the Temporal Client that have failed. High values of this metric can be caused by several factors: * Network Issues: Problems with the network connection between the Temporal Client and the Temporal Server can cause requests to fail. * Client Errors: If there's an issue with the Temporal Client, such as misconfiguration or resource exhaustion, it may fail to make requests correctly. * Operation Errors: Specific operations like `SignalWorkflowExecution` or `TerminateWorkflowExecution` can fail if they are trying to act on a closed Workflow Execution that no longer exists (because it completed and was removed from persistence when it hit Namespace retention time). * Rate Limiting: If the rate of requests exceeds the configured limits on the Temporal Server, additional requests may be rejected, increasing the counter. This is often indicated by a `ResourceExhausted` status code. * Request Size Limit: If the Worker tries to return an Activity response that is larger than the blob size limit (2MB), the service will reject it, causing a request failure. * Server Errors: If the Temporal Server is experiencing issues, it may fail to respond to requests correctly, leading to an increase in `temporal_request_failure_total`. To diagnose the cause of high `temporal_request_failure_total`, you can: 1. Check the status or code tag of the `temporal_request_failure_total` metric to see the type of errors that are occurring. 2. Look at the operation tag of the `temporal_request_failure_total` metric to see which operations are failing. 3. Monitor the Temporal Server logs and the Temporal Client logs for any error messages or warnings. 4. Check the network connection between the Temporal Client and the Temporal Server. ### High `temporal_request_latency`[​](https://docs.temporal.io/troubleshooting/performance-bottlenecks#high-temporal_request_latency "Direct link to high-temporal_request_latency") The [`temporal_request_latency`](https://docs.temporal.io/references/sdk-metrics#request_latency) metric measures the latency of gRPC requests made by the Temporal Client. High values for this metric can be caused by several factors: * Network Latency: The physical distance and network conditions between the Temporal Client and the Temporal Server can affect the latency of requests. * Network Transfer Time: Larger payloads take longer to transfer over the network, which affects request latency. For example, large payloads in `RespondWorkflowTaskCompleted` can affect the latency of the request. This is especially true when Workflows are scheduling multiple activities with large inputs. * Resource Exhaustion: Running out of resources (such as CPU or memory) on the client or server can cause delays in processing the request. * Client Configuration: Improper client configuration, such as setting thread pool sizes too aggressively or having memory constraints that are too low for the number of allocated threads, can lead to situations where Tasks overwhelm the client, causing increased latency. * Server Load: If the Temporal Server is under heavy load, it may take longer to respond to requests, leading to increased latency. To diagnose and address high `temporal_request_latency`: 1. Monitor the `temporal_request_latency` metric to identify when and where latency spikes are occurring. 2. Check the network connection between the Temporal Client and the Temporal Server. 3. Monitor the resource usage on both the Temporal Client and the Temporal Server. 4. Review your Temporal Client configuration to ensure it is optimized for your workload. 5. If you're using Temporal Cloud, check if the Cloud’s [service-latency](https://docs.temporal.io/cloud/metrics/reference#service-latency) metric spikes up and reach out to Temporal Support for help. ### `rate(temporal_long_request_total{operation="PollActivityTaskQueue"})`[​](https://docs.temporal.io/troubleshooting/performance-bottlenecks#ratetemporal_long_request_totaloperationpollactivitytaskqueue "Direct link to ratetemporal_long_request_totaloperationpollactivitytaskqueue") The [`rate(temporal_long_request_total{operation="PollActivityTaskQueue"})`](https://docs.temporal.io/references/sdk-metrics#long_request) expression measures the per-second average rate of `PollActivityTaskQueue` long poll requests over a certain period of time. `PollActivityTaskQueue` is an operation where Workers poll for Activity Tasks from the Task Queue. The `temporal_long_request_total` metric counts the number of these long poll requests. By applying the `rate()` function in Prometheus, you can calculate the per-second average rate of these requests over the time range specified in the Query. This can help you understand the load on your Temporal service and how often your Workers are polling for Activity Tasks. ### `rate(temporal_long_request_total{operation="PollWorkflowTaskQueue"})`[​](https://docs.temporal.io/troubleshooting/performance-bottlenecks#ratetemporal_long_request_totaloperationpollworkflowtaskqueue "Direct link to ratetemporal_long_request_totaloperationpollworkflowtaskqueue") The [`rate(temporal_long_request_total{operation="PollWorkflowTaskQueue"})`](https://docs.temporal.io/references/sdk-metrics#long_request) expression measures the per-second average rate of `PollWorkflowTaskQueue` long poll requests over a certain period of time. `PollWorkflowTaskQueue` is an operation where Workers poll for Workflow Tasks from the Task Queue. The `temporal_long_request_total` metric counts the number of these long poll requests. By applying the `rate()` function in Prometheus, you can calculate the per-second average rate of these requests over the time range specified in the query. This can help you understand the load on your Temporal service and how often your Workers are polling for Workflow Tasks. Caching[​](https://docs.temporal.io/troubleshooting/performance-bottlenecks#caching "Direct link to Caching") -------------------------------------------------------------------------------------------------------------- Temporal Workers rely on caching to optimize performance by reducing the overhead of fetching Workflow state from the history and Replaying. However, unlimited caching is impossible; there's a trade-off between the benefits of cached data and the memory consumed. These metrics allow you to balance performance gains with responsible memory usage. ### `temporal_sticky_cache_size`[​](https://docs.temporal.io/troubleshooting/performance-bottlenecks#temporal_sticky_cache_size "Direct link to temporal_sticky_cache_size") The [`temporal_sticky_cache_size`](https://docs.temporal.io/references/sdk-metrics#sticky_cache_size) metric represents the number of Workflow executions currently cached in a Worker's memory. The sticky cache is used to improve performance by keeping the Workflow state in memory, reducing the need to reconstruct the Workflow from its Event History for every Task. It’s particularly useful for latency-sensitive Workflows. There is a direct relationship between the sticky cache size and Worker memory consumption. As the cache size increases, so does the memory usage of the Worker. The maximum size of the sticky cache can be configured. For example, the default in the Go SDK is 10,000 Workflows. A larger sticky cache can improve performance by reducing the need to replay Workflow histories. However, it also increases memory usage, which can lead to issues if not properly managed. Monitor this metric alongside Worker memory usage. A sudden increase in `sticky_cache_size` can correlate with increased memory consumption and potential performance issues. If memory consumption is too high, you can reduce the maximum sticky cache size. Conversely, if you have available memory and want to improve performance, you might increase it. ### `temporal_sticky_cache_hit_total` and `temporal_sticky_cache_miss_total`[​](https://docs.temporal.io/troubleshooting/performance-bottlenecks#temporal_sticky_cache_hit_total-and-temporal_sticky_cache_miss_total "Direct link to temporal_sticky_cache_hit_total-and-temporal_sticky_cache_miss_total") The [`temporal_sticky_cache_hit_total`](https://docs.temporal.io/references/sdk-metrics#sticky_cache_hit) metric is a counter that measures the total number of times a Workflow Task found a cached Workflow Execution to run against, and the opposite is [`temporal_sticky_cache_miss_total`](https://docs.temporal.io/references/sdk-metrics#sticky_cache_miss) , which is a counter that measures the total number of times a Workflow Task did not find a cached Workflow Execution to run against. Sticky Execution is a feature where a Worker caches a Workflow Execution and creates a dedicated Task Queue to listen on. This improves performance because the Temporal Service only sends new events to the Worker instead of entire Event Histories, and the Workflow doesn't have to Replay. A “hit” means the Worker finds the Workflow in its cache when processing a Workflow Task, allowing immediate processing without fetching the full Event History from the server and Replaying. A "miss" means the Worker didn't find the Workflow in its cache, and it must fetch the Event History and Replay. Monitoring these two metrics and comparing them can help you understand how your sticky cache is being used. A high rate of cache hits with a low rate of cache misses indicates that your Workflows are being scheduled efficiently, with minimal need for fetching Event Histories and Replaying. ### `temporal_sticky_cache_total_forced_eviction_total`[​](https://docs.temporal.io/troubleshooting/performance-bottlenecks#temporal_sticky_cache_total_forced_eviction_total "Direct link to temporal_sticky_cache_total_forced_eviction_total") The [`temporal_sticky_cache_total_forced_eviction_total`](https://docs.temporal.io/references/sdk-metrics#sticky_cache_hit) metric is a counter that measures the total number of Workflow Executions that have been forcibly evicted from the sticky cache. Sticky Execution is a feature where a Worker caches a Workflow Execution and creates a dedicated Task Queue to listen on. This improves performance because the Temporal Service only sends new events to the Worker instead of entire Event Histories, and the Workflow doesn't have to Replay. A "forced eviction" in this context means that a Workflow Execution was removed from the cache before it completed, typically because the cache was full and needed to make room for other Workflow Executions. This means that if the Worker needs to process more Tasks for the evicted Workflow Execution, it will have to fetch the entire Event History from the Temporal Service and Replay. Monitoring the `temporal_sticky_cache_total_forced_eviction_total` metric can help you understand how often your Workflows are being evicted from the cache. A high rate of forced evictions could indicate that your cache size is too small for your workload, and you may need to increase the `WorkflowCacheSize` setting if your Worker resources can accommodate it. * [Task processing metrics](https://docs.temporal.io/troubleshooting/performance-bottlenecks#task-processing-metrics) * [`temporal_workflow_task_schedule_to_start_latency` spike](https://docs.temporal.io/troubleshooting/performance-bottlenecks#temporal_workflow_task_schedule_to_start_latency-spike) * [`temporal_activity_schedule_to_start_latency` spike](https://docs.temporal.io/troubleshooting/performance-bottlenecks#temporal_activity_schedule_to_start_latency-spike) * [`temporal_workflow_endtoend_latency` spike](https://docs.temporal.io/troubleshooting/performance-bottlenecks#temporal_workflow_endtoend_latency-spike) * [High `temporal_workflow_task_execution_latency`](https://docs.temporal.io/troubleshooting/performance-bottlenecks#high-temporal_workflow_task_execution_latency) * [High `workflow_task_replay_latency`](https://docs.temporal.io/troubleshooting/performance-bottlenecks#high-workflow_task_replay_latency) * [`temporal_activity_execution_latency` spike](https://docs.temporal.io/troubleshooting/performance-bottlenecks#temporal_activity_execution_latency-spike) * [Depletion of `temporal_worker_task_slots_available` for `WorkflowWorker`](https://docs.temporal.io/troubleshooting/performance-bottlenecks#depletion-of-temporal_worker_task_slots_available-for-workflowworker) * [Depletion of `temporal_worker_task_slots_available` for `ActivityWorker`](https://docs.temporal.io/troubleshooting/performance-bottlenecks#depletion-of-temporal_worker_task_slots_available-for-activityworker) * [Network requests](https://docs.temporal.io/troubleshooting/performance-bottlenecks#network-requests) * [High `temporal_long_request_failure`](https://docs.temporal.io/troubleshooting/performance-bottlenecks#high-temporal_long_request_failure) * [High `temporal_request_failure_total`](https://docs.temporal.io/troubleshooting/performance-bottlenecks#high-temporal_request_failure_total) * [High `temporal_request_latency`](https://docs.temporal.io/troubleshooting/performance-bottlenecks#high-temporal_request_latency) * [`rate(temporal_long_request_total{operation="PollActivityTaskQueue"})`](https://docs.temporal.io/troubleshooting/performance-bottlenecks#ratetemporal_long_request_totaloperationpollactivitytaskqueue) * [`rate(temporal_long_request_total{operation="PollWorkflowTaskQueue"})`](https://docs.temporal.io/troubleshooting/performance-bottlenecks#ratetemporal_long_request_totaloperationpollworkflowtaskqueue) * [Caching](https://docs.temporal.io/troubleshooting/performance-bottlenecks#caching) * [`temporal_sticky_cache_size`](https://docs.temporal.io/troubleshooting/performance-bottlenecks#temporal_sticky_cache_size) * [`temporal_sticky_cache_hit_total` and `temporal_sticky_cache_miss_total`](https://docs.temporal.io/troubleshooting/performance-bottlenecks#temporal_sticky_cache_hit_total-and-temporal_sticky_cache_miss_total) * [`temporal_sticky_cache_total_forced_eviction_total`](https://docs.temporal.io/troubleshooting/performance-bottlenecks#temporal_sticky_cache_total_forced_eviction_total) --- # Worker Versioning | Temporal Platform Documentation [Skip to main content](https://docs.temporal.io/production-deployment/worker-deployments/worker-versioning#__docusaurus_skipToContent_fallback) On this page Worker Versioning is a Temporal feature that allows you to confidently deploy new changes to the Workflows running on your Workers without breaking them. Temporal enables this by helping you manage different builds or versions, formally called [Worker Deployment Versions](https://docs.temporal.io/worker-versioning#deployment-versions) . For most teams, Worker Versioning should be the default recommendation for deploying Workflow code changes in production. If you can run versioned worker deployments, prefer Worker Versioning over patching. Worker Versioning unlocks important benefits for users of [blue-green or rainbow deployments](https://docs.temporal.io/production-deployment/worker-deployments/worker-versioning#deployment-systems) . * Ramping traffic gradually to a new Worker Deployment Version. * Verifying a new Deployment Version with tests before sending production traffic to it. * Instant rollback when you detect that a new Deployment Version is broken. * Improved error rates when adopting it. In addition, Worker Versioning introduces **Workflow Pinning**. For pinned Workflow Types, each execution runs entirely on the Worker Deployment Version where it started. You need not worry about making breaking code changes to running, pinned Workflows. To use Workflow Pinning, we recommend using [rainbow deployments](https://docs.temporal.io/production-deployment/worker-deployments/worker-versioning#deployment-systems) . tip Watch this Temporal Replay 2025 talk to learn more about Worker Versioning and see a demo. note Worker Versioning is currently available in Public Preview. Minimum versions: * Go SDK version [v1.35.0](https://github.com/temporalio/sdk-go/releases/tag/v1.35.0) * Python [v1.11](https://github.com/temporalio/sdk-python/releases/tag/1.11.0) * Java [v1.29](https://github.com/temporalio/sdk-java/releases/tag/v1.29.0) * Typescript [v1.12](https://github.com/temporalio/sdk-typescript/releases/tag/v1.12.0) * .NET [v1.7.0](https://github.com/temporalio/sdk-dotnet/releases/tag/1.7.0) * Ruby [v0.5.0](https://github.com/temporalio/sdk-ruby/releases/tag/v0.5.0) * Other SDKs: coming soon! Self-hosted users: * Minimum Temporal CLI version [v1.4.1](https://github.com/temporalio/cli/releases/tag/v1.4.1) * Minimum Temporal Server version: [v1.29.1](https://github.com/temporalio/temporal/releases/tag/v1.29.1) * Minimum Temporal UI Version [v2.38.0](https://github.com/temporalio/ui/releases/tag/v2.38.0) Getting Started with Worker Versioning[​](https://docs.temporal.io/production-deployment/worker-deployments/worker-versioning#definition "Direct link to Getting Started with Worker Versioning") -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- To get started with Worker Versioning, you should understand some concepts around versioning and deployments. * A **Worker Deployment** is a deployment or service across multiple versions. In a rainbow deployment, more than two active Deployment Versions can run at once. * A **Worker Deployment Version** is a version of a deployment or service. It can have multiple Workers polling on multiple Task Queues, but they all run the same build. * A **Build ID**, in combination with a Worker Deployment name, identifies a single Worker Deployment Version. * When a versioned worker polls on a task queue, that task queue becomes part of that Worker's version. That version's Worker Deployment controls how the task queue matches Workflow Tasks with Workers. * Using **Workflow Pinning**, you can declare each Workflow type to have a **Versioning Behavior**, either Pinned or Auto-Upgrade. * A **Pinned** Workflow is guaranteed to complete on a single Worker Deployment Version. * An **Auto-Upgrade** Workflow will automatically move to a new code version as you roll it out, specifically its Target Worker Deployment Version (defined below). Therefore, Auto-Upgrade Workflows are not restricted to a single Deployment Version and need to be kept replay-safe manually, i.e. with [patching](https://docs.temporal.io/workflow-definition#workflow-versioning) . * Both Pinned and Auto-Upgrade Workflows are guaranteed to start only on the Current or Ramping Version of their Worker Deployment. * Pinned Workflows are designed for use with rainbow deployments. See [Deployment Systems](https://docs.temporal.io/production-deployment/worker-deployments/worker-versioning#deployment-systems) . * Pinned Workflows don't need to be patched, as they run on the same worker and build until they complete. * If you expect your Workflow to run longer than you want your Worker Deployment Versions to exist, you should mark your Workflow Type as Auto-Upgrade. * Each Worker Deployment has a single [**Current Version**](https://docs.temporal.io/worker-versioning#versioning-definitions) which is where Workflows are routed to unless they were previously pinned on a different version. * Each Worker Deployment can have a [**Ramping Version**](https://docs.temporal.io/worker-versioning#versioning-definitions) which is where a configurable percentage of Workflows are routed to unless they were previously pinned on a different version. * For a given Workflow, its [**Target Worker Deployment Version**](https://docs.temporal.io/worker-versioning#versioning-definitions) is the version it will move to next. Setting up your deployment system[​](https://docs.temporal.io/production-deployment/worker-deployments/worker-versioning#deployment-systems "Direct link to Setting up your deployment system") ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ If you haven't already, you'll want to pick a container deployment solution for your Workers. You also need to pick among three common deployment strategies: * A **rolling deployment** strategy upgrades Workers in place with little control over how quickly they cut over and only a slow ability to roll Workers back. Rolling deploys have minimal footprint but tend to provide lower availability than the other strategies and are incompatible with Worker Versioning. * A **blue-green deployment** strategy maintains two "colors," or Worker Deployment Versions simultaneously and can control how traffic is routed between them. This allows you to maximize your uptime with features like instant rollback and ramping. Worker Versioning enables the routing control that blue-green deployments need. * A **rainbow deployment** strategy is like blue-green but with more colors, allowing Workflow Pinning. You can deploy new revisions of your Workflows freely while older versions drain. Using Worker Versioning, Temporal lets you know when all the Workflows of a given version are drained so that you can sunset it. note You also have the option to use the [Temporal Worker Controller](https://docs.temporal.io/production-deployment/worker-deployments/kubernetes-controller) to automatically enable rainbow deployments of your Workers if you're using Kubernetes. If you cannot yet support blue-green or rainbow style deployments, use [patching](https://docs.temporal.io/patching) as a fallback while you work toward a versioned deployment model. Configuring a Worker for Versioning[​](https://docs.temporal.io/production-deployment/worker-deployments/worker-versioning#configuring-a-worker-for-versioning "Direct link to Configuring a Worker for Versioning") --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- You'll need to add a few additional configuration parameters to your Workers to toggle on Worker Versioning. There are three new parameters, with different names depending on the language: * `UseVersioning`: This enables the Versioning functionality for this Worker. * A `Version` to identify the revision that this Worker will be allowed to execute. This is a combination of a deployment name and a build ID number. * (Optional) The [Default Versioning Behavior](https://docs.temporal.io/production-deployment/worker-deployments/worker-versioning#definition) . If unset, you'll be required to specify the behavior on each Workflow. Or you can default to Pinned or Auto-Upgrade. Follow the example for your SDK below: * Go * Java * Python * TypeScript * PHP * .NET * Ruby buildID:= mustGetEnv("MY_BUILD_ID")w := worker.New(c, myTaskQueue, worker.Options{ DeploymentOptions: worker.DeploymentOptions{ UseVersioning: true, Version: worker.WorkerDeploymentVersion{ DeploymentName: "llm_srv", BuildId: buildID, }, DefaultVersioningBehavior: workflow.VersioningBehaviorUnspecified, },}) import io.temporal.worker.WorkerOptions;import io.temporal.common.VersioningBehavior;import io.temporal.common.WorkerDeploymentVersion;import io.temporal.worker.WorkerDeploymentOptions;WorkerOptions.newBuilder() .setDeploymentOptions( WorkerDeploymentOptions.newBuilder() .setVersion(new WorkerDeploymentVersion("llm_srv", "1.0")) .setUseVersioning(true) .setDefaultVersioningBehavior(VersioningBehavior.AUTO_UPGRADE) .build()) .build(); from temporalio.common import WorkerDeploymentVersion, VersioningBehaviorfrom temporalio.worker import Worker, WorkerDeploymentConfigWorker( client, task_queue="mytaskqueue", workflows=workflows, activities=activities, deployment_config=WorkerDeploymentConfig( version=WorkerDeploymentVersion( deployment_name="llm_srv", build_id=my_env.build_id), use_worker_versioning=True, default_versioning_behavior=VersioningBehavior.UNSPECIFIED ),) const myWorker = await Worker.create({ workflowsPath: require.resolve('./workflows'), taskQueue, workerDeploymentOptions: { useWorkerVersioning: true, version: { buildId: '1.0', deploymentName: 'llm_srv' }, }, connection: nativeConnection,}); **PHP** example coming soon. var myWorker = new TemporalWorker( Client, new TemporalWorkerOptions(taskQueue) {DeploymentOptions = new(new("llm_srv", "1.0"), true) { DefaultVersioningBehavior = VersioningBehavior.Unspecified }, }.AddWorkflow()); worker = Temporalio::Worker.new( client: client, task_queue: task_queue, workflows: [MyWorkflow], deployment_options: Temporalio::Worker::DeploymentOptions.new( version: Temporalio::WorkerDeploymentVersion.new( deployment_name: 'llm_srv', build_id: '1.0' ), use_worker_versioning: true, default_versioning_behavior: Temporalio::VersioningBehavior::UNSPECIFIED )) Choosing a Versioning Behavior[​](https://docs.temporal.io/production-deployment/worker-deployments/worker-versioning#choosing-behavior "Direct link to Choosing a Versioning Behavior") ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- The right versioning behavior depends on how long your Workflows run relative to your deployment frequency. ### Decision guide[​](https://docs.temporal.io/production-deployment/worker-deployments/worker-versioning#decision-guide "Direct link to Decision guide") | Workflow Duration | Uses Continue-as-New? | Recommended Behavior | Patching Required? | | --- | --- | --- | --- | | **Short** (completes before next deploy) | N/A | `PINNED` | Never | | **Medium** (spans multiple deploys) | No | `AUTO_UPGRADE` | Yes | | **Long** (weeks to years) | Yes | `PINNED` + [upgrade on CaN](https://docs.temporal.io/production-deployment/worker-deployments/worker-versioning#upgrade-on-continue-as-new) | Never | | **Long** (weeks to years) | No | `AUTO_UPGRADE` + patching | Yes | ### Examples by Workflow type[​](https://docs.temporal.io/production-deployment/worker-deployments/worker-versioning#behavior-examples "Direct link to Examples by Workflow type") | Workflow Type | Duration | Recommended Behavior | Notes | | --- | --- | --- | --- | | Order processing | Minutes | `PINNED` | Completes before next deploy | | Payment retry | Hours | `PINNED` or `AUTO_UPGRADE` | Depends on deploy frequency | | Subscription billing | Days | `AUTO_UPGRADE` | May span multiple deploys | | Customer entity | Months-Years | `PINNED` + upgrade on CaN | Uses Continue-as-New pattern | | AI agent / Chatbot | Weeks | `PINNED` + upgrade on CaN | Long sleeps, uses CaN | | Compliance audit | Months | `AUTO_UPGRADE` + patching | Cannot use CaN (needs full history) | Long-running Workflows with Continue-as-New If your Workflow uses Continue-as-New to manage history size, you can upgrade to new Worker Deployment Versions at the CaN boundary without patching. See [Upgrading on Continue-as-New](https://docs.temporal.io/production-deployment/worker-deployments/worker-versioning#upgrade-on-continue-as-new) below. ### Default Versioning Behavior Considerations[​](https://docs.temporal.io/production-deployment/worker-deployments/worker-versioning#default-versioning-behavior-considerations "Direct link to Default Versioning Behavior Considerations") If you are using blue-green deployments, you should default to Auto-Upgrade and should not use Workflow Pinning. Otherwise, if your Worker and Workflows are new, we suggest not providing a `DefaultVersioningBehavior`. In general, each Workflow Type should be annotated as Auto-Upgrade or Pinned. If all of your Workflows will be short-running for the foreseeable future, you can default to Pinned. Many users who are migrating to Worker Versioning will start by defaulting to Auto-Upgrade until they have had time to annotate their Workflows. This default is the most similar to the legacy behavior. Once each Workflow Type is annotated, you can remove the `DefaultVersioningBehavior`. There is a possibility of a queue blocking limitation for new or Auto-Upgrade Workflows if there is a ramp, but one of the Current or Ramping versions is down or doesn't have enough capacity. This leads to other versions not getting Tasks or slowing down. For example, you have a Current Version and a Ramping Version at 50%. If all of your Current Version Workers go down, you would expect at least 50% of new Workflows to go to the Ramping Version. This won't happen because the Tasks for the Current Version are blocking the queue. note Keep in mind that Child Workflows of a parent or previous Auto-Upgrade Workflow default to Auto-Upgrade behavior and not Unspecified. You also want to make sure you understand how your Activities are going to work across different Worker Deployment Versions. Refer to the [Worker Versioning Activity behavior docs](https://docs.temporal.io/production-deployment/worker-deployments/worker-versioning#choosing-behavior) for more details. Rolling out changes with the CLI[​](https://docs.temporal.io/production-deployment/worker-deployments/worker-versioning#rolling-out-changes-with-the-cli "Direct link to Rolling out changes with the CLI") ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ Next, deploy your Worker with the additional configuration parameters. Before making any Workflow revisions, you can use the `temporal` CLI to check which of your Worker versions are currently polling: You can view the Versions that are part of a Deployment with `temporal worker deployment describe`: temporal worker deployment describe --name="$MY_DEPLOYMENT" To activate a Deployment Version, use `temporal worker deployment set-current-version`, specifying the deployment name and a Build ID: temporal worker deployment set-current-version \ --deployment-name "YourDeploymentName" \ --build-id "YourBuildID" To ramp a Deployment Version up to some percentage of your overall Worker fleet, use `set-ramping version`, with the same parameters and a ramping percentage: temporal worker deployment set-ramping-version \ --deployment-name "YourDeploymentName" \ --build-id "YourBuildID" \ --percentage=5 You can verify that Workflows are cutting over to that version with `describe -w YourWorkflowID`: temporal workflow describe -w YourWorkflowID That returns the new Version that the workflow is running on: Versioning Info: Behavior AutoUpgrade Version llm_srv.2.0 OverrideBehavior Unspecified Marking a Workflow Type as Pinned[​](https://docs.temporal.io/production-deployment/worker-deployments/worker-versioning#marking-a-workflow-type-as-pinned "Direct link to Marking a Workflow Type as Pinned") --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- You can mark a Workflow Type as pinned when you register it by adding an additional Pinned parameter. This will cause it to remain on its original deployed version: * Go * Java * Python * TypeScript * PHP * .NET * Ruby // w is the Worker configured as in the previous examplew.RegisterWorkflowWithOptions(HelloWorld, workflow.RegisterOptions{ // or workflow.VersioningBehaviorAutoUpgrade VersioningBehavior: workflow.VersioningBehaviorPinned,}) @WorkflowInterfacepublic interface HelloWorld { @WorkflowMethod String hello();}public static class HelloWorldImpl implements HelloWorld { @Override @WorkflowVersioningBehavior(VersioningBehavior.PINNED) public String hello() { return "Hello, World!"; }} @workflow.defn(versioning_behavior=VersioningBehavior.PINNED)class HelloWorld: @workflow.run async def run(self): return "hello world!" setWorkflowOptions({ versioningBehavior: 'PINNED' }, helloWorld);export async function helloWorld(): Promise { return 'hello world!';} **PHP** example coming soon. [Workflow(VersioningBehavior = VersioningBehavior.Pinned)]public class HelloWorld{ [WorkflowRun] public async Task RunAsync() { return "hello world!"; }} class HelloWorld < Temporalio::Workflow::Definition workflow_versioning_behavior Temporalio::VersioningBehavior::PINNED def execute 'hello world!' endend Moving a pinned Workflow[​](https://docs.temporal.io/production-deployment/worker-deployments/worker-versioning#moving-a-pinned-workflow "Direct link to Moving a pinned Workflow") ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ Sometimes you'll need to manually move a set of pinned Workflows off of a version that has a bug to a version with the fix. If you need to move a pinned Workflow to a new version, use `temporal workflow update-options`: temporal workflow update-options \ --workflow-id "$WORKFLOW_ID" \ --versioning-override-behavior pinned \ --versioning-override-deployment-name "$TARGET_DEPLOYMENT" \ --versioning-override-build-id "$TARGET_BUILD_ID" You can move several Workflows at once matching a `--query` parameter: temporal workflow update-options \ --query="TemporalWorkerDeploymentVersion=$TARGET_DEPLOYMENT:$BAD_BUILD_ID" \ --versioning-override-behavior pinned \ --versioning-override-deployment-name "$TARGET_DEPLOYMENT" \ --versioning-override-build-id "$FIXED_BUILD_ID" In this scenario, you may also need to use the other [Versioning APIs](https://docs.temporal.io/workflow-definition#workflow-versioning) to patch your Workflow in the "fixed" build, so that your target Worker can handle the moved Workflows correctly. If you made a [version-incompatible change](https://docs.temporal.io/workflow-definition#deterministic-constraints) to your Workflow, and you want to roll back to an earlier version, it's not possible to patch it. Considering using [Workflow Reset](https://docs.temporal.io/workflow-execution/event#reset) along with your move. "Reset-with-Move" allows you to atomically Reset your Workflow and set a Versioning Override on the newly reset Workflow, so when it resumes execution, all new Workflow Tasks will be executed on your new Worker. temporal workflow reset with-workflow-update-options \ --workflow-id "$WORKFLOW_ID" \ --event-id "$EVENT_ID" \ --reason "$REASON" \ --versioning-override-behavior pinned \ --versioning-override-deployment-name "$TARGET_DEPLOYMENT" \ --versioning-override-build-id "$TARGET_BUILD_ID" Migrating a Workflow from Pinned to Auto-Upgrade[​](https://docs.temporal.io/production-deployment/worker-deployments/worker-versioning#migrating-a-workflow-from-pinned-to-auto-upgrade "Direct link to Migrating a Workflow from Pinned to Auto-Upgrade") ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ There may be times when you need to migrate your Workflow from Pinned to Auto-Upgrade because you configured your Workflow Type with the wrong behavior or you've pinned a really long-running Workflow by mistake. Pinned Workflows can block version drainage, especially when they run for a long time. You could move the Workflow to a new build, but that would just push the problem to the next build. In order to make this change, you need to change the versioning behavior for your Workflow from Pinned to Auto-Upgrade. You can use `temporal workflow update-options` for this: temporal workflow update-options \ --workflow-id "$WORKFLOW_ID" \ --versioning-override-behavior auto_upgrade If you want to move all your Workflows of a certain type to this new configuration, you can do it with this command: temporal workflow update-options \ --query="WorkflowType='$WORKFLOW_TYPE'" \ --versioning-override-behavior auto_upgrade You can also filter on a certain build ID to limit the number of Workflows you apply it to: temporal workflow update-options \ --query="WorkflowType='$WORKFLOW_TYPE' AND TemporalWorkerDeploymentVersion='$TARGET_DEPLOYMENT:$OLD_VERSION'" \ --versioning-override-behavior auto_upgrade note When you change the behavior to Auto-Upgrade, the Workflow will resume work on the Workflow's Target Version. So if the Workflow's Target Version is different from the earlier Pinned Version, you should make sure you [patch](https://docs.temporal.io/patching#patching) the Workflow code. Upgrading on Continue-as-New[​](https://docs.temporal.io/production-deployment/worker-deployments/worker-versioning#upgrade-on-continue-as-new "Direct link to Upgrading on Continue-as-New") ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Long-running Workflows that use [Continue-as-New](https://docs.temporal.io/workflow-execution/continue-as-new) can upgrade to newer Worker Deployment Versions at Continue-as-New boundaries without requiring patching. This pattern is ideal for: * **Entity Workflows** that run for months or years * **Batch processing** Workflows that checkpoint with Continue-as-New * **AI agent Workflows** with long sleeps waiting for user input Public Preview This feature is in Public Preview as an experimental SDK-level option. ### How it works[​](https://docs.temporal.io/production-deployment/worker-deployments/worker-versioning#upgrade-on-can-how-it-works "Direct link to How it works") By default, Pinned Workflows stay on their original Worker Deployment Version even when they Continue-as-New. With the upgrade option enabled: 1. Each Workflow run remains pinned to its version (no patching needed during a run) 2. The Temporal Server tells the workflow when a new [Target Version](https://docs.temporal.io/worker-versioning#versioning-definitions) becomes available 3. When the Workflow performs Continue-as-New with the upgrade option, the new run starts on the [Target Version](https://docs.temporal.io/worker-versioning#versioning-definitions) ### Checking for new versions[​](https://docs.temporal.io/production-deployment/worker-deployments/worker-versioning#checking-for-new-versions "Direct link to Checking for new versions") When a new Worker Deployment Version becomes Current or Ramping, active Workflows can detect this through `target_worker_deployment_version_changed`: func (w *Workflows) ContinueAsNewWithVersionUpgradeV1( ctx workflow.Context, attempt int,) (string, error) { if attempt > 0 { return "v1.0", nil } // Check GetTargetWorkerDeploymentVersionChanged periodically. // GetTargetWorkerDeploymentVersionChanged is refreshed after each WFT completes. for { // Trigger a WFT when timer expires, thereby refreshing the GetTargetWorkerDeploymentVersionChanged flag. // Since this is just a test workflow, we aren't doing any real work. In a real workflow regularly // doing non-sleep workflow tasks, you would not need to artificially trigger a WFT to refresh the // GetTargetWorkerDeploymentVersionChanged flag. You could choose to check the field periodically, or you // might want to check before accepting updates, starting activities, or starting child workflows. err := workflow.Sleep(ctx, 10*time.Millisecond) if err != nil { return "", err } info := workflow.GetInfo(ctx) if info.GetTargetWorkerDeploymentVersionChanged() { return "", workflow.NewContinueAsNewErrorWithOptions( ctx, workflow.ContinueAsNewErrorOptions{ // Pass InitialVersioningBehavior=workflow.ContinueAsNewVersioningBehaviorAutoUpgrade // to make the new run start with AutoUpgrade behavior and use the Target Version of // its Worker Deployment. InitialVersioningBehavior: workflow.ContinueAsNewVersioningBehaviorAutoUpgrade, }, "ContinueAsNewWithVersionUpgrade", attempt+1, ) } }}func (w *Workflows) ContinueAsNewWithVersionUpgradeV2( ctx workflow.Context, attempt int,) (string, error) { return "v2.0", nil} ### Limitations[​](https://docs.temporal.io/production-deployment/worker-deployments/worker-versioning#upgrade-on-can-limitations "Direct link to Limitations") Current Limitations * **Lazy moving only:** Workflows must be invoked by executing a step to receive the Target-Version-Changed information. Sleeping Workflows won't be proactively get the Target-Version-Changed information. If you have idle workflows that you want to wake up so that they can check `GetTargetWorkerDeploymentVersionChanged`, you can send them a Signal. * **Interface compatibility:** When continuing as new to a different version, ensure your Workflow input provided by the previous version's workflow definition is compatible with the new version's workflow definition. If incompatible, the new run may fail on its first Workflow Task. Sunsetting an old Deployment Version[​](https://docs.temporal.io/production-deployment/worker-deployments/worker-versioning#sunsetting-an-old-deployment-version "Direct link to Sunsetting an old Deployment Version") ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ A Worker Deployment Version moves through the following states: 1. **Inactive**: The version exists because a Worker with that version has polled the server. If this version never becomes Active, it will never be Draining or Drained. 2. **Active**: The version is either Current or Ramping, so it is accepting new Workflows and existing Auto-Upgrade Workflows. 3. **Draining**: The version stopped being Current or Ramping, and it has open pinned Workflows running on it. It is possible to be Draining and have no open pinned Workflows for a short time, since the drainage status is updated periodically. 4. **Drained**: The version was draining and now all the pinned Workflows that were running on it are closed. You can see these statuses when you describe a Worker Deployment in the `WorkerDeploymentVersionStatus` of each `VersionSummary`, or by describing the version directly. When a version is Draining or Drained, that is displayed in a value called `DrainageStatus`. Periodically, the Temporal Service will refresh this status by counting any open pinned Workflows using that version. On each refresh, `DrainageInfo.last_checked_time` is updated. Eventually, `DrainageInfo` will report that the version is fully drained. At this point, no Workflows are still running on that version and no more will be automatically routed to it, so you can consider shutting down the running Workers. You can monitor this by checking `WorkerDeploymentInfo.VersionSummaries` or with `temporal worker deployment describe-version`: temporal worker deployment describe-version \ --deployment-name "YourDeploymentName" \ --build-id "YourBuildID" Worker Deployment Version: Version llm_srv.1.0 CreateTime 5 hours ago RoutingChangedTime 32 seconds ago RampPercentage 0 DrainageStatus draining DrainageLastChangedTime 31 seconds ago DrainageLastCheckedTime 31 seconds agoTask Queues: Name Type hello-world activity hello-world workflow If you have implemented [Queries](https://docs.temporal.io/sending-messages#sending-queries) on closed pinned Workflows, you may need to keep some Workers running to handle them. ### Adding a pre-deployment test[​](https://docs.temporal.io/production-deployment/worker-deployments/worker-versioning#adding-a-pre-deployment-test "Direct link to Adding a pre-deployment test") Before deploying a new Workflow revision, you can test it with synthetic traffic. To do this, use pinning in your tests, following the examples below * Go * Java * Python * TypeScript * PHP * .NET * Ruby workflowOptions := client.StartWorkflowOptions{ ID: "MyWorkflowId", TaskQueue: "MyTaskQueue", VersioningOverride: &client.PinnedVersioningOverride{ Version: worker.WorkerDeploymentVersion{ DeploymentName: "DeployName", BuildId: "1.0", }, },}// c is an initialized Clientwe, err := c.ExecuteWorkflow(context.Background(), workflowOptions, HelloWorld, "Hello") MyWorkflow handle = client.newWorkflowStub( MyWorkflow.class, WorkflowOptions.newBuilder() .setWorkflowId("MyWorkflowId") .setTaskQueue("MyTaskQueue") .setVersioningOverride(new VersioningOverride.PinnedVersioningOverride( new WorkerDeploymentVersion("DeployName", "1.0"))) .build());WorkflowExecution we = WorkflowClient.start(handle::execute, "Hello"); handle = client.start_workflow( MyWorkflow.run, "Hello", id="MyWorkflowId", task_queue="MyTaskQueue", versioning_override=PinnedVersioningOverride( WorkerDeploymentVersion("DeployName", "1.0") ),) const handle = await client.workflow.start('helloWorld', { taskQueue: 'MyTaskQueue', workflowId: 'MyWorkflowId', versioningOverride: { pinnedTo: { buildId: '1.0', deploymentName: 'deploy-name' }, },}); **PHP** example coming soon. var workerV1 = new WorkerDeploymentVersion("deploy-name", "1.0");var handle = await Client.StartWorkflowAsync( (HelloWorld wf) => wf.RunAsync(), new(id: "MyWorkflowId", taskQueue: "MyTaskQueue") { VersioningOverride = new VersioningOverride.Pinned(workerV1), }); worker_v1 = Temporalio::WorkerDeploymentVersion.new( deployment_name: 'deploy-name', build_id: '1.0')handle = env.client.start_workflow( HelloWorld, id: 'MyWorkflowId', task_queue: 'MyTaskQueue', versioning_override: Temporalio::VersioningOverride.pinned(worker_v1)) Garbage collection[​](https://docs.temporal.io/production-deployment/worker-deployments/worker-versioning#garbage-collection "Direct link to Garbage collection") ------------------------------------------------------------------------------------------------------------------------------------------------------------------ Worker Deployments are never garbage collected, but _Worker Deployment Versions_ (often referred to as Versions, Worker Versions, Deployment Versions) are. Versions are deleted to keep the total number of versions in one Worker Deployment less than or equal to [`matching.maxVersionsInDeployment`](https://github.com/temporalio/temporal/blob/a3a53266c002ae33b630a41977274f8b5b587031/common/dynamicconfig/constants.go#L1317-L1321) , which is currently set to 100 in Temporal Cloud, but that's a conservative number and it could be increased if needed. For example, when you deploy your 101st Worker Version in a Worker Deployment, the server looks at the oldest drained version in the Worker deployment. If it has had no pollers in the last 5 minutes, the server deletes it. If that version still has pollers, the server will try the next oldest version. If none of the 100 versions are eligible for deletion (ie. none of them are drained with no pollers), then no version will be deleted and the poll from the 101st version would fail. At that point, to successfully deploy your 101st version, you would need to increase `matching.maxVersionsInDeployment` or stop polling from one of the old drained versions to make it eligible for clean up. If you want to re-deploy a previously deleted version, start polling with a Worker that has the same build ID and Deployment Name as the deleted version and the server will recreate it. This covers the complete lifecycle of working with Worker Versioning. We are continuing to improve this feature, and we welcome any feedback or feature requests using the sidebar link! * [Getting Started with Worker Versioning](https://docs.temporal.io/production-deployment/worker-deployments/worker-versioning#definition) * [Setting up your deployment system](https://docs.temporal.io/production-deployment/worker-deployments/worker-versioning#deployment-systems) * [Configuring a Worker for Versioning](https://docs.temporal.io/production-deployment/worker-deployments/worker-versioning#configuring-a-worker-for-versioning) * [Choosing a Versioning Behavior](https://docs.temporal.io/production-deployment/worker-deployments/worker-versioning#choosing-behavior) * [Decision guide](https://docs.temporal.io/production-deployment/worker-deployments/worker-versioning#decision-guide) * [Examples by Workflow type](https://docs.temporal.io/production-deployment/worker-deployments/worker-versioning#behavior-examples) * [Default Versioning Behavior Considerations](https://docs.temporal.io/production-deployment/worker-deployments/worker-versioning#default-versioning-behavior-considerations) * [Rolling out changes with the CLI](https://docs.temporal.io/production-deployment/worker-deployments/worker-versioning#rolling-out-changes-with-the-cli) * [Marking a Workflow Type as Pinned](https://docs.temporal.io/production-deployment/worker-deployments/worker-versioning#marking-a-workflow-type-as-pinned) * [Moving a pinned Workflow](https://docs.temporal.io/production-deployment/worker-deployments/worker-versioning#moving-a-pinned-workflow) * [Migrating a Workflow from Pinned to Auto-Upgrade](https://docs.temporal.io/production-deployment/worker-deployments/worker-versioning#migrating-a-workflow-from-pinned-to-auto-upgrade) * [Upgrading on Continue-as-New](https://docs.temporal.io/production-deployment/worker-deployments/worker-versioning#upgrade-on-continue-as-new) * [How it works](https://docs.temporal.io/production-deployment/worker-deployments/worker-versioning#upgrade-on-can-how-it-works) * [Checking for new versions](https://docs.temporal.io/production-deployment/worker-deployments/worker-versioning#checking-for-new-versions) * [Limitations](https://docs.temporal.io/production-deployment/worker-deployments/worker-versioning#upgrade-on-can-limitations) * [Sunsetting an old Deployment Version](https://docs.temporal.io/production-deployment/worker-deployments/worker-versioning#sunsetting-an-old-deployment-version) * [Adding a pre-deployment test](https://docs.temporal.io/production-deployment/worker-deployments/worker-versioning#adding-a-pre-deployment-test) * [Garbage collection](https://docs.temporal.io/production-deployment/worker-deployments/worker-versioning#garbage-collection) --- # Temporal Worker deployments | Temporal Platform Documentation [Skip to main content](https://docs.temporal.io/production-deployment/worker-deployments#__docusaurus_skipToContent_fallback) A core feature of Temporal is that you are able to deploy your Workers to any infrastructure where your Workflow and Activity code will actually run. This way, you have total control over your runtime environment, and can be responsive to any security or scaling needs that may arise over time, whether you are using Temporal Cloud or self-hosting a Temporal Service. If you are just getting started, you want more guidance, or a refresher on Temporal concepts, our [Tutorials and Courses](https://learn.temporal.io/) help by using only one or two Temporal Workers to demonstrate core functionality. Once you have an understanding of the core concepts, the content in this section will provide clarity on real-world deployments that grow far beyond those examples. Our Worker Deployments guide provides documentation of Temporal product features that make it easier to scale and revise your Workflows. [Worker Versioning](https://docs.temporal.io/production-deployment/worker-deployments/worker-versioning) is the recommended default for safely deploying new Workflow code. It allows you to pin Workflows to individual versions of your workers, which are called Worker Deployment Versions. If your environment cannot yet support versioned worker deployments, you can fall back to patching Workflow code. However, new production deployments should prefer Worker Versioning whenever possible. You can optionally use the Temporal [Worker Controller](https://docs.temporal.io/production-deployment/worker-deployments/kubernetes-controller) to programmatically manage and scale your Worker deployments in Kubernetes pods. This section also covers specific Worker Deployment examples: * [**Deploy Workers to Amazon EKS**](https://docs.temporal.io/production-deployment/worker-deployments/deploy-workers-to-aws-eks) Containerize your Worker, publish it to Amazon Elastic Container Registry (ECR), and deploy it to Amazon Elastic Kubernetes Service (EKS) using the Temporal Python SDK. This guide covers the full deployment lifecycle and shows how to configure your Worker to connect to Temporal Cloud using Kubernetes-native tools like ConfigMaps and Secrets. Running Workers on EKS gives you fine-grained control over scaling, resource allocation, and availability—ideal for production systems that need reliability and flexibility in the cloud. --- # Troubleshoot the deadline-exceeded error | Temporal Platform Documentation [Skip to main content](https://docs.temporal.io/troubleshooting/deadline-exceeded-error#__docusaurus_skipToContent_fallback) On this page All requests made to the [Temporal Service](https://docs.temporal.io/temporal-service) by the Client or Worker are [gRPC requests](https://grpc.io/docs/what-is-grpc/core-concepts/#deadlines) . Sometimes, when these frontend requests can't be completed, you'll see this particular error message: `Context: deadline exceeded`. Network interruptions, timeouts, server overload, and Query errors are some of the causes of this error. The following sections discuss the nature of this error and how to troubleshoot it. ### Check system clocks[​](https://docs.temporal.io/troubleshooting/deadline-exceeded-error#check-system-clocks "Direct link to Check system clocks") Timing skew can cause the system clock on a Worker to drift behind the system clock of the Temporal Service. If the difference between the two clocks exceeds an Activity's Start-To-Close Timeout, an `Activity complete after timeout` error occurs. If you receive an `Activity complete after timeout` error alongside `Context: deadline exceeded`, check the clocks on the Temporal Service's system and the system of the Worker sending that error. If the Worker's clock doesn't match the Temporal Service, synchronize all clocks to an NTP server. ### Check Frontend Service logs[​](https://docs.temporal.io/troubleshooting/deadline-exceeded-error#check-frontend-service-logs "Direct link to Check Frontend Service logs") note Cloud users cannot access some of the logs needed to diagnose the source of the error. If you're using Temporal Cloud, create a [support ticket](https://docs.temporal.io/cloud/support#support-ticket) with as much information as possible, including the Namespace Name and the Workflow Ids of some Workflow Executions in which the issue occurs. [Frontend Service](https://docs.temporal.io/temporal-service/temporal-server#frontend-service) logs can show which parts of the Temporal Service aren't working. For the error to appear, a service pod or container must be up and running. OSS users can verify that the Frontend Service is connected and running by using the Temporal CLI. temporal operator cluster health --address 127.0.0.1:7233 Use [`grpc-health-probe`](https://github.com/grpc-ecosystem/grpc-health-probe) to check the Frontend Service, [Matching Service](https://docs.temporal.io/temporal-service/temporal-server#matching-service) , and [History Service](https://docs.temporal.io/temporal-service/temporal-server#history-service) . ./grpc-health-probe -addr=frontendAddress:frontendPort -service=temporal.api.workflowservice.v1.WorkflowService./grpc-health-probe -addr=matchingAddress:matchingPort -service=temporal.api.workflowservice.v1.MatchingService./grpc-health-probe -addr=historyAddress:historyPort -service=temporal.api.workflowservice.v1.HistoryService Logs can also be used to find failed Client [Query](https://docs.temporal.io/sending-messages#sending-queries) requests. ### Check your Temporal Service metrics[​](https://docs.temporal.io/troubleshooting/deadline-exceeded-error#check-your-temporal-service-metrics "Direct link to Check your Temporal Service metrics") Temporal Service metrics can be used to detect issues (such as `resource exhausted`) that impact Temporal Service health. A `resource exhausted` error can cause your client request to fail, which prompts the `deadline exceeded` error. Use the following query to check for errors in `RpsLimit`, `ConcurrentLimit` and `SystemOverloaded` on your metrics dashboard. sum(rate(service_errors_resource_exhausted{}[1m])) by (resource_exhausted_cause) Look for high latencies, short timeouts, and other abnormal [Temporal Service metrics](https://docs.temporal.io/references/cluster-metrics) . If the metrics come from a specific service (such as History Service), check the service's health and performance. ### Check Workflow logic[​](https://docs.temporal.io/troubleshooting/deadline-exceeded-error#check-workflow-logic "Direct link to Check Workflow logic") Check your [Client and Worker configuration](https://docs.temporal.io/references/configuration) files for missing or invalid target values, such as the following: * Server names * Network or host addresses * Certificates Invalid targets also cause `connection refused` errors alongside `deadline exceeded`. Check that the Client connects after updating your files. ### Advanced troubleshooting[​](https://docs.temporal.io/troubleshooting/deadline-exceeded-error#advanced-troubleshooting "Direct link to Advanced troubleshooting") In addition to the steps listed in the previous sections, check the areas mentioned in each of the following scenarios. ### After enabling mTLS[​](https://docs.temporal.io/troubleshooting/deadline-exceeded-error#after-enabling-mtls "Direct link to After enabling mTLS") Check the health of the Temporal Service with `temporal operator cluster health`. temporal operator cluster health --address [SERVER_ADDRESS] Add any missing [environment variables](https://docs.temporal.io/references/web-ui-environment-variables) to the configuration files, and correct any incorrect values. Server names and certificates must match between Frontend and internode. ### After restarting the Temporal Service[​](https://docs.temporal.io/troubleshooting/deadline-exceeded-error#after-restarting-the-temporal-service "Direct link to After restarting the Temporal Service") You might not be giving the Temporal Service enough time to respond and reconnect. Restart the Server, wait, and then check all services for connectivity and further errors. If the error persists, review your Workflow Execution History and server logs for more specific causes before continuing to troubleshoot. ### When executing or scheduling Workflows[​](https://docs.temporal.io/troubleshooting/deadline-exceeded-error#when-executing-or-scheduling-workflows "Direct link to When executing or scheduling Workflows") One or more services might be unable to connect to the [Frontend Service](https://docs.temporal.io/temporal-service/temporal-server#frontend-service) . The Workflow might be unable to complete requests within the given connection time. Increase the value of `frontend.keepAliveMaxConnectionAge` so that requests can be finished before the connection terminates. note If you increase `frontend.keepAliveMaxConnectionAge` values, consider monitoring your server performance for load. * * * Still unable to resolve your issue? * If you use Temporal Cloud, create a [support ticket](https://docs.temporal.io/cloud/support#support-ticket) . * If you use our open source software or Temporal Cloud, check for similar questions and possible solutions in our [community forum](https://community.temporal.io/) or [community Slack](https://temporal.io/slack) . * [Check system clocks](https://docs.temporal.io/troubleshooting/deadline-exceeded-error#check-system-clocks) * [Check Frontend Service logs](https://docs.temporal.io/troubleshooting/deadline-exceeded-error#check-frontend-service-logs) * [Check your Temporal Service metrics](https://docs.temporal.io/troubleshooting/deadline-exceeded-error#check-your-temporal-service-metrics) * [Check Workflow logic](https://docs.temporal.io/troubleshooting/deadline-exceeded-error#check-workflow-logic) * [Advanced troubleshooting](https://docs.temporal.io/troubleshooting/deadline-exceeded-error#advanced-troubleshooting) * [After enabling mTLS](https://docs.temporal.io/troubleshooting/deadline-exceeded-error#after-enabling-mtls) * [After restarting the Temporal Service](https://docs.temporal.io/troubleshooting/deadline-exceeded-error#after-restarting-the-temporal-service) * [When executing or scheduling Workflows](https://docs.temporal.io/troubleshooting/deadline-exceeded-error#when-executing-or-scheduling-workflows) --- # Temporal Server options reference | Temporal Platform Documentation [Skip to main content](https://docs.temporal.io/references/server-options#__docusaurus_skipToContent_fallback) On this page You can run the [Temporal Server](https://docs.temporal.io/temporal-service/temporal-server) as a Go application by including the server package `go.temporal.io/server/temporal` and using it to create and start a Temporal Server. The Temporal Server services can be run in various ways. We recommend this approach for a limited number of situations. s, err := temporal.NewServer()if err != nil { log.Fatal(err)}err = s.Start()if err != nil{ log.Fatal(err)} `NewServer()` accepts functions as parameters. Each function returns a `ServerOption` that is applied to the instance. Source code for parameter reference is here: [https://github.com/temporalio/temporal/blob/main/temporal/server\_option.go](https://github.com/temporalio/temporal/blob/main/temporal/server_option.go) ### WithConfig[​](https://docs.temporal.io/references/server-options#withconfig "Direct link to WithConfig") To launch a Temporal server, a configuration file is required. The server automatically searches for this configuration in the default location ./config/development.yaml when starting. If you need to use a custom configuration, you can specify it through the server's configuration option. For comprehensive details about configuration parameters and structure, refer to the [official configuration documentation](https://pkg.go.dev/go.temporal.io/server/common/config) . s, err := temporal.NewServer( temporal.WithConfig(cfg),) ### WithConfigLoader[​](https://docs.temporal.io/references/server-options#withconfigloader "Direct link to WithConfigLoader") Load a custom configuration from a file. s, err := temporal.NewServer( temporal.WithConfigLoader(configDir, env, zone),) ### ForServices[​](https://docs.temporal.io/references/server-options#forservices "Direct link to ForServices") Sets the list of all valid temporal services. The default can be used from the `go.temporal.io/server/temporal` package. s, err := temporal.NewServer( temporal.ForServices(temporal.Services),) ### InterruptOn[​](https://docs.temporal.io/references/server-options#interrupton "Direct link to InterruptOn") This option provides a channel that interrupts the server on the signal from that channel. * If `temporal.InterruptOn()` is not passed, `server.Start()` is never blocked and you need to call `server.Stop()` somewhere. * If `temporal.InterruptOn(nil)` is passed, `server.Start()` blocks forever until the process is killed. * If `temporal.InterruptOn(temporal.InterruptCh())` is passed, `server.Start()` blocks until you use Ctrl+C, which then gracefully shuts the server down. * If `temporal.Interrupt(someCustomChan)` is passed, `server.Start()` blocks until a signal is sent to `someCustomChan`. s, err := temporal.NewServer( temporal.InterruptOn(temporal.InterruptCh()),) ### WithAuthorizer[​](https://docs.temporal.io/references/server-options#withauthorizer "Direct link to WithAuthorizer") Sets a low level [authorization mechanism](https://docs.temporal.io/self-hosted-guide/security#authorizer-plugin) that determines whether to allow or deny inbound API calls. s, err := temporal.NewServer( temporal.WithAuthorizer(myAuthorizer),) ### WithTLSConfigFactory[​](https://docs.temporal.io/references/server-options#withtlsconfigfactory "Direct link to WithTLSConfigFactory") Overrides the default TLS configuration provider. `TLSConfigProvider` is defined in the `go.temporal.io/server/common/rpc/encryption` package. s, err := temporal.NewServer( temporal.WithTLSConfigFactory(yourTLSConfigProvider),) ### WithClaimMapper[​](https://docs.temporal.io/references/server-options#withclaimmapper "Direct link to WithClaimMapper") Configures a [mechanism to map roles](https://docs.temporal.io/self-hosted-guide/security#claim-mapper) to `Claims` for authorization. s, err := temporal.NewServer( temporal.WithClaimMapper(func(cfg *config.Config) authorization.ClaimMapper { logger := getYourLogger() // Replace with how you retrieve or initialize your logger return authorization.NewDefaultJWTClaimMapper( authorization.NewDefaultTokenKeyProvider(cfg, logger), cfg ) }),) ### WithCustomMetricsReporter[​](https://docs.temporal.io/references/server-options#withcustommetricsreporter "Direct link to WithCustomMetricsReporter") Sets a custom tally metric reporter. s, err := temporal.NewServer( temporal.WithCustomMetricsReporter(myReporter),) You can see the [Uber tally docs on custom reporter](https://github.com/uber-go/tally#report-your-metrics) and see a community implementation of [a reporter for Datadog's `dogstatsd` format](https://github.com/temporalio/temporal/pull/998#issuecomment-857884983) . * [WithConfig](https://docs.temporal.io/references/server-options#withconfig) * [WithConfigLoader](https://docs.temporal.io/references/server-options#withconfigloader) * [ForServices](https://docs.temporal.io/references/server-options#forservices) * [InterruptOn](https://docs.temporal.io/references/server-options#interrupton) * [WithAuthorizer](https://docs.temporal.io/references/server-options#withauthorizer) * [WithTLSConfigFactory](https://docs.temporal.io/references/server-options#withtlsconfigfactory) * [WithClaimMapper](https://docs.temporal.io/references/server-options#withclaimmapper) * [WithCustomMetricsReporter](https://docs.temporal.io/references/server-options#withcustommetricsreporter) --- # Evaluate Temporal | Temporal Platform Documentation [Skip to main content](https://docs.temporal.io/evaluate#__docusaurus_skipToContent_fallback) Temporal is designed to make developing distributed applications a delightful experience. Developers benefit from a clear approach to structure their code and visibility into the state of their application. Applications benefit from fault-tolerance and execution guarantees. Thousands of companies of all sizes are leveraging Temporal's capabilities for both mission critical and standard workloads. * [Why Temporal](https://docs.temporal.io/evaluate/why-temporal) * [Development and production features](https://docs.temporal.io/evaluate/development-production-features) * [Use cases](https://docs.temporal.io/evaluate/use-cases-design-patterns) * [Temporal Cloud](https://docs.temporal.io/cloud) * [Security](https://docs.temporal.io/security) --- # Temporal Platform references | Temporal Platform Documentation [Skip to main content](https://docs.temporal.io/references#__docusaurus_skipToContent_fallback) * [SDK metrics reference](https://docs.temporal.io/references/sdk-metrics) * [Commands reference](https://docs.temporal.io/references/commands) * [Events reference](https://docs.temporal.io/references/events) * [Web UI environment variables reference](https://docs.temporal.io/references/web-ui-environment-variables) * [Temporal Service configuration reference](https://docs.temporal.io/references/configuration) * [Temporal Web UI configuration reference](https://docs.temporal.io/references/web-ui-configuration) * [Temporal Cloud Operation reference](https://docs.temporal.io/references/operation-list) * [Go SDK API reference](https://pkg.go.dev/go.temporal.io/sdk) * [Java SDK API reference](https://www.javadoc.io/doc/io.temporal/temporal-sdk/latest/index.html) * [Python SDK API reference](https://python.temporal.io/) * [TypeScript SDK API reference](https://typescript.temporal.io/) * [.NET SDK API reference](https://dotnet.temporal.io/api/) * [PHP SDK API reference](https://php.temporal.io/namespaces/temporal.html) * [Glossary](https://docs.temporal.io/glossary) --- # Temporal Web UI configuration reference | Temporal Platform Documentation [Skip to main content](https://docs.temporal.io/references/web-ui-configuration#__docusaurus_skipToContent_fallback) On this page The Temporal Web UI Server uses a configuration file for many of the UI's settings. An example development.yaml file can be found in the [temporalio/ui-server repo](https://github.com/temporalio/ui-server/blob/main/config/development.yaml) . Multiple configuration files can be created for configuring specific areas of the UI, such as Auth or TLS. auth[​](https://docs.temporal.io/references/web-ui-configuration#auth "Direct link to auth") --------------------------------------------------------------------------------------------- Configures authorization for the Temporal Server. Settings apply when Auth is enabled. auth: enabled: true providers: label: sso # for internal use; in future may expose as button text type: oidc providerUrl: https://accounts.google.com issuerUrl: clientId: xxxxx-xxxx.apps.googleusercontent.com clientSecret: xxxxxxxxxxxxxxxxxxxx callbackUrl: https://xxxx.com:8080/auth/sso/callback scopes: - openid - profile - email batchActionsDisabled[​](https://docs.temporal.io/references/web-ui-configuration#batchactionsdisabled "Direct link to batchActionsDisabled") --------------------------------------------------------------------------------------------------------------------------------------------- Prevents the execution of Batch actions. batchActionsDisabled: false cloudUi[​](https://docs.temporal.io/references/web-ui-configuration#cloudui "Direct link to cloudUi") ------------------------------------------------------------------------------------------------------ Enables the Cloud UI. cloudUi: false codec[​](https://docs.temporal.io/references/web-ui-configuration#codec "Direct link to codec") ------------------------------------------------------------------------------------------------ Codec Server configuration. codec: endpoint: http://your-codec-server-endpoint passAccessToken: false includeCredentials: false decodeEventHistoryDownload: false cors[​](https://docs.temporal.io/references/web-ui-configuration#cors "Direct link to cors") --------------------------------------------------------------------------------------------- The name of the `cors` field stands for Cross-Origin Resource Sharing. Use this field to provide a list of domains that are authorized to access the UI Server APIs. cors: cookieInsecure: false allowOrigins: - http://localhost:3000 # used at development by https://github.com/temporalio/ui defaultNamespace[​](https://docs.temporal.io/references/web-ui-configuration#defaultnamespace "Direct link to defaultNamespace") --------------------------------------------------------------------------------------------------------------------------------- The default Namespace that the UI loads data for. Defaults to `default`. defaultNamespace: default disableWriteActions[​](https://docs.temporal.io/references/web-ui-configuration#disablewriteactions "Direct link to disableWriteActions") ------------------------------------------------------------------------------------------------------------------------------------------ Prevents the user from executing Workflow Actions on the Web UI. This option affects Bulk Actions for Recent Workflows as well as Workflow Actions on the Workflow Details page. disableWriteActions: false note `disableWriteActions` overrides the configuration values of each individual Workflow Action. Setting this variable to `true` disables all Workflow Actions on the Web UI. enableUi[​](https://docs.temporal.io/references/web-ui-configuration#enableui "Direct link to enableUi") --------------------------------------------------------------------------------------------------------- Enables the browser UI. This configuration can be set dynamically with the [TEMPORAL\_UI\_ENABLED](https://docs.temporal.io/references/web-ui-environment-variables#temporal_ui_enabled) environment variable. If disabled—that is, set to `false`—the UI server APIs remain available. enableUi: true feedbackUrl[​](https://docs.temporal.io/references/web-ui-configuration#feedbackurl "Direct link to feedbackUrl") ------------------------------------------------------------------------------------------------------------------ The URL to direct users to when they click on the Feedback button in the UI. If not specified, it defaults to the UI's GitHub Issue page. feedbackUrl: https://github.com/temporalio/ui/issues/new/choose forwardHeaders[​](https://docs.temporal.io/references/web-ui-configuration#forwardheaders "Direct link to forwardHeaders") --------------------------------------------------------------------------------------------------------------------------- Configures headers for forwarding. forwardHeaders: - hideLogs[​](https://docs.temporal.io/references/web-ui-configuration#hidelogs "Direct link to hideLogs") --------------------------------------------------------------------------------------------------------- If enabled, disables any server logs from being printed to the console. hideLogs: true hideWorkflowQueryErrors[​](https://docs.temporal.io/references/web-ui-configuration#hideworkflowqueryerrors "Direct link to hideWorkflowQueryErrors") ------------------------------------------------------------------------------------------------------------------------------------------------------ Hides any errors resulting from a Query to the Workflow. hideWorkflowQueryErrors: false notifyOnNewVersion[​](https://docs.temporal.io/references/web-ui-configuration#notifyonnewversion "Direct link to notifyOnNewVersion") --------------------------------------------------------------------------------------------------------------------------------------- When enabled—that is, when set to `true`—a notification appears in the UI when a newer version of the [Temporal Server](https://docs.temporal.io/temporal-service/temporal-server) is available. notifyOnNewVersion: true port[​](https://docs.temporal.io/references/web-ui-configuration#port "Direct link to port") --------------------------------------------------------------------------------------------- The port used by the Temporal Web UI Server and any APIs. port: 8080 publicPath[​](https://docs.temporal.io/references/web-ui-configuration#publicpath "Direct link to publicPath") --------------------------------------------------------------------------------------------------------------- The path used by the Temporal Web UI Server and any APIs. publicPath: '' refreshInterval[​](https://docs.temporal.io/references/web-ui-configuration#refreshinterval "Direct link to refreshInterval") ------------------------------------------------------------------------------------------------------------------------------ How often the configuration UI Server reads the configuration file for new values. Currently, only [tls](https://docs.temporal.io/references/web-ui-configuration#tls) configuration values are propagated during a refresh. refreshInterval: 1m showTemporalSystemNamespace[​](https://docs.temporal.io/references/web-ui-configuration#showtemporalsystemnamespace "Direct link to showTemporalSystemNamespace") ------------------------------------------------------------------------------------------------------------------------------------------------------------------ When enabled—that is, when set to `true`—the Temporal System Namespace becomes visible in the UI. The Temporal System Namespace lists Workflow Executions used by the Temporal Platform. showTemporalSystemNamespace: false temporalGrpcAddress[​](https://docs.temporal.io/references/web-ui-configuration#temporalgrpcaddress "Direct link to temporalGrpcAddress") ------------------------------------------------------------------------------------------------------------------------------------------ The frontend address for the Temporal Cluster. The default address is localhost (127.0.0.1:7233). temporalGrpcAddress: default tls[​](https://docs.temporal.io/references/web-ui-configuration#tls "Direct link to tls") ------------------------------------------------------------------------------------------ Transport Layer Security (TLS) configuration for the Temporal Server. Settings apply when TLS is enabled. tls: caFile: ../ca.cert certFile: ../cluster.pem keyFile: ../cluster.key caData: certData: keyData: enableHostVerification: true serverName: tls-server workflowCancelDisabled[​](https://docs.temporal.io/references/web-ui-configuration#workflowcanceldisabled "Direct link to workflowCancelDisabled") --------------------------------------------------------------------------------------------------------------------------------------------------- Prevents the user from canceling Workflow Executions from the Web UI. workflowCancelDisabled: false workflowResetDisabled[​](https://docs.temporal.io/references/web-ui-configuration#workflowresetdisabled "Direct link to workflowResetDisabled") ------------------------------------------------------------------------------------------------------------------------------------------------ Prevents the user from resetting Workflows from the Web UI. workflowResetDisabled: false workflowSignalDisabled[​](https://docs.temporal.io/references/web-ui-configuration#workflowsignaldisabled "Direct link to workflowSignalDisabled") --------------------------------------------------------------------------------------------------------------------------------------------------- Prevents the user from signaling Workflow Executions from the Web UI. workflowSignalDisabled: false workflowTerminateDisabled[​](https://docs.temporal.io/references/web-ui-configuration#workflowterminatedisabled "Direct link to workflowTerminateDisabled") ------------------------------------------------------------------------------------------------------------------------------------------------------------ Prevents the user from terminating Workflow Executions from the Web UI. workflowTerminateDisabled: false * [auth](https://docs.temporal.io/references/web-ui-configuration#auth) * [batchActionsDisabled](https://docs.temporal.io/references/web-ui-configuration#batchactionsdisabled) * [cloudUi](https://docs.temporal.io/references/web-ui-configuration#cloudui) * [codec](https://docs.temporal.io/references/web-ui-configuration#codec) * [cors](https://docs.temporal.io/references/web-ui-configuration#cors) * [defaultNamespace](https://docs.temporal.io/references/web-ui-configuration#defaultnamespace) * [disableWriteActions](https://docs.temporal.io/references/web-ui-configuration#disablewriteactions) * [enableUi](https://docs.temporal.io/references/web-ui-configuration#enableui) * [feedbackUrl](https://docs.temporal.io/references/web-ui-configuration#feedbackurl) * [forwardHeaders](https://docs.temporal.io/references/web-ui-configuration#forwardheaders) * [hideLogs](https://docs.temporal.io/references/web-ui-configuration#hidelogs) * [hideWorkflowQueryErrors](https://docs.temporal.io/references/web-ui-configuration#hideworkflowqueryerrors) * [notifyOnNewVersion](https://docs.temporal.io/references/web-ui-configuration#notifyonnewversion) * [port](https://docs.temporal.io/references/web-ui-configuration#port) * [publicPath](https://docs.temporal.io/references/web-ui-configuration#publicpath) * [refreshInterval](https://docs.temporal.io/references/web-ui-configuration#refreshinterval) * [showTemporalSystemNamespace](https://docs.temporal.io/references/web-ui-configuration#showtemporalsystemnamespace) * [temporalGrpcAddress](https://docs.temporal.io/references/web-ui-configuration#temporalgrpcaddress) * [tls](https://docs.temporal.io/references/web-ui-configuration#tls) * [workflowCancelDisabled](https://docs.temporal.io/references/web-ui-configuration#workflowcanceldisabled) * [workflowResetDisabled](https://docs.temporal.io/references/web-ui-configuration#workflowresetdisabled) * [workflowSignalDisabled](https://docs.temporal.io/references/web-ui-configuration#workflowsignaldisabled) * [workflowTerminateDisabled](https://docs.temporal.io/references/web-ui-configuration#workflowterminatedisabled) --- # Operations | Temporal Platform Documentation [Skip to main content](https://docs.temporal.io/references/operation-list#__docusaurus_skipToContent_fallback) Temporal Cloud [rate limits operations per second (OPS)](https://docs.temporal.io/cloud/limits#operations-per-second) per namespace. An operation is anything 1. a user does directly, or 2. Temporal does on behalf of the user in the background that results in load on Temporal Server. The exception is visibility queries: they do hit the Server (the query is passed from the server to the visibility store), but primarily the load is on the visibility system. Visibility rate limits are separate from OPS rate limits. Below is the list of operations, including: * operation name * description * priority (foreground is higher priority, background is lower priority) * effect of that operation being throttled | Operation ↕ | Description ↕ | Priority ↕ | Effect of Throttling ↕ | | --- | --- | --- | --- | | ActivityRetryTimer | Internal timer that schedules the retry of a failed Activity execution. | Background | Activity retry delayed. | | ActivityTimeout | Marks an Activity as timed‑out when its deadline passes. | Background | Activity task stays in-flight longer and timeout is delayed, so retry is also delayed (if there is a retry). | | CompleteNexusOperation | Finalises a Nexus operation after worker response. | Foreground | External call clean‑up/commit delayed. | | CreateSchedule | Creates a Temporal Schedule (cron‑like trigger). | Foreground | Client/SDK receives ResourceExhaustedError, leading to backoff and retry. | | CreateWorkflowRule | Adds a worker build‑ID versioning rule for a task queue. | Foreground | Rule enforcement is delayed. | | DeleteSchedule | Deletes an existing Schedule. | Foreground | Schedule may keep firing briefly. | | DeleteWorkerDeployment | Deletes a worker deployment record. | Foreground | Stale deployment metadata persists longer. | | DeleteWorkerDeploymentVersion | Deletes one build‑ID version entry under a deployment. | Foreground | Version appears valid until task processed. | | DeleteWorkflowExecution | Hard‑deletes workflow history and state. | Foreground | Storage is freed later; data visible longer. | | DeleteWorkflowRule | Deletes a worker build‑ID rule. | Foreground | Rule continues to exist until task handled. | | DeprecateNamespace | Marks a namespace as deprecated in cluster metadata. | Foreground | Clients still see it as active. | | DescribeBatchOperation | Returns metadata for a batch operation. | Foreground | Admin/UI read waits longer. | | DescribeDeployment | Shows worker deployment details. | Foreground | Info retrieval is delayed. | | DescribeNamespace | Returns namespace configuration. | Foreground | Admin read waits. | | DescribeSchedule | Reads a Schedule’s current state. | Foreground | Call responds slower. | | DescribeTaskQueue | Returns stats and configuration of a task queue. | Foreground | Monitoring dashboards lag. | | DescribeWorkerDeployment | Shows details of a specific worker deployment. | Foreground | Same delay in status. | | DescribeWorkerDeploymentVersion | Shows a particular build‑ID version record. | Foreground | Detail retrieval delayed. | | DescribeWorkflowExecution | Returns high‑level info for a workflow execution. | Foreground | Diagnostics/CLI wait longer. | | DescribeWorkflowRule | Reads a worker versioning rule. | Foreground | Admin tools wait. | | DispatchByEndpoint | Routes a Nexus task to workers by endpoint name. | Foreground | Task routing latencies increase. | | DispatchByNamespaceAndTaskQueue | Routes a Nexus task by namespace and task queue. | Foreground | Same: Nexus task starts later. | | ExecuteMultiOperation | Runs a compound operation. | Foreground | Operation is delayed | | GetClusterInfo | Returns information about cluster capabilities and versions. | Foreground | CLI/API calls take longer. | | GetCurrentDeployment | Fetches the cluster‑wide current worker deployment. | Foreground | Rollout tooling sees stale info. | | GetSearchAttributes | Lists custom search attribute definitions. | Foreground | SDK/CLI wait. | | GetSystemInfo | Returns system build & feature info. | Foreground | Diagnostic call delayed. | | GetWorkerBuildIdCompatibility | Returns build‑ID compatibility matrix for a task queue. | Foreground | Rollout decisions wait. | | GetWorkerVersioningRules | Lists worker versioning rules. | Foreground | Admin listing delayed. | | GetWorkflowExecutionHistory | Streams workflow history in forward order. | Foreground | History load in UI/CLI is slow. | | GetWorkflowExecutionHistoryReverse | Streams history in reverse order. | Foreground | Same slower history read. | | ListNamespaces | Lists all namespaces in the cluster. | Foreground | UI/CLI list paginates slower. | | ListScheduleMatchingTimes | Computes future fire‑times for a Schedule. | Foreground | Preview takes longer. | | ListTaskQueuePartitions | Lists partitions backing a task queue. | Foreground | Load‑balancing insight lag. | | ListWorkerDeployments | Lists all worker deployments. | Foreground | Deployment inventory delayed. | | ListWorkflowRules | Lists all build‑ID versioning rules. | Foreground | Admin list wait. | | PatchSchedule | Modifies an existing Schedule (e.g., add interval). | Foreground | Change becomes effective later. | | PauseActivity | Server‑side API to pause a long‑running Activity. | Foreground | Activity continues running until pause task processed. | | PollActivityTaskQueue | Worker long‑poll for Activity tasks. | Foreground | Activity poller receives ResourceExhaustedError, leading to automatic backoff and retry, which slows down activity task processing. | | PollNexusTaskQueue | Worker long‑poll for Nexus tasks. | Foreground | Nexus task poller receives ResourceExhaustedError, leading to automatic backoff and retry, which slows down Nexus task processing. | | PollWorkflowExecutionUpdate | Client poll for workflow‑update completion. | Foreground | Client waits extra for result. | | PollWorkflowTaskQueue | Worker long‑poll for Workflow Tasks. | Foreground | Workflow poller receives ResourceExhaustedError, leading to automatic backoff and retry, which slows down workflow progress. | | QueryWorkflow | Read‑only query on workflow state. | Foreground | Caller receives result later. | | RecordActivityTaskHeartbeat | Worker heartbeat for an Activity (by task token). | Foreground | Heartbeats delayed, risking false timeout. | | RecordActivityTaskHeartbeatById | Same heartbeat call using Activity/Run IDs. | Foreground | Same impact. | | RequestCancelWorkflowExecution | Client request to cancel a workflow. | Foreground | Cancellation propagates later. | | ResetActivity | Force‑restarts an Activity from scratch. | Foreground | Activity keeps running before reset takes effect. | | ResetStickyTaskQueue | Clears workflow’s sticky queue affinity. | Foreground | Tasks stay bound to prior worker longer. | | ResetWorkflowExecution | Server‑side rewind to past event and continue as new. | Foreground | Workflow continues in old state. | | RespondActivityTaskCanceled | Worker confirms Activity canceled. | Foreground | Workflow waits for ack. | | RespondActivityTaskCanceledById | Same by ID. | Foreground | Same wait. | | RespondActivityTaskCompleted | Worker returns Activity result. | Foreground | Workflow next step delayed. | | RespondActivityTaskCompletedById | Same by ID. | Foreground | Same delay. | | RespondActivityTaskFailed | Worker reports Activity failure. | Foreground | Retry/compensation delayed. | | RespondActivityTaskFailedById | Same by ID. | Foreground | Same delay. | | RespondNexusTaskCompleted | Worker returns Nexus task success. | Foreground | External call completion delayed. | | RespondNexusTaskFailed | Worker returns Nexus task failure. | Foreground | Error handling delayed. | | RespondQueryTaskCompleted | Worker returns query result. | Foreground | Client waits longer. | | RespondWorkflowTaskCompleted | Worker returns new commands after WFT. | Foreground | Workflow commands applied later. | | RespondWorkflowTaskFailed | Worker reports WFT failure. | Foreground | Retry/new task creation delayed. | | SetCurrentDeployment | Sets which worker deployment is current for a queue. | Foreground | Version switch postponed. | | SetCurrentDeploymentVersion | Sets current build‑ID version number. | Foreground | Rollout holds. | | SetWorkerDeploymentCurrentVersion | Sets version for a specific deployment. | Foreground | Deployment stays on old version. | | SetWorkerDeploymentRampingVersion | Sets percentage ramp for new version. | Foreground | Canary rollout paused. | | ShutdownWorker | Gracefully shuts down a running worker via server call. | Foreground | Worker keeps running a bit longer. | | SignalWithStartWorkflowExecution | Signals an existing run or starts a new one with a signal. | Foreground | Signal/start both delayed. | | SignalWorkflowExecution | Sends an asynchronous signal to a workflow. | Foreground | Signal arrives late. | | StartBatchOperation | Starts a batch admin operation (e.g., bulk reset). | Foreground | Batch begins later. | | StartWorkflowExecution | Creates a new workflow run. | Foreground | Start latency increases. | | StateMachineOutbound | Emits outbound commands generated by the new state‑machine core. | Background | Commands queue up, deferring their side‑effects. | | StateMachineTimer | Fires timers managed by the state‑machine core. | Background | Timed events (e.g., sleeps) occur later. | | StopBatchOperation | Stops/cancels a batch operation. | Foreground | Batch keeps working until task processed. | | TerminateWorkflowExecution | Force‑terminates a workflow run. | Foreground | Workflow continues running longer. | | TransferActivityTask | Dispatches an Activity task to a worker on its task queue. | Background | Activity task schedule is delayed. | | TransferCancelExecution | Sends a cancel request to the target workflow. | Background | Target workflow receives the cancel later. | | TransferCloseExecution | Close a running workflow execution | Background | Workflow remains open. | | TransferResetWorkflow | Initiates a reset of a workflow’s state/history. | Background | Reset is postponed; old state continues. | | TransferSignalExecution | Delivers a signal event to another workflow. | Background | Signal arrives late, delaying downstream logic. | | TransferStartChildExecution | Starts a configured child workflow run. | Background | Child workflow starts later than expected. | | TransferWorkflowTask | Schedules a Workflow Task for workers. | Background | Workflow task schedule is delayed. | | TriggerWorkflowRule | Manually triggers a worker build‑ID rule. | Foreground | Trigger effect deferred. | | UnpauseActivity | Resumes a previously paused Activity. | Foreground | Activity remains paused longer. | | UpdateActivityOptions | Updates retry/timeout options of a running Activity. | Foreground | New options take effect later. | | UpdateSchedule | Updates fields of an existing Schedule. | Foreground | Updated schedule behavior delayed. | | UpdateWorkerDeploymentVersionMetadata | Updates metadata on a deployment version. | Foreground | Metadata remains outdated. | | UpdateWorkflowExecution | Server‑side update (workflow update API). | Foreground | Update is accepted later. | | UpdateWorkflowExecutionOptions | Changes execution options (memo, search attributes). | Foreground | Option changes visible later. | | UserTimer | Fires a user‑defined timer created inside a workflow. | Background | User timer (including workflow sleep) fire event delayed. | | VisibilityCloseExecution | Updates a closed workflow in Visibility store. | Background | Closed workflow remains open in list workflow visibility API results until task is processed. | | VisibilityStartExecution | Creates a record for the workflow in Visibility store. | Background | New run is absent from lists in UI and CLI output until task processed. | | VisibilityUpsertExecution | Updates search attributes for a workflow run. | Background | Search attribute update is delayed. | | WorkflowBackoffTimer | Timer that delays retry or continued‑as‑new start per backoff policy. | Background | Workflow start delayed beyond intended backoff time. | | WorkflowExecutionTimeout | Enforces the max total execution duration of a workflow. | Background | Timeout of workflow execution chain is delayed. | | WorkflowRunTimeout | Enforces timeout of an individual workflow run. | Background | Timeout on workflow run is delayed. | | WorkflowTaskTimeout | Times out a Workflow Task that a worker hasn’t completed in time. | Background | Timeout workflow task remains outstanding; retried workflow task is delayed. | --- # Temporal Web UI environment variables reference | Temporal Platform Documentation [Skip to main content](https://docs.temporal.io/references/web-ui-environment-variables#__docusaurus_skipToContent_fallback) On this page You can use environment variables to dynamically alter the configuration of your Temporal Web UI. These can be used in many environments, such as with Docker. For example: docker run\-e TEMPORAL_ADDRESS=127.0.0.1:7233\-e TEMPORAL_UI_PORT=8080\-e TEMPORAL_UI_PUBLIC_PATH=path/to/webui\-e TEMPORAL_UI_ENABLED=true\-e TEMPORAL_CLOUD_UI=false\-e TEMPORAL_DEFAULT_NAMESPACE=default\-e TEMPORAL_FEEDBACK_URL=https://feedback.here\-e TEMPORAL_CONFIG_REFRESH_INTERVAL=0s\-e TEMPORAL_SHOW_TEMPORAL_SYSTEM_NAMESPACE=false\-e TEMPORAL_DISABLE_WRITE_ACTIONS=false\-e TEMPORAL_AUTH_ENABLED=true\-e TEMPORAL_AUTH_TYPE=oidc\-e TEMPORAL_AUTH_PROVIDER_URL=https://accounts.google.com\-e TEMPORAL_AUTH_ISSUER_URL=https://accounts.google.com\-e TEMPORAL_AUTH_CLIENT_ID=xxxxx-xxxx.apps.googleusercontent.com\-e TEMPORAL_AUTH_CLIENT_SECRET=xxxxxxxxxxxxxxx\-e TEMPORAL_AUTH_CALLBACK_URL=https://xxxx.com:8080/auth/sso/callback\-e TEMPORAL_AUTH_SCOPES=openid,email,profile\-e TEMPORAL_TLS_CA=../ca.cert\-e TEMPORAL_TLS_CERT=../cluster.pem\-e TEMPORAL_TLS_KEY=../cluster.key\-e TEMPORAL_TLS_ENABLE_HOST_VERIFICATION=true\-e TEMPORAL_TLS_SERVER_NAME=tls-server\-e TEMPORAL_CODEC_ENDPOINT=https://codec.server\-e TEMPORAL_CODEC_PASS_ACCESS_TOKEN=false\-e TEMPORAL_CODEC_INCLUDE_CREDENTIALS=false\-e TEMPORAL_HIDE_LOGS=false\temporalio/ui: The environment variables are defined in the [UI server configuration template file](https://github.com/temporalio/ui-server/blob/main/config/docker.yaml) and described in more detail below. `TEMPORAL_ADDRESS`[​](https://docs.temporal.io/references/web-ui-environment-variables#temporal_address "Direct link to temporal_address") ------------------------------------------------------------------------------------------------------------------------------------------- The [Frontend Service](https://docs.temporal.io/temporal-service/temporal-server#frontend-service) address for the Temporal Cluster. This variable can be set [in the base configuration file](https://docs.temporal.io/references/web-ui-configuration#temporalgrpcaddress) using `temporalGrpcAddress`. This variable is required for setting other environment variables. `TEMPORAL_UI_PORT`[​](https://docs.temporal.io/references/web-ui-environment-variables#temporal_ui_port "Direct link to temporal_ui_port") ------------------------------------------------------------------------------------------------------------------------------------------- The port used by the Temporal WebUI Server and the HTTP API. This variable is needed for `TEMPORAL_OPENAPI_ENABLED` and all auth-related settings to work properly. `TEMPORAL_UI_PUBLIC_PATH`[​](https://docs.temporal.io/references/web-ui-environment-variables#temporal_ui_public_path "Direct link to temporal_ui_public_path") ---------------------------------------------------------------------------------------------------------------------------------------------------------------- Stores a value such as "" or "/custom-path" that allows the UI to be served from a subpath. `TEMPORAL_UI_ENABLED`[​](https://docs.temporal.io/references/web-ui-environment-variables#temporal_ui_enabled "Direct link to temporal_ui_enabled") ---------------------------------------------------------------------------------------------------------------------------------------------------- Enables or disables the [browser UI](https://docs.temporal.io/references/web-ui-configuration#enableui) for the Temporal Cluster. Enabling the browser UI allows the Server to be accessed from your web browser. If disabled, the server cannot be viewed on the web, but the UI server APIs remain available for use. `TEMPORAL_CLOUD_UI`[​](https://docs.temporal.io/references/web-ui-environment-variables#temporal_cloud_ui "Direct link to temporal_cloud_ui") ---------------------------------------------------------------------------------------------------------------------------------------------- If enabled, use the alternate UI from Temporal Cloud. `TEMPORAL_DEFAULT_NAMESPACE`[​](https://docs.temporal.io/references/web-ui-environment-variables#temporal_default_namespace "Direct link to temporal_default_namespace") ------------------------------------------------------------------------------------------------------------------------------------------------------------------------- The default [Namespace](https://docs.temporal.io/namespaces) that the Web UI opens first. `TEMPORAL_FEEDBACK_URL`[​](https://docs.temporal.io/references/web-ui-environment-variables#temporal_feedback_url "Direct link to temporal_feedback_url") ---------------------------------------------------------------------------------------------------------------------------------------------------------- The URL that users are directed to when they click the Feedback button in the UI. If not specified, this variable defaults to the UI's GitHub Issue page. `TEMPORAL_CONFIG_REFRESH_INTERVAL`[​](https://docs.temporal.io/references/web-ui-environment-variables#temporal_config_refresh_interval "Direct link to temporal_config_refresh_interval") ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Determines how often the UI Server reads the configuration file for new values. `TEMPORAL_SHOW_TEMPORAL_SYSTEM_NAMESPACE`[​](https://docs.temporal.io/references/web-ui-environment-variables#temporal_show_temporal_system_namespace "Direct link to temporal_show_temporal_system_namespace") ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- If enabled, shows the System Namespace that handles internal Temporal Workflows in the Web UI. `TEMPORAL_DISABLE_WRITE_ACTIONS`[​](https://docs.temporal.io/references/web-ui-environment-variables#temporal_disable_write_actions "Direct link to temporal_disable_write_actions") ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Disables any button in the UI that allows the user to modify Workflows or Activities. `TEMPORAL_AUTH_ENABLED`[​](https://docs.temporal.io/references/web-ui-environment-variables#temporal_auth_enabled "Direct link to temporal_auth_enabled") ---------------------------------------------------------------------------------------------------------------------------------------------------------- Enables or disables Web UI authentication and authorization methods. When enabled, the Web UI will use the provider information in the [UI configuration file](https://docs.temporal.io/references/web-ui-configuration#auth) to verify the identity of users. All auth-related variables can be defined when `TEMPORAL_AUTH_ENABLED` is set to "true". Disabling the variable will retain given values. `TEMPORAL_AUTH_TYPE`[​](https://docs.temporal.io/references/web-ui-environment-variables#temporal_auth_type "Direct link to temporal_auth_type") ------------------------------------------------------------------------------------------------------------------------------------------------- Specifies the type of authentication. Defaults to `oidc`. `TEMPORAL_AUTH_PROVIDER_URL`[​](https://docs.temporal.io/references/web-ui-environment-variables#temporal_auth_provider_url "Direct link to temporal_auth_provider_url") ------------------------------------------------------------------------------------------------------------------------------------------------------------------------- The .well-known IDP discovery URL for authentication and authorization. This can be set as in the UI server configuration with [auth](https://docs.temporal.io/references/web-ui-configuration#auth) . `TEMPORAL_AUTH_ISSUER_URL`[​](https://docs.temporal.io/references/web-ui-environment-variables#temporal_auth_issuer_url "Direct link to temporal_auth_issuer_url") ------------------------------------------------------------------------------------------------------------------------------------------------------------------- The URL for the authentication or authorization issuer. This value is only needed when the issuer differs from the auth provider URL. `TEMPORAL_AUTH_CLIENT_ID`[​](https://docs.temporal.io/references/web-ui-environment-variables#temporal_auth_client_id "Direct link to temporal_auth_client_id") ---------------------------------------------------------------------------------------------------------------------------------------------------------------- The client ID used for authentication or authorization. This value is a required parameter. `TEMPORAL_AUTH_CLIENT_SECRET`[​](https://docs.temporal.io/references/web-ui-environment-variables#temporal_auth_client_secret "Direct link to temporal_auth_client_secret") ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- The client secret used for authentication and authorization. Client Secrets are used by the oAuth Client for authentication. `TEMPORAL_AUTH_CALLBACK_URL`[​](https://docs.temporal.io/references/web-ui-environment-variables#temporal_auth_callback_url "Direct link to temporal_auth_callback_url") ------------------------------------------------------------------------------------------------------------------------------------------------------------------------- The callback URL used by Temporal for authentication and authorization. Callback URLs are invoked by IDP after user has finished authenticating in IDP. `TEMPORAL_AUTH_SCOPES`[​](https://docs.temporal.io/references/web-ui-environment-variables#temporal_auth_scopes "Direct link to temporal_auth_scopes") ------------------------------------------------------------------------------------------------------------------------------------------------------- Specifies a set of scopes for auth. Typically, this is `openid`, `profile`, `email`. `TEMPORAL_TLS_CA`[​](https://docs.temporal.io/references/web-ui-environment-variables#temporal_tls_ca "Direct link to temporal_tls_ca") ---------------------------------------------------------------------------------------------------------------------------------------- The path for the Transport Layer Security (TLS) Certificate Authority file. In order to [configure TLS for your server](https://docs.temporal.io/references/web-ui-configuration#tls) , you'll need a CA certificate issued by a trusted Certificate Authority. Set this variable to properly locate and use the file. `TEMPORAL_TLS_CERT`[​](https://docs.temporal.io/references/web-ui-environment-variables#temporal_tls_cert "Direct link to temporal_tls_cert") ---------------------------------------------------------------------------------------------------------------------------------------------- The path for the Transport Layer Security (TLS) Certificate. In order to [configure TLS for your server](https://docs.temporal.io/references/web-ui-configuration#tls) , you'll need a self-signed certificate. Set the path to allow the environment to locate and use the certificate. `TEMPORAL_TLS_KEY`[​](https://docs.temporal.io/references/web-ui-environment-variables#temporal_tls_key "Direct link to temporal_tls_key") ------------------------------------------------------------------------------------------------------------------------------------------- The path for the Transport Layer Security (TLS) [key file](https://docs.temporal.io/references/web-ui-configuration#tls) . A key file is used to create private and public keys for encryption and signing. Together, these keys are used to create certificates. `TEMPORAL_TLS_CA_DATA`[​](https://docs.temporal.io/references/web-ui-environment-variables#temporal_tls_ca_data "Direct link to temporal_tls_ca_data") ------------------------------------------------------------------------------------------------------------------------------------------------------- Stores the data for a TLS CA file. This variable can be used instead of providing a path for `TEMPORAL_TLS_CA`. `TEMPORAL_TLS_CERT_DATA`[​](https://docs.temporal.io/references/web-ui-environment-variables#temporal_tls_cert_data "Direct link to temporal_tls_cert_data") ------------------------------------------------------------------------------------------------------------------------------------------------------------- Stores the data for a TLS cert file. This variable can be used instead of providing a path for `TEMPORAL_TLS_CERT`. `TEMPORAL_TLS_KEY_DATA`[​](https://docs.temporal.io/references/web-ui-environment-variables#temporal_tls_key_data "Direct link to temporal_tls_key_data") ---------------------------------------------------------------------------------------------------------------------------------------------------------- Stores the data for a TLS key file. This variable can be used instead of providing a path for `TEMPORAL_TLS_KEY`. `TEMPORAL_TLS_ENABLE_HOST_VERIFICATION`[​](https://docs.temporal.io/references/web-ui-environment-variables#temporal_tls_enable_host_verification "Direct link to temporal_tls_enable_host_verification") ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Enables or disables [Transport Layer Security (TLS) host verification](https://docs.temporal.io/references/web-ui-configuration#tls) . When enabled, TLS checks the Host Server to ensure that files are being sent to and from the correct URL. `TEMPORAL_TLS_SERVER_NAME`[​](https://docs.temporal.io/references/web-ui-environment-variables#temporal_tls_server_name "Direct link to temporal_tls_server_name") ------------------------------------------------------------------------------------------------------------------------------------------------------------------- The server on which to operate [Transport Layer Security (TLS) protocols](https://docs.temporal.io/references/web-ui-configuration#tls) . TLS allows the current server to transmit encrypted files to other URLs without having to reveal itself. Because of this, TLS operates a go-between server. `TEMPORAL_CODEC_ENDPOINT`[​](https://docs.temporal.io/references/web-ui-environment-variables#temporal_codec_endpoint "Direct link to temporal_codec_endpoint") ---------------------------------------------------------------------------------------------------------------------------------------------------------------- The endpoint for the [Codec Server](https://docs.temporal.io/codec-server) , if configured. `TEMPORAL_CODEC_PASS_ACCESS_TOKEN`[​](https://docs.temporal.io/references/web-ui-environment-variables#temporal_codec_pass_access_token "Direct link to temporal_codec_pass_access_token") ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Specifies whether to send a JWT access token as ‘authorization' header in requests with the Codec Server. `TEMPORAL_CODEC_INCLUDE_CREDENTIALS`[​](https://docs.temporal.io/references/web-ui-environment-variables#temporal_codec_include_credentials "Direct link to temporal_codec_include_credentials") ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Specifies whether to include credentials along with requests to the Codec Server. `TEMPORAL_FORWARD_HEADERS`[​](https://docs.temporal.io/references/web-ui-environment-variables#temporal_forward_headers "Direct link to temporal_forward_headers") ------------------------------------------------------------------------------------------------------------------------------------------------------------------- Forward-specified HTTP headers to direct from HTTP API requests to the Temporal gRPC backend. This is a comma-delimited list of the HTTP headers to be forwarded. `TEMPORAL_HIDE_LOGS`[​](https://docs.temporal.io/references/web-ui-environment-variables#temporal_hide_logs "Direct link to temporal_hide_logs") ------------------------------------------------------------------------------------------------------------------------------------------------- If enabled, does not print logs from the Temporal Service. * [`TEMPORAL_ADDRESS`](https://docs.temporal.io/references/web-ui-environment-variables#temporal_address) * [`TEMPORAL_UI_PORT`](https://docs.temporal.io/references/web-ui-environment-variables#temporal_ui_port) * [`TEMPORAL_UI_PUBLIC_PATH`](https://docs.temporal.io/references/web-ui-environment-variables#temporal_ui_public_path) * [`TEMPORAL_UI_ENABLED`](https://docs.temporal.io/references/web-ui-environment-variables#temporal_ui_enabled) * [`TEMPORAL_CLOUD_UI`](https://docs.temporal.io/references/web-ui-environment-variables#temporal_cloud_ui) * [`TEMPORAL_DEFAULT_NAMESPACE`](https://docs.temporal.io/references/web-ui-environment-variables#temporal_default_namespace) * [`TEMPORAL_FEEDBACK_URL`](https://docs.temporal.io/references/web-ui-environment-variables#temporal_feedback_url) * [`TEMPORAL_CONFIG_REFRESH_INTERVAL`](https://docs.temporal.io/references/web-ui-environment-variables#temporal_config_refresh_interval) * [`TEMPORAL_SHOW_TEMPORAL_SYSTEM_NAMESPACE`](https://docs.temporal.io/references/web-ui-environment-variables#temporal_show_temporal_system_namespace) * [`TEMPORAL_DISABLE_WRITE_ACTIONS`](https://docs.temporal.io/references/web-ui-environment-variables#temporal_disable_write_actions) * [`TEMPORAL_AUTH_ENABLED`](https://docs.temporal.io/references/web-ui-environment-variables#temporal_auth_enabled) * [`TEMPORAL_AUTH_TYPE`](https://docs.temporal.io/references/web-ui-environment-variables#temporal_auth_type) * [`TEMPORAL_AUTH_PROVIDER_URL`](https://docs.temporal.io/references/web-ui-environment-variables#temporal_auth_provider_url) * [`TEMPORAL_AUTH_ISSUER_URL`](https://docs.temporal.io/references/web-ui-environment-variables#temporal_auth_issuer_url) * [`TEMPORAL_AUTH_CLIENT_ID`](https://docs.temporal.io/references/web-ui-environment-variables#temporal_auth_client_id) * [`TEMPORAL_AUTH_CLIENT_SECRET`](https://docs.temporal.io/references/web-ui-environment-variables#temporal_auth_client_secret) * [`TEMPORAL_AUTH_CALLBACK_URL`](https://docs.temporal.io/references/web-ui-environment-variables#temporal_auth_callback_url) * [`TEMPORAL_AUTH_SCOPES`](https://docs.temporal.io/references/web-ui-environment-variables#temporal_auth_scopes) * [`TEMPORAL_TLS_CA`](https://docs.temporal.io/references/web-ui-environment-variables#temporal_tls_ca) * [`TEMPORAL_TLS_CERT`](https://docs.temporal.io/references/web-ui-environment-variables#temporal_tls_cert) * [`TEMPORAL_TLS_KEY`](https://docs.temporal.io/references/web-ui-environment-variables#temporal_tls_key) * [`TEMPORAL_TLS_CA_DATA`](https://docs.temporal.io/references/web-ui-environment-variables#temporal_tls_ca_data) * [`TEMPORAL_TLS_CERT_DATA`](https://docs.temporal.io/references/web-ui-environment-variables#temporal_tls_cert_data) * [`TEMPORAL_TLS_KEY_DATA`](https://docs.temporal.io/references/web-ui-environment-variables#temporal_tls_key_data) * [`TEMPORAL_TLS_ENABLE_HOST_VERIFICATION`](https://docs.temporal.io/references/web-ui-environment-variables#temporal_tls_enable_host_verification) * [`TEMPORAL_TLS_SERVER_NAME`](https://docs.temporal.io/references/web-ui-environment-variables#temporal_tls_server_name) * [`TEMPORAL_CODEC_ENDPOINT`](https://docs.temporal.io/references/web-ui-environment-variables#temporal_codec_endpoint) * [`TEMPORAL_CODEC_PASS_ACCESS_TOKEN`](https://docs.temporal.io/references/web-ui-environment-variables#temporal_codec_pass_access_token) * [`TEMPORAL_CODEC_INCLUDE_CREDENTIALS`](https://docs.temporal.io/references/web-ui-environment-variables#temporal_codec_include_credentials) * [`TEMPORAL_FORWARD_HEADERS`](https://docs.temporal.io/references/web-ui-environment-variables#temporal_forward_headers) * [`TEMPORAL_HIDE_LOGS`](https://docs.temporal.io/references/web-ui-environment-variables#temporal_hide_logs) --- # Safely deploying changes to Workflow code | Temporal Platform Documentation [Skip to main content](https://docs.temporal.io/develop/safe-deployments#__docusaurus_skipToContent_fallback) On this page Making changes safely to existing Workflow code requires care. Your Workflow code--as opposed to your Activity code--must be [deterministic](https://docs.temporal.io/workflow-definition#deterministic-constraints) . This means your changes to that code have to be as well. Changes to your Workflow code that qualify as non-deterministic need to be protected by either using [Worker Versioning](https://docs.temporal.io/production-deployment/worker-deployments/worker-versioning) to pin your Workflows to specific code revisions, or by using the [patching APIs](https://docs.temporal.io/workflow-definition#workflow-versioning) within your Workflow code. note We strongly recommend using Worker Versioning as users see improved error rates when adopting it. In this article, we’ll provide some advice on how you can safely validate changes to your Workflow code, ensuring that you won’t experience unexpected non-determinism errors in production when rolling them out. caution Eager start does not respect Worker versioning. An eagerly started Workflow may run on any available local Worker even if that Worker is not the Current or Ramping version of its Worker deployment. Use Replay Testing before and during your deployments[​](https://docs.temporal.io/develop/safe-deployments#use-replay-testing-before-and-during-your-deployments "Direct link to Use Replay Testing before and during your deployments") ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- The best way to verify that your code won’t cause non-determinism errors once deployed is to make use of [replay testing](https://docs.temporal.io/workflow-execution#replay) . Replay testing takes one or more existing [Workflow Histories](https://docs.temporal.io/workflow-execution/event#event-history) that ran against a previous version of Workflow code and runs them against your _current_ Workflow code, verifying that it is compatible with the provided history. In the case of Worker Versioning, you may have a [pinned Workflow](https://docs.temporal.io/worker-versioning#pinned) that you're switching over to the [current Worker deployment version](https://docs.temporal.io/worker-versioning#versioning-definitions) and you want to make sure that the changes don't introduce non-determinism errors. Or you may have an [Auto-Upgrade Workflow](https://docs.temporal.io/worker-versioning#auto-upgrade) that you want to run automated tests on to ensure the deployments don't trigger errors. There are multiple points in your development lifecycle where running replay tests can make sense. They exist on a spectrum, with shortest time to feedback on one end, and most representative of a production deployment on the other. * During development, replay testing lets you get feedback as early as possible on whether your changes are compatible. For example, you might include some integration tests that run your Workflows against the Temporal Test Server to produce histories which you then check in. You can use those checked-in histories for replay tests to verify you haven’t made breaking changes. * During pre-deployment validation (such as during some automated deployment validation) you can get feedback in a more representative environment. For example, you might fetch histories from a live Temporal environment (whether production or some kind of pre-production) and use them in replay tests. * At deployment time, your environment _is_ production, but you are using the new code to replay recent real-world Workflow histories. When you're writing changes to Workflow code, you can fetch some representative histories from your pre-production or production Temporal environment and verify they work with your changes. You can do the same with the pre-merge CI pipeline. However, if you are using encrypted Payloads, which is a typical and recommended setup in production, you may not be able to decrypt the fetched histories. Additionally if your Workflows contain any PII (which should be encrypted), make sure this information is scrubbed for the purposes of your tests, or err on the side of caution and don’t use this method. With that constraint in mind, we’ll focus on how you can perform replay tests in a production deployment of a Worker with new Workflow code. The core of how replay testing is done is the same regardless of when you choose to do it, so you can apply some of the lessons here to earlier stages in your development process. Implement a deployment-time replay test[​](https://docs.temporal.io/develop/safe-deployments#implement-a-deployment-time-replay-test "Direct link to Implement a deployment-time replay test") ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- The key to a successful safe deployment is to break it into two phases: a verification phase, where you’ll run the replay test, followed by the actual deployment of your new Worker code. You can accomplish this by wrapping your Worker application with some code that can choose whether it will run in verification mode, or in production. This is most easily done if you do not deploy your Workers side-by-side with other application code, which is a recommended best practice. If you do deploy your Workers as part of some other application, you will likely need to separate out a different entry point specifically for verification. ### Run a replay and real Worker with the same code[​](https://docs.temporal.io/develop/safe-deployments#run-a-replay-and-real-worker-with-the-same-code "Direct link to Run a replay and real Worker with the same code") The following code demonstrates how the same entry point could be used to either verify the new code using replay testing, or to actually run the Worker. import argparseimport asynciofrom datetime import datetime, timedeltafrom temporalio.client import Clientfrom temporalio.worker import Worker, Replayerasync def main(): parser = argparse.ArgumentParser(prog='MyTemporalWorker') parser.add_argument('mode', choices=['verify', 'run']) args = parser.parse_args() temporal_url = "localhost:7233" task_queue = "your-task-queue" my_workflows = [YourWorkflow] my_activities = [your_activity] client = await Client.connect(temporal_url) Everything up to this point is standard. You import the Workflow and Activity code, instantiate a parser with two modes, and create your Task Queue, Workflow, and Activity. You can pass in the `args.mode` from any appropriate spot in your code. If the mode is set to `verify`, you conduct the replay testing by specifying the time period to test, and passing in the Workflows corresponding to that time period. Note that the Workflows are consumed as histories, using [the `map_histories()` function](https://python.temporal.io/temporalio.client.WorkflowExecutionAsyncIterator.html#map_histories) . if args.mode == 'verify': start_time = (datetime.now() - timedelta(hours=10)).isoformat(timespec='seconds') workflows = client.list_workflows( f"TaskQueue={task_queue} and StartTime > '{start_time}'", limit = 100) histories = workflows.map_histories() replayer = Replayer( workflows=my_workflows, ) await replayer.replay_workflows(histories) return If any of the Workflows fail to replay, an error will be thrown. If no errors occur, you can return successfully to indicate success here, or communicate with an endpoint you've defined to indicate success or failure of the verification. You could switch to the `run` mode, and have this Worker transition to a real Worker that will start pulling from the Task Queue and processing Workflows: else: worker = Worker( client, task_queue=task_queue, workflows=my_workflows, activities=my_activities, ) await worker.run()if __name__ == "__main__": asyncio.run(main()) ### Use the multi-modal Worker[​](https://docs.temporal.io/develop/safe-deployments#use-the-multi-modal-worker "Direct link to Use the multi-modal Worker") The most straightforward way to use this bimodal Worker is to deploy one instance of it at the beginning of your deployment process in verify mode, see that it passes, and then proceed to deploy the rest of your new workers in run mode. * [Use Replay Testing before and during your deployments](https://docs.temporal.io/develop/safe-deployments#use-replay-testing-before-and-during-your-deployments) * [Implement a deployment-time replay test](https://docs.temporal.io/develop/safe-deployments#implement-a-deployment-time-replay-test) * [Run a replay and real Worker with the same code](https://docs.temporal.io/develop/safe-deployments#run-a-replay-and-real-worker-with-the-same-code) * [Use the multi-modal Worker](https://docs.temporal.io/develop/safe-deployments#use-the-multi-modal-worker) --- # Environment configuration | Temporal Platform Documentation [Skip to main content](https://docs.temporal.io/references/client-environment-configuration#__docusaurus_skipToContent_fallback) The following table details all available settings, their corresponding environment variables, and their TOML file paths. For more information on using environment variables and configuration files to set up your Temporal Client, refer to the [Environment Configuration](https://docs.temporal.io/develop/environment-configuration) . | Setting | Environment Variable | TOML Path | Description | | --- | --- | --- | --- | | Configuration File Path | `TEMPORAL_CONFIG_FILE` | **NA** | Path to the TOML configuration file | | Server Address | `TEMPORAL_ADDRESS` | `profile..address` | The host and port of the Temporal Frontend service (e.g., "localhost:7233"). | | Namespace | `TEMPORAL_NAMESPACE` | `profile..namespace` | The Temporal Namespace to connect to. | | API Key | `TEMPORAL_API_KEY` | `profile..api_key` | An API key for authentication. If present, TLS is enabled by default. | | Enable/Disable TLS | `TEMPORAL_TLS` | `profile..tls.disabled` | Set to "true" to enable TLS, "false" to disable. In TOML, disabled = true turns TLS off. | | Client Certificate | `TEMPORAL_TLS_CLIENT_CERT_DATA` | `profile..tls.client_cert_data` | The raw PEM data containing the client's public TLS certificate. Alternatively, you can use `TEMPORAL_TLS_CLIENT_CERT_PATH` to provide a path to the certificate or the TOML `profile..tls.client_cert_path`. | | Client Certificate Path | `TEMPORAL_TLS_CLIENT_CERT_PATH` | `profile..tls.client_cert_path` | A filesystem path to the client's public TLS certificate. Alternatively, you can provide the raw PEM data using `TEMPORAL_TLS_CLIENT_CERT_DATA` or the TOML `profile..tls.client_cert_data`. | | Client Key | `TEMPORAL_TLS_CLIENT_KEY_DATA` | `profile..tls.client_key_data` | The raw PEM data containing the client's private TLS key. Alternatively, you can use `TEMPORAL_TLS_CLIENT_KEY_PATH` to provide a path to the key or the TOML `profile..tls.client_key_path`. | | Client Key Path | `TEMPORAL_TLS_CLIENT_KEY_PATH` | `profile..tls.client_key_path` | A filesystem path to the client's private TLS key. Alternatively, you can provide the raw PEM data using `TEMPORAL_TLS_CLIENT_KEY_DATA` or the TOML `profile..tls.client_key_data`. | | Server CA Cert | `TEMPORAL_TLS_SERVER_CA_CERT_DATA` | `profile..tls.server_ca_cert_data` | The raw PEM data for the Certificate Authority certificate used to verify the server. Alternatively, you can use `TEMPORAL_TLS_SERVER_CA_CERT_PATH` to provide a path or the TOML `profile..tls.server_ca_cert_path`. | | Server CA Cert Path | `TEMPORAL_TLS_SERVER_CA_CERT_PATH` | `profile..tls.server_ca_cert_path` | A filesystem path to the Certificate Authority certificate. Alternatively, you can provide the raw PEM data using `TEMPORAL_TLS_SERVER_CA_CERT_DATA` or the TOML `profile..tls.server_ca_cert_data`. | | TLS Server Name | `TEMPORAL_TLS_SERVER_NAME` | `profile..tls.server_name` | Overrides the server name used for Server Name Indication (SNI) in the TLS handshake. | | Disable Host Verification | `TEMPORAL_TLS_DISABLE_HOST_VERIFICATION` | `profile..tls.disable_host_verification` | A boolean to disable server hostname verification. Use with caution. Not supported by all SDKs. | | Codec Endpoint | `TEMPORAL_CODEC_ENDPOINT` | `profile..codec.endpoint` | The endpoint for a remote Data Converter. This is not supported by all SDKs. SDKs that support this configuration don't apply it by default. Intended mostly for CLI use. | | Codec Auth | `TEMPORAL_CODEC_AUTH` | `profile..codec.auth` | The authorization header value for the remote data converter. | | gRPC Metadata | `TEMPORAL_GRPC_META_*` | `profile..grpc_meta` | Sets gRPC headers. The part after `_META_` becomes the header key (e.g., `_SOME_KEY` -> `some-key`). | --- # Temporal Cluster dynamic configuration reference | Temporal Platform Documentation [Skip to main content](https://docs.temporal.io/references/dynamic-configuration#__docusaurus_skipToContent_fallback) On this page Temporal Cluster provides [dynamic configuration](https://docs.temporal.io/temporal-service/configuration#dynamic-configuration) keys that you can update and apply to a running Cluster without restarting your services. The dynamic configuration keys are set with default values when you create your Cluster configuration. You can override these values as you test your Cluster setup for optimal performance according to your workload requirements. For the complete list of dynamic configuration keys, see [https://github.com/temporalio/temporal/blob/main/common/dynamicconfig/constants.go](https://github.com/temporalio/temporal/blob/main/common/dynamicconfig/constants.go) . Ensure that you check server release notes for any changes to these keys and values. For the default values of dynamic configuration keys, check the following links: * [Frontend Service](https://github.com/temporalio/temporal/blob/5783e781504d8ffac59f9848b830868f3139b980/service/frontend/service.go#L176) * [History Service](https://github.com/temporalio/temporal/blob/5783e781504d8ffac59f9848b830868f3139b980/service/history/configs/config.go#L309) * [Matching Service](https://github.com/temporalio/temporal/blob/5783e781504d8ffac59f9848b830868f3139b980/service/matching/config.go#L125) * [Worker Service](https://github.com/temporalio/temporal/blob/5783e781504d8ffac59f9848b830868f3139b980/service/worker/service.go#L193) Setting dynamic configuration is optional. Change these values only if you need to override the default values to achieve better performance on your Temporal Cluster. Also, ensure that you test your changes before setting these in production. Format[​](https://docs.temporal.io/references/dynamic-configuration#format "Direct link to Format") ---------------------------------------------------------------------------------------------------- To override the default dynamic configuration values, specify your custom values and constraints for the dynamic configuration keys that you want to change in a YAML configuration file. Use the following format when creating your dynamic configuration file. testGetBoolPropertyKey: - value: false - value: true constraints: namespace: 'your-namespace' - value: false constraints: namespace: 'your-other-namespace'testGetDurationPropertyKey: - value: '1m' constraints: namespace: 'your-namespace' taskQueueName: 'longIdleTimeTaskqueue'testGetFloat64PropertyKey: - value: 12.0 constraints: namespace: 'your-namespace'testGetMapPropertyKey: - value: key1: 1 key2: 'value 2' key3: - false - key4: true key5: 2.0 ### Constraints[​](https://docs.temporal.io/references/dynamic-configuration#constraints "Direct link to Constraints") You can define constraints on some dynamic configuration keys to set specific values that apply on a Namespace or Task Queue level. Not defining constraints on a dynamic configuration key sets the values across the Cluster. * To set global values for the configuration key with no constraints, use the following: frontend.globalNamespaceRPS: # Total per-Namespace RPC rate limit applied across the Cluster. - value: 5000 * For keys that can be customized at Namespace level, you can specify multiple values for different Namespaces in addition to one default value that applies globally to all Namespaces. To set values at a Namespace level, use `namespace` (String) as shown in the following example. frontend.persistenceNamespaceMaxQPS: # Rate limit on the number of queries the Frontend sends to the Persistence store. - constraints: {} # Sets default value that applies to all Namespaces value: 2000 # The default value for this key is 0. - constraints: { namespace: 'namespace1' } # Sets limit on number of queries that can be sent from "namespace1" Namespace to the Persistence store. value: 4000 - constraints: { namespace: 'namespace2' } value: 1000 * For keys that can be customized at a Task Queue level, you can specify Task Queue name and Task type in addition to Namespace. To set values at a Task Queue level, use `taskQueueName` (String) with `taskType` (optional; supported values: `Workflow` and `Activity`). For example if you have Workflow Executions creating a large number of Workflow and Activity tasks per second, you can add more partitions to your Task Queues (default is 4) to handle the high throughput of tasks. To do this, add the following to your dynamic configuration file. Note that if changing the number of partitions, you must set the same count for both read and write operations on Task Queues. matching.numTaskqueueReadPartitions: # Number of Task Queue partitions for read operations. - constraints: { namespace: 'namespace1', taskQueueName: 'tq' } # Applies to the "tq" Task Queue for both Workflows and Activities. value: 8 # The default value for this key is 4. Task Queues that need to support high traffic require higher number of partitions. Set these values in accordance to your poller count. - constraints: { namespace: 'namespace1', taskQueueName: 'other-tq', taskType: 'Activity', } # Applies to the "other_tq" Task Queue for Activities specifically. value: 20 - constraints: { namespace: 'namespace2' } # Applies to all task queues in "namespace2". value: 10 - constraints: {} # Applies to all other task queues in "namespace1" and all other Namespaces. value: 16matching.numTaskqueueWritePartitions: # Number of Task Queue partitions for write operations. - constraints: { namespace: 'namespace1', taskQueueName: 'tq' } # Applies to the "tq" Task Queue for both Workflows and Activities. value: 8 # The default value for this key is 4. Task Queues that need to support high traffic require higher number of partitions. Set these values in accordance to your poller count. - constraints: { namespace: 'namespace1', taskQueueName: 'other-tq', taskType: 'Activity', } # Applies to the "other_tq" Task Queue for Activities specifically. value: 20 - constraints: { namespace: 'namespace2' } # Applies to all task queues in "namespace2". value: 10 - constraints: {} # Applies to all other task queues in "namespace1" and all other Namespaces. value: 16 For more examples on how dynamic configuration is set, see [samples-server](https://github.com/temporalio/samples-server/blob/main/tls/config/dynamicconfig/development.yaml) . Commonly used dynamic configuration keys[​](https://docs.temporal.io/references/dynamic-configuration#commonly-used-dynamic-configuration-keys "Direct link to Commonly used dynamic configuration keys") ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- The following table lists commonly used dynamic configuration keys that can be used for rate limiting requests to the Temporal Cluster. Setting dynamic configuration keys is optional. If you choose to update these values for your Temporal Cluster, ensure that you are provisioning enough resources to handle the load. All values listed here are for Temporal server v1.21. Check [server release notes](https://github.com/temporalio/temporal/releases) to verify any potential breaking changes when upgrading your versions. ### Service-level RPS limits[​](https://docs.temporal.io/references/dynamic-configuration#service-level-rps-limits "Direct link to Service-level RPS limits") The Requests Per Second (RPS) dynamic configuration keys set the rate at which requests can be made to each service in your Cluster. When scaling your services, tune the RPS to test your workload and set acceptable provisioning benchmarks. Exceeding these limits results in `ResourceExhaustedError`. | Dynamic configuration key | Type | Description | Default value | | --- | --- | --- | --- | | Frontend | | | | | `frontend.rps` | Int | Rate limit (requests/second) for requests accepted by each Frontend Service host. | 2400 | | `frontend.namespaceRPS` | Int | Rate limit (requests/second) for requests accepted by each Namespace on the Frontend Service. | 2400 | | `frontend.namespaceCount` | Int | Limit on the number of concurrent Task Queue polls per Namespace per Frontend Service host. | 1200 | | `frontend.globalNamespaceRPS` | Int | Rate limit (requests/second) for requests accepted per Namespace, applied across Cluster. The limit is evenly distributed among available Frontend Service instances. If this is set, it overrides the per-instance limit (`frontend.namespaceRPS`). | 0 | | `internal-frontend.globalNamespaceRPS` | Int | Rate limit (requests/second) for requests accepted on each Internal-Frontend Service host applied across the Cluster. | 0 | | History | | | | | `history.rps` | Int | Rate limit (requests/second) for requests accepted by each History Service host. | 3000 | | Matching | | | | | `matching.rps` | Int | Rate limit (requests/second) for requests accepted by each Matching Service host. | 1200 | | `matching.numTaskqueueReadPartitions` | Int | Number of read partitions for a Task Queue. Must be set with `matching.numTaskqueueWritePartitions`. | 4 | | `matching.numTaskqueueWritePartitions` | Int | Number of write partitions for a Task Queue. | 4 | ### QPS limits for Persistence store[​](https://docs.temporal.io/references/dynamic-configuration#qps-limits-for-persistence-store "Direct link to QPS limits for Persistence store") The Queries Per Second (QPS) dynamic configuration keys set the maximum number of queries a service can make per second to the Persistence store. Persistence store rate limits are evaluated synchronously. Adjust these keys according to your database capacity and workload. If the number of queries made to the Persistence store exceeds the dynamic configuration value, you will see latencies and timeouts on your tasks. | Dynamic configuration key | Type | Description | Default value | | --- | --- | --- | --- | | Frontend | | | | | `frontend.persistenceMaxQPS` | Int | Maximum number queries per second that the Frontend Service host can send to the Persistence store. | 2000 | | `frontend.persistenceNamespaceMaxQPS` | Int | Maximum number of queries per second that each Namespace on the Frontend Service host can send to the Persistence store.
If the value set for this config is less than or equal to 0, the value set for `frontend.persistenceMaxQPS` will apply. | 0 | | History | | | | | `history.persistenceMaxQPS` | Int | Maximum number of queries per second that the History host can send to the Persistence store. | 9000 | | `history.persistenceNamespaceMaxQPS` | Int | Maximum number of queries per second for each Namespace that the History host can send to the Persistence store.
If the value set for this config is less than or equal to 0, then the value set for `history.persistenceMaxQPS` will apply. | 0 | | Matching | | | | | `matching.persistenceMaxQPS` | Int | Maximum number of queries per second that the Matching Service host can send to the Persistence store. | 9000 | | `matching.persistenceNamespaceMaxQPS` | Int | Maximum number of queries per second that the Matching host can send to the Persistence store for each Namespace.
If the value set for this config is less than or equal to 0, the value set for `matching.persistenceMaxQPS` will apply. | 0 | | Worker | | | | | `worker.persistenceMaxQPS` | Int | Maximum number of queries per second that the Worker Service host can send to the Persistence store. | 100 | | `worker.persistenceNamespaceMaxQPS` | Int | Maximum number of queries per second that the Worker host can send to the Persistence store for each Namespace.
If the value set for this config is less than or equal to 0, the value set for `worker.persistenceMaxQPS` will apply. | 0 | | Visibility | | | | | `system.visibilityPersistenceMaxReadQPS` | Int | Maximum number queries per second that Visibility database can receive for read operations. | 9000 | | `system.visibilityPersistenceMaxWriteQPS` | Int | Maximum number of queries per second that Visibility database can receive for write operations. | 9000 | ### Activity and Workflow default policy setting[​](https://docs.temporal.io/references/dynamic-configuration#activity-and-workflow-default-policy-setting "Direct link to Activity and Workflow default policy setting") You can define default values for Activity and Workflow [Retry Policies](https://docs.temporal.io/encyclopedia/retry-policies) at the Cluster level with the following dynamic configuration keys. | Dynamic configuration key | Type | Description | Default value | | --- | --- | --- | --- | | `history.defaultActivityRetryPolicy` | Map (key-value pair elements) | Server configuration for an Activity Retry Policy when it is not explicitly set for the Activity in your code. | [Default values for retry Policy](https://docs.temporal.io/encyclopedia/retry-policies#default-values-for-retry-policy) | | `history.defaultWorkflowRetryPolicy` | Map (key-value pair elements) | Retry Policy for unset fields where the user has set an explicit `RetryPolicy`, but not specified all the fields. | [Default values for retry Policy](https://docs.temporal.io/encyclopedia/retry-policies#default-values-for-retry-policy) | ### Size limit settings[​](https://docs.temporal.io/references/dynamic-configuration#size-limit-settings "Direct link to Size limit settings") The Persistence store in the Cluster has default size limits set for optimal performance. The dynamic configuration keys relating to some of these are listed below. The default values on these keys are based on extensive testing. You can change these values, but ensure that you are provisioning enough database resources to handle the changed values. For details on platform limits, see the [Temporal Platform limits sheet](https://docs.temporal.io/self-hosted-guide/defaults) . | Dynamic configuration key | Type | Description | Default value | | --- | --- | --- | --- | | `limit.maxIDLength` | Int | Length limit for various Ids, including: `Namespace`, `TaskQueue`, `WorkflowID`, `ActivityID`, `TimerID`, `WorkflowType`, `ActivityType`, `SignalName`, `MarkerName`, `ErrorReason`/`FailureReason`/`CancelCause`, `Identity`, and `RequestID`. | 1000 | | `limit.blobSize.warn` | Int | Limit, in bytes, for BLOBs size in an Event when a warning is thrown in the server logs. | 512 KB (512 × 1024) | | `limit.blobSize.error` | Int | Limit, in bytes, for BLOBs size in an Event when an error occurs in the transaction. | 2 MB (2 × 1024 × 1024) | | `limit.historySize.warn` | Int | Limit, in bytes, at which a warning is thrown for the Workflow Execution Event History size. | 10 MB (10 × 1024 × 1024) | | `limit.historySize.error` | Int | Limit, in bytes, at which an error occurs in the Workflow Execution for exceeding allowed size. | 50 MB (50 × 1024 × 1024) | | `limit.historyCount.warn` | Int | Limit, in count, at which a warning is thrown for the Workflow Execution Event History size. | 10,240 Events | | `limit.historyCount.error` | Int | Limit, in count, at which an error occurs in the Workflow Execution for exceeding allowed number of Events. | 51,200 events | | `limit.numPendingActivities.error` | Int | Maximum number of pending Activities that a Workflow Execution can have before the `ScheduleActivityTask` fails with an error. | 2000 | | `limit.numPendingSignals.error` | Int | Maximum number of pending Signals that a Workflow Execution can have before the `SignalExternalWorkflowExecution` commands from this Workflow fail with an error. | 2000 | | `history.maximumSignalsPerExecution` | Int | Maximum number of Signals that a Workflow Execution can receive before it throws an `Invalid Argument` error. | 10000 | | `limit.numPendingCancelRequests.error` | Int | Maximum number of pending requests to cancel other Workflows that a Workflow Execution can have before the `RequestCancelExternalWorkflowExecution` commands fail with an error. | 2000 | | `limit.numPendingChildExecutions.error` | Int | Maximum number of pending Child Workflows that a Workflow Execution can have before the `StartChildWorkflowExecution` commands fail with an error. | 2000 | | `frontend.visibilityMaxPageSize` | Int | Maximum number of Workflow Executions shown from the ListWorkflowExecutions API in one page. | 1000 | ### Secondary visibility settings[​](https://docs.temporal.io/references/dynamic-configuration#secondary-visibility-settings "Direct link to Secondary visibility settings") Secondary visibility configuration keys enable Dual Visibility on your Temporal Cluster. This can be useful when migrating a Visibility database or creating a backup Visibility store. | Dynamic configuration key | Type | Description | Default value | | --- | --- | --- | --- | | `system.enableReadFromSecondaryVisibility` | Boolean | Enables reading from the [secondary visibility store](https://docs.temporal.io/dual-visibility)
, and can be set per Namespace. Allowed values are `true` or `false`. | `false` | | `system.secondaryVisibilityWritingMode` | | Enables writing Visibility data to the secondary Visibility store and can be set per Namespace. Setting this value to `on` disables write operations to the primary Visibility store. Allowed values:
`off`: Enables writing to primary Visibility store only.
`on`: Enables writing to secondary Visibility store only.
`dual`: Enables writing to both primary and secondary Visibility stores. | `off` | ### Server version check settings[​](https://docs.temporal.io/references/dynamic-configuration#server-version-check-settings "Direct link to Server version check settings") The Temporal server reports the server version and the version of the SDK that it is connected to in order to determine if the Web UI should show a banner that states a new version is available to install. This can be disabled by defining the following value or by setting the `TEMPORAL_VERSION_CHECK_DISABLED` environment variable to `1`. | Dynamic configuration key | Type | Description | Default value | | --- | --- | --- | --- | | `frontend.enableServerVersionCheck` | Boolean | Enables the Temporal server to report version information about the current server and SDK. Allowed values are `true` or `false`. | `true` | ### Nexus settings[​](https://docs.temporal.io/references/dynamic-configuration#nexus-settings "Direct link to Nexus settings") Settings related to the management of Nexus. For more detailed explanation of these configs see the [in-repo Nexus architecture doc](https://github.com/temporalio/temporal/blob/6cb70c0785fd7ba4574ed91d772fe17dff4981b4/docs/architecture/nexus.md) . | Dynamic configuration key | Type | Description | Default value | | --- | --- | --- | --- | | `system.enableNexus` | Boolean | Enables Nexus Features. Removed in 1.31.0 (Nexus is always enabled). | `true` (since 1.27) | | `component.nexusoperations.useSystemCallbackURL` | String | Controls whether to use temporal://system as the callback URL generated by the server. (Server 1.30 or newer) | `false` | | `component.nexusoperations.callback.endpoint.template` | String | Defines the URL template used to construct Nexus callback URLs. (Only required for the experimental external endpoint target feature or servers older than 1.30.) | unset | | `component.callbacks.allowedAddresses` | Object | Defines the security allow-list of callback URL patterns that the server will accept; used to restrict what callback endpoints can be invoked. (Only required for the experimental external endpoint target feature or servers older than 1.30.) | unset | * [Format](https://docs.temporal.io/references/dynamic-configuration#format) * [Constraints](https://docs.temporal.io/references/dynamic-configuration#constraints) * [Commonly used dynamic configuration keys](https://docs.temporal.io/references/dynamic-configuration#commonly-used-dynamic-configuration-keys) * [Service-level RPS limits](https://docs.temporal.io/references/dynamic-configuration#service-level-rps-limits) * [QPS limits for Persistence store](https://docs.temporal.io/references/dynamic-configuration#qps-limits-for-persistence-store) * [Activity and Workflow default policy setting](https://docs.temporal.io/references/dynamic-configuration#activity-and-workflow-default-policy-setting) * [Size limit settings](https://docs.temporal.io/references/dynamic-configuration#size-limit-settings) * [Secondary visibility settings](https://docs.temporal.io/references/dynamic-configuration#secondary-visibility-settings) * [Server version check settings](https://docs.temporal.io/references/dynamic-configuration#server-version-check-settings) * [Nexus settings](https://docs.temporal.io/references/dynamic-configuration#nexus-settings) --- # Parent Close Policy | Temporal Platform Documentation [Skip to main content](https://docs.temporal.io/parent-close-policy#__docusaurus_skipToContent_fallback) On this page This page discusses [Parent Close Policy](https://docs.temporal.io/parent-close-policy#parent-close-policy) . What is a Parent Close Policy?[​](https://docs.temporal.io/parent-close-policy#parent-close-policy "Direct link to What is a Parent Close Policy?") ---------------------------------------------------------------------------------------------------------------------------------------------------- A Parent Close Policy determines what happens to a Child Workflow Execution if its Parent changes to a Closed status (Completed, Failed, or Timed out). * [How to set a Parent Close Policy using the Go SDK](https://docs.temporal.io/develop/go/workflows/child-workflows#parent-close-policy) * [How to set a Parent Close Policy using the Java SDK](https://docs.temporal.io/develop/java/workflows/child-workflows#parent-close-policy) * [How to set a Parent Close Policy using the PHP SDK](https://docs.temporal.io/develop/php/workflows/child-workflows#parent-close-policy) * [How to set a Parent Close Policy using the Python SDK](https://docs.temporal.io/develop/python/workflows/child-workflows#parent-close-policy) * [How to set a Parent Close Policy using the TypeScript SDK](https://docs.temporal.io/develop/typescript/workflows/child-workflows#parent-close-policy) * [How to set a Parent Close Policy using the .NET SDK](https://docs.temporal.io/develop/dotnet/workflows/child-workflows#parent-close-policy) There are three possible values: * **Abandon:** the Child Workflow Execution is not affected. * **Request Cancel:** a Cancellation request is sent to the Child Workflow Execution. * **Terminate** (default): the Child Workflow Execution is forcefully Terminated. [`ParentClosePolicy`](https://github.com/temporalio/api/blob/c1f04d0856a3ba2995e92717607f83536b5a44f5/temporal/api/enums/v1/workflow.proto#L44) proto definition. Each Child Workflow Execution may have its own Parent Close Policy. This policy applies only to Child Workflow Executions and has no effect otherwise. ![Parent Close Policy entity relationship](https://docs.temporal.io/diagrams/parent-close-policy.svg) Parent Close Policy entity relationship You can set policies per child, which means you can opt out of propagating terminates / cancels on a per-child basis. This is useful for starting Child Workflows asynchronously (see [relevant issue here](https://community.temporal.io/t/best-way-to-create-an-async-child-workflow/114) or the corresponding SDK docs). * [What is a Parent Close Policy?](https://docs.temporal.io/parent-close-policy#parent-close-policy) --- # Self-hosted Temporal Service guide | Temporal Platform Documentation [Skip to main content](https://docs.temporal.io/self-hosted-guide#__docusaurus_skipToContent_fallback) On this page Welcome to the self-hosted Temporal Service guide. This guide shows you how to self-host open source infrastructure software that orchestrates your durable applications. Do you need a production Temporal Service? If you're still developing and testing your application locally, you may not need a production Temporal Service. Use the [Temporal CLI development server](https://docs.temporal.io/cli/server#start-dev) — a single binary with no external dependencies: `temporal server start-dev` This starts a complete Temporal Service with Web UI on your local machine. We recommend this for local development regardless of whether you plan to use Temporal Cloud or self-host in production. See the [Temporal CLI server](https://docs.temporal.io/cli/server) page for configuration options. Plan and deploy your service[​](https://docs.temporal.io/self-hosted-guide#plan-and-deploy-your-service "Direct link to Plan and deploy your service") ------------------------------------------------------------------------------------------------------------------------------------------------------- * [Deployment](https://docs.temporal.io/self-hosted-guide/deployment) : Choose a deployment approach (Docker, Kubernetes, or manual) and set up a production-ready Temporal Service. * [Embedded server](https://docs.temporal.io/self-hosted-guide/embedded-server) : Run Temporal in-process as a Go library for local development and testing scenarios. * [Defaults](https://docs.temporal.io/self-hosted-guide/defaults) : Review platform limits and default settings that can affect Workflow and Activity behavior. * [Production checklist](https://docs.temporal.io/self-hosted-guide/production-checklist) : Validate readiness for scale, reliability, operations, and long-term maintainability. Operate your self-hosted service[​](https://docs.temporal.io/self-hosted-guide#operate-your-self-hosted-service "Direct link to Operate your self-hosted service") ------------------------------------------------------------------------------------------------------------------------------------------------------------------- * [Namespaces](https://docs.temporal.io/self-hosted-guide/namespaces) : Create and manage Namespace isolation, retention, and related configuration. * [Security](https://docs.temporal.io/self-hosted-guide/security) : Configure TLS/mTLS, authentication, authorization, and related hardening controls. * [Monitoring](https://docs.temporal.io/self-hosted-guide/monitoring) : Collect and visualize service and SDK metrics to troubleshoot and track health. * [Visibility](https://docs.temporal.io/self-hosted-guide/visibility) : Configure Visibility storage so you can list, filter, and search Workflow Executions. * [Upgrading server](https://docs.temporal.io/self-hosted-guide/upgrade-server#upgrade-server) : Perform safe, sequential server and schema upgrades. Protect data and enable advanced features[​](https://docs.temporal.io/self-hosted-guide#protect-data-and-enable-advanced-features "Direct link to Protect data and enable advanced features") ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- * [Data encryption](https://docs.temporal.io/production-deployment/data-encryption) : Use Payload Codecs and Codec Server patterns to protect sensitive Workflow data. * [Archival](https://docs.temporal.io/self-hosted-guide/archival) : Move closed Event Histories and Visibility records to blob storage for longer retention. * [Multi-Cluster Replication](https://docs.temporal.io/self-hosted-guide/multi-cluster-replication) : Replicate Workflow state across clusters for failover and disaster recovery. * [Temporal Nexus](https://docs.temporal.io/production-deployment/self-hosted-guide/nexus) : Enable Nexus in self-hosted environments to connect Temporal Applications across boundaries. * [Plan and deploy your service](https://docs.temporal.io/self-hosted-guide#plan-and-deploy-your-service) * [Operate your self-hosted service](https://docs.temporal.io/self-hosted-guide#operate-your-self-hosted-service) * [Protect data and enable advanced features](https://docs.temporal.io/self-hosted-guide#protect-data-and-enable-advanced-features) --- # Temporal Failures reference | Temporal Platform Documentation [Skip to main content](https://docs.temporal.io/references/failures#__docusaurus_skipToContent_fallback) On this page A Failure is Temporal's representation of various types of errors that occur in the system. There are different types of Failures, and each has a different type in the SDKs and different information in the protobuf messages (which are used to communicate with the Temporal Service and appear in [Event History](https://docs.temporal.io/workflow-execution/event#event-history) ). Temporal Failure[​](https://docs.temporal.io/references/failures#temporal-failure "Direct link to Temporal Failure") --------------------------------------------------------------------------------------------------------------------- Most SDKs have a base class that the other Failures extend: * TypeScript: [TemporalFailure](https://typescript.temporal.io/api/classes/common.TemporalFailure) * Java: [TemporalFailure](https://www.javadoc.io/doc/io.temporal/temporal-sdk/latest/io/temporal/failure/TemporalFailure.html) * Python: [FailureError](https://python.temporal.io/temporalio.exceptions.FailureError.html) * PHP: [TemporalFailure](https://php.temporal.io/classes/Temporal-Exception-Failure-TemporalFailure.html) The base [Failure proto message](https://api-docs.temporal.io/#temporal.api.failure.v1.Failure) has these fields: * `string message` * `string stack_trace` * `string source`: The SDK this Failure originated in (for example, `"TypeScriptSDK"`). In some SDKs, this field is used to rehydrate the call stack into an exception object. * `Failure cause`: The `Failure` message of the cause of this Failure (if applicable). * `Payload encoded_attributes`: Contains the encoded `message` and `stack_trace` fields when using a [Failure Converter](https://docs.temporal.io/failure-converter) . Application Failure[​](https://docs.temporal.io/references/failures#application-failure "Direct link to Application Failure") ------------------------------------------------------------------------------------------------------------------------------ Workflow, and Activity, and Nexus Operation code use Application Failures to communicate application-specific failures that happen. This is the only type of Temporal Failure created and thrown by user code. * TypeScript: [ApplicationFailure](https://typescript.temporal.io/api/classes/common.ApplicationFailure) * Java: [ApplicationFailure](https://www.javadoc.io/doc/io.temporal/temporal-sdk/latest/io/temporal/failure/ApplicationFailure.html) * Go: [ApplicationError](https://pkg.go.dev/go.temporal.io/sdk/temporal#ApplicationError) * Python: [ApplicationError](https://python.temporal.io/temporalio.exceptions.ApplicationError.html) * PHP: [ApplicationFailure](https://php.temporal.io/classes/Temporal-Exception-Failure-ApplicationFailure.html) * Proto: [ApplicationFailureInfo](https://api-docs.temporal.io/#temporal.api.failure.v1.ApplicationFailureInfo) and [Failure](https://api-docs.temporal.io/#temporal.api.failure.v1.Failure) ### Errors in Workflows[​](https://docs.temporal.io/references/failures#errors-in-workflows "Direct link to Errors in Workflows") An error in a Workflow can cause either a **Workflow Task Failure** (the Task will be retried) or a **Workflow Execution Failure** (the Workflow is marked as failed). Only Workflow exceptions that are Temporal Failures cause the Workflow Execution to fail; all other exceptions cause the Workflow Task to fail and be retried (in Go, any error returned from the Workflow fails the Workflow Execution, and a panic fails the Workflow Task). Most types of Temporal Failures are raised by the Temporal Service, like a [Cancelled Failure](https://docs.temporal.io/references/failures#cancelled-failure) when the Workflow is Cancelled or an [Activity Failure](https://docs.temporal.io/references/failures#activity-failure) when an Activity fails. In contrast, you can explicitly fail the Workflow Execution by throwing an Application Failure (returning any error in Go) in Workflow Definition code. #### Workflow Task Failures[​](https://docs.temporal.io/references/failures#workflow-task-failures "Direct link to Workflow Task Failures") A **Workflow Task Failure** is an unexpected situation failing to process a Workflow Task. This could be triggered by a non-Temporal exception being raised (panicking in Go) in your Workflow code. Any exception that does not extend Temporal's `FailureError` exception is considered a Workflow Task Failure. These types of failures will cause the Workflow Task to be retried until the Workflow Execution Timeout, which is unlimited by default. #### Workflow Execution Failures[​](https://docs.temporal.io/references/failures#workflow-execution-failures "Direct link to Workflow Execution Failures") An `ApplicationError`, an extension of `FailureError`, can be raised in a Workflow to fail the Workflow Execution. Workflow Execution Failures put the Workflow Execution into the "Failed" state and no more attempts will be made in progressing this execution. If you are creating custom exceptions you would need to extend the [`ApplicationError`](https://docs.temporal.io/references/failures#application-failure) class—a child class of [`FailureError`](https://docs.temporal.io/references/failures#temporal-failure) . ### Errors in Activities[​](https://docs.temporal.io/references/failures#errors-in-activities "Direct link to Errors in Activities") In Activities, you can either throw an Application Failure or another Error to fail the Activity Task. In the latter case, the error is converted to an Application Failure. During conversion, the following Application Failure fields are set: * `type` is set to the error's type name. * `message` is set to the error message. * `non_retryable` is set to false. * `details` are left unset. * `cause` is a Failure converted from the error's `cause` property. * `next_retry_delay` is left unset. * call stack is copied. When an [Activity Execution](https://docs.temporal.io/activity-execution) fails, the Application Failure from the last Activity Task is the `cause` field of the [ActivityFailure](https://docs.temporal.io/references/failures#activity-failure) . This ActivityFailure is thrown by the Workflow's call to the Activity, and it can be handled in the Workflow Definition. ### Errors in Nexus Operations[​](https://docs.temporal.io/references/failures#errors-in-nexus-operations "Direct link to Errors in Nexus Operations") Nexus Operations can end up in completed, failed, canceled, and timed out states. Under the hood, the Nexus Operation machinery breaks up the lifecycle of an Operation into one or more StartOperation requests and completion callbacks, and automatically retries these requests as long they fail with retryable errors. The Workflow-specified schedule-to-close timeout is enforced by the caller's machinery and is the only way for an Operation to transition to the timed out state. Operations can end up in the other three states either when the operation handler returns a synchronous response or error, or when an asynchronous Operation (for example, one backed by a workflow) eventually reaches a terminal state. A Nexus Operation handler can return either retryable or non-retryable errors to indicate to the caller's Nexus machinery whether to retry a given request. Requests that time out before a response is sent to the caller are automatically retried. By default, errors are considered retryable, unless specified below: * Non retryable Application Failures * Unsuccessful Operation errors that can resolve an operation as either failed or canceled * [Handler errors](https://github.com/nexus-rpc/api/blob/main/SPEC.md#predefined-handler-errors) with the following types: `BAD_REQUEST`, `UNAUTHENTICATED`, `UNAUTHORIZED`, `NOT_FOUND`, and `RESOURCE_EXHAUSTED` #### Nexus Operation Task Failures[​](https://docs.temporal.io/references/failures#nexus-operation-task-failures "Direct link to Nexus Operation Task Failures") A Nexus Operation Task Failure is an unexpected situation failing to process a Nexus Operation Task in a handler. This could be triggered by throwing an unknown error in your Nexus handler code. These types of failures will cause the Nexus Operation Task to be retried. #### Nexus Operation Execution Failures[​](https://docs.temporal.io/references/failures#nexus-operation-execution-failures "Direct link to Nexus Operation Execution Failures") A non-retryable Application Failure can be thrown by a Nexus Operation handler to fail the overall Nexus Operation Execution. Nexus Operation Execution Failures put the Nexus Operation Execution into the "Failed" state and no more attempts will be made to complete the Nexus Operation. #### Propagation of Workflow errors[​](https://docs.temporal.io/references/failures#propagation-of-workflow-errors "Direct link to Propagation of Workflow errors") Application Errors thrown from a Workflow created by a Nexus NewWorkflowRunOperation handler will be automatically propagated to the caller as a non-retryable error and result in a Nexus Operation Execution Failure. #### Using Failures in a Nexus handler[​](https://docs.temporal.io/references/failures#using-failures-in-a-nexus-handler "Direct link to Using Failures in a Nexus handler") In a Nexus Operation handler, you can throw an Application Failure, a Nexus Error or another Error to fail the individual Nexus Operation Task or fail the overall Nexus Operation Execution. Unknown errors are converted to a retryable Application Failure. During conversion, the following fields are set on the Application Failure: * `non_retryable` is set to false. * `type` is set to the error's type name. * `message` is set to the error message. #### Retryable failures[​](https://docs.temporal.io/references/failures#retryable-failures "Direct link to Retryable failures") Retryable Nexus Operation Task failures, such as an unknown error, are automatically retried with a built-in Retry Policy. When a Nexus Task fails, the caller Workflow records an event attempt failure on the pending Nexus Operation and sets the following fields: * `state` is set to the new state, for example BackingOff. * `attempt` is set to an incremented count. * `next_attempt_schedule_time` is set when the Nexus Task will be retried. * `last_attempt_failure` is set with the following fields: * `message` is set to the error message. * `failure_info` is set to the Application Failure. For example, an unknown error thrown in a Nexus handler will surface as: temporal workflow describe -w my-workflow-id...Pending Nexus Operations: 1 Endpoint myendpoint Service my-hello-service Operation echo OperationToken State BackingOff Attempt 6 ScheduleToCloseTimeout 0s NextAttemptScheduleTime 20 seconds from now LastAttemptCompleteTime 11 seconds ago LastAttemptFailure {"message":"unexpected response status: "500 Internal Server Error": internal error","applicationFailureInfo":{}} ### Non-retryable[​](https://docs.temporal.io/references/failures#non-retryable "Direct link to Non-retryable") When an Activity or Workflow throws an Application Failure, the Failure's `type` field is matched against a Retry Policy's list of [non-retryable errors](https://docs.temporal.io/encyclopedia/retry-policies#non-retryable-errors) to determine whether to retry the Activity or Workflow. Activities and Workflow can also avoid retrying by setting an Application Failure's `non_retryable` flag to `true`. When a Nexus Operation handler throws an Application Failure, it is retried by default using a built-in Retry Policy that cannot be customized. Nexus Operation handlers can avoid retrying by setting an Application Failure's `non_retryable` flag to true. When a non-retryable error is returned from a Nexus handler, the overall Nexus Operation Execution is failed and the error is returned to the caller’s Workflow Execution as a Nexus Operation Failure. ### Setting the Next Retry Delay[​](https://docs.temporal.io/references/failures#activity-next-retry-delay "Direct link to Setting the Next Retry Delay") By setting the Next Retry Delay for a given Application Failure, you can tell the server to wait that amount of time before trying the Activity or Workflow again. This will override whatever the Retry Policy would have computed for your specific exception. Java: [NextRetryDelay](https://docs.temporal.io/develop/java/activities/timeouts#activity-next-retry-delay) TypeScript: [nextRetryDelay](https://docs.temporal.io/develop/typescript/activities/timeouts#activity-next-retry-delay) PHP: [NextRetryDelay](https://docs.temporal.io/develop/php/activities/timeouts#activity-next-retry-delay) ### Nexus errors[​](https://docs.temporal.io/references/failures#nexus-errors "Direct link to Nexus errors") #### Default mapping[​](https://docs.temporal.io/references/failures#default-mapping "Direct link to Default mapping") By default, Application Failures thrown from a Nexus Operation handler will be mapped to the following underlying Nexus Failures, based on what `non_retryable` is set to: | `non_retryable` | Nexus error | HTTP status code | | --- | --- | --- | | false (default) | HandlerErrorTypeInternal | 500 Internal Server Error | | true | UnsuccessfulOperationError | 424 Failed Dependency | #### Use Nexus Errors directly[​](https://docs.temporal.io/references/failures#use-nexus-errors-directly "Direct link to Use Nexus Errors directly") For improved semantics and mapping to HTTP status codes for external Nexus callers (when supported), we recommend that Nexus Operation handlers throw a Nexus Error directly, which includes the list below with associated retry semantics. For example the Nexus Go SDK provides * `nexus.HandlerError(nexus.HandlerErrorType, msg)` * `nexus.UnsuccessfulOperationError{state, failure}` #### Retryable Nexus errors[​](https://docs.temporal.io/references/failures#retryable-nexus-errors "Direct link to Retryable Nexus errors") | Nexus error type | `non_retryable` | | --- | --- | | HandlerErrorTypeResourceExhausted | false | | HandlerErrorTypeInternal | false | | HandlerErrorTypeNotImplemented | false | | HandlerErrorTypeUnavailable | false | #### Non-retryable Nexus errors[​](https://docs.temporal.io/references/failures#non-retryable-nexus-errors "Direct link to Non-retryable Nexus errors") | Nexus error type | `non_retryable` | | --- | --- | | HandlerErrorTypeBadRequest | true | | HandlerErrorTypeUnauthenticated | true | | HandlerErrorTypeUnauthorized | true | | HandlerErrorTypeNotFound | true | | UnsuccessfulOperationError | true | Cancelled Failure[​](https://docs.temporal.io/references/failures#cancelled-failure "Direct link to Cancelled Failure") ------------------------------------------------------------------------------------------------------------------------ When [Cancellation](https://docs.temporal.io/activity-execution#cancellation) of a Workflow, Activity or Nexus Operation is requested, SDKs represent the cancellation to the user in language-specific ways. For example, in TypeScript, in some cases a Cancelled Failure is thrown directly by a Workflow API function, and in other cases the Cancelled Failure is wrapped in a different Failure. To check both types of cases, TypeScript has the [isCancellation](https://typescript.temporal.io/api/namespaces/workflow#iscancellation) helper. When a Workflow, Activity or Nexus Operation is successfully Cancelled, a Cancelled Failure is the `cause` field of the Activity Failure, Nexus Operation Failure or "Workflow failed" error. * TypeScript: [CancelledFailure](https://typescript.temporal.io/api/classes/common.CancelledFailure) * Java: [CanceledFailure](https://www.javadoc.io/doc/io.temporal/temporal-sdk/latest/io/temporal/failure/CanceledFailure.html) * Go: [CanceledError](https://pkg.go.dev/go.temporal.io/sdk/temporal#CanceledError) * Python: [CancelledError](https://python.temporal.io/temporalio.exceptions.CancelledError.html) * PHP: [CanceledFailure](https://php.temporal.io/classes/Temporal-Exception-Failure-CanceledFailure.html) * Proto: [CanceledFailureInfo](https://api-docs.temporal.io/#temporal.api.failure.v1.CanceledFailureInfo) and [Failure](https://api-docs.temporal.io/#temporal.api.failure.v1.Failure) Activity Failure[​](https://docs.temporal.io/references/failures#activity-failure "Direct link to Activity Failure") --------------------------------------------------------------------------------------------------------------------- An Activity Failure is delivered to the Workflow Execution when an Activity fails. It contains information about the failure and the Activity Execution; for example, the Activity Type and Activity Id. The reason for the failure is in the `cause` field. For example, if an Activity Execution times out, the `cause` is a [Timeout Failure](https://docs.temporal.io/references/failures#timeout-failure) . * TypeScript: [ActivityFailure](https://typescript.temporal.io/api/classes/common.ActivityFailure) * Java: [ActivityFailure](https://www.javadoc.io/doc/io.temporal/temporal-sdk/latest/io/temporal/failure/ActivityFailure.html) * Go: [ActivityError](https://pkg.go.dev/go.temporal.io/sdk/temporal#ActivityError) * Python: [ActivityError](https://python.temporal.io/temporalio.exceptions.ActivityError.html) * PHP: [ActivityFailure](https://php.temporal.io/classes/Temporal-Exception-Failure-ActivityFailure.html) * Proto: [ActivityFailureInfo](https://api-docs.temporal.io/#temporal.api.failure.v1.ActivityFailureInfo) and [Failure](https://api-docs.temporal.io/#temporal.api.failure.v1.Failure) Nexus Operation Failure[​](https://docs.temporal.io/references/failures#nexus-operation-failure "Direct link to Nexus Operation Failure") ------------------------------------------------------------------------------------------------------------------------------------------ A Nexus Operation Failure is delivered to the Workflow Execution when a Nexus Operation fails. It contains information about the failure and the Nexus Operation Execution; for example, the Nexus Operation name and Nexus Operation token. The reason for the failure is in the message and cause (typically an Application Error or a Canceled Error). * Go: NexusOperationError * Proto: NexusOperationFailureInfo A Nexus Operation Failure includes the following fields: * Endpoint is set to the name of the endpoint. * Service is set to the name of the service. * Operation is set to the name of the operation. * Operation\_token is set if this is an async operation, which can be used to perform additional actions like cancelling the operation. * Scheduled\_event\_id is set to the caller’s event id that scheduled the operation. * Message is set to a generic unsuccessful error message. * Cause is set to the underlying Application Failure with the following fields: * Non-retryable is set to true. * Type is set to the error's type name. * Message is set to the error message. * Nexus\_error\_code is set to the underlying Nexus error code. Child Workflow Failure[​](https://docs.temporal.io/references/failures#child-workflow-failure "Direct link to Child Workflow Failure") --------------------------------------------------------------------------------------------------------------------------------------- A Child Workflow Failure is delivered to the Workflow Execution when a Child Workflow Execution fails. It contains information about the failure and the Child Workflow Execution; for example, the Workflow Type and Workflow Id. The reason for the failure is in the `cause` field. * TypeScript: [ChildWorkflowFailure](https://typescript.temporal.io/api/classes/common.ChildWorkflowFailure) * Java: [ChildWorkflowFailure](https://www.javadoc.io/doc/io.temporal/temporal-sdk/latest/io/temporal/failure/ChildWorkflowFailure.html) * Go: [ChildWorkflowExecutionError](https://pkg.go.dev/go.temporal.io/sdk/temporal#ChildWorkflowExecutionError) * Python: [ChildWorkflowError](https://python.temporal.io/temporalio.exceptions.ChildWorkflowError.html) * PHP: [ChildWorkflowFailure](https://php.temporal.io/classes/Temporal-Exception-Failure-ChildWorkflowFailure.html) * Proto: [ChildWorkflowExecutionFailureInfo](https://api-docs.temporal.io/#temporal.api.failure.v1.ChildWorkflowExecutionFailureInfo) and [Failure](https://api-docs.temporal.io/#temporal.api.failure.v1.Failure) Timeout Failure[​](https://docs.temporal.io/references/failures#timeout-failure "Direct link to Timeout Failure") ------------------------------------------------------------------------------------------------------------------ A Timeout Failure represents the timeout of an Activity or Workflow. When an Activity times out, the last Heartbeat details it emitted is attached. * TypeScript: [TimeoutFailure](https://typescript.temporal.io/api/classes/common.TimeoutFailure) * Java: [TimeoutFailure](https://www.javadoc.io/doc/io.temporal/temporal-sdk/latest/io/temporal/failure/TimeoutFailure.html) * Go: [TimeoutError](https://pkg.go.dev/go.temporal.io/sdk/temporal#TimeoutError) * Python: [TimeoutError](https://python.temporal.io/temporalio.exceptions.TimeoutError.html) * PHP: [TimeoutFailure](https://php.temporal.io/classes/Temporal-Exception-Failure-TimeoutFailure.html) * Proto: [TimeoutFailureInfo](https://api-docs.temporal.io/#temporal.api.failure.v1.TimeoutFailureInfo) and [Failure](https://api-docs.temporal.io/#temporal.api.failure.v1.Failure) Terminated Failure[​](https://docs.temporal.io/references/failures#terminated-failure "Direct link to Terminated Failure") --------------------------------------------------------------------------------------------------------------------------- A Terminated Failure is used as the `cause` of an error when a Workflow is terminated, and you receive the error in one of the following locations: * Inside a Workflow that's waiting for the result of a Child Workflow. * When waiting for the result of a Workflow on the Client. In the SDKs: * TypeScript: [TerminatedFailure](https://typescript.temporal.io/api/classes/common.TerminatedFailure) * Java: [TerminatedFailure](https://www.javadoc.io/doc/io.temporal/temporal-sdk/latest/io/temporal/failure/TerminatedFailure.html) * Go: [TerminatedError](https://pkg.go.dev/go.temporal.io/sdk/temporal#TerminatedError) * Python: [TerminatedError](https://python.temporal.io/temporalio.exceptions.TerminatedError.html) * PHP: [TerminatedFailure](https://php.temporal.io/classes/Temporal-Exception-Failure-TerminatedFailure.html) * Proto: [TerminatedFailureInfo](https://api-docs.temporal.io/#temporal.api.failure.v1.TerminatedFailureInfo) and [Failure](https://api-docs.temporal.io/#temporal.api.failure.v1.Failure) Server Failure[​](https://docs.temporal.io/references/failures#server-failure "Direct link to Server Failure") --------------------------------------------------------------------------------------------------------------- A Server Failure is used for errors that originate in the Temporal Service. * TypeScript: [ServerFailure](https://typescript.temporal.io/api/classes/common.ServerFailure) * Java: [ServerFailure](https://www.javadoc.io/doc/io.temporal/temporal-sdk/latest/io/temporal/failure/ServerFailure.html) * Go: [ServerError](https://pkg.go.dev/go.temporal.io/sdk/temporal#ServerError) * Python: [ServerError](https://python.temporal.io/temporalio.exceptions.ServerError.html) * PHP: [ServerFailure](https://php.temporal.io/classes/Temporal-Exception-Failure-ServerFailure.html) * Proto: [ServerFailureInfo](https://api-docs.temporal.io/#temporal.api.failure.v1.ServerFailureInfo) and [Failure](https://api-docs.temporal.io/#temporal.api.failure.v1.Failure) * [Temporal Failure](https://docs.temporal.io/references/failures#temporal-failure) * [Application Failure](https://docs.temporal.io/references/failures#application-failure) * [Errors in Workflows](https://docs.temporal.io/references/failures#errors-in-workflows) * [Workflow Task Failures](https://docs.temporal.io/references/failures#workflow-task-failures) * [Workflow Execution Failures](https://docs.temporal.io/references/failures#workflow-execution-failures) * [Errors in Activities](https://docs.temporal.io/references/failures#errors-in-activities) * [Errors in Nexus Operations](https://docs.temporal.io/references/failures#errors-in-nexus-operations) * [Nexus Operation Task Failures](https://docs.temporal.io/references/failures#nexus-operation-task-failures) * [Nexus Operation Execution Failures](https://docs.temporal.io/references/failures#nexus-operation-execution-failures) * [Propagation of Workflow errors](https://docs.temporal.io/references/failures#propagation-of-workflow-errors) * [Using Failures in a Nexus handler](https://docs.temporal.io/references/failures#using-failures-in-a-nexus-handler) * [Retryable failures](https://docs.temporal.io/references/failures#retryable-failures) * [Non-retryable](https://docs.temporal.io/references/failures#non-retryable) * [Setting the Next Retry Delay](https://docs.temporal.io/references/failures#activity-next-retry-delay) * [Nexus errors](https://docs.temporal.io/references/failures#nexus-errors) * [Default mapping](https://docs.temporal.io/references/failures#default-mapping) * [Use Nexus Errors directly](https://docs.temporal.io/references/failures#use-nexus-errors-directly) * [Retryable Nexus errors](https://docs.temporal.io/references/failures#retryable-nexus-errors) * [Non-retryable Nexus errors](https://docs.temporal.io/references/failures#non-retryable-nexus-errors) * [Cancelled Failure](https://docs.temporal.io/references/failures#cancelled-failure) * [Activity Failure](https://docs.temporal.io/references/failures#activity-failure) * [Nexus Operation Failure](https://docs.temporal.io/references/failures#nexus-operation-failure) * [Child Workflow Failure](https://docs.temporal.io/references/failures#child-workflow-failure) * [Timeout Failure](https://docs.temporal.io/references/failures#timeout-failure) * [Terminated Failure](https://docs.temporal.io/references/failures#terminated-failure) * [Server Failure](https://docs.temporal.io/references/failures#server-failure) --- # OSS Temporal Service metrics reference | Temporal Platform Documentation [Skip to main content](https://docs.temporal.io/references/cluster-metrics#__docusaurus_skipToContent_fallback) On this page OSS Temporal Service metrics The information on this page is relevant to open source [Temporal Service deployments](https://docs.temporal.io/temporal-service) . See [Cloud metrics](https://docs.temporal.io/cloud/metrics/) for metrics emitted by [Temporal Cloud](https://docs.temporal.io/cloud/overview) . See [SDK metrics](https://docs.temporal.io/references/sdk-metrics) for metrics emitted by the [SDKs](https://docs.temporal.io/encyclopedia/temporal-sdks) . A Temporal Service emits a range of metrics to help operators get visibility into the Temporal Service's performance and to set up alerts. All metrics emitted by the Temporal Service are listed in [metric\_defs.go](https://github.com/temporalio/temporal/blob/main/common/metrics/metric_defs.go) . For details on setting up metrics in your Temporal Service configuration, see the [Temporal Service configuration reference](https://docs.temporal.io/references/configuration#global) . The [dashboards repository](https://github.com/temporalio/dashboards) contains community-driven Grafana dashboard templates that can be used as a starting point for monitoring the Temporal Service and SDK metrics. You can use these templates as references to build your own dashboards. For any metrics that are missing in the dashboards, use [metric\_defs.go](https://github.com/temporalio/temporal/blob/main/common/metrics/metric_defs.go) as a reference. Note that, apart from these metrics emitted by the Temporal Service, you should also monitor infrastructure-specific metrics like CPU, memory, and network for all hosts that are running Temporal Service services. Common metrics[​](https://docs.temporal.io/references/cluster-metrics#common-metrics "Direct link to Common metrics") ---------------------------------------------------------------------------------------------------------------------- Temporal emits metrics for each gRPC service request. These metrics are emitted with `type`, `operation`, and `namespace` tags, which provide visibility into Service usage and show the request rates across Services, Namespaces, and Operations. * Use the `operation` tag in your query to get request rates, error rates, or latencies per operation. * Use the `service_name` tag with the [service role tag values](https://github.com/temporalio/temporal/blob/bba148cf1e1642fd39fa0174423b183d5fc62d95/common/metrics/defs.go#L108) to get details for the specific service. All common tags that you can add in your query are defined in the [metric\_defs.go](https://github.com/temporalio/temporal/blob/main/common/metrics/metric_defs.go) file. For example, to see service requests by operation on the Frontend Service, use the following: `sum by (operation) (rate(service_requests{service_name="frontend"}[2m]))` Note: All metrics queries in this topic are [Prometheus queries](https://prometheus.io/docs/prometheus/latest/querying/basics/) . The following list describes some metrics you can get started with. ### `service_requests`[​](https://docs.temporal.io/references/cluster-metrics#service_requests "Direct link to service_requests") Shows service requests received per Task Queue. Example: Service requests by operation `sum(rate(service_requests{operation=\"AddWorkflowTask\"}[2m]))` ### `service_latency`[​](https://docs.temporal.io/references/cluster-metrics#service_latency "Direct link to service_latency") Shows latencies for all Client request operations. Usually these are the starting point to investigate which operation is experiencing high-latency issues. Example: P95 service latency by operation for the Frontend Service `histogram_quantile(0.95, sum(rate(service_latency_bucket{service_name="frontend"}[5m])) by (operation, le))` ### `service_error_with_type`[​](https://docs.temporal.io/references/cluster-metrics#service_error_with_type "Direct link to service_error_with_type") (Available only in v1.17.0+) Identifies errors encountered by the service. Example: Service errors by type for the Frontend Service `sum(rate(service_error_with_type{service_name="frontend"}[5m])) by (error_type)` ### `client_errors`[​](https://docs.temporal.io/references/cluster-metrics#client_errors "Direct link to client_errors") An indicator for connection issues between different Server roles. Example: Client errors `sum(rate(client_errors{service_name="frontend",service_role="history"}[5m]))` In addition to these, you can define some service-specific metrics to get performance details for each service. Start with the following list, and use [metric\_defs.go](https://github.com/temporalio/temporal/blob/main/common/metrics/metric_defs.go) to define additional metrics as required. Matching Service metrics[​](https://docs.temporal.io/references/cluster-metrics#matching-service-metrics "Direct link to Matching Service metrics") ---------------------------------------------------------------------------------------------------------------------------------------------------- ### `poll_success`[​](https://docs.temporal.io/references/cluster-metrics#poll_success "Direct link to poll_success") Shows for Tasks that are successfully matched to a poller. Example: `sum(rate(poll_success{}[5m]))` ### `poll_timeouts`[​](https://docs.temporal.io/references/cluster-metrics#poll_timeouts "Direct link to poll_timeouts") Shows when no Tasks are available for the poller within the poll timeout. Example: `sum(rate(poll_timeouts{}[5m]))` ### `asyncmatch_latency`[​](https://docs.temporal.io/references/cluster-metrics#asyncmatch_latency "Direct link to asyncmatch_latency") Measures the time from creation to delivery for async matched Tasks. The larger this latency, the longer Tasks are sitting in the queue waiting for your Workers to pick them up. Example: `histogram_quantile(0.95, sum(rate(asyncmatch_latency_bucket{service_name="matching"}[5m])) by (operation, le))` ### `no_poller_tasks`[​](https://docs.temporal.io/references/cluster-metrics#no_poller_tasks "Direct link to no_poller_tasks") Emitted whenever a task is added to a task queue that has no poller, and is a counter metric. This is usually an indicator that either the Worker or the starter programs are using the wrong Task Queue. History Service metrics[​](https://docs.temporal.io/references/cluster-metrics#history-service-metrics "Direct link to History Service metrics") ------------------------------------------------------------------------------------------------------------------------------------------------- A History Task is an internal Task in Temporal that is created as part of a transaction to update Workflow state and is processed by the Temporal History service. It is critical to ensure that the History Task processing system is healthy. The following key metrics can be used to monitor the History Service health: ### `task_requests`[​](https://docs.temporal.io/references/cluster-metrics#task_requests "Direct link to task_requests") Emitted on every Task process request. Example: `sum(rate(task_requests{operation=~"TransferActive.*"}[1m]))` ### `task_errors`[​](https://docs.temporal.io/references/cluster-metrics#task_errors "Direct link to task_errors") Emitted on every Task process error. Example: `sum(rate(task_errors{operation=~"TransferActive.*"}[1m]))` ### `task_attempt`[​](https://docs.temporal.io/references/cluster-metrics#task_attempt "Direct link to task_attempt") Number of attempts on each Task Execution. A Task is retried forever, and each retry increases the attempt count. Example: `histogram_quantile(0.95, sum(rate(task_attempt_bucket{operation=~"TransferActive.*"}[1m])) by (operation, le))` ### `task_latency_processing`[​](https://docs.temporal.io/references/cluster-metrics#task_latency_processing "Direct link to task_latency_processing") Shows the processing latency per attempt. Example: `histogram_quantile(0.95, sum(rate(task_latency_processing_bucket{operation=~"TransferActive.*",service_name="history"}[1m])) by (operation, le))` ### `task_latency`[​](https://docs.temporal.io/references/cluster-metrics#task_latency "Direct link to task_latency") Measures the in-memory latency across multiple attempts. ### `task_latency_queue`[​](https://docs.temporal.io/references/cluster-metrics#task_latency_queue "Direct link to task_latency_queue") Measures the duration, end-to-end, from when the Task should be executed (from the time it was fired) to when the Task is done. ### `task_latency_load`[​](https://docs.temporal.io/references/cluster-metrics#task_latency_load "Direct link to task_latency_load") (Available only in v1.18.0+) Measures the duration from Task generation to Task loading (Task schedule to start latency for persistence queue). ### `task_latency_schedule`[​](https://docs.temporal.io/references/cluster-metrics#task_latency_schedule "Direct link to task_latency_schedule") (Available only in v1.18.0+) Measures the duration from Task submission (to the Task scheduler) to processing (Task schedule to start latency for in-memory queue). ### `queue_latency_schedule`[​](https://docs.temporal.io/references/cluster-metrics#queue_latency_schedule "Direct link to queue_latency_schedule") (Available only in v1.18.0+) Measures the time to schedule 100 Tasks in one Task channel in the host-level Task scheduler. If fewer than 100 Tasks are in the Task channel for 30 seconds, the latency is scaled to 100 Tasks upon emission. Note: This is still an experimental metric and is subject to change. ### `service_latency_userlatency`[​](https://docs.temporal.io/references/cluster-metrics#service_latency_userlatency "Direct link to service_latency_userlatency") Shows the latency introduced because of Workflow logic. For example, if you have one Workflow scheduling many Activities or Child Workflows at the same time, it can cause a per-Workflow lock contention. The wait period for the per-Workflow lock is counted as `userlatency`. The `operation` tag contains details about Task type and Active versus Standby statuses, and can be used to get request rates, error rates, or latencies per operation, which can help identify issues caused by database problems. Persistence metrics[​](https://docs.temporal.io/references/cluster-metrics#persistence-metrics "Direct link to Persistence metrics") ------------------------------------------------------------------------------------------------------------------------------------- Temporal Server emits metrics for every persistence database read and write. Some of the most important ones are the following: ### `persistence_requests`[​](https://docs.temporal.io/references/cluster-metrics#persistence_requests "Direct link to persistence_requests") Emitted on every persistence request. Examples: * Prometheus query for getting the total number of persistence requests by operation for the History Service: `sum by (operation) (rate(persistence_requests{service_name="history"}[1m]))` * Prometheus query for getting the total number of persistence requests by operation for the Matching Service: `sum by (operation) (rate(persistence_requests{service_name="matching"}[1m]))` ### `persistence_errors`[​](https://docs.temporal.io/references/cluster-metrics#persistence_errors "Direct link to persistence_errors") Shows all persistence errors. This metric is a good indicator for connection issues between the Temporal Service and the persistence store. Example: * Prometheus query for getting all persistence errors by service (history) `sum (rate(persistence_errors{service_name="history"}[1m]))` ### `persistence_error_with_type`[​](https://docs.temporal.io/references/cluster-metrics#persistence_error_with_type "Direct link to persistence_error_with_type") Shows all errors related to the persistence store with type, and contain an `error_type` tag. * Prometheus query for getting persistence errors with type by (history) and by error type: `sum(rate(persistence_error_with_type{service_name="history"}[1m])) by (error_type)` ### `persistence_latency`[​](https://docs.temporal.io/references/cluster-metrics#persistence_latency "Direct link to persistence_latency") Shows the latency on persistence operations. Example: * Prometheus query for getting latency by percentile: `histogram_quantile(0.95, sum(rate(persistence_latency_bucket{service_name="history"}[1m])) by (operation, le))` Schedule metrics[​](https://docs.temporal.io/references/cluster-metrics#schedule-metrics "Direct link to Schedule metrics") ---------------------------------------------------------------------------------------------------------------------------- Temporal emits metrics that track the performance and outcomes of these Scheduled Executions. Below are additional metrics that can help you monitor and optimize your Scheduled Workflow Executions. ### `schedule_buffer_overruns`[​](https://docs.temporal.io/references/cluster-metrics#schedule_buffer_overruns "Direct link to schedule_buffer_overruns") Indicates instances where the buffer for holding Scheduled Workflows exceeds its maximum capacity. This scenario typically occurs when schedules with a `buffer_all` overlap policy have their average run length exceeding the average schedule interval. Example: To monitor buffer overruns. `sum(rate(schedule_buffer_overruns{namespace="$namespace"}[5m]))` ### `schedule_missed_catchup_window`[​](https://docs.temporal.io/references/cluster-metrics#schedule_missed_catchup_window "Direct link to schedule_missed_catchup_window") Tracks occurrences when the system fails to execute a Scheduled Action within the defined catchup window. Missed catchup windows can result from extended outages beyond the configured catchup period. Example: To identify missed catchup opportunities. `sum(rate(schedule_missed_catchup_window{namespace="$namespace"}[5m]))` ### `schedule_rate_limited`[​](https://docs.temporal.io/references/cluster-metrics#schedule_rate_limited "Direct link to schedule_rate_limited") Reflects instances where the creation of Workflows by a Schedule is throttled due to rate limiting policies within a Namespace. This metric is crucial for identifying scheduling patterns that frequently hit rate limits, potentially causing missed catchup windows. Example: To assess the impact of rate limiting on Scheduled Executions. `sum(rate(schedule_rate_limited{namespace="$namespace"}[5m]))` ### `schedule_action_success`[​](https://docs.temporal.io/references/cluster-metrics#schedule_action_success "Direct link to schedule_action_success") Measures the successful execution of Workflows as per their schedules or through manual triggers. This metric confirms that Workflows are running as expected without delays or errors. Example: To track the success rate of Scheduled Workflow Executions. `sum(rate(schedule_action_success{namespace="$namespace"}[5m]))` Workflow metrics[​](https://docs.temporal.io/references/cluster-metrics#workflow-metrics "Direct link to Workflow metrics") ---------------------------------------------------------------------------------------------------------------------------- These metrics pertain to Workflow statistics. ### `workflow_cancel`[​](https://docs.temporal.io/references/cluster-metrics#workflow_cancel "Direct link to workflow_cancel") Number of Workflows canceled before completing execution. ### `workflow_continued_as_new`[​](https://docs.temporal.io/references/cluster-metrics#workflow_continued_as_new "Direct link to workflow_continued_as_new") Number of Workflow Executions that were Continued-As-New from a past execution. ### `workflow_failed`[​](https://docs.temporal.io/references/cluster-metrics#workflow_failed "Direct link to workflow_failed") Number of Workflows that failed before completion. ### `workflow_success`[​](https://docs.temporal.io/references/cluster-metrics#workflow_success "Direct link to workflow_success") Number of Workflows that successfully completed. ### `workflow_timeout`[​](https://docs.temporal.io/references/cluster-metrics#workflow_timeout "Direct link to workflow_timeout") Number of Workflows that timed out before completing execution. Nexus metrics[​](https://docs.temporal.io/references/cluster-metrics#nexus-metrics "Direct link to Nexus metrics") ------------------------------------------------------------------------------------------------------------------- These metrics pertain to Nexus Operations. ### Nexus Machinery in the History Service[​](https://docs.temporal.io/references/cluster-metrics#nexus-machinery-in-the-history-service "Direct link to Nexus Machinery in the History Service") See [architecture document](https://github.com/temporalio/temporal/blob/5d55d6c707bd68d8f3274c57ae702331adf05e6e/docs/architecture/nexus.md#scheduler) for more info. #### In-Memory Buffer[​](https://docs.temporal.io/references/cluster-metrics#in-memory-buffer "Direct link to In-Memory Buffer") `dynamic_worker_pool_scheduler_enqueued_tasks`: A counter that is incremented when a task is enqueued to the buffer. `dynamic_worker_pool_scheduler_dequeued_tasks`: A counter that is incremented when a task is dequeued from the buffer. `dynamic_worker_pool_scheduler_rejected_tasks`: A counter that is incremented when the buffer is full and adding the task is rejected. `dynamic_worker_pool_scheduler_buffer_size`: A gauge that periodically samples the size of the buffer. ### Concurrency Limiter[​](https://docs.temporal.io/references/cluster-metrics#concurrency-limiter "Direct link to Concurrency Limiter") `dynamic_worker_pool_scheduler_active_workers`: A gauge that periodically samples the number of running goroutines. #### Rate Limiter[​](https://docs.temporal.io/references/cluster-metrics#rate-limiter "Direct link to Rate Limiter") `rate_limited_task_runnable_wait_time`: A histogram representing the time a task spends waiting for the rate limiter. #### Circuit Breaker[​](https://docs.temporal.io/references/cluster-metrics#circuit-breaker "Direct link to Circuit Breaker") `circuit_breaker_executable_blocked`: A counter that is incremented every time a task execution is blocked by the circuit breaker. #### Task Executors[​](https://docs.temporal.io/references/cluster-metrics#task-executors "Direct link to Task Executors") `nexus_outbound_requests`: A counter representing the number of Nexus outbound requests made by the history service. `nexus_outbound_latency`: A histogram representing the latency of outbound Nexus requests made by the history service. `callback_outbound_requests`: A counter representing the number of callback outbound requests made by the history service. `callback_outbound_latency`: A histogram representing the latency histogram of outbound callback requests made by the history service. ### Nexus Machinery on the Frontend Service[​](https://docs.temporal.io/references/cluster-metrics#nexus-machinery-on-the-frontend-service "Direct link to Nexus Machinery on the Frontend Service") #### `nexus_requests`[​](https://docs.temporal.io/references/cluster-metrics#nexus_requests "Direct link to nexus_requests") The number of Nexus requests received by the service. Type: Counter #### `nexus_latency`[​](https://docs.temporal.io/references/cluster-metrics#nexus_latency "Direct link to nexus_latency") Latency of Nexus requests. Type: Histogram #### `nexus_request_preprocess_errors`[​](https://docs.temporal.io/references/cluster-metrics#nexus_request_preprocess_errors "Direct link to nexus_request_preprocess_errors") The number of Nexus requests for which pre-processing failed. Type: Counter #### `nexus_completion_requests`[​](https://docs.temporal.io/references/cluster-metrics#nexus_completion_requests "Direct link to nexus_completion_requests") The number of Nexus completion (callback) requests received by the service. Type: Counter #### `nexus_completion_latency`[​](https://docs.temporal.io/references/cluster-metrics#nexus_completion_latency "Direct link to nexus_completion_latency") Latency histogram of Nexus completion (callback) requests. Type: Histogram #### `nexus_completion_request_preprocess_errors`[​](https://docs.temporal.io/references/cluster-metrics#nexus_completion_request_preprocess_errors "Direct link to nexus_completion_request_preprocess_errors") The number of Nexus completion requests for which pre-processing failed. Type: Counter * [Common metrics](https://docs.temporal.io/references/cluster-metrics#common-metrics) * [`service_requests`](https://docs.temporal.io/references/cluster-metrics#service_requests) * [`service_latency`](https://docs.temporal.io/references/cluster-metrics#service_latency) * [`service_error_with_type`](https://docs.temporal.io/references/cluster-metrics#service_error_with_type) * [`client_errors`](https://docs.temporal.io/references/cluster-metrics#client_errors) * [Matching Service metrics](https://docs.temporal.io/references/cluster-metrics#matching-service-metrics) * [`poll_success`](https://docs.temporal.io/references/cluster-metrics#poll_success) * [`poll_timeouts`](https://docs.temporal.io/references/cluster-metrics#poll_timeouts) * [`asyncmatch_latency`](https://docs.temporal.io/references/cluster-metrics#asyncmatch_latency) * [`no_poller_tasks`](https://docs.temporal.io/references/cluster-metrics#no_poller_tasks) * [History Service metrics](https://docs.temporal.io/references/cluster-metrics#history-service-metrics) * [`task_requests`](https://docs.temporal.io/references/cluster-metrics#task_requests) * [`task_errors`](https://docs.temporal.io/references/cluster-metrics#task_errors) * [`task_attempt`](https://docs.temporal.io/references/cluster-metrics#task_attempt) * [`task_latency_processing`](https://docs.temporal.io/references/cluster-metrics#task_latency_processing) * [`task_latency`](https://docs.temporal.io/references/cluster-metrics#task_latency) * [`task_latency_queue`](https://docs.temporal.io/references/cluster-metrics#task_latency_queue) * [`task_latency_load`](https://docs.temporal.io/references/cluster-metrics#task_latency_load) * [`task_latency_schedule`](https://docs.temporal.io/references/cluster-metrics#task_latency_schedule) * [`queue_latency_schedule`](https://docs.temporal.io/references/cluster-metrics#queue_latency_schedule) * [`service_latency_userlatency`](https://docs.temporal.io/references/cluster-metrics#service_latency_userlatency) * [Persistence metrics](https://docs.temporal.io/references/cluster-metrics#persistence-metrics) * [`persistence_requests`](https://docs.temporal.io/references/cluster-metrics#persistence_requests) * [`persistence_errors`](https://docs.temporal.io/references/cluster-metrics#persistence_errors) * [`persistence_error_with_type`](https://docs.temporal.io/references/cluster-metrics#persistence_error_with_type) * [`persistence_latency`](https://docs.temporal.io/references/cluster-metrics#persistence_latency) * [Schedule metrics](https://docs.temporal.io/references/cluster-metrics#schedule-metrics) * [`schedule_buffer_overruns`](https://docs.temporal.io/references/cluster-metrics#schedule_buffer_overruns) * [`schedule_missed_catchup_window`](https://docs.temporal.io/references/cluster-metrics#schedule_missed_catchup_window) * [`schedule_rate_limited`](https://docs.temporal.io/references/cluster-metrics#schedule_rate_limited) * [`schedule_action_success`](https://docs.temporal.io/references/cluster-metrics#schedule_action_success) * [Workflow metrics](https://docs.temporal.io/references/cluster-metrics#workflow-metrics) * [`workflow_cancel`](https://docs.temporal.io/references/cluster-metrics#workflow_cancel) * [`workflow_continued_as_new`](https://docs.temporal.io/references/cluster-metrics#workflow_continued_as_new) * [`workflow_failed`](https://docs.temporal.io/references/cluster-metrics#workflow_failed) * [`workflow_success`](https://docs.temporal.io/references/cluster-metrics#workflow_success) * [`workflow_timeout`](https://docs.temporal.io/references/cluster-metrics#workflow_timeout) * [Nexus metrics](https://docs.temporal.io/references/cluster-metrics#nexus-metrics) * [Nexus Machinery in the History Service](https://docs.temporal.io/references/cluster-metrics#nexus-machinery-in-the-history-service) * [In-Memory Buffer](https://docs.temporal.io/references/cluster-metrics#in-memory-buffer) * [Concurrency Limiter](https://docs.temporal.io/references/cluster-metrics#concurrency-limiter) * [Rate Limiter](https://docs.temporal.io/references/cluster-metrics#rate-limiter) * [Circuit Breaker](https://docs.temporal.io/references/cluster-metrics#circuit-breaker) * [Task Executors](https://docs.temporal.io/references/cluster-metrics#task-executors) * [Nexus Machinery on the Frontend Service](https://docs.temporal.io/references/cluster-metrics#nexus-machinery-on-the-frontend-service) * [`nexus_requests`](https://docs.temporal.io/references/cluster-metrics#nexus_requests) * [`nexus_latency`](https://docs.temporal.io/references/cluster-metrics#nexus_latency) * [`nexus_request_preprocess_errors`](https://docs.temporal.io/references/cluster-metrics#nexus_request_preprocess_errors) * [`nexus_completion_requests`](https://docs.temporal.io/references/cluster-metrics#nexus_completion_requests) * [`nexus_completion_latency`](https://docs.temporal.io/references/cluster-metrics#nexus_completion_latency) * [`nexus_completion_request_preprocess_errors`](https://docs.temporal.io/references/cluster-metrics#nexus_completion_request_preprocess_errors) --- # Deploying a Temporal Service | Temporal Platform Documentation [Skip to main content](https://docs.temporal.io/self-hosted-guide/deployment#__docusaurus_skipToContent_fallback) On this page There are many ways to self-host a [Temporal Service](https://docs.temporal.io/temporal-service) . The right way for you depends entirely on your use case and where you plan to run it. This page provides instructions for deploying a Temporal Service for sustained workloads that exceed what the [development server](https://docs.temporal.io/cli#start-dev-server) is designed to handle. For local development or testing, you can use the Temporal CLI to [start a local development Temporal Service](https://docs.temporal.io/cli#start-dev-server) . Temporal Hosts Should Not Be Exposed to the Open Internet In self-hosted deployments, the Temporal Service is a critical control and persistence component and should be secured similarly to a database. Temporal services should run on hosts that are not accessible from the public internet, with network configurations that restrict access to trusted internal networks only. For step-by-step guides on deploying and configuring Temporal, refer to our [Infrastructure tutorials](https://learn.temporal.io/tutorials/infrastructure/) . Use Docker Compose[​](https://docs.temporal.io/self-hosted-guide/deployment#use-docker-compose "Direct link to Use Docker Compose") ------------------------------------------------------------------------------------------------------------------------------------ You can run a Temporal Service in [Docker](https://docs.docker.com/engine/install) containers using [Docker Compose](https://docs.docker.com/compose/install) . ### Prerequisites[​](https://docs.temporal.io/self-hosted-guide/deployment#prerequisites "Direct link to Prerequisites") * You have Docker Compose installed. * Docker is running and the daemon is available. * Git is installed and available. ### Procedure[​](https://docs.temporal.io/self-hosted-guide/deployment#procedure "Direct link to Procedure") 1. Clone the [temporalio/samples-server](https://github.com/temporalio/samples-server) repository. 2. Change into the `compose` directory. cd samples-server/compose 3. Run the `docker compose up` command. This uses the default configuration from the `docker-compose.yaml` file, which includes a PostgreSQL database, an Elasticsearch instance, and exposes the Temporal gRPC Frontend on port 7233. docker compose up The Temporal Web UI will be available at `http://localhost:8080`. 4. (Optional) Review [the additional configuration options](https://github.com/temporalio/samples-server/tree/main/compose#other-configuration-files) available in the samples-server repository and use `docker compose up` with the corresponding configuration file to try them out. The configurations include different databases, visibility stores, and TLS settings. Use Temporal Server binaries[​](https://docs.temporal.io/self-hosted-guide/deployment#use-temporal-server-binaries "Direct link to Use Temporal Server binaries") ------------------------------------------------------------------------------------------------------------------------------------------------------------------ You can run a complete Temporal Server by deploying two Go binaries -- the [core Temporal Server](https://github.com/temporalio/temporal/releases/) , and the [Temporal UI Server](https://github.com/temporalio/ui-server/releases) . Each service can be deployed separately. Refer to [How to Configure a Temporal Service without a Proxy](https://learn.temporal.io/tutorials/infrastructure/configuring-sqlite-binary/) to deploy each service using `systemd`. If you need to run the Temporal Server behind a reverse proxy, refer to our tutorials to deploy the Temporal Service behind an [Nginx reverse proxy](https://learn.temporal.io/tutorials/infrastructure/nginx-sqlite-binary/) or an [Envoy edge proxy](https://learn.temporal.io/tutorials/infrastructure/envoy-sqlite-binary/) . ### Configuration templating[​](https://docs.temporal.io/self-hosted-guide/deployment#configuration-templating "Direct link to Configuration templating") Configuration templating is how the Temporal Server turns a template config file into the final `config.yaml` it runs with. It lets you reuse one template across environments by filling in values from environment variables. For example, database endpoints, TLS paths, or feature flags. If you are **not** using a custom config template, you can skip this section. The default configuration is rendered automatically by the server and embedded in the binary. #### Template compatibility[​](https://docs.temporal.io/self-hosted-guide/deployment#template-compatibility "Direct link to Template compatibility") If you use a custom configuration template, be aware of the following: * The server renders templates with embedded `sprig`, so any `dockerize`\-specific syntax or helpers will fail * Some template syntax differs, particularly `.Env` and `default` function usage. * Refer to the [sprig documentation](http://masterminds.github.io/sprig/) for supported template functions * Use `temporal-server render-config` to verify your templates render correctly #### Helm Chart configuration[​](https://docs.temporal.io/self-hosted-guide/deployment#helm-chart-configuration "Direct link to Helm Chart configuration") When deploying with Helm charts versions 0.73.1 or later, you may need to adjust the following configuration options depending on the images you are using. | Configuration Option | Description | Default | | --- | --- | --- | | `server.useEntrypointScript` | Whether to use entrypoint script that autodetects `dockerize` vs `sprig`. | `false` | | `server.configMapsToMount` | Which config template to mount: `"dockerize"`, `"sprig"`, or `"both"`. | `"dockerize"` | | `server.setConfigFilePath` | Set `TEMPORAL_SERVER_CONFIG_FILE_PATH` environment variable. | `false` | Refer to the following guidelines to determine if you need to adjust the configuration options: * The default settings work if you are only using pre-1.30 images with 0.73.1 or later Helm chart. * If you are using 1.30+ images with 0.73.1 or later Helm chart, you need to set `server.configMapsToMount` to `"sprig"` and `server.setConfigFilePath` to `true`. Keep the `server.useEntrypointScript` as `false`. * If you need use the Helm chart with both pre-1.30 and 1.30+ images, you need to set `server.configMapsToMount` to `"both"` and `server.useEntrypointScript` to `true`. Keep the `server.setConfigFilePath` as `false`. Import the Server package[​](https://docs.temporal.io/self-hosted-guide/deployment#import-the-server-package "Direct link to Import the Server package") --------------------------------------------------------------------------------------------------------------------------------------------------------- The Temporal Server is a standalone Go application that can be [imported](https://docs.temporal.io/references/server-options) into another project. You might want to do this to pass custom plugins or any other customizations through the [Server Options](https://docs.temporal.io/references/server-options) . Then you can build and run a binary that contains your customizations. This requires Go v1.19 or later, as specified in the Temporal Server [Build prerequisites](https://github.com/temporalio/temporal/blob/main/CONTRIBUTING.md#build-prerequisites) . Use Helm charts[​](https://docs.temporal.io/self-hosted-guide/deployment#use-helm-charts "Direct link to Use Helm charts") --------------------------------------------------------------------------------------------------------------------------- [Temporal Helm charts](https://github.com/temporalio/helm-charts) enable you to get a Temporal Service running on [Kubernetes](https://kubernetes.io/) by deploying the Temporal Server services to individual pods and connecting them to your existing database and Elasticsearch instances. The Temporal Helm charts repo contains [extensive documentation](https://github.com/temporalio/helm-charts/blob/main/README.md) about Kubernetes deployments. Helm Chart version compatibility If you are using Temporal Server images 1.30+, you must upgrade to Helm chart version 0.73.1 or later. Helm chart versions below 0.73.1 are **not compatible** with `server` and `admin-tools` images **version 1.30 and later**. You **cannot** override old chart versions with newer images. * [Use Docker Compose](https://docs.temporal.io/self-hosted-guide/deployment#use-docker-compose) * [Prerequisites](https://docs.temporal.io/self-hosted-guide/deployment#prerequisites) * [Procedure](https://docs.temporal.io/self-hosted-guide/deployment#procedure) * [Use Temporal Server binaries](https://docs.temporal.io/self-hosted-guide/deployment#use-temporal-server-binaries) * [Configuration templating](https://docs.temporal.io/self-hosted-guide/deployment#configuration-templating) * [Template compatibility](https://docs.temporal.io/self-hosted-guide/deployment#template-compatibility) * [Helm Chart configuration](https://docs.temporal.io/self-hosted-guide/deployment#helm-chart-configuration) * [Import the Server package](https://docs.temporal.io/self-hosted-guide/deployment#import-the-server-package) * [Use Helm charts](https://docs.temporal.io/self-hosted-guide/deployment#use-helm-charts) --- # Upgrade the Temporal Server | Temporal Platform Documentation [Skip to main content](https://docs.temporal.io/self-hosted-guide/upgrade-server#__docusaurus_skipToContent_fallback) On this page How to upgrade the Temporal Server version[​](https://docs.temporal.io/self-hosted-guide/upgrade-server#upgrade-server "Direct link to How to upgrade the Temporal Server version") ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ If a newer version of the [Temporal Server](https://docs.temporal.io/temporal-service/temporal-server) is available, a notification appears in the Temporal Web UI. info If you are using a version that is older than 1.0.0, reach out to us at [community.temporal.io](http://community.temporal.io/) to ask how to upgrade. First check to see if an upgrade to the database schema is required for the version you wish to upgrade to. If a database schema upgrade is required, it will be called out directly in the [release notes](https://github.com/temporalio/temporal/releases) . Some releases require changes to the schema, and some do not. We ensure that any consecutive versions are compatible in terms of database schema upgrades, features, and system behavior; however there is no guarantee that there is compatibility between _any_ two non-consecutive versions. ### Key considerations[​](https://docs.temporal.io/self-hosted-guide/upgrade-server#key-considerations "Direct link to Key considerations") When upgrading the Temporal Server, there are two key considerations to keep in mind: 1. **Sequential Upgrades:** Temporal Server should be upgraded sequentially. That is, if you're on version (v1.n.x), your next upgrade should be to (v1.n+1.x) or the closest available subsequent version. This sequence should be repeated until your desired version is reached. 2. **Data Compatibility:** During an upgrade, the Temporal Server either updates or restructures the existing version data to match the data format of the newer version. Temporal Server ensures backward compatibility only between two successive minor versions. Consequently, skipping versions during an upgrade may lead to older data formats becoming unreadable. If the previous data format cannot be interpreted and converted to the newer format, the upgrade process will be unsuccessful. ### Step-by-Step Upgrade Procedure:[​](https://docs.temporal.io/self-hosted-guide/upgrade-server#step-by-step-upgrade-procedure "Direct link to Step-by-Step Upgrade Procedure:") Upgrading the Temporal Server requires a methodical approach to ensure data integrity, compatibility, and seamless transition between versions. The following documentation outlines the step-by-step process to successfully upgrade your Temporal Server. When upgrading your Temporal Server version, ensure that you upgrade sequentially. 1. **Upgrade Database Schema:** Before initiating the Temporal Server upgrade, use one of the recommended upgrade tools to update your database schema. This ensures it is aligned with the version of Temporal Server you aim to upgrade to. 2. **Upgrade Temporal Server:** Once the database schema is updated, proceed to upgrade the Temporal Server deployment to the next sequential version. 3. **Iterative Upgrades** (optional): Continue this process (steps 1 and 2) iteratively until you reach the desired Temporal Server version. By adhering to the above guidelines and following the step-by-step procedure, you can ensure a smooth and successful upgrade of your Temporal Server. The Temporal Server upgrade updates or rewrites the old version data with the format introduced in the newer version. Because Temporal Server guarantees backward compatibility between two consecutive minor versions, and because older versions of the code are eventually removed from the code base, skipping versions when upgrading might cause older formats to become unrecognizable. If the old format of the data can't be read to be rewritten to the new format, the upgrades fail. Check the [Temporal Server releases](https://github.com/temporalio/temporal/releases) and follow these releases in order. You can skip patch versions; use the latest patch of a minor version when upgrading. Also, be aware that each upgrade requires the History Service to load all Shards and update the Shard metadata, so allow approximately 10 minutes on each version for these processes to complete before upgrading to the next version. Use one of the upgrade tools to upgrade your database schema to be compatible with the Temporal Server version being upgraded to. ### Upgrade Cassandra schema[​](https://docs.temporal.io/self-hosted-guide/upgrade-server#upgrade-cassandra-schema "Direct link to Upgrade Cassandra schema") If you are using Cassandra for your Temporal Service's persistence, use the `temporal-cassandra-tool` to upgrade both the default Persistence and Visibility schemas. **Example default schema upgrade:** temporal_v1.2.1 $ temporal-cassandra-tool \ --tls \ --tls-ca-file <...> \ --user \ --password \ --endpoint \ --keyspace temporal \ --timeout 120 \ update \ --schema-dir ./schema/cassandra/temporal/versioned **Example visibility schema upgrade:** temporal_v1.2.1 $ temporal-cassandra-tool \ --tls \ --tls-ca-file <...> \ --user \ --password \ --endpoint \ --keyspace temporal_visibility \ --timeout 120 \ update \ --schema-dir ./schema/cassandra/visibility/versioned ### Upgrade Elasticsearch schema[​](https://docs.temporal.io/self-hosted-guide/upgrade-server#upgrade-elasticsearch-schema "Direct link to Upgrade Elasticsearch schema") If you are using Elasticsearch for your Temporal Service's Visibility, use the `temporal-elasticsearch-tool` to upgrade the schema. **Example schema upgrade:** echo "Updating index mappings: $ES_VISIBILITY_INDEX"temporal-elasticsearch-tool \ --endpoint "$ES_SCHEME://$ES_HOST:$ES_PORT" \ --user "$ES_USER" \ --password "$ES_PWD" \ update-schema \ --index "$ES_VISIBILITY_INDEX" ### Upgrade PostgreSQL or MySQL schema[​](https://docs.temporal.io/self-hosted-guide/upgrade-server#upgrade-postgresql-or-mysql-schema "Direct link to Upgrade PostgreSQL or MySQL schema") If you are using MySQL or PostgreSQL use the `temporal-sql-tool`, which works similarly to the `temporal-cassandra-tool`. Refer to this [Makefile](https://github.com/temporalio/temporal/blob/v1.4.1/Makefile#L367-L383) for context. #### PostgreSQL[​](https://docs.temporal.io/self-hosted-guide/upgrade-server#postgresql "Direct link to PostgreSQL") **Example default schema upgrade:** ./temporal-sql-tool \ --tls \ --tls-enable-host-verification \ --tls-cert-file \ --tls-key-file \ --tls-ca-file \ --ep localhost -p 5432 -u temporal -pw temporal --pl postgres --db temporal update-schema -d ./schema/postgresql/v96/temporal/versioned **Example visibility schema upgrade:** ./temporal-sql-tool \ --tls \ --tls-enable-host-verification \ --tls-cert-file \ --tls-key-file \ --tls-ca-file \ --ep localhost -p 5432 -u temporal -pw temporal --pl postgres --db temporal_visibility update-schema -d ./schema/postgresql/v96/visibility/versioned If you're upgrading PostgreSQL to v12 or later to enable advanced Visibility features with Temporal Server v1.20, upgrade your PostgreSQL version first, and then run `temporal-sql-tool` with the `postgres12` plugin, as shown in the following example: ./temporal-sql-tool \ --tls \ --tls-enable-host-verification \ --tls-cert-file \ --tls-key-file \ --tls-ca-file \ --ep localhost -p 5432 -u temporal -pw temporal --pl postgres12 --db temporal_visibility update-schema -d ./schema/postgresql/v12/visibility/versioned #### MySQL[​](https://docs.temporal.io/self-hosted-guide/upgrade-server#mysql "Direct link to MySQL") **Example default schema upgrade:** ./temporal-sql-tool \ --tls \ --tls-enable-host-verification \ --tls-cert-file \ --tls-key-file \ --tls-ca-file \ --ep localhost -p 3036 -u root -pw root --pl mysql --db temporal update-schema -d ./schema/mysql/v57/temporal/versioned/ **Example visibility schema upgrade:** ./temporal-sql-tool \ --tls \ --tls-enable-host-verification \ --tls-cert-file \ --tls-key-file \ --tls-ca-file \ --ep localhost -p 3036 -u root -pw root --pl mysql --db temporal_visibility update-schema -d ./schema/mysql/v57/visibility/versioned/ If you're upgrading MySQL to v8.0.17 or later to enable advanced Visibility features with Temporal Server v1.20, upgrade your MySQL version first, and then run `temporal-sql-tool` with the `mysql8` plugin, as shown in the following example: ./temporal-sql-tool \ --tls \ --tls-enable-host-verification \ --tls-cert-file \ --tls-key-file \ --tls-ca-file \ --ep localhost -p 5432 -u temporal -pw temporal --pl mysql8 --db temporal_visibility update-schema -d ./schema/mysql/v8/visibility/versioned. ### Roll-out technique[​](https://docs.temporal.io/self-hosted-guide/upgrade-server#roll-out-technique "Direct link to Roll-out technique") We recommend preparing a staging Temporal Service and then doing the following to verify the upgrade is successful: 1. Create some simulation load on the staging Temporal Service. 2. Upgrade the database schema in the staging Temporal Service. 3. Wait and observe for a few minutes to verify that there is no unstable behavior from both the server and the simulation load logic. 4. Upgrade the server. 5. Now do the same to the live environment Temporal Service. * [How to upgrade the Temporal Server version](https://docs.temporal.io/self-hosted-guide/upgrade-server#upgrade-server) * [Key considerations](https://docs.temporal.io/self-hosted-guide/upgrade-server#key-considerations) * [Step-by-Step Upgrade Procedure:](https://docs.temporal.io/self-hosted-guide/upgrade-server#step-by-step-upgrade-procedure) * [Upgrade Cassandra schema](https://docs.temporal.io/self-hosted-guide/upgrade-server#upgrade-cassandra-schema) * [Upgrade Elasticsearch schema](https://docs.temporal.io/self-hosted-guide/upgrade-server#upgrade-elasticsearch-schema) * [Upgrade PostgreSQL or MySQL schema](https://docs.temporal.io/self-hosted-guide/upgrade-server#upgrade-postgresql-or-mysql-schema) * [PostgreSQL](https://docs.temporal.io/self-hosted-guide/upgrade-server#postgresql) * [MySQL](https://docs.temporal.io/self-hosted-guide/upgrade-server#mysql) * [Roll-out technique](https://docs.temporal.io/self-hosted-guide/upgrade-server#roll-out-technique) --- # Interceptors | Temporal Platform Documentation [Skip to main content](https://docs.temporal.io/encyclopedia/interceptors#__docusaurus_skipToContent_fallback) On this page Interceptors let you add cross-cutting behavior before and after SDK operations such as starting a Workflow, executing an Activity, or handling a Signal. They work like middleware: each interceptor wraps the next, forming a chain that executes around the underlying operation. Common use cases: * Observability (logging, metrics, tracing) * Authorization and authentication checks * Header manipulation (propagating metadata) * Input/output validation Implementing Interceptors[​](https://docs.temporal.io/encyclopedia/interceptors#implementing-interceptors "Direct link to Implementing Interceptors") ------------------------------------------------------------------------------------------------------------------------------------------------------ Here are SDK-specific guides: * [Python](https://docs.temporal.io/develop/python/workers/interceptors) * [TypeScript](https://docs.temporal.io/develop/typescript/workers/interceptors) * [Implementing Interceptors](https://docs.temporal.io/encyclopedia/interceptors#implementing-interceptors) --- # Server frontend API reference | Temporal Platform Documentation [Skip to main content](https://docs.temporal.io/self-hosted-guide/server-frontend-api-reference#__docusaurus_skipToContent_fallback) On this page While it's usually easiest to interact with [Temporal Server](https://docs.temporal.io/temporal-service/temporal-server) via a [Client SDK](https://docs.temporal.io/encyclopedia/temporal-sdks#temporal-client) or the [Temporal CLI](https://docs.temporal.io/cli) , you can also use its gRPC API. Our Client and Worker SDKs use the gRPC API. The API reference is located here: [`api-docs.temporal.io`](https://api-docs.temporal.io/) Use with code[​](https://docs.temporal.io/self-hosted-guide/server-frontend-api-reference#use-with-code "Direct link to Use with code") ---------------------------------------------------------------------------------------------------------------------------------------- Usually you interact with the API via high-level methods like `client.workflow.start()`. However, Client SDKs also expose the underlying gRPC services. For instance, the TypeScript SDK has: * WorkflowService: [`Client.connection.workflowService`](https://typescript.temporal.io/api/classes/client.Connection#workflowservice) * OperatorService: [`Client.connection.operatorService`](https://typescript.temporal.io/api/classes/client.Connection/#operatorservice) * HealthService: [`Client.connection.healthService`](https://typescript.temporal.io/api/classes/client.Connection/#healthservice) If you're not using an SDK Client (rare), you can generate gRPC client stubs by: * Cloning [`temporalio/api`](https://github.com/temporalio/api) (repo with the protobuf files) * Generating code in [your language](https://grpc.io/docs/languages/) Use manually[​](https://docs.temporal.io/self-hosted-guide/server-frontend-api-reference#use-manually "Direct link to Use manually") ------------------------------------------------------------------------------------------------------------------------------------- To query the API manually via command line or a GUI, first: * If you don't already have a Server to connect to, run [`temporal server start-dev`](https://docs.temporal.io/cli/server#start-dev) * Clone this repo: git clone https://github.com/temporalio/api.gitcd api ### With command line[​](https://docs.temporal.io/self-hosted-guide/server-frontend-api-reference#with-command-line "Direct link to With command line") Install [`evans`](https://github.com/ktr0731/evans#installation) . cd /path/to/apievans --proto temporal/api/workflowservice/v1/service.proto --port 7233 To connect to Temporal Cloud, set the host, cert, cert key, and TLS flag: evans --proto temporal/api/workflowservice/v1/service.proto --host devrel.a2dd6.tmprl.cloud --port 7233 --tls --cert /Users/me/certs/temporal.pem --certkey /Users/me/certs/temporal.key Once inside the evans prompt, you can run commands like `help`, `show service` to list available methods, and `call ListWorkflowExecutions`. ### With a GUI[​](https://docs.temporal.io/self-hosted-guide/server-frontend-api-reference#with-a-gui "Direct link to With a GUI") * Install [BloomRPC](https://github.com/bloomrpc/bloomrpc#installation) . * Open the app * Select "Import Paths" button on the top-left and enter the path to the cloned repo: `/path/to/api` * Select the "Import protos" + button and select this file: /path/to/api/temporal/api/workflowservice/v1/service.proto * A list of methods should appear in the sidebar. Select one. * Edit the JSON in the left pane. * Hit `Cmd/Ctrl-Enter` or click the play button to get a response from the server on the right. ![ListWorkflowExecutions](https://docs.temporal.io/img/proto/ListWorkflowExecutions.png) ListWorkflowExecutions One downside compared to [command line](https://docs.temporal.io/self-hosted-guide/server-frontend-api-reference#with-command-line) is it doesn't show enum names, just numbers like `"task_queue_type": 1`. ![DescribeTaskQueue](https://docs.temporal.io/img/proto/DescribeTaskQueue.png) DescribeTaskQueue * [Use with code](https://docs.temporal.io/self-hosted-guide/server-frontend-api-reference#use-with-code) * [Use manually](https://docs.temporal.io/self-hosted-guide/server-frontend-api-reference#use-manually) * [With command line](https://docs.temporal.io/self-hosted-guide/server-frontend-api-reference#with-command-line) * [With a GUI](https://docs.temporal.io/self-hosted-guide/server-frontend-api-reference#with-a-gui) --- # Temporal Cluster configuration reference | Temporal Platform Documentation [Skip to main content](https://docs.temporal.io/references/configuration#__docusaurus_skipToContent_fallback) On this page Much of the behavior of a Temporal Cluster is configured using the `development.yaml` file and may contain the following top-level sections: * [`global`](https://docs.temporal.io/references/configuration#global) * [`persistence`](https://docs.temporal.io/references/configuration#persistence) * [`log`](https://docs.temporal.io/references/configuration#log) * [`clusterMetadata`](https://docs.temporal.io/references/configuration#clustermetadata) * [`services`](https://docs.temporal.io/references/configuration#services) * [`publicClient`](https://docs.temporal.io/references/configuration#publicclient) * [`archival`](https://docs.temporal.io/references/configuration#archival) * [`namespaceDefaults`](https://docs.temporal.io/references/configuration#namespacedefaults) * [`dcRedirectionPolicy`](https://docs.temporal.io/references/configuration#dcredirectionpolicy) * [`dynamicConfigClient`](https://docs.temporal.io/references/configuration#dynamicconfigclient) Changing any properties in the `development.yaml` file requires a process restart for changes to take effect. Configuration parsing code is available [here](https://github.com/temporalio/temporal/blob/main/common/config/config.go) . global[​](https://docs.temporal.io/references/configuration#global "Direct link to global") -------------------------------------------------------------------------------------------- The `global` section contains process-wide configuration. See below for a minimal configuration (optional parameters are commented out.) global: membership: broadcastAddress: '127.0.0.1' metrics: prometheus: framework: 'tally' listenAddress: '127.0.0.1:8000' ### membership[​](https://docs.temporal.io/references/configuration#membership "Direct link to membership") The `membership` section controls the following membership layer parameters. #### maxJoinDuration[​](https://docs.temporal.io/references/configuration#maxjoinduration "Direct link to maxJoinDuration") The amount of time the service will attempt to join the gossip layer before failing. Default is 10s. #### broadcastAddress[​](https://docs.temporal.io/references/configuration#broadcastaddress "Direct link to broadcastAddress") Used by gossip protocol to communicate with other hosts in the same Cluster for membership info. Use IP address that is reachable by other hosts in the same Cluster. If there is only one host in the Cluster, you can use 127.0.0.1. Check `net.ParseIP` for supported syntax, only IPv4 is supported. ### metrics[​](https://docs.temporal.io/references/configuration#metrics "Direct link to metrics") Configures the Cluster's metric subsystem. Specific providers are configured using provider names as the keys. * [`statsd`](https://docs.temporal.io/references/configuration#statsd) * `prometheus` * `m3` #### prefix[​](https://docs.temporal.io/references/configuration#prefix "Direct link to prefix") The prefix to be applied to all outgoing metrics. #### tags[​](https://docs.temporal.io/references/configuration#tags "Direct link to tags") The set of key-value pairs to be reported as part of every metric. #### excludeTags[​](https://docs.temporal.io/references/configuration#excludetags "Direct link to excludeTags") A map from tag name string to tag values string list. This is useful to exclude some tags that might have unbounded cardinality. The value string list can be used to whitelist values of that excluded tag to continue to be included. For example, if you want to exclude `task_queue` because it has unbounded cardinality, but you still want to see a whitelisted value for `task_queue`. #### statsd[​](https://docs.temporal.io/references/configuration#statsd "Direct link to statsd") caution `statsd` is not supported natively by Temporal. The `statsd` sections supports the following settings: * `hostPort`: The host:port of the statsd server. * `prefix`: Specific prefix in reporting to `statsd`. * `flushInterval`: Maximum interval for sending packets. (_Default_ 300ms). * `flushBytes`: Specifies the maximum UDP packet size you wish to send. (_Default_ 1432 bytes). #### prometheus[​](https://docs.temporal.io/references/configuration#prometheus "Direct link to prometheus") The `prometheus` sections supports the following settings: * `framework`: The framework to use, currently supports `opentelemetry` and `tally`, default is `tally`. We plan to switch default to `opentelemetry` once its API become stable. * `listenAddress`: Address for Prometheus to scrape metrics from. The Temporal Server uses the Prometheus client API, and the `listenAddress` configuration is used to listen for metrics. * `handlerPath`: Metrics handler path for scraper; default is `/metrics`. #### m3[​](https://docs.temporal.io/references/configuration#m3 "Direct link to m3") The `m3` sections supports the following settings: * `hostPort`: The host:port of the M3 server. * `service`: The service tag that this client emits. * `queue`: M3 reporter queue size, default is 4k. * `packetSize`: M3 reporter max packet size, default is 32k. ### pprof[​](https://docs.temporal.io/references/configuration#pprof "Direct link to pprof") * `port`: If specified, this will initialize pprof upon process start on the listed port. ### tls[​](https://docs.temporal.io/references/configuration#tls "Direct link to tls") The `tls` section controls the SSL/TLS settings for network communication and contains two subsections, `internode` and `frontend`. The `internode` section governs internal service communication among roles where the `frontend` governs SDK client communication to the Frontend Service role. Each of these subsections contain a `server` section and a `client` section. The `server` contains the following parameters: * `certFile`: The path to the file containing the PEM-encoded public key of the certificate to use. * `keyFile`: The path to the file containing the PEM-encoded private key of the certificate to use. * `requireClientAuth`: _boolean_ - Requires clients to authenticate with a certificate when connecting, otherwise known as mutual TLS. * `clientCaFiles`: A list of paths to files containing the PEM-encoded public key of the Certificate Authorities you wish to trust for client authentication. This value is ignored if `requireClientAuth` is not enabled. tip See the [server samples repo](https://github.com/temporalio/samples-server/tree/master/tls) for sample TLS configurations. Below is an example enabling Server TLS (https) between SDKs and the Frontend APIs: global: tls: frontend: server: certFile: /path/to/cert/file keyFile: /path/to/key/file client: serverName: dnsSanInFrontendCertificate Note, the `client` section generally needs to be provided to specify an expected DNS SubjectName contained in the presented server certificate via the `serverName` field; this is needed as Temporal uses IP to IP communication. You can avoid specifying this if your server certificates contain the appropriate IP Subject Alternative Names. Additionally, the `rootCaFiles` field needs to be provided when the client's host does not trust the Root CA used by the server. The example below extends the above example to manually specify the Root CA used by the Frontend Services: global: tls: frontend: server: certFile: /path/to/cert/file keyFile: /path/to/key/file client: serverName: dnsSanInFrontendCertificate rootCaFiles: - /path/to/frontend/server/CA/files Below is an additional example of a fully secured cluster using mutual TLS for both frontend and internode communication with manually specified CAs: global: tls: internode: server: certFile: /path/to/internode/cert/file keyFile: /path/to/internode/key/file requireClientAuth: true clientCaFiles: - /path/to/internode/serverCa client: serverName: dnsSanInInternodeCertificate rootCaFiles: - /path/to/internode/serverCa frontend: server: certFile: /path/to/frontend/cert/file keyFile: /path/to/frontend/key/file requireClientAuth: true clientCaFiles: - /path/to/internode/serverCa - /path/to/sdkClientPool1/ca - /path/to/sdkClientPool2/ca client: serverName: dnsSanInFrontendCertificate rootCaFiles: - /path/to/frontend/serverCa **Note:** In the case that client authentication is enabled, the `internode.server` certificate is used as the client certificate among services. This adds the following requirements: * The `internode.server` certificate must be specified on all roles, even for a frontend-only configuration. * Internode server certificates must be minted with either **no** Extended Key Usages or **both** ServerAuth and ClientAuth EKUs. * If your Certificate Authorities are untrusted, such as in the previous example, the internode server Ca will need to be specified in the following places: * `internode.server.clientCaFiles` * `internode.client.rootCaFiles` * `frontend.server.clientCaFiles` persistence[​](https://docs.temporal.io/references/configuration#persistence "Direct link to persistence") ----------------------------------------------------------------------------------------------------------- The `persistence` section holds configuration for the data store/persistence layer. The following example shows a minimal specification for a password-secured Cluster using Cassandra. persistence: defaultStore: default visibilityStore: cass-visibility # The primary Visibility store. secondaryVisibilityStore: es-visibility # A secondary Visibility store added to enable Dual Visibility. numHistoryShards: 512 datastores: default: cassandra: hosts: '127.0.0.1' keyspace: 'temporal' user: 'username' password: 'password' cass-visibility: cassandra: hosts: '127.0.0.1' keyspace: 'temporal_visibility' es-visibility: elasticsearch: version: 'v7' logLevel: 'error' url: scheme: 'http' host: '127.0.0.1:9200' indices: visibility: temporal_visibility_v1_dev closeIdleConnectionsInterval: 15s The following top level configuration items are required: ### numHistoryShards[​](https://docs.temporal.io/references/configuration#numhistoryshards "Direct link to numHistoryShards") _Required_ - The number of history shards to create when initializing the Cluster. **Warning:** This value is immutable and will be ignored after the first run. Please ensure you set this value appropriately high enough to scale with the worst case peak load for this Cluster. ### defaultStore[​](https://docs.temporal.io/references/configuration#defaultstore "Direct link to defaultStore") _Required_ - The name of the data store definition that should be used by the Temporal server. ### visibilityStore[​](https://docs.temporal.io/references/configuration#visibilitystore "Direct link to visibilityStore") _Required_ - The name of the primary data store definition that should be used to set up [Visibility](https://docs.temporal.io/temporal-service/visibility) on the Temporal Cluster. ### secondaryVisibilityStore[​](https://docs.temporal.io/references/configuration#secondaryvisibilitystore "Direct link to secondaryVisibilityStore") _Optional_ - The name of the secondary data store definition that should be used to set up [Dual Visibility](https://docs.temporal.io/dual-visibility) on the Temporal Cluster. ### datastores[​](https://docs.temporal.io/references/configuration#datastores "Direct link to datastores") _Required_ - contains named data store definitions to be referenced. Each definition is defined with a heading declaring a name (ie: `default:` and `visibility:` above), which contains a data store definition. Data store definitions must be either `cassandra` or `sql`. #### cassandra[​](https://docs.temporal.io/references/configuration#cassandra "Direct link to cassandra") A `cassandra` data store definition can contain the following values: * `hosts`: _Required_ - "," separated Cassandra endpoints, e.g. "192.168.1.2,192.168.1.3,192.168.1.4". * `port`: Default: 9042 - Cassandra port used for connection by `gocql` client. * `user`: Cassandra username used for authentication by `gocql` client. * `password`: Cassandra password used for authentication by `gocql` client. * `keyspace`: _Required_ - the Cassandra keyspace. * `datacenter`: The data center filter arg for Cassandra. * `maxConns`: The max number of connections to this data store for a single TLS configuration. * `tls`: See TLS below. #### sql[​](https://docs.temporal.io/references/configuration#sql "Direct link to sql") A `sql` data store definition can contain the following values: * `user`: Username used for authentication. * `password`: Password used for authentication. * `pluginName`: _Required_ - SQL database type. * _Valid values_: `mysql` or `postgres`. * `databaseName` - _required_ - the name of SQL database to connect to. * `connectAddr` - _required_ - the remote address of the database, e.g. "192.168.1.2". * `connectProtocol` - _required_ - the protocol that goes with the `connectAddr` * _Valid values_: `tcp` or `unix` * `connectAttributes` - a map of key-value attributes to be sent as part of connect `data_source_name` url. * `maxConns` - the max number of connections to this data store. * `maxIdleConns` - the max number of idle connections to this data store * `maxConnLifetime` - is the maximum time a connection can be alive. * `tls` - See below. #### tls[​](https://docs.temporal.io/references/configuration#tls-1 "Direct link to tls") The `tls` and `mtls` sections can contain the following values: * `enabled` - _boolean_. * `serverName` - name of the server hosting the data store. * `certFile` - path to the cert file. * `keyFile` - path to the key file. * `caFile` - path to the ca file. * `enableHostVerification` - _boolean_ - `true` to verify the hostname and server cert (like a wildcard for Cassandra cluster). This option is basically the inverse of `InSecureSkipVerify`. See `InSecureSkipVerify` in [http://golang.org/pkg/crypto/tls/](http://golang.org/pkg/crypto/tls/) for more info. Note: `certFile` and `keyFile` are optional depending on server config, but both fields must be omitted to avoid using a client certificate. log[​](https://docs.temporal.io/references/configuration#log "Direct link to log") ----------------------------------------------------------------------------------- The `log` section is optional and contains the following possible values: * `stdout` - _boolean_ - `true` if the output needs to go to standard out. * `level` - sets the logging level. * _Valid values_ - debug, info, warn, error or fatal, default to info. * `outputFile` - path to output log file. clusterMetadata[​](https://docs.temporal.io/references/configuration#clustermetadata "Direct link to clusterMetadata") ----------------------------------------------------------------------------------------------------------------------- `clusterMetadata` contains the local cluster information. The information is used in [Multi-Cluster Replication](https://docs.temporal.io/temporal-service/multi-cluster-replication) . An example `clusterMetadata` section: clusterMetadata: enableGlobalNamespace: true failoverVersionIncrement: 10 masterClusterName: 'active' currentClusterName: 'active' clusterInformation: active: enabled: true initialFailoverVersion: 0 rpcAddress: '127.0.0.1:7233' #replicationConsumer: #type: kafka * `currentClusterName` - _required_ - the name of the current cluster. **Warning:** This value is immutable and will be ignored after the first run. * `enableGlobalNamespace` - _Default:_ `false`. * `replicationConsumer` - determines which method to use to consume replication tasks. The type may be either `kafka` or `rpc`. * `failoverVersionIncrement` - the increment of each cluster version when failover happens. * `masterClusterName` - the master cluster name, only the master cluster can register/update namespace. All clusters can do namespace failover. * `clusterInformation` - contains the local cluster name to `ClusterInformation` definition. The local cluster name should be consistent with `currentClusterName`. `ClusterInformation` sections consist of: * `enabled` - _boolean_ - whether a remote cluster is enabled for replication. * `initialFailoverVersion` * `rpcAddress` - indicate the remote service address (host:port). Host can be DNS name. Use `dns:///` prefix to enable round-robin between IP address for DNS name. services[​](https://docs.temporal.io/references/configuration#services "Direct link to services") -------------------------------------------------------------------------------------------------- The `services` section contains configuration keyed by service role type. There are four supported service roles: * `frontend` * `matching` * `worker` * `history` Below is a minimal example of a `frontend` service definition under `services`: services: frontend: rpc: grpcPort: 8233 membershipPort: 8933 bindOnIP: '0.0.0.0' There are two sections defined under each service heading: ### rpc[​](https://docs.temporal.io/references/configuration#rpc "Direct link to rpc") _Required_ `rpc` contains settings related to the way a service interacts with other services. The following values are supported: * `grpcPort`: Port on which gRPC will listen. * `membershipPort`: Port used to communicate with other hosts in the same Cluster for membership info. Each service should use different port. If there are multiple Temporal Clusters in your environment (Kubernetes for example), and they have network access to each other, each Cluster should use a different membership port. * `bindOnLocalHost`: Determines whether uses `127.0.0.1` as the listener address. * `bindOnIP`: Used to bind service on specific IP, or `0.0.0.0`. Check `net.ParseIP` for supported syntax, only IPv4 is supported, mutually exclusive with `BindOnLocalHost` option. **Note:** Port values are currently expected to be consistent among role types across all hosts. publicClient[​](https://docs.temporal.io/references/configuration#publicclient "Direct link to publicClient") -------------------------------------------------------------------------------------------------------------- The `publicClient` is a required section describing the configuration needed for a worker to connect to Temporal server for background server maintenance. * `hostPort` IPv4 host port or DNS name to reach Temporal frontend, [reference](https://github.com/grpc/grpc/blob/master/doc/naming.md) Example: publicClient: hostPort: 'localhost:8933' Use `dns:///` prefix to enable round-robin between IP address for DNS name. archival[​](https://docs.temporal.io/references/configuration#archival "Direct link to archival") -------------------------------------------------------------------------------------------------- _Optional_ Archival is an optional configuration needed to set up the [Archival store](https://docs.temporal.io/temporal-service/archival) . It can be enabled on `history` and `visibility` data. The following list describes supported values for each configuration on the `history` and `visibility` data. * `state`: State for Archival setting. Supported values are `enabled`, `disabled`. This value must be `enabled` to use Archival with any Namespace in your Cluster. * `enabled`: Enables Archival in your Cluster setup. When set to `enabled`, `URI` and `namespaceDefaults` values must be provided. * `disabled`: Disables Archival in your Cluster setup. When set to `disabled`, the `enableRead` value must be set to `false`, and under `namespaceDefaults`, `state` must be set to `disabled`, with no values set for `provider` and `URI` fields. * `enableRead`: Supported values are `true` or `false`. Set to `true` to allow read operations from the archived Event History data. * `provider`: Location where data should be archived. Subprovider configs are `filestore`, `gstorage`, `s3`, or `your_custom_provider`. Default configuration specifies `filestore`. Example: * To enable Archival in your Cluster configuration: # Cluster-level Archival config enabledarchival: # Event History configuration history: # Archival is enabled for the History Service data. state: 'enabled' enableRead: true # Namespaces can use either the local filestore provider or the Google Cloud provider. provider: filestore: fileMode: '0666' dirMode: '0766' gstorage: credentialsPath: '/tmp/gcloud/keyfile.json' # Configuration for archiving Visibility data. visibility: # Archival is enabled for Visibility data. state: 'enabled' enableRead: true provider: filestore: fileMode: '0666' dirMode: '0766' * To disable Archival in your Cluster configuration: # Cluster-level Archival config disabledarchival: history: state: 'disabled' enableRead: false visibility: state: 'disabled' enableRead: falsenamespaceDefaults: archival: history: state: 'disabled' visibility: state: 'disabled' For more details on Archival setup, see [Set up Archival](https://docs.temporal.io/self-hosted-guide/archival#set-up-archival) . namespaceDefaults[​](https://docs.temporal.io/references/configuration#namespacedefaults "Direct link to namespaceDefaults") ----------------------------------------------------------------------------------------------------------------------------- _Optional_ Sets default Archival configuration for each Namespace using `namespaceDefaults` for `history` and `visibility` data. * `state`: Default state of the Archival for the Namespace. Supported values are `enabled` or `disabled`. * `URI`: Default URI for the Namespace. For more details on setting Namespace defaults on Archival, see [Create an Archiving Namespace in Archival setup](https://docs.temporal.io/self-hosted-guide/archival#create-an-archiving-namespace) Example: # Default values for a Namespace if none are provided at creation.namespaceDefaults: # Archival defaults. archival: # Event History defaults. history: state: 'enabled' # New Namespaces will default to the local provider. URI: 'file:///tmp/temporal_archival/development' visibility: state: 'disabled' URI: 'file:///tmp/temporal_vis_archival/development' dcRedirectionPolicy[​](https://docs.temporal.io/references/configuration#dcredirectionpolicy "Direct link to dcRedirectionPolicy") ----------------------------------------------------------------------------------------------------------------------------------- _Optional_ Contains the Frontend datacenter API redirection policy that you can use for cross-DC replication. Supported values: * `policy`: Supported values are `noop`, `selected-apis-forwarding`, and `all-apis-forwarding`. * `noop`: Not setting a value or setting `noop` means no redirection. This is the default value. * `selected-apis-forwarding`: Sets up forwarding for the following APIs to the active Cluster based on the Namespace. * `StartWorkflowExecution` * `SignalWithStartWorkflowExecution` * `SignalWorkflowExecution` * `RequestCancelWorkflowExecution` * `TerminateWorkflowExecution` * `QueryWorkflow` * `all-apis-forwarding`: Sets up forwarding for all APIs on the Namespace in the active Cluster. Example: #...dcRedirectionPolicy: policy: 'selected-apis-forwarding'#... dynamicConfigClient[​](https://docs.temporal.io/references/configuration#dynamicconfigclient "Direct link to dynamicConfigClient") ----------------------------------------------------------------------------------------------------------------------------------- _Optional_ Configuration for setting up file-based [dynamic configuration](https://docs.temporal.io/temporal-service/configuration#dynamic-configuration) client for the Cluster. This setting is required if specifying dynamic configuration. Supported configuration values are as follows: * `filepath`: Specifies the path where the dynamic configuration YAML file is stored. The path should be relative to the root directory. * `pollInterval`: Interval between the file-based client polls to check for dynamic configuration updates. The minimum period you can set is 5 seconds. Example: dynamicConfigClient: filepath: 'config/dynamicconfig/development-cass.yaml' pollInterval: '10s' * [global](https://docs.temporal.io/references/configuration#global) * [membership](https://docs.temporal.io/references/configuration#membership) * [maxJoinDuration](https://docs.temporal.io/references/configuration#maxjoinduration) * [broadcastAddress](https://docs.temporal.io/references/configuration#broadcastaddress) * [metrics](https://docs.temporal.io/references/configuration#metrics) * [prefix](https://docs.temporal.io/references/configuration#prefix) * [tags](https://docs.temporal.io/references/configuration#tags) * [excludeTags](https://docs.temporal.io/references/configuration#excludetags) * [statsd](https://docs.temporal.io/references/configuration#statsd) * [prometheus](https://docs.temporal.io/references/configuration#prometheus) * [m3](https://docs.temporal.io/references/configuration#m3) * [pprof](https://docs.temporal.io/references/configuration#pprof) * [tls](https://docs.temporal.io/references/configuration#tls) * [persistence](https://docs.temporal.io/references/configuration#persistence) * [numHistoryShards](https://docs.temporal.io/references/configuration#numhistoryshards) * [defaultStore](https://docs.temporal.io/references/configuration#defaultstore) * [visibilityStore](https://docs.temporal.io/references/configuration#visibilitystore) * [secondaryVisibilityStore](https://docs.temporal.io/references/configuration#secondaryvisibilitystore) * [datastores](https://docs.temporal.io/references/configuration#datastores) * [cassandra](https://docs.temporal.io/references/configuration#cassandra) * [sql](https://docs.temporal.io/references/configuration#sql) * [tls](https://docs.temporal.io/references/configuration#tls-1) * [log](https://docs.temporal.io/references/configuration#log) * [clusterMetadata](https://docs.temporal.io/references/configuration#clustermetadata) * [services](https://docs.temporal.io/references/configuration#services) * [rpc](https://docs.temporal.io/references/configuration#rpc) * [publicClient](https://docs.temporal.io/references/configuration#publicclient) * [archival](https://docs.temporal.io/references/configuration#archival) * [namespaceDefaults](https://docs.temporal.io/references/configuration#namespacedefaults) * [dcRedirectionPolicy](https://docs.temporal.io/references/configuration#dcredirectionpolicy) * [dynamicConfigClient](https://docs.temporal.io/references/configuration#dynamicconfigclient) --- # Low latency - Temporal feature | Temporal Platform Documentation [Skip to main content](https://docs.temporal.io/evaluate/development-production-features/low-latency#__docusaurus_skipToContent_fallback) Temporal Cloud provides features that significantly reduce latency compared to self-hosted instances, making your applications faster, more performant, and more efficient. In the world of modern applications, low latency is crucial for ensuring minimal delay in Workflow Executions. This low-latency architecture ensures rapid Workflow Execution and responsiveness, critical for time-sensitive applications and high-performance systems. Temporal Cloud's custom persistence layer incorporates three key components that contribute to low latency: * **Better Sharding:** Distributes load across multiple databases, preventing bottlenecks. Enables independent resizing, improving scalability and handling high-traffic events without delay. * **Write-Ahead Log (WAL):** Aggregates updates before writing to the database, reducing write latency. Stores writes in an append-only format, reducing latency and database size by batching updates before writing to the database. * **Tiered Storage of Workflow Event History:** Offloads completed Workflow Event Histories, improving database efficiency. Temporal Cloud provides lower latency, making it suitable for latency-sensitive, large-scale, or business-critical applications. Related 📚 * [Exploring Temporal Cloud Automation Features](https://temporal.io/blog/exploring-temporal-cloud-automation-features) * [High Availability and Disaster Recovery with Temporal Cloud](https://temporal.io/blog/high-availability-and-disaster-recovery-with-temporal-cloud) * [Higher throughput and lower latency: Temporal Cloud’s custom persistence layer](https://temporal.io/blog/higher-throughput-and-lower-latency-temporal-clouds-custom-persistence-layer) * [How to Migrate Your Self-Hosted Service to Temporal Cloud](https://temporal.io/blog/how-to-migrate-your-self-hosted-service-to-temporal-cloud) * [Scaling your self-hosted instance](https://temporal.io/blog/scaling-temporal-the-basics) * [Benchmarking Latency: Temporal Cloud vs. Self-Hosted Temporal](https://temporal.io/blog/benchmarking-latency-temporal-cloud-vs-self-hosted-temporal) * [Temporal Cloud’s Latency SLO](https://docs.temporal.io/cloud/service-availability#latency) * [Replay Conference Talk: Custom Persistence Layer](https://www.youtube.com/watch?v=SQv9ot-jB6o&list=PLl9kRkvFJrlREHL7fiEKBWTp5QuFeYS2r&index=5) --- # Temporal Commands reference | Temporal Platform Documentation [Skip to main content](https://docs.temporal.io/references/commands#__docusaurus_skipToContent_fallback) On this page A [Command](https://docs.temporal.io/workflow-execution#command) is a requested action issued by a [Worker](https://docs.temporal.io/workers#worker) to the [Temporal Service](https://docs.temporal.io/temporal-service) after a [Workflow Task Execution](https://docs.temporal.io/tasks#workflow-task-execution) completes. The following is a complete list of possible Commands. ### CompleteWorkflowExecution[​](https://docs.temporal.io/references/commands#completeworkflowexecution "Direct link to CompleteWorkflowExecution") This Command is triggered when the Workflow Function Execution returns. It indicates to the Temporal Service that the [Workflow Execution](https://docs.temporal.io/workflow-execution) is complete. The corresponding [Event](https://docs.temporal.io/workflow-execution/event#event) for this Command is one of the few Events that will be the last in a Workflow Execution [Event History](https://docs.temporal.io/workflow-execution/event#event-history) . * Awaitable: No, a Workflow Execution can not await on the action resulting from this Command. * Corresponding Event: [WorkflowExecutionCompleted](https://docs.temporal.io/references/events#workflowexecutioncompleted) ### ContinueAsNewWorkflowExecution[​](https://docs.temporal.io/references/commands#continueasnewworkflowexecution "Direct link to ContinueAsNewWorkflowExecution") This Command is triggered when there is a call to [Continue-As-New](https://docs.temporal.io/workflow-execution/continue-as-new) from within the [Workflow](https://docs.temporal.io/workflows) . The corresponding Event for this Command is one of the few Events that will be the last in a Workflow Execution Event History. * Awaitable: No, a Workflow Execution can not await on the action resulting from this Command. * Corresponding Event: [WorkflowExecutionContinuedAsNew](https://docs.temporal.io/references/events#workflowexecutioncontinuedasnew) ### FailWorkflowExecution[​](https://docs.temporal.io/references/commands#failworkflowexecution "Direct link to FailWorkflowExecution") This Command is triggered when the Workflow Execution returns an error or an exception is thrown. * Awaitable: No, a Workflow Execution can not await on the action resulting from this Command. * Corresponding Event: [WorkflowExecutionFailed](https://docs.temporal.io/references/events#workflowexecutionfailed) ### CancelWorkflowExecution[​](https://docs.temporal.io/references/commands#cancelworkflowexecution "Direct link to CancelWorkflowExecution") This Command is triggered when the Workflow has successfully cleaned up after receiving a Cancellation Request (which will be present as [WorkflowExecutionCancelRequestedEvent](https://docs.temporal.io/references/events#workflowexecutioncancelrequested) in the Event History). The Corresponding Event for this Command is one of the few Events that will be the last in a Workflow Execution Event History. * Awaitable: No, a Workflow Execution can not await on the action resulting from this Command. * Corresponding Event: [WorkflowExecutionCanceled](https://docs.temporal.io/references/events#workflowexecutioncanceled) ### StartChildWorkflowExecution[​](https://docs.temporal.io/references/commands#startchildworkflowexecution "Direct link to StartChildWorkflowExecution") This Command is triggered by a call to spawn a [Child Workflow Execution](https://docs.temporal.io/child-workflows) . * Awaitable: Yes, a Workflow Execution can await on the action resulting from this Command. * Corresponding Event: [ChildWorkflowExecutionStarted](https://docs.temporal.io/references/events#childworkflowexecutionstarted) By default, you cannot have more than 2,000 pending Child Workflows. ### SignalExternalWorkflowExecution[​](https://docs.temporal.io/references/commands#signalexternalworkflowexecution "Direct link to SignalExternalWorkflowExecution") This Command is triggered by a call to [Signal](https://docs.temporal.io/sending-messages#sending-signals) another Workflow Execution. * Awaitable: Yes, a Workflow Execution can await on the action resulting from this Command. * Corresponding Event: [SignalExternalWorkflowExecutionInitiated](https://docs.temporal.io/references/events#signalexternalworkflowexecutioninitiated) By default, you cannot have more than 2,000 pending Signals to other Workflows. ### RequestCancelExternalWorkflowExecution[​](https://docs.temporal.io/references/commands#requestcancelexternalworkflowexecution "Direct link to RequestCancelExternalWorkflowExecution") This Command is triggered by a call to request cancellation of another Workflow Execution. * Awaitable: Yes, a Workflow Execution can await on the action resulting from this Command. * Corresponding Event: [RequestCancelExternalWorkflowExecutionInitiated](https://docs.temporal.io/references/events#requestcancelexternalworkflowexecutioninitiated) By default, you cannot have more than 2,000 pending Signals to other Workflows. ### ScheduleActivityTask[​](https://docs.temporal.io/references/commands#scheduleactivitytask "Direct link to ScheduleActivityTask") This Command is triggered by a call to execute an [Activity](https://docs.temporal.io/activities) . * Awaitable: Yes, a Workflow Execution can await on the action resulting from this Command. * Corresponding Event: [ActivityTaskScheduled](https://docs.temporal.io/references/events#activitytaskscheduled) By default, you cannot schedule more than 2,000 Activities concurrently. ### RequestCancelActivityTask[​](https://docs.temporal.io/references/commands#requestcancelactivitytask "Direct link to RequestCancelActivityTask") This Command is triggered by a call to request the cancellation of an [Activity Task](https://docs.temporal.io/tasks#activity-task) . * Awaitable: No, a Workflow Execution can not await on the action resulting from this Command. * Corresponding Event: [ActivityTaskCancelRequested](https://docs.temporal.io/references/events#activitytaskcancelrequested) ### StartTimer[​](https://docs.temporal.io/references/commands#starttimer "Direct link to StartTimer") This Command is triggered by a call to start a Timer. * Awaitable: Yes, a Workflow Execution can await on the action resulting from this Command. * Corresponding Event: [TimerStarted](https://docs.temporal.io/references/events#timerstarted) ### CancelTimer[​](https://docs.temporal.io/references/commands#canceltimer "Direct link to CancelTimer") This Command is triggered by a call to cancel a Timer. * Awaitable: No, a Workflow Execution can not await on the action resulting from this Command. * Corresponding Event: [TimerCanceled](https://docs.temporal.io/references/events#timercanceled) ### RecordMarker[​](https://docs.temporal.io/references/commands#recordmarker "Direct link to RecordMarker") This Command is triggered by the SDK. * Awaitable: No, a Workflow Execution can not await on the action resulting from this Command. * Corresponding Event: [MarkerRecorded](https://docs.temporal.io/references/events#markerrecorded) ### UpsertWorkflowSearchAttributes[​](https://docs.temporal.io/references/commands#upsertworkflowsearchattributes "Direct link to UpsertWorkflowSearchAttributes") This Command is triggered by a call to "upsert" Workflow [Search Attributes](https://docs.temporal.io/search-attribute) . * Awaitable: No, a Workflow Execution can not await on the action resulting from this Command. * Corresponding Event: [UpsertWorkflowSearchAttributes](https://docs.temporal.io/references/events#upsertworkflowsearchattributes) ### ProtocolMessageCommand[​](https://docs.temporal.io/references/commands#protocolmessagecommand "Direct link to ProtocolMessageCommand") This Command helps guarantee ordering constraints for features such as Updates. This Command points at the message from which the Event is created. Therefore, just from the Command, you can't predict the resulting Event type. ### ScheduleNexusOperation[​](https://docs.temporal.io/references/commands#schedulenexusoperation "Direct link to ScheduleNexusOperation") This Command is triggered by a call to execute a Nexus Operation in the caller Workflow. * Awaitable: Yes, a Workflow Execution can await on the action resulting from this Command. * Corresponding Event: [NexusOperationScheduled](https://docs.temporal.io/references/events#nexusoperationscheduled) By default, you can't schedule more than 30 Nexus Operations concurrently, see [Limits](https://docs.temporal.io/workflow-execution/limits#workflow-execution-nexus-operation-limits) for details. ### CancelNexusOperation[​](https://docs.temporal.io/references/commands#cancelnexusoperation "Direct link to CancelNexusOperation") This Command is triggered by a call to request the cancellation of a Nexus Operation. * Awaitable: No, a Workflow Execution can not await on the action resulting from this Command. * Corresponding Event: [NexusOperationCancelRequested](https://docs.temporal.io/references/events#nexusoperationcancelrequested) * [CompleteWorkflowExecution](https://docs.temporal.io/references/commands#completeworkflowexecution) * [ContinueAsNewWorkflowExecution](https://docs.temporal.io/references/commands#continueasnewworkflowexecution) * [FailWorkflowExecution](https://docs.temporal.io/references/commands#failworkflowexecution) * [CancelWorkflowExecution](https://docs.temporal.io/references/commands#cancelworkflowexecution) * [StartChildWorkflowExecution](https://docs.temporal.io/references/commands#startchildworkflowexecution) * [SignalExternalWorkflowExecution](https://docs.temporal.io/references/commands#signalexternalworkflowexecution) * [RequestCancelExternalWorkflowExecution](https://docs.temporal.io/references/commands#requestcancelexternalworkflowexecution) * [ScheduleActivityTask](https://docs.temporal.io/references/commands#scheduleactivitytask) * [RequestCancelActivityTask](https://docs.temporal.io/references/commands#requestcancelactivitytask) * [StartTimer](https://docs.temporal.io/references/commands#starttimer) * [CancelTimer](https://docs.temporal.io/references/commands#canceltimer) * [RecordMarker](https://docs.temporal.io/references/commands#recordmarker) * [UpsertWorkflowSearchAttributes](https://docs.temporal.io/references/commands#upsertworkflowsearchattributes) * [ProtocolMessageCommand](https://docs.temporal.io/references/commands#protocolmessagecommand) * [ScheduleNexusOperation](https://docs.temporal.io/references/commands#schedulenexusoperation) * [CancelNexusOperation](https://docs.temporal.io/references/commands#cancelnexusoperation) --- # Temporal development and production features | Temporal Platform Documentation [Skip to main content](https://docs.temporal.io/evaluate/development-production-features#__docusaurus_skipToContent_fallback) Through a Temporal SDK, Temporal provides a wide range of features that enable developers to build applications that serve a wide range of use cases. * **[Core application primitives](https://docs.temporal.io/evaluate/development-production-features/core-application) **: Develop and run your application with Workflows, Activities, and Workers. * **[Testing suite](https://docs.temporal.io/evaluate/development-production-features/testing-suite) **: Each Temporal SDK comes with a testing suite that enables developers to test their applications as they would any other. * **[Scheduled Workflows](https://docs.temporal.io/evaluate/development-production-features/schedules) **: Start a business process at a specific time or on a given time interval. * **[Interrupt a Workflow](https://docs.temporal.io/evaluate/development-production-features/interrupt-workflow) **: Cancel or terminate a business process (Workflow) that is already in progress and compensate for any steps already taken. * **Runtime safeguards**: Prevent avoidable errors and issues from executing during runtime. * **[Failure detection and mitigation](https://docs.temporal.io/evaluate/development-production-features/failure-detection) **: Detect failures with timeouts and configure automatic retries to mitigate them. * **[Temporal Nexus](https://docs.temporal.io/evaluate/nexus) **: Connect Temporal Applications across (and within) isolated Namespaces for improved modularity, security, debugging, and fault isolation. Nexus supports cross-team, cross-domain, and multi-region use cases. * **[Workflow message passing](https://docs.temporal.io/evaluate/development-production-features/workflow-message-passing) **: Build responsive applications that react to events at runtime and enable data retrieval from ongoing Workflows. * **Versioning**: Support multiple versions of your business logic for long-running business processes. * **[Observability](https://docs.temporal.io/evaluate/development-production-features/observability) **: List business processes, view their state, and set up dashboards with metrics. * **[Debugging](https://docs.temporal.io/evaluate/development-production-features/debugging) **: Surface errors and step through code to find issues. * **[Data encryption](https://docs.temporal.io/evaluate/development-production-features/data-encryption) **: Transform data and protect the privacy of the users of your application. * **[Throughput composability](https://docs.temporal.io/evaluate/development-production-features/throughput-composability) **: Breakup business processes by data streams, team ownership, or other organization factors. * **[Cloud Automation](https://docs.temporal.io/evaluate/development-production-features/cloud-automation) **: Simplify cloud management and boost security with Temporal's Cloud Automation. * **[Low Latency](https://docs.temporal.io/evaluate/development-production-features/low-latency) **: Making your applications faster, more performant, and more efficient. * **[Multi-tenancy](https://docs.temporal.io/evaluate/development-production-features/multi-tenancy) **: Enhances efficiency and cost-effectiveness. For detailed information on Temporal feature release stages and criteria, see this [Product Release Stages Guide](https://docs.temporal.io/evaluate/development-production-features/release-stages) . --- # Codec Server - Temporal Platform feature guide | Temporal Platform Documentation [Skip to main content](https://docs.temporal.io/production-deployment/data-encryption#__docusaurus_skipToContent_fallback) On this page Temporal Server stores and persists the data handled in your Workflow Execution. Encrypting this data ensures that any sensitive application data is secure when handled by the Temporal Server. For example, if you have sensitive information passed in the following objects that are persisted in the Workflow Execution Event History, use encryption to secure it: * Inputs and outputs/results in your [Workflow](https://docs.temporal.io/workflow-execution) , [Activity](https://docs.temporal.io/activity-execution) , and [Child Workflow](https://docs.temporal.io/child-workflows) * [Signal](https://docs.temporal.io/sending-messages#sending-signals) inputs * [Memo](https://docs.temporal.io/workflow-execution#memo) * Headers (verify if applicable to your SDK) * [Query](https://docs.temporal.io/sending-messages#sending-queries) inputs and results * Results of [Local Activities](https://docs.temporal.io/local-activity) and [Side Effects](https://docs.temporal.io/workflow-execution/event#side-effect) * [Application errors and failures](https://docs.temporal.io/references/failures) . Failure messages and call stacks are not encoded as codec-capable Payloads by default; you must explicitly enable encoding these common attributes on failures. For more details, see [Failure Converter](https://docs.temporal.io/failure-converter) . Using encryption ensures that your sensitive data exists unencrypted only on the Client and the Worker Process that is executing the Workflows and Activities, on hosts that you control. By default, your data is serialized to a [Payload](https://docs.temporal.io/dataconversion#payload) by a [Data Converter](https://docs.temporal.io/dataconversion) . To encrypt your Payload, configure your custom encryption logic with a [Payload Codec](https://docs.temporal.io/payload-codec) and set it with a [custom Data Converter](https://docs.temporal.io/default-custom-data-converters#custom-data-converter) . A Payload Codec does byte-to-byte conversion to transform your Payload (for example, by implementing compression and/or encryption and decryption) and is an optional step that happens between the Client and the [Payload Converter](https://docs.temporal.io/payload-converter) : ![Remote data encoding architecture](https://docs.temporal.io/diagrams/remote-data-encoding.svg) Remote data encoding architecture You can run your Payload Codec with a [Codec Server](https://docs.temporal.io/codec-server) and use the Codec Server endpoints in the Web UI and CLI to decode your encrypted Payload locally. For details on how to set up a Codec Server, see [Codec Server setup](https://docs.temporal.io/production-deployment/data-encryption#codec-server-setup) . However, if you plan to set up [remote data encoding](https://docs.temporal.io/remote-data-encoding) for your data, ensure that you consider all security implications of running encryption remotely before implementing it. When implementing a custom codec, it is recommended to perform your compression or encryption on the entire input Payload and store the result in the data field of a new Payload with a different encoding metadata field. This ensures that the input Payload's metadata is preserved. When the encoded Payload is sent to be decoded, you can verify the metadata field before applying the decryption. If your Payload is not encoded, it is recommended to pass the unencoded data to the decode function instead of failing the conversion. Examples for implementing encryption: * [Go sample](https://github.com/temporalio/samples-go/tree/main/encryption) * [Java sample](https://github.com/temporalio/samples-java/tree/main/core/src/main/java/io/temporal/samples/encryptedpayloads) * [Python sample](https://github.com/temporalio/samples-python/tree/main/encryption) * [TypeScript sample](https://github.com/temporalio/samples-typescript/tree/main/encryption) * [.NET sample](https://github.com/temporalio/samples-dotnet/tree/main/src/Encryption) Codec Server setup[​](https://docs.temporal.io/production-deployment/data-encryption#codec-server-setup "Direct link to Codec Server setup") --------------------------------------------------------------------------------------------------------------------------------------------- Use a Codec Server to programmatically decode your encoded [payloads](https://docs.temporal.io/dataconversion#payload) . A Codec Server is an HTTP server that uses your custom Codec logic to decode your data remotely. The Codec Server is independent of the Temporal Service and decodes your encrypted payloads through predefined endpoints. You create, operate, and manage access to your Codec Server in your own environment. The Temporal CLI and the Web UI in turn provide built-in hooks to call the Codec Server to decode encrypted payloads on demand. The Codec Server is independent of the Temporal Server and decodes your encrypted payloads through endpoints. When you configure a Codec Server endpoint in the Temporal Web UI or CLI, the Web UI and CLI use the remote endpoint to receive decoded payloads from the Codec Server. See [API contract requirements](https://docs.temporal.io/production-deployment/data-encryption#api-contract-specifications) . Decoded payloads can then be displayed in the Workflow Execution Event History on the Web UI. Note that when you use a Codec Server, the decoded payloads are decoded and returned on the client side only; payloads on the Temporal Server (whether on Temporal Cloud or a self-hosted Temporal Service) remain encrypted. Because you create, operate, and manage access to your Codec Server in your controlled environment, ensure that you consider the following: * When you register a Codec Server endpoint with your Web UI, expect the Codec Server to receive multiple requests per Workflow Execution. * Ensure that you secure access to your Codec Server. For details, see [Authorization](https://docs.temporal.io/production-deployment/data-encryption#authorization) . You might need some form of [Key management infrastructure](https://docs.temporal.io/key-management) for sharing your encryption keys between the Workers and your Codec Server. * You will need to enable [CORS](https://developer.mozilla.org/en-US/docs/Web/HTTP/CORS) on the HTTP/HTTPS endpoints in your Codec Server to receive requests from the Temporal Web UI. * You may introduce latency in the Web UI when sending and receiving payloads to the Codec Server. Your Codec Server should share logic with the custom [Payload Codec](https://docs.temporal.io/payload-codec) used elsewhere in your application. ### API contract specifications[​](https://docs.temporal.io/production-deployment/data-encryption#api-contract-specifications "Direct link to API contract specifications") When you create your Codec Server to handle requests from the Web UI, the following requirements must be met. #### Endpoints[​](https://docs.temporal.io/production-deployment/data-encryption#endpoints "Direct link to Endpoints") The Web UI and CLI send a POST to a `/decode` endpoint. In your Codec Server, create a `/decode` path and pass the incoming payload to the decode method in your Payload Codec. For examples on how to create your Codec Server, see the following Codec Server implementation samples: * [Go](https://github.com/temporalio/samples-go/tree/main/codec-server) * [Java](https://github.com/temporalio/sdk-java/tree/master/temporal-remote-data-encoder) * [Python](https://github.com/temporalio/samples-python/blob/main/encryption/codec_server.py) * [TypeScript](https://github.com/temporalio/samples-typescript/blob/main/encryption/src/codec-server.ts) * [.NET](https://github.com/temporalio/samples-dotnet/blob/main/src/Encryption/CodecServer/Program.cs) You can also add a [verification step](https://docs.temporal.io/production-deployment/data-encryption#authorization) to check whether the incoming request has the required authorization to access the decode logic in your Payload Codec. #### Headers[​](https://docs.temporal.io/production-deployment/data-encryption#headers "Direct link to Headers") Each request from the Web UI to your Codec Server includes the following headers: * `Content-Type: application/json`: Ensure that your Codec Server can accommodate this [MIME type](https://developer.mozilla.org/en-US/docs/Web/HTTP/Basics_of_HTTP/MIME_types) . * `X-Namespace: {namespace}`: This is a custom HTTP Header. Ensure that the [CORS](https://developer.mozilla.org/en-US/docs/Web/HTTP/CORS) configuration in your Codec Server includes this header. * \[Optional\] `Authorization: `: Include this in your CORS configuration when enabling authorization with your Codec Server. For details on setting up authorization, see [Authorization](https://docs.temporal.io/production-deployment/data-encryption#authorization) . #### Request body[​](https://docs.temporal.io/production-deployment/data-encryption#request-body "Direct link to Request body") The general specification for the `POST` request body contains payloads. By default, all field values in your payload are base64 encoded, regardless of whether they are encrypted by your custom codec implementation. The following example shows a sample `POST` request body with base64 encoding. { "payloads": [{ "metadata": { "encoding": }, "data": }, ...]} #### CORS[​](https://docs.temporal.io/production-deployment/data-encryption#cors "Direct link to CORS") By default, in cross-origin Fetch/XHR invocations, browsers will not send credentials. Enable [Cross-Origin Resource Sharing (CORS)](https://developer.mozilla.org/en-US/docs/Web/HTTP/CORS) requests on your Codec Server to receive HTTP/HTTPS requests from the Temporal Web UI. At a minimum, enable the following responses from your Codec Server to allow requests coming from the Temporal Web UI: * `Access-Control-Allow-Origin` * `Access-Control-Allow-Methods` * `Access-Control-Allow-Headers` For example, for Temporal Cloud Web UI hosted at [https://cloud.temporal.io](https://cloud.temporal.io/) , enable the following in your Codec Server: * `Access-Control-Allow-Origin: https://cloud.temporal.io` * `Access-Control-Allow-Methods: POST, GET, OPTIONS` * `Access-Control-Allow-Headers: X-Namespace, Content-Type` For details on what a sample request/response looks like from the Temporal Web UI, see [Sample Request/Response](https://docs.temporal.io/production-deployment/data-encryption#sample-requestresponse) . If setting authorization, include `Authorization` in your `Access-Control-Allow-Headers`. For details on setting up authorization, see [Authorization](https://docs.temporal.io/production-deployment/data-encryption#authorization) . #### Authorization[​](https://docs.temporal.io/production-deployment/data-encryption#authorization "Direct link to Authorization") It is important to establish how you will provide access to your Codec Server. Because it is designed to decode potentially sensitive data with a single API call, access to a production Codec Server should be restricted. Depending on your infrastructure and risk levels, it might be sufficient to restrict HTTP ingress to your Codec Server (such as by using a VPN like [WireGuard](https://www.wireguard.com/) ). The Temporal Web UI can communicate with a Codec Server that is only accessible on `localhost`, so this is a legitimate security pattern. However, if your Codec Server is exposed to the internet at all, you will likely need an authentication solution. If you are already using an organization-wide authentication provider, you should integrate it with your Codec Server. Remember, a Codec Server is just a standalone HTTP server, so you can use existing libraries for OAuth, [Auth0](https://auth0.com/) , or any other protocol. [This repository](https://github.com/pvsone/codec-cors-credentials) contains an example of using Auth0 to handle browser-based auth to a Codec Server. To enable authorization from the Web UI (for both a self-hosted Temporal Service and Temporal Cloud), your Codec Server must be an HTTPS Server. **Temporal Cloud** The Temporal Cloud UI provides an option to pass access tokens (JWT) to your Codec Server endpoints. Use the access tokens to validate access and then return decoded payloads from the Codec Server. You can enable this by selecting **Pass access token** in your Codec Server endpoint interface where you add your endpoint. Enabling this option in the Temporal Cloud UI adds an authorization header to each request sent to the Codec Server endpoint that you set. In your Codec Server implementation, verify the signature on this access token (in your authorization header) against [our JWKS endpoint](https://login.tmprl.cloud/.well-known/jwks.json) . The token provided from Temporal Cloud UI contains the email identifier of the person requesting access to the payloads. Based on the permissions you have provided to the user in your access control systems, set conditions in your Codec Server whether to return decoded payloads or just return the original encoded payloads. **Self-hosted Temporal Service** On a self-hosted Temporal Service, configure [authorization in the Web UI configuration](https://docs.temporal.io/references/web-ui-configuration#auth) in your Temporal Service setup. With this enabled, you can pass access tokens to your Codec Server and validate the requests from the Web UI to the Codec Server endpoints that you set. Note that with a self-hosted Temporal Service, you must explicitly configure authorization specifications for the Web UI and CLI. #### Sample request/response[​](https://docs.temporal.io/production-deployment/data-encryption#sample-requestresponse "Direct link to Sample request/response") Consider the following sample request/response when creating and hosting a Codec Server with the following specifications: * Scheme: `https` * Host: `dev.mydomain.com/codec` * Path: `/decode` HTTP/1.1 POST /decodeHost: https://dev.mydomain.com/codecContent-Type: application/jsonX-Namespace: myapp-dev.acctid123Authorization: Bearer {"payloads":[{"metadata":{"encoding":"anNvbi9wcm90b2J1Zg==","messageType":"dGVtcG9yYWxfc2hvcC5vcmNoZXN0cmF0aW9ucy52MS5TdGFydFNob3BwaW5nQ2FydFJlcXVlc3Q="},"data":"eyJjYXJ0SWQiOiJleGFtcGxlLWNhcnQiLCJzaG9wcGVySWQiOiJ5b3VyLXNob3BwZXItaWQtZXhhbXBsZSIsImVtYWlsIjoieW91ci1lbWFpbEBkb21haW4uY29tIn0"}]}200 OKContent-Type: application/json{ "payloads": [{ "metadata":{ "encoding": "json/protobuf", "messageType": "temporal_shop.orchestrations.v1.StartShoppingCartRequest" }, "data":{ "cartId":"example-cart", "shopperId":"your-shopper-id-example", "email":"your-email@domain.com" }}]} You can also perform remote encoding on an `/encode` endpoint, which looks the same in reverse: * Scheme: `https` * Host: `dev.mydomain.com/codec` * Path: `/encode` HTTP/1.1 POST /encodeHost: https://dev.mydomain.com/codecContent-Type: application/jsonX-Namespace: myapp-dev.acctid123Authorization: Bearer {"payloads":[{"metadata":{"encoding":"json/protobuf","messageType":"temporal_shop.orchestrations.v1.StartShoppingCartRequest"},"data":{"cartId":"example-cart","shopperId":"your-shopper-id-example","email":"your-email@domain.com"}}]}200 OKContent-Type: application/json{ "payloads": [ { "metadata": { "encoding": "anNvbi9wcm90b2J1Zg==", "messageType": "dGVtcG9yYWxfc2hvcC5vcmNoZXN0cmF0aW9ucy52MS5TdGFydFNob3BwaW5nQ2FydFJlcXVlc3Q=" }, "data": "eyJjYXJ0SWQiOiJleGFtcGxlLWNhcnQiLCJzaG9wcGVySWQiOiJ5b3VyLXNob3BwZXItaWQtZXhhbXBsZSIsImVtYWlsIjoieW91ci1lbWFpbEBkb21haW4uY29tIn0" } ]} ### Set your Codec Server endpoints with Web UI and CLI[​](https://docs.temporal.io/production-deployment/data-encryption#set-your-codec-server-endpoints-with-web-ui-and-cli "Direct link to Set your Codec Server endpoints with Web UI and CLI") After you create your Codec Server and expose the requisite endpoints, set the endpoints in your Web UI and CLI. #### Web UI[​](https://docs.temporal.io/production-deployment/data-encryption#web-ui "Direct link to Web UI") On Temporal Cloud and self-hosted Temporal Service, you can configure a Codec Server endpoint to be used for a Namespace in the Web UI. ![Codec Server endpoint Namespace setting](https://docs.temporal.io/img/info/set-codec-endpoint-form.png) Codec Server endpoint Namespace setting caution If your Codec Server is on a private network, your browser may block Temporal Web UI from accessing it. On Chrome, your browser may prompt you to allow access to the Codec Server endpoint. Make sure to allow access to the Codec Server endpoint. Refer to the [Chrome for Developers blog: New permission prompt for Local Network Access](https://developer.chrome.com/blog/local-network-access/) for details on this permission prompt. If your browser has blocked Temporal's access to your Codec Server, refer to [Chrome documentation](https://support.google.com/chrome/answer/114662) for details on how to change the site settings. In **Site settings**, look for the **Local network** setting and change it to **Allow** to only change this setting for Temporal without affecting other sites. To set a Codec Server endpoint on a Namespace, do the following. 1. In the Web UI, go to Namespaces, select the Namespace where you want to configure the Codec Server endpoint, and click **Edit**. 2. In the **Codec Server** section on the Namespace configuration page, enter your Codec Server endpoint and port number. 3. Optional: If your Codec Server is configured to [authenticate requests](https://docs.temporal.io/production-deployment/data-encryption#authorization) from Temporal Web UI, enable **Pass access token** to send a JWT access token with the HTTPS requests. 4. Optional: If your Codec Server is configured to [verify origins of requests](https://docs.temporal.io/production-deployment/data-encryption#cors) , enable **Include cross-origin credentials**. On Temporal Cloud, you must have [Namespace Admin privileges](https://docs.temporal.io/cloud/manage-access/roles-and-permissions#namespace-level-permissions) to add a Codec Server endpoint on the Namespace. Setting a Codec Server endpoint on a Cloud Namespace enables it for all users on the Namespace. Setting a Codec Server endpoint on a self-hosted Temporal Service enables it for the entire Temporal Service. You can use a single Codec Server to handle different encoding and decoding routes for each namespace. You can also override the global Codec Server setting at the browser level. This can be useful when developing, testing, or troubleshooting encoding functionality. ![Codec Server endpoint browser setting](https://docs.temporal.io/img/info/data-encoder-button.png) Codec Server endpoint browser setting To set a browser override for the Namespace-level endpoint, do the following. 1. Navigate to **Workflows** in your Namespace. 2. In the top-right corner, select **Configure Codec Server**. 3. Select whether you want to use the Namespace-level (or Temporal Service-level for self-hosted Temporal Service) or the browser-level Codec Endpoint setting as the default for your browser. In Temporal Cloud: * **Use Namespace-level settings, where available. Otherwise, use my browser setting.** Uses the Namespace-level Codec Server endpoint by default. If no endpoint is set on the Namespace, your browser setting is applied. * **Use my browser setting and ignore Namespace-level setting.** Applies your browser-level setting by default, overriding the Namespace-level Codec Server endpoint. 4. Enter your Codec Server endpoint and port number. 5. Optional: If your Codec Server is configured to [authenticate requests](https://docs.temporal.io/production-deployment/data-encryption#authorization) from Temporal Web UI, enable **Pass access token** to send a JWT access token with the HTTPS requests. 6. Optional: If your Codec Server is configured to [verify origins of requests](https://docs.temporal.io/production-deployment/data-encryption#cors) , enable **Include cross-origin credentials**. In a self-hosted Temporal Service with dedicated UI Server configuration, you can also set the codec endpoint in the UI server [configuration file](https://docs.temporal.io/references/web-ui-configuration#codec) : codec: endpoint: {{ default .Env.TEMPORAL_CODEC_ENDPOINT "{namespace}"}} #### CLI[​](https://docs.temporal.io/production-deployment/data-encryption#cli "Direct link to CLI") You can configure a Codec Server endpoint with the Temporal CLI using the `--codec-endpoint` flag. For example, if you are running your Codec Server on `http://localhost:8888`, you can use `env set` to set the endpoint globally: temporal env set --codec-endpoint "http://localhost:8888" If your Codec Server endpoint is not set globally, provide the `--codec-endpoint` option with each command. For example, to see the decoded output of the Workflow Execution "yourWorkflow" in the Namespace "yourNamespace", run: temporal --codec-endpoint "http://localhost:8888" --namespace "yourNamespace" workflow show --workflow-id "yourWorkflow" --run-id "" --output "table" For details, see the [CLI reference](https://docs.temporal.io/cli/) . If your Codec Server requires authentication, the Temporal CLI will also accept a `--codec-auth` parameter to supply an authorization header: temporal workflow show \ --workflow-id converters_workflowID \ --codec-endpoint 'http://localhost:8081/{namespace}' \ --codec-auth 'auth-header' ### Working with Large Payloads[​](https://docs.temporal.io/production-deployment/data-encryption#working-with-large-payloads "Direct link to Working with Large Payloads") Codec Servers can be used for more than encryption and decryption of sensitive data. Codec Server behavior is left up to implementers -- they can also call external services or perform other tasks, as long as they hook in at the encoding and decoding stages of a Workflow payload. By default, Temporal limits payload size to 4MB. If this limitation is problematic for your use case, you could implement a codec that persists your payloads to an object store outside of workflow histories. An example implementation is available from [DataDog](https://github.com/DataDog/temporal-large-payload-codec) . ### Temporal Nexus[​](https://docs.temporal.io/production-deployment/data-encryption#temporal-nexus "Direct link to Temporal Nexus") The Data Converter works the same for a Nexus Operation as it does for other payloads sent between a Worker and Temporal Cloud. Both the caller and handler Workers must use compatible Data Converters to pass operation inputs and results between them. See [Nexus Payload Encryption & Data Converter](https://docs.temporal.io/nexus/security#payload-encryption-data-converter) for details. * [Codec Server setup](https://docs.temporal.io/production-deployment/data-encryption#codec-server-setup) * [API contract specifications](https://docs.temporal.io/production-deployment/data-encryption#api-contract-specifications) * [Endpoints](https://docs.temporal.io/production-deployment/data-encryption#endpoints) * [Headers](https://docs.temporal.io/production-deployment/data-encryption#headers) * [Request body](https://docs.temporal.io/production-deployment/data-encryption#request-body) * [CORS](https://docs.temporal.io/production-deployment/data-encryption#cors) * [Authorization](https://docs.temporal.io/production-deployment/data-encryption#authorization) * [Sample request/response](https://docs.temporal.io/production-deployment/data-encryption#sample-requestresponse) * [Set your Codec Server endpoints with Web UI and CLI](https://docs.temporal.io/production-deployment/data-encryption#set-your-codec-server-endpoints-with-web-ui-and-cli) * [Web UI](https://docs.temporal.io/production-deployment/data-encryption#web-ui) * [CLI](https://docs.temporal.io/production-deployment/data-encryption#cli) * [Working with Large Payloads](https://docs.temporal.io/production-deployment/data-encryption#working-with-large-payloads) * [Temporal Nexus](https://docs.temporal.io/production-deployment/data-encryption#temporal-nexus) --- # Embedding Temporal server as a Go library | Temporal Platform Documentation [Skip to main content](https://docs.temporal.io/self-hosted-guide/embedded-server#__docusaurus_skipToContent_fallback) On this page You can run Temporal server as an embedded Go library instead of deploying it as a separate service. This approach is useful for testing and development scenarios where you want to run Temporal in-process without managing external infrastructure. Not for production use Embedded deployments with SQLite are suitable for **testing and development only**. For production workloads, deploy Temporal as a service using [MySQL, PostgreSQL, or Cassandra](https://docs.temporal.io/temporal-service/persistence) as the persistence layer. Reference implementation[​](https://docs.temporal.io/self-hosted-guide/embedded-server#reference-implementation "Direct link to Reference implementation") ----------------------------------------------------------------------------------------------------------------------------------------------------------- The recommended way to run an embedded Temporal server is to use the Temporal CLI's dev server implementation as a reference. The CLI's [devserver package](https://github.com/temporalio/cli/tree/main/internal/devserver) provides a complete implementation that handles: * SQLite configuration and schema setup * Namespace creation * Service configuration * Port allocation You can study and adapt this implementation for your own embedded use case. Basic server API[​](https://docs.temporal.io/self-hosted-guide/embedded-server#basic-server-api "Direct link to Basic server API") ----------------------------------------------------------------------------------------------------------------------------------- The core API for embedding Temporal is `temporal.NewServer()`: import ( "go.temporal.io/server/temporal" "go.temporal.io/server/common/config")server, err := temporal.NewServer( temporal.ForServices(temporal.DefaultServices), temporal.WithConfig(cfg), temporal.InterruptOn(temporal.InterruptCh()),)if err != nil { log.Fatal(err)}if err := server.Start(); err != nil { log.Fatal(err)} The challenge is building the `config.Config` struct correctly, especially for SQLite which requires: 1. **Schema setup** - SQLite databases need schema initialization via `sqliteschema.SetupSchema()` 2. **Namespace creation** - Namespaces can be pre-created via `sqliteschema.CreateNamespaces()` 3. **Service configuration** - All four services (frontend, history, matching, worker) need proper port configuration Configuration from file[​](https://docs.temporal.io/self-hosted-guide/embedded-server#configuration-from-file "Direct link to Configuration from file") -------------------------------------------------------------------------------------------------------------------------------------------------------- For non-SQLite databases, you can load configuration from a YAML file: cfg, err := config.Load( config.WithConfigFile("/path/to/config.yaml"),)if err != nil { log.Fatal(err)}server, err := temporal.NewServer( temporal.ForServices(temporal.DefaultServices), temporal.WithConfig(cfg),) Or load from a directory with environment-specific files: cfg, err := config.Load( config.WithConfigDir("./config"), config.WithEnv("development"),) Server options reference[​](https://docs.temporal.io/self-hosted-guide/embedded-server#server-options-reference "Direct link to Server options reference") ----------------------------------------------------------------------------------------------------------------------------------------------------------- The `temporal.NewServer()` function accepts options to customize the server. See [Server Options Reference](https://docs.temporal.io/references/server-options) for the complete list. Key options include: | Option | Description | | --- | --- | | `ForServices([]string)` | Services to run (default: frontend, history, matching, worker) | | `WithConfig(*config.Config)` | Server configuration | | `WithLogger(log.Logger)` | Custom logger | | `WithAuthorizer(authorization.Authorizer)` | Custom authorization | | `WithClaimMapper(func)` | Role/claim mapping for auth | | `WithCustomMetricsHandler(metrics.Handler)` | Custom metrics handler | | `WithDynamicConfigClient(dynamicconfig.Client)` | Runtime configuration | | `InterruptOn(chan)` | Channel for graceful shutdown | SQLite limitations[​](https://docs.temporal.io/self-hosted-guide/embedded-server#sqlite-limitations "Direct link to SQLite limitations") ----------------------------------------------------------------------------------------------------------------------------------------- SQLite is intended for testing and development only: * **Single writer**: SQLite supports only one writer at a time, limiting write throughput * **No durability in memory mode**: In-memory mode loses data on restart * **Not scalable**: Cannot handle production workloads * **Single shard**: Use `NumHistoryShards: 1` for SQLite For production, use MySQL, PostgreSQL, or Cassandra with a properly scaled multi-node deployment. Examples[​](https://docs.temporal.io/self-hosted-guide/embedded-server#examples "Direct link to Examples") ----------------------------------------------------------------------------------------------------------- For complete working examples, see: * [Temporal CLI dev server](https://github.com/temporalio/cli/tree/main/internal/devserver) - Reference implementation for SQLite embedding * [samples-server repository](https://github.com/temporalio/samples-server) - Server extensibility examples: * [Authorizer](https://github.com/temporalio/samples-server/tree/main/extensibility/authorizer) - Custom authorization and claim mapping * [Metrics handler](https://github.com/temporalio/samples-server/tree/main/extensibility/metrics-handler) - Custom metrics handling * [TLS](https://github.com/temporalio/samples-server/tree/main/tls) - TLS configuration for secure communication * [Docker Compose](https://github.com/temporalio/samples-server/tree/main/compose) - Database configurations (PostgreSQL, MySQL, Cassandra) Related[​](https://docs.temporal.io/self-hosted-guide/embedded-server#related "Direct link to Related") -------------------------------------------------------------------------------------------------------- * [Server Options Reference](https://docs.temporal.io/references/server-options) * [Deployment](https://docs.temporal.io/self-hosted-guide/deployment) * [Visibility Storage](https://docs.temporal.io/self-hosted-guide/visibility) * [Reference implementation](https://docs.temporal.io/self-hosted-guide/embedded-server#reference-implementation) * [Basic server API](https://docs.temporal.io/self-hosted-guide/embedded-server#basic-server-api) * [Configuration from file](https://docs.temporal.io/self-hosted-guide/embedded-server#configuration-from-file) * [Server options reference](https://docs.temporal.io/self-hosted-guide/embedded-server#server-options-reference) * [SQLite limitations](https://docs.temporal.io/self-hosted-guide/embedded-server#sqlite-limitations) * [Examples](https://docs.temporal.io/self-hosted-guide/embedded-server#examples) * [Related](https://docs.temporal.io/self-hosted-guide/embedded-server#related) --- # Task Queue Priority and Fairness | Temporal Platform Documentation [Skip to main content](https://docs.temporal.io/develop/task-queue-priority-fairness#__docusaurus_skipToContent_fallback) On this page [Task Queue Priority](https://docs.temporal.io/develop/task-queue-priority-fairness#task-queue-priority) and [Task Queue Fairness](https://docs.temporal.io/develop/task-queue-priority-fairness#task-queue-fairness) are two ways to manage the distribution of work within Task Queues. Priority focuses on how Tasks are prioritized within a single Task Queue. Fairness aims to prevent one set of Tasks from blocking others within the same priority level. You can use Priority and Fairness individually within your Task Queues or you can use them together for more complex scenarios. Support, stability, and dependency info Priority and Fairness are currently in [Public Preview](https://docs.temporal.io/evaluate/development-production-features/release-stages#public-preview) . Priority is a free feature in Temporal Cloud and for self-hosted Temporal Services. Fairness will be a **paid** feature in Temporal Cloud and billing will be enabled in the near future. You can find more details about the pricing structure for Fairness [here](https://docs.temporal.io/cloud/actions#fairness) . Task Queue Priority[​](https://docs.temporal.io/develop/task-queue-priority-fairness#task-queue-priority "Direct link to Task Queue Priority") ----------------------------------------------------------------------------------------------------------------------------------------------- **Task Queue Priority** lets you control the execution order of Workflows, Activities, and Child Workflows based on assigned priority values within a _single_ Task Queue. Each priority level acts as a sub-queue that separates Tasks within a single Task Queue so that high priority Tasks can cut in front of low priority Tasks. [![Detailed view of how Priority works](https://docs.temporal.io/img/develop/task-queue-priority-fairness/priority-details.png)](https://docs.temporal.io/img/develop/task-queue-priority-fairness/priority-details.png) ### When to use Priority[​](https://docs.temporal.io/develop/task-queue-priority-fairness#when-to-use-priority "Direct link to When to use Priority") If you need a way to specify the order your Tasks execute in, you can use Priority to manage that. Priority lets you differentiate between your Tasks, like batch and real-time Tasks, so that you can use a single pool of Workers for efficient resource allocation, while ensuring real-time Tasks are processed ahead of batch Tasks. You can also use this as a way to run urgent Tasks immediately and override others. For example, if you are running an e-commerce platform, you may want to process payment related Tasks before less time-sensitive Tasks like internal inventory management. ### How to use Priority[​](https://docs.temporal.io/develop/task-queue-priority-fairness#how-to-use-priority "Direct link to How to use Priority") Priority is available for both self-hosted Temporal instances and Temporal Cloud and it will be automatically enabled. If you apply priority keys, the feature will be in use. info You can toggle Priority enablement on a Task Queue, Namespace, or globally by setting the dynamic config `matching.useNewMatcher` to `true` or `false`. You can select a priority level by setting the _priority key_ to a value within the integer range `[1,5]`. A lower value implies higher priority, so `1` is the highest priority level. If you don't specify a Priority, a Task defaults to a Priority of `3`. Activities will inherit the priority level of their Workflow if a separate Activity priority level isn't set. When you set a priority level within your Task Queues, this means that they will **all** be processed in priority order. For example, all of your priority level `1` Tasks will execute before your priority level `2` Tasks and so on. So your lower priority Tasks will be blocked until the higher priority Tasks finish running. Tasks with the same priority level are scheduled to run in first-in-first-out (FIFO) order. If you need more flexibility to allocate resources to Tasks of the same type, like processing payments for multiple e-commerce platforms, check out [the Fairness section](https://docs.temporal.io/develop/task-queue-priority-fairness#task-queue-fairness) . You can do this via the Temporal CLI where you set the priority key parameter on a Workflow: temporal workflow start \ --type ChargeCustomer \ --task-queue my-task-queue \ --workflow-id my-workflow-id \ --input '{"customerId":"12345"}' \ --priority-key 1 Or choose your SDK below to see an example of setting priority for your Workflows: * Go * Java * Python * TypeScript * PHP * .NET * Ruby workflowOptions := client.StartWorkflowOptions{ ID: "my-workflow-id", TaskQueue: "my-task-queue", Priority: temporal.Priority{PriorityKey: 5},}we, err := c.ExecuteWorkflow(context.Background(), workflowOptions, MyWorkflow) WorkflowOptions options = WorkflowOptions.newBuilder() .setTaskQueue("my-task-queue") .setPriority(Priority.newBuilder().setPriorityKey(5).build()) .build();WorkflowClient client = WorkflowClient.newInstance(service); MyWorkflow workflow =client.newWorkflowStub(MyWorkflow.class, options); workflow.run(); await client.start_workflow( MyWorkflow.run, args="hello", id="my-workflow-id", task_queue="my-task-queue", priority=Priority(priority_key=1),) **TypeScript** example coming soon. **PHP** example coming soon. var handle = await Client.StartWorkflowAsync( (MyWorkflow wf) => wf.RunAsync("hello"), new StartWorkflowOptions( id: "my-workflow-id", taskQueue: "my-task-queue" ) { Priority = new Priority(1), }); **Ruby** example coming soon. Choose your SDK below to see an example of setting priority for your Activities: * Go * Java * Python * TypeScript * PHP * .NET * Ruby ao := workflow.ActivityOptions{ StartToCloseTimeout: time.Minute, Priority: temporal.Priority{PriorityKey: 3},}ctx := workflow.WithActivityOptions(ctx, ao)err := workflow.ExecuteActivity(ctx, MyActivity).Get(ctx, nil) ActivityOptions options = ActivityOptions.newBuilder() .setStartToCloseTimeout(Duration.ofMinutes(1)) .setPriority(Priority.newBuilder().setPriorityKey(3).build()) .build();MyActivity activity = Workflow.newActivityStub(MyActivity.class, options); activity.perform(); await workflow.execute_activity( say_hello, "hi", priority=Priority(priority_key=3), start_to_close_timeout=timedelta(seconds=5),) **TypeScript** example coming soon. **PHP** example coming soon. await Workflow.ExecuteActivityAsync( () => SayHello("hi"), new() { StartToCloseTimeout = TimeSpan.FromSeconds(5), Priority = new(3), } ); **Ruby** example coming soon. Choose your SDK below to see an example of setting priority for your Child Workflows: * Go * Java * Python * TypeScript * PHP * .NET * Ruby cwo := workflow.ChildWorkflowOptions{ WorkflowID: "child-workflow-id", TaskQueue: "child-task-queue", Priority: temporal.Priority{PriorityKey: 1},}ctx := workflow.WithChildOptions(ctx, cwo)err := workflow.ExecuteChildWorkflow(ctx, MyChildWorkflow).Get(ctx, nil) ChildWorkflowOptions childOptions = ChildWorkflowOptions.newBuilder() .setTaskQueue("child-task-queue") .setWorkflowId("child-workflow-id") .setPriority(Priority.newBuilder().setPriorityKey(1).build()) .build();MyChildWorkflow child = Workflow.newChildWorkflowStub(MyChildWorkflow.class, childOptions); child.run(); await workflow.execute_child_workflow( MyChildWorkflow.run, args="hello child", priority=Priority(priority_key=1),) **TypeScript** example coming soon. **PHP** example coming soon. await Workflow.ExecuteChildWorkflowAsync( (MyChildWorkflow wf) => wf.RunAsync("hello child"), new() { Priority = new(1) }); **Ruby** example coming soon. Task Queue Fairness[​](https://docs.temporal.io/develop/task-queue-priority-fairness#task-queue-fairness "Direct link to Task Queue Fairness") ----------------------------------------------------------------------------------------------------------------------------------------------- Task Queue Fairness lets you distribute Tasks based on _fairness keys_ and their _fairness weight_ within a Task Queue. The fairness keys are used to describe your Task structure. Each fairness key creates its own "virtual queue" within a Task Queue, allowing you to organize Tasks into logical groups like tenants, applications, or workload types. These virtual queues operate using a round-robin dispatch mechanism, meaning the system cycles through each fairness key in turn when selecting the next Task to dispatch. The round-robin approach essentially prevents any single fairness key from hogging Worker capacity. This ensures that Tasks from one fairness key can't completely block Tasks from other keys, even if one key has a much larger backlog than the others. When Workers become available, each fairness key gets an opportunity to dispatch Tasks rather than processing all Tasks from one key before moving to the next. This creates a fairer distribution where smaller tenants or lower-volume workloads still receive regular processing time, rather than waiting behind higher volume operations. The Tasks associated with each fairness key can be dispatched based on the _fairness weight_ that has been assigned to the key. There should only be one fairness weight assigned to each fairness key within a single Task Queue. Having multiple fairness weights on a fairness key will result in unspecific behavior. Fairness weight can be used to give more or less resources to a fairness key. By default, fairness keys have a fairness weight of 1.0. Tasks belonging to a fairness key with weight of 2.0 will be dispatched twice as often as other keys with a weight of 1.0. Using the fairness keys and their corresponding fairness weights lets you define levels with weighted capacities. For example, you can have free, basic, and premium levels and Fairness makes sure that an influx of premium Tasks don't overwhelm your resources and block free and basic Tasks. info When you start using fairness keys, it switches your active Task Queues to fairness mode. That means new Tasks are created with Fairness and the existing queued Tasks get processed before any of the new ones. ### When to use Fairness[​](https://docs.temporal.io/develop/task-queue-priority-fairness#when-to-use-fairness "Direct link to When to use Fairness") Fairness is intended to address common situations like: * Multi-tenant applications with big and small tenants where small tenants shouldn't be blocked by big ones. * Assigning Tasks to different capacity bands and then, for example, dispatching 80% from one band and 20% from another without limiting overall capacity when one band is empty. It sequences Tasks in the Task Queue probabilistically using a weighted distribution based on: * Fairness weights you set * The current backlog of Tasks * A data structure that tracks how you've distributed Tasks for different fairness keys As an example, imagine a workload with three tenants, _tenant-big_, _tenant-mid_, _tenant-small_, that have varying numbers of Tasks at all times. Your _tenant-big_ has a large number of Tasks that can overwhelm your Task Queue and prevent _tenant-mid_ and _tenant-small_ from running their Tasks. With Fairness, you can give each tenant a different fairness key to make sure _tenant-big_ doesn't use all of the Task Queue resources and block the others. In this case, _tenant-mid_ and _tenant-small_ will have Tasks run in between _tenant-big_ Tasks so that they are executed "fairly". Although, if all your Tasks can be dispatched immediately, then you don't need to use fairness. This same scenario can apply to batch jobs where certain jobs run more often than others or processing orders from multiple vendors where several vendors have the majority of the orders. Fairness applies at Task dispatch time based on information about the Tasks passing through the Task Queue and considers each Task as having equal cost until dispatch. It doesn't consider any Task execution that is currently being done by Workers. So if you look at Tasks being processed by Workers, you might not see "fairness" across tenants. For example, if you already have Tasks from _tenant-big_ being processed, when Tasks for _tenant-small_ are dispatched it may still look like _tenant-big_ is using the most resources. There are two ways to use Task Queue Fairness: without Priority and with Priority. ### How Fairness works[​](https://docs.temporal.io/develop/task-queue-priority-fairness#how-fairness-works "Direct link to How Fairness works") If you use Fairness without Priority, Tasks with different fairness keys will use a weighted distribution based on the fairness weights to allocate resources in the Task Queue. For example, say you have three fairness keys to describe customer tiers: _free-tier_, _basic-tier_, and _premium-tier_. You give _premium-tier_ a fairness weight of 5.0, _basic-tier_ a fairness weight of 3.0, and _free-tier_ a fairness weight of 2.0. With Fairness, that means 50% of the time _premium-tier_ Tasks dispatch, 30% of the time _basic-tier_ Tasks dispatch, and 20% of the time _free-tier_ Tasks dispatch from the Task Queue backlog. If there are Tasks in the Task Queue backlog that have the same fairness key, then they're dispatched in [FIFO order](https://docs.temporal.io/task-queue#task-ordering) . This is how you are able to ensure that one tier doesn't use all the resources and block other Tasks in the Task Queue backlog from dispatching. When you update fairness keys or fairness weights, the Task Queues will only reflect these changes for Tasks that haven't dispatched yet. [![High-level of how fairness works](https://docs.temporal.io/img/develop/task-queue-priority-fairness/fairness-details.png)](https://docs.temporal.io/img/develop/task-queue-priority-fairness/fairness-details.png) Tasks that do not have a `fairness_key` set are grouped together under an implicit empty-string key. All unkeyed Tasks share this single default bucket and participate in the same round-robin dispatch alongside named fairness keys, with a default weight of 1.0. This means Fairness adoption can be incremental: you can assign fairness keys to some tenants but not others. Unkeyed Tasks do not bypass Fairness; they compete as one group alongside all explicitly keyed Tasks. ### Using Fairness and Priority together[​](https://docs.temporal.io/develop/task-queue-priority-fairness#using-fairness-and-priority-together "Direct link to Using Fairness and Priority together") When you use Fairness and Priority together, Priority determines which sub-queue Tasks go into. A single Task Queue with Priority implemented will have different sub-queues based on priority levels. Fairness will apply to the Tasks within each priority level. [![High-level of how priority and fairness work together](https://docs.temporal.io/img/develop/task-queue-priority-fairness/priority-fairness.png)](https://docs.temporal.io/img/develop/task-queue-priority-fairness/priority-fairness.png) ### How to use Fairness[​](https://docs.temporal.io/develop/task-queue-priority-fairness#how-to-use-fairness "Direct link to How to use Fairness") Fairness is available for both self-hosted Temporal instances and Temporal Cloud. If you start using _fairness keys_ in your API calls, it will automatically be enabled in Temporal Cloud. If you're self-hosting Temporal, use the latest pre-release development server and set `matching.useNewMatcher` and `matching.enableFairness` to `true` in the [dynamic config](https://github.com/temporalio/temporal/blob/a3a53266c002ae33b630a41977274f8b5b587031/common/dynamicconfig/constants.go#L1345-L1348) on the relevant Task Queues or Namespaces. You'll also need to set `matching.enableMigration` to `true` in order to support draining Tasks in existing backlogs after Fairness is enabled. Enabling `matching.useNewMatcher` and `matching.enableFairness` is only applicable for self-hosted Temporal instances. There is a toggle coming to Temporal Cloud soon to enable Priority and Fairness at the Namespace level. info Fairness will be a paid feature in Temporal Cloud and billing will be enabled in the near future. You will be notified before billing is enabled for your Namespaces. When Temporal begins billing for this feature, you will be able to enable or disable Fairness at the Namespace level and billing will be disabled at the next calendar hour after it is disabled. You can do this via the Temporal CLI where you set the fairness key and weight parameters for your Workflow: temporal workflow start \ --type ChargeCustomer \ --task-queue my-task-queue \ --workflow-id my-workflow-id \ --input '{"customerId":"12345"}' \ --priority-key 1 \ --fairness-key a-key \ --fairness-weight 3.14 Or choose your SDK below to see an example of setting fairness for your Workflows: * Go * Java * Python * TypeScript * PHP * .NET * Ruby workflowOptions := client.StartWorkflowOptions{ ID: "my-workflow-id", TaskQueue: "my-task-queue", Priority: temporal.Priority{ PriorityKey: 1, FairnessKey: "a-key", FairnessWeight: 3.14, },}we, err := c.ExecuteWorkflow(context.Background(), workflowOptions, MyWorkflow) WorkflowOptions options = WorkflowOptions.newBuilder() .setTaskQueue("my-task-queue") .setPriority(Priority.newBuilder().setPriorityKey(5).setFairnessKey("a-key").setFairnessWeight(3.14).build()) .build();WorkflowClient client = WorkflowClient.newInstance(service);MyWorkflow workflow = client.newWorkflowStub(MyWorkflow.class, options);workflow.run(); await client.start_workflow( MyWorkflow.run, args="hello", id="my-workflow-id", task_queue="my-task-queue", priority=Priority(priority_key=3, fairness_key="a-key", fairness_weight=3.14),) const handle = await startWorkflow(workflows.priorityWorkflow, { args: [false, 1], priority: { priorityKey: 3, fairnessKey: 'a-key', fairnessWeight: 3.14 },}); **PHP** example coming soon. var handle = await Client.StartWorkflowAsync( (MyWorkflow wf) => wf.RunAsync("hello"), new StartWorkflowOptions( id: "my-workflow-id", taskQueue: "my-task-queue" ) { Priority = new Priority( priorityKey: 3, fairnessKey: "a-key", fairnessWeight: 3.14 ) }); client.start_workflow( MyWorkflow, "input-arg", id: "my-workflow-id", task_queue: "my-task-queue", priority: Temporalio::Priority.new( priority_key: 3, fairness_key: "a-key", fairness_weight: 3.14 )) Choose your SDK below to see an example of setting fairness for your Activities: * Go * Java * Python * TypeScript * PHP * .NET * Ruby ao := workflow.ActivityOptions{ StartToCloseTimeout: time.Minute, Priority: temporal.Priority{ PriorityKey: 1, FairnessKey: "a-key", FairnessWeight: 3.14, },}ctx := workflow.WithActivityOptions(ctx, ao)err := workflow.ExecuteActivity(ctx, MyActivity).Get(ctx, nil) ActivityOptions options = ActivityOptions.newBuilder() .setStartToCloseTimeout(Duration.ofMinutes(1)) .setPriority(Priority.newBuilder().setPriorityKey(3).setFairnessKey("a-key").setFairnessWeight(3.14).build()) .build();MyActivity activity = Workflow.newActivityStub(MyActivity.class, options);activity.perform(); await workflow.execute_activity( say_hello, "hi", priority=Priority(priority_key=3, fairness_key="a-key", fairness_weight=3.14), start_to_close_timeout=timedelta(seconds=5),) const handle = await startWorkflow(workflows.priorityWorkflow, { args: [false, 1], priority: { priorityKey: 3, fairnessKey: 'a-key', fairnessWeight: 3.14 },}); **PHP** example coming soon. var handle = await Client.StartWorkflowAsync( (MyWorkflow wf) => wf.RunAsync("hello"), new StartWorkflowOptions( id: "my-workflow-id", taskQueue: "my-task-queue" ) { Priority = new Priority( priorityKey: 3, fairnessKey: "a-key", fairnessWeight: 3.14 ) }); client.start_activity( MyActivity, "input-arg", id: "my-workflow-id", task_queue: "my-task-queue", priority: Temporalio::Priority.new( priority_key: 3, fairness_key: "a-key", fairness_weight: 3.14 )) Choose your SDK below to see an example of setting fairness for your Child Workflows: * Go * Java * Python * TypeScript * PHP * .NET * Ruby cwo := workflow.ChildWorkflowOptions{ WorkflowID: "child-workflow-id", TaskQueue: "child-task-queue", Priority: temporal.Priority{ PriorityKey: 1, FairnessKey: "a-key", FairnessWeight: 3.14, },}ctx := workflow.WithChildOptions(ctx, cwo)err := workflow.ExecuteChildWorkflow(ctx, MyChildWorkflow).Get(ctx, nil) ChildWorkflowOptions childOptions = ChildWorkflowOptions.newBuilder() .setTaskQueue("child-task-queue") .setWorkflowId("child-workflow-id") .setPriority(Priority.newBuilder().setPriorityKey(1).setFairnessKey("a-key").setFairnessWeight(3.14).build()) .build();MyChildWorkflow child = Workflow.newChildWorkflowStub(MyChildWorkflow.class, childOptions);child.run(); await workflow.execute_child_workflow( MyChildWorkflow.run, args="hello child", priority=Priority(priority_key=3, fairness_key="a-key", fairness_weight=3.14),) const handle = await startChildWorkflow(workflows.priorityWorkflow, { args: [false, 1], priority: { priorityKey: 3, fairnessKey: 'a-key', fairnessWeight: 3.14 },}); **PHP** example coming soon. var handle = await Client.StartWorkflowAsync( (MyWorkflow wf) => wf.RunAsync("hello"), new StartWorkflowOptions( id: "my-workflow-id", taskQueue: "my-task-queue" ) { Priority = new Priority( priorityKey: 3, fairnessKey: "a-key", fairnessWeight: 3.14 ) }); client.start_child_workflow( MyChildWorkflow, "input-arg", id: "my-child-workflow-id", task_queue: "my-task-queue", priority: Temporalio::Priority.new( priority_key: 3, fairness_key: "a-key", fairness_weight: 3.14 )) ### Set rate limits at the Task Queue level[​](https://docs.temporal.io/develop/task-queue-priority-fairness#set-rate-limits-at-the-task-queue-level "Direct link to Set rate limits at the Task Queue level") When you're starting to scale your Temporal Services, you may decide to set [Requests Per Second (RPS)](https://docs.temporal.io/references/dynamic-configuration#service-level-rps-limits) limits to test your workload or experiment with provisioning benchmarks. You can set the RPS limit at the Task Queue level with `queue-rps-limit` in the CLI. The whole queue rate limits the dispatch rate of Tasks regardless of the fairness key. Tasks won't be dispatched faster than the specified limit when averaged over a few seconds, although you may see small bursts due to partitioning. note The whole queue rate limit is the same feature that's available through the Worker Options in the SDKs. If it's set through the API, that limit takes precedence over the limit set through Worker Options. If you want to make sure that a specific fairness key has limits to throttle Tasks, you can also set an RPS limit based on fairness keys with `fairness-key-rps-limit-default` in the CLI. This could be how you distinguish customer tiers in a way that only allows a defined number of Tasks to be processed by that tier. temporal task-queue config set \ --task-queue my-task-queue \ --task-queue-type activity \ --namespace my-namespace \ --queue-rps-limit 500 \ --queue-rps-limit-reason "overall limit" \ --fairness-key-rps-limit-default 33.3 \ --fairness-key-rps-limit-reason "per-key limit" The per-fairness-key rate limit works in conjunction with Task Queue Fairness. If you think of Fairness as dividing the queue into one virtual queue for each key, then the per-fairness-key rate limit is a limit on each individual virtual queue. Some important notes on the per-fairness-key limit: * The whole queue limit and per-fairness-key limit may be set independently: none, one or the other, or both may be set. If both are set, then the more restrictive one applies. * The per-fairness-key limit for a key is scaled by the fairness weight assigned to that key. So if the per-fairness-key limit for a queue is set to 10, then all keys with the default weight (1.0) will have a limit of 10 tasks/second. But if a particular key is given a weight of 2.5, then the per-key rate limit for that key will be 25 tasks/second. * Since the dispatch rate for each key should be proportional to its weight, if any key is hitting the per-key limit, then nearly all of them are. The way it works is if the next Task to be dispatched hits the per-key limit, then dispatch will wait until it can go. * Usually there isn't actually any blocking, but there can be when the fairness weight for a key is changed between when a Task is scheduled and when it's dispatched. If the fairness weight for a key is lowered, for example, the new lower per-key rate limit will be respected. Since those Tasks were originally scheduled with the higher rate, they will block other Tasks as they're dispatched. This limitation will be improved in the future. ### Fairness weight overrides[​](https://docs.temporal.io/develop/task-queue-priority-fairness#fairness-weight-overrides "Direct link to Fairness weight overrides") In many cases, supplying the weight of a fairness key along with the key itself is straightforward. In some situations, it's convenient to leave the weights out of client code and instead control the weights of a small number of keys through a config API. You can override the weights of up to 1000 keys through the config API. When an override is set for a key, the weight attached to the Task, through Workflow or Activity priority metadata, will be ignored, and the overridden weight will be used instead. Weight overrides are stored per Task Queue, including type, so they must be set for both Workflow and Activity Task Queues to take effect for both. ### Limitations of Fairness[​](https://docs.temporal.io/develop/task-queue-priority-fairness#limitations-of-fairness "Direct link to Limitations of Fairness") * There isn't a limit on the number of fairness keys you can use, but their accuracy can degrade as you add more. * Task Queues are internally [partitioned](https://docs.temporal.io/task-queue#task-ordering) and Tasks are distributed to partitions randomly. This could interfere with fairness. Depending on your use case, you can reach out to Temporal Support to get your Task Queues set to a single partition. * The fairness weight applies at schedule time, not at dispatch time. So it only affects newly-scheduled Tasks, not currently backlogged ones. This means if you need to throttle a single fairness key in the existing backlog of Tasks, you won't be able to. * When you use Worker Versioning and you're moving Workflows from one version to another, Priority will still apply between versions. Fairness isn't guaranteed between versions. For example, you may have Tasks that were originally queued on Worker version _alpha_, Tasks that were queued on Worker version _beta_, and some Tasks were moved from _alpha_ to _beta_. Fairness is only guaranteed when Tasks are originally queued on the same Worker version. So there might be some discrepancies on the Tasks moved from _alpha_ to _beta_. * Backlogged tasks are durable and unaffected by server restarts. Fairness ordering is preserved across restarts for the most active keys; less active keys may briefly dispatch new tasks ahead of their existing backlog until ordering normalizes. * [Task Queue Priority](https://docs.temporal.io/develop/task-queue-priority-fairness#task-queue-priority) * [When to use Priority](https://docs.temporal.io/develop/task-queue-priority-fairness#when-to-use-priority) * [How to use Priority](https://docs.temporal.io/develop/task-queue-priority-fairness#how-to-use-priority) * [Task Queue Fairness](https://docs.temporal.io/develop/task-queue-priority-fairness#task-queue-fairness) * [When to use Fairness](https://docs.temporal.io/develop/task-queue-priority-fairness#when-to-use-fairness) * [How Fairness works](https://docs.temporal.io/develop/task-queue-priority-fairness#how-fairness-works) * [Using Fairness and Priority together](https://docs.temporal.io/develop/task-queue-priority-fairness#using-fairness-and-priority-together) * [How to use Fairness](https://docs.temporal.io/develop/task-queue-priority-fairness#how-to-use-fairness) * [Set rate limits at the Task Queue level](https://docs.temporal.io/develop/task-queue-priority-fairness#set-rate-limits-at-the-task-queue-level) * [Fairness weight overrides](https://docs.temporal.io/develop/task-queue-priority-fairness#fairness-weight-overrides) * [Limitations of Fairness](https://docs.temporal.io/develop/task-queue-priority-fairness#limitations-of-fairness) --- # What is Temporal? | Temporal Platform Documentation [Skip to main content](https://docs.temporal.io/temporal#__docusaurus_skipToContent_fallback) On this page Temporal is a scalable and reliable runtime for durable function executions called [Temporal Workflow Executions](https://docs.temporal.io/workflow-execution) . Said another way, it's a platform that guarantees the [Durable Execution](https://docs.temporal.io/temporal#durable-execution) of your application code. It enables you to develop as if failures don't even exist. Your application will run reliably even if it encounters problems, such as network outages or server crashes, which would be catastrophic for a typical application. The Temporal Platform handles these types of problems, allowing you to focus on the business logic, instead of writing application code to detect and recover from failures. ![The Temporal System](https://docs.temporal.io/diagrams/temporal-system-simple.svg) The Temporal System Durable Execution[​](https://docs.temporal.io/temporal#durable-execution "Direct link to Durable Execution") ------------------------------------------------------------------------------------------------------------- Durable Execution in the context of Temporal refers to the ability of a Workflow Execution to maintain its state and progress even in the face of failures, crashes, or server outages. This is achieved through Temporal's use of an [Event History](https://docs.temporal.io/workflow-execution/event#event-history) , which records the state of a Workflow Execution at each step. If a failure occurs, the Workflow Execution can resume from the last recorded event, ensuring that progress isn't lost. This durability is a key feature of Temporal Workflow Executions, making them reliable and resilient. It enables application code to execute effectively once and to completion, regardless of whether it takes seconds or years. What is the Temporal Platform?[​](https://docs.temporal.io/temporal#temporal-platform "Direct link to What is the Temporal Platform?") --------------------------------------------------------------------------------------------------------------------------------------- The Temporal Platform consists of a [Temporal Service](https://docs.temporal.io/temporal-service) and [Worker Processes](https://docs.temporal.io/workers#worker-process) . Together these components create a runtime for Workflow Executions. The Temporal Platform consists of a supervising software typically called the [Temporal Service](https://docs.temporal.io/temporal-service)  and application code bundled as Worker Processes. Together these components create a runtime for your application. ![The Temporal Platform](https://docs.temporal.io/diagrams/temporal-platform-simple.svg) The Temporal Platform A Temporal Service consists of the [Temporal Server](https://github.com/temporalio/temporal) , written in Go, and a database. Our software as a service (SaaS) offering, Temporal Cloud, offers an alternative to hosting the Temporal Service yourself. Worker Processes are hosted and operated by you and execute your code. Workers run using one of our SDKs. ![Basic component topology of the Temporal Platform](https://docs.temporal.io/diagrams/temporal-platform-component-topology.svg) Basic component topology of the Temporal Platform What is a Temporal Application?[​](https://docs.temporal.io/temporal#temporal-application "Direct link to What is a Temporal Application?") -------------------------------------------------------------------------------------------------------------------------------------------- A Temporal Application is a set of [Temporal Workflow Executions](https://docs.temporal.io/workflow-execution) . Each Temporal Workflow Execution has exclusive access to its local state, executes concurrently to all other Workflow Executions, and communicates with other Workflow Executions and the environment via message passing. A Temporal Application can consist of millions to billions of Workflow Executions. Workflow Executions are lightweight A Workflow Execution consumes few compute resources; in fact, if a Workflow Execution is suspended, such as when it is in a waiting state, the Workflow Execution consumes no compute resources at all. **Reentrant Process** A Temporal Workflow Execution is a Reentrant Process. A Reentrant Process is resumable, recoverable, and reactive. * Resumable: Ability of a process to continue execution after execution was suspended on an _awaitable_. * Recoverable: Ability of a process to continue execution after execution was suspended on a _failure_. * Reactive: Ability of a process to react to external events. Therefore, a Temporal Workflow Execution executes a [Temporal Workflow Definition](https://docs.temporal.io/workflow-definition) , also called a Temporal Workflow Function, your application code, exactly once and to completion—whether your code executes for seconds or years, in the presence of arbitrary load and arbitrary failures. What is a Failure?[​](https://docs.temporal.io/temporal#failure "Direct link to What is a Failure?") ----------------------------------------------------------------------------------------------------- [Temporal Failures](https://docs.temporal.io/references/failures) are representations (in the SDKs and Event History) of various types of errors that occur in the system. Failure handling is an essential part of development. For more information, including the difference between application-level and platform-level failures, see [Handling Failure From First Principles](https://dominik-tornow.medium.com/handling-failures-from-first-principles-1ed976b1b869) . For the practical application of those concepts in Temporal, see [Failure Handling in Practice](https://temporal.io/blog/failure-handling-in-practice) . For languages that throw (or raise) errors (or exceptions), throwing an error that is not a Temporal Failure from a Workflow fails the Workflow Task (and the Task will be retried until it succeeds), whereas throwing a Temporal Failure (or letting a Temporal Failure propagate from Temporal calls, like an [Activity Failure](https://docs.temporal.io/references/failures#activity-failure) from an Activity call) fails the Workflow Execution. For more information, see [Application Failure](https://docs.temporal.io/references/failures#application-failure) . * [Durable Execution](https://docs.temporal.io/temporal#durable-execution) * [What is the Temporal Platform?](https://docs.temporal.io/temporal#temporal-platform) * [What is a Temporal Application?](https://docs.temporal.io/temporal#temporal-application) * [What is a Failure?](https://docs.temporal.io/temporal#failure) --- # Self-hosted Multi-Cluster Replication | Temporal Platform Documentation [Skip to main content](https://docs.temporal.io/self-hosted-guide/multi-cluster-replication#__docusaurus_skipToContent_fallback) On this page Multi-Cluster Replication is a feature which asynchronously replicates Workflow Executions from active Clusters to other passive Clusters, for backup and state reconstruction. When necessary, for higher availability, Cluster operators can failover to any of the backup Clusters. Temporal's Multi-Cluster Replication feature is considered **experimental** and not subject to normal [versioning and support policy](https://docs.temporal.io/temporal-service/temporal-server#versions-and-support) . Temporal automatically forwards Start, Signal, and Query requests to the active Cluster. This feature must be enabled through a Dynamic Config flag per [Global Namespace](https://docs.temporal.io/global-namespace) . When the feature is enabled, Tasks are sent to the Parent Task Queue partition that matches that Namespace, if it exists. All Visibility APIs can be used against active and standby Clusters. This enables [Temporal UI](https://docs.temporal.io/web-ui) to work seamlessly for Global Namespaces. Applications making API calls directly to the Temporal Visibility API continue to work even if a Global Namespace is in standby mode. However, they might see a lag due to replication delay when querying the Workflow Execution state from a standby Cluster. #### Namespace Versions[​](https://docs.temporal.io/self-hosted-guide/multi-cluster-replication#namespace-versions "Direct link to Namespace Versions") A _version_ is a concept in Multi-Cluster Replication that describes the chronological order of events per Namespace. With Multi-Cluster Replication, all Namespace change events and Workflow Execution History events are replicated asynchronously for high throughput. This means that data across clusters is **not** strongly consistent. To guarantee that Namespace data and Workflow Execution data will achieve eventual consistency (especially when there is a data conflict during a failover), a **version** is introduced and attached to Namespaces. All Workflow Execution History entries generated in a Namespace will also come with the version attached to that Namespace. All participating Clusters are pre-configured with a unique initial version and a shared version increment: * `initial version < shared version increment` When performing failover for a Namespace from one Cluster to another Cluster, the version attached to the Namespace will be changed by the following rule: * for all versions which follow `version % (shared version increment) == (active cluster's initial version)`, find the smallest version which has `version >= old version in namespace` When there is a data conflict, a comparison will be made and Workflow Execution History entries with the highest version will be considered the source of truth. When a cluster is trying to mutate a Workflow Execution History, the version will be checked. A cluster can mutate a Workflow Execution History only if the following is true: * The version in the Namespace belongs to this cluster, i.e. `(version in namespace) % (shared version increment) == (this cluster's initial version)` * The version of this Workflow Execution History's last entry (event) is equal or less than the version in the Namespace, i.e. `(last event's version) <= (version in namespace)` Namespace version change example Assuming the following scenario: * Cluster A comes with initial version: 1 * Cluster B comes with initial version: 2 * Shared version increment: 10 T = 0: Namespace α is registered, with active Cluster set to Cluster A namespace α's version is 1all workflows events generated within this namespace, will come with version 1 T = 1: namespace β is registered, with active Cluster set to Cluster B namespace β's version is 2all workflows events generated within this namespace, will come with version 2 T = 2: Namespace α is updated to with active Cluster set to Cluster B namespace α's version is 2all workflows events generated within this namespace, will come with version 2 T = 3: Namespace β is updated to with active Cluster set to Cluster A namespace β's version is 11all workflows events generated within this namespace, will come with version 11 #### Version history[​](https://docs.temporal.io/self-hosted-guide/multi-cluster-replication#version-history "Direct link to Version history") Version history is a concept which provides a high level summary of version information in regards to Workflow Execution History. Whenever there is a new Workflow Execution History entry generated, the version from Namespace will be attached. The Workflow Execution's mutable state will keep track of all history entries (events) and the corresponding version. Version history example (without data conflict) * Cluster A comes with initial version: 1 * Cluster B comes with initial version: 2 * Shared version increment: 10 T = 0: adding event with event ID == 1 & version == 1 View in both Cluster A & B | -------- | --------------- | --------------- | ------- || Events | Version History | | || -------- | --------------- | --------------- | ------- || Event ID | Event Version | Event ID | Version || -------- | ------------- | --------------- | ------- || 1 | 1 | 1 | 1 || -------- | ------------- | --------------- | ------- | T = 1: adding event with event ID == 2 & version == 1 View in both Cluster A & B | -------- | --------------- | --------------- | ------- || Events | Version History | | || -------- | --------------- | --------------- | ------- || Event ID | Event Version | Event ID | Version || -------- | ------------- | --------------- | ------- || 1 | 1 | 2 | 1 || 2 | 1 | | || -------- | ------------- | --------------- | ------- | T = 2: adding event with event ID == 3 & version == 1 View in both Cluster A & B | -------- | --------------- | --------------- | ------- || Events | Version History | | || -------- | --------------- | --------------- | ------- || Event ID | Event Version | Event ID | Version || -------- | ------------- | --------------- | ------- || 1 | 1 | 3 | 1 || 2 | 1 | | || 3 | 1 | | || -------- | ------------- | --------------- | ------- | T = 3: Namespace failover triggered, Namespace version is now 2 adding event with event ID == 4 & version == 2 View in both Cluster A & B | -------- | --------------- | --------------- | ------- || Events | Version History | | || -------- | --------------- | --------------- | ------- || Event ID | Event Version | Event ID | Version || -------- | ------------- | --------------- | ------- || 1 | 1 | 3 | 1 || 2 | 1 | 4 | 2 || 3 | 1 | | || 4 | 2 | | || -------- | ------------- | --------------- | ------- | T = 4: adding event with event ID == 5 & version == 2 View in both Cluster A & B | -------- | --------------- | --------------- | ------- || Events | Version History | | || -------- | --------------- | --------------- | ------- || Event ID | Event Version | Event ID | Version || -------- | ------------- | --------------- | ------- || 1 | 1 | 3 | 1 || 2 | 1 | 5 | 2 || 3 | 1 | | || 4 | 2 | | || 5 | 2 | | || -------- | ------------- | --------------- | ------- | Since Temporal is AP, during failover (change of active Temporal Cluster Namespace), there can exist cases where more than one Cluster can modify a Workflow Execution, causing divergence of Workflow Execution History. Below shows how the version history will look like under such conditions. Version history example (with data conflict) Below, shows version history of the same Workflow Execution in 2 different Clusters. * Cluster A comes with initial version: 1 * Cluster B comes with initial version: 2 * Cluster C comes with initial version: 3 * Shared version increment: 10 T = 0: View in both Cluster B & C | -------- | --------------- | --------------- | ------- || Events | Version History | | || -------- | --------------- | --------------- | ------- || Event ID | Event Version | Event ID | Version || -------- | ------------- | --------------- | ------- || 1 | 1 | 2 | 1 || 2 | 1 | 3 | 2 || 3 | 2 | | || -------- | ------------- | --------------- | ------- | T = 1: adding event with event ID == 4 & version == 2 in Cluster B | -------- | --------------- | --------------- | ------- || Events | Version History | | || -------- | --------------- | --------------- | ------- || Event ID | Event Version | Event ID | Version || -------- | ------------- | --------------- | ------- || 1 | 1 | 2 | 1 || 2 | 1 | 4 | 2 || 3 | 2 | | || 4 | 2 | | || -------- | ------------- | --------------- | ------- | T = 1: namespace failover to Cluster C, adding event with event ID == 4 & version == 3 in Cluster C | -------- | --------------- | --------------- | ------- || Events | Version History | | || -------- | --------------- | --------------- | ------- || Event ID | Event Version | Event ID | Version || -------- | ------------- | --------------- | ------- || 1 | 1 | 2 | 1 || 2 | 1 | 3 | 2 || 3 | 2 | 4 | 3 || 4 | 3 | | || -------- | ------------- | --------------- | ------- | T = 2: replication task from Cluster C arrives in Cluster B Note: below is a tree structure | -------------- | ------------- || Events | || -------------- | ------------- || Event ID | Event Version || ------------- | ------------- || 1 | 1 || 2 | 1 || 3 | 2 || ------------- | ------------- || | || ------------- | ------------- || | || -------------- | ------------- | | -------- | ------------- || Event ID | Event Version | | Event ID | Event Version || ------------- | ------------- | | -------- | ------------- || 4 | 2 | | 4 | 3 || -------------- | ------------- | | -------- | ------------- || --------------- | ----------- || Version History | || --------------- | ------------ || Event ID | Version || --------------- | ------------ || 2 | 1 || 3 | 2 || --------------- | ------------ || -------- | ------- | | -------- | ------- || Event ID | Version | | Event ID | Version || -------- | ------- | | -------- | ------- || 4 | 2 | | 4 | 3 || -------- | ------- | | -------- | ------- | T = 2: replication task from Cluster B arrives in Cluster C, same as above #### Conflict resolution[​](https://docs.temporal.io/self-hosted-guide/multi-cluster-replication#conflict-resolution "Direct link to Conflict resolution") When a Workflow Execution History diverges, proper conflict resolution is applied. In Multi-cluster Replication, Workflow Execution History Events are modeled as a tree, as shown in the second example in [Version History](https://docs.temporal.io/self-hosted-guide/multi-cluster-replication#version-history) . Workflow Execution Histories that diverge will have more than one history branch. Among all history branches, the history branch with the highest version is considered the `current branch` and the Workflow Execution's mutable state is a summary of the current branch. Whenever there is a switch between Workflow Execution History branches, a complete rebuild of the Workflow Execution's mutable state will occur. Temporal Multi-Cluster Replication relies on asynchronous replication of Events across Clusters, so in the case of a failover it is possible to have an Activity Task dispatched again to the newly active Cluster due to a replication task lag. This also means that whenever a Workflow Execution is updated after a failover by the new Cluster, any previous replication tasks for that Execution cannot be applied. This results in loss of some progress made by the Workflow Execution in the previous active Cluster. During such conflict resolution, Temporal re-injects any external Events like Signals in the new Event History before discarding replication tasks. Even though some progress could roll back during failovers, Temporal provides the guarantee that Workflow Executions won't get stuck and will continue to make forward progress. Activity Execution completions are not forwarded across Clusters. Any outstanding Activities will eventually time out based on the configuration. Your application should have retry logic in place so that the Activity gets retried and dispatched again to a Worker after the failover to the new Cluster. Handling this is similar to handling an Activity Task timeout caused by a Worker restarting. #### Zombie Workflows[​](https://docs.temporal.io/self-hosted-guide/multi-cluster-replication#zombie-workflows "Direct link to Zombie Workflows") There is an existing contract that for any Namespace and Workflow Id combination, there can be at most one run (Namespace + Workflow Id + Run Id) open / executing. Multi-cluster Replication aims to keep the Workflow Execution History as up-to-date as possible among all participating Clusters. Due to the nature of Multi-cluster Replication (for example, Workflow Execution History events are replicated asynchronously) different Runs (same Namespace and Workflow Id) can arrive at the target Cluster at different times, sometimes out of order, as shown below: +-----------+ +----------------+ +-----------+| Cluster A | | Network Layer | | Cluster B |+-----------+ +----------------+ +-----------+ | | | | Run 1 Replication | | |---------------------> | | | | | | Run 2 Replication | | |---------------------> | | | | | | | Run 2 Replication | | |---------------------> | | | | | | Run 1 Replication | | |---------------------> | | | | Because Run 2 appears in Cluster B first, Run 1 cannot be replicated as "runnable" due to the rule `at most one Run open` (see above), thus the "zombie" Workflow Execution state is introduced. A "zombie" state is one in which a Workflow Execution which cannot be actively mutated by a Cluster (assuming the corresponding Namespace is active in this Cluster). A zombie Workflow Execution can only be changed by a replication Task. Run 1 will be replicated similar to Run 2, except when Run 1's execution will become a "zombie" before Run 1 reaches completion. #### Workflow Task processing[​](https://docs.temporal.io/self-hosted-guide/multi-cluster-replication#workflow-task-processing "Direct link to Workflow Task processing") In the context of Multi-cluster Replication, a Workflow Execution's mutable state is an entity which tracks all pending tasks. Prior to the introduction of Multi-cluster Replication, Workflow Execution History entries (events) are from a single branch, and the Temporal Server will only append new entries (events) to the Workflow Execution History. After the introduction of Multi-cluster Replication, it is possible that a Workflow Execution can have multiple Workflow Execution History branches. Tasks generated according to one history branch may become invalidated by switching history branches during conflict resolution. Example: T = 0: task A is generated according to Event Id: 4, version: 2 | -------- | ------------- || Events || -------- | ------------- || Event ID | Event Version || -------- | ------------- || 1 | 1 || 2 | 1 || 3 | 2 || -------- | ------------- || || || -------- | ------------- || Event ID | Event Version || -------- | ------------- || 4 | 2 | <-- task A belongs to this event| -------- | ------------- | T = 1: conflict resolution happens, Workflow Execution's mutable state is rebuilt and history Event Id: 4, version: 3 is written down to persistence | --------- | ------------- || Events || --------- | ------------- || Event ID | Event Version || -------- | ------------- || 1 | 1 || 2 | 1 || 3 | 2 || -------- | ------------- || || --------- | ------------- || | | -------- | ------------- | | -------- | ------------- || Event ID | Event Version | | Event ID | Event Version || -------- | ------------- | | -------- | ------------- || 4 | 2 | <-- task A belongs to this event | 4 | 3 | <-- current branch / mutable state| -------- | ------------- | | -------- | ------------- | T = 2: task A is loaded. At this time, due to the rebuild of a Workflow Execution's mutable state (conflict resolution), Task A is no longer relevant (Task A's corresponding Event belongs to non-current branch). Task processing logic will verify both the Event Id and version of the Task against a corresponding Workflow Execution's mutable state, then discard task A. How to set up Multi-Cluster Replication[​](https://docs.temporal.io/self-hosted-guide/multi-cluster-replication#set-up-multi-cluster-replication "Direct link to How to set up Multi-Cluster Replication") ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- The [Multi-Cluster Replication](https://docs.temporal.io/self-hosted-guide/multi-cluster-replication) feature asynchronously replicates Workflow Execution Event Histories from active Clusters to other passive Clusters, and can be enabled by setting the appropriate values in the `clusterMetadata` section of your configuration file. 1. `enableGlobalNamespace` must be set to `true`. 2. `failoverVersionIncrement` has to be equal across connected Clusters. 3. `initialFailoverVersion` in each Cluster has to assign a different value. No equal value is allowed across connected Clusters. After the above conditions are satisfied, you can start to configure a multi-cluster setup. #### Set up Multi-Cluster Replication prior to v1.14[​](https://docs.temporal.io/self-hosted-guide/multi-cluster-replication#set-up-multi-cluster-replication-prior-to-v114 "Direct link to Set up Multi-Cluster Replication prior to v1.14") You can set this up with [`clusterMetadata` configuration](https://docs.temporal.io/references/configuration#clustermetadata) ; however, this is meant to be only a conceptual guide rather than a detailed tutorial. tip If you need help when setting up, please reach out to our [community Slack](https://temporalio.slack.com/) . Good places to start include the **#support-community** channel, searching through previous conversations, and asking our unusually excellent Temporal-trained large language model (visit **#ask-ai**). Need a Slack invitation? Here's an [invitation link](https://temporal.io/slack) . For example: # cluster AclusterMetadata: enableGlobalNamespace: true failoverVersionIncrement: 100 masterClusterName: "clusterA" currentClusterName: "clusterA" clusterInformation: clusterA: enabled: true initialFailoverVersion: 1 rpcAddress: "127.0.0.1:7233" clusterB: enabled: true initialFailoverVersion: 2 rpcAddress: "127.0.0.1:8233"# cluster BclusterMetadata: enableGlobalNamespace: true failoverVersionIncrement: 100 masterClusterName: "clusterA" currentClusterName: "clusterB" clusterInformation: clusterA: enabled: true initialFailoverVersion: 1 rpcAddress: "127.0.0.1:7233" clusterB: enabled: true initialFailoverVersion: 2 rpcAddress: "127.0.0.1:8233" #### Set up Multi-Cluster Replication in v1.14 and later[​](https://docs.temporal.io/self-hosted-guide/multi-cluster-replication#set-up-multi-cluster-replication-in-v114-and-later "Direct link to Set up Multi-Cluster Replication in v1.14 and later") You still need to set up local cluster [`clusterMetadata` configuration](https://docs.temporal.io/references/configuration#clustermetadata) For example: # cluster AclusterMetadata: enableGlobalNamespace: true failoverVersionIncrement: 100 masterClusterName: "clusterA" currentClusterName: "clusterA" clusterInformation: clusterA: enabled: true initialFailoverVersion: 1 rpcAddress: "127.0.0.1:7233"# cluster BclusterMetadata: enableGlobalNamespace: true failoverVersionIncrement: 100 masterClusterName: "clusterB" currentClusterName: "clusterB" clusterInformation: clusterB: enabled: true initialFailoverVersion: 2 rpcAddress: "127.0.0.1:8233" Then you can use the Temporal CLI tool to add cluster connections. All operations should be executed in both Clusters. # Add a clustertemporal operator cluster upsert --frontend_address="127.0.2.1:8233"# Disable connectionstemporal operator cluster upsert --frontend_address="localhost:8233" --enable_connection false# Delete connectionstemporal operator cluster remove --name="someClusterName" * [Namespace Versions](https://docs.temporal.io/self-hosted-guide/multi-cluster-replication#namespace-versions) * [Version history](https://docs.temporal.io/self-hosted-guide/multi-cluster-replication#version-history) * [Conflict resolution](https://docs.temporal.io/self-hosted-guide/multi-cluster-replication#conflict-resolution) * [Zombie Workflows](https://docs.temporal.io/self-hosted-guide/multi-cluster-replication#zombie-workflows) * [Workflow Task processing](https://docs.temporal.io/self-hosted-guide/multi-cluster-replication#workflow-task-processing) * [How to set up Multi-Cluster Replication](https://docs.temporal.io/self-hosted-guide/multi-cluster-replication#set-up-multi-cluster-replication) * [Set up Multi-Cluster Replication prior to v1.14](https://docs.temporal.io/self-hosted-guide/multi-cluster-replication#set-up-multi-cluster-replication-prior-to-v114) * [Set up Multi-Cluster Replication in v1.14 and later](https://docs.temporal.io/self-hosted-guide/multi-cluster-replication#set-up-multi-cluster-replication-in-v114-and-later) --- # Self-hosted Temporal Service defaults | Temporal Platform Documentation [Skip to main content](https://docs.temporal.io/self-hosted-guide/defaults#__docusaurus_skipToContent_fallback) Looking for Temporal Cloud defaults? See the [Temporal Cloud defaults and limits page](https://docs.temporal.io/cloud/limits) This page details many of the defaults coded into the Temporal Platform that can produce errors and warnings. Errors are hard limits that fail when reached. Warnings are soft limits that produce a warning log on the server side. info These limits might apply specifically to each Workflow Execution and do not pertain to the entire Temporal Platform or individual Namespaces. * **Identifiers:** By default, the maximum length for identifiers (such as Workflow Id, Workflow Type, and Task Queue name) is 1000 characters. * This is configurable with the `limit.maxIDLength` dynamic config variable, set to 255 in [this SQL example](https://github.com/temporalio/samples-server/blob/main/compose/dynamicconfig/development-sql.yaml) . * The character format is UTF-8. * **gRPC:** gRPC has a limit of 4 MB for [each message received](https://github.com/grpc/grpc/blob/v1.36.2/include/grpc/impl/codegen/grpc_types.h#L466) . * **Event batch size:** The `DefaultTransactionSizeLimit` limit is [4 MB](https://github.com/temporalio/temporal/pull/1363) . This is the largest transaction size allowed for the persistence of Event Histories. * **Blob size limit** for Payloads (including Workflow context and each Workflow and Activity argument and return value; _[source](https://github.com/temporalio/temporal/blob/v1.7.0/service/frontend/service.go#L133-L134) _): * Temporal warns at 256 KB: `Blob size exceeds limit.` * Temporal errors at 2 MB: `ErrBlobSizeExceedsLimit: Blob data size exceeds limit.` * Refer to [Troubleshoot blob size limit error](https://docs.temporal.io/troubleshooting/blob-size-limit-error) . * **Workflow Execution Update limits**: * A single Workflow Execution can have a maximum of 10 in-flight Updates and 2000 total Updates in History. * **History total size limit** (leading to a terminated Workflow Execution): * Temporal warns at 10 MB: [history size exceeds warn limit](https://github.com/temporalio/temporal/blob/v1.7.0/service/history/workflowExecutionContext.go#L1238) . * Temporal errors at 50 MB: [history size exceeds error limit](https://github.com/temporalio/temporal/blob/v1.7.0/service/history/workflowExecutionContext.go#L1204) . * This is configurable with [HistorySizeLimitError and HistorySizeLimitWarn](https://github.com/temporalio/temporal/blob/v1.7.0/service/history/configs/config.go#L380-L381) . * **History total count limit** (leading to a terminated Workflow Execution): * Temporal warns after 10,240 Events: [history size exceeds warn limit](https://github.com/temporalio/temporal/blob/v1.7.0/service/history/workflowExecutionContext.go#L1238) . * Temporal errors after 51,200 Events: [history size exceeds error limit](https://github.com/temporalio/temporal/blob/v1.7.0/service/history/workflowExecutionContext.go#L1204) . * This is configurable with [HistoryCountLimitError and HistoryCountLimitWarn](https://github.com/temporalio/temporal/blob/v1.7.0/service/history/configs/config.go#L382-L383) . * **Concurrent limit** * The following Commands are limited: * `ScheduleActivityTask` * `SignalExternalWorkflowExecution` * `RequestCancelExternalWorkflowExecution` * `StartChildWorkflowExecution` * These will fail if the concurrent pending count exceeds 2,000. For optimal performance, limit concurrent operations to 500 or fewer. This reduces Workflow's Event History size and decreases the loading time in the Web UI. * As of v1.21, the open source Temporal Service has a default limit of 2,000 pending Activities, Child Workflows, Signals, or Workflow cancellation requests, but you can override the limits in the dynamic configuration using these variables: * `limit.numPendingActivities.error` * `limit.numPendingSignals.error` * `limit.numPendingCancelRequests.error` * `limit.numPendingChildExecutions.error` * By default, [Batch jobs](https://docs.temporal.io/cli/batch) are limited to one job at a time. * [Custom Search Attributes limits](https://docs.temporal.io/search-attribute#custom-search-attribute-limits) For details on dynamic configuration keys, see [Dynamic configuration reference](https://docs.temporal.io/references/dynamic-configuration) . --- # Managing Namespaces | Temporal Platform Documentation [Skip to main content](https://docs.temporal.io/self-hosted-guide/namespaces#__docusaurus_skipToContent_fallback) On this page Open source Temporal This page covers namespace operations for **open source Temporal**. For core namespace concepts, see [Temporal Namespace](https://docs.temporal.io/namespaces) . For Temporal Cloud, see [Temporal Cloud Namespaces](https://docs.temporal.io/cloud/namespaces) . A [Namespace](https://docs.temporal.io/namespaces) is a unit of isolation within the Temporal Platform. Before you can run Workflows, you must register at least one Namespace with your Temporal Service. Create a Namespace[​](https://docs.temporal.io/self-hosted-guide/namespaces#create-a-namespace "Direct link to Create a Namespace") ------------------------------------------------------------------------------------------------------------------------------------ Registering a Namespace creates it on the Temporal Service. When you register a Namespace, you must set a [Retention Period](https://docs.temporal.io/temporal-service/temporal-server#retention-period) that determines how long closed Workflow execution history is kept. You can create Namespaces using: * **Temporal CLI** (recommended): [`temporal operator namespace create`](https://docs.temporal.io/cli/operator#create) * **Go SDK**: [`RegisterNamespace`](https://docs.temporal.io/develop/go/client/namespaces#register-namespace) * **Java SDK**: [`RegisterNamespace`](https://docs.temporal.io/develop/java/client/namespaces#register-namespace) * **TypeScript SDK**: [Namespace management](https://docs.temporal.io/develop/typescript/client/namespaces#register-namespace) ### The default Namespace[​](https://docs.temporal.io/self-hosted-guide/namespaces#the-default-namespace "Direct link to The default Namespace") If no Namespace is specified, SDKs and CLI use the `default` Namespace. You must register this Namespace before using it. For local development, the [`temporal server start-dev`](https://docs.temporal.io/cli/server#start-dev) command automatically creates the `default` Namespace. For all other deployment methods, create the `default` Namespace manually using the Temporal CLI: temporal operator namespace create --namespace default Namespace registration takes up to 15 seconds to complete. Wait for this process to finish before making calls to the Namespace. Manage Namespaces[​](https://docs.temporal.io/self-hosted-guide/namespaces#manage-namespaces "Direct link to Manage Namespaces") --------------------------------------------------------------------------------------------------------------------------------- Common namespace management operations: | Operation | CLI Command | Description | | --- | --- | --- | | List | [`temporal operator namespace list`](https://docs.temporal.io/cli/operator#list) | List all registered Namespaces | | Describe | [`temporal operator namespace describe`](https://docs.temporal.io/cli/operator#describe) | Get details for a Namespace | | Update | [`temporal operator namespace update`](https://docs.temporal.io/cli/operator#update) | Update Namespace configuration | | Delete | [`temporal operator namespace delete`](https://docs.temporal.io/cli/operator#delete) | Delete a Namespace and all its data | For SDK-based namespace management: * [Go SDK namespace management](https://docs.temporal.io/develop/go/client/namespaces#manage-namespaces) * [Java SDK namespace management](https://docs.temporal.io/develop/java/client/namespaces#manage-namespaces) * [TypeScript SDK namespace management](https://docs.temporal.io/develop/typescript/client/namespaces#manage-namespaces) ### Deprecate vs Delete[​](https://docs.temporal.io/self-hosted-guide/namespaces#deprecate-vs-delete "Direct link to Deprecate vs Delete") * **Deprecate**: Prevents new Workflow Executions from starting, but existing Workflows continue to run. * **Delete**: Removes the Namespace and all Workflow Executions. This is irreversible. Security[​](https://docs.temporal.io/self-hosted-guide/namespaces#security "Direct link to Security") ------------------------------------------------------------------------------------------------------ Use a custom [Authorizer](https://docs.temporal.io/self-hosted-guide/security#authorizer-plugin) on your Frontend Service to control who can create, update, or deprecate Namespaces. Without an Authorizer configured, Temporal uses the `nopAuthority` authorizer that allows all API calls unconditionally. For Temporal Cloud, [role-based access controls](https://docs.temporal.io/cloud/manage-access/roles-and-permissions#namespace-level-permissions) provide namespace-level authorization without custom configuration. Best practices[​](https://docs.temporal.io/self-hosted-guide/namespaces#best-practices "Direct link to Best practices") ------------------------------------------------------------------------------------------------------------------------ For namespace naming conventions, organizational patterns, and production safeguards, see [Namespace Best Practices](https://docs.temporal.io/best-practices/managing-namespace) . * [Create a Namespace](https://docs.temporal.io/self-hosted-guide/namespaces#create-a-namespace) * [The default Namespace](https://docs.temporal.io/self-hosted-guide/namespaces#the-default-namespace) * [Manage Namespaces](https://docs.temporal.io/self-hosted-guide/namespaces#manage-namespaces) * [Deprecate vs Delete](https://docs.temporal.io/self-hosted-guide/namespaces#deprecate-vs-delete) * [Security](https://docs.temporal.io/self-hosted-guide/namespaces#security) * [Best practices](https://docs.temporal.io/self-hosted-guide/namespaces#best-practices) --- # Temporal SDK metrics reference | Temporal Platform Documentation [Skip to main content](https://docs.temporal.io/references/sdk-metrics#__docusaurus_skipToContent_fallback) On this page SDK metrics The information on this page is relevant to [Temporal SDKs](https://docs.temporal.io/encyclopedia/temporal-sdks) . See [Cloud metrics](https://docs.temporal.io/cloud/metrics/) for metrics emitted by [Temporal Cloud](https://docs.temporal.io/cloud/overview) . See [Cluster metrics](https://docs.temporal.io/references/cluster-metrics) for metrics emitted by the [OSS Cluster](https://docs.temporal.io/temporal-service) . Some SDKs may emit metrics beyond what is listed in this SDK Metrics reference. Only metrics included in this Metrics reference have guaranteed, defined behavior. Other metrics are considered deprecated, inconsistent or experimental. The Temporal SDKs emit a set of metrics from Temporal Client usage and Worker Processes. * [How to emit metrics using the Go SDK](https://docs.temporal.io/develop/go/platform/observability#metrics) * [How to emit metrics using the Java SDK](https://docs.temporal.io/develop/java/platform/observability#metrics) * [How to emit metrics using the Python SDK](https://docs.temporal.io/develop/python/platform/observability#metrics) * [How to emit metrics using the TypeScript SDK](https://docs.temporal.io/develop/typescript/platform/observability#metrics) * [How to emit metrics using the .NET SDK](https://docs.temporal.io/develop/dotnet/platform/observability#metrics) * [How to emit metrics using the Ruby SDK](https://docs.temporal.io/develop/ruby/platform/observability#metrics) * [How to tune Worker performance based on metrics](https://docs.temporal.io/develop/worker-performance) All metrics are prefixed with `temporal_` before being exported to their configured destination. (The prefix has been removed in parts of this reference.) Currently, some metrics are specific to certain SDKs. TypeScript, Python, .NET, and Ruby SDK metrics are defined in the Core SDK. PHP and Go metrics are defined in the Go SDK. Java metrics are defined in the Java SDK. Metrics are defined in the following locations. * [Core SDK Worker metrics](https://github.com/temporalio/sdk-core/blob/master/crates/sdk-core/src/telemetry/metrics.rs) * [Core SDK Client metrics](https://github.com/temporalio/sdk-core/blob/master/crates/client/src/metrics.rs) * [Java SDK Worker metrics](https://github.com/temporalio/sdk-java/blob/master/temporal-sdk/src/main/java/io/temporal/worker/MetricsType.java) * [Java SDK Client metrics](https://github.com/temporalio/sdk-java/blob/master/temporal-serviceclient/src/main/java/io/temporal/serviceclient/MetricsType.java) * [Go SDK Worker and Client metrics](https://github.com/temporalio/sdk-go/blob/c32b04729cc7691f80c16f80eed7f323ee5ce24f/internal/common/metrics/constants.go) Metric units across SDKs The unit of measurement for metrics can vary based on which SDK they are being reported from: **Core-based SDKs:** Metrics of the type Histogram are measured in _milliseconds_ by default. This can be customized to use seconds for SDKs using [Core SDK](https://docs.temporal.io/glossary#core-sdk) . The Core SDK is a shared common core library used by several Temporal SDKs, including TypeScript, Python, and .NET. **Java and Go SDKs:** Metrics of the type Histogram are measured in _seconds_. Each metric may have some combination of the following keys attached to them: * `task-queue`: Task Queue that the Worker Entity is polling * `namespace`: Namespace the Worker is bound to * `poller_type`: One of the following: * `workflow_task` * `activity_task` * `nexus_task` (Go and Java only) * `sticky_workflow_task` * `worker_type`: One of the following: * `ActivityWorker` * `WorkflowWorker` * `LocalActivityWorker` (Go and Java only) * `NexusWorker` (Go and Java only) * `activity_type`: The name of the Activity Function the metric is associated with * `workflow_type`: The name of the Workflow Function the metric is associated with * `operation`: RPC method name; available for metrics related to Temporal Client gRPC requests Some keys may not be available in every SDK, and Histogram metrics may have different buckets in each SDK. | Metric name | Emitted by | Metric type | Availability | | --- | --- | --- | --- | | [temporal\_activity\_execution\_cancelled](https://docs.temporal.io/references/sdk-metrics#activity_execution_cancelled) | Worker | Counter | Java | | [temporal\_activity\_execution\_failed](https://docs.temporal.io/references/sdk-metrics#activity_execution_failed) | Worker | Counter | Core, Go, Java | | [temporal\_activity\_execution\_latency](https://docs.temporal.io/references/sdk-metrics#activity_execution_latency) | Worker | Histogram | Core, Go, Java | | [temporal\_activity\_poll\_no\_task](https://docs.temporal.io/references/sdk-metrics#activity_poll_no_task) | Worker | Counter | Core, Go, Java | | [temporal\_activity\_schedule\_to\_start\_latency](https://docs.temporal.io/references/sdk-metrics#activity_schedule_to_start_latency) | Worker | Histogram | Core, Go, Java | | [temporal\_activity\_succeed\_endtoend\_latency](https://docs.temporal.io/references/sdk-metrics#activity_succeed_endtoend_latency) | Worker | Histogram | Core, Go, Java | | [temporal\_activity\_task\_error](https://docs.temporal.io/references/sdk-metrics#activity_task_error) | Worker | Counter | Go | | [temporal\_corrupted\_signals](https://docs.temporal.io/references/sdk-metrics#corrupted_signals) | Worker | Counter | Go, Java | | [temporal\_local\_activity\_execution\_cancelled](https://docs.temporal.io/references/sdk-metrics#local_activity_execution_cancelled) | Worker | Counter | Core, Go, Java | | [temporal\_local\_activity\_execution\_failed](https://docs.temporal.io/references/sdk-metrics#local_activity_execution_failed) | Worker | Counter | Core, Go, Java | | [temporal\_local\_activity\_execution\_latency](https://docs.temporal.io/references/sdk-metrics#local_activity_execution_latency) | Worker | Histogram | Core, Go, Java | | [temporal\_local\_activity\_succeeded\_endtoend\_latency](https://docs.temporal.io/references/sdk-metrics#local_activity_succeeded_endtoend_latency) | Worker | Histogram | Core, Go, Java | | [temporal\_local\_activity\_total](https://docs.temporal.io/references/sdk-metrics#local_activity_total) | Worker | Counter | Core, Go, Java | | [temporal\_long\_request](https://docs.temporal.io/references/sdk-metrics#long_request) | Service Client | Counter | Core, Go, Java | | [temporal\_long\_request\_failure](https://docs.temporal.io/references/sdk-metrics#long_request_failure) | Service Client | Counter | Core, Go, Java | | [temporal\_long\_request\_latency](https://docs.temporal.io/references/sdk-metrics#long_request_latency) | Service Client | Histogram | Core, Go, Java | | [temporal\_nexus\_poll\_no\_task](https://docs.temporal.io/references/sdk-metrics#nexus_poll_no_task) | Worker | Counter | Core, Go, Java | | [temporal\_nexus\_task\_schedule\_to\_start\_latency](https://docs.temporal.io/references/sdk-metrics#nexus_task_schedule_to_start_latency) | Worker | Histogram | Core, Go, Java | | [temporal\_nexus\_task\_execution\_failed](https://docs.temporal.io/references/sdk-metrics#nexus_task_execution_failed) | Worker | Counter | Core, Go, Java | | [temporal\_nexus\_task\_execution\_latency](https://docs.temporal.io/references/sdk-metrics#nexus_task_execution_latency) | Worker | Histogram | Core, Go, Java | | [temporal\_nexus\_task\_endtoend\_latency](https://docs.temporal.io/references/sdk-metrics#nexus_task_endtoend_latency) | Worker | Histogram | Core, Go, Java | | [temporal\_num\_pollers](https://docs.temporal.io/references/sdk-metrics#num_pollers) | Worker | Gauge | Core, Go | | [temporal\_poller\_start](https://docs.temporal.io/references/sdk-metrics#poller_start) | Worker | Counter | Go, Java | | [temporal\_request](https://docs.temporal.io/references/sdk-metrics#request) | Service Client | Counter | Core, Go, Java | | [temporal\_request\_failure](https://docs.temporal.io/references/sdk-metrics#request_failure) | Service Client | Counter | Core, Go, Java | | [temporal\_request\_latency](https://docs.temporal.io/references/sdk-metrics#request_latency) | Service Client | Histogram | Core, Go, Java | | [temporal\_resource\_slots\_mem\_usage](https://docs.temporal.io/references/sdk-metrics#resource_slots_cpu_usage) | Worker | Gauge | Core, Java | | [temporal\_resource\_slots\_cpu\_usage](https://docs.temporal.io/references/sdk-metrics#resource_slots_mem_usage) | Worker | Gauge | Core, Java | | [temporal\_sticky\_cache\_hit](https://docs.temporal.io/references/sdk-metrics#sticky_cache_hit) | Worker | Counter | Core, Go, Java | | [temporal\_sticky\_cache\_miss](https://docs.temporal.io/references/sdk-metrics#sticky_cache_miss) | Worker | Counter | Core, Go, Java | | [temporal\_sticky\_cache\_size](https://docs.temporal.io/references/sdk-metrics#sticky_cache_size) | Worker | Gauge | Core, Go, Java | | [temporal\_sticky\_cache\_total\_forced\_eviction](https://docs.temporal.io/references/sdk-metrics#sticky_cache_total_forced_eviction) | Worker | Counter | Go, Java | | [temporal\_unregistered\_activity\_invocation](https://docs.temporal.io/references/sdk-metrics#unregistered_activity_invocation) | Worker | Counter | Go | | [temporal\_worker\_start](https://docs.temporal.io/references/sdk-metrics#worker_start) | Worker | Counter | Core, Go, Java | | [temporal\_worker\_task\_slots\_available](https://docs.temporal.io/references/sdk-metrics#worker_task_slots_available) | Worker | Gauge | Core, Go, Java | | [temporal\_worker\_task\_slots\_used](https://docs.temporal.io/references/sdk-metrics#worker_task_slots_used) | Worker | Gauge | Core, Go, Java | | [temporal\_workflow\_active\_thread\_count](https://docs.temporal.io/references/sdk-metrics#workflow_active_thread_count) | Worker | Gauge | Java | | [temporal\_workflow\_cancelled](https://docs.temporal.io/references/sdk-metrics#workflow_cancelled) | Worker | Counter | Core, Go, Java | | [temporal\_workflow\_completed](https://docs.temporal.io/references/sdk-metrics#workflow_completed) | Worker | Counter | Core, Go, Java | | [temporal\_workflow\_continue\_as\_new](https://docs.temporal.io/references/sdk-metrics#workflow_continue_as_new) | Worker | Counter | Core, Go, Java | | [temporal\_workflow\_endtoend\_latency](https://docs.temporal.io/references/sdk-metrics#workflow_endtoend_latency) | Worker | Histogram | Core, Go, Java | | [temporal\_workflow\_failed](https://docs.temporal.io/references/sdk-metrics#workflow_failed) | Worker | Counter | Core, Go, Java | | [temporal\_workflow\_task\_execution\_failed](https://docs.temporal.io/references/sdk-metrics#workflow_task_execution_failed) | Worker | Counter | Core, Go, Java | | [temporal\_workflow\_task\_execution\_latency](https://docs.temporal.io/references/sdk-metrics#workflow_task_execution_latency) | Worker | Histogram | Core, Go, Java | | [temporal\_workflow\_task\_queue\_poll\_empty](https://docs.temporal.io/references/sdk-metrics#workflow_task_queue_poll_empty) | Worker | Counter | Core, Go, Java | | [temporal\_workflow\_task\_queue\_poll\_succeed](https://docs.temporal.io/references/sdk-metrics#workflow_task_queue_poll_succeed) | Worker | Counter | Core, Go, Java | | [temporal\_workflow\_task\_replay\_latency](https://docs.temporal.io/references/sdk-metrics#workflow_task_replay_latency) | Worker | Histogram | Core, Go, Java | | [temporal\_workflow\_task\_schedule\_to\_start\_latency](https://docs.temporal.io/references/sdk-metrics#workflow_task_schedule_to_start_latency) | Worker | Histogram | Core, Go, Java | ### activity\_execution\_cancelled[​](https://docs.temporal.io/references/sdk-metrics#activity_execution_cancelled "Direct link to activity_execution_cancelled") An Activity Execution was canceled. * Type: Counter * Available in: Java * Tags: `activity_type`, `namespace`, `task_queue` ### activity\_execution\_failed[​](https://docs.temporal.io/references/sdk-metrics#activity_execution_failed "Direct link to activity_execution_failed") An Activity Execution failed. This does not include local Activity Failures in the Go and Java SDKs (see [local\_activity\_execution\_failed](https://docs.temporal.io/references/sdk-metrics#local_activity_execution_failed) ). * Type: Counter * Available in: Core, Go, Java * Tags: `activity_type`, `namespace`, `task_queue` ### activity\_execution\_latency[​](https://docs.temporal.io/references/sdk-metrics#activity_execution_latency "Direct link to activity_execution_latency") Time to complete an Activity Execution, from the time the Activity Task is generated to the time the language SDK responded with a completion (failure or success). * Type: Histogram * Available in: Core, Go, Java * Tags: `activity_type`, `namespace`, `task_queue` ### activity\_poll\_no\_task[​](https://docs.temporal.io/references/sdk-metrics#activity_poll_no_task "Direct link to activity_poll_no_task") An Activity Worker poll for an Activity Task timed out, and no Activity Task is available to pick from the Task Queue. * Type: Counter * Available in: Core, Go, Java * Tags: `namespace`, `task_queue` ### activity\_schedule\_to\_start\_latency[​](https://docs.temporal.io/references/sdk-metrics#activity_schedule_to_start_latency "Direct link to activity_schedule_to_start_latency") The Schedule-To-Start time of an Activity Task in seconds. A [Schedule-To-Start Timeout](https://docs.temporal.io/encyclopedia/detecting-activity-failures#schedule-to-start-timeout) can be set when an Activity Execution is spawned. This metric is useful for ensuring Activity Tasks are being processed from the queue in a timely manner. Some SDKs may include the `activity_type` label, but the metric should not vary by type, as it does not influence the rate at which tasks are pulled from the queue. * Type: Histogram * Available in: Core, Go, Java * Tags: `namespace`, `task_queue` ### activity\_succeed\_endtoend\_latency[​](https://docs.temporal.io/references/sdk-metrics#activity_succeed_endtoend_latency "Direct link to activity_succeed_endtoend_latency") Total latency of successfully finished Activity Executions from the time they are scheduled to the time they are completed. This metric is not recorded for async Activity completion. * Type: Histogram * Available in: Core, Go, Java * Tags: `activity_type`, `namespace`, `task_queue` ### activity\_task\_error[​](https://docs.temporal.io/references/sdk-metrics#activity_task_error "Direct link to activity_task_error") An internal error or panic occurred during Activity Task handling or execution. * Type: Counter * Available in: Go, * Tags: `activity_type`, `namespace`, `task_queue`, `workflow_type` ### corrupted\_signals[​](https://docs.temporal.io/references/sdk-metrics#corrupted_signals "Direct link to corrupted_signals") Number of Signals whose payload could not be deserialized. * Type: Counter * Available in: Go, Java * Tags: `namespace`, `task_queue`, `workflow_type` ### local\_activity\_execution\_cancelled[​](https://docs.temporal.io/references/sdk-metrics#local_activity_execution_cancelled "Direct link to local_activity_execution_cancelled") A Local Activity Execution was canceled. * Type: Counter * Available in: Core, Go, Java * Tags: `activity_type`, `namespace`, `task_queue` ### local\_activity\_execution\_failed[​](https://docs.temporal.io/references/sdk-metrics#local_activity_execution_failed "Direct link to local_activity_execution_failed") A Local Activity Execution failed. * Type: Counter * Available in: Core, Go, Java * Tags: `activity_type`, `namespace`, `task_queue` ### local\_activity\_execution\_latency[​](https://docs.temporal.io/references/sdk-metrics#local_activity_execution_latency "Direct link to local_activity_execution_latency") Time to complete a Local Activity Execution, from the time the first Activity Task is generated to the time the SDK responds that the execution is complete. * Type: Histogram * Available in: Core, Go, Java * Tags: `activity_type`, `namespace`, `task_queue` ### local\_activity\_succeeded\_endtoend\_latency[​](https://docs.temporal.io/references/sdk-metrics#local_activity_succeeded_endtoend_latency "Direct link to local_activity_succeeded_endtoend_latency") Total latency of successfully finished Local Activity Executions (from schedule to completion). * Type: Histogram * Available in: Core, Go, Java * Tags: `activity_type`, `namespace`, `task_queue` ### local\_activity\_total[​](https://docs.temporal.io/references/sdk-metrics#local_activity_total "Direct link to local_activity_total") Total number of [Local Activity Executions](https://docs.temporal.io/local-activity) . * Type: Counter * Available in: Core, Go, Java * Tags: `activity_type`, `namespace`, `task_queue` ### long\_request[​](https://docs.temporal.io/references/sdk-metrics#long_request "Direct link to long_request") Temporal Client made an RPC long poll request. * Type: Counter * Available in: Core, Go, Java * Tags: `namespace`, `operation` ### long\_request\_failure[​](https://docs.temporal.io/references/sdk-metrics#long_request_failure "Direct link to long_request_failure") Temporal Client made an RPC long poll request that failed. This number is included into the total `long_request` counter for long poll RPC requests. * Type: Counter * Available in: Core, Go, Java * Tags: `namespace`, `operation` ### long\_request\_latency[​](https://docs.temporal.io/references/sdk-metrics#long_request_latency "Direct link to long_request_latency") Latency of a Temporal Client gRPC long poll request. * Type: Histogram * Available in: Core, Go, Java * Tags: `namespace`, `operation` ### nexus\_poll\_no\_task[​](https://docs.temporal.io/references/sdk-metrics#nexus_poll_no_task "Direct link to nexus_poll_no_task") A Nexus Worker poll for a Nexus Task timed out, and no Nexus Task is available to pick from the Task Queue. * Type: Counter * Available in: Go, Java * Tags: `namespace`, `task_queue` ### nexus\_task\_schedule\_to\_start\_latency[​](https://docs.temporal.io/references/sdk-metrics#nexus_task_schedule_to_start_latency "Direct link to nexus_task_schedule_to_start_latency") The Schedule-To-Start time of a Nexus Task in seconds. The schedule time is taken from when the corresponding request hit the Frontend service to when the SDK started processing the task. This time is limited by the `Request-Timeout` header given to the Frontend when handling this request. * Type: Histogram * Available in: Core, Go, Java * Tags: `namespace`, `task_queue` ### nexus\_task\_execution\_failed[​](https://docs.temporal.io/references/sdk-metrics#nexus_task_execution_failed "Direct link to nexus_task_execution_failed") Handling of a Nexus Task resulted in an error. This includes any error returned from a user handler and unexpected internal errors in the SDK. * Type: Counter * Available in: Go, Java * Tags: `namespace`, `task_queue`, `nexus_service`, `nexus_operation`, `failure_reason` Valid values for the `failure_reason` tag: * `internal_sdk_error`: There was an unexpected internal error within the SDK while handling the Nexus task. Indicates a bug in the SDK. * `handler_error_{TYPE}`: The user handler code returned a predefined error, as specified in the [Nexus spec](https://github.com/nexus-rpc/api/blob/main/SPEC.md#predefined-handler-errors) . If the handler returns an unexpected error, the TYPE is set to `INTERNAL`. * `timeout`: The user handler code did not return within the request timeout. * `operation_failed`: The user handler code has indicated that the operation has failed. In Go, this maps to an `UnsuccessfulOperationError` with a `failed` state. * `operation_canceled`: The user handler code has indicated that the operation has completed as canceled. In Go, this maps to an `UnsuccessfulOperationError` with a `canceled` state. ### nexus\_task\_execution\_latency[​](https://docs.temporal.io/references/sdk-metrics#nexus_task_execution_latency "Direct link to nexus_task_execution_latency") Time to complete a Nexus Task, from the time the Nexus Task processing starts in the SDK to the time the user handler completes. * Type: Histogram * Available in: Go, Java * Tags: `namespace`, `task_queue`, `nexus_service`, `nexus_operation` ### nexus\_task\_endtoend\_latency[​](https://docs.temporal.io/references/sdk-metrics#nexus_task_endtoend_latency "Direct link to nexus_task_endtoend_latency") Total latency of Nexus Tasks from the time the corresponding request hit the Frontend to after the SDK gets acknowledgment from the server for task completion. * Type: Histogram * Available in: Go, Java * Tags: `namespace`, `task_queue`, `nexus_service`, `nexus_operation` ### num\_pollers[​](https://docs.temporal.io/references/sdk-metrics#num_pollers "Direct link to num_pollers") Current number of Worker Entities that are polling. * Type: Gauge * Available in: Core, Go, Java * Tags: `namespace`, `poller_type`, `task_queue` ### poller\_start[​](https://docs.temporal.io/references/sdk-metrics#poller_start "Direct link to poller_start") A Worker Entity poller was started. * Type: Counter * Available in: Go, Java * Tags: `namespace`, `task_queue` ### request[​](https://docs.temporal.io/references/sdk-metrics#request "Direct link to request") Temporal Client made an RPC request. * Type: Counter * Available in: Core, Go, Java * Tags: `namespace`, `operation` ### request\_failure[​](https://docs.temporal.io/references/sdk-metrics#request_failure "Direct link to request_failure") Temporal Client made an RPC request that failed. This number is included into the total `request` counter for RPC requests. * Type: Counter * Available in: Core, Go, Java * Tags: `namespace`, `operation` ### request\_latency[​](https://docs.temporal.io/references/sdk-metrics#request_latency "Direct link to request_latency") Latency of a Temporal Client gRPC request. * Type: Histogram * Available in: Core, Go, Java * Tags: `namespace`, `operation` ### resource\_slots\_cpu\_usage[​](https://docs.temporal.io/references/sdk-metrics#resource_slots_cpu_usage "Direct link to resource_slots_cpu_usage") CPU usage as a value between 0 and 100. As perceived by the resource-based slots tuner, if enabled. * Type: Gauge * Available in: Core, Java ### resource\_slots\_mem\_usage[​](https://docs.temporal.io/references/sdk-metrics#resource_slots_mem_usage "Direct link to resource_slots_mem_usage") Memory usage as a value between 0 and 100. As perceived by the resource-based slots tuner, if enabled. * Type: Gauge * Available in: Core, Java ### sticky\_cache\_hit[​](https://docs.temporal.io/references/sdk-metrics#sticky_cache_hit "Direct link to sticky_cache_hit") A Workflow Task found a cached Workflow Execution to run against. * Type: Counter * Available in: Core, Go, Java * Tags: `namespace`, `task_queue` ### sticky\_cache\_miss[​](https://docs.temporal.io/references/sdk-metrics#sticky_cache_miss "Direct link to sticky_cache_miss") A Workflow Task did not find a cached Workflow execution to run against. * Type: Counter * Available in: Core, Go, Java * Tags: `namespace`, `task_queue` ### sticky\_cache\_size[​](https://docs.temporal.io/references/sdk-metrics#sticky_cache_size "Direct link to sticky_cache_size") Current cache size, expressed in number of Workflow Executions. * Type: Gauge * Available in: Core, Go, Java * Tags: `namespace` (TypeScript, Java), `task_queue` (TypeScript) ### sticky\_cache\_total\_forced\_eviction[​](https://docs.temporal.io/references/sdk-metrics#sticky_cache_total_forced_eviction "Direct link to sticky_cache_total_forced_eviction") A Workflow Execution has been forced from the cache intentionally. * Type: Counter * Available in: Go, Java * Tags: `namespace`, `task_queue` ### unregistered\_activity\_invocation[​](https://docs.temporal.io/references/sdk-metrics#unregistered_activity_invocation "Direct link to unregistered_activity_invocation") A request to spawn an Activity Execution is not registered with the Worker. * Type: Counter * Available in: Go, * Tags: `activity_type`, `namespace`, `task_queue`, `workflow_type` ### worker\_start[​](https://docs.temporal.io/references/sdk-metrics#worker_start "Direct link to worker_start") A Worker Entity has been registered, created, or started. * Type: Counter * Available in: Core, Go, Java * Tags: `namespace`, `task_queue`, `worker_type` ### worker\_task\_slots\_available[​](https://docs.temporal.io/references/sdk-metrics#worker_task_slots_available "Direct link to worker_task_slots_available") The total number of Workflow, Activity, Local Activity, or Nexus Task execution slots that are currently available. Use the `worker_type` key to differentiate execution slots. The Worker type specifies an ability to perform certain tasks. For example, Workflow Workers execute Workflow Tasks, Activity Workers execute Activity Tasks, and so forth. * Type: Gauge * Available in: Core, Go, Java * Tags: `namespace`, `task_queue`, `worker_type` ### worker\_task\_slots\_used[​](https://docs.temporal.io/references/sdk-metrics#worker_task_slots_used "Direct link to worker_task_slots_used") The total number of Workflow, Activity, Local Activity, or Nexus Tasks execution slots in current use. Use the `worker_type` key to differentiate execution slots. The Worker type specifies an ability to perform certain tasks. For example, Workflow Workers execute Workflow Tasks, Activity Workers execute Activity Tasks, and so forth. * Type: Gauge * Available in: Core, Go, Java * Tags: `namespace`, `task_queue`, `worker_type` ### workflow\_active\_thread\_count[​](https://docs.temporal.io/references/sdk-metrics#workflow_active_thread_count "Direct link to workflow_active_thread_count") Total amount of Workflow threads in the Worker Process. * Type: Gauge * Available in: Java ### workflow\_cancelled[​](https://docs.temporal.io/references/sdk-metrics#workflow_cancelled "Direct link to workflow_cancelled") Workflow Execution ended because of a cancellation request. * Type: Counter * Available in: Core, Go, Java * Tags: `namespace`, `task_queue`, `workflow_type` ### workflow\_completed[​](https://docs.temporal.io/references/sdk-metrics#workflow_completed "Direct link to workflow_completed") A Workflow Execution completed successfully. * Type: Counter * Available in: Core, Go, Java * Tags: `namespace`, `task_queue`, `workflow_type` ### workflow\_continue\_as\_new[​](https://docs.temporal.io/references/sdk-metrics#workflow_continue_as_new "Direct link to workflow_continue_as_new") A Workflow ended with Continue-As-New. * Type: Counter * Available in: Core, Go, Java * Tags: `namespace`, `task_queue`, `workflow_type` ### workflow\_endtoend\_latency[​](https://docs.temporal.io/references/sdk-metrics#workflow_endtoend_latency "Direct link to workflow_endtoend_latency") Total Workflow Execution time from schedule to completion for a single Workflow Run. (A retried Workflow Execution is a separate Run.) * Type: Histogram * Available in: Core, Go, Java * Tags: `namespace`, `task_queue`, `workflow_type` ### workflow\_failed[​](https://docs.temporal.io/references/sdk-metrics#workflow_failed "Direct link to workflow_failed") A Workflow Execution failed. * Type: Counter * Available in: Core, Go, Java * Tags: `namespace`, `task_queue`, `workflow_type` ### workflow\_task\_execution\_failed[​](https://docs.temporal.io/references/sdk-metrics#workflow_task_execution_failed "Direct link to workflow_task_execution_failed") A Workflow Task Execution failed. * Type: Counter * Available in: Core, Go, Java * Tags: `namespace`, `task_queue`, `workflow_type`, `failure_reason` Valid values for the `failure_reason` tag: * `NonDeterminismError`: The Workflow Task failed due to a non-determinism error. * `WorkflowError`: The Workflow Task failed for any other reason. ### workflow\_task\_execution\_latency[​](https://docs.temporal.io/references/sdk-metrics#workflow_task_execution_latency "Direct link to workflow_task_execution_latency") Workflow Task Execution time. * Type: Histogram * Available in: Core, Go, Java * Tags: `namespace`, `task_queue`, `workflow_type` ### workflow\_task\_queue\_poll\_empty[​](https://docs.temporal.io/references/sdk-metrics#workflow_task_queue_poll_empty "Direct link to workflow_task_queue_poll_empty") A Workflow Worker polled a Task Queue and timed out without picking up a Workflow Task. * Type: Counter * Available in: Core, Go, Java * Tags: `namespace`, `task_queue` ### workflow\_task\_queue\_poll\_succeed[​](https://docs.temporal.io/references/sdk-metrics#workflow_task_queue_poll_succeed "Direct link to workflow_task_queue_poll_succeed") A Workflow Worker polled a Task Queue and successfully picked up a Workflow Task. * Type: Counter * Available in: Core, Go, Java * Tags: `namespace`, `task_queue` ### workflow\_task\_replay\_latency[​](https://docs.temporal.io/references/sdk-metrics#workflow_task_replay_latency "Direct link to workflow_task_replay_latency") Time to catch up on replaying a Workflow Task. * Type: Histogram * Available in: Core, Go, Java * Tags: `namespace`, `task_queue`, `workflow_type` ### workflow\_task\_schedule\_to\_start\_latency[​](https://docs.temporal.io/references/sdk-metrics#workflow_task_schedule_to_start_latency "Direct link to workflow_task_schedule_to_start_latency") The Schedule-To-Start time of a Workflow Task. * Type: Histogram * Available in: Core, Go, Java * Tags: `namespace`, `task_queue` * [activity\_execution\_cancelled](https://docs.temporal.io/references/sdk-metrics#activity_execution_cancelled) * [activity\_execution\_failed](https://docs.temporal.io/references/sdk-metrics#activity_execution_failed) * [activity\_execution\_latency](https://docs.temporal.io/references/sdk-metrics#activity_execution_latency) * [activity\_poll\_no\_task](https://docs.temporal.io/references/sdk-metrics#activity_poll_no_task) * [activity\_schedule\_to\_start\_latency](https://docs.temporal.io/references/sdk-metrics#activity_schedule_to_start_latency) * [activity\_succeed\_endtoend\_latency](https://docs.temporal.io/references/sdk-metrics#activity_succeed_endtoend_latency) * [activity\_task\_error](https://docs.temporal.io/references/sdk-metrics#activity_task_error) * [corrupted\_signals](https://docs.temporal.io/references/sdk-metrics#corrupted_signals) * [local\_activity\_execution\_cancelled](https://docs.temporal.io/references/sdk-metrics#local_activity_execution_cancelled) * [local\_activity\_execution\_failed](https://docs.temporal.io/references/sdk-metrics#local_activity_execution_failed) * [local\_activity\_execution\_latency](https://docs.temporal.io/references/sdk-metrics#local_activity_execution_latency) * [local\_activity\_succeeded\_endtoend\_latency](https://docs.temporal.io/references/sdk-metrics#local_activity_succeeded_endtoend_latency) * [local\_activity\_total](https://docs.temporal.io/references/sdk-metrics#local_activity_total) * [long\_request](https://docs.temporal.io/references/sdk-metrics#long_request) * [long\_request\_failure](https://docs.temporal.io/references/sdk-metrics#long_request_failure) * [long\_request\_latency](https://docs.temporal.io/references/sdk-metrics#long_request_latency) * [nexus\_poll\_no\_task](https://docs.temporal.io/references/sdk-metrics#nexus_poll_no_task) * [nexus\_task\_schedule\_to\_start\_latency](https://docs.temporal.io/references/sdk-metrics#nexus_task_schedule_to_start_latency) * [nexus\_task\_execution\_failed](https://docs.temporal.io/references/sdk-metrics#nexus_task_execution_failed) * [nexus\_task\_execution\_latency](https://docs.temporal.io/references/sdk-metrics#nexus_task_execution_latency) * [nexus\_task\_endtoend\_latency](https://docs.temporal.io/references/sdk-metrics#nexus_task_endtoend_latency) * [num\_pollers](https://docs.temporal.io/references/sdk-metrics#num_pollers) * [poller\_start](https://docs.temporal.io/references/sdk-metrics#poller_start) * [request](https://docs.temporal.io/references/sdk-metrics#request) * [request\_failure](https://docs.temporal.io/references/sdk-metrics#request_failure) * [request\_latency](https://docs.temporal.io/references/sdk-metrics#request_latency) * [resource\_slots\_cpu\_usage](https://docs.temporal.io/references/sdk-metrics#resource_slots_cpu_usage) * [resource\_slots\_mem\_usage](https://docs.temporal.io/references/sdk-metrics#resource_slots_mem_usage) * [sticky\_cache\_hit](https://docs.temporal.io/references/sdk-metrics#sticky_cache_hit) * [sticky\_cache\_miss](https://docs.temporal.io/references/sdk-metrics#sticky_cache_miss) * [sticky\_cache\_size](https://docs.temporal.io/references/sdk-metrics#sticky_cache_size) * [sticky\_cache\_total\_forced\_eviction](https://docs.temporal.io/references/sdk-metrics#sticky_cache_total_forced_eviction) * [unregistered\_activity\_invocation](https://docs.temporal.io/references/sdk-metrics#unregistered_activity_invocation) * [worker\_start](https://docs.temporal.io/references/sdk-metrics#worker_start) * [worker\_task\_slots\_available](https://docs.temporal.io/references/sdk-metrics#worker_task_slots_available) * [worker\_task\_slots\_used](https://docs.temporal.io/references/sdk-metrics#worker_task_slots_used) * [workflow\_active\_thread\_count](https://docs.temporal.io/references/sdk-metrics#workflow_active_thread_count) * [workflow\_cancelled](https://docs.temporal.io/references/sdk-metrics#workflow_cancelled) * [workflow\_completed](https://docs.temporal.io/references/sdk-metrics#workflow_completed) * [workflow\_continue\_as\_new](https://docs.temporal.io/references/sdk-metrics#workflow_continue_as_new) * [workflow\_endtoend\_latency](https://docs.temporal.io/references/sdk-metrics#workflow_endtoend_latency) * [workflow\_failed](https://docs.temporal.io/references/sdk-metrics#workflow_failed) * [workflow\_task\_execution\_failed](https://docs.temporal.io/references/sdk-metrics#workflow_task_execution_failed) * [workflow\_task\_execution\_latency](https://docs.temporal.io/references/sdk-metrics#workflow_task_execution_latency) * [workflow\_task\_queue\_poll\_empty](https://docs.temporal.io/references/sdk-metrics#workflow_task_queue_poll_empty) * [workflow\_task\_queue\_poll\_succeed](https://docs.temporal.io/references/sdk-metrics#workflow_task_queue_poll_succeed) * [workflow\_task\_replay\_latency](https://docs.temporal.io/references/sdk-metrics#workflow_task_replay_latency) * [workflow\_task\_schedule\_to\_start\_latency](https://docs.temporal.io/references/sdk-metrics#workflow_task_schedule_to_start_latency) --- # Self-hosted Archival setup | Temporal Platform Documentation [Skip to main content](https://docs.temporal.io/self-hosted-guide/archival#__docusaurus_skipToContent_fallback) On this page Use Archival to back up closed Workflow Execution [Event Histories](https://docs.temporal.io/workflow-execution/event#event-history) and Visibility records from Temporal Service persistence to blob storage. * [How to create a custom Archiver](https://docs.temporal.io/self-hosted-guide/archival#custom-archiver) * [How to set up Archival](https://docs.temporal.io/self-hosted-guide/archival#set-up-archival) When a Workflow Execution closes, Temporal schedules close-processing tasks for both Visibility records and Event History Archival. Archival then runs asynchronously after a randomized delay. By default, that delay is up to 5 minutes set by `history.archivalProcessorArchiveDelay`, and is capped by the Namespace [Retention Period](https://docs.temporal.io/temporal-service/temporal-server#retention-period) . The closed execution still stays in Temporal persistence until retention cleanup runs. For some time, the same closed execution can exist in both persistence and archival storage. Archival enables Workflow Execution data to persist beyond retention without overwhelming the Temporal Service persistence store. Use this to keep closed Workflow data available for compliance, audits, and debugging without keeping all closed data in your primary persistence store. Experimental feature Archival is an **experimental** feature and not subject to normal [versioning and support policy](https://docs.temporal.io/temporal-service/temporal-server#versions-and-support) . Archival is not supported when running Temporal through Docker. It's disabled by default when installing the system manually and when deploying through [helm charts](https://github.com/temporalio/helm-charts/blob/main/charts/temporal/templates/server-configmap.yaml) . It can be enabled in the server [configuration](https://github.com/temporalio/temporal/blob/main/config/development.yaml) . ### Set up Archival[​](https://docs.temporal.io/self-hosted-guide/archival#set-up-archival "Direct link to Set up Archival") To set up [Archival](https://docs.temporal.io/temporal-service/archival) , decide the following: * **Which provider to use:** S3, Google Cloud, local file system, or custom. * **Which URI to use:** URI scheme and path identify the provider and destination. * **Which Namespace should use Archival:** Archival must be enabled at both the Temporal Service level and the Namespace level. Take the following steps to set up Archival: 1. [Choose an Archival provider](https://docs.temporal.io/self-hosted-guide/archival#choose-an-archival-provider) . 2. [Configure Archival options](https://docs.temporal.io/self-hosted-guide/archival#configure-archival-options) . 3. [Create an Archiving Namespace](https://docs.temporal.io/self-hosted-guide/archival#create-an-archiving-namespace) . #### Choose an Archival provider[​](https://docs.temporal.io/self-hosted-guide/archival#choose-an-archival-provider "Direct link to Choose an Archival provider") Temporal directly supports several providers: * **Local file system**: The [filestore archiver](https://github.com/temporalio/temporal/tree/main/common/archiver/filestore) is used to archive data in the file system of whatever host the Temporal server is running on. In the case of [temporal helm-charts](https://github.com/temporalio/helm-charts) , the archive data is stored in the `history` pod. APIs do not function with the filestore archive. This provider is used mainly for local installations and testing and should not be relied on for production environments. * **Google Cloud**: The [gcloud archiver](https://github.com/temporalio/temporal/tree/main/common/archiver/gcloud) is used to connect and archive data with [Google Cloud](https://cloud.google.com/storage) . * **S3**: The [s3store archiver](https://github.com/temporalio/temporal/tree/main/common/archiver/s3store) is used to connect and archive data with [S3](https://aws.amazon.com/s3) . * **Custom**: If you want to use a provider that is not currently supported, you can [create your own archiver](https://docs.temporal.io/self-hosted-guide/archival#custom-archiver) to support it. Save the provider URI so you can pass it when you create a Namespace with Archival enabled. #### Configure Archival options[​](https://docs.temporal.io/self-hosted-guide/archival#configure-archival-options "Direct link to Configure Archival options") Configure Archival in [`config/development.yaml`](https://github.com/temporalio/temporal/blob/main/config/development.yaml#L93) : # Temporal Service level Archival configarchival: # Event History configuration history: # Archival is enabled at the Temporal Service level state: 'enabled' enableRead: true # Namespaces can use either the local filestore provider or the Google Cloud provider provider: filestore: fileMode: '0666' dirMode: '0766' gstorage: credentialsPath: '/tmp/gcloud/keyfile.json'# Default values for a Namespace if none are provided at creationnamespaceDefaults: # Archival defaults archival: # Event History defaults history: state: 'enabled' # New Namespaces will default to the local provider URI: 'file:///tmp/temporal_archival/development' You can disable Archival by setting `archival.history.state` and `namespaceDefaults.archival.history.state` to `"disabled"`. Example: archival: history: state: 'disabled'namespaceDefaults: archival: history: state: 'disabled' The following table shows the available configuration options and their accepted values: | Config | Acceptable values | Description | | --- | --- | --- | | `archival.history.state` | `enabled`, `disabled` | Must be `enabled` to use the Archival feature with any Namespace in the Temporal Service. | | `archival.history.enableRead` | `true`, `false` | Must be `true` to read from the archived Event History. | | `archival.history.provider` | Sub provider configs are `filestore`, `gstorage`, `s3`, or `your_custom_provider`. | Default config specifies `filestore`. | | `archival.history.provider.filestore.fileMode` | File permission string | File permissions of the archived files. We recommend using the default value of `"0666"` to avoid read/write issues. | | `archival.history.provider.filestore.dirMode` | File permission string | Directory permissions of the archive directory. We recommend using the default value of `"0766"` to avoid read/write issues. | | `namespaceDefaults.archival.history.state` | `enabled`, `disabled` | Default state of the Archival feature whenever a new Namespace is created without specifying the Archival state. | | `namespaceDefaults.archival.history.URI` | Valid URI | Must be a URI of the file store location and match a schema that correlates to a provider. | Additional resources: [Temporal Service configuration reference](https://docs.temporal.io/references/configuration) . #### Create an Archiving Namespace[​](https://docs.temporal.io/self-hosted-guide/archival#create-an-archiving-namespace "Direct link to Create an Archiving Namespace") Although Archival is configured at the Temporal Service level, it operates independently within each Namespace. If you don't specify an Archival URI during Namespace creation, the Namespace uses `namespaceDefaults.archival.history.URI` from `config/development.yaml`. The Archival URI cannot be changed after the Namespace is created. Each Namespace supports only a single Archival URI, but each Namespace can use a different URI. A Namespace can safely switch Archival between `enabled` and `disabled` states as long as Archival is enabled at the Temporal Service level. Archival is supported in [Global Namespaces](https://docs.temporal.io/global-namespace) (Namespaces that span multiple clusters). When Archival is running in a Global Namespace, it first runs on the active cluster; later it runs on the standby cluster. Before archiving, a history check is done to see what has been previously archived. #### Test your Archival setup[​](https://docs.temporal.io/self-hosted-guide/archival#test-your-archival-setup "Direct link to Test your Archival setup") To test Archival locally, start by running a Temporal server: ./temporal-server start Then register a new Namespace with Archival enabled. ./temporal operator namespace create --namespace="my-namespace" --global false --history-archival-state="enabled" --retention="4d" note If the retention period isn't set, it defaults to 72h. The minimum retention period is one day. For retention maximums, check [Temporal Service Retention Period limits](https://docs.temporal.io/temporal-service/temporal-server#retention-period) for your server version. Setting the retention period to 0 results in the error _A valid retention period is not set on request_. Next, run a sample Workflow such as the [helloworld temporal sample](https://github.com/temporalio/temporal-go-samples/tree/master/helloworld) . When the Workflow Execution closes, Temporal schedules archival processing. #### Retrieve archived history[​](https://docs.temporal.io/self-hosted-guide/archival#retrieve-archived-history "Direct link to Retrieve archived history") You can retrieve archived Event Histories by copying the `workflowId` and `runId` of the completed Workflow from the log output and running the following command: ./temporal workflow show --workflow-id="my-workflow-id" --run-id="my-run-id" --namespace="my-namespace" ### Create a custom Archiver[​](https://docs.temporal.io/self-hosted-guide/archival#custom-archiver "Direct link to Create a custom Archiver") To archive data with a given provider, using the [Archival](https://docs.temporal.io/temporal-service/archival) feature, Temporal must have a corresponding Archiver component installed. The platform does not limit you to the existing providers. To use a provider that is not currently supported, you can create your own Archiver. #### Create a new package[​](https://docs.temporal.io/self-hosted-guide/archival#create-a-new-package "Direct link to Create a new package") The first step is to create a new package for your implementation in [/common/archiver](https://github.com/temporalio/temporal/tree/main/common/archiver) . Create a directory in the archiver folder and arrange the structure to look like the following: temporal/common/archiver - filestore/ -- Filestore implementation - provider/ - provider.go -- Provider of archiver instances - yourImplementation/ - historyArchiver.go -- HistoryArchiver implementation - historyArchiver_test.go -- Unit tests for HistoryArchiver - visibilityArchiver.go -- VisibilityArchiver implementations - visibilityArchiver_test.go -- Unit tests for VisibilityArchiver #### Archiver interfaces[​](https://docs.temporal.io/self-hosted-guide/archival#archiver-interfaces "Direct link to Archiver interfaces") Next, define objects that implement the [HistoryArchiver](https://github.com/temporalio/temporal/blob/main/common/archiver/interface.go#L80) and the [VisibilityArchiver](https://github.com/temporalio/temporal/blob/main/common/archiver/interface.go#L121) interfaces. The objects should live in `historyArchiver.go` and `visibilityArchiver.go`, respectively. #### Update provider[​](https://docs.temporal.io/self-hosted-guide/archival#update-provider "Direct link to Update provider") Update the `GetHistoryArchiver` and `GetVisibilityArchiver` methods of the `archiverProvider` object in the [/common/archiver/provider/provider.go](https://github.com/temporalio/temporal/blob/main/common/archiver/provider/provider.go) file so that it knows how to create an instance of your archiver. #### Add configs[​](https://docs.temporal.io/self-hosted-guide/archival#add-configs "Direct link to Add configs") Add configs for your archiver to the `config/development.yaml` file and then modify the [HistoryArchiverProvider](https://github.com/temporalio/temporal/blob/main/common/config/config.go#L376) and [VisibilityArchiverProvider](https://github.com/temporalio/temporal/blob/main/common/config/config.go#L393) structs in `/common/common/config.go` accordingly. #### Custom archiver FAQ[​](https://docs.temporal.io/self-hosted-guide/archival#custom-archiver-faq "Direct link to Custom archiver FAQ") **If my custom Archive method can automatically be retried by the caller, how can I record and access progress between retries?** Handle this situation by using `ArchiverOptions`. Here is an example: func(a * Archiver) Archive(ctx context.Context, URI string, request * ArchiveRequest, opts...ArchiveOption) error { featureCatalog: = GetFeatureCatalog(opts...) // this function is defined in options.go var progress progress // Check if the feature for recording progress is enabled. if featureCatalog.ProgressManager != nil { if err: = featureCatalog.ProgressManager.LoadProgress(ctx, & prevProgress); err != nil { // log some error message and return error if needed. } } // Your archiver implementation... // Record current progress if featureCatalog.ProgressManager != nil { if err: = featureCatalog.ProgressManager.RecordProgress(ctx, progress); err != nil { // log some error message and return error if needed. } }} **If my `Archive` method encounters an error that is non-retryable, how do I indicate to the caller that it should not retry?** func(a * Archiver) Archive(ctx context.Context, URI string, request * ArchiveRequest, opts...ArchiveOption) error { featureCatalog: = GetFeatureCatalog(opts...) // this function is defined in options.go err: = youArchiverImpl() if nonRetryableErr(err) { if featureCatalog.NonRetryableError != nil { return featureCatalog.NonRetryableError() // when the caller gets this error type back it will not retry anymore. } }} **How does my history archiver implementation read history?** The archiver package provides a utility called [HistoryIterator](https://github.com/temporalio/temporal/blob/main/common/archiver/historyIterator.go) which is a wrapper of [ExecutionManager](https://github.com/temporalio/temporal/blob/main/common/persistence/data_interfaces.go#L1014) . `HistoryIterator` is more simple than the `HistoryManager`, which is available in the BootstrapContainer, so archiver implementations can choose to use it when reading Workflow histories. See the [historyIterator.go](https://github.com/temporalio/temporal/blob/main/common/archiver/history_iterator.go) file for more details. Use the [filestore historyArchiver implementation](https://github.com/temporalio/temporal/tree/main/common/archiver/filestore) as an example. **Should my archiver define its own error types?** Each archiver is free to define and return its own errors. However, many common errors that exist between archivers are already defined in [common/archiver/constants.go](https://github.com/temporalio/temporal/blob/main/common/archiver/constants.go) . **Is there a generic query syntax for the visibility archiver?** Currently, no. But this is something we plan to do in the future. As for now, try to make your syntax similar to the one used by our advanced list Workflow API. * [s3store](https://github.com/temporalio/temporal/tree/main/common/archiver/s3store#visibility-query-syntax) * [gcloud](https://github.com/temporalio/temporal/tree/main/common/archiver/gcloud#visibility-query-syntax) * [Set up Archival](https://docs.temporal.io/self-hosted-guide/archival#set-up-archival) * [Choose an Archival provider](https://docs.temporal.io/self-hosted-guide/archival#choose-an-archival-provider) * [Configure Archival options](https://docs.temporal.io/self-hosted-guide/archival#configure-archival-options) * [Create an Archiving Namespace](https://docs.temporal.io/self-hosted-guide/archival#create-an-archiving-namespace) * [Test your Archival setup](https://docs.temporal.io/self-hosted-guide/archival#test-your-archival-setup) * [Retrieve archived history](https://docs.temporal.io/self-hosted-guide/archival#retrieve-archived-history) * [Create a custom Archiver](https://docs.temporal.io/self-hosted-guide/archival#custom-archiver) * [Create a new package](https://docs.temporal.io/self-hosted-guide/archival#create-a-new-package) * [Archiver interfaces](https://docs.temporal.io/self-hosted-guide/archival#archiver-interfaces) * [Update provider](https://docs.temporal.io/self-hosted-guide/archival#update-provider) * [Add configs](https://docs.temporal.io/self-hosted-guide/archival#add-configs) * [Custom archiver FAQ](https://docs.temporal.io/self-hosted-guide/archival#custom-archiver-faq) --- # List Filter | Temporal Platform Documentation [Skip to main content](https://docs.temporal.io/list-filter#__docusaurus_skipToContent_fallback) On this page This page discusses [List Filter](https://docs.temporal.io/list-filter#list-filter) . What is a List Filter?[​](https://docs.temporal.io/list-filter#list-filter "Direct link to What is a List Filter?") -------------------------------------------------------------------------------------------------------------------- The [Visibility](https://docs.temporal.io/temporal-service/visibility) List API requires you to provide a List Filter as an SQL-like string parameter. A List Filter includes [Search Attribute](https://docs.temporal.io/search-attribute) names, Search Attribute values, and [operators](https://docs.temporal.io/list-filter#supported-operators) so that it can retrieve a filtered list of Workflow Executions from the Visibility Store. List Filter [Search Attribute](https://docs.temporal.io/search-attribute) names are case sensitive. A single [Namespace](https://docs.temporal.io/namespaces) scopes each List Filter. A List Filter using a time range provides a resolution of 1 ns on [Elasticsearch](https://docs.temporal.io/self-hosted-guide/visibility#elasticsearch) and 1 µs for [SQL databases](https://docs.temporal.io/self-hosted-guide/visibility) . ### Supported operators[​](https://docs.temporal.io/list-filter#supported-operators "Direct link to Supported operators") List Filters support the following operators: * **`=, !=, >, >=, <, <=`** * **`AND, OR, ()`** * **`BETWEEN ... AND`** * **`IN`** * **STARTS\_WITH** note The **ORDER BY** operator is currently not supported in Temporal Cloud. The default ordering is: `ClosedTime DESC NULL FIRST`, `StartTime DESC`. Custom Search Attributes of the `Text` type cannot be used in **ORDER BY** clauses. ### Partial string match[​](https://docs.temporal.io/list-filter#partial-string-match "Direct link to Partial string match") There are different options for partial string matching when the type of the Search Attribute is [Text](https://docs.temporal.io/list-filter#text) versus [Keyword](https://docs.temporal.io/list-filter#keyword) . #### Text[​](https://docs.temporal.io/list-filter#text "Direct link to Text") Search Attributes of type `Text` are [broken up into words](https://www.elastic.co/guide/en/elasticsearch/reference/current/analysis-standard-tokenizer.html) that match with the `=` operator. For example, if you have a custom `Text` Search Attribute named `Description` with either of the following values— my-business-id-foobarmy business id foobar —then the following List Filter matches— Description = 'foobar' —but a partial word does not: // Doesn't matchDescription = 'foo' #### Keyword[​](https://docs.temporal.io/list-filter#keyword "Direct link to Keyword") For Search Attributes of type `Keyword` like `WorkflowId`, perform partial string matching using STARTS\_WITH for prefixes and BETWEEN for suffixes. * `WorkflowId STARTS_WITH "order-"` matches Workflow Ids with the "order-" prefix, regardless of the following text. order-order-1234order-abracadabraorder-~~~abracadabra * `WorkflowId BETWEEN "order-" AND "order-~"` matches Workflow Ids that have characters after `order-` with ASCII values lower than `~` (126, the highest-value printable character), such as the following: order-order-1234order-abracadabra It does not match `order-~~`. Filter Composition Quick Reference **Composition** * Data types: * String literals with single or double quotes * Numbers (Integer and Floating Point) * Booleans * Comparison: `=`, `!=`, `>`, `>=`, `<`, `<=` * Expressions/Operators: * `IN array` * `BETWEEN value AND value` * `STARTS_WITH string` * `IS NULL`, `IS NOT NULL` * `expr AND expr`, `expr OR expr`, `( expr )` * Array: `( comma-separated-values )` **Please note** * Wrap attributes with backticks if it contains characters not in `[a-zA-Z0-9]`. * `STARTS_WITH` is only available for Keyword Search Attributes. ### Efficient API usage[​](https://docs.temporal.io/list-filter#efficient-api-usage "Direct link to Efficient API usage") If the Advanced List Filter API retrieves a substantial number of Workflow Executions (more than 10,000), the response time might be longer. Beginning with Temporal Server v1.20, you can employ the `CountWorkflow` API to efficiently count the number of [Workflow Executions](https://docs.temporal.io/workflow-execution) . To paginate the results using the `ListWorkflow` API, use the page token to retrieve the next page. Continue until the page token becomes `null`/`nil`. #### List Filter examples[​](https://docs.temporal.io/list-filter#list-filter-examples "Direct link to List Filter examples") Here are examples of List Filters set with the [Temporal CLI](https://docs.temporal.io/cli/workflow#list) : WorkflowType = "main.YourWorkflowDefinition" and ExecutionStatus != "Running" and (StartTime > "2021-06-07T16:46:34.236-08:00" or CloseTime > "2021-06-07T16:46:34-08:00") When you use the preceding example, you receive a list of Workflows fulfilling the following criteria: * Workflow Type is `main.YourWorkflowDefinition`. * Workflow isn't in a running state. * Workflow either started after "2021-06-07T16:46:34.236-08:00" or closed after "2021-06-07T16:46:34-08:00". The following are additional examples of List Filters. WorkflowId = '' WorkflowId = '' or WorkflowId = '' WorkflowId IN ('', '') WorkflowId = '' and ExecutionStatus = 'Running' WorkflowId = '' or ExecutionStatus = 'Running' WorkflowId = '' and StartTime > '2021-08-22T15:04:05+00:00' ExecutionTime between '2021-08-22T15:04:05+00:00' and '2021-08-28T15:04:05+00:00' ExecutionTime < '2021-08-28T15:04:05+00:00' or ExecutionTime > '2021-08-22T15:04:05+00:00' WorkflowType STARTS_WITH '' ### Search Attribute aliasing[​](https://docs.temporal.io/list-filter#search-attribute-aliasing "Direct link to Search Attribute aliasing") Temporal prefixes most [default Search Attributes](https://docs.temporal.io/search-attribute#default-search-attribute) with `Temporal` to avoid naming conflicts with custom Search Attributes. To make it easier to reference default Search Attributes in List Filters, Temporal supports aliasing, which lets you use the non-prefixed name of a default Search Attribute. However, if you choose to define a custom Search Attribute with the same name as the non-prefixed alias of a default Search Attribute, your custom Search Attribute overrides the alias. Server Version Requirement Search Attribute aliasing requires Temporal Server version 1.30 and greater. For example, the default Search Attribute `TemporalWorkflowVersioningBehavior` has the alias `WorkflowVersioningBehavior`. If you haven't defined a custom Search Attribute named `WorkflowVersioningBehavior`, you can use either name in a List Filter, and both refer to the same Search Attribute. -- Using the original attribute nameWorkflowVersioningBehavior = 'pinned'-- Using the Temporal-prefixed alias (equivalent)TemporalWorkflowVersioningBehavior = 'pinned' #### Alias resolution with custom Search Attributes[​](https://docs.temporal.io/list-filter#alias-resolution-with-custom-search-attributes "Direct link to Alias resolution with custom Search Attributes") When resolving a Search Attribute in a List Filter, Temporal Server checks for matches in the following order: 1. Custom Search Attributes defined in the current Namespace 2. Default Search Attributes This means that if you define a custom Search Attribute with the same name as the alias of a default Search Attribute, the non `Temporal` prefixed name will refer to your custom attribute. You can still search with the default Search Attribute by using the `Temporal` prefix. For example, if you have a custom Search Attribute named `SchedulePaused`, List Filters using the following Search Attributes will return different results: -- If you have a custom Search Attribute named 'SchedulePaused'-- This will use your custom attribute, not the default Search AttributeSchedulePaused = true-- The original system attribute still works by using the Temporal prefixTemporalSchedulePaused = true `SchedulePaused` will refer to your custom Search Attribute, while `TemporalSchedulePaused` will refer to the default Search Attribute. * [What is a List Filter?](https://docs.temporal.io/list-filter#list-filter) * [Supported operators](https://docs.temporal.io/list-filter#supported-operators) * [Partial string match](https://docs.temporal.io/list-filter#partial-string-match) * [Text](https://docs.temporal.io/list-filter#text) * [Keyword](https://docs.temporal.io/list-filter#keyword) * [Efficient API usage](https://docs.temporal.io/list-filter#efficient-api-usage) * [List Filter examples](https://docs.temporal.io/list-filter#list-filter-examples) * [Search Attribute aliasing](https://docs.temporal.io/list-filter#search-attribute-aliasing) * [Alias resolution with custom Search Attributes](https://docs.temporal.io/list-filter#alias-resolution-with-custom-search-attributes) --- # Temporal product release stages guide | Temporal Platform Documentation [Skip to main content](https://docs.temporal.io/evaluate/development-production-features/release-stages#__docusaurus_skipToContent_fallback) On this page CHANGELOG To stay up-to-date with the latest feature changes, visit the [changelog](https://temporal.io/change-log) . This Product Release Stages Guide provides an understanding of how Temporal features are released. It describes and lists the criteria for each release stage, so that you can make informed decisions about the adoption of each new feature. Product Release Guide Expectations: | | Pre-release | Public Preview | General Availability | | --- | --- | --- | --- | | **Features access** | Self-hosted Temporal users: Everyone; Temporal Cloud: Invite only. | Everyone. Temporal Cloud may limit the number of users being onboarded to ensure stability. | Everyone. | | **Feature completeness** | Limited functionality. | Core functionality is complete. | Mature and feature complete. | | **API stability** | Experimental; API is subject to change. | API breaking changes are kept to a minimum. | API is stable. | | **Feature region Availability** | Limited regions. | Most regions. | All [regions](https://docs.temporal.io/cloud/regions)
. | | **Feature support** | Community and engineering team. | [Formal support](https://docs.temporal.io/cloud/support#support-ticket)
. | [Formal support](https://docs.temporal.io/cloud/support#support-ticket)
. | | **Feature recommended usage** | Experimental. | Production use cases. | Production usage. | | **Feature Cloud pricing** | No additional cost. | Pricing changes are kept to a minimum. | Pricing is stable. | | **Feature Interoperability** | Limited. | Features are compatible with each other, unless otherwise stated. | Features are compatible with each other. | Pre-release[​](https://docs.temporal.io/evaluate/development-production-features/release-stages#pre-release "Direct link to Pre-release") ------------------------------------------------------------------------------------------------------------------------------------------ **Access:** Most Pre-release features are released in the open source Temporal software and are publicly available. However, some features which are explicit to hosting Temporal Services, such as [API Keys](https://docs.temporal.io/cloud/api-keys) , may be specific to Temporal Cloud. In Temporal Cloud, Pre-release features are invite-only: Temporal will work directly with a group of existing Temporal Cloud customers to be part of testing of each Pre-release feature. These customers are invited to provide feedback to the Temporal team. **Classification:** New features in Pre-release may not be fully mature and may have bugs. Users acknowledge and agree that Pre-release features are provided on an “as-is” basis, and that they are provided without any indemnification, support, warranties, or representation of any kind. **Feedback:** Feedback is highly encouraged and important for guiding Temporal feature development. We encourage you to share your experience so that you can influence the future direction of Temporal. **Availability:** Temporal may modify features before they become Generally Available, or may even decide to remove them. This means there is no guarantee that a new feature will become Generally Available. A Pre-release feature can be deprecated at any time. Pre-release features may be disabled by default, and can be enabled via configuration. Temporal Cloud customers can contact the Temporal account team or [Temporal Support Team](https://docs.temporal.io/cloud/support#support-ticket) to gain Pre-release access. Public Preview[​](https://docs.temporal.io/evaluate/development-production-features/release-stages#public-preview "Direct link to Public Preview") --------------------------------------------------------------------------------------------------------------------------------------------------- **Access:** New features in Public preview are available to everyone. **Classification:** Features in public preview may undergo further development and testing before they are made Generally Available. These features are being refined and are recommended for production usage. **Feedback:** Temporal users are invited to share feedback via the [Community Slack](http://t.mp/slack) , by reaching out directly to the Temporal team at [product@temporal.io](mailto:product@temporal.io) , or by creating issues in the relevant [GitHub repository](https://github.com/temporalio) . Temporal also encourages Temporal Cloud users to submit feedback via [support ticket](https://docs.temporal.io/cloud/support#support-ticket) . This feedback will assist in guiding the improvements for General Availability. **Availability:** New Features in Public Preview may evolve. The APIs may undergo changes; however, Temporal's goal is to maintain backward compatibility. General Availability[​](https://docs.temporal.io/evaluate/development-production-features/release-stages#general-availability "Direct link to General Availability") --------------------------------------------------------------------------------------------------------------------------------------------------------------------- **Access:** Features in General Availability are available to everyone. **Classification:** The feature is now fully developed, tested, and available for use without further anticipated changes. **Feedback:** Temporal users are invited to share feedback via the [Community Slack](http://t.mp/slack) , by reaching out directly to the Temporal team at [product@temporal.io](mailto:product@temporal.io) , or by creating issues in the relevant [GitHub repository](https://github.com/temporalio) . **Availability:** Features in General Availability are released with stable APIs and recommended for production use with a committed SLA. Exceptions There may be exceptions for different features, but this is the typical expectation. Any variation will be documented. * [Pre-release](https://docs.temporal.io/evaluate/development-production-features/release-stages#pre-release) * [Public Preview](https://docs.temporal.io/evaluate/development-production-features/release-stages#public-preview) * [General Availability](https://docs.temporal.io/evaluate/development-production-features/release-stages#general-availability) --- # Multi-tenant application patterns | Temporal Platform Documentation [Skip to main content](https://docs.temporal.io/production-deployment/multi-tenant-patterns#__docusaurus_skipToContent_fallback) On this page Many SaaS providers and large enterprise platform teams use a single Temporal [Namespace](https://docs.temporal.io/namespaces) with [per-tenant Task Queues](https://docs.temporal.io/production-deployment/multi-tenant-patterns#1-task-queues-per-tenant-recommended) to power their multi-tenant applications. This approach maximizes resource efficiency while maintaining logical separation between tenants. This guide covers architectural patterns, design considerations, and practical examples for building multi-tenant applications with Temporal. Architectural principles[​](https://docs.temporal.io/production-deployment/multi-tenant-patterns#architectural-principles "Direct link to Architectural principles") --------------------------------------------------------------------------------------------------------------------------------------------------------------------- When designing a multi-tenant Temporal application, follow these principles: * **Define your tenant model** - Determine what constitutes a tenant in your business (customers, pricing tiers, teams, etc.) * **Prefer simplicity** - Start with the simplest pattern that meets your needs * **Understand Temporal limits** - Design within the constraints of your Temporal deployment * **Test at scale** - Performance testing must drive your capacity decisions * **Plan for growth** - Consider how you'll onboard new tenants and scale workers Architectural patterns[​](https://docs.temporal.io/production-deployment/multi-tenant-patterns#architectural-patterns "Direct link to Architectural patterns") --------------------------------------------------------------------------------------------------------------------------------------------------------------- There are three main patterns for multi-tenant applications in Temporal, listed from most to least recommended: ### 1\. Task queues per tenant (Recommended)[​](https://docs.temporal.io/production-deployment/multi-tenant-patterns#1-task-queues-per-tenant-recommended "Direct link to 1. Task queues per tenant (Recommended)") **Use different [Task Queues](https://docs.temporal.io/task-queue) for each tenant's [Workflows](https://docs.temporal.io/workflows) and [Activities](https://docs.temporal.io/activities) .** This is the recommended pattern for most use cases. Each tenant gets dedicated Task Queue(s), with [Workers](https://docs.temporal.io/workers) polling multiple tenant Task Queues in a single process. **Pros:** * Strong isolation between tenants * Efficient resource utilization * Flexible worker scaling * Easy to add new tenants * Can handle thousands of tenants per [Namespace](https://docs.temporal.io/namespaces) **Cons:** * Requires worker configuration management * Potential for uneven resource distribution * Need to prevent "noisy neighbor" issues at the worker level Related 📚 [Task Queue Isolation Pattern Details](https://docs.temporal.io/production-deployment/multi-tenant-patterns#task-queue-isolation-pattern) feature-guide ### 2\. Shared Workflow Task Queues, separate Activity Task Queues[​](https://docs.temporal.io/production-deployment/multi-tenant-patterns#2-shared-workflow-task-queues-separate-activity-task-queues "Direct link to 2. Shared Workflow Task Queues, separate Activity Task Queues") **Share [Workflow Task Queues](https://docs.temporal.io/task-queue) but use different [Activity Task Queues](https://docs.temporal.io/task-queue) per tenant.** Use this pattern when [Workflows](https://docs.temporal.io/workflows) are lightweight but [Activities](https://docs.temporal.io/activities) have heavy resource requirements or external dependencies that need isolation. **Pros:** * Easier worker management than full isolation * Activity-level tenant isolation * Good for compute-intensive Activities **Cons:** * Less isolation than pattern #1 * Workflow visibility is shared * More complex to reason about ### 3\. Namespace per tenant[​](https://docs.temporal.io/production-deployment/multi-tenant-patterns#3-namespace-per-tenant "Direct link to 3. Namespace per tenant") **Use a separate [Namespace](https://docs.temporal.io/namespaces) for each tenant.** Only practical for a small number (< 50) of high-value tenants due to operational overhead. **Pros:** * Complete isolation between tenants * Per-tenant rate limiting * Maximum security **Cons:** * Higher operational overhead * Credential and connectivity management per [Namespace](https://docs.temporal.io/namespaces) * Requires more [Workers](https://docs.temporal.io/workers) (minimum 2 per Namespace for high availability) * Expensive at scale Related 📚 [Namespace Isolation in Temporal Cloud](https://docs.temporal.io/evaluate/development-production-features/multi-tenancy#namespace-isolation) Task Queue isolation pattern[​](https://docs.temporal.io/production-deployment/multi-tenant-patterns#task-queue-isolation-pattern "Direct link to Task Queue isolation pattern") --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- This section details the recommended pattern for most multi-tenant applications. ### Worker design[​](https://docs.temporal.io/production-deployment/multi-tenant-patterns#worker-design "Direct link to Worker design") When a [Worker](https://docs.temporal.io/workers) starts up: 1. **Load tenant configuration** - Retrieve the list of tenants this Worker should handle (from config file, API, or database) 2. **Create [Task Queues](https://docs.temporal.io/task-queue) ** - For each tenant, generate a unique Task Queue name (e.g., `customer-{tenant-id}`) 3. **Register [Workflows](https://docs.temporal.io/workflows) and [Activities](https://docs.temporal.io/activities) ** - Register your Workflow and Activity implementations once, passing the tenant-specific Task Queue name 4. **Poll multiple Task Queues** - A single Worker process polls all assigned tenant Task Queues // Example: Go worker polling multiple tenant Task Queuesfor _, tenant := range assignedTenants { taskQueue := fmt.Sprintf("customer-%s", tenant.ID) worker := worker.New(client, taskQueue, worker.Options{}) worker.RegisterWorkflow(YourWorkflow) worker.RegisterActivity(YourActivity)} ### Routing requests to Task Queues[​](https://docs.temporal.io/production-deployment/multi-tenant-patterns#routing-requests-to-task-queues "Direct link to Routing requests to Task Queues") Your application needs to route [Workflow](https://docs.temporal.io/workflows) starts and other operations to the correct tenant [Task Queue](https://docs.temporal.io/task-queue) : // Example: Starting a Workflow for a specific tenanttaskQueue := fmt.Sprintf("customer-%s", tenantID)workflowOptions := client.StartWorkflowOptions{ ID: workflowID, TaskQueue: taskQueue,} Consider creating an API or service that: * Maps tenant IDs to Task Queue names * Tracks which [Workers](https://docs.temporal.io/workers) are handling which tenants * Allows both your application and Workers to read the mappings of: 1. Tenant IDs to Task Queues 2. Workers to tenants ### Capacity planning[​](https://docs.temporal.io/production-deployment/multi-tenant-patterns#capacity-planning "Direct link to Capacity planning") Key questions to answer through performance testing: **[Namespace](https://docs.temporal.io/namespaces) capacity:** * How many concurrent [Task Queue](https://docs.temporal.io/task-queue) pollers can your Namespace support? * What are your [Actions Per Second (APS)](https://docs.temporal.io/cloud/limits#actions-per-second) limits? * What are your [Operations Per Second (OPS)](https://docs.temporal.io/references/operation-list) limits? **[Worker](https://docs.temporal.io/workers) capacity:** * How many tenants can a single Worker process handle? * What are the CPU and memory requirements per tenant? * How many concurrent [Workflow](https://docs.temporal.io/workflows) executions per tenant? * How many concurrent [Activity](https://docs.temporal.io/activities) executions per tenant? **SDK configuration to tune:** * `MaxConcurrentWorkflowTaskExecutionSize` * `MaxConcurrentActivityExecutionSize` * `MaxConcurrentWorkflowTaskPollers` * `MaxConcurrentActivityTaskPollers` * Worker replicas (in Kubernetes deployments) ### Provisioning new tenants[​](https://docs.temporal.io/production-deployment/multi-tenant-patterns#provisioning-new-tenants "Direct link to Provisioning new tenants") Automate tenant onboarding with a Temporal [Workflow](https://docs.temporal.io/workflows) : 1. Create a tenant onboarding Workflow that: * Validates tenant information * Provisions infrastructure * Deploys/updates [Worker](https://docs.temporal.io/workers) configuration * Triggers Worker restarts or scaling * Verifies the tenant is operational 2. Store tenant-to-Worker mappings in a database or configuration service 3. Update Worker deployments to pick up new tenant assignments Practical example[​](https://docs.temporal.io/production-deployment/multi-tenant-patterns#practical-example "Direct link to Practical example") ------------------------------------------------------------------------------------------------------------------------------------------------ **Scenario:** A SaaS company has 1,000 customers and expects to grow to 5,000 customers over 3 years. They have 2 [Workflows](https://docs.temporal.io/workflows) and ~25 [Activities](https://docs.temporal.io/activities) per Workflow. All customers are on the same tier (no segmentation yet). ### Assumptions[​](https://docs.temporal.io/production-deployment/multi-tenant-patterns#assumptions "Direct link to Assumptions") | Item | Value | | --- | --- | | Current customers | 1,000 | | Workflow Task Queues per customer | 1 | | Activity Task Queues per customer | 1 | | Max Task Queue pollers per Namespace | 5,000 | | SDK concurrent Workflow task pollers | 5 | | SDK concurrent Activity task pollers | 5 | | Max concurrent Workflow executions | 200 | | Max concurrent Activity executions | 200 | ### Capacity calculations[​](https://docs.temporal.io/production-deployment/multi-tenant-patterns#capacity-calculations "Direct link to Capacity calculations") **[Task Queue](https://docs.temporal.io/task-queue) poller limits:** * Each [Worker](https://docs.temporal.io/workers) uses 10 pollers per tenant (5 Workflow + 5 Activity) * Maximum Workers in [Namespace](https://docs.temporal.io/namespaces) : 5,000 pollers ÷ 10 = **500 Workers** **Worker capacity:** * Each Worker can theoretically handle 200 [Workflows](https://docs.temporal.io/workflows) and 200 [Activities](https://docs.temporal.io/activities) concurrently * Conservative estimate: **250 tenants per Worker** (accounting for overhead) * For 1,000 customers: **4 Workers minimum** (plus replicas for HA) * For 5,000 customers: **20 Workers minimum** (plus replicas for HA) **Namespace capacity:** * At 250 tenants per Worker, need 2 Workers per group of tenants (for HA) * Maximum tenants in Namespace: (500 Workers ÷ 2) × 250 = **62,500 tenants** note These are theoretical calculations based on SDK defaults. **Always perform load testing** to determine actual capacity for your specific workload. Monitor CPU, memory, and Temporal metrics during testing. While testing, also pay attention to your [metrics capacity and cardinality](https://docs.temporal.io/cloud/metrics/openmetrics/api-reference#managing-high-cardinality) . ### Worker assignment strategies[​](https://docs.temporal.io/production-deployment/multi-tenant-patterns#worker-assignment-strategies "Direct link to Worker assignment strategies") **Option 1: Static configuration** * Each [Worker](https://docs.temporal.io/workers) reads a config file listing assigned tenant IDs * Simple to implement * Requires deployment to add tenants **Option 2: Dynamic API** * Workers call an API on startup to get assigned tenants * Workers identified by static ID (1 to N) * API returns tenant list based on Worker ID * More flexible, no deployment needed for new tenants Best practices[​](https://docs.temporal.io/production-deployment/multi-tenant-patterns#best-practices "Direct link to Best practices") --------------------------------------------------------------------------------------------------------------------------------------- ### Monitoring[​](https://docs.temporal.io/production-deployment/multi-tenant-patterns#monitoring "Direct link to Monitoring") Track these [metrics](https://docs.temporal.io/references/sdk-metrics) per tenant: * [Workflow completion](https://docs.temporal.io/cloud/metrics/openmetrics/metrics-reference#workflow-completion-metrics) rates * [Activity execution](https://docs.temporal.io/cloud/metrics/openmetrics/metrics-reference#task-queue-metrics) rates * [Task Queue backlog](https://docs.temporal.io/cloud/metrics/openmetrics/metrics-reference#task-queue-metrics) * [Worker resource utilization](https://docs.temporal.io/references/sdk-metrics#worker_task_slots_used) * [Workflow failure rates](https://docs.temporal.io/encyclopedia/detecting-workflow-failures) ### Handling noisy neighbors[​](https://docs.temporal.io/production-deployment/multi-tenant-patterns#handling-noisy-neighbors "Direct link to Handling noisy neighbors") Even with [Task Queue](https://docs.temporal.io/task-queue) isolation, monitor for tenants that: * Generate excessive load * Have high failure rates * Cause [Worker](https://docs.temporal.io/workers) resource exhaustion Strategies: * Implement per-tenant rate limiting in your application * Move problematic tenants to dedicated Workers * Use [Workflow](https://docs.temporal.io/workflows) /[Activity](https://docs.temporal.io/activities) timeouts aggressively ### Tenant lifecycle[​](https://docs.temporal.io/production-deployment/multi-tenant-patterns#tenant-lifecycle "Direct link to Tenant lifecycle") Plan for: * **Onboarding** - Automated provisioning [Workflow](https://docs.temporal.io/workflows) * **Scaling** - When to add new [Workers](https://docs.temporal.io/workers) for growing tenants * **Offboarding** - Graceful tenant removal and data cleanup * **Rebalancing** - Redistributing tenants across Workers ### Search Attributes[​](https://docs.temporal.io/production-deployment/multi-tenant-patterns#search-attributes "Direct link to Search Attributes") Use [Search Attributes](https://docs.temporal.io/search-attribute) to enable tenant-scoped queries: // Add tenant ID as a Search AttributesearchAttributes := map[string]interface{}{ "TenantId": tenantID,} This allows filtering [Workflows](https://docs.temporal.io/workflows) by tenant in the UI and SDK: TenantId = 'customer-123' AND ExecutionStatus = 'Running' Related resources[​](https://docs.temporal.io/production-deployment/multi-tenant-patterns#related-resources "Direct link to Related resources") ------------------------------------------------------------------------------------------------------------------------------------------------ Related 📚 * [Multi-tenancy Overview](https://docs.temporal.io/evaluate/development-production-features/multi-tenancy) feature-guide * [Temporal Cloud Limits](https://docs.temporal.io/cloud/limits) * [Visibility and Search Attributes](https://docs.temporal.io/visibility) feature-guide * [Architectural principles](https://docs.temporal.io/production-deployment/multi-tenant-patterns#architectural-principles) * [Architectural patterns](https://docs.temporal.io/production-deployment/multi-tenant-patterns#architectural-patterns) * [1\. Task queues per tenant (Recommended)](https://docs.temporal.io/production-deployment/multi-tenant-patterns#1-task-queues-per-tenant-recommended) * [2\. Shared Workflow Task Queues, separate Activity Task Queues](https://docs.temporal.io/production-deployment/multi-tenant-patterns#2-shared-workflow-task-queues-separate-activity-task-queues) * [3\. Namespace per tenant](https://docs.temporal.io/production-deployment/multi-tenant-patterns#3-namespace-per-tenant) * [Task Queue isolation pattern](https://docs.temporal.io/production-deployment/multi-tenant-patterns#task-queue-isolation-pattern) * [Worker design](https://docs.temporal.io/production-deployment/multi-tenant-patterns#worker-design) * [Routing requests to Task Queues](https://docs.temporal.io/production-deployment/multi-tenant-patterns#routing-requests-to-task-queues) * [Capacity planning](https://docs.temporal.io/production-deployment/multi-tenant-patterns#capacity-planning) * [Provisioning new tenants](https://docs.temporal.io/production-deployment/multi-tenant-patterns#provisioning-new-tenants) * [Practical example](https://docs.temporal.io/production-deployment/multi-tenant-patterns#practical-example) * [Assumptions](https://docs.temporal.io/production-deployment/multi-tenant-patterns#assumptions) * [Capacity calculations](https://docs.temporal.io/production-deployment/multi-tenant-patterns#capacity-calculations) * [Worker assignment strategies](https://docs.temporal.io/production-deployment/multi-tenant-patterns#worker-assignment-strategies) * [Best practices](https://docs.temporal.io/production-deployment/multi-tenant-patterns#best-practices) * [Monitoring](https://docs.temporal.io/production-deployment/multi-tenant-patterns#monitoring) * [Handling noisy neighbors](https://docs.temporal.io/production-deployment/multi-tenant-patterns#handling-noisy-neighbors) * [Tenant lifecycle](https://docs.temporal.io/production-deployment/multi-tenant-patterns#tenant-lifecycle) * [Search Attributes](https://docs.temporal.io/production-deployment/multi-tenant-patterns#search-attributes) * [Related resources](https://docs.temporal.io/production-deployment/multi-tenant-patterns#related-resources) --- # Worker tuning quick reference | Temporal Platform Documentation [Skip to main content](https://docs.temporal.io/develop/worker-tuning-reference#__docusaurus_skipToContent_fallback) On this page This page provides a quick reference for Worker configuration options and their default values across Temporal SDKs. Use this guide alongside the comprehensive [Worker performance](https://docs.temporal.io/develop/worker-performance) documentation for detailed tuning guidance. Worker performance is constrained by three primary resources: | Resource | Description | | --- | --- | | **Compute** | CPU-bound operations, concurrent Task execution | | **Memory** | Workflow cache, thread pools | | **IO** | Network calls to Temporal Service, polling | How a Worker works[​](https://docs.temporal.io/develop/worker-tuning-reference#how-a-worker-works "Direct link to How a Worker works") --------------------------------------------------------------------------------------------------------------------------------------- Workers poll a [Task Queue](https://docs.temporal.io/task-queue) in Temporal Cloud or a self-hosted Temporal Service, execute Tasks, and respond with the result. ┌─────────────────┐ Poll for Tasks ┌──────────────────┐│ - Worker │ ◄─────────────────────── │ Temporal Service ││ - Workflows │ │ ││ - Activities │ ───────────────────────► │ │└─────────────────┘ Respond with results └──────────────────┘ Multiple Workers can poll the same Task Queue, providing horizontal scalability. ### How Worker failure recovery works[​](https://docs.temporal.io/develop/worker-tuning-reference#how-worker-failure-recovery-works "Direct link to How Worker failure recovery works") When a Worker crashes or experiences a host outage: 1. The Workflow Task times out 2. Another available Worker picks up the Task 3. The new Worker replays the Event History to reconstruct state 4. Execution continues from where it left off For more details on Worker architecture, see [What is a Temporal Worker?](https://docs.temporal.io/workers) Compute settings[​](https://docs.temporal.io/develop/worker-tuning-reference#compute-settings "Direct link to Compute settings") --------------------------------------------------------------------------------------------------------------------------------- Compute settings control how many Tasks a Worker can execute concurrently. ### Compute configuration options[​](https://docs.temporal.io/develop/worker-tuning-reference#compute-configuration-options "Direct link to Compute configuration options") | Setting | Description | | --- | --- | | `MaxConcurrentWorkflowTaskExecutionSize` | Maximum concurrent Workflow Tasks | | `MaxConcurrentActivityTaskExecutionSize` | Maximum concurrent Activity Tasks | | `MaxConcurrentLocalActivityTaskExecutionSize` | Maximum concurrent Local Activities | | `MaxWorkflowThreadCount` / `workflowThreadPoolSize` | Thread pool for Workflow execution | ### Compute defaults by SDK[​](https://docs.temporal.io/develop/worker-tuning-reference#compute-defaults-by-sdk "Direct link to Compute defaults by SDK") | SDK | MaxConcurrentWorkflowTaskExecutionSize | MaxConcurrentActivityTaskExecutionSize | MaxConcurrentLocalActivityTaskExecutionSize | MaxWorkflowThreadCount | | --- | --- | --- | --- | --- | | **Go** | 1,000 | 1,000 | 1,000 | \- | | **Java** | 200 | 200 | 200 | 600 | | **TypeScript** | 40 | 100 | 100 | 1 (reuseV8Context) | | **Python** | 100 | 100 | 100 | \- | | **.NET** | 100 | 100 | 100 | \- | ### Resource-based slot suppliers[​](https://docs.temporal.io/develop/worker-tuning-reference#resource-based-slot-suppliers "Direct link to Resource-based slot suppliers") Instead of fixed slot counts, you can use resource-based slot suppliers that automatically adjust available Task slots based on CPU and memory utilization. For implementation details, see [Slot suppliers](https://docs.temporal.io/develop/worker-performance#slot-suppliers) . Memory settings[​](https://docs.temporal.io/develop/worker-tuning-reference#memory-settings "Direct link to Memory settings") ------------------------------------------------------------------------------------------------------------------------------ Memory settings control the Workflow cache size and thread pool allocation. ### Memory configuration options[​](https://docs.temporal.io/develop/worker-tuning-reference#memory-configuration-options "Direct link to Memory configuration options") | Setting | Description | | --- | --- | | `MaxCachedWorkflows` / `StickyWorkflowCacheSize` | Number of Workflows to keep in cache | | `MaxWorkflowThreadCount` | Thread pool size | | `reuseV8Context` (TypeScript) | Reuse V8 context for Workflows | ### Memory defaults by SDK[​](https://docs.temporal.io/develop/worker-tuning-reference#memory-defaults-by-sdk "Direct link to Memory defaults by SDK") | SDK | MaxCachedWorkflows / StickyWorkflowCacheSize | | --- | --- | | **Go** | 10,000 | | **Java** | 600 | | **TypeScript** | Dynamic (e.g., 2000 for 4 GiB RAM) | | **Python** | 1,000 | | **.NET** | 10,000 | For cache tuning guidance, see [Workflow cache tuning](https://docs.temporal.io/develop/worker-performance#workflow-cache-tuning) . IO settings[​](https://docs.temporal.io/develop/worker-tuning-reference#io-settings "Direct link to IO settings") ------------------------------------------------------------------------------------------------------------------ IO settings control the number of pollers and rate limits for Task Queue interactions. ### IO configuration options[​](https://docs.temporal.io/develop/worker-tuning-reference#io-configuration-options "Direct link to IO configuration options") | Setting | Description | | --- | --- | | `MaxConcurrentWorkflowTaskPollers` | Number of concurrent Workflow pollers | | `MaxConcurrentActivityTaskPollers` | Number of concurrent Activity pollers | | `Namespace APS` | Actions per second limit for Namespace | | `TaskQueueActivitiesPerSecond` | Activity rate limit per Task Queue | ### IO defaults by SDK[​](https://docs.temporal.io/develop/worker-tuning-reference#io-defaults-by-sdk "Direct link to IO defaults by SDK") | SDK | MaxConcurrentWorkflowTaskPollers | MaxConcurrentActivityTaskPollers | Namespace APS | TaskQueueActivitiesPerSecond | | --- | --- | --- | --- | --- | | **Go** | 2 | 2 | 400 | Unlimited | | **Java** | 5 | 5 | \- | \- | | **TypeScript** | 10 | 10 | \- | \- | | **Python** | 5 | 5 | \- | \- | | **.NET** | 5 | 5 | \- | \- | ### Poller autoscaling[​](https://docs.temporal.io/develop/worker-tuning-reference#poller-autoscaling "Direct link to Poller autoscaling") Use poller autoscaling to automatically adjust the number of concurrent polls based on workload. For configuration details, see [Configuring poller options](https://docs.temporal.io/develop/worker-performance#configuring-poller-options) . Metrics reference by resource type[​](https://docs.temporal.io/develop/worker-tuning-reference#metrics-reference-by-resource-type "Direct link to Metrics reference by resource type") --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Use these metrics to identify bottlenecks and guide tuning decisions. For the complete metrics reference, see [SDK metrics](https://docs.temporal.io/references/sdk-metrics) . ### Compute-related metrics[​](https://docs.temporal.io/develop/worker-tuning-reference#compute-related-metrics "Direct link to Compute-related metrics") | Worker configuration option | SDK metric | | --- | --- | | `MaxConcurrentWorkflowTaskExecutionSize` | [`worker_task_slots_available {worker_type = WorkflowWorker}`](https://docs.temporal.io/references/sdk-metrics#worker_task_slots_available) | | `MaxConcurrentActivityTaskExecutionSize` | [`worker_task_slots_available {worker_type = ActivityWorker}`](https://docs.temporal.io/references/sdk-metrics#worker_task_slots_available) | | `MaxWorkflowThreadCount` | [`workflow_active_thread_count`](https://docs.temporal.io/references/sdk-metrics#workflow_active_thread_count)
(Java only) | | CPU-intensive logic | [`workflow_task_execution_latency`](https://docs.temporal.io/references/sdk-metrics#workflow_task_execution_latency) | Also monitor your machine's CPU consumption (for example, `container_cpu_usage_seconds_total` in Kubernetes). ### Memory-related metrics[​](https://docs.temporal.io/develop/worker-tuning-reference#memory-related-metrics "Direct link to Memory-related metrics") | Worker configuration option | SDK metric | | --- | --- | | `StickyWorkflowCacheSize` | [`sticky_cache_total_forced_eviction`](https://docs.temporal.io/references/sdk-metrics#sticky_cache_total_forced_eviction)
, [`sticky_cache_size`](https://docs.temporal.io/references/sdk-metrics#sticky_cache_size)
, [`sticky_cache_hit`](https://docs.temporal.io/references/sdk-metrics#sticky_cache_hit)
, [`sticky_cache_miss`](https://docs.temporal.io/references/sdk-metrics#sticky_cache_miss) | Also monitor your machine's memory consumption (for example, `container_memory_usage_bytes` in Kubernetes). ### IO-related metrics[​](https://docs.temporal.io/develop/worker-tuning-reference#io-related-metrics "Direct link to IO-related metrics") | Worker configuration option | SDK metric | | --- | --- | | `MaxConcurrentWorkflowTaskPollers` | [`num_pollers {poller_type = workflow_task}`](https://docs.temporal.io/references/sdk-metrics#num_pollers) | | `MaxConcurrentActivityTaskPollers` | [`num_pollers {poller_type = activity_task}`](https://docs.temporal.io/references/sdk-metrics#num_pollers) | | Network latency | [`request_latency {namespace, operation}`](https://docs.temporal.io/references/sdk-metrics#request_latency) | ### Task Queue metrics[​](https://docs.temporal.io/develop/worker-tuning-reference#task-queue-metrics "Direct link to Task Queue metrics") | Metric | Description | | --- | --- | | [`poll_success_sync_count`](https://docs.temporal.io/cloud/metrics/reference#temporal_cloud_v0_poll_success_sync_count) | Sync match rate (Tasks immediately assigned to Workers) | | [`approximate_backlog_count`](https://docs.temporal.io/cloud/metrics/openmetrics/metrics-reference#temporal_cloud_v1_approximate_backlog_count) | Approximate number of Tasks in a Task Queue | Task Queue statistics are also available via the `DescribeTaskQueue` API: * `ApproximateBacklogCount` * `ApproximateBacklogAge` * `TasksAddRate` * `TasksDispatchRate` * `BacklogIncreaseRate` For more on Task Queue metrics, see [Available Task Queue information](https://docs.temporal.io/develop/worker-performance#task-queue-metrics) . ### Failure metrics[​](https://docs.temporal.io/develop/worker-tuning-reference#failure-metrics "Direct link to Failure metrics") | Metric | Description | | --- | --- | | [`long_request_failure`](https://docs.temporal.io/references/sdk-metrics#long_request_failure) | Failures for long-running operations (polling, history retrieval) | | [`request_failure`](https://docs.temporal.io/references/sdk-metrics#request_failure) | Failures for standard operations (Task completion responses) | Common failure codes: * `RESOURCE_EXHAUSTED` - Rate limits exceeded * `DEADLINE_EXCEEDED` - Operation timeout * `NOT_FOUND` - Resource not found Worker tuning tips[​](https://docs.temporal.io/develop/worker-tuning-reference#worker-tuning-tips "Direct link to Worker tuning tips") --------------------------------------------------------------------------------------------------------------------------------------- 1. **Scale test before production**: Validate your configuration under realistic load. 2. **Infrastructure matters**: Workers don't operate in a vacuum. Consider network latency, database performance, and external service dependencies. 3. **Tune and observe**: Make incremental changes and monitor metrics before making additional adjustments. 4. **Identify the bottleneck**: Use the [theory of constraints](https://en.wikipedia.org/wiki/Theory_of_constraints) . Improving non-bottleneck resources won't improve overall throughput. For detailed tuning guidance, see: * [Worker performance](https://docs.temporal.io/develop/worker-performance) * [Worker deployment and performance best practices](https://docs.temporal.io/best-practices/worker) * [Performance bottlenecks troubleshooting](https://docs.temporal.io/troubleshooting/performance-bottlenecks) Related resources[​](https://docs.temporal.io/develop/worker-tuning-reference#related-resources "Direct link to Related resources") ------------------------------------------------------------------------------------------------------------------------------------ * [What is a Temporal Worker?](https://docs.temporal.io/workers) - Conceptual overview * [Worker performance](https://docs.temporal.io/develop/worker-performance) - Comprehensive tuning guide * [Worker deployment and performance](https://docs.temporal.io/best-practices/worker) - Best practices * [SDK metrics reference](https://docs.temporal.io/references/sdk-metrics) - Complete metrics documentation * [Worker Versioning](https://docs.temporal.io/production-deployment/worker-deployments/worker-versioning) - Safe deployments * [Workers in production](https://temporal.io/blog/workers-in-production) - Blog post * [Introduction to Worker Tuning](https://temporal.io/blog/an-introduction-to-worker-tuning) - Blog post * [How a Worker works](https://docs.temporal.io/develop/worker-tuning-reference#how-a-worker-works) * [How Worker failure recovery works](https://docs.temporal.io/develop/worker-tuning-reference#how-worker-failure-recovery-works) * [Compute settings](https://docs.temporal.io/develop/worker-tuning-reference#compute-settings) * [Compute configuration options](https://docs.temporal.io/develop/worker-tuning-reference#compute-configuration-options) * [Compute defaults by SDK](https://docs.temporal.io/develop/worker-tuning-reference#compute-defaults-by-sdk) * [Resource-based slot suppliers](https://docs.temporal.io/develop/worker-tuning-reference#resource-based-slot-suppliers) * [Memory settings](https://docs.temporal.io/develop/worker-tuning-reference#memory-settings) * [Memory configuration options](https://docs.temporal.io/develop/worker-tuning-reference#memory-configuration-options) * [Memory defaults by SDK](https://docs.temporal.io/develop/worker-tuning-reference#memory-defaults-by-sdk) * [IO settings](https://docs.temporal.io/develop/worker-tuning-reference#io-settings) * [IO configuration options](https://docs.temporal.io/develop/worker-tuning-reference#io-configuration-options) * [IO defaults by SDK](https://docs.temporal.io/develop/worker-tuning-reference#io-defaults-by-sdk) * [Poller autoscaling](https://docs.temporal.io/develop/worker-tuning-reference#poller-autoscaling) * [Metrics reference by resource type](https://docs.temporal.io/develop/worker-tuning-reference#metrics-reference-by-resource-type) * [Compute-related metrics](https://docs.temporal.io/develop/worker-tuning-reference#compute-related-metrics) * [Memory-related metrics](https://docs.temporal.io/develop/worker-tuning-reference#memory-related-metrics) * [IO-related metrics](https://docs.temporal.io/develop/worker-tuning-reference#io-related-metrics) * [Task Queue metrics](https://docs.temporal.io/develop/worker-tuning-reference#task-queue-metrics) * [Failure metrics](https://docs.temporal.io/develop/worker-tuning-reference#failure-metrics) * [Worker tuning tips](https://docs.temporal.io/develop/worker-tuning-reference#worker-tuning-tips) * [Related resources](https://docs.temporal.io/develop/worker-tuning-reference#related-resources) --- # Monitor Temporal Platform metrics | Temporal Platform Documentation [Skip to main content](https://docs.temporal.io/self-hosted-guide/monitoring#__docusaurus_skipToContent_fallback) On this page The Temporal Service and SDKs emit metrics that can be used to monitor performance and troubleshoot issues. You can relay these metrics to any monitoring and observability platform. This guide will provide an example of configuring [Prometheus](https://prometheus.io/) and [Grafana](https://grafana.com/) to work with the observability metrics emitted from Temporal. This solution can work on its own, or serve as a baseline for you to further customize and integrate with other observability tooling. For example, it is also possible to use the [OpenTelemetry Collector](https://temporal.io/code-exchange/temporal-opentelemetry) in your stack instead of scraping metrics directly with Prometheus, or [Datadog](https://docs.temporal.io/self-hosted-guide/monitoring#datadog) as a frontend instead of Grafana. This configuration assumes that you have [Docker](https://www.docker.com/) installed and are running a [Temporal dev server](https://temporal.io/setup/start-development-server) via the CLI. Prometheus[​](https://docs.temporal.io/self-hosted-guide/monitoring#prometheus "Direct link to Prometheus") ------------------------------------------------------------------------------------------------------------ This section discusses exporting metrics from Temporal SDKs, and setting up Prometheus to collect metrics on Temporal Service, Temporal Client, and Temporal Worker performance. The Temporal Service and SDKs emit all metrics by default. However, you must enable Prometheus in your application code (using the Temporal SDKs) and your Temporal Service configuration to collect the metrics emitted from your SDK and Temporal Service. First, you'll need to create a `prometheus.yml` configuration file with some target ports to collect metrics from. Here is a sample with one Temporal Service metrics target and two Temporal Worker (SDK) metrics targets: global: scrape_interval: 10sscrape_configs: - job_name: 'temporalmetrics' metrics_path: /metrics scheme: http static_configs: # Temporal Service metrics target - targets: - 'host.docker.internal:8000' labels: group: 'server-metrics' # Local app targets (set in SDK code) - targets: - 'host.docker.internal:8077' - 'host.docker.internal:8078' labels: group: 'sdk-metrics' In this example, Prometheus is configured to scrape at 10-second intervals and to listen for Temporal Service metrics on `host.docker.internal:8000` and SDK metrics on two targets, `host.docker.internal:8077` and `host.docker.internal:8078`. The `8077` and `8078` ports must be set on `WorkflowServiceStubs` in your application code with your preferred SDK -- there is an example of this in the next section. You can set up as many targets as required. info For further Prometheus configuration options, refer to the [Prometheus documentation](https://prometheus.io/docs/prometheus/latest/configuration/configuration/) . You can use Docker to run the official Prometheus image with this configuration: docker run -p 9090:9090 -v /path/to/prometheus.yml /etc/prometheus/prometheus.yml prom/prometheus Next, launch your Temporal dev server from the CLI with an additional `--metrics-port 8000` parameter: temporal server start-dev --metrics-port 8000 info Refer to the [Temporal Cluster configuration reference](https://docs.temporal.io/references/configuration#global) to expose metrics from a production service. You should now have both Prometheus and a Temporal Service running locally, with Temporal providing Service metrics to Prometheus. Next, you'll want to configure SDK metrics as well. ### SDK metrics setup[​](https://docs.temporal.io/self-hosted-guide/monitoring#sdk-metrics-setup "Direct link to SDK metrics setup") SDK metrics are emitted by Temporal Workers and other Clients, and must be configured in your application code. The Metrics section in the Observability guide details how to create hooks for all supported SDKs: * [Go](https://docs.temporal.io/develop/go/platform/observability#metrics) * [Java](https://docs.temporal.io/develop/java/platform/observability#metrics) * [PHP](https://docs.temporal.io/develop/php/platform/observability) * [Python](https://docs.temporal.io/develop/python/platform/observability#metrics) * [TypeScript](https://docs.temporal.io/develop/typescript/platform/observability#metrics) * [.NET](https://docs.temporal.io/develop/dotnet/platform/observability#metrics) * [Ruby](https://docs.temporal.io/develop/ruby/platform/observability#metrics) For end-to-end examples of how to expose metrics from each SDK, see the metrics samples: * [Go SDK Sample](https://github.com/temporalio/samples-go/tree/main/metrics) * [Java SDK Sample](https://github.com/temporalio/samples-java/tree/main/core/src/main/java/io/temporal/samples/metrics) * [Python SDK Sample](https://github.com/temporalio/samples-python/tree/main/prometheus) * [TypeScript SDK Sample](https://github.com/temporalio/samples-typescript/tree/main/interceptors-opentelemetry) * [.NET SDK Sample](https://github.com/temporalio/samples-dotnet/tree/main/src/OpenTelemetry) Some of these may require you to set different metrics port numbers based on the Prometheus example here, which is configured to scrape port `8077` and `8078` by default. Follow the instructions from each of the samples to run Workflows and begin emitting metrics. This will allow you to populate a dashboard in the next section and understand how to further customize Temporal observability for your needs. ### Verifying Prometheus configuration[​](https://docs.temporal.io/self-hosted-guide/monitoring#verifying-prometheus-configuration "Direct link to Verifying Prometheus configuration") Once your Workflows are running and emitting metrics, you can visit [http://localhost:9090/targets](http://localhost:9090/targets) on your local Prometheus instance to verify that it is able to scrape the provided endpoints. ![Prometheus scrape targets](https://docs.temporal.io/assets/images/prometheus-targets-425943a103bdfe0251cd5892f330d658.png) This example shows a response from the server metrics endpoint, provided by the Temporal dev server, and two SDK metrics endpoints, as defined in the Prometheus configuration. To create this example, we used the Go and Python metrics samples, running on port 8077 and 8088 respectively. If you are not pushing data to exactly 3 metrics endpoints, your environment may be different. Next, you can visit the [local Prometheus query endpoint](http://localhost:9090/query) to manually run [PromQL](https://prometheus.io/docs/prometheus/latest/querying/basics/) queries on your exported metrics, or proceed to the next section to configure Grafana to generate dashboards from those metrics. Grafana[​](https://docs.temporal.io/self-hosted-guide/monitoring#grafana "Direct link to Grafana") --------------------------------------------------------------------------------------------------- With [Prometheus](https://docs.temporal.io/self-hosted-guide/monitoring#prometheus) configured, deploy Grafana as a metrics frontend, and configure it to use Prometheus as a data source. As before, you can use Docker to run the official Grafana image: docker run -d -p 3000:3000 grafana/grafana-enterprise This will deploy a Grafana instance with a default username and password of `admin`/`admin`. In production, you would want to [configure authentication](https://grafana.com/docs/grafana/latest/setup-grafana/configure-security/configure-authentication/generic-oauth/) and control port access to Grafana. info For more information on how to customize your Grafana setup, see the [Grafana documentation](https://grafana.com/docs/grafana/latest/setup-grafana/) . Next, configure Grafana to use Prometheus as the data source. To do this, click on "Add new data source" from the "Connections" menu in the Grafana sidebar, and add Prometheus from the list. You will be prompted to add additional configuration parameters. If you are following this guide using Docker, use `http://host.docker.internal:9090` as the Prometheus address. This is a [DNS name provided by Docker Desktop](https://docs.docker.com/desktop/features/networking/#use-cases-and-workarounds) which resolves to the internal IP address used by the host machine, and allows you to connect applications across Docker containers without additional configuration rules. This is the only parameter you will need to set for your Prometheus configuration. After providing it, scroll down to the "Save and Test" button, and you can validate Prometheus as a data source for this Grafana instance. ![Grafana data sources](https://docs.temporal.io/assets/images/grafana-data-sources-34a3d47303731470e2abdb6699bc8a3b.png) In this example, Grafana is set to pull metrics from Prometheus at the port 9090, as defined in the Prometheus configuration. Now, you'll just need to add some of our provided dashboards for visualizing Temporal metrics. ### Dashboard setup[​](https://docs.temporal.io/self-hosted-guide/monitoring#dashboard-setup "Direct link to Dashboard setup") We provide community-driven Grafana dashboards that can be used for monitoring Temporal Server and SDK metrics in a [dashboards](https://github.com/temporalio/dashboards/) repo. Follow the instructions in that repo's README to import the dashboards to Grafana. This way, you can create at least one dashboard for monitoring server metrics: ![Grafana server metrics](https://docs.temporal.io/assets/images/grafana-server-metrics-d704168a8130db3f74a4e2bdee63be31.png) And at least one other dashboard for monitoring SDK metrics: ![Grafana SDK metrics](https://docs.temporal.io/assets/images/grafana-sdk-metrics-d4e217417b3b2aa13cb942b9f0b222c4.png) info You can provide additional queries in your dashboard to report other data as needed. For more details on configuring Grafana dashboards, see the [Grafana Dashboards documentation](https://grafana.com/docs/grafana/latest/dashboards/) . From here, you can configure Grafana [Alerts](https://grafana.com/docs/grafana/latest/alerting/) for any monitored parameters, add custom metrics to your Temporal SDK code, and use these observability features to help scale your Temporal deployment. Refer to the [Cluster metrics](https://docs.temporal.io/references/cluster-metrics) and [SDK metrics](https://docs.temporal.io/references/sdk-metrics) reference for more. Configuring Temporal Service health checks[​](https://docs.temporal.io/self-hosted-guide/monitoring#health-checks "Direct link to Configuring Temporal Service health checks") ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- The [Frontend Service](https://docs.temporal.io/temporal-service/temporal-server#frontend-service) supports TCP or [gRPC](https://github.com/grpc/grpc/blob/875066b61e3b57af4bb1d6e36aabe95a4f6ba4f7/src/proto/grpc/health/v1/health.proto#L45) health checks on port 7233. If you use [Nomad](https://www.nomadproject.io/) to manage your containers, the [check stanza](https://developer.hashicorp.com/nomad/docs/job-specification/check) would look like this for TCP: service { check { type = "tcp" port = 7233 interval = "10s" timeout = "2s" } or like this for gRPC (requires Consul ≥ `1.0.5`): service { check { type = "grpc" port = 7233 interval = "10s" timeout = "2s" } Installing via Helm Chart[​](https://docs.temporal.io/self-hosted-guide/monitoring#installing-via-helm-chart "Direct link to Installing via Helm Chart") --------------------------------------------------------------------------------------------------------------------------------------------------------- If you are installing and running Temporal via [Helm chart](https://github.com/temporalio/helm-charts) , you can also [provide additional parameters](https://github.com/temporalio/helm-charts?tab=readme-ov-file#exploring-metrics-via-grafana) to populate and explore a Grafana dashboard out of the box. Datadog[​](https://docs.temporal.io/self-hosted-guide/monitoring#datadog "Direct link to Datadog") --------------------------------------------------------------------------------------------------- Datadog has a Temporal integration for collecting Temporal Service metrics. Once you've [configured Prometheus](https://docs.temporal.io/self-hosted-guide/monitoring#prometheus) , you can configure the [Datadog Agent](https://docs.datadoghq.com/integrations/temporal/) . If you are using [Temporal Cloud](https://docs.temporal.io/cloud/overview) , you can also [integrate Datadog directly](https://docs.datadoghq.com/integrations/temporal-cloud/) , without needing to use Prometheus. * [Prometheus](https://docs.temporal.io/self-hosted-guide/monitoring#prometheus) * [SDK metrics setup](https://docs.temporal.io/self-hosted-guide/monitoring#sdk-metrics-setup) * [Verifying Prometheus configuration](https://docs.temporal.io/self-hosted-guide/monitoring#verifying-prometheus-configuration) * [Grafana](https://docs.temporal.io/self-hosted-guide/monitoring#grafana) * [Dashboard setup](https://docs.temporal.io/self-hosted-guide/monitoring#dashboard-setup) * [Configuring Temporal Service health checks](https://docs.temporal.io/self-hosted-guide/monitoring#health-checks) * [Installing via Helm Chart](https://docs.temporal.io/self-hosted-guide/monitoring#installing-via-helm-chart) * [Datadog](https://docs.temporal.io/self-hosted-guide/monitoring#datadog) --- # What is a Temporal Retry Policy? | Temporal Platform Documentation [Skip to main content](https://docs.temporal.io/encyclopedia/retry-policies#__docusaurus_skipToContent_fallback) On this page A Retry Policy is a collection of settings that tells Temporal how and when to try again after something fails in a Workflow Execution or Activity Task Execution. Overview[​](https://docs.temporal.io/encyclopedia/retry-policies#overview "Direct link to Overview") ----------------------------------------------------------------------------------------------------- Temporal's default behavior is to automatically retry an Activity that fails, so transient or intermittent failures require no action on your part. This behavior is defined by the Retry Policy. A Retry Policy is declarative. You do not need to implement your own logic for handling the retries; you only need to specify the desired behavior and Temporal will provide it. In contrast to the Activities it contains, a Workflow Execution itself is not associated with a Retry Policy by default. This may seem counterintuitive, but Workflows and Activities perform different roles. Activities are intended for operations that may fail, so having a default Retry Policy increases the likelihood that they will ultimately complete successfully, even if the initial attempt failed. On the other hand, Workflows must be deterministic and are not intended to perform failure-prone operations. While it is possible to assign a Retry Policy to a Workflow Execution, this is not the default and it is uncommon to do so. Retry Policies do not apply to Workflow Task Executions, which retry until the Workflow Execution Timeout (which is unlimited by default) with an exponential backoff and a max interval of 10 minutes. A Retry Policy instructs the Temporal Service how to retry a failure of either a [Workflow Execution](https://docs.temporal.io/workflow-execution) or an [Activity Task Execution](https://docs.temporal.io/tasks#activity-task-execution) . Try out the [Activity retry simulator](https://docs.temporal.io/develop/activity-retry-simulator) to visualize how a Retry Policy works. * * * Related 📚 * [![](https://docs.temporal.io/img/sdks/svgs/golang.svg)Set a custom Retry Policy for an Activity in Go](https://docs.temporal.io/develop/go/activities/timeouts#activity-retries) feature-guide * [![](https://docs.temporal.io/img/sdks/svgs/java.svg)Set a custom Retry Policy for an Activity in Java](https://docs.temporal.io/develop/java/activities/timeouts#activity-retries) feature-guide * [![](https://docs.temporal.io/img/sdks/svgs/php.svg)Set a custom Retry Policy for an Activity in PHP](https://docs.temporal.io/develop/php/activities/timeouts#activity-retries) feature-guide * [![](https://docs.temporal.io/img/sdks/svgs/python.svg)Set a custom Retry Policy for an Activity in Python](https://docs.temporal.io/develop/python/activities/timeouts#activity-retries) feature-guide * [![](https://docs.temporal.io/img/sdks/svgs/typescript.svg)Set a custom Retry Policy for an Activity in TypeScript](https://docs.temporal.io/develop/typescript/activities/timeouts#activity-retries) feature-guide * * * Related 📚 * [![](https://docs.temporal.io/img/sdks/svgs/golang.svg)Set a Retry Policy for a Workflow in Go](https://docs.temporal.io/develop/go/workflows/timeouts#workflow-retries) feature-guide * [![](https://docs.temporal.io/img/sdks/svgs/java.svg)Set a Retry Policy for a Workflow in Java](https://docs.temporal.io/develop/java/workflows/timeouts#workflow-retries) feature-guide * [![](https://docs.temporal.io/img/sdks/svgs/php.svg)Set a Retry Policy for a Workflow in PHP](https://docs.temporal.io/develop/php/workflows/timeouts#workflow-retries) feature-guide * [![](https://docs.temporal.io/img/sdks/svgs/python.svg)Set a Retry Policy for a Workflow in Python](https://docs.temporal.io/develop/python/workflows/timeouts#workflow-retries) feature-guide * [![](https://docs.temporal.io/img/sdks/svgs/typescript.svg)Set a Retry Policy for a Workflow in TypeScript](https://docs.temporal.io/develop/typescript/workflows/timeouts#workflow-retries) feature-guide Default behavior[​](https://docs.temporal.io/encyclopedia/retry-policies#default-behavior "Direct link to Default behavior") ----------------------------------------------------------------------------------------------------------------------------- Activities in Temporal are associated with a Retry Policy by default, while Workflows are not. The Temporal SDK provides a Retry Policy instance with default behavior. While this object is not specific to either a Workflow or Activity, you'll use different methods to apply it to the execution of each. This section details the default retry behavior for both Activities and Workflows to provide context for any further customization. ### Activity Execution[​](https://docs.temporal.io/encyclopedia/retry-policies#activity-execution "Direct link to Activity Execution") Temporal's default behavior is to automatically retry an Activity, with a short delay between each attempt that increases exponentially, until it either succeeds or is canceled. When a subsequent request succeeds, your Workflow code will resume as if the failure never occurred. When an Activity Task Execution is retried, the Temporal Service places a new [Activity Task](https://docs.temporal.io/tasks#activity-task) into its respective [Activity Task Queue](https://docs.temporal.io/task-queue) , which results in a new Activity Task Execution. The default Retry Policy uses exponential backoff with a 2.0 backoff coefficient, starting with a 1-second initial interval and capping at a maximum interval of 100 seconds. By default, the maximum attempt of retries are set to zero which is evaluated as unlimited and non-retryable errors default to none. For detailed information about all Retry Policy attributes and their default values, see the [Properties](https://docs.temporal.io/encyclopedia/retry-policies#properties) section. ### Workflow Execution[​](https://docs.temporal.io/encyclopedia/retry-policies#workflow-execution "Direct link to Workflow Execution") Unlike Activities, Workflow Executions do not retry by default. When a Workflow Execution is spawned, it is not associated with a default Retry Policy and thus does not retry by default. Temporal provides guidance around idempotence of Activity code with the expectation that Activities will need to re-execute upon failure; this is not typically true of Workflows. In most use cases, a Workflow failure would indicate an issue with the design or deployment of your application; for example, a permanent failure that may require different input data. Retrying an entire Workflow Execution is not recommended due to Temporal's deterministic design. Since Workflows replay the same sequence of events to reach the same state, retrying the whole workflow would repeat the same logic without resolving the underlying issue that caused the failure. This repetition does not address problems related to external dependencies or unchanged conditions and can lead to unnecessary resource consumption and higher costs. Instead, it's more efficient to retry only the failed Activities. This approach targets specific points of failure, allowing the workflow to progress without redundant operations, thereby saving on resources and ensuring a more focused and effective error recovery process. If you need to retry parts of your Workflow Definition, we recommend you implement this in your Workflow code. Custom Retry Policy[​](https://docs.temporal.io/encyclopedia/retry-policies#custom-retry-policy "Direct link to Custom Retry Policy") -------------------------------------------------------------------------------------------------------------------------------------- To use a custom Retry Policy, provide it as an options parameter when starting a Workflow Execution or Activity Execution. Only certain scenarios merit starting a Workflow Execution with a custom Retry Policy, such as the following: * A [Temporal Cron Job](https://docs.temporal.io/cron-job) or some other stateless, always-running Workflow Execution that can benefit from retries. * A file-processing or media-encoding Workflow Execution that downloads files to a host. Properties[​](https://docs.temporal.io/encyclopedia/retry-policies#properties "Direct link to Properties") ----------------------------------------------------------------------------------------------------------- ### Default values for Retry Policy[​](https://docs.temporal.io/encyclopedia/retry-policies#default-values-for-retry-policy "Direct link to Default values for Retry Policy") Initial Interval = 1 secondBackoff Coefficient = 2.0Maximum Interval = 100 × Initial IntervalMaximum Attempts = ∞Non-Retryable Errors = [] ### Initial Interval[​](https://docs.temporal.io/encyclopedia/retry-policies#initial-interval "Direct link to Initial Interval") * **Description:** Amount of time that must elapse before the first retry occurs. * **The default value is 1 second.** * **Use case:** This is used as the base interval time for the [Backoff Coefficient](https://docs.temporal.io/encyclopedia/retry-policies#backoff-coefficient) to multiply against. ### Backoff Coefficient[​](https://docs.temporal.io/encyclopedia/retry-policies#backoff-coefficient "Direct link to Backoff Coefficient") * **Description:** The value dictates how much the _retry interval_ increases. * **The default value is 2.0.** * A backoff coefficient of 1.0 means that the retry interval always equals the [Initial Interval](https://docs.temporal.io/encyclopedia/retry-policies#initial-interval) . * **Use case:** Use this attribute to increase the interval between retries. By having a backoff coefficient greater than 1.0, the first few retries happen relatively quickly to overcome intermittent failures, but subsequent retries happen farther and farther apart to account for longer outages. Use the [Maximum Interval](https://docs.temporal.io/encyclopedia/retry-policies#maximum-interval) attribute to prevent the coefficient from increasing the retry interval too much. ### Maximum Interval[​](https://docs.temporal.io/encyclopedia/retry-policies#maximum-interval "Direct link to Maximum Interval") * **Description:** Specifies the maximum interval between retries. * **The default value is 100 times the [Initial Interval](https://docs.temporal.io/encyclopedia/retry-policies#initial-interval) .** * **Use case:** This attribute is useful for [Backoff Coefficients](https://docs.temporal.io/encyclopedia/retry-policies#backoff-coefficient) that are greater than 1.0 because it prevents the retry interval from growing infinitely. ### Maximum Attempts[​](https://docs.temporal.io/encyclopedia/retry-policies#maximum-attempts "Direct link to Maximum Attempts") * **Description:** Specifies the maximum number of execution attempts that can be made in the presence of failures. * **The default is unlimited.** * If this limit is exceeded, the execution fails without retrying again. When this happens an error is returned. * Setting the value to 0 also means unlimited. * Setting the value to 1 means a single execution attempt and no retries. * Setting the value to a negative integer results in an error when the execution is invoked. * **Use case:** Use this attribute to ensure that retries do not continue indefinitely. In most cases, we recommend using the Workflow Execution Timeout for [Workflows](https://docs.temporal.io/workflows) or the Schedule-To-Close Timeout for Activities to limit the total duration of retries, rather than using this attribute. ### Non-Retryable Errors[​](https://docs.temporal.io/encyclopedia/retry-policies#non-retryable-errors "Direct link to Non-Retryable Errors") Non-Retryable Errors specify errors that shouldn't be retried. By default, none are specified. Errors are matched against the `type` field of the [Application Failure](https://docs.temporal.io/references/failures#application-failure) . If one of those errors occurs, a retry does not occur. If you know of errors that should not trigger a retry, you can specify that and if they occur, the execution is not retried. #### Non-Retryable Errors for Activities[​](https://docs.temporal.io/encyclopedia/retry-policies#non-retryable-errors-for-activities "Direct link to Non-Retryable Errors for Activities") When writing software applications, you will encounter three types of failures: transient, intermittent, and permanent. While transient and intermittent failures may resolve themselves upon retrying without further intervention, permanent failures will not. Permanent failures, by definition, require you to make some change to your logic or your input. Therefore, it is better to surface them than to retry them. Non-Retryable Errors are errors that will not be retried, regardless of a Retry Policy. * Ruby * Python * TypeScript * Java * Go * .NET To raise a non-retryable error, specify the `non_retryable` flag when raising an `ApplicationError`: raise Temporalio::Error::ApplicationError.new( "Invalid credit card number: #{credit_card_number}", type: 'InvalidChargeAmount', non_retryable: true) This will designate the `ApplicationError` as non-retryable. To raise a non-retryable error, specify the `non_retryable` flag when raising an `ApplicationError`: raise ApplicationError( f"Invalid credit card number: {credit_card_number}", type="InvalidChargeAmount", non_retryable=True,) This will designate the `ApplicationError` as non-retryable. To throw a non-retryable error, add `nonRetryable: true` to `ApplicationFailure.create({})`: throw ApplicationFailure.create({ message: `Invalid charge amount: ${chargeAmount} (must be above zero)`, details: [chargeAmount], nonRetryable: true,}); This will designate the Error as non-retryable. To throw a non-retryable error, use the `newNonRetryableFailure` method: throw ApplicationFailure.newNonRetryableFailure( "Invalid credit card number: " + creditCardNumber, InvalidChargeAmountException.class.getName()); This will designate the `ApplicationFailure` as non-retryable. To return a non-retryable error, replace your call to `NewApplicationError()` with `NewNonRetryableApplicationError()`: temporal.NewNonRetryableApplicationError("Credit Card Charge Error", "CreditCardError", nil, nil) This will designate the Error as non-retryable. To throw a non-retryable error, specify the `nonRetryable` flag when throwing an `ApplicationFailureException`: var attempt = ActivityExecutionContext.Current.Info.Attempt;throw new ApplicationFailureException( $"Something bad happened on attempt {attempt}", errorType: "my_failure_type", nonRetryable: true); This will designate the `ApplicationFailureException` as non-retryable. Use non-retryable errors in your code sparingly. * Ruby * Python * TypeScript * Java * Go * .NET If you do not specify the failure as non-retryable within the definition, you can always mark that error type as non-retryable in your Activity's Retry Policy, but an `ApplicationError` with the `non_retryable` keyword argument set to `true` will always be non-retryable. If you do not specify the failure as non-retryable within the definition, you can always mark that error type as non-retryable in your Activity's Retry Policy, but an `ApplicationError` with the `non_retryable` keyword argument set to `True` will always be non-retryable. If you do not specify the failure as non-retryable within the definition, you can always mark that error type as non-retryable in your Activity's Retry Policy, but an error with `nonRetryable: true` set will always be non-retryable. If you throw a regular `newFailure()`, you can always mark that error _type_ as non-retryable in your Activity's Retry Policy, but a `newNonRetryableFailure()` will always be non-retryable. If you return a regular `NewApplicationError()`, you can always mark that error _type_ as non-retryable in your Activity's Retry Policy, but a `NewNonRetryableApplicationError()` will always be non-retryable. If you do not specify the failure as non-retryable within the definition, you can always mark that error type as non-retryable in your Activity's Retry Policy, but an `ApplicationFailureException` with the `nonRetryable` parameter set to `true` will always be non-retryable. For example, checking for bad input data is a reasonable time to use a non-retryable error. If the Activity cannot proceed with the input it has, that error should be surfaced immediately so that the input can be corrected on the next attempt. If responsibility for your application is distributed across multiple maintainers, or if you are developing a library to integrate into somebody else's application, you can think of the decision to hardcode non-retryable errors as following a "caller vs. implementer" dichotomy. Anyone who is calling your Activity would be able to make decisions about their Retry Policy, but only the implementer can decide whether an error should never be retryable out of the box. Retry interval[​](https://docs.temporal.io/encyclopedia/retry-policies#retry-interval "Direct link to Retry interval") ----------------------------------------------------------------------------------------------------------------------- The wait time before a retry is the _retry interval_. A retry interval is the smaller of two values: * The [Initial Interval](https://docs.temporal.io/encyclopedia/retry-policies#initial-interval) multiplied by the [Backoff Coefficient](https://docs.temporal.io/encyclopedia/retry-policies#backoff-coefficient) raised to the power of the number of retries. * The [Maximum Interval](https://docs.temporal.io/encyclopedia/retry-policies#maximum-interval) . ![Diagram that shows the retry interval and its formula](https://docs.temporal.io/img/info/retry-interval-diagram.png) Diagram that shows the retry interval and its formula ### Per-error next Retry delay[​](https://docs.temporal.io/encyclopedia/retry-policies#per-error-next-retry-delay "Direct link to Per-error next Retry delay") Sometimes, your Activity or Workflow raises a special exception that needs a different retry interval from the Retry Policy. To accomplish this, you may throw an [Application Failure](https://docs.temporal.io/references/failures#application-failure) with the next Retry delay field set. This value will replace and override whatever the retry interval would be on the Retry Policy. Note that your retries will still cap out under the Retry Policy's Maximum Attempts, as well as overall timeouts. For an Activity, its Schedule-to-Close Timeout applies. For a Workflow, the Execution Timeout applies. Related 📚 * [![](https://docs.temporal.io/img/sdks/svgs/java.svg)Customize retry delays per error in the Java SDK.](https://docs.temporal.io/develop/java/activities/timeouts#activity-next-retry-delay) feature-guide * [![](https://docs.temporal.io/img/sdks/svgs/typescript.svg)Customize retry delays per error in the TypeScript SDK](https://docs.temporal.io/develop/typescript/activities/timeouts#activity-next-retry-delay) feature-guide Event History[​](https://docs.temporal.io/encyclopedia/retry-policies#event-history "Direct link to Event History") -------------------------------------------------------------------------------------------------------------------- There are some subtle nuances to how Events are recorded to an Event History when a Retry Policy comes into play. * For an Activity Execution, the [ActivityTaskStarted](https://docs.temporal.io/references/events#activitytaskstarted) Event will not show up in the Workflow Execution Event History until the Activity Execution has completed or failed (having exhausted all retries). This is to avoid filling the Event History with noise. Use the Describe API to get a pending Activity Execution's attempt count. * When a Workflow fails and has a Retry Policy, the failed run ends with `WorkflowExecutionFailed`, with `retryState=IN_PROGRESS` and `newExecutionRunId` set, and the Temporal Service starts a new Workflow Execution. The new Workflow Execution is created immediately, but the first Workflow Task won’t be scheduled until the backoff duration is exhausted. That duration is recorded as the `first_workflow_task_backoff` field on the new run’s `WorkflowExecutionStartedEventAttributes`. * [Overview](https://docs.temporal.io/encyclopedia/retry-policies#overview) * [Default behavior](https://docs.temporal.io/encyclopedia/retry-policies#default-behavior) * [Activity Execution](https://docs.temporal.io/encyclopedia/retry-policies#activity-execution) * [Workflow Execution](https://docs.temporal.io/encyclopedia/retry-policies#workflow-execution) * [Custom Retry Policy](https://docs.temporal.io/encyclopedia/retry-policies#custom-retry-policy) * [Properties](https://docs.temporal.io/encyclopedia/retry-policies#properties) * [Default values for Retry Policy](https://docs.temporal.io/encyclopedia/retry-policies#default-values-for-retry-policy) * [Initial Interval](https://docs.temporal.io/encyclopedia/retry-policies#initial-interval) * [Backoff Coefficient](https://docs.temporal.io/encyclopedia/retry-policies#backoff-coefficient) * [Maximum Interval](https://docs.temporal.io/encyclopedia/retry-policies#maximum-interval) * [Maximum Attempts](https://docs.temporal.io/encyclopedia/retry-policies#maximum-attempts) * [Non-Retryable Errors](https://docs.temporal.io/encyclopedia/retry-policies#non-retryable-errors) * [Non-Retryable Errors for Activities](https://docs.temporal.io/encyclopedia/retry-policies#non-retryable-errors-for-activities) * [Retry interval](https://docs.temporal.io/encyclopedia/retry-policies#retry-interval) * [Per-error next Retry delay](https://docs.temporal.io/encyclopedia/retry-policies#per-error-next-retry-delay) * [Event History](https://docs.temporal.io/encyclopedia/retry-policies#event-history) --- # Sticky Execution | Temporal Platform Documentation [Skip to main content](https://docs.temporal.io/sticky-execution#__docusaurus_skipToContent_fallback) On this page This page discusses [Sticky Execution](https://docs.temporal.io/sticky-execution#sticky-execution) . What is a Sticky Execution?[​](https://docs.temporal.io/sticky-execution#sticky-execution "Direct link to What is a Sticky Execution?") ---------------------------------------------------------------------------------------------------------------------------------------- Workers cache the state of the Workflow they execute. To make this caching more effective, Temporal employs a performance optimization known as "Sticky Execution", which directs Workflow Tasks to the same Worker that previously processed tasks for a specific Workflow Execution. ### How Sticky Execution Works[​](https://docs.temporal.io/sticky-execution#how-sticky-execution-works "Direct link to How Sticky Execution Works") Once Workflow Execution begins, the Temporal Service schedules a Workflow Task and puts it into a Task Queue with the name you specify. Any Worker that polls that Task Queue is eligible to accept the Task and begin executing the Workflow. The Worker that picks up this Workflow Task will continue polling the original Task Queue, but will also begin polling an additional Task Queue, which the Temporal Service shares exclusively with that specific Worker. This queue, which has an automatically-generated name, is known as a **Sticky Queue**. The Worker caches the Workflow state in memory, which improves performance by reducing the need to reconstruct the Workflow from its Event History for every Task. As the Workflow Execution progresses, the Temporal Service schedules additional Workflow Tasks into this Worker-specific Sticky Queue. If the Worker fails to start a Workflow Task in the Sticky Queue shortly after it's scheduled (within five seconds by default), the Temporal Service disables stickiness for that Workflow Execution. When stickiness is disabled, the Temporal Service reschedules the Workflow Task in the original queue, allowing any Worker to pick it up and continue the Workflow Execution. If a Workflow Task fails, the Worker removes that Workflow Execution from its cache (as it's now in an unknown state), which invalidates the Sticky Execution. The Workflow Task is then put back into the original Task Queue. ### Why Sticky Execution?[​](https://docs.temporal.io/sticky-execution#why-sticky-execution "Direct link to Why Sticky Execution?") The main benefit of Sticky Execution is improved performance. By caching the Workflow state in memory and directing tasks to the same Worker, it reduces the need to reconstruct the Workflow from its Event History for every Task, which is particularly useful for latency-sensitive Workflows. Sticky Execution is the default behavior of the Temporal Platform and only applies to Workflow Tasks. Since Event History is associated with a Workflow, the concept of Sticky Execution is not relevant to Activity Tasks. * [How to set a `StickyScheduleToStartTimeout` on a individual Worker in Go](https://docs.temporal.io/develop/go/workers/run-worker-process#stickyscheduletostarttimeout) Sticky Executions are the default behavior of the Temporal Platform. * [What is a Sticky Execution?](https://docs.temporal.io/sticky-execution#sticky-execution) * [How Sticky Execution Works](https://docs.temporal.io/sticky-execution#how-sticky-execution-works) * [Why Sticky Execution?](https://docs.temporal.io/sticky-execution#why-sticky-execution) --- # Patching | Temporal Platform Documentation [Skip to main content](https://docs.temporal.io/patching#__docusaurus_skipToContent_fallback) On this page This page discusses [Patching](https://docs.temporal.io/patching#patching) . What is Patching?[​](https://docs.temporal.io/patching#patching "Direct link to What is Patching?") ---------------------------------------------------------------------------------------------------- A Patch defines a logical branch in a Workflow for a specific change, similar to a feature flag. It applies a code change to new Workflow Executions while avoiding disruptive changes to in-progress Workflow Executions. When you want to make substantive code changes that may affect existing Workflow executions, create a patch. Note that there's no need to patch [Pinned Workflows](https://docs.temporal.io/worker-versioning) . ### Detailed Description of the `patched()` Function[​](https://docs.temporal.io/patching#detailed-description-of-the-patched-function "Direct link to detailed-description-of-the-patched-function") This applies to the `patched()` function in the Python, .NET, and Ruby SDKs. #### Behavior When Not Replaying[​](https://docs.temporal.io/patching#behavior-when-not-replaying "Direct link to Behavior When Not Replaying") If the execution is not replaying, when it encounters a call to `patched()`, it first checks the event history. * If the patch ID is not in the event history, the execution adds a marker to the event history, upserts a search attribute, and returns `true`. This happens in the first block of the patch ID. * If the patch ID is in the event history, the execution doesn't modify the history, and returns `true`. This happens in a patch ID's subsequent blocks, because the event history was updated in the first block. There is a caveat to this behavior, which we will cover below. #### Behavior When Replaying With Marker Before-Or-At Current Location[​](https://docs.temporal.io/patching#behavior-when-replaying-with-marker-before-or-at-current-location "Direct link to Behavior When Replaying With Marker Before-Or-At Current Location") If the execution is replaying and has a call to `patched()`, and if the event history has a marker from a call to `patched()` in the same place (which means it will match the original event history), then it writes a marker to the replay event history and returns `true`. This is similar to the behavior of the non-replay case, and also happens in a given patch ID's first block. If the code has a call to `patched()`, and the event history has a marker with that Patch ID earlier in the history, it will return `true` and will not modify the replay event history. This is also similar to the behavior of the non-replay case, and also happens in a given patch ID's subsequent blocks. #### Behavior When Replaying With Marker After Current Location[​](https://docs.temporal.io/patching#behavior-when-replaying-with-marker-after-current-location "Direct link to Behavior When Replaying With Marker After Current Location") If the Event History's Marker Event is after the current execution point, that means the new patch is too early. The execution will encounter the new patch before the original. The execution will attempt to write the marker to the replay event history, but it will throw a non-deterministic exception because the replay and original event histories don't match. #### Behavior When Replaying With No Marker For that Patch ID[​](https://docs.temporal.io/patching#behavior-when-replaying-with-no-marker-for-that-patch-id "Direct link to Behavior When Replaying With No Marker For that Patch ID") During a Replay, if there is no marker for a given patch ID, the execution will return `false` and will not add a marker to the event history. In addition, all future calls to `patched()` with that ID will return `false` -- even after it is done replaying and is running new code. The [preceding section](https://docs.temporal.io/patching#behavior-when-not-replaying) states that if the execution is not replaying, the `patched()` function will always return `true`. If the marker doesn't exist, it will be added, and if the marker already exists, it won't be re-added. However, this behavior doesn't occur if there was already a call to `patched()` with that ID in the replay code, but not in the event history. In this situation, the function won't return `true`. #### A Summary of the Two Potentially Unexpected Behaviors[​](https://docs.temporal.io/patching#a-summary-of-the-two-potentially-unexpected-behaviors "Direct link to A Summary of the Two Potentially Unexpected Behaviors") Recapping the potentially unexpected behaviors that may occur during a Replay: If the execution hits a call to `patched()`, but that patch ID isn't _at or before that point_ in the event history, you may not realize that the event history _after_ the current execution location matters. This behavior occurs because: * If that patch ID exists later, you get a non-determinism error * If the patch doesn't exist later, you don't get a non-determinism error, and the call returns `false` If the execution hits a call to `patched()` with an ID that doesn't exist in the history, then not only will it return `false` in that occurence, but it will also return `false` if the execution surpasses the Replay threshold and is running new code. #### Implications of the Behaviors[​](https://docs.temporal.io/patching#implications-of-the-behaviors "Direct link to Implications of the Behaviors") If you deploy new code while Workflows are executing, any Workflows that were in the middle of executing will Replay up to the point they were at when the Worker was shut down. When they do this Replay, they will not follow the `patched()` branches in the code. For the rest of the execution after they have replayed to the point before the deployment and worker restart, they will either: * Use new code if there was no call to `patched()` in the replay code * If there was a call to `patched()` in the replay code, they will run the non-patched code during and after replay This might sound odd, but it's actually exactly what's needed because that means that if the future patched code depends on earlier patched code, then it won't use the new code -- it will use the old code instead. But if there's new code in the future, and there was no code earlier in the body that required the new patch, then it can switch over to the new code, which it will do. Note that this behavior means that the Workflow _does not always run the newest code_. It only does that if not replaying or if replay is surpassed and there hasn't been a call to `patched()` (with that ID) throughout the replay. #### Recommendations[​](https://docs.temporal.io/patching#recommendations "Direct link to Recommendations") Based on this behavior and the implications, when patching in new code, always put the newest code at the top of an if-patched-block. if patched('v3'): # This is the newest version of the code. # put this at the top, so when it is running # a fresh execution and not replaying, # this patched statement will return true # and it will run the new code. passelif patched('v2'): passelse: pass The following sample shows how `patched()` will behave in a conditional block that's arranged differently. In this case, the code's conditional block doesn't have the newest code at the top. Because `patched()` will return `True` when not Replaying (except with the preceding caveats), this snippet will run the `v2` branch instead of `v3` in new executions. if patched('v2'): # This is bad because when doing a new execution (i.e. not replaying), # patched statements evaluate to True (and put a marker # in the event history), which means that new executions # will use v2, and miss v3 below passelif patched('v3'): passelse: pass * [What is Patching?](https://docs.temporal.io/patching#patching) * [Detailed Description of the `patched()` Function](https://docs.temporal.io/patching#detailed-description-of-the-patched-function) * [Behavior When Not Replaying](https://docs.temporal.io/patching#behavior-when-not-replaying) * [Behavior When Replaying With Marker Before-Or-At Current Location](https://docs.temporal.io/patching#behavior-when-replaying-with-marker-before-or-at-current-location) * [Behavior When Replaying With Marker After Current Location](https://docs.temporal.io/patching#behavior-when-replaying-with-marker-after-current-location) * [Behavior When Replaying With No Marker For that Patch ID](https://docs.temporal.io/patching#behavior-when-replaying-with-no-marker-for-that-patch-id) * [A Summary of the Two Potentially Unexpected Behaviors](https://docs.temporal.io/patching#a-summary-of-the-two-potentially-unexpected-behaviors) * [Implications of the Behaviors](https://docs.temporal.io/patching#implications-of-the-behaviors) * [Recommendations](https://docs.temporal.io/patching#recommendations) --- # Temporal Workflow message passing - Signals, Queries, & Updates | Temporal Platform Documentation [Skip to main content](https://docs.temporal.io/encyclopedia/workflow-message-passing#__docusaurus_skipToContent_fallback) On this page Workflows can be thought of as stateful web services that can receive messages. The Workflow can have powerful message handlers akin to endpoints that react to the incoming messages in combination with the current state of the Workflow. Temporal supports three types of messages: Signals, Queries, and Updates: * Queries are read requests. They can read the current state of the Workflow but cannot block in doing so. * Signals are asynchronous write requests. They cause changes in the running Workflow, but you cannot await any response or error. * Updates are synchronous, tracked write requests. The sender of the Update can wait for a response on completion or an error on failure. How to choose between Signals, Updates, and Queries as a Workflow author?[​](https://docs.temporal.io/encyclopedia/workflow-message-passing#choosing-messages "Direct link to How to choose between Signals, Updates, and Queries as a Workflow author?") ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- This section will help you write Workflows that receive messages. ### For write requests[​](https://docs.temporal.io/encyclopedia/workflow-message-passing#for-write-requests "Direct link to For write requests") Unlike Signals, Updates must be synchronous and must wait for the Worker running the Workflow to acknowledge the request. The following table compares when to use **Signals** versus **Updates**. | **Requirement type** | **Use Signals when...** | **Use Updates when...** | | --- | --- | --- | | **Asynchronous communication** | Clients want to quickly move on after sending an asynchronous message. | Clients want to track the completion of the message. | | **Result handling** | Clients are okay with “fire and forget” — no result or exception needed. | Clients need a result or exception without performing a query. | | **Worker availability** | Clients don't depend on the Worker being available. | You want to validate the Update before accepting it into the Workflow and its history. | | **Concurrency and throughput** | You don’t want to limit the number of messages processed concurrently by a single Workflow. | You don’t need more concurrent Updates per Workflow than the allowed limits for [Cloud](https://docs.temporal.io/cloud/limits#per-workflow-execution-update-limits)
or [Self-Hosted](https://docs.temporal.io/self-hosted-guide/defaults)
. | | **Latency sensitivity** | Since clients don’t expect a result, latency is often not relevant when using Signals. | Clients want a low-latency end-to-end operation and are willing to wait for completion or validation. | ### For read requests[​](https://docs.temporal.io/encyclopedia/workflow-message-passing#for-read-requests "Direct link to For read requests") You normally want to do a Query, because: * Queries are efficient–they never add entries to the [Workflow Event History](https://docs.temporal.io/workflow-execution/event#event-history) , whereas an Update would (if accepted). * Queries can operate on completed Workflows. However, because Queries cannot block, sometimes Updates are best. When your goal is to do a read once the Workflow achieves a certain desired state, you have two options: * You could poll periodically with Queries until the Workflow is ready. * You could write your read operation as an Update, which will give you better efficiency and latency, though it will write an entry to the [Workflow Event History](https://docs.temporal.io/workflow-execution/event#event-history) . ### For read/write requests[​](https://docs.temporal.io/encyclopedia/workflow-message-passing#for-readwrite-requests "Direct link to For read/write requests") Use an Update for synchronous read/write requests. If your request must be asynchronous, consider sending a Signal followed by polling with a Query. * [How to choose between Signals, Updates, and Queries as a Workflow author?](https://docs.temporal.io/encyclopedia/workflow-message-passing#choosing-messages) * [For write requests](https://docs.temporal.io/encyclopedia/workflow-message-passing#for-write-requests) * [For read requests](https://docs.temporal.io/encyclopedia/workflow-message-passing#for-read-requests) * [For read/write requests](https://docs.temporal.io/encyclopedia/workflow-message-passing#for-readwrite-requests) --- # Sending Signals, Queries, & Updates | Temporal Platform Documentation [Skip to main content](https://docs.temporal.io/sending-messages#__docusaurus_skipToContent_fallback) On this page This section will help you write clients that send messages to Workflows which includes: * [Sending Signals](https://docs.temporal.io/sending-messages#sending-signals) * [Sending Updates](https://docs.temporal.io/sending-messages#sending-updates) * [Sending Queries](https://docs.temporal.io/sending-messages#sending-queries) ### Sending Signals[​](https://docs.temporal.io/sending-messages#sending-signals "Direct link to Sending Signals") You can send Signals from any Temporal Client, the Temporal CLI, or you can Signal one Workflow to another. You can also Signal-With-Start to lazily initialize a Workflow while sending a Signal. #### Send a Signal from a Temporal Client or the CLI[​](https://docs.temporal.io/sending-messages#send-a-signal-from-a-temporal-client-or-the-cli "Direct link to Send a Signal from a Temporal Client or the CLI") Related 📚 * [![](https://docs.temporal.io/img/assets/terminal.svg)Send a Signal using the Temporal CLI](https://docs.temporal.io/cli/workflow#signal) feature-guide * [![](https://docs.temporal.io/img/sdks/svgs/golang.svg)Send Signals with the Go SDK](https://docs.temporal.io/develop/go/workflows/message-passing#send-signal-from-client) feature-guide * [![](https://docs.temporal.io/img/sdks/svgs/java.svg)Send Signals with the Java SDK](https://docs.temporal.io/develop/java/workflows/message-passing#send-signal-from-client) feature-guide * [![](https://docs.temporal.io/img/sdks/svgs/php.svg)Send Signals with the PHP SDK](https://docs.temporal.io/develop/php/workflows/message-passing#send-signal-from-client) feature-guide * [![](https://docs.temporal.io/img/sdks/svgs/python.svg)Send Signals with the Python SDK](https://docs.temporal.io/develop/python/workflows/message-passing#send-signal-from-client) feature-guide * [![](https://docs.temporal.io/img/sdks/svgs/typescript.svg)Send Signals with the TypeScript SDK](https://docs.temporal.io/develop/typescript/workflows/message-passing#send-signal-from-client) feature-guide * [![](https://docs.temporal.io/img/sdks/svgs/dotnet.svg)Send Signals with the .NET SDK](https://docs.temporal.io/develop/dotnet/workflows/message-passing#send-signal-from-client) feature-guide #### Send a Signal from one Workflow to another[​](https://docs.temporal.io/sending-messages#send-a-signal-from-one-workflow-to-another "Direct link to Send a Signal from one Workflow to another") Related 📚 * [![](https://docs.temporal.io/img/sdks/svgs/golang.svg)Send Signals from Workflows with the Go SDK](https://docs.temporal.io/develop/go/workflows/message-passing#send-signal-from-workflow) feature-guide * [![](https://docs.temporal.io/img/sdks/svgs/java.svg)Send Signals from Workflows with the Java SDK](https://docs.temporal.io/develop/java/workflows/message-passing#send-signal-from-workflow) feature-guide * [![](https://docs.temporal.io/img/sdks/svgs/php.svg)Send Signals from Workflows with the PHP SDK](https://docs.temporal.io/develop/php/workflows/message-passing#send-signal-from-workflow) feature-guide * [![](https://docs.temporal.io/img/sdks/svgs/python.svg)Send Signals from Workflows with the Python SDK](https://docs.temporal.io/develop/python/workflows/message-passing#send-signal-from-workflow) feature-guide * [![](https://docs.temporal.io/img/sdks/svgs/typescript.svg)Send Signals from Workflows with the TypeScript SDK](https://docs.temporal.io/develop/typescript/workflows/message-passing#send-signal-from-workflow) feature-guide * [![](https://docs.temporal.io/img/sdks/svgs/dotnet.svg)Send Signals from Workflows with the .NET SDK](https://docs.temporal.io/develop/dotnet/workflows/message-passing#send-signal-from-workflow) feature-guide #### Signal-With-Start[​](https://docs.temporal.io/sending-messages#signal-with-start "Direct link to Signal-With-Start") Signal-With-Start is a great tool for lazily initializing Workflows. When you send this operation, if there is a running Workflow Execution with the given Workflow Id, it will be Signaled. Otherwise, a new Workflow Execution starts and is immediately sent the Signal. Related 📚 * [![](https://docs.temporal.io/img/sdks/svgs/golang.svg)Signal-With-Start using the Go SDK](https://docs.temporal.io/develop/go/workflows/message-passing#signal-with-start) feature-guide * [![](https://docs.temporal.io/img/sdks/svgs/java.svg)Signal-With-Start using the Java SDK](https://docs.temporal.io/develop/java/workflows/message-passing#signal-with-start) feature-guide * [![](https://docs.temporal.io/img/sdks/svgs/php.svg)Signal-With-Start using the PHP SDK](https://docs.temporal.io/develop/php/workflows/message-passing#signal-with-start) feature-guide * [![](https://docs.temporal.io/img/sdks/svgs/python.svg)Signal-With-Start using the Python SDK](https://docs.temporal.io/develop/python/workflows/message-passing#signal-with-start) feature-guide * [![](https://docs.temporal.io/img/sdks/svgs/typescript.svg)Signal-With-Start using the TypeScript SDK](https://docs.temporal.io/develop/typescript/workflows/message-passing#signal-with-start) feature-guide * [![](https://docs.temporal.io/img/sdks/svgs/dotnet.svg)Signal-With-Start using the .NET SDK](https://docs.temporal.io/develop/dotnet/workflows/message-passing#signal-with-start) feature-guide ### Sending Updates[​](https://docs.temporal.io/sending-messages#sending-updates "Direct link to Sending Updates") note To use the Workflow Update feature in versions prior to v1.25.0, it must be manually enabled. Set the [frontend.enableUpdateWorkflowExecution](https://github.com/temporalio/temporal/blob/main/common/dynamicconfig/constants.go) and [frontend.enableUpdateWorkflowExecutionAsyncAccepted](https://github.com/temporalio/temporal/blob/main/common/dynamicconfig/constants.go) dynamic config values to `true`. For example, with the Temporal CLI, run these commands: temporal server start-dev --dynamic-config-value frontend.enableUpdateWorkflowExecution=truetemporal server start-dev --dynamic-config-value frontend.enableUpdateWorkflowExecutionAsyncAccepted=true Updates can be sent from a Temporal Client or the Temporal CLI to a Workflow Execution. This call is synchronous and will call into the corresponding Update handler. If you’d rather make an asynchronous request, you should use Signals. In most languages (except Go), you may call `executeUpdate` to complete an Update and get its result. Alternatively, to start an Update, you may call `startUpdate` and pass in the Workflow Update Stage as an argument. You have two choices on what to await: * Accepted - wait until the Worker is contacted, which ensures that the Update is persisted. See [Update Validators](https://docs.temporal.io/handling-messages#update-validators) for more information. * Completed - wait until the handler finishes and returns a result. (This is equivalent to `executeUpdate`.) The start call will give you a handle you can use to track the Update, determine whether it was Accepted, and ultimately get its result or an error. If you want to send an Update to another Workflow such as a Child Workflow from within a Workflow, you should do so within an Activity and use the Temporal Client as normal. There are limits on the total number of Updates that may occur during a Workflow Execution run, and also on the number of concurrent in-progress Updates that a Workflow Execution may have. Use [Update Validators](https://docs.temporal.io/handling-messages#update-validators) and [Update IDs](https://docs.temporal.io/handling-messages#exactly-once-message-processing) to stay within the system limits in both [Cloud](https://docs.temporal.io/cloud/limits#per-workflow-execution-update-limits) and [Self-Hosted](https://docs.temporal.io/self-hosted-guide/defaults) . Related 📚 * [![](https://docs.temporal.io/img/sdks/svgs/golang.svg)Send Updates in Go](https://docs.temporal.io/develop/go/workflows/message-passing#send-update-from-client) feature-guide * [![](https://docs.temporal.io/img/sdks/svgs/java.svg)Send Updates in Java](https://docs.temporal.io/develop/java/workflows/message-passing#send-update-from-client) feature-guide * [![](https://docs.temporal.io/img/sdks/svgs/php.svg)Send Updates in PHP](https://docs.temporal.io/develop/php/workflows/message-passing#send-update-from-client) feature-guide * [![](https://docs.temporal.io/img/sdks/svgs/python.svg)Send Updates in Python](https://docs.temporal.io/develop/python/workflows/message-passing#send-update-from-client) feature-guide * [![](https://docs.temporal.io/img/sdks/svgs/typescript.svg)Send Updates in TypeScript](https://docs.temporal.io/develop/typescript/workflows/message-passing#send-update-from-client) feature-guide * [![](https://docs.temporal.io/img/sdks/svgs/dotnet.svg)Send Updates in .NET](https://docs.temporal.io/develop/dotnet/workflows/message-passing#send-update-from-client) feature-guide #### Update-With-Start[​](https://docs.temporal.io/sending-messages#update-with-start "Direct link to Update-With-Start") tip For open source server users, Temporal Server version [Temporal Server version 1.28](https://github.com/temporalio/temporal/releases/tag/v1.28.0) is recommended. Update-with-Start sends an Update request, starting a Workflow if necessary. A [`WorkflowIDConflictPolicy`](https://docs.temporal.io/workflow-execution/workflowid-runid#workflow-id-conflict-policy) must be specified. Workflow ID and Update ID can be used as idempotency keys as follows: * If the Workflow exists and you provided an Update ID, and the Update exists in the latest Workflow Run, then Update-With-Start attaches to the existing Update (regardless of `WorkflowIDConflictPolicy`) * If the Workflow is closed, it attaches only if the Update has completed. * Otherwise it uses [`WorkflowIDConflictPolicy`](https://docs.temporal.io/workflow-execution/workflowid-runid#workflow-id-conflict-policy) and [`WorkflowIDReusePolicy`](https://docs.temporal.io/workflow-execution/workflowid-runid#workflow-id-reuse-policy) as usual to determine whether to start a Workflow, and then starts a new Update immediately. Update-With-Start is great for latency-sensitive use cases: * **Lazy Initialization** - Instead of making separate Start Workflow and Update Workflow calls, Update-With-Start allows you to send them together in a single roundtrip. For example, a shopping cart can be modeled using Update-With-Start. Updates let you add and remove items from the cart. Update-With-Start lets the customer start shopping, whether the cart already exists or they've just started shopping. It ensures the cart, modeled by a Workflow Execution, exists before applying any Update that changes the state of items within the cart. Set your `WorkflowIDConflictPolicy` to `USE_EXISTING` for this pattern. * **Early Return** - Using Update-With-Start you can begin a new Workflow Execution and synchronously receive a response, while the Workflow Execution continues to run to completion. For example, you might model a payment process using Update-With-Start. This allows you to send the payment validation results back to the client synchronously, while the transaction Workflow continues in the background. Set your `WorkflowIDConflictPolicy` to `FAIL` and use a unique Update ID for this pattern if you want to assert it does not reuse an existing Workflow. caution Unlike Signal-with-Start - Update-With-Start is _not_ atomic. If the Update can't be delivered, for example, because there's no running Worker available, a new Workflow Execution will still start. The SDKs will retry the Update-With-Start request, but there is no guarantee that the Update will succeed. Related 📚 * [![](https://docs.temporal.io/img/sdks/svgs/golang.svg)Update-With-Start with the Go SDK](https://docs.temporal.io/develop/go/workflows/message-passing#update-with-start) feature-guide * [![](https://docs.temporal.io/img/sdks/svgs/java.svg)Update-With-Start with the Java SDK](https://docs.temporal.io/develop/java/workflows/message-passing#update-with-start) feature-guide * [![](https://docs.temporal.io/img/sdks/svgs/php.svg)Update-With-Start with the PHP SDK](https://docs.temporal.io/develop/php/workflows/message-passing#update-with-start) feature-guide * [![](https://docs.temporal.io/img/sdks/svgs/python.svg)Update-With-Start with the Python SDK](https://docs.temporal.io/develop/python/workflows/message-passing#update-with-start) feature-guide * [![](https://docs.temporal.io/img/sdks/svgs/typescript.svg)Update-With-Start with the TypeScript SDK](https://docs.temporal.io/develop/typescript/workflows/message-passing#update-with-start) feature-guide * [![](https://docs.temporal.io/img/sdks/svgs/dotnet.svg)Update-With-Start with the .NET SDK](https://docs.temporal.io/develop/dotnet/workflows/message-passing#update-with-start) feature-guide ### Sending Queries[​](https://docs.temporal.io/sending-messages#sending-queries "Direct link to Sending Queries") Queries can be sent from a Temporal Client or the Temporal CLI to a Workflow Execution--even if this Workflow has Completed. This call is synchronous and will call into the corresponding Query handler. You can also send a built-in "Stack Trace Query" for debugging. Related 📚 * [![](https://docs.temporal.io/img/assets/terminal.svg)Send a Query using the Temporal CLI](https://docs.temporal.io/cli/workflow#query) feature-guide * [![](https://docs.temporal.io/img/sdks/svgs/golang.svg)Send a Query with the Go SDK](https://docs.temporal.io/develop/go/workflows/message-passing#send-query) feature-guide * [![](https://docs.temporal.io/img/sdks/svgs/java.svg)Send a Query with the Java SDK](https://docs.temporal.io/develop/java/workflows/message-passing#send-query) feature-guide * [![](https://docs.temporal.io/img/sdks/svgs/php.svg)Send a Query with the PHP SDK](https://docs.temporal.io/develop/php/workflows/message-passing#send-query) feature-guide * [![](https://docs.temporal.io/img/sdks/svgs/python.svg)Send a Query with the Python SDK](https://docs.temporal.io/develop/python/workflows/message-passing#send-query) feature-guide * [![](https://docs.temporal.io/img/sdks/svgs/typescript.svg)Send a Query with the TypeScript SDK](https://docs.temporal.io/develop/typescript/workflows/message-passing#send-query) feature-guide * [![](https://docs.temporal.io/img/sdks/svgs/dotnet.svg)Send a Query with the .NET SDK](https://docs.temporal.io/develop/dotnet/workflows/message-passing#send-query) feature-guide #### Stack Trace Query[​](https://docs.temporal.io/sending-messages#stack-trace-query "Direct link to Stack Trace Query") In many SDKs, the Temporal Client exposes a predefined `__stack_trace` Query that returns the call stack of all the threads owned by that Workflow Execution. This is a great way to troubleshoot a Workflow Execution in production. For example, if a Workflow Execution has been stuck at a state for longer than an expected period of time, you can send a `__stack_trace` Query to return the current call stack. The `__stack_trace` Query name does not require special handling in your Workflow code. note Stack Trace Queries are available only for running Workflow Executions. * [Sending Signals](https://docs.temporal.io/sending-messages#sending-signals) * [Sending Updates](https://docs.temporal.io/sending-messages#sending-updates) * [Sending Queries](https://docs.temporal.io/sending-messages#sending-queries) --- # Rails integration - Ruby SDK | Temporal Platform Documentation [Skip to main content](https://docs.temporal.io/develop/ruby/integrations/rails-integration#__docusaurus_skipToContent_fallback) On this page Temporal Ruby SDK is a generic Ruby library that can work in any Ruby environment. However, there are some common conventions for Rails users to be aware of. See the [rails\_app sample](https://github.com/temporalio/samples-ruby/tree/main/rails_app) for an example of using Temporal from Rails. ActiveRecord[​](https://docs.temporal.io/develop/ruby/integrations/rails-integration#activerecord "Direct link to ActiveRecord") --------------------------------------------------------------------------------------------------------------------------------- For ActiveRecord, or other general/ORM models that are used for a different purpose, it is not recommended to try to reuse them as Temporal models. Eventually model purposes diverge and models for a Temporal workflows/activities should be specific to their use for clarity and compatibility reasons. Also many Ruby ORMs do many lazy things and therefore provide unclear serialization semantics. Instead, consider having models specific for Workflows/Activities and translate to/from existing models as needed. See the [ActiveModel section](https://docs.temporal.io/develop/ruby/best-practices/converters-and-encryption#active-model) on how to do this with ActiveModel objects. Lazy/Eager Loading[​](https://docs.temporal.io/develop/ruby/integrations/rails-integration#lazyeager-loading "Direct link to Lazy/Eager Loading") -------------------------------------------------------------------------------------------------------------------------------------------------- By default, Rails eagerly loads all application code on application start in production, but lazily loads it in non-production environments. Temporal Workflows by default disallow use of IO during the Workflow run. With lazy loading enabled in dev/test environments, when an Activity class is referenced in a Workflow before it has been explicitly required, it can give an error like: Cannot access File path from inside a workflow. If this is known to be safe, the code can be run in a Temporalio::Workflow::Unsafe.illegal_call_tracing_disabled block. This comes from bootsnap via zeitwerk because it is lazily loading a class/module at Workflow runtime. It is not good to lazily load code during a Workflow run because it can be side effecting. Workflows and the classes they reference should be eagerly loaded. To resolve this, either always eagerly load (e.g. `config.eager_load = true`) or explicitly require what is used by a workflow at the top of the file. Note, this only affects non-production environments. * [ActiveRecord](https://docs.temporal.io/develop/ruby/integrations/rails-integration#activerecord) * [Lazy/Eager Loading](https://docs.temporal.io/develop/ruby/integrations/rails-integration#lazyeager-loading) --- # Temporal Platform's production readiness checklist | Temporal Platform Documentation [Skip to main content](https://docs.temporal.io/self-hosted-guide/production-checklist#__docusaurus_skipToContent_fallback) On this page This page describes common challenges customers face who self-host Temporal and it shares recommendations to mitigate those issues. Temporal at its core is about durability and reliability. To ensure this durability and reliability, a Temporal Service must be deployed according to best practices. This guide provides a path to demonstrate that Temporal consumers can be confident in a Temporal Service and provides a list of key tests you as a user should perform against the service. Self-Hosting Challenge Areas[​](https://docs.temporal.io/self-hosted-guide/production-checklist#self-hosting-challenge-areas "Direct link to Self-Hosting Challenge Areas") ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Significant engineering and ongoing effort is required to resolve several potential challenges: * Scalability with spiky or growing workloads * Global hosting * Uptime, availability and reliability * Management and control plane * Latency, which must be kept low and consistent * [Security](https://docs.temporal.io/self-hosted-guide/security) * Maintenance and upgrades * Expert support to users of the service * Cost management Each of these components is an essential part of building a mission critical Temporal Service. Without demonstrated architectural durability, the value of Temporal's [Durable Execution](https://temporal.io/how-it-works) model is compromised. Scalability with Variable or Growing Workloads[​](https://docs.temporal.io/self-hosted-guide/production-checklist#scaling-and-metrics "Direct link to Scalability with Variable or Growing Workloads") ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Workloads can be highly variable, and you may experience sustained workload spikes. Temporal recommends scaling your clusters to well above the average throughput. See [Scaling Temporal: The Basics](https://temporal.io/blog/scaling-temporal-the-basics) for an introduction to the topic. Temporal server throughput is often limited by the number of [Shards](https://docs.temporal.io/temporal-service/temporal-server#history-shard) configured for the Temporal Service. A Shard is a unit within a Temporal Service by which concurrent Workflow Execution throughput can be scaled. Shard capacity, and often overall cluster throughput, is set at build time for a cluster and that cluster setting cannot be adjusted later. Adding more Shards if needed requires a cluster rebuild, and a migration to the new cluster. The requirements of your Temporal Service will vary widely based on your intended production workload. You will want to run your own proof of concept tests and watch for key metrics to understand the system health and scaling needs. **Load testing.** You can use [the Omes benchmarking tool](https://github.com/temporalio/omes/) , see how we ourselves [stress test Temporal](https://temporal.io/blog/temporal-deep-dive-stress-testing/) , or write your own. All metrics emitted by the server are [listed in Temporal's source](https://github.com/temporalio/temporal/blob/main/common/metrics/defs.go) . There are also equivalent metrics that you can configure from the client side. At a high level, you will want to track these 3 categories of metrics: * **Service metrics**: For each request made by the service handler we emit `service_requests`, `service_errors`, and `service_latency` metrics with `type`, `operation`, and `namespace` tags. This gives you basic visibility into service usage and allows you to look at request rates across services, namespaces and even operations. * **Persistence metrics**: The Server emits `persistence_requests`, `persistence_errors` and `persistence_latency` metrics for each persistence operation. These metrics include the `operation` tag such that you can get the request rates, error rates or latencies per operation. These are super useful in identifying issues caused by the database. * **Workflow Execution stats**: The Server also emits counters for when Workflow Executions are complete. These are useful in getting overall stats about Workflow Execution completions. Use `workflow_success`, `workflow_failed`, `workflow_timeout`, `workflow_terminate` and `workflow_cancel` counters for each type of Workflow Execution completion. These include the `namespace` tag. Availability[​](https://docs.temporal.io/self-hosted-guide/production-checklist#availability "Direct link to Availability") ---------------------------------------------------------------------------------------------------------------------------- A high level of availability and reliability (99.99%) is a requirement for mission critical deployments. Temporal recommends testing for this availability level while load testing. We also recommend validating this level of reliability while doing server upgrades, to ensure no loss of service availability. Temporal Clusters can be deployed in as many regions as needed to meet various requirements: * Data Residency * Latency * Security / Isolation This can multiply the effort to implement and maintain clusters. [Temporal Cloud is available in various cloud provider regions](https://docs.temporal.io/cloud/service-availability) . Management and Control Plane[​](https://docs.temporal.io/self-hosted-guide/production-checklist#management-and-control-plane "Direct link to Management and Control Plane") ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Temporal success leads to larger Temporal deployments. Needs can increase, and can go from having one or two production use cases in a single region to many use cases in many regions. Running multiple Temporal Services is complex work, as each needs its own setup, tuning, and configuration. Needing to monitor and manage all your Temporal Services in a unified way leads to operational management pain. Consider adding a layer on top of Temporal to manage multiple Temporal Services: a control plane. A control plane manages and directs data flow, deciding where data packets should be sent. A Temporal Service data plane can streamline operations and improve efficiency. Since Temporal does not ship its own open source data plane, rolling your own can be complex and take effort to add. Temporal Cloud provides exactly that support. With Temporal Cloud, all Namespaces in all regions can be managed from a single view. [Temporal Cloud](https://temporal.io/cloud) also has RBAC functionality that can delegate responsibilities for individual Namespaces. Self-hosted Temporal does not support RBAC or audit logging out of the box. Temporal Cloud provides RBAC and SSO support, audit logging, data encryption, third party penetration test validation, and SOC 2-Type II and HIPAA compliance. Maintenance and Upgrades[​](https://docs.temporal.io/self-hosted-guide/production-checklist#maintenance-and-upgrades "Direct link to Maintenance and Upgrades") ---------------------------------------------------------------------------------------------------------------------------------------------------------------- Temporal recommends keeping up-to-date and not falling behind on your server versions. Temporal Server is proactively updated, and releases as often as every two weeks. Temporal recommends [upgrading sequentially](https://docs.temporal.io/self-hosted-guide/upgrade-server) , not skipping any minor versions, although you can skip patch versions. No support is guaranteed for Temporal Server, but very old servers will be hard for even the community to support, so we encourage you to keep up to date. You must create and maintain the infrastructure to host and run your self-hosted Temporal installation, such as Kubernetes, as well as data stores for persistence. Server upgrades can negatively affect self-hosted Temporal Service availability. Temporal recommends load and availability testing during the upgrade process to understand the performance implications. Temporal Cloud updates are managed by the Temporal Cloud team; Cloud upgrades are seamless. Expert Support[​](https://docs.temporal.io/self-hosted-guide/production-checklist#expert-support "Direct link to Expert Support") ---------------------------------------------------------------------------------------------------------------------------------- Temporal recommends that customer platform teams who are building out a Temporal service gain deep experience across the lifecycle and breadth of a Temporal application. Specific activities include: * [Worker tuning](https://docs.temporal.io/develop/worker-performance) * [Worker best practices](https://docs.temporal.io/workers) * Code reviews * Design guidance * Training * Code reviews * Security reviews * [Metrics](https://docs.temporal.io/references/sdk-metrics) and monitoring * Technical onboarding [Temporal support](https://docs.temporal.io/cloud/support) provides guidance on all of the above. Cost Management[​](https://docs.temporal.io/self-hosted-guide/production-checklist#cost-management "Direct link to Cost Management") ------------------------------------------------------------------------------------------------------------------------------------- Running a mission critical, global Temporal Service can be expensive. Temporal Server is a complex system to run and scale. Temporal recommends performance testing and planning scaling as your performance requirements evolve. Following our guidance can oversize your self-hosted Temporal Server installs, but this is necessary to handle unpredictable spiky workloads. Performance testing can help you right-size your environments. Running mission critical Temporal as a Service requires multiple Temporal Clusters for high availability and global coverage. It is a good practice to have trained, experienced administrators familiar with Temporal Service architecture to maintain your Temporal servers and provide a mission critical service. Staffing, training and skill development can be significant costs to maintaining a Temporal Service. [Temporal Cloud](https://temporal.io/cloud) can be significantly less expensive to set up and scale. * [Self-Hosting Challenge Areas](https://docs.temporal.io/self-hosted-guide/production-checklist#self-hosting-challenge-areas) * [Scalability with Variable or Growing Workloads](https://docs.temporal.io/self-hosted-guide/production-checklist#scaling-and-metrics) * [Availability](https://docs.temporal.io/self-hosted-guide/production-checklist#availability) * [Management and Control Plane](https://docs.temporal.io/self-hosted-guide/production-checklist#management-and-control-plane) * [Maintenance and Upgrades](https://docs.temporal.io/self-hosted-guide/production-checklist#maintenance-and-upgrades) * [Expert Support](https://docs.temporal.io/self-hosted-guide/production-checklist#expert-support) * [Cost Management](https://docs.temporal.io/self-hosted-guide/production-checklist#cost-management) --- # Temporal Workflow | Temporal Platform Documentation [Skip to main content](https://docs.temporal.io/workflows#__docusaurus_skipToContent_fallback) On this page This guide provides a comprehensive overview of Temporal Workflows and covers the following: * [Workflow Definition](https://docs.temporal.io/workflow-definition) * [Workflow Execution](https://docs.temporal.io/workflow-execution) * [Schedules](https://docs.temporal.io/schedule) * [Dynamic Handler](https://docs.temporal.io/dynamic-handler) * [Cron Job](https://docs.temporal.io/cron-job) Intro to Workflows[​](https://docs.temporal.io/workflows#intro-to-workflows "Direct link to Intro to Workflows") ----------------------------------------------------------------------------------------------------------------- Conceptually, a workflow defines a sequence of steps. With Temporal, those steps are defined by writing code, known as a Workflow Definition, and are carried out by running that code, which results in a Workflow Execution. In day-to-day conversations, the term Workflow might refer to Workflow Type, a Workflow Definition, or a Workflow Execution. 1. A **Workflow Definition** is the code that defines your Workflow. 2. The **Workflow Type** is the name that maps to a Workflow Definition. It's an identifier that makes it possible to distinguish one type of Workflow (such as order processing) from another (such as customer onboarding). 3. A **Workflow Execution** is a running Workflow, which is created by combining a Workflow Definition with a request to execute it. You can execute a Workflow Definition any number of times, potentially providing different input each time (i.e., a Workflow Definition for order processing might process order #123 in one execution and order #567 in another execution). It is the actual instance of the Workflow Definition running in the Temporal Platform. You'll develop those Workflows by writing code in a general-purpose programming language such as Go, Java, TypeScript, or Python. The code you write is the same code that will be executed at runtime, so you can use your favorite tools and libraries to develop Temporal Workflows. Temporal Workflows are resilient. They can run—and keep running—for years, even if the underlying infrastructure fails. If the application itself crashes, Temporal will automatically recreate its pre-failure state so it can continue right where it left off. Each Workflow Execution progresses through a series of **Commands** and **Events**, which are recorded in an **Event History**. Workflows must follow deterministic constraints to ensure consistent replay behavior. * [Intro to Workflows](https://docs.temporal.io/workflows#intro-to-workflows) --- # Self-hosted Temporal Nexus | Temporal Platform Documentation [Skip to main content](https://docs.temporal.io/production-deployment/self-hosted-guide/nexus#__docusaurus_skipToContent_fallback) On this page SUPPORT, STABILITY, and DEPENDENCY INFO Temporal Nexus is now [Generally Available](https://docs.temporal.io/evaluate/development-production-features/release-stages#general-availability) . NEW TO NEXUS? This page explains how to self-host Nexus. To learn about Nexus, see the [how Nexus works page](https://docs.temporal.io/nexus) . To evaluate whether Nexus fits your use case, see the [evaluation guide](https://docs.temporal.io/evaluate/nexus) . Enable Nexus[​](https://docs.temporal.io/production-deployment/self-hosted-guide/nexus#enable-nexus "Direct link to Enable Nexus") ----------------------------------------------------------------------------------------------------------------------------------- Nexus can be configured by setting static configuration and dynamic configuration entries. note Nexus is supported in single-cluster setups only. See [Nexus Architecture](https://github.com/temporalio/temporal/blob/main/docs/architecture/nexus.md) for operational details. note Replace `$PUBLIC_URL` with a URL value that's accessible to external callers or internally within the cluster. Currently, external Nexus calls are considered experimental so it should be safe to use the address of an internal load balancer for the Frontend Service. To enable Nexus in your deployment: 1. Enable the HTTP API in the server's static configuration. services: frontend: rpc: # NOTE: keep other fields as they were httpPort: 7243clusterMetadata: # NOTE: keep other fields as they were clusterInformation: active: # NOTE: keep other fields as they were httpAddress: $PUBLIC_URL:7243 2. Set the required dynamic configuration 1. **Prior to version 1.30.X**, you must set the public callback URL and the allowed callback addresses. **NOTE**: the callback endpoint template and allowed addresses should be set when using the experimental "external" endpoint targets. component.nexusoperations.callback.endpoint.template: # The URL must be publicly accessible if the callback is meant to be called by external services. # When using Nexus for cross namespace calls, the URL's host is irrelevant as the address is resolved using # membership. The URL is a Go template that interpolates the `NamepaceName` and `NamespaceID` variables. - value: https://$PUBLIC_URL:7243/namespaces/{{.NamespaceName}}/nexus/callbackcomponent.callbacks.allowedAddresses: # Limits which callback URLs are accepted by the server. # Wildcard patterns (*) and insecure (HTTP) callbacks are intended for development only. # For production, restrict allowed hosts and set AllowInsecure to false # whenever HTTPS/TLS is supported. Allowing HTTP increases MITM and data exposure risk. - value: - Pattern: "*" # Update to restrict allowed callers, e.g. "*.example.com" AllowInsecure: true # In production, set to false and ensure traffic is HTTPS/TLS encrypted 2. **Version 1.30.X+**: Nexus is enabled by default. Only the system callback URL is needed. component.nexusoperations.useSystemCallbackURL: - value: true Build and use Nexus Services[​](https://docs.temporal.io/production-deployment/self-hosted-guide/nexus#build-and-use-nexus-services "Direct link to Build and use Nexus Services") ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- See [how Nexus works](https://docs.temporal.io/nexus) for an architectural overview, then follow an SDK guide to build your first Nexus Service. SDK GUIDES * [Go](https://docs.temporal.io/develop/go/nexus/feature-guide) | [Java](https://docs.temporal.io/develop/java/nexus) | [Python](https://docs.temporal.io/develop/python/nexus) | [TypeScript](https://docs.temporal.io/develop/typescript/nexus) | [.NET](https://docs.temporal.io/develop/dotnet/nexus) * [Enable Nexus](https://docs.temporal.io/production-deployment/self-hosted-guide/nexus#enable-nexus) * [Build and use Nexus Services](https://docs.temporal.io/production-deployment/self-hosted-guide/nexus#build-and-use-nexus-services) --- # Plugins | Temporal Platform Documentation [Skip to main content](https://docs.temporal.io/encyclopedia/plugins#__docusaurus_skipToContent_fallback) On this page A Plugin bundles multiple extensibility primitives - interceptors, context propagators, data converters, and built-in Workflow/Activity/Nexus definitions - into a single reusable package. Plugins let platform teams and library authors ship ready-made functionality that application developers can adopt with a single registration call. Common use cases: * AI Agent SDKs (e.g., OpenAI Agents, Pydantic AI) * Observability packages (tracing, logging, metrics) * Encryption or compliance middleware * Shared infrastructure integrations (messaging, payments, LLM calls) Implementing Plugins[​](https://docs.temporal.io/encyclopedia/plugins#implementing-plugins "Direct link to Implementing Plugins") ---------------------------------------------------------------------------------------------------------------------------------- See the [Plugins guide](https://docs.temporal.io/develop/plugins-guide) for how to build and use plugins across all supported SDKs. * [Implementing Plugins](https://docs.temporal.io/encyclopedia/plugins#implementing-plugins) --- # Dual Visibility | Temporal Platform Documentation [Skip to main content](https://docs.temporal.io/dual-visibility#__docusaurus_skipToContent_fallback) On this page This page discusses [Dual Visibility](https://docs.temporal.io/dual-visibility#dual-visibility) . What is Dual Visibility?[​](https://docs.temporal.io/dual-visibility#dual-visibility "Direct link to What is Dual Visibility?") -------------------------------------------------------------------------------------------------------------------------------- Dual Visibility is a feature that lets you set a secondary Visibility store in addition to a primary store in your Temporal Service. Setting up Dual Visibility is optional and can be used to [migrate your Visibility database](https://docs.temporal.io/self-hosted-guide/visibility#migrating-visibility-database) or create a backup Visibility store. For example, if you have Cassandra configured as your Visibility database, you can set up a supported SQL database as your secondary Visibility store and gradually migrate your data to the secondary store before deprecating your primary one. A Dual Visibility setup requires two Visibility store configurations: * **Primary Visibility:** The primary Visibility store where Visibility data is written to and read from by default. The primary Visibility store is set with the `visibilityStore` configuration key in your Temporal Service. * **Secondary Visibility:** A secondary storage for your Visibility data. The secondary Visibility store is set with the `secondaryVisibilityStore` configuration key in your Temporal Service. For configuration details, see [Dual Visibility setup](https://docs.temporal.io/self-hosted-guide/visibility#dual-visibility) . The following combinations are allowed in a Dual Visibility setting. | Primary | Secondary | | --- | --- | | Standard (Cassandra or SQL) | Advanced (SQL or Elasticsearch) | | Advanced (SQL) | Advanced (SQL) | | Advanced (Elasticsearch) | Advanced (Elasticsearch) | With Dual Visibility, you can read from only one Visibility store at a time, but can configure your Temporal Service to write to primary only, secondary only, or to both primary and secondary Visibility stores. When migrating from one Visibility store database to another, set up the database you want to migrate to as your secondary Visibility store. You can plan your migration using specific dynamic configuration keys that help you transition your read and write operations from the primary to the secondary Visibility store. For details on migrating your Visibility store databases, see [Dual Visibility](https://docs.temporal.io/self-hosted-guide/visibility#dual-visibility) . * [What is Dual Visibility?](https://docs.temporal.io/dual-visibility#dual-visibility) --- # Global Namespace | Temporal Platform Documentation [Skip to main content](https://docs.temporal.io/global-namespace#__docusaurus_skipToContent_fallback) On this page This page provides an overview of Global Namespace. What is a Global Namespace?[​](https://docs.temporal.io/global-namespace#global-namespace "Direct link to What is a Global Namespace?") ---------------------------------------------------------------------------------------------------------------------------------------- A Global Namespace is a [Namespace](https://docs.temporal.io/namespaces) that exists across Clusters when [Multi-Cluster Replication](https://docs.temporal.io/temporal-service/multi-cluster-replication) is set up. * [How to register a Global Namespace](https://docs.temporal.io/cli/operator#create) * [How to change the active Cluster for a Global Namespace](https://docs.temporal.io/cli/operator#update) The Global Namespace feature enables Workflow Executions to progress through another Cluster in the event of a failover. A Global Namespace may be replicated to any number of Clusters, but is active in only one Cluster at any given time. For a failover to be successful, Worker Processes must be polling for Tasks for the Global Namespace on all Clusters. A Global Namespace has a failover version. Because a failover can be triggered from any Cluster, the failover version prevents certain conflicts from occurring if a failover is mistakenly triggered simultaneously on two Clusters. Only the active Cluster dispatches [Tasks](https://docs.temporal.io/tasks#task) ; however, certain conflicts are possible. Unlike regular Namespaces, which provide at-most-once semantics for an Activity Execution, Global Namespaces can support only at-least-once semantics (see [Conflict resolution](https://docs.temporal.io/temporal-service/multi-cluster-replication#conflict-resolution) ). Worker Processes on the standby Clusters are idle until a failover occurs and their Cluster becomes active. Temporal Application API calls made to a non-active Cluster are rejected with a **NamespaceNotActiveError** which contains the name of the current active Cluster. It is the responsibility of the Temporal Application to call the Cluster that is currently active. * [What is a Global Namespace?](https://docs.temporal.io/global-namespace#global-namespace) --- # Event History walkthrough with the .NET SDK | Temporal Platform Documentation [Skip to main content](https://docs.temporal.io/encyclopedia/event-history/event-history-dotnet#__docusaurus_skipToContent_fallback) On this page In order to understand how Workflow Replay works, this page will go through the following walkthroughs: 1. [How Workflow Code Maps to Commands](https://docs.temporal.io/encyclopedia/event-history/event-history-dotnet#How-Workflow-Code-Maps-To-Commands) 2. [How Workflow Commands Map to Events](https://docs.temporal.io/encyclopedia/event-history/event-history-dotnet#How-Workflow-Commands-Map-To-Events) 3. [How History Replay Provides Durable Execution](https://docs.temporal.io/encyclopedia/event-history/event-history-dotnet#How-History-Replay-Provides-Durable-Execution) 4. [Example of a Non-Deterministic Workflow](https://docs.temporal.io/encyclopedia/event-history/event-history-dotnet#Example-of-Non-Deterministic-Workflow) How Workflow Code Maps to Commands[​](https://docs.temporal.io/encyclopedia/event-history/event-history-dotnet#How-Workflow-Code-Maps-To-Commands "Direct link to How Workflow Code Maps to Commands") ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- This walkthrough will cover how the Workflow code maps to Commands that get sent to the Temporal Service, letting the Temporal Service know what to do. Image 1 out of 11 ![Slide 1](https://learn.temporal.io/courses/temporal-102/dotnet/event-history-walkthrough/code-commands/code-commands.001.jpeg)❮❯ How Workflow Commands Map to Events[​](https://docs.temporal.io/encyclopedia/event-history/event-history-dotnet#How-Workflow-Commands-Map-To-Events "Direct link to How Workflow Commands Map to Events") ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- The Commands that are sent to the Temporal Service are then turned into Events, which build up the Event History. The Event History is a detailed log of Events that occur during the lifecycle of a Workflow Execution, such as the execution of Workflow Tasks or Activity Tasks. Event Histories are persisted to the database used by the Temporal Service, so they're durable, and will even survive a crash of the Temporal Service itself. These Events are what are used to recreate a Workflow Execution's state in the case of failure. Image 1 out of 14 ![Slide 1](https://learn.temporal.io/courses/temporal-102/dotnet/event-history-walkthrough/commands-events/commands-events.001.jpeg)❮❯ How History Replay Provides Durable Execution[​](https://docs.temporal.io/encyclopedia/event-history/event-history-dotnet#How-History-Replay-Provides-Durable-Execution "Direct link to How History Replay Provides Durable Execution") ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Now that you have seen how code maps to Commands, and how Commands map to Events, this next walkthrough will take a look at how Temporal uses Replay with the Events to provide Durable Execution and restore a Workflow Execution in the case of a failure. This code walkthrough will begin by walking through a Workflow Execution, describing how the code maps to Commands and Events. There will then be a Worker crash halfway through, explaining how Temporal uses Replay to recover the state of the Workflow Execution, ultimately resulting in a completed execution that's identical to one that had not crashed. Image 1 out of 57 ![Slide 1](https://learn.temporal.io/courses/temporal-102/dotnet/event-history-walkthrough/history-replay/history-replay.001.jpeg)❮❯ Example of a Non-Deterministic Workflow[​](https://docs.temporal.io/encyclopedia/event-history/event-history-dotnet#Example-of-Non-Deterministic-Workflow "Direct link to Example of a Non-Deterministic Workflow") -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Now that Replay has been covered, this section will explain why Workflows need to be [deterministic](https://docs.temporal.io/workflow-definition#deterministic-constraints) in order for Replay to work. A Workflow is deterministic if every execution of its Workflow Definition produces the same Commands in the same sequence given the same input. As mentioned in the [`How History Replay Provides Durable Execution`](https://docs.temporal.io/encyclopedia/event-history/event-history-dotnet#How-History-Replay-Provides-Durable-Execution) walkthrough, in the case of a failure, a Worker requests the Event History to replay it. During Replay, the Worker runs the Workflow code again to produce a set of Commands which is compared against the sequence of Commands in the Event History. When there’s a mismatch between the expected sequence of Commands the Worker expects based on the Event History and the actual sequence produced during Replay (due to non-determinism), Replay will be unable to continue. To better understand why Workflows need to be deterministic, it's helpful to look at a Workflow Definition that violates it. In this case, this code will walk through a Workflow Definition that breaks the determinism constraint with a random number generator. Image 1 out of 12 ![Slide 1](https://learn.temporal.io/courses/temporal-102/dotnet/event-history-walkthrough/nondeterministic-workflow/nondeterministic-workflow.014.jpeg)❮❯ Note that non-deterministic failures do not fail the Workflow Execution by default. A non-deterministic failure is considered a [Workflow Task Failure](https://docs.temporal.io/references/failures#workflow-task-failures) which is considered a transient failure, meaning it retries over and over. Users can also fix the source of non-determinism, perhaps by removing the Activity, and then restart the Workers. This means that this type of failure can recover by itself. You can also use a strategy called versioning to address this non-determinism error. See [versioning](https://docs.temporal.io/develop/dotnet/workflows/versioning) to learn more. For more information on how Temporal handles Durable Execution or to see these slides in a video format with more explanation, check out our free, self-paced courses: [Temporal 102](https://learn.temporal.io/courses/temporal_102/) and [Versioning Workflows](https://learn.temporal.io/courses/versioning/) . Temporal Applications Support Non-Deterministic Operations[​](https://docs.temporal.io/encyclopedia/event-history/event-history-dotnet#temporal-applications-support-non-deterministic-operations "Direct link to Temporal Applications Support Non-Deterministic Operations") ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- We want to emphasize that although your Workflows themselves need to be deterministic, your application itself does not! Remember that pretty much anything that interacts with the external world is inherently non-deterministic: * Calling LLM APIs * Querying databases * Reading or writing files * Making HTTP requests to external services **Good news**: Your Temporal application can absolutely handle all of these operations. While your Workflow must be deterministic, your application absolutely can handle any type of non-deterministic operation, including those listed above. This gives you the best of both worlds—the crash-proof reliability of a Workflow and the resiliency of Activities which have built-in support for retries. * [How Workflow Code Maps to Commands](https://docs.temporal.io/encyclopedia/event-history/event-history-dotnet#How-Workflow-Code-Maps-To-Commands) * [How Workflow Commands Map to Events](https://docs.temporal.io/encyclopedia/event-history/event-history-dotnet#How-Workflow-Commands-Map-To-Events) * [How History Replay Provides Durable Execution](https://docs.temporal.io/encyclopedia/event-history/event-history-dotnet#How-History-Replay-Provides-Durable-Execution) * [Example of a Non-Deterministic Workflow](https://docs.temporal.io/encyclopedia/event-history/event-history-dotnet#Example-of-Non-Deterministic-Workflow) * [Temporal Applications Support Non-Deterministic Operations](https://docs.temporal.io/encyclopedia/event-history/event-history-dotnet#temporal-applications-support-non-deterministic-operations) --- # Event History walkthrough with the Go SDK | Temporal Platform Documentation [Skip to main content](https://docs.temporal.io/encyclopedia/event-history/event-history-go#__docusaurus_skipToContent_fallback) On this page In order to understand how Workflow Replay works, this page will go through the following walkthroughs: 1. [How Workflow Code Maps to Commands](https://docs.temporal.io/encyclopedia/event-history/event-history-go#How-Workflow-Code-Maps-To-Commands) 2. [How Workflow Commands Map to Events](https://docs.temporal.io/encyclopedia/event-history/event-history-go#How-Workflow-Commands-Map-To-Events) 3. [How History Replay Provides Durable Execution](https://docs.temporal.io/encyclopedia/event-history/event-history-go#How-History-Replay-Provides-Durable-Execution) 4. [Example of a Non-Deterministic Workflow](https://docs.temporal.io/encyclopedia/event-history/event-history-go#Example-of-Non-Deterministic-Workflow) How Workflow Code Maps to Commands[​](https://docs.temporal.io/encyclopedia/event-history/event-history-go#How-Workflow-Code-Maps-To-Commands "Direct link to How Workflow Code Maps to Commands") --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- This walkthrough will cover how the Workflow code maps to Commands that get sent to the Temporal Service, letting the Temporal Service know what to do. Image 1 out of 11 ![Slide 1](https://learn.temporal.io/courses/temporal-102/go/event-history-walkthrough/code-commands/code-commands.001.jpeg)❮❯ How Workflow Commands Map to Events[​](https://docs.temporal.io/encyclopedia/event-history/event-history-go#How-Workflow-Commands-Map-To-Events "Direct link to How Workflow Commands Map to Events") ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ The Commands that are sent to the Temporal Service are then turned into Events, which build up the Event History. The Event History is a detailed log of Events that occur during the lifecycle of a Workflow Execution, such as the execution of Workflow Tasks or Activity Tasks. Event Histories are persisted to the database used by the Temporal Service, so they're durable, and will even survive a crash of the Temporal Service itself. These Events are what are used to recreate a Workflow Execution's state in the case of failure. Image 1 out of 14 ![Slide 1](https://learn.temporal.io/courses/temporal-102/go/event-history-walkthrough/commands-events/commands-events.001.jpeg)❮❯ How History Replay Provides Durable Execution[​](https://docs.temporal.io/encyclopedia/event-history/event-history-go#How-History-Replay-Provides-Durable-Execution "Direct link to How History Replay Provides Durable Execution") ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ Now that you have seen how code maps to Commands, and how Commands map to Events, this next walkthrough will take a look at how Temporal uses Replay with the Events to provide Durable Execution and restore a Workflow Execution in the case of a failure. This code walkthrough will begin by walking through a Workflow Execution, describing how the code maps to Commands and Events. There will then be a Worker crash halfway through, explaining how Temporal uses Replay to recover the state of the Workflow Execution, ultimately resulting in a completed execution that's identical to one that had not crashed. Image 1 out of 65 ![Slide 1](https://learn.temporal.io/courses/temporal-102/go/event-history-walkthrough/history-walkthrough/history-walkthrough.001.jpeg)❮❯ Example of a Non-Deterministic Workflow[​](https://docs.temporal.io/encyclopedia/event-history/event-history-go#Example-of-Non-Deterministic-Workflow "Direct link to Example of a Non-Deterministic Workflow") ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Now that Replay has been covered, this section will explain why Workflows need to be [deterministic](https://docs.temporal.io/workflow-definition#deterministic-constraints) in order for Replay to work. A Workflow is deterministic if every execution of its Workflow Definition produces the same Commands in the same sequence given the same input. As mentioned in the [`How History Replay Provides Durable Execution`](https://docs.temporal.io/encyclopedia/event-history/event-history-go#How-History-Replay-Provides-Durable-Execution) walkthrough, in the case of a failure, a Worker requests the Event History to replay it. During Replay, the Worker runs the Workflow code again to produce a set of Commands which is compared against the sequence of Commands in the Event History. When there’s a mismatch between the expected sequence of Commands the Worker expects based on the Event History and the actual sequence produced during Replay (due to non-determinism), Replay will be unable to continue. To better understand why Workflows need to be deterministic, it's helpful to look at a Workflow Definition that violates it. In this case, this code will walk through a Workflow Definition that breaks the determinism constraint with a random number generator. Image 1 out of 12 ![Slide 1](https://learn.temporal.io/courses/temporal-102/go/event-history-walkthrough/nondeterministic-workflow/nondeterministic-workflow.001.jpeg)❮❯ Note that non-deterministic failures do not fail the Workflow Execution by default. A non-deterministic failure is considered a [Workflow Task Failure](https://docs.temporal.io/references/failures#workflow-task-failures) which is considered a transient failure, meaning it retries over and over. Users can also fix the source of non-determinism, perhaps by removing the Activity, and then restart the Workers. This means that this type of failure can recover by itself. You can also use a strategy called versioning to address this non-determinism error. See [versioning](https://docs.temporal.io/develop/go/workflows/versioning) to learn more. For more information on how Temporal handles Durable Execution or to see these slides in a video format with more explanation, check out our free, self-paced courses: [Temporal 102](https://learn.temporal.io/courses/temporal_102/) and [Versioning Workflows](https://learn.temporal.io/courses/versioning/) . Temporal Applications Support Non-Deterministic Operations[​](https://docs.temporal.io/encyclopedia/event-history/event-history-go#temporal-applications-support-non-deterministic-operations "Direct link to Temporal Applications Support Non-Deterministic Operations") --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- We want to emphasize that although your Workflows themselves need to be deterministic, your application itself does not! Remember that pretty much anything that interacts with the external world is inherently non-deterministic: * Calling LLM APIs * Querying databases * Reading or writing files * Making HTTP requests to external services **Good news**: Your Temporal application can absolutely handle all of these operations. While your Workflow must be deterministic, your application absolutely can handle any type of non-deterministic operation, including those listed above. This gives you the best of both worlds—the crash-proof reliability of a Workflow and the resiliency of Activities which have built-in support for retries. * [How Workflow Code Maps to Commands](https://docs.temporal.io/encyclopedia/event-history/event-history-go#How-Workflow-Code-Maps-To-Commands) * [How Workflow Commands Map to Events](https://docs.temporal.io/encyclopedia/event-history/event-history-go#How-Workflow-Commands-Map-To-Events) * [How History Replay Provides Durable Execution](https://docs.temporal.io/encyclopedia/event-history/event-history-go#How-History-Replay-Provides-Durable-Execution) * [Example of a Non-Deterministic Workflow](https://docs.temporal.io/encyclopedia/event-history/event-history-go#Example-of-Non-Deterministic-Workflow) * [Temporal Applications Support Non-Deterministic Operations](https://docs.temporal.io/encyclopedia/event-history/event-history-go#temporal-applications-support-non-deterministic-operations) --- # Event History walkthrough with the Python SDK | Temporal Platform Documentation [Skip to main content](https://docs.temporal.io/encyclopedia/event-history/event-history-python#__docusaurus_skipToContent_fallback) On this page In order to understand how Workflow Replay works, this page will go through the following walkthroughs: 1. [How Workflow Code Maps to Commands](https://docs.temporal.io/encyclopedia/event-history/event-history-python#How-Workflow-Code-Maps-To-Commands) 2. [How Workflow Commands Map to Events](https://docs.temporal.io/encyclopedia/event-history/event-history-python#How-Workflow-Commands-Map-To-Events) 3. [How History Replay Provides Durable Execution](https://docs.temporal.io/encyclopedia/event-history/event-history-python#How-History-Replay-Provides-Durable-Execution) 4. [Example of a Non-Deterministic Workflow](https://docs.temporal.io/encyclopedia/event-history/event-history-python#Example-of-Non-Deterministic-Workflow) How Workflow Code Maps to Commands[​](https://docs.temporal.io/encyclopedia/event-history/event-history-python#How-Workflow-Code-Maps-To-Commands "Direct link to How Workflow Code Maps to Commands") ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- This walkthrough will cover how the Workflow code maps to Commands that get sent to the Temporal Service, letting the Temporal Service know what to do. Image 1 out of 11 ![Slide 1](https://learn.temporal.io/courses/temporal-102/python/event-history-walkthrough/code-commands/code-commands.001.jpeg)❮❯ How Workflow Commands Map to Events[​](https://docs.temporal.io/encyclopedia/event-history/event-history-python#How-Workflow-Commands-Map-To-Events "Direct link to How Workflow Commands Map to Events") ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- The Commands that are sent to the Temporal Service are then turned into Events, which build up the Event History. The Event History is a detailed log of Events that occur during the lifecycle of a Workflow Execution, such as the execution of Workflow Tasks or Activity Tasks. Event Histories are persisted to the database used by the Temporal Service, so they're durable, and will even survive a crash of the Temporal Service itself. These Events are what are used to recreate a Workflow Execution's state in the case of failure. Image 1 out of 14 ![Slide 1](https://learn.temporal.io/courses/temporal-102/python/event-history-walkthrough/commands-events/commands-events.001.jpeg)❮❯ How History Replay Provides Durable Execution[​](https://docs.temporal.io/encyclopedia/event-history/event-history-python#How-History-Replay-Provides-Durable-Execution "Direct link to How History Replay Provides Durable Execution") ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Now that you have seen how code maps to Commands, and how Commands map to Events, this next walkthrough will take a look at how Temporal uses Replay with the Events to provide Durable Execution and restore a Workflow Execution in the case of a failure. This code walkthrough will begin by walking through a Workflow Execution, describing how the code maps to Commands and Events. There will then be a Worker crash halfway through, explaining how Temporal uses Replay to recover the state of the Workflow Execution, ultimately resulting in a completed execution that's identical to one that had not crashed. Image 1 out of 61 ![Slide 1](https://learn.temporal.io/courses/temporal-102/python/event-history-walkthrough/history-replay/history-replay.001.jpeg)❮❯ Example of a Non-Deterministic Workflow[​](https://docs.temporal.io/encyclopedia/event-history/event-history-python#Example-of-Non-Deterministic-Workflow "Direct link to Example of a Non-Deterministic Workflow") -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Now that Replay has been covered, this section will explain why Workflows need to be [deterministic](https://docs.temporal.io/workflow-definition#deterministic-constraints) in order for Replay to work. A Workflow is deterministic if every execution of its Workflow Definition produces the same Commands in the same sequence given the same input. As mentioned in the [`How History Replay Provides Durable Execution`](https://docs.temporal.io/encyclopedia/event-history/event-history-python#How-History-Replay-Provides-Durable-Execution) walkthrough, in the case of a failure, a Worker requests the Event History to replay it. During Replay, the Worker runs the Workflow code again to produce a set of Commands which is compared against the sequence of Commands in the Event History. When there’s a mismatch between the expected sequence of Commands the Worker expects based on the Event History and the actual sequence produced during Replay (due to non-determinism), Replay will be unable to continue. To better understand why Workflows need to be deterministic, it's helpful to look at a Workflow Definition that violates it. In this case, this code will walk through a Workflow Definition that breaks the determinism constraint with a random number generator. Image 1 out of 13 ![Slide 1](https://learn.temporal.io/courses/temporal-102/python/event-history-walkthrough/nondeterministic-workflow/nondeterministic-workflow.001.jpeg)❮❯ Note that non-deterministic failures do not fail the Workflow Execution by default. A non-deterministic failure is considered a [Workflow Task Failure](https://docs.temporal.io/references/failures#workflow-task-failures) which is considered a transient failure, meaning it retries over and over. Users can also fix the source of non-determinism, perhaps by removing the Activity, and then restart the Workers. This means that this type of failure can recover by itself. You can also use a strategy called versioning to address this non-determinism error. See [versioning](https://docs.temporal.io/develop/python/workflows/versioning) to learn more. For more information on how Temporal handles Durable Execution or to see these slides in a video format with more explanation, check out our free, self-paced courses: [Temporal 102](https://learn.temporal.io/courses/temporal_102/) and [Versioning Workflows](https://learn.temporal.io/courses/versioning/) . Temporal Applications Support Non-Deterministic Operations[​](https://docs.temporal.io/encyclopedia/event-history/event-history-python#temporal-applications-support-non-deterministic-operations "Direct link to Temporal Applications Support Non-Deterministic Operations") ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- We want to emphasize that although your Workflows themselves need to be deterministic, your application itself does not! Remember that pretty much anything that interacts with the external world is inherently non-deterministic: * Calling LLM APIs * Querying databases * Reading or writing files * Making HTTP requests to external services **Good news**: Your Temporal application can absolutely handle all of these operations. While your Workflow must be deterministic, your application absolutely can handle any type of non-deterministic operation, including those listed above. This gives you the best of both worlds—the crash-proof reliability of a Workflow and the resiliency of Activities which have built-in support for retries. * [How Workflow Code Maps to Commands](https://docs.temporal.io/encyclopedia/event-history/event-history-python#How-Workflow-Code-Maps-To-Commands) * [How Workflow Commands Map to Events](https://docs.temporal.io/encyclopedia/event-history/event-history-python#How-Workflow-Commands-Map-To-Events) * [How History Replay Provides Durable Execution](https://docs.temporal.io/encyclopedia/event-history/event-history-python#How-History-Replay-Provides-Durable-Execution) * [Example of a Non-Deterministic Workflow](https://docs.temporal.io/encyclopedia/event-history/event-history-python#Example-of-Non-Deterministic-Workflow) * [Temporal Applications Support Non-Deterministic Operations](https://docs.temporal.io/encyclopedia/event-history/event-history-python#temporal-applications-support-non-deterministic-operations) --- # Event History walkthrough with the TypeScript SDK | Temporal Platform Documentation [Skip to main content](https://docs.temporal.io/encyclopedia/event-history/event-history-typescript#__docusaurus_skipToContent_fallback) On this page In order to understand how Workflow Replay works, this page will go through the following walkthroughs: 1. [How Workflow Code Maps to Commands](https://docs.temporal.io/encyclopedia/event-history/event-history-typescript#How-Workflow-Code-Maps-To-Commands) 2. [How Workflow Commands Map to Events](https://docs.temporal.io/encyclopedia/event-history/event-history-typescript#How-Workflow-Commands-Map-To-Events) 3. [How History Replay Provides Durable Execution](https://docs.temporal.io/encyclopedia/event-history/event-history-typescript#How-History-Replay-Provides-Durable-Execution) 4. [Example of a Non-Deterministic Workflow](https://docs.temporal.io/encyclopedia/event-history/event-history-typescript#Example-of-Non-Deterministic-Workflow) How Workflow Code Maps to Commands[​](https://docs.temporal.io/encyclopedia/event-history/event-history-typescript#How-Workflow-Code-Maps-To-Commands "Direct link to How Workflow Code Maps to Commands") ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- This walkthrough will cover how the Workflow code maps to Commands that get sent to the Temporal Service, letting the Temporal Service know what to do. Image 1 out of 12 ![Slide 1](https://learn.temporal.io/courses/temporal-102/typescript/event-history-walkthrough/code-commands/code-commands.001.jpeg)❮❯ How Workflow Commands Map to Events[​](https://docs.temporal.io/encyclopedia/event-history/event-history-typescript#How-Workflow-Commands-Map-To-Events "Direct link to How Workflow Commands Map to Events") -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- The Commands that are sent to the Temporal Service are then turned into Events, which build up the Event History. The Event History is a detailed log of Events that occur during the lifecycle of a Workflow Execution, such as the execution of Workflow Tasks or Activity Tasks. Event Histories are persisted to the database used by the Temporal Service, so they're durable, and will even survive a crash of the Temporal Service itself. These Events are what are used to recreate a Workflow Execution's state in the case of failure. Image 1 out of 14 ![Slide 1](https://learn.temporal.io/courses/temporal-102/typescript/event-history-walkthrough/commands-events/commands-events.001.jpeg)❮❯ How History Replay Provides Durable Execution[​](https://docs.temporal.io/encyclopedia/event-history/event-history-typescript#How-History-Replay-Provides-Durable-Execution "Direct link to How History Replay Provides Durable Execution") -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Now that you have seen how code maps to Commands, and how Commands map to Events, this next walkthrough will take a look at how Temporal uses Replay with the Events to provide Durable Execution and restore a Workflow Execution in the case of a failure. This code walkthrough will begin by walking through a Workflow Execution, describing how the code maps to Commands and Events. There will then be a Worker crash halfway through, explaining how Temporal uses Replay to recover the state of the Workflow Execution, ultimately resulting in a completed execution that's identical to one that had not crashed. Image 1 out of 61 ![Slide 1](https://learn.temporal.io/courses/temporal-102/typescript/event-history-walkthrough/history-replay/history-replay.001.jpeg)❮❯ Example of a Non-Deterministic Workflow[​](https://docs.temporal.io/encyclopedia/event-history/event-history-typescript#Example-of-Non-Deterministic-Workflow "Direct link to Example of a Non-Deterministic Workflow") ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ Now that Replay has been covered, this section will explain why Workflows need to be [deterministic](https://docs.temporal.io/workflow-definition#deterministic-constraints) in order for Replay to work. A Workflow is deterministic if every execution of its Workflow Definition produces the same Commands in the same sequence given the same input. As mentioned in the [`How History Replay Provides Durable Execution`](https://docs.temporal.io/encyclopedia/event-history/event-history-typescript#How-History-Replay-Provides-Durable-Execution) walkthrough, in the case of a failure, a Worker requests the Event History to replay it. During Replay, the Worker runs the Workflow code again to produce a set of Commands which is compared against the sequence of Commands in the Event History. When there’s a mismatch between the expected sequence of Commands the Worker expects based on the Event History and the actual sequence produced during Replay (due to non-determinism), Replay will be unable to continue. To better understand why Workflows need to be deterministic, it's helpful to look at a Workflow Definition that violates it. In this case, this code will walk through a Workflow Definition that breaks the determinism constraint with a random number generator. Image 1 out of 13 ![Slide 1](https://learn.temporal.io/courses/temporal-102/typescript/event-history-walkthrough/nondeterministic-workflow/nondeterministic-workflow.001.jpeg)❮❯ Note that non-deterministic failures do not fail the Workflow Execution by default. A non-deterministic failure is considered a [Workflow Task Failure](https://docs.temporal.io/references/failures#workflow-task-failures) which is considered a transient failure, meaning it retries over and over. Users can also fix the source of non-determinism, perhaps by removing the Activity, and then restart the Workers. This means that this type of failure can recover by itself. You can also use a strategy called versioning to address this non-determinism error. See [versioning](https://docs.temporal.io/develop/typescript/workflows/versioning) to learn more. For more information on how Temporal handles Durable Execution or to see these slides in a video format with more explanation, check out our free, self-paced courses: [Temporal 102](https://learn.temporal.io/courses/temporal_102/) and [Versioning Workflows](https://learn.temporal.io/courses/versioning/) . Temporal Applications Support Non-Deterministic Operations[​](https://docs.temporal.io/encyclopedia/event-history/event-history-typescript#temporal-applications-support-non-deterministic-operations "Direct link to Temporal Applications Support Non-Deterministic Operations") ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- We want to emphasize that although your Workflows themselves need to be deterministic, your application itself does not! Remember that pretty much anything that interacts with the external world is inherently non-deterministic: * Calling LLM APIs * Querying databases * Reading or writing files * Making HTTP requests to external services **Good news**: Your Temporal application can absolutely handle all of these operations. While your Workflow must be deterministic, your application absolutely can handle any type of non-deterministic operation, including those listed above. This gives you the best of both worlds—the crash-proof reliability of a Workflow and the resiliency of Activities which have built-in support for retries. * [How Workflow Code Maps to Commands](https://docs.temporal.io/encyclopedia/event-history/event-history-typescript#How-Workflow-Code-Maps-To-Commands) * [How Workflow Commands Map to Events](https://docs.temporal.io/encyclopedia/event-history/event-history-typescript#How-Workflow-Commands-Map-To-Events) * [How History Replay Provides Durable Execution](https://docs.temporal.io/encyclopedia/event-history/event-history-typescript#How-History-Replay-Provides-Durable-Execution) * [Example of a Non-Deterministic Workflow](https://docs.temporal.io/encyclopedia/event-history/event-history-typescript#Example-of-Non-Deterministic-Workflow) * [Temporal Applications Support Non-Deterministic Operations](https://docs.temporal.io/encyclopedia/event-history/event-history-typescript#temporal-applications-support-non-deterministic-operations) --- # Visibility | Temporal Platform Documentation [Skip to main content](https://docs.temporal.io/temporal-service/visibility#__docusaurus_skipToContent_fallback) On this page This page discusses [Visibility](https://docs.temporal.io/temporal-service/visibility#visibility) . What is Visibility?[​](https://docs.temporal.io/temporal-service/visibility#visibility "Direct link to What is Visibility?") ----------------------------------------------------------------------------------------------------------------------------- The term [Visibility](https://docs.temporal.io/visibility) , within the Temporal Platform, refers to the subsystems and APIs that enable an operator to view, filter, and search for Workflow Executions that currently exist within a Temporal Service. The [Visibility store](https://docs.temporal.io/self-hosted-guide/visibility) in your Temporal Service stores persisted Workflow Execution Event History data and is set up as a part of your [Persistence store](https://docs.temporal.io/temporal-service/persistence) to enable listing and filtering details about Workflow Executions that exist on your Temporal Service. * [How to set up a Visibility store](https://docs.temporal.io/self-hosted-guide/visibility) With Temporal Server v1.21, you can set up [Dual Visibility](https://docs.temporal.io/dual-visibility) to migrate your Visibility store from one database to another. Support for separate standard and advanced Visibility setups will be deprecated from Temporal Server v1.21 onwards. Check [Supported databases](https://docs.temporal.io/self-hosted-guide/visibility) for updates. * [What is Visibility?](https://docs.temporal.io/temporal-service/visibility#visibility) --- # Multi-Cluster Replication | Temporal Platform Documentation [Skip to main content](https://docs.temporal.io/temporal-service/multi-cluster-replication#__docusaurus_skipToContent_fallback) On this page This page discusses the following: * [Multi-Cluster Replication](https://docs.temporal.io/temporal-service/multi-cluster-replication#multi-cluster-replication) * [Namespace Versions](https://docs.temporal.io/temporal-service/multi-cluster-replication#namespace-versions) * [Version History](https://docs.temporal.io/temporal-service/multi-cluster-replication#version-history) * [Conflict Resolution](https://docs.temporal.io/temporal-service/multi-cluster-replication#conflict-resolution) * [Zombie Workflows](https://docs.temporal.io/temporal-service/multi-cluster-replication#zombie-workflows) * [Workflow Task Processing](https://docs.temporal.io/temporal-service/multi-cluster-replication#workflow-task-processing) What is Multi-Cluster Replication?[​](https://docs.temporal.io/temporal-service/multi-cluster-replication#multi-cluster-replication "Direct link to What is Multi-Cluster Replication?") ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Multi-Cluster Replication is a feature which asynchronously replicates Workflow Executions from active Clusters to other passive Clusters, for backup and state reconstruction. When necessary, for higher availability, Cluster operators can failover to any of the backup Clusters. Temporal's Multi-Cluster Replication feature is considered **experimental** and not subject to normal [versioning and support policy](https://docs.temporal.io/temporal-service/temporal-server#versions-and-support) . Temporal automatically forwards Start, Signal, and Query requests to the active Cluster. This feature must be enabled through a Dynamic Config flag per [Global Namespace](https://docs.temporal.io/global-namespace) . When the feature is enabled, Tasks are sent to the Parent Task Queue partition that matches that Namespace, if it exists. All Visibility APIs can be used against active and standby Clusters. This enables [Temporal UI](https://docs.temporal.io/web-ui) to work seamlessly for Global Namespaces. Applications making API calls directly to the Temporal Visibility API continue to work even if a Global Namespace is in standby mode. However, they might see a lag due to replication delay when querying the Workflow Execution state from a standby Cluster. Namespace Versions[​](https://docs.temporal.io/temporal-service/multi-cluster-replication#namespace-versions "Direct link to Namespace Versions") -------------------------------------------------------------------------------------------------------------------------------------------------- A _version_ is a concept in Multi-Cluster Replication that describes the chronological order of events per Namespace. With Multi-Cluster Replication, all Namespace change events and Workflow Execution History events are replicated asynchronously for high throughput. This means that data across clusters is **not** strongly consistent. To guarantee that Namespace data and Workflow Execution data will achieve eventual consistency (especially when there is a data conflict during a failover), a **version** is introduced and attached to Namespaces. All Workflow Execution History entries generated in a Namespace will also come with the version attached to that Namespace. All participating Clusters are pre-configured with a unique initial version and a shared version increment: * `initial version < shared version increment` When performing failover for a Namespace from one Cluster to another Cluster, the version attached to the Namespace will be changed by the following rule: * for all versions which follow `version % (shared version increment) == (active cluster's initial version)`, find the smallest version which has `version >= old version in namespace` When there is a data conflict, a comparison will be made and Workflow Execution History entries with the highest version will be considered the source of truth. When a cluster is trying to mutate a Workflow Execution History, the version will be checked. A cluster can mutate a Workflow Execution History only if the following is true: * The version in the Namespace belongs to this cluster, i.e. `(version in namespace) % (shared version increment) == (this cluster's initial version)` * The version of this Workflow Execution History's last entry (event) is equal or less than the version in the Namespace, i.e. `(last event's version) <= (version in namespace)` Namespace version change example Assuming the following scenario: * Cluster A comes with initial version: 1 * Cluster B comes with initial version: 2 * Shared version increment: 10 T = 0: Namespace α is registered, with active Cluster set to Cluster A namespace α's version is 1all workflows events generated within this namespace, will come with version 1 T = 1: namespace β is registered, with active Cluster set to Cluster B namespace β's version is 2all workflows events generated within this namespace, will come with version 2 T = 2: Namespace α is updated to with active Cluster set to Cluster B namespace α's version is 2all workflows events generated within this namespace, will come with version 2 T = 3: Namespace β is updated to with active Cluster set to Cluster A namespace β's version is 11all workflows events generated within this namespace, will come with version 11 Version history[​](https://docs.temporal.io/temporal-service/multi-cluster-replication#version-history "Direct link to Version history") ----------------------------------------------------------------------------------------------------------------------------------------- Version history is a concept which provides a high level summary of version information in regards to Workflow Execution History. Whenever there is a new Workflow Execution History entry generated, the version from Namespace will be attached. The Workflow Execution's mutable state will keep track of all history entries (events) and the corresponding version. Version history example (without data conflict) * Cluster A comes with initial version: 1 * Cluster B comes with initial version: 2 * Shared version increment: 10 T = 0: adding event with event ID == 1 & version == 1 View in both Cluster A & B | -------- | --------------- | --------------- | ------- || Events | Version History || -------- | --------------- | --------------- | ------- || Event ID | Event Version | Event ID | Version || -------- | ------------- | --------------- | ------- || 1 | 1 | 1 | 1 || -------- | ------------- | --------------- | ------- | T = 1: adding event with event ID == 2 & version == 1 View in both Cluster A & B | -------- | --------------- | --------------- | ------- || Events | Version History || -------- | --------------- | --------------- | ------- || Event ID | Event Version | Event ID | Version || -------- | ------------- | --------------- | ------- || 1 | 1 | 2 | 1 || 2 | 1 | | || -------- | ------------- | --------------- | ------- | T = 2: adding event with event ID == 3 & version == 1 View in both Cluster A & B | -------- | --------------- | --------------- | ------- || Events | Version History || -------- | --------------- | --------------- | ------- || Event ID | Event Version | Event ID | Version || -------- | ------------- | --------------- | ------- || 1 | 1 | 3 | 1 || 2 | 1 | | || 3 | 1 | | || -------- | ------------- | --------------- | ------- | T = 3: Namespace failover triggered, Namespace version is now 2 adding event with event ID == 4 & version == 2 View in both Cluster A & B | -------- | --------------- | --------------- | ------- || Events | Version History || -------- | --------------- | --------------- | ------- || Event ID | Event Version | Event ID | Version || -------- | ------------- | --------------- | ------- || 1 | 1 | 3 | 1 || 2 | 1 | 4 | 2 || 3 | 1 | | || 4 | 2 | | || -------- | ------------- | --------------- | ------- | T = 4: adding event with event ID == 5 & version == 2 View in both Cluster A & B | -------- | --------------- | --------------- | ------- || Events | Version History || -------- | --------------- | --------------- | ------- || Event ID | Event Version | Event ID | Version || -------- | ------------- | --------------- | ------- || 1 | 1 | 3 | 1 || 2 | 1 | 5 | 2 || 3 | 1 | | || 4 | 2 | | || 5 | 2 | | || -------- | ------------- | --------------- | ------- | Since Temporal is AP, during failover (change of active Temporal Service Namespace), there can exist cases where more than one Cluster can modify a Workflow Execution, causing divergence of Workflow Execution History. Below shows how the version history will look like under such conditions. Version history example (with data conflict) Below, shows version history of the same Workflow Execution in 2 different Clusters. * Cluster A comes with initial version: 1 * Cluster B comes with initial version: 2 * Cluster C comes with initial version: 3 * Shared version increment: 10 T = 0: View in both Cluster B & C | -------- | --------------- | --------------- | ------- || Events | Version History || -------- | --------------- | --------------- | ------- || Event ID | Event Version | Event ID | Version || -------- | ------------- | --------------- | ------- || 1 | 1 | 2 | 1 || 2 | 1 | 3 | 2 || 3 | 2 | | || -------- | ------------- | --------------- | ------- | T = 1: adding event with event ID == 4 & version == 2 in Cluster B | -------- | --------------- | --------------- | ------- || Events | Version History || -------- | --------------- | --------------- | ------- || Event ID | Event Version | Event ID | Version || -------- | ------------- | --------------- | ------- || 1 | 1 | 2 | 1 || 2 | 1 | 4 | 2 || 3 | 2 | | || 4 | 2 | | || -------- | ------------- | --------------- | ------- | T = 1: namespace failover to Cluster C, adding event with event ID == 4 & version == 3 in Cluster C | -------- | --------------- | --------------- | ------- || Events | Version History || -------- | --------------- | --------------- | ------- || Event ID | Event Version | Event ID | Version || -------- | ------------- | --------------- | ------- || 1 | 1 | 2 | 1 || 2 | 1 | 3 | 2 || 3 | 2 | 4 | 3 || 4 | 3 | | || -------- | ------------- | --------------- | ------- | T = 2: replication task from Cluster C arrives in Cluster B Note: below are a tree structures | ------------- | ------------- || Events || ------------- | ------------- || Event ID | Event Version || ------------- | ------------- || 1 | 1 || 2 | 1 || 3 | 2 || -------- | ------------- || || || -------- | ------------- | | -------- | ------------- || Event ID | Event Version | | Event ID | Event Version || -------- | ------------- | | -------- | ------------- || 4 | 2 | | 4 | 3 || -------- | ------------- | | -------- | ------------- || --------------- | ------------------- || Version History || --------------- | ------------------- || Event ID | Version || --------------- | ------- || 2 | 1 || 3 | 2 || --------------- | ------------------- || || || --------------- | ------- | | --------------- | ------- || Event ID | Version | | Event ID | Version || --------------- | ------- | | --------------- | ------- || 4 | 2 | | 4 | 3 || --------------- | ------- | | --------------- | ------- | T = 2: replication task from Cluster B arrives in Cluster C, same as above Conflict resolution[​](https://docs.temporal.io/temporal-service/multi-cluster-replication#conflict-resolution "Direct link to Conflict resolution") ----------------------------------------------------------------------------------------------------------------------------------------------------- When a Workflow Execution History diverges, proper conflict resolution is applied. In Multi-cluster Replication, Workflow Execution History Events are modeled as a tree, as shown in the second example in [Version History](https://docs.temporal.io/temporal-service/multi-cluster-replication#version-history) . Workflow Execution Histories that diverge will have more than one history branch. Among all history branches, the history branch with the highest version is considered the `current branch` and the Workflow Execution's mutable state is a summary of the current branch. Whenever there is a switch between Workflow Execution History branches, a complete rebuild of the Workflow Execution's mutable state will occur. Temporal Multi-Cluster Replication relies on asynchronous replication of Events across Clusters, so in the case of a failover it is possible to have an Activity Task dispatched again to the newly active Cluster due to a replication task lag. This also means that whenever a Workflow Execution is updated after a failover by the new Cluster, any previous replication tasks for that Execution cannot be applied. This results in loss of some progress made by the Workflow Execution in the previous active Cluster. During such conflict resolution, Temporal re-injects any external Events like Signals in the new Event History before discarding replication tasks. Even though some progress could roll back during failovers, Temporal provides the guarantee that Workflow Executions won't get stuck and will continue to make forward progress. Activity Execution completions are not forwarded across Clusters. Any outstanding Activities will eventually time out based on the configuration. Your application should have retry logic in place so that the Activity gets retried and dispatched again to a Worker after the failover to the new Cluster. Handling this is similar to handling an Activity Task timeout caused by a Worker restarting. Zombie Workflows[​](https://docs.temporal.io/temporal-service/multi-cluster-replication#zombie-workflows "Direct link to Zombie Workflows") -------------------------------------------------------------------------------------------------------------------------------------------- There is an existing contract that for any Namespace and Workflow Id combination, there can be at most one run (Namespace + Workflow Id + Run Id) open / executing. Multi-cluster Replication aims to keep the Workflow Execution History as up-to-date as possible among all participating Clusters. Due to the nature of Multi-cluster Replication (for example, Workflow Execution History events are replicated asynchronously) different Runs (same Namespace and Workflow Id) can arrive at the target Cluster at different times, sometimes out of order, as shown below: +-----------+ +----------------+ +-----------+| Cluster A | | Network Layer | | Cluster B |+-----------+ +----------------+ +-----------+ | | | | Run 1 Replication | | |---------------------> | | | | | | Run 2 Replication | | |---------------------> | | | | | | | Run 2 Replication | | |---------------------> | | | | | | Run 1 Replication | | |---------------------> | | | | Because Run 2 appears in Cluster B first, Run 1 cannot be replicated as "runnable" due to the rule `at most one Run open` (see above), thus the "zombie" Workflow Execution state is introduced. A "zombie" state is one in which a Workflow Execution which cannot be actively mutated by a Cluster (assuming the corresponding Namespace is active in this Cluster). A zombie Workflow Execution can only be changed by a replication Task. Run 1 will be replicated similar to Run 2, except when Run 1's execution will become a "zombie" before Run 1 reaches completion. Workflow Task processing[​](https://docs.temporal.io/temporal-service/multi-cluster-replication#workflow-task-processing "Direct link to Workflow Task processing") -------------------------------------------------------------------------------------------------------------------------------------------------------------------- In the context of Multi-cluster Replication, a Workflow Execution's mutable state is an entity which tracks all pending tasks. Prior to the introduction of Multi-cluster Replication, Workflow Execution History entries (events) are from a single branch, and the Temporal Server will only append new entries (events) to the Workflow Execution History. After the introduction of Multi-cluster Replication, it is possible that a Workflow Execution can have multiple Workflow Execution History branches. Tasks generated according to one history branch may become invalidated by switching history branches during conflict resolution. Example: T = 0: task A is generated according to Event Id: 4, version: 2 | -------- | ------------- || Events || -------- | ------------- || Event ID | Event Version || -------- | ------------- || 1 | 1 || 2 | 1 || 3 | 2 || -------- | ------------- || || || -------- | ------------- || Event ID | Event Version || -------- | ------------- || 4 | 2 | <-- task A belongs to this event || -------- | ------------- | T = 1: conflict resolution happens, Workflow Execution's mutable state is rebuilt and history Event Id: 4, version: 3 is written down to persistence | --------- | ------------- || Events || --------- | ------------- || Event ID | Event Version || -------- | ------------- || 1 | 1 || 2 | 1 || 3 | 2 || -------- | ------------- || || --------- | ------------- || | | -------- | ------------- | | -------- | ------------- || Event ID | Event Version | | Event ID | Event Version || -------- | ------------- | | -------- | ------------- || 4 | 2 | <-- task A belongs to this event | 4 | 3 | <-- current branch / mutable state| -------- | ------------- | | -------- | ------------- | T = 2: task A is loaded. At this time, due to the rebuild of a Workflow Execution's mutable state (conflict resolution), Task A is no longer relevant (Task A's corresponding Event belongs to non-current branch). Task processing logic will verify both the Event Id and version of the Task against a corresponding Workflow Execution's mutable state, then discard task A. * [What is Multi-Cluster Replication?](https://docs.temporal.io/temporal-service/multi-cluster-replication#multi-cluster-replication) * [Namespace Versions](https://docs.temporal.io/temporal-service/multi-cluster-replication#namespace-versions) * [Version history](https://docs.temporal.io/temporal-service/multi-cluster-replication#version-history) * [Conflict resolution](https://docs.temporal.io/temporal-service/multi-cluster-replication#conflict-resolution) * [Zombie Workflows](https://docs.temporal.io/temporal-service/multi-cluster-replication#zombie-workflows) * [Workflow Task processing](https://docs.temporal.io/temporal-service/multi-cluster-replication#workflow-task-processing) --- # Run a development server | Temporal Platform Documentation [Skip to main content](https://docs.temporal.io/develop/run-a-development-server#__docusaurus_skipToContent_fallback) On this page How to install the Temporal CLI and run a development server[​](https://docs.temporal.io/develop/run-a-development-server#run-a-development-server "Direct link to How to install the Temporal CLI and run a development server") ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- This page describes how to install the [Temporal CLI](https://docs.temporal.io/cli) and run a development Temporal Service. The local development Temporal Service comes packaged with the [Temporal Web UI](https://docs.temporal.io/web-ui) . For information on deploying and running a self-hosted production Temporal Service, see the [Self-hosted guide](https://docs.temporal.io/self-hosted-guide) , or sign up for [Temporal Cloud](https://docs.temporal.io/cloud) and let us run your production Temporal Service for you. Temporal CLI is a tool for interacting with a Temporal Service from the command line and it includes a distribution of the Temporal Server and Web UI. This local development Temporal Service runs as a single process with zero runtime dependencies and it supports persistence to disk and in-memory mode through SQLite. **Install the Temporal CLI** The Temporal CLI is available on macOS, Windows, and Linux. ### macOS[​](https://docs.temporal.io/develop/run-a-development-server#macos "Direct link to macOS") **How to install the Temporal CLI on macOS** Choose one of the following install methods to install the Temporal CLI on macOS: **Install the Temporal CLI with Homebrew** brew install temporal **Install the Temporal CLI from CDN** 1. Select the platform and architecture needed. * Download for Darwin amd64: [https://temporal.download/cli/archive/latest?platform=darwin&arch=amd64](https://temporal.download/cli/archive/latest?platform=darwin&arch=amd64) * Download for Darwin arm64: [https://temporal.download/cli/archive/latest?platform=darwin&arch=arm64](https://temporal.download/cli/archive/latest?platform=darwin&arch=arm64) 2. Extract the downloaded archive. 3. Add the `temporal` binary to your PATH. ### Linux[​](https://docs.temporal.io/develop/run-a-development-server#linux "Direct link to Linux") **How to install the Temporal CLI on Linux** Choose one of the following install methods to install the Temporal CLI on Linux: **Install the Temporal CLI with Homebrew** brew install temporal **Install the Temporal CLI from CDN** 1. Select the platform and architecture needed. * Download for Linux amd64: [https://temporal.download/cli/archive/latest?platform=linux&arch=amd64](https://temporal.download/cli/archive/latest?platform=linux&arch=amd64) * Download for Linux arm64: [https://temporal.download/cli/archive/latest?platform=linux&arch=arm64](https://temporal.download/cli/archive/latest?platform=linux&arch=arm64) 2. Extract the downloaded archive. 3. Add the `temporal` binary to your PATH. ### Windows[​](https://docs.temporal.io/develop/run-a-development-server#windows "Direct link to Windows") **How to install the Temporal CLI on Windows** Follow these instructions to install the Temporal CLI on Windows: **Install the Temporal CLI from CDN** 1. Select the platform and architecture needed and download the binary. * Download for Windows amd64: [https://temporal.download/cli/archive/latest?platform=windows&arch=amd64](https://temporal.download/cli/archive/latest?platform=windows&arch=amd64) * Download for Windows arm64: [https://temporal.download/cli/archive/latest?platform=windows&arch=arm64](https://temporal.download/cli/archive/latest?platform=windows&arch=arm64) 2. Extract the downloaded archive. 3. Add the `temporal.exe` binary to your PATH. ### Start the Temporal Development Server[​](https://docs.temporal.io/develop/run-a-development-server#start-the-temporal-development-server "Direct link to Start the Temporal Development Server") Start the Temporal Development Server by using the `server start-dev` command. temporal server start-dev This command automatically starts the Web UI, creates the default [Namespace](https://docs.temporal.io/namespaces) , and uses an in-memory database. The Temporal Server should be available on `localhost:7233` and the Temporal Web UI should be accessible at [`http://localhost:8233`](http://localhost:8233/) . The server's startup configuration can be customized using command line options. For a full list of options, run: temporal server start-dev --help * [How to install the Temporal CLI and run a development server](https://docs.temporal.io/develop/run-a-development-server#run-a-development-server) * [macOS](https://docs.temporal.io/develop/run-a-development-server#macos) * [Linux](https://docs.temporal.io/develop/run-a-development-server#linux) * [Windows](https://docs.temporal.io/develop/run-a-development-server#windows) * [Start the Temporal Development Server](https://docs.temporal.io/develop/run-a-development-server#start-the-temporal-development-server) --- # Search Attributes | Temporal Platform Documentation [Skip to main content](https://docs.temporal.io/search-attribute#__docusaurus_skipToContent_fallback) On this page This page discusses the following: * [Search Attributes](https://docs.temporal.io/search-attribute#search-attribute) * [Default Search Attributes](https://docs.temporal.io/search-attribute#default-search-attribute) * [Custom Search Attributes](https://docs.temporal.io/search-attribute#custom-search-attribute) What is a Search Attribute?[​](https://docs.temporal.io/search-attribute#search-attribute "Direct link to What is a Search Attribute?") ---------------------------------------------------------------------------------------------------------------------------------------- A Search Attribute is an indexed field used in a [List Filter](https://docs.temporal.io/list-filter) to filter a list of [Workflow Executions](https://docs.temporal.io/workflow-execution) that have the Search Attribute in their metadata. Each Search Attribute is a key-value pair metadata object included in a Workflow Execution's Visibility information. This information is available in the Visibility store. Do not use sensitive data or PII in Search Attributes Do not include sensitive data, secrets, or personally identifiable information (PII) in Search Attribute **names or values**. Search Attribute values are stored unencrypted in the Visibility store and are not processed by a custom [Payload Codec](https://docs.temporal.io/payload-codec#payload-codec) . The Temporal Server must be able to read these values in plain text to support filtering and ordering, so encryption is not possible without breaking search functionality. Attribute names are also visible in Namespace configuration, query expressions, and Temporal UI. Using sensitive data in either names or values risks exposure to anyone with Namespace access and may violate data protection regulations such as GDPR, HIPAA, or SOC 2. Temporal provides some [default Search Attributes](https://docs.temporal.io/search-attribute#default-search-attribute) , such as `ExecutionStatus`, the current state of your Workflow Executions. You can also create [custom Search Attribute](https://docs.temporal.io/search-attribute#custom-search-attribute) keys in your Visibility store and assign values when starting a Workflow Execution or in Workflow code. When using [Continue-As-New](https://docs.temporal.io/workflow-execution/continue-as-new) or a [Temporal Cron Job](https://docs.temporal.io/cron-job) , Search Attribute keys are carried over to the new Workflow Run by default. Search Attribute values are only available for as long as the Workflow is. Search Attributes are most effective for search purposes or tasks requiring collection-based result sets. For business logic in which you need to get information about a Workflow Execution, consider one of the following: * Storing state in a local variable and exposing it with a Query. * Storing state in an external datastore through Activities and fetching it directly from the store. If your business logic requires high throughput or low latency, store and fetch the data through Activities. You might experience lag due to time passing between the Workflow's state change and the Activity updating the Visibility store. ### Default Search Attributes[​](https://docs.temporal.io/search-attribute#default-search-attribute "Direct link to Default Search Attributes") A Temporal Service has a set of default Search Attributes already available. Default Search Attributes are set globally in any Namespace. These Search Attributes are created when the initial index is created. | NAME | TYPE | DEFINITION | | --- | --- | --- | | BatcherUser | Keyword | Used by internal batcher Workflow that runs in `TemporalBatcher` Namespace division to indicate the user who started the batch operation. | | BinaryChecksums | Keyword List | List of binary Ids of Workers that run the Workflow Execution. Deprecated since server version 1.21 in favor of the `BuildIds` search attribute. | | BuildIds | Keyword List | List of Worker Build Ids that have processed the Workflow Execution, formatted as `versioned:{BuildId}` or `unversioned:{BuildId}`, or the sentinel `unversioned` value. Available from server version 1.21. | | CloseTime | Datetime | The time at which the Workflow Execution completed. | | ExecutionDuration | Int | The time needed to run the Workflow Execution (in nanoseconds). Available only for closed Workflows. | | ExecutionStatus | Keyword | The current state of the Workflow Execution. | | ExecutionTime | Datetime | The time at which the Workflow Execution actually begins running; same as `StartTime` for most cases but different for Cron Workflows and retried Workflows. | | HistoryLength | Int | The number of events in the history of Workflow Execution. Available only for closed Workflows. | | HistorySizeBytes | Long | The size of the Event History. | | RunId | Keyword | Identifies the current Workflow Execution Run. | | StartTime | Datetime | The time at which the Workflow Execution started. | | StateTransitionCount | Int | The number of times that Workflow Execution has persisted its state. Available only for closed Workflows. | | TaskQueue | Keyword | Task Queue used by Workflow Execution. | | TemporalChangeVersion | Keyword List | Stores change/version pairs if the GetVersion API is enabled. | | TemporalReportedProblems | Keyword List | Stores information about Workflow task failures. Formatted as `category= cause=`. | | TemporalScheduledStartTime | Datetime | The time that the Workflow is schedule to start according to the Schedule Spec. Can be manually triggered. Set on Schedules. | | TemporalScheduledById | Keyword | The Id of the Schedule that started the Workflow. | | TemporalSchedulePaused | Boolean | Indicates whether the Schedule has been paused. Set on Schedules. | | TemporalWorkerDeployment | Keyword | Indicates the name of the associated Worker Deployment. | | TemporalWorkerDeploymentVersion | Keyword | Indicates the Version string of the associated Worker Deployment, in the format `:`. | | TemporalWorkflowVersioningBehavior | Keyword | Indicates the associated Worker Versioning behavior ("Pinned", "Auto-Upgrade", or null if not using Worker Versioning). | | WorkflowId | Keyword | Identifies the Workflow Execution. | | WorkflowType | Keyword | The type of Workflow. | * All default Search Attributes are reserved and read-only. You cannot create a custom one with the same name or alter the existing one. * Search Attributes are not encrypted in the system. Do not use sensitive data or PII as either the Search Attribute name or value. * To use default Search Attributes with the `Temporal` prefix in a List Filter, you can use their non-prefixed alias. Refer to [Search Attribute aliasing](https://docs.temporal.io/list-filter#search-attribute-aliasing) for details. * ExecutionStatus values correspond to Workflow Execution statuses: Running, Completed, Failed, Canceled, Terminated, ContinuedAsNew, TimedOut. * StartTime, CloseTime, and ExecutionTime are stored as dates but are supported by queries that use either EpochTime in nanoseconds or a string in [RFC3339Nano format](https://pkg.go.dev/time#pkg-constants) (such as "2006-01-02T15:04:05.999999999Z07:00"). * ExecutionDuration is stored in nanoseconds but is supported by queries that use integers in nanoseconds, [Golang duration format](https://pkg.go.dev/time#ParseDuration) , or "hh:mm:ss" format. * CloseTime, HistoryLength, StateTransitionCount, and ExecutionDuration are present only in a closed Workflow Execution. * ExecutionTime can differ from StartTime in retry and cron use cases. You can use the default Search Attributes in a List Filter, such as in the Temporal Web UI or with the `temporal workflow list` commands, under the following conditions: * Without advanced Visibility, you can only use the `=` operator with a single default Search Attribute in your List Filter. For example: `temporal workflow list --query "ExecutionStatus = 'Completed'"` or `temporal workflow list --query "WorkflowType = 'YourWorkflow'"`. * With advanced Visibility, you can combine default Search Attributes in a List Filter to get a list of specific Workflow Executions. For example: `temporal workflow list --query "WorkflowType = 'main.YourWorkflowDefinition' and ExecutionStatus != 'Running' and (StartTime > '2022-06-07T16:46:34.236-08:00' or CloseTime < '2022-06-08T16:46:34-08:00')"` ### Custom Search Attributes[​](https://docs.temporal.io/search-attribute#custom-search-attribute "Direct link to Custom Search Attributes") You can [create custom Search Attributes](https://docs.temporal.io/self-hosted-guide/visibility#create-custom-search-attributes) with unique key names that are relevant to your business needs. Use custom Search Attributes in a List Filter, such as in the Temporal Web UI or with the `temporal workflow list` commands, under the following conditions: * Without advanced Visibility, you cannot use a custom Search Attribute in your List Filter. * With advanced Visibility, you can create multiple custom Search Attributes and use them in combinations with List Filters to get a list of specific Workflow Executions. For example: `temporal workflow list --query "WorkflowType = 'main.YourWorkflowDefinition' and YourCustomSA = 'YourCustomSAValue' and (StartTime > '2022-06-07T16:46:34.236-08:00' or CloseTime < '2022-06-08T16:46:34-08:00')"` * With Temporal Server v1.19 and earlier, you must [integrate Elasticsearch](https://docs.temporal.io/self-hosted-guide/visibility#elasticsearch) to use custom Search Attributes with List Filters. * With Temporal Server v1.20 and later, custom Search Attribute capabilities are available on MySQL (v8.0.17 or later), PostgreSQL (v12 and later), and SQLite (v3.31.0 and later), in addition to Elasticsearch. If you use Elasticsearch as your Visibility store, your custom Search Attributes apply globally and can be used across Namespaces. However, if using any of the [supported SQL databases](https://docs.temporal.io/self-hosted-guide/visibility) with Temporal Server v1.20 and later, your custom Search Attributes are associated with a specific Namespace and can be used for Workflow Executions in that Namespace. See [custom Search Attributes limits](https://docs.temporal.io/search-attribute#custom-search-attribute-limits) for limits on the number and size of custom Search Attributes you can create. #### Supported types[​](https://docs.temporal.io/search-attribute#supported-types "Direct link to Supported types") Custom Search Attributes must be one of the following types: * Bool * Datetime * Double * Int * Keyword * KeywordList * Text Note: * **Double** is backed up by `scaled_float` Elasticsearch type with scale factor 10000 (4 decimal digits). * **Datetime** is backed up by `date` type with milliseconds precision in Elasticsearch 6 and `date_nanos` type with nanoseconds precision in Elasticsearch 7. * **Int** is 64-bit integer (`long` Elasticsearch type). * **Keyword** and **Text** types are concepts taken from Elasticsearch. Each word in a **Text** is considered a searchable keyword. For a UUID, that can be problematic because Elasticsearch indexes each portion of the UUID separately. To have the whole string considered as a searchable keyword, use the **Keyword** type. For example, if the key `ProductId` has the value of `2dd29ab7-2dd8-4668-83e0-89cae261cfb1`: * As a **Keyword** it would be matched only by `ProductId = "2dd29ab7-2dd8-4668-83e0-89cae261cfb1`. * As a **Text** it would be matched by `ProductId = 2dd8`, which could cause unwanted matches. * With Temporal Server v1.19 and earlier, the **Keyword** type can store a list of values. * With Temporal Server v1.20 and later, the **Keyword** type supports only a single value. To store a list of values, use **KeywordList**. * The **Text** type cannot be used in the "Order By" clause. #### Custom Search Attributes limits[​](https://docs.temporal.io/search-attribute#custom-search-attribute-limits "Direct link to Custom Search Attributes limits") The following table lists the maximum number of custom Search Attributes you can create per Namespace by supported Visibility database. | Search Attribute type | MySQL (v8.0.17 and later) | PostgreSQL (v12 and later) | SQLite (v3.31.0 and later) | Temporal Cloud | | --- | --- | --- | --- | --- | | Bool | 3 | 3 | 3 | 20 | | Datetime | 3 | 3 | 3 | 20 | | Double | 3 | 3 | 3 | 20 | | Int | 3 | 3 | 3 | 20 | | Keyword | 10 | 10 | 10 | 40 | | KeywordList | 3 | 3 | 3 | 5 | | Text | 3 | 3 | 3 | 5 | Temporal does not impose a limit on the number of custom Search Attributes you can create with Elasticsearch. However, [Elasticsearch sets a default mapping limit](https://www.elastic.co/guide/en/elasticsearch/reference/8.6/mapping-settings-limit.html) that may apply. Custom Search Attributes are an advanced Visibility feature and are not supported on Cassandra. Size limits for a custom Search Attribute: * The default single Search Attribute **value** size limit is 2 KB. * The maximum total Search Attribute size is 40 KB. * The maximum total characters per Search Attribute value is 255. For Temporal Cloud specific configurations, see the [Defaults, limits, and configurable settings -Temporal Cloud](https://docs.temporal.io/cloud/limits#number-of-custom-search-attributes) guide. ### Usage[​](https://docs.temporal.io/search-attribute#usage "Direct link to Usage") Search Attributes available in your Visibility store can be used with Workflow Executions for the Temporal Service. To actually have results from the use of a [List Filter](https://docs.temporal.io/list-filter) , Search Attributes must be added to a Workflow Execution as metadata. * To create custom Search Attributes in your Visibility store, see [Create custom Search Attributes](https://docs.temporal.io/self-hosted-guide/visibility#create-custom-search-attributes) . * To remove a custom Search Attribute from the Visibility store, see [Remove custom Search Attributes](https://docs.temporal.io/self-hosted-guide/visibility#remove-custom-search-attributes) . Removing custom Search Attributes is not supported on Temporal Cloud. * To rename a custom Search Attribute on Temporal Cloud, see [`tcld namespace search-attributes rename`](https://docs.temporal.io/cloud/tcld/namespace/#rename) . With Workflows you can do the following: * Set the value of Search Attributes in your Workflow * Update the value set for a Search Attribute from within the Workflow code * Remove the value set for a Search Attribute from within the Workflow code Manage Search Attributes by SDK * [How to manage Search Attributes using the Go SDK](https://docs.temporal.io/develop/go/platform/observability#visibility) * [How to manage Search Attributes using the Java SDK](https://docs.temporal.io/develop/java/platform/observability#visibility) * [How to manage Search Attributes using the PHP SDK](https://docs.temporal.io/develop/php/platform/observability#visibility) * [How to manage Search Attributes using the Python SDK](https://docs.temporal.io/develop/python/platform/observability#visibility) * [How to manage Search Attributes using the TypeScript SDK](https://docs.temporal.io/develop/typescript/platform/observability#visibility) * [How to manage Search Attributes using the .NET SDK](https://docs.temporal.io/develop/dotnet/platform/observability#search-attributes) * To get a list of Search Attributes using the Temporal CLI, issue `temporal operator search-attribute list`. See [Search Attributes](https://docs.temporal.io/search-attribute) . After you add and set your Search Attributes, use your default or custom Search Attributes in a List Filter. * [What is a Search Attribute?](https://docs.temporal.io/search-attribute#search-attribute) * [Default Search Attributes](https://docs.temporal.io/search-attribute#default-search-attribute) * [Custom Search Attributes](https://docs.temporal.io/search-attribute#custom-search-attribute) * [Supported types](https://docs.temporal.io/search-attribute#supported-types) * [Custom Search Attributes limits](https://docs.temporal.io/search-attribute#custom-search-attribute-limits) * [Usage](https://docs.temporal.io/search-attribute#usage) --- # Task Routing and Worker sessions | Temporal Platform Documentation [Skip to main content](https://docs.temporal.io/task-routing#__docusaurus_skipToContent_fallback) On this page This page discusses the following: * [Task Routing](https://docs.temporal.io/task-routing#task-routing) * [Worker Sessions](https://docs.temporal.io/task-routing#worker-session) What is Task Routing?[​](https://docs.temporal.io/task-routing#task-routing "Direct link to What is Task Routing?") -------------------------------------------------------------------------------------------------------------------- Task Routing is simply when a Task Queue is paired with one or more Workers, primarily for Activity Task Executions. This could also mean employing multiple Task Queues, each one paired with a Worker Process. Task Routing has many applicable use cases. Some SDKs provide a [Session API](https://docs.temporal.io/task-routing#worker-session) that provides a straightforward way to ensure that Activity Tasks are executed with the same Worker without requiring you to manually specify Task Queue names. It also includes features like concurrent session limitations and worker failure detection. ### Flow control[​](https://docs.temporal.io/task-routing#flow-control "Direct link to Flow control") A Worker that consumes from a Task Queue asks for an Activity Task only when it has available capacity, so it is never overloaded by request spikes. If Activity Tasks get created faster than Workers can process them, they are backlogged in the Task Queue. ### Throttling[​](https://docs.temporal.io/task-routing#throttling "Direct link to Throttling") The rate at which each Activity Worker polls for and processes Activity Tasks is configurable per Worker. Workers do not exceed this rate even if it has spare capacity. There is also support for global Task Queue rate limiting. This limit works across all Workers for the given Task Queue. It is frequently used to limit load on a downstream service that an Activity calls into. ### Specific environments[​](https://docs.temporal.io/task-routing#specific-environments "Direct link to Specific environments") In some cases, you might need to execute Activities in a dedicated environment. To send Activity Tasks to this environment, use a dedicated Task Queue. #### Route Activity Tasks to a specific host[​](https://docs.temporal.io/task-routing#route-activity-tasks-to-a-specific-host "Direct link to Route Activity Tasks to a specific host") In some use cases, such as file processing or machine learning model training, an Activity Task must be routed to a specific Worker Process or Worker Entity. For example, suppose that you have a Workflow with the following three separate Activities: * Download a file. * Process the file in some way. * Upload a file to another location. The first Activity, to download the file, could occur on any Worker on any host. However, the second and third Activities must be executed by a Worker on the same host where the first Activity downloaded the file. In a real-life scenario, you might have many Worker Processes scaled over many hosts. You would need to develop your Temporal Application to route Tasks to specific Worker Processes when needed. Code samples: * [Go file processing example](https://github.com/temporalio/samples-go/tree/main/fileprocessing) * [Java file processing example](https://github.com/temporalio/samples-java/tree/main/core/src/main/java/io/temporal/samples/fileprocessing) * [PHP file processing example](https://github.com/temporalio/samples-php/tree/master/app/src/FileProcessing) #### Route Activity Tasks to a specific process[​](https://docs.temporal.io/task-routing#route-activity-tasks-to-a-specific-process "Direct link to Route Activity Tasks to a specific process") Some Activities load large datasets and cache them in the process. The Activities that rely on those datasets should be routed to the same process. In this case, a unique Task Queue would exist for each Worker Process involved. #### Workers with different capabilities[​](https://docs.temporal.io/task-routing#workers-with-different-capabilities "Direct link to Workers with different capabilities") Some Workers might exist on GPU boxes versus non-GPU boxes. In this case, each type of box would have its own Task Queue and a Workflow can pick one to send Activity Tasks. ### Multiple priorities[​](https://docs.temporal.io/task-routing#multiple-priorities "Direct link to Multiple priorities") If your use case involves more than one priority, you can create one Task Queue per priority, with a Worker pool per priority. ### Versioning[​](https://docs.temporal.io/task-routing#versioning "Direct link to Versioning") Task Routing is the simplest way to version your code. If you have a new backward-incompatible Activity Definition, start by using a different Task Queue. What is a Worker Session?[​](https://docs.temporal.io/task-routing#worker-session "Direct link to What is a Worker Session?") ------------------------------------------------------------------------------------------------------------------------------ A Worker Session is a feature provided by some SDKs that provides a straightforward API for [Task Routing](https://docs.temporal.io/task-routing#task-routing) to ensure that Activity Tasks are executed with the same Worker without requiring you to manually specify Task Queue names. It also includes features like concurrent session limitations and Worker failure detection. * [How to use Worker Sessions](https://docs.temporal.io/develop/go/workers/sessions) * [What is Task Routing?](https://docs.temporal.io/task-routing#task-routing) * [Flow control](https://docs.temporal.io/task-routing#flow-control) * [Throttling](https://docs.temporal.io/task-routing#throttling) * [Specific environments](https://docs.temporal.io/task-routing#specific-environments) * [Route Activity Tasks to a specific host](https://docs.temporal.io/task-routing#route-activity-tasks-to-a-specific-host) * [Route Activity Tasks to a specific process](https://docs.temporal.io/task-routing#route-activity-tasks-to-a-specific-process) * [Workers with different capabilities](https://docs.temporal.io/task-routing#workers-with-different-capabilities) * [Multiple priorities](https://docs.temporal.io/task-routing#multiple-priorities) * [Versioning](https://docs.temporal.io/task-routing#versioning) * [What is a Worker Session?](https://docs.temporal.io/task-routing#worker-session) --- # Worker performance | Temporal Platform Documentation [Skip to main content](https://docs.temporal.io/develop/worker-performance#__docusaurus_skipToContent_fallback) On this page This page documents metrics and configurations that drive the efficiency of your Worker fleet. It provides coverage of performance metric families, Worker configuration options, Task Queue information, backlog counts, Task rates, and how to evaluate Worker availability. This content covers practical methods for querying Task Queue information, and strategies for tuning Workers and Task Queue processing so you manage your resources effectively. info All metrics on this page are prepended with the `temporal_` prefix. For example, `worker_task_slots_available` is actually `temporal_worker_task_slots_available` when used. The omitted prefix makes the names more readable and descriptive. Worker performance concepts[​](https://docs.temporal.io/develop/worker-performance#worker-performance-concepts "Direct link to Worker performance concepts") ------------------------------------------------------------------------------------------------------------------------------------------------------------- A Worker's performance characteristics are affected by, but not limited to, the following elements. ### Task slots[​](https://docs.temporal.io/develop/worker-performance#slots "Direct link to Task slots") A **Worker Task Slot**, represents the capacity of a Temporal Worker to execute a single concurrent Task. Slots are crucial for managing the workload and performance of Workers in a Temporal application. They're used for both Workflow and Activity Tasks. When a Worker starts processing a Task, it occupies one slot. The number of available slots directly affects how many tasks a Worker can handle simultaneously. ### Slot suppliers[​](https://docs.temporal.io/develop/worker-performance#slot-suppliers "Direct link to Slot suppliers") A **Slot Supplier** defines a strategy to provide slots for a Worker, increasing or decreasing the Worker's slot count. The supplier determines when it's acceptable to begin a new Task. Each supplier manages one slot type. There are slot types for Activity, Workflow, Nexus, or Local Activity Tasks. An available slot determines whether or not a Worker is willing to poll for, and execute, a new Task of that type. Slot supplier strategies include manual assignment of fixed slot counts and resource-balanced "auto-tuner" assignment. Resource-based suppliers adjust slot counts based on CPU and memory resources. Available slot suppliers include: * **Fixed Size Slot Suppliers**: Hands out slots up to a preset limit. This is useful if you have a concrete idea of how many resources your tasks are going to consume, and can easily determine an upper bound on how many should run at once. When you need the absolute best performance, review your hardware and environment characteristics. This information lets you calculate an appropriate fixed-size limit. Evaluate the maximum number of slots you can support without oversubscribing or hitting out-of-memory conditions ("OOMing"). Using that value with a fixed-size supplier provides optimal results with the least overhead. * **Resource-Based Slot Suppliers**: Hands out slots based on real-time CPU and memory usage. You set target utilization for both CPU and memory and the Slot Supplier tries to reach those values without exceeding them under load. A resource-based supplier will account for memory limits imposed in containerized environments. It dynamically adjusts the number of available slots for different task types with respect to current system resources. info When running in a containerized environment, all SDKs use cgroups for both CPU and memory. CPU is accounted for at the container level. * **Custom Slot Suppliers**: Hands out slots based on the custom logic that you define. Use this approach when you need complete control over when Workers accept and execute Tasks. For implementation details, see [Implement Custom Slot Suppliers](https://docs.temporal.io/develop/worker-performance#custom-slot-implementation) . caution * You cannot guarantee that the targets for resource-based suppliers won't ever be exceeded. Resources consumed during a task can't be known ahead of time. * Read about [choosing an appropriate slot supplier type](https://docs.temporal.io/develop/worker-performance#choosing-slot-supplier-types) before picking one. * Worker tuners supersede the existing `maxConcurrentXXXTask` style Worker options. Using both styles will cause an error at Worker initialization time. ### Worker tuning[​](https://docs.temporal.io/develop/worker-performance#worker-tuning "Direct link to Worker tuning") Worker tuning is the process of defining customized slot suppliers for the different task slots of a Worker to fine-tune its performance. You use special types called **Worker tuners** that assign slot suppliers to various Task Types, including Worker, Activity, Nexus, and Local Activity Tasks. For more on how to configure and use Worker tuners, refer to [Worker runtime performance tuning](https://docs.temporal.io/develop/worker-performance#worker-performance-tuning) . caution Worker tuners supersede the existing `maxConcurrentXXXTask` style Worker options. Using both styles will cause an error at Worker initialization time. ### Task Pollers[​](https://docs.temporal.io/develop/worker-performance#task-pollers "Direct link to Task Pollers") A Worker's **Task Pollers** play a crucial role in the Temporal architecture by efficiently ingesting work to Workers to support scalable, resilient Workflow Execution. Pollers create long-polling connections to the Temporal Service and actively poll a Task Queue for Tasks to process. When a Task Poller receives a Task, it delivers the Task to the appropriate Executor Slot for processing. Temporal SDKs implement support for _Poller Autoscaling_, which dynamically adjusts the number of pollers in use to maximize throughput for a given number of workers and the size of the task backlog. Temporal recommends using Poller Autoscaling for the majority of use cases, as manually setting the number of pollers too high or too low for your workload will result in decreased performance. To configure Poller Autoscaling, see [Configuring Poller Options](https://docs.temporal.io/develop/worker-performance#configuring-poller-options) and samples for each Temporal SDK. ### Eager task execution[​](https://docs.temporal.io/develop/worker-performance#eager-task-execution "Direct link to Eager task execution") caution Eager start does not respect Worker versioning. An eagerly started Workflow may run on any available local Worker even if that Worker is not the Current or Ramping version of its Worker deployment. As a latency optimization, Activity and Workflow Tasks may be started eagerly in a local Worker under the right circumstances. #### Eager Activity Start[​](https://docs.temporal.io/develop/worker-performance#eager-activity-start "Direct link to Eager Activity Start") Eager Activity Start may happen automatically if the Worker processing a Workflow Task has also registered the Activity Definition being called. If it does, it may try to reserve an Activity Slot for the execution of the Activity, and the server may respond to the Workflow Task completion with the Activity Task for the worker to execute immediately. #### Eager Workflow Start[​](https://docs.temporal.io/develop/worker-performance#eager-workflow-start "Direct link to Eager Workflow Start") SUPPORT, STABILITY, and DEPENDENCY INFO Eager Workflow Start is available in [Public Preview](https://docs.temporal.io/evaluate/development-production-features/release-stages#public-preview) in the Go, Java, Python, and .NET SDKs. Eager Workflow Start is enabled for all Temporal Cloud users and self-hosted Temporal Server 1.29.0+. No additional configuration or access request is needed. However, you must set Request-Eager-Start to true when starting each Workflow for Eager Workflow Start to be used. Eager Workflow Start reduces the latency required to initiate a Workflow execution. It is recommended for short-lived Workflows that use Local Activities to interact with external services, especially when these interactions are initiated in the first Workflow Task and the Workflow is deployed near the Temporal Server to minimize network delay. This feature is particularly beneficial for Workflows with a “happy path” that must begin external interactions within tens of milliseconds, while still relying on Temporal’s server-driven retries and compensation mechanisms to ensure reliability in failure scenarios. **Quick Start** Eager Workflow Start requires the Starter and the Worker to share a Client located in the same process and setting the `request_eager_start` (or similar name) to true in the Start Workflow call. When set, and the Worker has a Workflow Task slot available and the Workflow Definition registered, the Worker can execute the first task of the Workflow locally without first making a round-trip to the Temporal Server. This is typically most useful in combination with a Local Activity executing in the first Workflow Task, since other Workflow API calls that require waiting on something will force a round-trip. RESOURCES * [Go SDK - Code sample](https://github.com/temporalio/samples-go/tree/main/eager-workflow-start) * [Java SDK - Code sample](https://github.com/temporalio/samples-java/blob/main/core/src/main/java/io/temporal/samples/hello/HelloEagerWorkflowStart.java) * Python SDK - use `request_eager_start` when calling `start_workflow` or `execute_workflow` * .NET SDK - use `RequestEagerStart` in your `WorkflowOptions` when starting a workflow * [Blog: Improving Latency with Eager Workflow Start](https://temporal.io/blog/improving-latency-with-eager-workflow-start) **How it works** The traditional way to start a Workflow decouples the starter program from the worker by sharing a Task Queue name between them, similar to a publish/subscribe pattern. This has many advantages: for example, we can reliably schedule a Workflow Execution without a running Worker, or separate the Worker and Workflow implementation from the Starter application and host them independently. But decoupling also makes it harder to optimize for latency. Instead, when the **Starter and Worker are collocated in the same process** and aware of each other, they can interact while bypassing the server, saving a few time-intensive operations. ![Eager Workflow Start](https://docs.temporal.io/img/develop/worker-performance/eager-workflow-start-flow.png) Eager Workflow Start The above figure shows Eager Workflow Start in action: 1. The process begins with the Starter setting `request_eager_start` (or similar name) to true in the Start Workflow Options. 2. The SDK will try to locate a local Worker that is willing to execute the first Workflow Task, and reserve an execution slot for it. 3. If successful, the SDK will provide a hint to the server that eager mode is preferred for the new Workflow. 4. The server not only registers the start of the Workflow in history, it also assigns the first Workflow Task to the Starter, all in the same DB update. 5. The first task is included in the server response, no matching step required. 6. The SDK extracts the task from the response, and dispatches it to the local worker. To recover from errors, Eager Workflow Start falls back to the non-eager path. For example, when the first Task is returned eagerly, but the local Worker fails or times out while processing the task, the server retries this task non-eagerly after WorkflowTaskTimeout. Performance metrics for tuning[​](https://docs.temporal.io/develop/worker-performance#metrics "Direct link to Performance metrics for tuning") ----------------------------------------------------------------------------------------------------------------------------------------------- The Temporal SDKs emit metrics from Temporal Client usage and Worker Processes. Performance tuning uses three important SDK metric groups: ### Slot availability metrics[​](https://docs.temporal.io/develop/worker-performance#slot-availability-metrics "Direct link to Slot availability metrics") Temporal's [`worker_task_slots_available`](https://docs.temporal.io/references/sdk-metrics#worker_task_slots_available) and `worker_task_slots_used` gauges can report the number of available executor “slots” that are currently available and unoccupied for a Worker type. Tag these with `worker_type=WorkflowWorker` for Workflow Task Workers or `worker_type=ActivityWorker` for Activity Workers. tip Unlike `worker_task_slots_used`, `worker_task_slots_available` can only be used with fixed size slot suppliers and can't be used with resource-based slot suppliers. ### Latency metrics[​](https://docs.temporal.io/develop/worker-performance#latency-metrics "Direct link to Latency metrics") Temporal provides two latency timers: [`workflow_task_schedule_to_start_latency`](https://docs.temporal.io/references/sdk-metrics#workflow_task_schedule_to_start_latency) for Workflow Tasks and [`activity_schedule_to_start_latency`](https://docs.temporal.io/references/sdk-metrics#activity_schedule_to_start_latency) for Activities. A Schedule-To-Start latency is the time from when an Task is scheduled (that is, placed in a Queue) to when a Worker starts (that is, picks up from the Task Queue) that Task. These metrics help ensure that Tasks are being processed from the queue in a timely manner. For more information about `schedule_to_start` timeout and latency, see [Schedule-To-Start Timeout](https://docs.temporal.io/encyclopedia/detecting-activity-failures#schedule-to-start-timeout) . ### Cache metrics[​](https://docs.temporal.io/develop/worker-performance#cache-metrics "Direct link to Cache metrics") The [`sticky_cache_size`](https://docs.temporal.io/references/sdk-metrics#sticky_cache_size) and [`workflow_active_thread_count`](https://docs.temporal.io/references/sdk-metrics#workflow_active_thread_count) metrics report the size of the Workflow cache and the number of cached Workflow threads. Worker performance options[​](https://docs.temporal.io/develop/worker-performance#configuration "Direct link to Worker performance options") --------------------------------------------------------------------------------------------------------------------------------------------- Each Worker can be configured by providing custom Worker options (`WorkerOptions`) at instantiation. Options are specific to individual Workers and do not affect other members of your fleet. ### Executor slot options[​](https://docs.temporal.io/develop/worker-performance#executor-slot-options "Direct link to Executor slot options") The `maxConcurrentWorkflowTaskExecutionSize` and `maxConcurrentActivityExecutionSize` options define the number of total available Workflow Task and Activity Task slots for a Worker. caution * Worker tuners supersede the existing `maxConcurrentXXXTask` style Worker options. Using both styles will cause an error at Worker initialization time. ### Configuring Poller Options[​](https://docs.temporal.io/develop/worker-performance#configuring-poller-options "Direct link to Configuring Poller Options") #### Recommended Approach[​](https://docs.temporal.io/develop/worker-performance#recommended-approach "Direct link to Recommended Approach") The Temporal SDKs support Poller Autoscaling, which automatically selects an appropriate number of pollers based on need. Using this feature results in more efficient poller usage, better throughput, and schedule-to-start latency improvements. You can enable this feature by setting the `*_task_poller_behavior` options to `PollerBehaviorAutoscaling`. Names may vary slightly depending on the SDK. For specific examples of enabling Poller Autoscaling, see the SDK Examples section below. Poller Autoscaling will be the default configuration in future versions of Temporal SDKs. tip `PollerBehaviorAutoscaling` is only enabled in Temporal Server v1.28.0 and later. #### Manual Configuration[​](https://docs.temporal.io/develop/worker-performance#manual-configuration "Direct link to Manual Configuration") There are options available to manually configure minimum, maximum, and initial poller counts, but it is not recommended to set these values manually for production use cases. To set these values manually, the following options are available: * `maxConcurrentWorkflowTaskPollers` (in the JavaSDK: `workflowPollThreadCount`) * `maxConcurrentActivityTaskPollers` (in the JavaSDK: `activityPollThreadCount`) These options define the maximum count of pollers performing poll requests on Workflow and Activity Task Queues, respectively. #### SDK Examples[​](https://docs.temporal.io/develop/worker-performance#sdk-examples "Direct link to SDK Examples") * Go * Java * Python * TypeScript * PHP * .NET * Ruby [Go SDK docs](https://pkg.go.dev/go.temporal.io/sdk/worker#PollerBehaviorAutoscalingOptions) w := worker.New(c, "my-task-queue", worker.Options{ WorkflowTaskPollerBehavior: worker.NewPollerBehaviorAutoscaling(worker.PollerBehaviorAutoscalingOptions{}), ActivityTaskPollerBehavior: worker.NewPollerBehaviorAutoscaling(worker.PollerBehaviorAutoscalingOptions{}), NexusTaskPollerBehavior: worker.NewPollerBehaviorAutoscaling(worker.PollerBehaviorAutoscalingOptions{}),}) [Java SDK docs](https://javadoc.io/doc/io.temporal/temporal-sdk/latest/io/temporal/worker/tuning/PollerBehaviorAutoscaling.html) public class WorkerExample { public static void main(String[] args) { WorkflowServiceStubs service = WorkflowServiceStubs.newLocalServiceStubs(); WorkflowClient client = WorkflowClient.newInstance(service); WorkerFactory factory = WorkerFactory.newInstance(client); WorkerOptions workerOptions = WorkerOptions.newBuilder() .setWorkflowTaskPollersBehavior(new PollerBehaviorAutoscaling()) .setActivityTaskPollersBehavior(new PollerBehaviorAutoscaling()) .setNexusTaskPollersBehavior(new PollerBehaviorAutoscaling()) .build(); Worker worker = factory.newWorker("my-task-queue", workerOptions); }} [Python SDK docs](https://python.temporal.io/temporalio.worker.PollerBehaviorAutoscaling.html) worker = Worker( client, task_queue="my-task-queue", workflows=[MyWorkflow], activities=[my_activity], workflow_task_poller_behavior=PollerBehaviorAutoscaling(), activity_task_poller_behavior=PollerBehaviorAutoscaling(), nexus_task_poller_behavior=PollerBehaviorAutoscaling(),) [TypeScript SDK docs](https://typescript.temporal.io/api/interfaces/proto.temporal.api.sdk.v1.WorkerConfig.IAutoscalingPollerBehavior) const worker = await Worker.create({ connection, taskQueue: 'my-task-queue', workflowsPath: require.resolve('./workflows'), activities, workflowTaskPollerBehavior: PollerBehavior.autoscaling(), activityTaskPollerBehavior: PollerBehavior.autoscaling(), nexusTaskPollerBehavior: PollerBehavior.autoscaling(),}); **PHP** example coming soon. [DotNet SDK docs](https://dotnet.temporal.io/api/Temporalio.Worker.Tuning.PollerBehavior.Autoscaling.html) using var worker = new TemporalWorker( client, new TemporalWorkerOptions("my-task-queue") { WorkflowTaskPollerBehavior = new PollerBehavior.Autoscaling(), ActivityTaskPollerBehavior = new PollerBehavior.Autoscaling(), NexusTaskPollerBehavior = new PollerBehavior.Autoscaling(), } .AddWorkflow() .AddActivity(MyActivities.MyActivity)); [Ruby SDK docs](https://ruby.temporal.io/Temporalio/Worker/PollerBehavior/Autoscaling.html) worker = Temporalio::Worker.new( client, 'my-task-queue', workflows: [MyWorkflow], activities: [MyActivity], workflow_task_poller_behavior: Temporalio::Worker::PollerBehavior::Autoscaling.new, activity_task_poller_behavior: Temporalio::Worker::PollerBehavior::Autoscaling.new, nexus_task_poller_behavior: Temporalio::Worker::PollerBehavior::Autoscaling.new,) ### Cache options (Java SDK)[​](https://docs.temporal.io/develop/worker-performance#cache-options "Direct link to Cache options (Java SDK)") A Workflow Cache is created and shared between all Workers on a single host. It's designed to limit the resources used by the cache for each host/process. These options are defined on `WorkerFactoryOptions`: * `WorkerFactoryOptions#workflowCacheSize` defines the maximum number of cached Workflow Executions. Each cached Workflow contains at least one Workflow thread and its resources (memory, etc.). * `maxWorkflowThreadCount` defines the maximum number of Workflow threads that may exist concurrently at any time. These cache options limit the resource consumption of the in-memory Workflow cache. Workflow cache options are shared between all Workers because the Workflow cache is tightly integrated with the resource consumption of the entire host. This includes memory and the total thread count, which should be limited per host/JVM. For Go, use [`SetStickyWorkflowCacheSize`](https://pkg.go.dev/go.temporal.io/sdk/worker#SetStickyWorkflowCacheSize) . For Python, use the `max_cached_workflows` Worker option. ### "Large value" drawbacks[​](https://docs.temporal.io/develop/worker-performance#large-value-drawbacks "Direct link to "Large value" drawbacks") There are drawbacks when you use "large values everywhere." As with any multithreading system, specifying excessively large values without monitoring with the SDK and system metrics leads to constant resource contention/stealing This decreases the total throughput and increases latency jitter of the system. ### Invariants (JavaSDK only)[​](https://docs.temporal.io/develop/worker-performance#invariants "Direct link to Invariants (JavaSDK only)") These properties should always be true for a Worker's configuration. Perform this sanity check after the adjustments to Worker settings. 1. `workflowCacheSize` should be ≤ `maxWorkflowThreadCount`. Each Workflow has at least one Workflow thread. 2. `maxConcurrentWorkflowTaskExecutionSize` should be ≤ `maxWorkflowThreadCount`. Having more Worker slots than the Workflow cache size will lead to resource contention/stealing between executors and unpredictable delays. It's recommended that `maxWorkflowThreadCount` be at least 2x of `maxConcurrentWorkflowTaskExecutionSize`. 3. `maxConcurrentWorkflowTaskPollers` should be significantly ≤ `maxConcurrentWorkflowTaskExecutionSize`. And `maxConcurrentActivityTaskPollers` should be significantly ≤ `maxConcurrentActivityExecutionSize`. The number of pollers should always be lower than the number of executors. Worker runtime performance tuning[​](https://docs.temporal.io/develop/worker-performance#worker-performance-tuning "Direct link to Worker runtime performance tuning") ----------------------------------------------------------------------------------------------------------------------------------------------------------------------- Worker tuning manages the assignment of slot suppliers. A **Worker Tuner** instance exists per-Worker, providing slot suppliers for different slot types (Activity, Workflow, Nexus, or Local Activity Tasks). A tuner assigns different suppliers to each slot type. For example, it might provide a fixed assignment slot supplier for Workflows and use a resource-based supplier for Activities. ### Choosing slot supplier types[​](https://docs.temporal.io/develop/worker-performance#choosing-slot-supplier-types "Direct link to Choosing slot supplier types") Temporal offers three types of slot suppliers: fixed assignment, resource-based, and custom. Here’s how to choose the best approach based on your system requirements and workload characteristics. When choosing whether to opt for fixed assignment or resource-based suppliers, consider: * Workflow Tasks make minimal demands on the CPU and, normally, do not consume much memory. They are well-served by fixed-sized slot suppliers. * When very low Task completion latency & maximum throughput is important, avoid resourced-based auto-tuning slot suppliers. * Reserve auto-tuned resource-based slot suppliers for deployments with workloads that have resource usage patterns you don't fully understand, and don't care to. It can offer **good-enough** performance for workloads you don't want to spend time profiling. Scenarios with tasks that have variable, or very high, per-task resource needs should rely on fixed-size suppliers and manual tuning. The resource-based tuner can never perform as well as a fixed-size tuner with appropriately chosen configuration, but can offer reasonable performance without the need for profiling. The following use cases are well suited to resource-based auto-tuning slot suppliers: * **You want acceptable performance with minimum effort**: Resource-based suppliers can provide reasonable performance without the need for profiling your workfload. They can be a great way to get started with new workloads. If a workload becomes very performance-sensitive, we suggest you profile it and choose appropriate fixed-size numbers. * **Fluctuating workloads with low per-Task consumption**: The resource-based supplier works well when each Task consumes few resources but may run for a (relatively) long time. For example: HTTP calls or other blocking I/Os that spend most of their time waiting on external events. * **Protection from out-of-memory & over-subscription in the face of unpredictable per-task consumption:** Do your Tasks often consume an unpredictable number of resources? Do you want to avoid crashes without setting an overly-conservative fixed limit? In these cases, the resource-based supplier is a good match. Keep in mind that auto-tuning can never do a _perfect_ job and may sometimes exceed your requested system limits for CPU and memory. For the highest level of control over slot allocation, consider custom slot suppliers. This allows you to tailor the logic of how slots are allocated based on your system requirements. Custom suppliers provide flexibility to optimize for specific use cases that fixed assignment and resource-based suppliers may not fully address. Choosing the right slot supplier depends on your workload complexity and the control you need over resource allocation. For predictable tasks, variable workloads, or complex dynamic scenarios, Temporal slot suppliers can meet your needs. ### Implement Custom Slot Suppliers[​](https://docs.temporal.io/develop/worker-performance#custom-slot-implementation "Direct link to Implement Custom Slot Suppliers") Implement your own Slot Supplier to control how Workers are allocated Tasks and manage the processing of Workflows, Activities, and Nexus Operations. Custom Slot Suppliers let you fine-tune task processing based on your application's needs. Each SDK's reference documentation explains the specifics of the interface, but the core concepts are consistent across SDKs: | Language | Slot Supplier Reference | | --- | --- | | ![](https://docs.temporal.io/img/sdks/png/golang.png) | [`SlotSupplier`](https://pkg.go.dev/go.temporal.io/sdk/worker#SlotSupplier) | | ![](https://docs.temporal.io/img/sdks/png/java.png) | [`SlotSupplier`](https://www.javadoc.io/doc/io.temporal/temporal-sdk/latest/io/temporal/worker/tuning/SlotSupplier.html) | | ![](https://docs.temporal.io/img/sdks/png/python.png) | [`CustomSlotSupplier`](https://python.temporal.io/temporalio.worker.CustomSlotSupplier.html) | | ![](https://docs.temporal.io/img/sdks/png/typescript.png) | [`CustomSlotSupplier`](https://typescript.temporal.io/api/interfaces/worker.CustomSlotSupplier) | | ![](https://docs.temporal.io/img/sdks/png/dotnet.png) | [`CustomSlotSupplier`](https://dotnet.temporal.io/api/Temporalio.Worker.Tuning.CustomSlotSupplier.html) | Slot Suppliers issue `SlotPermit`s. These represent the right to use a slot of a specific type, namely Workflow, Activity, Local Activity, or Nexus. You control whether a Worker can perform certain tasks by issuing or withholding permits. Custom Slot Suppliers must implement these functions: * `reserveSlot` - Called before polling for new tasks. Your implementation can block and must return a Slot Permit once it decides to accept new work. * `tryReserveSlot` - Called for slot reservations in cases like eager activity processing. This must not block. * `markSlotUsed` - Called when a slot is about to be used for a task (not while it’s held during polling). It provides information about the task. * `releaseSlot` - Called when a slot is no longer needed, whether or not it was used. Custom policies require more effort, but provide finer control over Task processing. By implementing your own Slot Supplier, you can tailor how Workflows, Activities, and Nexus Operations are handled, optimizing performance for your specific needs. ### Slot supplier throttles[​](https://docs.temporal.io/develop/worker-performance#slot-supplier-throttles "Direct link to Slot supplier throttles") Auto-tuned suppliers may diverge from requested thresholds. The resources a given Task will use can't be known ahead of time. There is a fundamental tradeoff between how quickly a slot supplier is willing to accept Tasks and how well it can respect the defined thresholds. Slot throttling is a mechanism to control the rate at which new slots for concurrent tasks are made available for processing. This concept is part of the resource-based auto-tuning feature for Workers. By waiting a brief period between making slots available, the Worker can assess how resource usage has changed since the last task began processing. This throttle is called `rampThrottle` in the SDK options for resource-based slot suppliers. It defines the minimum time the Worker will wait between handing out new slots after passing the minimum slots number. **A higher `rampThrottle` trades off performance for safety.** For example: If a just-started worker were to have no throttle, and there was a backlog of Tasks, it might immediately accept 100 Tasks at once. If each Task allocated 1GB of RAM, the Worker would likely run out of memory and crash. The throttle enforces a wait before handing out new slots (after a minimum number of slots have been occupied) so you can measure newly consumed resources. Performance tuning examples[​](https://docs.temporal.io/develop/worker-performance#examples "Direct link to Performance tuning examples") ------------------------------------------------------------------------------------------------------------------------------------------ The following examples show how to create and provision composite Worker tuners and set other performance related options. Each tuner provides slot suppliers for various Task types. These examples focus on Activities and Local Activities, since Workflow Tasks normally do not need resource-based tuning. ### Go SDK[​](https://docs.temporal.io/develop/worker-performance#go-sdk "Direct link to Go SDK") **Resource-based tuner:** [features/snippets/worker\_tuner/worker\_tuner.go](https://github.com/temporalio/features/blob/main/features/snippets/worker_tuner/worker_tuner.go) func resourceBasedTuner() (worker.Options, error) { tuner, err := worker.NewResourceBasedTuner(worker.ResourceBasedTunerOptions{ TargetMem: 0.8, TargetCpu: 0.9, InfoSupplier: sysinfo.SysInfoProvider(), }) if err != nil { return worker.Options{}, err } return worker.Options{ Tuner: tuner, }, nil} **Composite tuner:** A composite tuner lets you mix different slot supplier strategies for each Task type. For example, you can use fixed-size slot suppliers for Workflow and Nexus Tasks while using resource-based slot suppliers for Activity and Local Activity Tasks. [features/snippets/worker\_tuner/worker\_tuner.go](https://github.com/temporalio/features/blob/main/features/snippets/worker_tuner/worker_tuner.go) func compositeTuner() (worker.Options, error) { options := worker.DefaultResourceControllerOptions() options.MemTargetPercent = 0.8 options.CpuTargetPercent = 0.9 options.InfoSupplier = sysinfo.SysInfoProvider() controller := worker.NewResourceController(options) wfSS, err := worker.NewFixedSizeSlotSupplier(10) if err != nil { return worker.Options{}, err } actSS, err := worker.NewResourceBasedSlotSupplier(controller, worker.DefaultActivityResourceBasedSlotSupplierOptions()) if err != nil { return worker.Options{}, err } laSS, err := worker.NewResourceBasedSlotSupplier(controller, worker.DefaultActivityResourceBasedSlotSupplierOptions()) if err != nil { return worker.Options{}, err } nexusSS, err := worker.NewFixedSizeSlotSupplier(10) if err != nil { return worker.Options{}, err } compositeTuner, err := worker.NewCompositeTuner(worker.CompositeTunerOptions{ WorkflowSlotSupplier: wfSS, ActivitySlotSupplier: actSS, LocalActivitySlotSupplier: laSS, NexusSlotSupplier: nexusSS, }) if err != nil { return worker.Options{}, err } return worker.Options{ Tuner: compositeTuner, }, nil} ### Java SDK[​](https://docs.temporal.io/develop/worker-performance#java-sdk "Direct link to Java SDK") // Just resource basedWorkerOptions.newBuilder() .setWorkerTuner( ResourceBasedTuner.newBuilder() .setControllerOptions( ResourceBasedControllerOptions.newBuilder(0.8, 0.9).build()) .build()) .build())// Combining different typesSlotSupplier workflowTaskSlotSupplier = new FixedSizeSlotSupplier<>(10);SlotSupplier activityTaskSlotSupplier = ResourceBasedSlotSupplier.createForActivity( resourceController, ResourceBasedTuner.DEFAULT_ACTIVITY_SLOT_OPTIONS);SlotSupplier localActivitySlotSupplier = ResourceBasedSlotSupplier.createForLocalActivity( resourceController, ResourceBasedTuner.DEFAULT_ACTIVITY_SLOT_OPTIONS);SlotSupplier nexusSlotSupplier = new FixedSizeSlotSupplier<>(10);WorkerOptions.newBuilder() .setWorkerTuner( new CompositeTuner( workflowTaskSlotSupplier, activityTaskSlotSupplier, localActivitySlotSupplier, nexusSlotSupplier)) .build(); ### TypeScript SDK[​](https://docs.temporal.io/develop/worker-performance#typescript-sdk "Direct link to TypeScript SDK") // Just resource basedconst resourceBasedTunerOptions: ResourceBasedTunerOptions = { targetMemoryUsage: 0.8, targetCpuUsage: 0.9,};const workerOptions = { tuner: { tunerOptions: resourceBasedTunerOptions, },};// Combining different typesconst resourceBasedTunerOptions: ResourceBasedTunerOptions = { targetMemoryUsage: 0.8, targetCpuUsage: 0.9,};const workerOptions = { tuner: { activityTaskSlotSupplier: { type: 'resource-based', tunerOptions: resourceBasedTunerOptions, }, workflowTaskSlotSupplier: { type: 'fixed-size', numSlots: 10, }, localActivityTaskSlotSupplier: { type: 'resource-based', tunerOptions: resourceBasedTunerOptions, }, },}; ### Python SDK[​](https://docs.temporal.io/develop/worker-performance#python-sdk "Direct link to Python SDK") # Just a resource based tuner, with poller autoscalingtuner = WorkerTuner.create_resource_based( target_memory_usage=0.5, target_cpu_usage=0.5,)worker = Worker( client, task_queue="foo", tuner=tuner, workflow_task_poller_behavior=PollerBehaviorAutoscaling(), activity_task_poller_behavior=PollerBehaviorAutoscaling())# Combining different types, with poller autoscalingresource_based_options = ResourceBasedTunerConfig(0.8, 0.9)tuner = WorkerTuner.create_composite( workflow_supplier=FixedSizeSlotSupplier(10), activity_supplier=ResourceBasedSlotSupplier( ResourceBasedSlotConfig(), resource_based_options, ), local_activity_supplier=ResourceBasedSlotSupplier( ResourceBasedSlotConfig(), resource_based_options, ),)worker = Worker( client, task_queue="foo", tuner=tuner, workflow_task_poller_behavior=PollerBehaviorAutoscaling(), activity_task_poller_behavior=PollerBehaviorAutoscaling()) ### .NET C# SDK[​](https://docs.temporal.io/develop/worker-performance#net-c-sdk "Direct link to .NET C# SDK") // Just resource basedvar worker = new TemporalWorker( Client, new TemporalWorkerOptions("my-task-queue") { Tuner = WorkerTuner.CreateResourceBased(0.8, 0.9), });// Combining different typesvar resourceTunerOptions = new ResourceBasedTunerOptions(0.8, 0.9);var worker = new TemporalWorker( Client, new TemporalWorkerOptions("my-task-queue") { Tuner = new WorkerTuner( new FixedSizeSlotSupplier(10), new ResourceBasedSlotSupplier( new ResourceBasedSlotSupplierOptions(), resourceTunerOptions), new ResourceBasedSlotSupplier( new ResourceBasedSlotSupplierOptions(), resourceTunerOptions)), }); Workflow Cache Tuning[​](https://docs.temporal.io/develop/worker-performance#workflow-cache-tuning "Direct link to Workflow Cache Tuning") ------------------------------------------------------------------------------------------------------------------------------------------- When the number of cached Workflow Executions reported by `sticky_cache_size` hits `workflowCacheSize` _or_ the number of threads reported by the `workflow_active_thread_count` metrics gauge hits `maxWorkflowThreadCount`, Workflow Executions will start to be evicted from the cache. An evicted Workflow Execution will need to be replayed when it gets any action that may advance it. If the Workflow Cache limits described above are hit, and Worker hosts have enough free RAM and are not close to reasonable thread limits, then you may choose to increase `workflowCacheSize` and `maxWorkflowThreadCount` limits to decrease the overall latency and cost of the Replays in the system. If the opposite occurs, consider decreasing the limits. note In CoreSDK based SDKs, like TypeScript, this metric works differently and should be monitored and adjusted on a per Worker and Task Queue basis. The `maxWorkflowThreadCount` and `workflow_active_thread_count` parameters are for the Java SDK only. Available Task Queue information[​](https://docs.temporal.io/develop/worker-performance#task-queue-metrics "Direct link to Available Task Queue information") -------------------------------------------------------------------------------------------------------------------------------------------------------------- Support, stability, and dependency info The information listed in this section is readable using the `DescribeTaskQueueEnhanced` method in the [Go SDK](https://github.com/temporalio/sdk-go/blob/74320648ab0e4178b1fedde01672f9b5b9f6c898/client/client.go) , with the [Temporal CLI](https://github.com/temporalio/cli/releases/tag/v1.1.0) `task-queue describe` command, and using `DescribeTaskQueue` through RPC. The Temporal Service reports information separately for each Task Queue type (not aggregated). Use the following Task Queue properties to retrieve and evaluate information about Task Queue health and performance. Available data include: * [`ApproximateBacklogCount`](https://docs.temporal.io/develop/worker-performance#ApproximateBacklogCountAndAge) and [`ApproximateBacklogAge`](https://docs.temporal.io/develop/worker-performance#ApproximateBacklogCountAndAge) * [`TasksAddRate`](https://docs.temporal.io/develop/worker-performance#TasksAddRate-and-TasksDispatchRate) and [`TasksDispatchRate`](https://docs.temporal.io/develop/worker-performance#TasksAddRate-and-TasksDispatchRate) * [`BacklogIncreaseRate`](https://docs.temporal.io/develop/worker-performance#BacklogIncreaseRate) (derived from [`TasksAddRate`](https://docs.temporal.io/develop/worker-performance#TasksAddRate-and-TasksDispatchRate) and [`TasksDispatchRate`](https://docs.temporal.io/develop/worker-performance#TasksAddRate-and-TasksDispatchRate) ) ### `ApproximateBacklogCount` and `ApproximateBacklogAge`[​](https://docs.temporal.io/develop/worker-performance#ApproximateBacklogCountAndAge "Direct link to ApproximateBacklogCountAndAge") `ApproximateBacklogCount` represents the approximate count of Tasks currently backlogged in this Task Queue. The number may include expired Tasks as well as active Tasks, but it will eventually converge to the correct count over time. `ApproximateBacklogAge` returns the approximate age of the oldest Task in the backlog. The age is based on the creation time of the Task at the head of the queue. You can rely on both these counts when making scaling decisions. Please note: [Sticky queues](https://docs.temporal.io/sticky-execution) will affect these values, but only for a few seconds. That's because Tasks sent to Sticky queues are not included in the returned values for `ApproximateBacklogCount` and `ApproximateBacklogAge`. Inaccuracies diminish as the backlog grows. ### `TasksAddRate` and `TasksDispatchRate`[​](https://docs.temporal.io/develop/worker-performance#TasksAddRate-and-TasksDispatchRate "Direct link to TasksAddRate-and-TasksDispatchRate") Reports the approximate Tasks-per-second added to or dispatched from a Task Queue. This rate is averaged over the most recent 30-second time interval. The calculations include Tasks that were added to or dispatched from the backlog as well as Tasks that were immediately dispatched and bypassed the backlog (sync-matched). The actual Task delivery count may be significantly higher than the number reported by these two values: * Eager dispatch refers to a Temporal feature where Activities can be requested by an SDK using one Workflow Task completion response. Tasks using Eager dispatch do not pass through Task Queues. * Tasks passed to Sticky Task Queues not included in the returned values for `TasksAddRate` and `TasksDispatchRate`. ### `BacklogIncreaseRate`[​](https://docs.temporal.io/develop/worker-performance#BacklogIncreaseRate "Direct link to BacklogIncreaseRate") Approximates the _net_ Tasks per second added to the backlog, averaged over the most recent 30 seconds. This is calculated as: TasksAddRate - TasksDispatchRate * Positive values of `X` indicate the backlog is growing by about `X` Tasks per second. * Negative values of `X` indicate the backlog is shrinking by about `X` Tasks per second. While individual `add` and `dispatch` rates may be inaccurate due to Eager and Sticky Task Queues, the `BacklogIncreaseRate` reliably reflects the rate at which the backlog is shrinking or growing for backlogs older than a few seconds. Evaluate Task Queue performance[​](https://docs.temporal.io/develop/worker-performance#evaluate-worker-loads "Direct link to Evaluate Task Queue performance") --------------------------------------------------------------------------------------------------------------------------------------------------------------- A [Task Queue](https://docs.temporal.io/task-queue) is a lightweight, dynamically allocated queue. [Worker Entities](https://docs.temporal.io/workers#worker-entity) poll the queue for [Tasks](https://docs.temporal.io/tasks#task) and retrieve Tasks to work on. Tasks are contexts that a Worker progresses using a specific Workflow Execution, Activity Execution, or a Nexus Task Execution. Each Task Queue type offers its Tasks to compatible Workers for Task completion. The Temporal Service dynamically creates different [Task Queue types](https://docs.temporal.io/task-queue) including Activity Task Queues, Workflow Task Queues, and Nexus Task Queues. With an accurate estimate of backlog Tasks, you can determine the optimal number of Workers to deploy. Balance your Worker count with the number of Tasks to achieve the best performance. This approach minimizes Task backlog saturation and reduces idle Workers. Task Queue data provide numerical insights into your Task Queue activity and backlog characteristics. Use these numbers to tune your production deployments. Evaluate your Worker loads and assess whether you need to scale up or reduce your Worker deployment. RATE LIMITS [Visibility API rate limits](https://docs.temporal.io/cloud/limits#visibility-api-rate-limit) apply to Task Queue performance data requests. ### Query Task Queue info with Temporal CLI[​](https://docs.temporal.io/develop/worker-performance#cli-task-queue-info "Direct link to Query Task Queue info with Temporal CLI") The Temporal CLI helps you monitor and evaluate Worker performance. Issue the following command to display a list of active Workers that have recently polled a Task Queue: temporal task-queue describe \ --task-queue YourTaskQueueName \ [additional options] This command retrieves poller information, backlog statistics, and task reachability for Task types (available in Temporal Server v1.25.0, Temporal CLI 1.1 and later). danger Task reachability status is experimental. Determining Task reachability incurs a non-trivial computing cost. This feature may significantly change or be removed in a future release. ### Query Task Queue info with the Go SDK[​](https://docs.temporal.io/develop/worker-performance#go-sdk-task-queue-info "Direct link to Query Task Queue info with the Go SDK") Retrieve Task Queue data using the Go SDK by calling `DescribeTaskQueueEnhanced`. Specify the Task Queue name and set `ReportStats` to `true`, as in the following example: for _, taskQueueName := range taskQueueNames { resp, err := s.client.DescribeTaskQueueEnhanced(ctx, client.DescribeTaskQueueEnhancedOptions{ TaskQueue: taskQueueName, ReportStats: true, }) if err != nil { log.Printf("Error describing task queue %s: %v", taskQueueName, err) } // Get the backlog count from the enhanced response backlogCount += getBacklogCount(resp) } ### Evaluate Worker availability and capacity issues[​](https://docs.temporal.io/develop/worker-performance#worker-capacity-issues "Direct link to Evaluate Worker availability and capacity issues") Each Temporal [Server](https://docs.temporal.io/temporal-service/temporal-server) records the last time of each poll request. This time is displayed in the `temporal task-queue describe` output. * A `LastAccessTime` value exceeding one minute may indicate that the Worker fleet is at capacity or that Workers have shut down or been removed. * Values under 5 minutes typically suggest the Worker fleet is at capacity. "At capacity" means that all Workflow and Activity slots are full. * Values over 5 minutes since the last poll request usually suggest that Workers have shut down or been removed. Workers are removed if 5 minutes have passed since the last poll request. ### Manage your Worker fleet[​](https://docs.temporal.io/develop/worker-performance#manage-your-worker-fleet "Direct link to Manage your Worker fleet") You can adjust the number of Workers to enhance Workflow Execution performance and manage your fleet size. For instance, a large backlog of Tasks with too few Workers will slow down Workflow Execution completions and decrease processing efficiency. Adding more Workers boosts speeds up completion rates and improves throughput. An empty backlog indicates low Worker utilization, allowing you to reduce your fleet and associated costs. The values provided by `temporal task-queue describe` can help you manage your Worker fleet deployment: * `ApproximateBacklogAge` shows how long Tasks have been waiting to be dispatched. If this time grows too long, more Workers can boost Workflow efficiency. * Calculate the demand per Worker by dividing the number of backlogged Tasks (`ApproximateBacklogCount`) by the number of Workers. Determine if your task processing rate is within an acceptable range for your needs using the per-Worker demand (how many Tasks each Worker has yet to process), the backlog consumption rate (`TasksDispatchRate`, the rate at which Workers are processing Tasks), and the dispatch latency (`ApproximateBacklogAge`, the time the oldest Task has been waiting to be assigned to a Worker). * The backlog increase rate (`BacklogIncreaseRate`) shows the changing demand on your Workers over time. As this rate increases, you may need to add more Workers until demand and capacity are balanced. As it decreases, you may be able to reduce your Worker fleet. Task Queue processing tuning[​](https://docs.temporal.io/develop/worker-performance#task-queues-processing-tuning "Direct link to Task Queue processing tuning") ----------------------------------------------------------------------------------------------------------------------------------------------------------------- The following steps limit delays in Task Queue processing due to insufficient or unbalanced Workers. Review these steps if you notice high `schedule_to_start` metrics. The steps are arranged in the recommended order of execution. ### Hosts and Resources provisioning[​](https://docs.temporal.io/develop/worker-performance#hosts-and-resources-provisioning "Direct link to Hosts and Resources provisioning") If currently provisioned Worker hosts are fully utilized (near full CPU usage, high load average, etc), additional Workers hosts have to be provisioned to increase the capacity of the Workers pool. **It's possible to have too many Workers** Monitor the poll success (`poll_success`/`poll_success_sync`) and poll timeout `poll_timeouts` Server metric counters. Poll Success Rate = (`poll_success` + `poll_success_sync`) / (`poll_success` + `poll_success_sync` + `poll_timeouts`) Poll Success Rate should be >90% in most cases of systems with a steady load. For high volume and low latency, try to target >95%. If you see 1. low Poll Success Rate, and 2. low `schedule_to_start_latency`, and 3. low Worker hosts resource utilization at the same time, then you might have too many workers, consider sizing down. ### Worker Executor Slots sizing[​](https://docs.temporal.io/develop/worker-performance#worker-executor-slots-sizing "Direct link to Worker Executor Slots sizing") The main area to focus on when tuning is the number of Worker Executor Slots. Increase the maximum number of working slots by adjusting `maxConcurrentWorkflowTaskExecutionSize` or `maxConcurrentActivityExecutionSize` if both of the following conditions are met: 1. The Worker hosts are underutilized (no bottlenecks on CPU, load average, etc.). 2. The `worker_task_slots_available` metric from the corresponding Worker type frequently shows a depleted number of available Worker slots. Alternatively, consider using a resource-based slot supplier as described [here](https://docs.temporal.io/develop/worker-performance#slot-suppliers) . ### Poller count[​](https://docs.temporal.io/develop/worker-performance#poller-count "Direct link to Poller count") Sometimes, it can be appropriate to increase the number of task pollers. This is usually more common in situations where your Workers have somewhat high latency when communicating with the server. You can simply use automated poller tuning to handle this automatically. Consider manual adjustment if: 1. The Worker hosts are underutilized, for example, there are no bottlenecks on CPU, load average, etc. 2. `worker_task_slots_available` metric from the corresponding Worker type shows that a significant percentage of Worker slots are available on a regular basis. 3. The `schedule_to_start` metric is abnormally long. Then consider increasing the number of pollers by adjusting `maxConcurrentWorkflowTaskPollers` or `maxConcurrentActivityTaskPollers`, depending on which type of `schedule_to_start` metric is elevated. ### Rate Limiting[​](https://docs.temporal.io/develop/worker-performance#rate-limiting "Direct link to Rate Limiting") If, after adjusting the poller and executors count as specified earlier, you still observe an elevated `schedule_to_start`, underutilized Worker hosts, or high `worker_task_slots_available`, you might want to check the following: * If server-side rate limiting per Task Queue is set by `WorkerOptions#maxTaskQueueActivitiesPerSecond`, remove the limit or adjust the value up. (See [Go](https://docs.temporal.io/develop/go/workers/run-worker-process#taskqueueactivitiespersecond) and [Java](https://www.javadoc.io/doc/io.temporal/temporal-sdk/latest/io/temporal/worker/WorkerOptions.Builder.html) .) * If Worker-side rate limiting per Worker is set by `WorkerOptions#maxWorkerActivitiesPerSecond`, remove the limit. (See [Go](https://docs.temporal.io/develop/go/workers/run-worker-process#workeractivitiespersecond) , [TypeScript](https://typescript.temporal.io/api/interfaces/worker.WorkerOptions#maxconcurrentactivitytaskexecutions) , and [Java](https://www.javadoc.io/doc/io.temporal/temporal-sdk/latest/io/temporal/worker/WorkerOptions.Builder.html) .) Related reading[​](https://docs.temporal.io/develop/worker-performance#related-reading "Direct link to Related reading") ------------------------------------------------------------------------------------------------------------------------- * [Worker tuning quick reference](https://docs.temporal.io/develop/worker-tuning-reference) - SDK defaults and metrics by resource type * [Workers in production operation guide](https://temporal.io/blog/workers-in-production) * [Full set of SDK Metrics reference](https://docs.temporal.io/references/sdk-metrics) * [Worker performance concepts](https://docs.temporal.io/develop/worker-performance#worker-performance-concepts) * [Task slots](https://docs.temporal.io/develop/worker-performance#slots) * [Slot suppliers](https://docs.temporal.io/develop/worker-performance#slot-suppliers) * [Worker tuning](https://docs.temporal.io/develop/worker-performance#worker-tuning) * [Task Pollers](https://docs.temporal.io/develop/worker-performance#task-pollers) * [Eager task execution](https://docs.temporal.io/develop/worker-performance#eager-task-execution) * [Performance metrics for tuning](https://docs.temporal.io/develop/worker-performance#metrics) * [Slot availability metrics](https://docs.temporal.io/develop/worker-performance#slot-availability-metrics) * [Latency metrics](https://docs.temporal.io/develop/worker-performance#latency-metrics) * [Cache metrics](https://docs.temporal.io/develop/worker-performance#cache-metrics) * [Worker performance options](https://docs.temporal.io/develop/worker-performance#configuration) * [Executor slot options](https://docs.temporal.io/develop/worker-performance#executor-slot-options) * [Configuring Poller Options](https://docs.temporal.io/develop/worker-performance#configuring-poller-options) * [Cache options (Java SDK)](https://docs.temporal.io/develop/worker-performance#cache-options) * ["Large value" drawbacks](https://docs.temporal.io/develop/worker-performance#large-value-drawbacks) * [Invariants (JavaSDK only)](https://docs.temporal.io/develop/worker-performance#invariants) * [Worker runtime performance tuning](https://docs.temporal.io/develop/worker-performance#worker-performance-tuning) * [Choosing slot supplier types](https://docs.temporal.io/develop/worker-performance#choosing-slot-supplier-types) * [Implement Custom Slot Suppliers](https://docs.temporal.io/develop/worker-performance#custom-slot-implementation) * [Slot supplier throttles](https://docs.temporal.io/develop/worker-performance#slot-supplier-throttles) * [Performance tuning examples](https://docs.temporal.io/develop/worker-performance#examples) * [Go SDK](https://docs.temporal.io/develop/worker-performance#go-sdk) * [Java SDK](https://docs.temporal.io/develop/worker-performance#java-sdk) * [TypeScript SDK](https://docs.temporal.io/develop/worker-performance#typescript-sdk) * [Python SDK](https://docs.temporal.io/develop/worker-performance#python-sdk) * [.NET C# SDK](https://docs.temporal.io/develop/worker-performance#net-c-sdk) * [Workflow Cache Tuning](https://docs.temporal.io/develop/worker-performance#workflow-cache-tuning) * [Available Task Queue information](https://docs.temporal.io/develop/worker-performance#task-queue-metrics) * [`ApproximateBacklogCount` and `ApproximateBacklogAge`](https://docs.temporal.io/develop/worker-performance#ApproximateBacklogCountAndAge) * [`TasksAddRate` and `TasksDispatchRate`](https://docs.temporal.io/develop/worker-performance#TasksAddRate-and-TasksDispatchRate) * [`BacklogIncreaseRate`](https://docs.temporal.io/develop/worker-performance#BacklogIncreaseRate) * [Evaluate Task Queue performance](https://docs.temporal.io/develop/worker-performance#evaluate-worker-loads) * [Query Task Queue info with Temporal CLI](https://docs.temporal.io/develop/worker-performance#cli-task-queue-info) * [Query Task Queue info with the Go SDK](https://docs.temporal.io/develop/worker-performance#go-sdk-task-queue-info) * [Evaluate Worker availability and capacity issues](https://docs.temporal.io/develop/worker-performance#worker-capacity-issues) * [Manage your Worker fleet](https://docs.temporal.io/develop/worker-performance#manage-your-worker-fleet) * [Task Queue processing tuning](https://docs.temporal.io/develop/worker-performance#task-queues-processing-tuning) * [Hosts and Resources provisioning](https://docs.temporal.io/develop/worker-performance#hosts-and-resources-provisioning) * [Worker Executor Slots sizing](https://docs.temporal.io/develop/worker-performance#worker-executor-slots-sizing) * [Poller count](https://docs.temporal.io/develop/worker-performance#poller-count) * [Rate Limiting](https://docs.temporal.io/develop/worker-performance#rate-limiting) * [Related reading](https://docs.temporal.io/develop/worker-performance#related-reading) --- # Context Propagation | Temporal Platform Documentation [Skip to main content](https://docs.temporal.io/encyclopedia/context-propagation#__docusaurus_skipToContent_fallback) On this page Context propagation lets you pass custom key-value data from a Client to Workflows, and from Workflows to Activities and Child Workflows, without threading values through every function signature. Common use cases: * Propagating distributed tracing IDs (e.g., OpenTelemetry trace context) * Passing tenant IDs for multi-tenant applications * Forwarding auth tokens or request-scoped metadata Each SDK provides a **context propagator** interface you implement to control which values are injected and extracted. You register propagators on the Client, and the SDK calls them automatically at every boundary. Implementing Context Propagation[​](https://docs.temporal.io/encyclopedia/context-propagation#implementing-context-propagation "Direct link to Implementing Context Propagation") ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Here are SDK-specific guides: * [Go](https://docs.temporal.io/develop/go/best-practices/context-propagation) * [Implementing Context Propagation](https://docs.temporal.io/encyclopedia/context-propagation#implementing-context-propagation) --- # Job Queue | Temporal Platform Documentation [Skip to main content](https://docs.temporal.io/evaluate/development-production-features/job-queue#__docusaurus_skipToContent_fallback) On this page What is a Job Queue?[​](https://docs.temporal.io/evaluate/development-production-features/job-queue#what-is-a-job-queue "Direct link to What is a Job Queue?") --------------------------------------------------------------------------------------------------------------------------------------------------------------- A job is a single, discrete unit of work that runs asynchronously in the background such as sending an email, processing a webhook, syncing data, or executing a single function reliably. A job queue is the system that manages these jobs: accepting work, dispatching it to workers, retrying on failure, and providing visibility into what's running and what failed. **Standalone Activities are Temporal's job queue.** They let you use Temporal Activities as background jobs, in addition to using the same Activities as steps inside a Workflow. You write an Activity once and can run it either as a background job or as part of a multi-step Workflow. Temporal provides stronger guarantees, better visibility, and more control than traditional job queues - while remaining cost-effective for high-volume use cases and offering a clean upgrade path to multi-step workflow orchestration. ### Overview[​](https://docs.temporal.io/evaluate/development-production-features/job-queue#overview "Direct link to Overview") Standalone Activities add the ability to execute any Temporal Activity as a top-level Activity Execution for durable job processing. #### Unified programming model & worker deployment[​](https://docs.temporal.io/evaluate/development-production-features/job-queue#unified-programming-model--worker-deployment "Direct link to Unified programming model & worker deployment") * Write an Activity once and use it anywhere - with a unified Activity programming model * Optional heartbeats support checkpointing for long-running jobs * Deploy to an Activity Worker once, and invoke standalone or from within a Workflow #### Execution lifecycle[​](https://docs.temporal.io/evaluate/development-production-features/job-queue#execution-lifecycle "Direct link to Execution lifecycle") * Jobs are submitted as Standalone Activity Executions * Each job is durably persisted with Temporal reliability, so jobs are not lost * Jobs are scheduled with priority, fairness, deduplication and no head-of-line blocking * Workers poll task queues and execute Activities (you run your own Workers) * Temporal ensures retries, timeouts, and exponential backoff policy is enforced #### Observability & lifecycle controls[​](https://docs.temporal.io/evaluate/development-production-features/job-queue#observability--lifecycle-controls "Direct link to Observability & lifecycle controls") * Full job visibility (list, search) with detailed execution state, retry count, errors & results * OpenMetrics support * Lifecycle controls: cancel, pause, unpause, reset, terminate * Manual completion for external integrations & on-call management Related 📚 * [![](https://docs.temporal.io/img/assets/link-preview-icon.svg)Standalone Activity concepts and getting started](https://docs.temporal.io/standalone-activity) encyclopedia * [![](https://docs.temporal.io/img/sdks/svgs/golang.svg)Go SDK - Standalone Activities quick start and code sample](https://docs.temporal.io/develop/go/activities/standalone-activities) feature-guide * [![](https://docs.temporal.io/img/sdks/svgs/python.svg)Python SDK - Standalone Activities quick start and code sample](https://docs.temporal.io/develop/python/activities/standalone-activities) feature-guide * [![](https://docs.temporal.io/img/sdks/svgs/dotnet.svg).NET SDK - Standalone Activities quick start and code sample](https://docs.temporal.io/develop/dotnet/activities/standalone-activities) feature-guide * [What is a Job Queue?](https://docs.temporal.io/evaluate/development-production-features/job-queue#what-is-a-job-queue) * [Overview](https://docs.temporal.io/evaluate/development-production-features/job-queue#overview) --- # Task Queues and naming best practices | Temporal Platform Documentation [Skip to main content](https://docs.temporal.io/task-queue/naming#__docusaurus_skipToContent_fallback) The Temporal Service maintains a set of Task Queues, which Workers poll to see what work needs to be done. Each Task Queue is identified by a name, which is provided to the Temporal Service when launching a Workflow Execution. * Python * Go * Java * Typescript * .NET **Excerpt of code used to start the Workflow in Python** client = await Client.connect("localhost:7233", namespace="default")# Execute a workflowresult = await client.execute_workflow( GreetingWorkflow.run, name, id="my-workflow", task_queue="my-task-queue-name",) **Excerpt of code used to configure the Worker in Python** worker = Worker( client, task_queue="my-task-queue-name", workflows=[GreetingWorkflow], activities=[activities.say_hello],) **Excerpt of code used to start the Workflow in Go** options := client.StartWorkflowOptions{ ID: "my-workflow", TaskQueue: "my-task-queue-name",}run, err := c.ExecuteWorkflow(ctx, options, ProcessOrderWorkflow, input) **Excerpt of code used to configure the Worker in Go** w := worker.New(c, "my-task-queue-name", worker.Options{}) **Excerpt of code used to start the Workflow in Java** WorkflowOptions options = WorkflowOptions.newBuilder() .setWorkflowId("my-workflow") .setTaskQueue("my-task-queue-name") .build();MyWorkflow workflow = client.newWorkflowStub(MyWorkflow.class, options); **Excerpt of code used to configure the Worker in Java** Worker worker = factory.newWorker("my-task-queue-name"); **Excerpt of code used to start the Workflow in TypeScript** await client.workflow.start(OrderProcessingWorkflow, { args: [order], taskQueue: 'my-task-queue', workflowId: `workflow-order-${order.id},`,}); **Excerpt of code used to configure the Worker in TypeScript** const worker = await Worker.create({ taskQueue: 'my-task-queue', connection, workflowsPath: require.resolve('./workflows'), activities,}); **Excerpt of code used to start the Workflow in C# and .NET** var options = new WorkflowOptions( id: "translation-workflow", taskQueue: "my-task-queue");// Run workflowvar result = await client.ExecuteWorkflowAsync( (TranslationWorkflow wf) => wf.RunAsync(input), options); **Excerpt of code used to configure the Worker in C# and .NET** using var worker = new TemporalWorker( client, new TemporalWorkerOptions("my-task-queue") .AddAllActivities(activities) .AddWorkflow()); Since Task Queues are created dynamically when they are first used, a mismatch between these two values does not result in an error. Instead, it will result in the creation of two different Task Queues. Consequently, the Worker will not receive any tasks from the Temporal Service and the Workflow Execution will not progress. Therefore, we recommend that you define the Task Queue name in a constant that is referenced by the Client and Worker if possible, as this will ensure that they always use the same value. * Python * Go * Java * Typescript * .NET **Excerpt of code used to define a constant with the Task Queue name in Python (in a shared.py file)** TASK_QUEUE_NAME = "my-task-queue-name" **Excerpt of code used to start the Workflow, referencing the constant defined with the Task Queue name in Python** from shared import TASK_QUEUE_NAME...client = await Client.connect("localhost:7233", namespace="default")# Execute a workflowresult = await client.execute_workflow( GreetingWorkflow.run, name, id="my-workflow", task_queue=TASK_QUEUE_NAME,) **Excerpt of code used to configure the Worker, referencing the constant defined with the Task Queue name in Python** worker = Worker( client, task_queue=TASK_QUEUE_NAME, workflows=[GreetingWorkflow], activities=[activities.say_hello],) **Excerpt of code used to define a constant with the Task Queue name in Go** package app const TaskQueueName = "my-taskqueue-name" **Excerpt of code used to start the Workflow, referencing the constant defined with the Task Queue name in Go** options := client.StartWorkflowOptions{ ID: "my-workflow", TaskQueue: app.TaskQueueName,}run, err := c.ExecuteWorkflow(ctx, options, ProcessOrderWorkflow, input) **Excerpt of code used to configure the Worker, referencing the constant defined with the Task Queue name in Go** w := worker.New(c, app.TaskQueueName, worker.Options{}) **Excerpt of code used to define a constant with the Task Queue name in Java** package app;public class Constants { public static final String taskQueueName = "my-task-queue-name";} **Excerpt of code used to start the Workflow, referencing the constant defined with the Task Queue name in Java** WorkflowOptions options = WorkflowOptions.newBuilder() .setWorkflowId("my-workflow") .setTaskQueue(Constants.taskQueueName) .build();MyWorkflow workflow = client.newWorkflowStub(MyWorkflow.class, options); **Excerpt of code used to configure the Worker, referencing the constant defined with the Task Queue name in Java** Worker worker = factory.newWorker(Constants.taskQueueName); **Excerpt of code used to define a constant with the Task Queue name in TypeScript** const TASK_QUEUE_NAME = 'my-taskqueue-name'; **Excerpt of code used to start the Workflow, referencing the constant defined with the Task Queue name in TypeScript** import { TASK_QUEUE_NAME } from './shared';// additional code would followawait client.workflow.start(OrderProcessingWorkflow, { args: [order], taskQueue: TASK_QUEUE_NAME, workflowId: `workflow-order-${order.id},`,}); **Excerpt of code used to configure the Worker, referencing the constant defined with the Task Queue name in TypeScript** import { TASK_QUEUE_NAME } from './shared';// additional code would followconst worker = await Worker.create({ taskQueue: TASK_QUEUE_NAME, connection, workflowsPath: require.resolve('./workflows'), activities,}); **Excerpt of code used to define a constant with the Task Queue name in C# and .NET** public static class WorkflowConstants{ public const string TaskQueueName = "translation-tasks";} **Excerpt of code used to start the Workflow, referencing the constant defined with the Task Queue name in C# and .NET** var options = new WorkflowOptions( id: "translation-workflow", taskQueue: WorkflowConstants.TaskQueueName);// Run workflowvar result = await client.ExecuteWorkflowAsync( (TranslationWorkflow wf) => wf.RunAsync(input), options); **Excerpt of code used to configure the Worker, referencing the constant defined with the Task Queue name in C# and .NET** using var worker = new TemporalWorker( client, new TemporalWorkerOptions(WorkflowConstants.TaskQueueName) .AddAllActivities(activities) .AddWorkflow()); However, it’s not always possible to do define the Task Queue name in a constant, such as when the Client used to start the Workflow is running on another system or is implemented in a different programming language. --- # Worker processes - Ruby SDK | Temporal Platform Documentation [Skip to main content](https://docs.temporal.io/develop/ruby/workers/run-worker-process#__docusaurus_skipToContent_fallback) On this page Run Worker Process[​](https://docs.temporal.io/develop/ruby/workers/run-worker-process#run-worker-process "Direct link to Run Worker Process") ----------------------------------------------------------------------------------------------------------------------------------------------- The [Worker Process](https://docs.temporal.io/workers#worker-process) is where Workflow Functions and Activity Functions are actually executed. In a Temporal application deployment, you ship and scale as many Workers as you need to handle the load of your Workflows and Activities. * Each [Worker Entity](https://docs.temporal.io/workers#worker-entity) in the Worker Process must register the exact Workflow Types and Activity Types it may execute. * Each Worker Entity must also associate itself with exactly one [Task Queue](https://docs.temporal.io/task-queue) . * Each Worker Entity polling the same Task Queue must be registered with the same Workflow Types and Activity Types. A [Worker Entity](https://docs.temporal.io/workers#worker-entity) is the component within a Worker Process that listens to a specific Task Queue. A Worker Entity contains a Workflow Worker and/or an Activity Worker, which makes progress on Workflow Executions and Activity Executions, respectively. Workers are implemented in each Temporal SDK, and can be deployed with just a bit of boilerplate. To create a Worker, use `Temporalio::Worker.new()`, providing the Worker options which include Task Queue, Workflows, and Activities and more. The following code example creates a Worker that polls for tasks from the Task Queue and executes the Workflow. When a Worker is created, it accepts a list of Workflows, a list of Activities, or both. # Create a client to localhost on default namespaceclient = Temporalio::Client.connect('localhost:7233', 'default')# Create a worker with the client, activities, and workflowsworker = Temporalio::Worker.new( client:, task_queue: 'my-task-queue', workflows: [MyWorkflow], # This provides the activity instance which means it is reused for each attempt, but # just the class can be provided to instantiate for each attempt activities: [MyActivity.new])# Run the worker until SIGINT. There are other ways to wait for shutdown, or a block can# be provided that will shutdown when the block completesworker.run(shutdown_signals: ['SIGINT']) To run multiple workers, `Temporalio::Worker.run_all` may be used instead. All Workers listening to the same Task Queue name must be registered to handle the exact same Workflows Types and Activity Types. If a Worker polls a Task for a Workflow Type or Activity Type it does not know about, it fails that Task. However, the failure of the Task does not cause the associated Workflow Execution to fail. * [Run Worker Process](https://docs.temporal.io/develop/ruby/workers/run-worker-process#run-worker-process) --- # Handling Signals, Queries, & Updates | Temporal Platform Documentation [Skip to main content](https://docs.temporal.io/handling-messages#__docusaurus_skipToContent_fallback) On this page When Signals, Updates, and Queries arrive at your Workflow, the handlers for these messages will operate on the current state of your Workflow and can use the fields you have set. In this section, we’ll give you an overview of how messages work with Temporal and cover how to write correct and robust handlers by covering topics like atomicity, guaranteeing completion before the Workflow exits, exceptions, and idempotency. Handling Messages[​](https://docs.temporal.io/handling-messages#handling-messages "Direct link to Handling Messages") ---------------------------------------------------------------------------------------------------------------------- ### Message handler concurrency[​](https://docs.temporal.io/handling-messages#message-handler-concurrency "Direct link to Message handler concurrency") If your Workflow receives messages, you may need to consider how those messages interact with one another or with the main Workflow method. Behind the scenes, Temporal is running a loop that looks like this: ![Diagram that shows the execution ordering of Workflows](https://docs.temporal.io/img/info/messages-workflow-loop.png) Diagram that shows the execution ordering of Workflows Every time the Workflow wakes up--generally, it wakes up when it needs to--it will process messages in the order they were received, followed by making progress in the Workflow’s main method. This execution is on a single thread–while this means you don’t have to worry about parallelism, you do need to worry about concurrency if you have written Signal and Update handlers that can block. These can run interleaved with the main Workflow and with one another, resulting in potential race conditions. These methods should be made reentrant. #### Initializing the Workflow first[​](https://docs.temporal.io/handling-messages#workflow-initializers "Direct link to Initializing the Workflow first") Initialize your Workflow's state before handling messages. This prevents your handler from reading uninitialized instance variables. To see why, refer to the [diagram](https://docs.temporal.io/handling-messages#message-handler-concurrency) . It shows that your Workflow processes messages before the first run of your Workflow's main method. The message handler runs first in several scenarios, such as: * When using [Signal-with-Start](https://docs.temporal.io/sending-messages#signal-with-start) . * When your Worker experiences delays, such as when the Task Queue it polls gets backlogged. * When messages arrive immediately after a Workflow continues as new but before it resumes. For all languages except Go and TypeScript, use your constructor to set up state. Annotate your constructor as a Workflow Initializer and take the same arguments as your Workflow's main method. Note that you can't make blocking calls from your constructor. If you need to block, make your Signal or Update handler [wait](https://docs.temporal.io/handling-messages#waiting) for an initialization flag. In Go and TypeScript, register any message handlers only after completing initialization. ### Message handler patterns[​](https://docs.temporal.io/handling-messages#message-handler-patterns "Direct link to Message handler patterns") Here are several common patterns for write operations, Signal and Update handlers. They don't apply to pure read operations, i.e. Queries or [Update Validators](https://docs.temporal.io/handling-messages#update-validators) : * Returning immediately from a handler * Waiting for the Workflow to be ready to process them * Kicking off activities and other asynchronous tasks * Injecting work into the main Workflow * Finishing handlers before the Workflow completes * Ensuring your messages are processed exactly once #### Synchronous handlers[​](https://docs.temporal.io/handling-messages#synchronous-handlers "Direct link to Synchronous handlers") Synchronous handlers don’t kick off any long-running operations or otherwise block. They're guaranteed to run atomically. #### Waiting[​](https://docs.temporal.io/handling-messages#waiting "Direct link to Waiting") A Signal or Update handler can block waiting for the Workflow to reach a certain state using a Wait Condition. See the links below to find out how to use this with your SDK. #### Running asynchronous tasks[​](https://docs.temporal.io/handling-messages#running-asynchronous-tasks "Direct link to Running asynchronous tasks") Sometimes, you need your message handler to wait for long-running operations such as executing an Activity. When this happens, the handler will yield control back to [the loop](https://docs.temporal.io/handling-messages#message-handler-concurrency) . This means that your handlers can have race conditions if you’re not careful. You can guard your handlers with concurrency primitives like mutexes or semaphores, but you should use versions of these primitives provided for Workflows in most languages. See the links below for examples of how to use them in your SDK. #### Inject work into the main Workflow[​](https://docs.temporal.io/handling-messages#injecting-work-into-main-workflow "Direct link to Inject work into the main Workflow") Sometimes you want to process work provided by messages in the main Workflow. Perhaps you’d like to accumulate several messages before acting on any of them. For example, message handlers might put work into a queue, which can then be picked up and processed in an event loop that you yourself write. This option is considered advanced but offers powerful flexibility. And if you serialize the handling of your messages inside your main Workflow, you can avoid using concurrency primitives like mutexes and semaphores. See the links above for how to do this in your SDK. #### Finishing handlers before the Workflow completes[​](https://docs.temporal.io/handling-messages#finishing-message-handlers "Direct link to Finishing handlers before the Workflow completes") You should generally finish running all handlers before the Workflow run completes or continues as new. For some Workflows, this means you should explicitly check to make sure that all the handlers have completed before finishing. You can await a condition called All Handlers Finished at the end of your Workflow. If you don’t need to ensure that your handlers complete, you may specify your handler’s Handler Unfinished Policy as Abandon to turn off the warnings. However, note that clients waiting for Updates will get Not Found errors if they're waiting for Updates that never complete before the Workflow run completes. See the links below for how to ensure handlers are finished in your SDK. #### Message IDs and handling Continue-As-New[​](https://docs.temporal.io/handling-messages#exactly-once-message-processing "Direct link to Message IDs and handling Continue-As-New") Usually, you'll want your message handlers to run exactly once--to be idempotent--in cases where the same Signal or Update is delivered twice. For Updates, Temporal handles this for you on the server, by deduplicating according to the Update ID. The Update ID is set automatically to a UUID, but you can set it yourself. For Signals, you should use a custom idempotency key that you send as part of your own signal inputs, implementing the deduplication in your Workflow code. However, if you are using Updates with [Continue-As-New](https://docs.temporal.io/workflow-execution/continue-as-new) you should implement the deduplication in your Workflow code, since Update ID deduplication by the server is per Workflow run. info In addition to these application-level identifiers, both Signals and Updates automatically use request IDs to deduplicate retried client calls. You do not need to do anything to enable this. See the links below for examples of handling idempotency and Continue-As-New in your SDK. #### Authoring message handler patterns[​](https://docs.temporal.io/handling-messages#authoring-message-handler-patterns "Direct link to Authoring message handler patterns") See examples of the above patterns. Related 📚 * [![](https://docs.temporal.io/img/sdks/svgs/dotnet.svg)Author message handler patterns in .NET](https://docs.temporal.io/develop/dotnet/workflows/message-passing) feature-guide * [![](https://docs.temporal.io/img/sdks/svgs/golang.svg)Author message handler patterns in Go](https://docs.temporal.io/develop/go/workflows/message-passing#message-handler-patterns) feature-guide * [![](https://docs.temporal.io/img/sdks/svgs/java.svg)Author message handler patterns in Java](https://docs.temporal.io/develop/java/workflows/message-passing) feature-guide * [![](https://docs.temporal.io/img/sdks/svgs/php.svg)Author message handler patterns in PHP](https://docs.temporal.io/develop/php/workflows/message-passing) feature-guide * [![](https://docs.temporal.io/img/sdks/svgs/python.svg)Author message handler patterns in Python](https://docs.temporal.io/develop/python/workflows/message-passing#message-handler-patterns) feature-guide * [![](https://docs.temporal.io/img/sdks/svgs/typescript.svg)Author message handler patterns in TypeScript](https://docs.temporal.io/develop/typescript/workflows/message-passing#message-handler-patterns) feature-guide ### Update Validators[​](https://docs.temporal.io/handling-messages#update-validators "Direct link to Update Validators") When you define an Update handler, you may optionally define an Update Validator: a read operation that's responsible for accepting or rejecting the Update. You can use Validators to verify arguments or make sure the Workflow is ready to accept your Updates. * If it accepts, the Update will become part of your Workflow’s history and the client will be notified that the operation has been Accepted. The Update handler will then run until it returns a value. * If it rejects, the client will be informed that it was Rejected, and the Workflow will have no indication that it was ever requested, similar to a Query handler. note Like Queries, Validators are not allowed to block. Once the Update handler is finished and has returned a value, the operation is considered Completed. Related 📚 * [![](https://docs.temporal.io/img/sdks/svgs/golang.svg)Validate updates in Go](https://docs.temporal.io/develop/go/workflows/message-passing#updates) feature-guide * [![](https://docs.temporal.io/img/sdks/svgs/java.svg)Validate updates in Java](https://docs.temporal.io/develop/java/workflows/message-passing#updates) feature-guide * [![](https://docs.temporal.io/img/sdks/svgs/dotnet.svg)Validate updates in .NET](https://docs.temporal.io/develop/dotnet/workflows/message-passing#updates) feature-guide * [![](https://docs.temporal.io/img/sdks/svgs/python.svg)Validate updates in Python](https://docs.temporal.io/develop/python/workflows/message-passing#updates) feature-guide * [![](https://docs.temporal.io/img/sdks/svgs/typescript.svg)Validate updates in TypeScript](https://docs.temporal.io/develop/typescript/workflows/message-passing#updates) feature-guide * [![](https://docs.temporal.io/img/sdks/svgs/php.svg)Validate updates in PHP](https://docs.temporal.io/develop/php/workflows/message-passing#handle-updates) feature-guide ### Exceptions in message handlers[​](https://docs.temporal.io/handling-messages#exceptions "Direct link to Exceptions in message handlers") When throwing an exception in a message handler, you should decide whether to make it an [Application Failure](https://docs.temporal.io/references/failures#application-failure) . The implications are different between Signals and Updates. caution The following content applies in every SDK except the Go SDK. See below. #### Exceptions in Signals[​](https://docs.temporal.io/handling-messages#exceptions-in-signals "Direct link to Exceptions in Signals") In Signal handlers, throw [Application Failures](https://docs.temporal.io/references/failures#application-failure) only for unrecoverable errors, because the entire Workflow will fail. Similarly, allowing a failing Activity or Child Workflow to exhaust its retries, so that it throws an [Activity Failure](https://docs.temporal.io/references/failures#activity-failure) or [Child Workflow Failure](https://docs.temporal.io/references/failures#child-workflow-failure) will cause the entire Workflow to fail. Note that for Activities, this will only happen if you change the default Activity [Retry Policy](https://docs.temporal.io/encyclopedia/retry-policies) , since by default they retry forever. If you throw any other exception, by default, it will cause a [Workflow Task Failure](https://docs.temporal.io/references/failures#workflow-task-failures) . This means the Workflow will get stuck and will retry the handler periodically until the exception is fixed, for example by a code change. #### Exceptions in Updates[​](https://docs.temporal.io/handling-messages#exceptions-in-updates "Direct link to Exceptions in Updates") Doing any of the following will fail the Update and cause the client to receive the error: * Reject the Update by throwing any exception from your [Validator](https://docs.temporal.io/handling-messages#update-validators) . * Allow a failing Activity or Child Workflow to exhaust its retries, so that it throws an [Activity Failure](https://docs.temporal.io/references/failures#activity-failure) or [Child Workflow Failure](https://docs.temporal.io/references/failures#child-workflow-failure) . Note that for Activities, this will only happen if you change the default Activity [Retry Policy](https://docs.temporal.io/encyclopedia/retry-policies) , since by default they retry forever. * Throw an [Application Failure](https://docs.temporal.io/references/failures#application-failure) from your Update handler. Unlike with Signals, the Workflow will keep going in these cases. If you throw any other exception, by default, it will cause a [Workflow Task Failure](https://docs.temporal.io/references/failures#workflow-task-failures) . This means the Workflow will get stuck and will retry the handler periodically until the exception is fixed, for example by a code change or infrastructure coming back online. Note that this will cause a delay for clients waiting for an Update result. #### Errors and panics in message handlers in the Go SDK[​](https://docs.temporal.io/handling-messages#errors-and-panics-in-message-handlers-in-the-go-sdk "Direct link to Errors and panics in message handlers in the Go SDK") In Go, returning an error behaves like an [Application Failure](https://docs.temporal.io/references/failures#application-failure) in the other SDKs. Panics behave like non-Application Failure exceptions in other languages, in that they cause a [Workflow Task Failure](https://docs.temporal.io/references/failures#workflow-task-failures) . ### Writing Signal Handlers[​](https://docs.temporal.io/handling-messages#writing-signal-handlers "Direct link to Writing Signal Handlers") Use these links to see a simple Signal handler. Related 📚 * [![](https://docs.temporal.io/img/sdks/svgs/golang.svg)Handle Signals in Go](https://docs.temporal.io/develop/go/workflows/message-passing#signals) feature-guide * [![](https://docs.temporal.io/img/sdks/svgs/java.svg)Handle Signals in Java](https://docs.temporal.io/develop/java/workflows/message-passing#signals) feature-guide * [![](https://docs.temporal.io/img/sdks/svgs/python.svg)Handle Signals in Python](https://docs.temporal.io/develop/python/workflows/message-passing#signals) feature-guide * [![](https://docs.temporal.io/img/sdks/svgs/typescript.svg)Handle Signals in TypeScript](https://docs.temporal.io/develop/typescript/workflows/message-passing#signals) feature-guide * [![](https://docs.temporal.io/img/sdks/svgs/dotnet.svg)Handle Signals in .NET](https://docs.temporal.io/develop/dotnet/workflows/message-passing#signals) feature-guide * [![](https://docs.temporal.io/img/sdks/svgs/php.svg)Handle Signals in PHP](https://docs.temporal.io/develop/php/workflows/message-passing#handle-signal) feature-guide ### Writing Update Handlers[​](https://docs.temporal.io/handling-messages#writing-update-handlers "Direct link to Writing Update Handlers") Use these links to see a simple update handler. Related 📚 * [![](https://docs.temporal.io/img/sdks/svgs/golang.svg)Handle Updates in Go](https://docs.temporal.io/develop/go/workflows/message-passing#updates) feature-guide * [![](https://docs.temporal.io/img/sdks/svgs/java.svg)Handle Updates in Java](https://docs.temporal.io/develop/java/workflows/message-passing#updates) feature-guide * [![](https://docs.temporal.io/img/sdks/svgs/python.svg)Handle Updates in Python](https://docs.temporal.io/develop/python/workflows/message-passing#updates) feature-guide * [![](https://docs.temporal.io/img/sdks/svgs/typescript.svg)Handle Updates in TypeScript](https://docs.temporal.io/develop/typescript/workflows/message-passing#updates) feature-guide * [![](https://docs.temporal.io/img/sdks/svgs/dotnet.svg)Handle Updates in .NET](https://docs.temporal.io/develop/dotnet/workflows/message-passing#updates) feature-guide * [![](https://docs.temporal.io/img/sdks/svgs/php.svg)Handle Updates in PHP](https://docs.temporal.io/develop/php/workflows/message-passing#handle-updates) feature-guide ### Writing Query Handlers[​](https://docs.temporal.io/handling-messages#writing-query-handlers "Direct link to Writing Query Handlers") Author queries using these per-language guides. Related 📚 * [![](https://docs.temporal.io/img/sdks/svgs/golang.svg)Handle Queries in Go](https://docs.temporal.io/develop/go/workflows/message-passing#queries) feature-guide * [![](https://docs.temporal.io/img/sdks/svgs/java.svg)Handle Queries in Java](https://docs.temporal.io/develop/java/workflows/message-passing#queries) feature-guide * [![](https://docs.temporal.io/img/sdks/svgs/python.svg)Handle Queries in Python](https://docs.temporal.io/develop/python/workflows/message-passing#queries) feature-guide * [![](https://docs.temporal.io/img/sdks/svgs/typescript.svg)Handle Queries in TypeScript](https://docs.temporal.io/develop/typescript/workflows/message-passing#queries) feature-guide * [![](https://docs.temporal.io/img/sdks/svgs/dotnet.svg)Handle Queries in .NET](https://docs.temporal.io/develop/dotnet/workflows/message-passing#queries) feature-guide * [![](https://docs.temporal.io/img/sdks/svgs/php.svg)Handle Queries in PHP](https://docs.temporal.io/develop/php/workflows/message-passing#handle-query) feature-guide * [Handling Messages](https://docs.temporal.io/handling-messages#handling-messages) * [Message handler concurrency](https://docs.temporal.io/handling-messages#message-handler-concurrency) * [Message handler patterns](https://docs.temporal.io/handling-messages#message-handler-patterns) * [Update Validators](https://docs.temporal.io/handling-messages#update-validators) * [Exceptions in message handlers](https://docs.temporal.io/handling-messages#exceptions) * [Writing Signal Handlers](https://docs.temporal.io/handling-messages#writing-signal-handlers) * [Writing Update Handlers](https://docs.temporal.io/handling-messages#writing-update-handlers) * [Writing Query Handlers](https://docs.temporal.io/handling-messages#writing-query-handlers) --- # Enriching the user interface - Ruby SDK | Temporal Platform Documentation [Skip to main content](https://docs.temporal.io/develop/ruby/platform/enriching-ui#__docusaurus_skipToContent_fallback) On this page Temporal supports adding context to Workflows and Events with metadata. This helps users identify and understand Workflows and their operations. Adding Summary and Details to Workflows[​](https://docs.temporal.io/develop/ruby/platform/enriching-ui#adding-summary-and-details-to-workflows "Direct link to Adding Summary and Details to Workflows") --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- ### Starting a Workflow[​](https://docs.temporal.io/develop/ruby/platform/enriching-ui#starting-a-workflow "Direct link to Starting a Workflow") When starting a Workflow, you can provide a static summary and details to help identify the Workflow in the UI: require 'temporalio/client'# Create clientclient = Temporalio::Client.connect('localhost:7233')# Start a workflow with static summary and detailshandle = client.start_workflow( 'YourWorkflow', 'workflow input', id: 'your-workflow-id', task_queue: 'your-task-queue', static_summary: 'Order processing for customer #12345', static_details: 'Processing premium order with expedited shipping') `static_summary:` is a single-line description that appears in the Workflow list view, limited to 200 bytes. `static_details:` can be multi-line and provides more comprehensive information that appears in the Workflow details view, with a larger limit of 20K bytes. The input format is standard Markdown excluding images, HTML, and scripts. You can also use `execute_workflow` for synchronous execution: # Execute workflow synchronouslyresult = client.execute_workflow( 'YourWorkflow', 'workflow input', id: 'your-workflow-id', task_queue: 'your-task-queue', static_summary: 'Order processing for customer #12345', static_details: 'Processing premium order with expedited shipping') #### Inside the Workflow[​](https://docs.temporal.io/develop/ruby/platform/enriching-ui#inside-the-workflow "Direct link to Inside the Workflow") Within a Workflow, you can get and set the _current workflow details_. Unlike static summary/details set at Workflow start, this value can be updated throughout the life of the Workflow. Current Workflow details also takes Markdown format (excluding images, HTML, and scripts) and can span multiple lines. require 'temporalio'class YourWorkflow < Temporalio::Workflow::Definition def execute(input) # Get the current details current_details = Temporalio::Workflow.current_details Temporalio::Workflow.logger.info("Current details: #{current_details}") # Set/update the current details Temporalio::Workflow.current_details = 'Updated workflow details with new status' 'Workflow completed' endend #### Adding Summary to Activities and Timers[​](https://docs.temporal.io/develop/ruby/platform/enriching-ui#adding-summary-to-activities-and-timers "Direct link to Adding Summary to Activities and Timers") You can attach a `summary:` to activities when starting them from within a Workflow: require 'temporalio'class YourWorkflow < Temporalio::Workflow::Definition def execute(input) # Execute an activity with a summary result = Temporalio::Workflow.execute_activity( 'YourActivity', input, start_to_close_timeout: 10, summary: 'Processing user data' ) result endend Similarly, you can attach a `summary:` to timers within a Workflow: require 'temporalio'class YourWorkflow < Temporalio::Workflow::Definition def execute(input) # Create a timer with a summary Temporalio::Workflow.sleep(300, summary: 'Waiting for payment confirmation') 'Timer completed' endend The input format for `summary:` is a string, and limited to 200 bytes. Viewing Summary and Details in the UI[​](https://docs.temporal.io/develop/ruby/platform/enriching-ui#viewing-summary-and-details-in-the-ui "Direct link to Viewing Summary and Details in the UI") --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Once you've added summaries and details to your Workflows, Activities, and Timers, you can view this enriched information in the Temporal Web UI. Navigate to your Workflow's details page to see the metadata displayed in two key locations: ### Workflow Overview Section[​](https://docs.temporal.io/develop/ruby/platform/enriching-ui#workflow-overview-section "Direct link to Workflow Overview Section") At the top of the workflow details page, you'll find the workflow-level metadata: * **Summary & Details** - Displays the static summary and static details set when starting the workflow * **Current Details** - Displays the dynamic details that can be updated during workflow execution All Workflow details support standard Markdown formatting (excluding images, HTML, and scripts), allowing you to create rich, structured information displays. ### Event History[​](https://docs.temporal.io/develop/ruby/platform/enriching-ui#event-history "Direct link to Event History") Individual events in the Workflow's Event History display their associated summaries when available: Workflow, Activity and Timer summaries appear in purple text next to their corresponding Events, providing immediate context without requiring you to expand the event details. When you do expand an event, the summary is also prominently displayed in the detailed view. * [Adding Summary and Details to Workflows](https://docs.temporal.io/develop/ruby/platform/enriching-ui#adding-summary-and-details-to-workflows) * [Starting a Workflow](https://docs.temporal.io/develop/ruby/platform/enriching-ui#starting-a-workflow) * [Viewing Summary and Details in the UI](https://docs.temporal.io/develop/ruby/platform/enriching-ui#viewing-summary-and-details-in-the-ui) * [Workflow Overview Section](https://docs.temporal.io/develop/ruby/platform/enriching-ui#workflow-overview-section) * [Event History](https://docs.temporal.io/develop/ruby/platform/enriching-ui#event-history) --- # Detecting application failures | Temporal Platform Documentation [Skip to main content](https://docs.temporal.io/encyclopedia/detecting-application-failures#__docusaurus_skipToContent_fallback) In Temporal, timeouts detect application failures. The system can then automatically mitigate these failures through retries. Both Workflows and Activities have dedicated timeout configurations and can be configured with a RetryPolicy. * [Detecting Workflow failures](https://docs.temporal.io/encyclopedia/detecting-workflow-failures) * [Detecting Activity failures](https://docs.temporal.io/encyclopedia/detecting-activity-failures) * [Retry Policies](https://docs.temporal.io/encyclopedia/retry-policies) --- # Extensibility | Temporal Platform Documentation [Skip to main content](https://docs.temporal.io/encyclopedia/extensibility#__docusaurus_skipToContent_fallback) Temporal offers many mechanisms to augment the functionality of Workflows and Activities. These allow you to customize how data is serialized, propagate metadata across execution boundaries, and inject cross-cutting behavior like tracing and logging. * [Data Conversion](https://docs.temporal.io/dataconversion) - Customize how arguments and return values are serialized, compressed, or encrypted * [Context Propagation](https://docs.temporal.io/encyclopedia/context-propagation) - Pass custom metadata (tracing IDs, tenant IDs, auth tokens) across Workflow, Activity, and Child Workflow boundaries * [Interceptors](https://docs.temporal.io/encyclopedia/interceptors) - Add cross-cutting behavior (observability, authorization, header manipulation) before and after SDK operations * [Plugins](https://docs.temporal.io/encyclopedia/plugins) - Bundle interceptors, context propagators, data converters, and built-in definitions into reusable packages --- # Worker Shutdown Behavior | Temporal Platform Documentation [Skip to main content](https://docs.temporal.io/encyclopedia/workers/worker-shutdown#__docusaurus_skipToContent_fallback) On this page When a Worker shuts down, it stops polling for new tasks and begins the shutdown sequence. In the case of in-flight Workflow Tasks, shutdown may cause them to fail if they aren’t completed in time, after exhausting Retry Policy attempts. There are two types of shutdown behavior that can occur, depending on whether an idea of “graceful shutdown” is configured. Graceful Shutdown[​](https://docs.temporal.io/encyclopedia/workers/worker-shutdown#graceful-shutdown "Direct link to Graceful Shutdown") ----------------------------------------------------------------------------------------------------------------------------------------- Graceful shutdown configures how much time a Worker has to complete its current task before shutting down. An Activity is able to determine that the Worker it’s running on is being shut down, through the Activity context. > Core SDKs - `graceful_shutdown_period` > Go - `WorkerStopTimeout` > Java - `shutdown()` followed by `awaitTermination(timeout, unit)` ### Workflow tasks[​](https://docs.temporal.io/encyclopedia/workers/worker-shutdown#workflow-tasks "Direct link to Workflow tasks") Any in-flight Workflow Tasks are (attempted to be) completed. The only reason they may not immediately, is if Workflow code is (incorrectly) blocking, or because of Local Activities (see below). ### Activities[​](https://docs.temporal.io/encyclopedia/workers/worker-shutdown#activities "Direct link to Activities") Activities are allowed to complete during the graceful shutdown period. ### Local Activities[​](https://docs.temporal.io/encyclopedia/workers/worker-shutdown#local-activities "Direct link to Local Activities") Because Local Activities run within a Workflow Task, current and future Local Activities within the same Workflow Task will be allowed to run and complete, assuming there is no additional command to yield to. If the Local Activity is unable to complete by the graceful shutdown period, the Local Activity attempt is sent a cancel signal. In this case, no new Local Activities will be retried or started, and the Worker is shut down. The Worker still waits for the current Workflow Task to complete, meaning you can eventually hit your Workflow Task or execution timeout, unless another Worker is spun up. Non-Graceful Period Shutdown[​](https://docs.temporal.io/encyclopedia/workers/worker-shutdown#non-graceful-period-shutdown "Direct link to Non-Graceful Period Shutdown") -------------------------------------------------------------------------------------------------------------------------------------------------------------------------- This behavior is for either no graceful period being specified, or if the shutdown has taken longer than the configured graceful period. In all cases, the Activity context is canceled and the Worker will finish shutdown when the current Workflow Task completes (with either success or failure). note Go and Core SDKs behave differently when we pass task timeout and the Activity or Local Activity is still running: **Go** - The shutdown completes, but the Activity will continue to run and use a slot. **Core** - The Worker shutdown will not complete while the Activity completes. ### Local Activities[​](https://docs.temporal.io/encyclopedia/workers/worker-shutdown#local-activities-1 "Direct link to Local Activities") The Local Activity is sent a cancel signal, then the Workflow Task heartbeats stop, and no new Local Activities will be retried or started. The Worker still waits for the current Workflow Task to complete, meaning you can eventually hit your Workflow Task or execution timeout, unless another Worker is spun up. General Developer Guidance[​](https://docs.temporal.io/encyclopedia/workers/worker-shutdown#general-developer-guidance "Direct link to General Developer Guidance") -------------------------------------------------------------------------------------------------------------------------------------------------------------------- * Ensure Activities and Local Activities **honor context cancellation** or other shutdown signals. * Expect that **long or hung Local Activities may block shutdown** unless you fail early. It is recommended that Local Activities should already generally be used for short Activities. * [Graceful Shutdown](https://docs.temporal.io/encyclopedia/workers/worker-shutdown#graceful-shutdown) * [Workflow tasks](https://docs.temporal.io/encyclopedia/workers/worker-shutdown#workflow-tasks) * [Activities](https://docs.temporal.io/encyclopedia/workers/worker-shutdown#activities) * [Local Activities](https://docs.temporal.io/encyclopedia/workers/worker-shutdown#local-activities) * [Non-Graceful Period Shutdown](https://docs.temporal.io/encyclopedia/workers/worker-shutdown#non-graceful-period-shutdown) * [Local Activities](https://docs.temporal.io/encyclopedia/workers/worker-shutdown#local-activities-1) * [General Developer Guidance](https://docs.temporal.io/encyclopedia/workers/worker-shutdown#general-developer-guidance) --- # Set up your local with the Ruby SDK | Temporal Platform Documentation [Skip to main content](https://docs.temporal.io/develop/ruby/set-up-local-ruby#__docusaurus_skipToContent_fallback) This guide walks you through setting up the Temporal Ruby SDK and running your first Workflow. In just a few steps, you'll install the SDK and start a local development server. To validate that your local environment is correctly installed, we will execute a Workflow that will output "Hello, Temporal". Installation[​](https://docs.temporal.io/develop/ruby/set-up-local-ruby#installation "Direct link to Installation") -------------------------------------------------------------------------------------------------------------------- This step sets up a new Ruby project using Bundler and installs the Temporal Ruby SDK. We recommend using [Bundler](https://bundler.io/) to manage your Ruby project dependencies, including the Temporal SDK. These tutorials assume Ruby 3.4.3 or higher. Follow the steps to create a directory, initialize the project with a `Gemfile`, and add the Temporal SDK. **Note:** * Only macOS ARM/x64 and Linux ARM/x64 are supported. * Source gem is published but **cannot be built directly**. * Windows (MinGW) is not supported. * `fibers`/`async` are only supported on Ruby **3.3+**. * See [Platform Support](https://docs.temporal.io/develop/ruby/set-up-local-ruby#) for full details. **1\. Check your Ruby version:** ruby -v You should see output like `ruby 3.4.3`. Ruby 3.2+ is required. We recommend Ruby 3.4.3. **2\. Create your project folder:** mkdir temporal-project cd temporal-project **3\. Initialize with Bundler:** bundle init **4\. Add the Temporal Ruby SDK:** bundle add temporalio You should see output like: Fetching gem metadata from https://rubygems.org/... Resolving dependencies... Installing temporalio 0.4.0 (arm64-darwin) Bundle complete! 1 Gemfile dependency, 6 gems now installed. **5\. Install dependencies:** bundle install Install Temporal CLI[​](https://docs.temporal.io/develop/ruby/set-up-local-ruby#install-temporal-cli "Direct link to Install Temporal CLI") -------------------------------------------------------------------------------------------------------------------------------------------- The fastest way to get a development version of the Temporal Service running on your local machine is to use [Temporal CLI](https://docs.temporal.io/cli) . Choose your operating system to install Temporal CLI. * macOS * Windows * Linux Install the Temporal CLI using Homebrew: brew install temporal Download the Temporal CLI archive for your architecture: * [Windows amd64](https://temporal.download/cli/archive/latest?platform=windows&arch=amd64) * [Windows arm64](https://temporal.download/cli/archive/latest?platform=windows&arch=arm64) Extract it and add `temporal.exe` to your PATH. Download the Temporal CLI for your architecture: * [Linux amd64](https://temporal.download/cli/archive/latest?platform=linux&arch=amd64) * [Linux arm64](https://temporal.download/cli/archive/latest?platform=linux&arch=arm64) Extract the archive and move the `temporal` binary into your PATH, for example: sudo mv temporal /usr/local/bin Start the development server[​](https://docs.temporal.io/develop/ruby/set-up-local-ruby#start-the-development-server "Direct link to Start the development server") -------------------------------------------------------------------------------------------------------------------------------------------------------------------- Once you've installed Temporal CLI and added it to your PATH, open a new Terminal window and run the following command. This command starts a local Temporal Service. It starts the Web UI, creates the default Namespace, and uses an in-memory database. The Temporal Service will be available on localhost:7233. The Temporal Web UI will be available at [http://localhost:8233](http://localhost:8233/) . Leave the local Temporal Service running as you work through tutorials and other projects. You can stop the Temporal Service at any time by pressing CTRL+C. Once you have everything installed, you're ready to build apps with Temporal on your local machine. After installing, open a new Terminal window and start the development server: temporal server start-dev #### Change the Web UI port The Temporal Web UI may be on a different port in some examples or tutorials. To change the port for the Web UI, use the `--ui-port` option when starting the server: temporal server start-dev --ui-port 8080 The Temporal Web UI will now be available at http://localhost:8080. Run Hello World: Test Your Installation[​](https://docs.temporal.io/develop/ruby/set-up-local-ruby#run-hello-world-test-your-installation "Direct link to Run Hello World: Test Your Installation") ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Now let's verify your setup is working by creating and running a complete Temporal application with both a Workflow and Activity. This test will confirm that: * The Temporal Ruby SDK is properly installed * Your local Temporal Service is running * You can successfully create and execute Workflows and Activities * The communication between components is functioning correctly ### 1\. Create the Activity[​](https://docs.temporal.io/develop/ruby/set-up-local-ruby#1-create-the-activity "Direct link to 1. Create the Activity") Create an Activity file (say\_hello\_activity.rb): require 'temporalio/activity'# Implementation of a simple activityclass SayHelloActivity < Temporalio::Activity::Definition def execute(name) "Hello, #{name}!" endend ### 2\. Create the Workflow[​](https://docs.temporal.io/develop/ruby/set-up-local-ruby#2-create-the-workflow "Direct link to 2. Create the Workflow") Create a Workflow file (say\_hello\_workflow.rb): require 'temporalio/workflow'require_relative 'say_hello_activity'class SayHelloWorkflow < Temporalio::Workflow::Definition def execute(name) Temporalio::Workflow.execute_activity( SayHelloActivity, name, schedule_to_close_timeout: 300 ) endend ### 3\. Create and Run the Worker[​](https://docs.temporal.io/develop/ruby/set-up-local-ruby#3-create-and-run-the-worker "Direct link to 3. Create and Run the Worker") With your Activity and Workflow defined, you need a Worker to execute them. Workers are a crucial part of your Temporal application as they're what actually execute the tasks defined in your Workflows and Activities. For more information on Workers, see [Understanding Temporal](https://docs.temporal.io/evaluate/understanding-temporal#workers) and a [deep dive into Workers](https://docs.temporal.io/workers) . Create a Worker file (worker.rb): require 'temporalio/client'require 'temporalio/worker'require_relative 'say_hello_activity'require_relative 'say_hello_workflow'# Create a clientclient = Temporalio::Client.connect('localhost:7233', 'default')# Create a worker with the client, activities, and workflowsworker = Temporalio::Worker.new( client:, task_queue: 'my-task-queue', workflows: [SayHelloWorkflow], # There are various forms an activity can take, see "Activities" section for details activities: [SayHelloActivity])# Run the worker until SIGINT. This can be done in many ways, see "Workers" section for details.worker.run(shutdown_signals: ['SIGINT']) Run the Worker: ruby worker.rb ### 4\. Execute the Workflow[​](https://docs.temporal.io/develop/ruby/set-up-local-ruby#4-execute-the-workflow "Direct link to 4. Execute the Workflow") Now that your Worker is running, it's time to start a Workflow Execution. Create a separate file called starter.rb: require 'temporalio/client'require_relative 'say_hello_workflow'# Create a clientclient = Temporalio::Client.connect('localhost:7233', 'default')# Run workflowresult = client.execute_workflow( SayHelloWorkflow, 'Temporal', # This is the input to the workflow id: 'my-workflow-id', task_queue: 'my-task-queue')puts "Result: #{result}" Then run: ruby starter.rb ### Verify Success[​](https://docs.temporal.io/develop/ruby/set-up-local-ruby#verify-success "Direct link to Verify Success") If everything is working correctly, you should see: * Worker processing the workflow and activity * Output: `Workflow result: Hello, Temporal!` * Workflow Execution details in the [Temporal Web UI](http://localhost:8233/) [### Next: Run your first Temporal Application\ \ Create a basic Workflow and run it with the Temporal Ruby SDK\ \ →](https://learn.temporal.io/getting_started/ruby/first_program_in_ruby/) --- # Schedule | Temporal Platform Documentation [Skip to main content](https://docs.temporal.io/schedule#__docusaurus_skipToContent_fallback) On this page This page discusses [Schedule](https://docs.temporal.io/schedule#schedule) . What is a Schedule?[​](https://docs.temporal.io/schedule#schedule "Direct link to What is a Schedule?") -------------------------------------------------------------------------------------------------------- A Schedule contains instructions for starting a [Workflow Execution](https://docs.temporal.io/workflow-execution) at specific times. Schedules provide a more flexible and user-friendly approach than [Temporal Cron Jobs](https://docs.temporal.io/cron-job) . * [How to enable Schedules](https://docs.temporal.io/schedule#limitations) * [How to operate Schedules using the Temporal CLI](https://docs.temporal.io/cli/schedule) A Schedule has an identity and is independent of a Workflow Execution. This differs from a Temporal Cron Job, which relies on a cron schedule as a property of the Workflow Execution. info For triggering a Workflow Execution at a specific one-time future point rather than on a recurring schedule, the [Start Delay](https://docs.temporal.io/workflow-execution/timers-delays#delay-workflow-execution) option should be used instead of a Schedule. ### Action[​](https://docs.temporal.io/schedule#action "Direct link to Action") The Action of a Schedule is where the Workflow Execution properties are established, such as Workflow Type, Task Queue, parameters, and timeouts. Workflow Executions started by a Schedule have the following additional properties: * The Action's timestamp is appended to the Workflow Id. * The `TemporalScheduledStartTime` [Search Attribute](https://docs.temporal.io/search-attribute) is added to the Workflow Execution. The value is the Action's timestamp. * The `TemporalScheduledById` Search Attribute is added to the Workflow Execution. The value is the Schedule Id. ### Spec[​](https://docs.temporal.io/schedule#spec "Direct link to Spec") The Schedule Spec defines when the Action should be taken. Unless many Schedules have Actions scheduled at the same time, Actions should generally start within 1 second of the specified time. There are two kinds of Schedule Spec: * A simple interval, like "every 30 minutes" (aligned to start at the Unix epoch, and optionally including a phase offset). * A calendar-based expression, similar to the "cron expressions" supported by lots of software, including the older Temporal Cron feature. These two kinds have multiple representations, depending on the interface or SDK you're using, but they all support the same features. In the Temporal CLI, for example, an interval is specified as a string like `45m` to mean every 45 minutes, or `6h/5h` to mean every 6 hours but at the start of the fifth hour within each period. In the Temporal CLI, a calendar expression can be specified as either a traditional cron string with five (or six or seven) positional fields, or as JSON with named fields: { "year": "2022", "month": "Jan,Apr,Jul,Oct", "dayOfMonth": "1,15", "hour": "11-14"} The following calendar JSON fields are available: * `year` * `month` * `dayOfMonth` * `dayOfWeek` * `hour` * `minute` * `second` * `comment` Each field can contain a comma-separated list of ranges (or the `*` wildcard), and each range can include a slash followed by a skip value. The `hour`, `minute`, and `second` fields default to `0` while the others default to `*`, so you can describe many useful specs with only a few fields. For `month`, names of months may be used instead of integers (case-insensitive, abbreviations permitted). For `dayOfWeek`, day-of-week names may be used. The `comment` field is optional and can be used to include a free-form description of the intent of the calendar spec, useful for complicated specs. No matter which form you supply, calendar and interval specs are converted to canonical representations. What you see when you "describe" or "list" a Schedule might not look exactly like what you entered, but it has the same meaning. #### Phase offset[​](https://docs.temporal.io/schedule#phase-offset "Direct link to Phase offset") By default, intervals align to the Unix epoch (January 1, 1970 at midnight UTC), so without a phase offset, a 7-day interval would fire every Thursday at whatever time midnight UTC equates to in your local time, which may not be convenient for your use case. Use a phase offset when you need your recurring Schedule to fire at a specific, human-meaningful time like every 10 days at 5pm UTC on a particular date, rather than accepting the arbitrary time the epoch alignment produces. There is an `offset` parameter you can set so an `interval` spec fires exactly at a pre-determined date and time. Here's an example of how you can calculate the `offset` and apply it to your Schedules. Timestamps are measured with UNIX time. That means 7 days is 604,800 seconds (7 \* 24 \* 60 \* 60), 5 days is 432,000 (5 \* 24 \* 60 \* 60), etc. The Schedule fires when the following condition is true: ((currentTimestampUTC - phase offset) % interval) == 0 So using an interval of 7 days and a phase offset of 0, that means the Schedule fires on `December 28th 2023, at midnight UTC`: ((1703721600 - 0) % 604800) = 0 For an interval of 5 days and a phase offset of 0, the Schedule fires on `December 29th 2023` ((1703808000 - 0) % 432000) = 0 You can calculate the phase offset you need for any interval. Take the UNIX timestamp of any date and time you want your Schedule to execute, then calculate the following: phase offset = referenceTimestampUTC % interval For example, if you want a Schedule that will fire every 10 days, including Jun 19th 2024 at 5pm UTC, you need to calculate the UNIX timestamp: UNIX Timestamp of 2024-06-19T17:00:00 UTC = 1718816400 Then you need to calculate the number of seconds for your interval: Interval of 10 days = (10 * 24 * 60 * 60) = 864000 Finally, you can calculate the phase offset you need: phase offset = 1718816400 % 864000 = 320400 So you will set your Schedule interval to `864000s` (10 days) and phase to `320400s` and you get exactly every 10 days at 5pm UTC. #### Other Spec features[​](https://docs.temporal.io/schedule#other-spec-features "Direct link to Other Spec features") **Multiple intervals/calendar expressions:** A Spec can have combinations of multiple intervals and/or calendar expressions to define a specific Schedule. **Time bounds:** Provide an absolute start or end time (or both) with a Spec to ensure that no actions are taken before the start time or after the end time. **Exclusions:** A Spec can contain exclusions in the form of zero or more calendar expressions. This can be used to express scheduling like "each Monday at noon except for holidays". You'll have to provide your own set of exclusions and include it in each schedule; there are no pre-defined sets. (This feature isn't currently exposed in the Temporal CLI or the Temporal Web UI.) **Jitter:** If given, a random offset between zero and the maximum jitter is added to each Action time (but bounded by the time until the next scheduled Action). **Time zones:** By default, calendar-based expressions are interpreted in UTC. Temporal recommends using UTC to avoid various surprising properties of time zones. If you don't want to use UTC, you can provide the name of a time zone. The time zone definition is loaded on the Temporal Server Worker Service from either disk or the fallback embedded in the binary. For more operational control, embed the contents of the time zone database file in the Schedule Spec itself. (Note: this isn't currently exposed in the Temporal CLI or the web UI.) ### Pause[​](https://docs.temporal.io/schedule#pause "Direct link to Pause") A Schedule can be Paused. When a Schedule is Paused, the Spec has no effect. However, you can still force manual actions by using the [temporal schedule trigger](https://docs.temporal.io/cli/schedule#trigger) command. To assist communication among developers and operators, a “notes” field can be updated on pause or resume to store an explanation for the current state. ### Backfill[​](https://docs.temporal.io/schedule#backfill "Direct link to Backfill") A Schedule can be Backfilled. When a Schedule is Backfilled, all the Actions that would have been taken over a specified time period are taken now (in parallel if the `AllowAll` [Overlap Policy](https://docs.temporal.io/schedule#overlap-policy) is used; sequentially if `BufferAll` is used). You might use this to fill in runs from a time period when the Schedule was paused due to an external condition that's now resolved, or a period before the Schedule was created. ### Limit number of Actions[​](https://docs.temporal.io/schedule#limit-number-of-actions "Direct link to Limit number of Actions") A Schedule can be limited to a certain number of scheduled Actions (that is, not trigger immediately). After that it will act as if it were paused. ### Policies[​](https://docs.temporal.io/schedule#policies "Direct link to Policies") A Schedule supports a set of Policies that enable customizing behavior. #### Overlap Policy[​](https://docs.temporal.io/schedule#overlap-policy "Direct link to Overlap Policy") The Overlap Policy controls what happens when it is time to start a Workflow Execution but a previously started Workflow Execution is still running. The following options are available: * `Skip`: **Default**. Nothing happens; the Workflow Execution is not started. * `BufferOne`: Starts the Workflow Execution as soon as the current one completes. The buffer is limited to one. If another Workflow Execution is supposed to start, but one is already in the buffer, only the one in the buffer eventually starts. * `BufferAll`: Allows an unlimited number of Workflows to buffer. They are started sequentially. * `CancelOther`: Cancels the running Workflow Execution, and then starts the new one after the old one completes cancellation. * `TerminateOther`: Terminates the running Workflow Execution and starts the new one immediately. * `AllowAll` Starts any number of concurrent Workflow Executions. With this policy (and only this policy), more than one Workflow Execution, started by the Schedule, can run simultaneously. #### Catchup Window[​](https://docs.temporal.io/schedule#catchup-window "Direct link to Catchup Window") The Temporal Service might be down or unavailable at the time when a Schedule should take an Action. When it comes back up, the Catchup Window controls which missed Actions should be taken at that point. The default is one year, meaning Actions will be taken unless over one year late. If your Actions are more time-sensitive, you can set the Catchup Window to a smaller value (minimum ten seconds), accepting that an outage longer than the window could lead to missed Actions. (But you can always [Backfill](https://docs.temporal.io/schedule#backfill) .) #### Pause-on-failure[​](https://docs.temporal.io/schedule#pause-on-failure "Direct link to Pause-on-failure") If this policy is set, a Workflow Execution started by a Schedule that ends with a failure or timeout (but not Cancellation or Termination) causes the Schedule to automatically pause. Note that with the `AllowAll` Overlap Policy, this pause might not apply to the next Workflow Execution, because the next Workflow Execution might have started before the failed one finished. It applies only to Workflow Executions that were scheduled to start after the failed one finished. ### Last completion result[​](https://docs.temporal.io/schedule#last-completion-result "Direct link to Last completion result") A Workflow started by a Schedule can obtain the completion result from the most recent successful run. (How you do this depends on the SDK you're using.) For overlap policies that don't allow overlap, “the most recent successful run” is straightforward to define. For the `AllowAll` policy, it refers to the run that completed most recently, at the time that the run in question is started. Consider the following overlapping runs: time --------------------------------------------> A |----------------------| B |-------| C |---------------| D |--------------T If D asks for the last completion result at time T, it gets the result of A. Not B, even though B started more recently, because A completed later. And not C, even though C completed after A, because the result for D is captured when D is started, not when it's queried. Failures and timeouts do not affect the last completion result. note When a Schedule triggers a Workflow that completes successfully and yields a result, the result from the initial Schedule execution can be accessed by the subsequent scheduled execution through `LastCompletionResult`. Be aware that if, during the subsequent run, the Workflow employs the [Continue-As-New](https://docs.temporal.io/workflow-execution/continue-as-new) feature, `LastCompletionResult` won't be accessible for this new Workflow iteration. It is important to note that the [status](https://docs.temporal.io/workflow-execution#workflow-execution-status) of the subsequent run is marked as `Continued-As-New` and not as `Completed`. caution A scheduled Workflow Execution may complete with a result up to the maximum blob size (2 MiB by default). However, due to internal limitations, results that are within 1 KiB of this limit cannot be passed to the next execution. So, for example, a Workflow Execution that returns a result of size 2,096,640 bytes (which is above 2MiB - 1KiB limit) will be allowed to complete successfully, but that value will not be available as a last completion result. This limitation may be lifted in the future. ### Last failure[​](https://docs.temporal.io/schedule#last-failure "Direct link to Last failure") A Workflow started by a Schedule can obtain the details of the failure of the most recent run that ended at the time when the Workflow in question was started. Unlike last completion result, a _successful_ run _does_ reset the last failure. ### Limitations[​](https://docs.temporal.io/schedule#limitations "Direct link to Limitations") Internally, a Schedule is implemented as a Workflow. If you're using Elasticsearch, these Workflow Executions are hidden from normal views. * [What is a Schedule?](https://docs.temporal.io/schedule#schedule) * [Action](https://docs.temporal.io/schedule#action) * [Spec](https://docs.temporal.io/schedule#spec) * [Phase offset](https://docs.temporal.io/schedule#phase-offset) * [Other Spec features](https://docs.temporal.io/schedule#other-spec-features) * [Pause](https://docs.temporal.io/schedule#pause) * [Backfill](https://docs.temporal.io/schedule#backfill) * [Limit number of Actions](https://docs.temporal.io/schedule#limit-number-of-actions) * [Policies](https://docs.temporal.io/schedule#policies) * [Overlap Policy](https://docs.temporal.io/schedule#overlap-policy) * [Catchup Window](https://docs.temporal.io/schedule#catchup-window) * [Pause-on-failure](https://docs.temporal.io/schedule#pause-on-failure) * [Last completion result](https://docs.temporal.io/schedule#last-completion-result) * [Last failure](https://docs.temporal.io/schedule#last-failure) * [Limitations](https://docs.temporal.io/schedule#limitations) --- # Temporal Server | Temporal Platform Documentation [Skip to main content](https://docs.temporal.io/temporal-service/temporal-server#__docusaurus_skipToContent_fallback) On this page This page discusses the following: * [Frontend Service](https://docs.temporal.io/temporal-service/temporal-server#frontend-service) * [History Service](https://docs.temporal.io/temporal-service/temporal-server#history-service) * [History Shard](https://docs.temporal.io/temporal-service/temporal-server#history-shard) * [Matching Service](https://docs.temporal.io/temporal-service/temporal-server#matching-service) * [Worker Service](https://docs.temporal.io/temporal-service/temporal-server#worker-service) * [Retention Period](https://docs.temporal.io/temporal-service/temporal-server#retention-period) What is the Temporal Server?[​](https://docs.temporal.io/temporal-service/temporal-server#temporal-server "Direct link to What is the Temporal Server?") --------------------------------------------------------------------------------------------------------------------------------------------------------- The Temporal Server consists of four independently scalable services: * Frontend gateway: for rate limiting, routing, authorizing. * History subsystem: maintains data (mutable state, queues, and timers). * Matching subsystem: hosts Task Queues for dispatching. * Worker Service: for internal background Workflows. For example, a real-life production deployment can have 5 Frontend, 15 History, 17 Matching, and 3 Worker Services per Temporal Service. The Temporal Server services can run independently or be grouped together into shared processes on one or more physical or virtual machines. For live (production) environments, we recommend that each service runs independently, because each one has different scaling requirements and troubleshooting becomes easier. The History, Matching, and Worker Services can scale horizontally within a Temporal Service. The Frontend Service scales differently than the others because it has no sharding or partitioning; it is just stateless. Each service is aware of the others, including scaled instances, through a membership protocol via [Ringpop](https://github.com/temporalio/ringpop-go) . ### Versions and support[​](https://docs.temporal.io/temporal-service/temporal-server#versions-and-support "Direct link to Versions and support") tip We release new versions of the Temporal SDKs and Temporal Server software independently of one another. That said, All SDK versions support all server versions. To take advantage of bug fixes, performance improvements, and new features, please upgrade both SDK and servers to the latest versions on a regular cadence. All Temporal Server releases abide by the [Semantic Versioning Specification](https://semver.org/) . We support upgrade paths from every version beginning with Temporal v1.7.0. For details on upgrading your Temporal Service, see [Upgrade Server](https://docs.temporal.io/self-hosted-guide/upgrade-server#upgrade-server) . We provide maintenance support for previously published minor and major versions by continuing to release critical bug fixes related to security, the prevention of data loss, and reliability, whenever they are found. We aim to publish incremental upgrade guides for each minor and major version, which include specifics about dependency upgrades that we have tested for (such as Cassandra 3.0 -> 3.11). We offer maintenance support of the last three **minor** versions after a release and do not plan to "backport" patches beyond that. We offer maintenance support of **major** versions for at least 12 months after a GA release, and we provide at least 6 months' notice before EOL/deprecating support. **Dependencies** Temporal offers official support for, and is tested against, dependencies with the exact versions described in the `go.mod` file of the corresponding release tag. (For example, [v1.5.1](https://github.com/temporalio/temporal/tree/v1.5.1) dependencies are documented in [the go.mod for v1.5.1](https://github.com/temporalio/temporal/blob/v1.5.1/go.mod) .) What is a Frontend Service?[​](https://docs.temporal.io/temporal-service/temporal-server#frontend-service "Direct link to What is a Frontend Service?") -------------------------------------------------------------------------------------------------------------------------------------------------------- The Frontend Service is a stateless gateway service that exposes a strongly typed [Proto API](https://github.com/temporalio/api/blob/master/temporal/api/workflowservice/v1/service.proto) . The Frontend Service is responsible for rate limiting, authorizing, validating, and routing all inbound calls. ![Frontend Service](https://docs.temporal.io/diagrams/temporal-frontend-service.svg) Frontend Service Types of inbound calls include the following: * [Namespace](https://docs.temporal.io/namespaces) CRUD * External events * Worker polls * [Visibility](https://docs.temporal.io/temporal-service/visibility) requests * [Temporal CLI](https://docs.temporal.io/cli) (the Temporal CLI) operations * Calls from a remote Temporal Service related to [Multi-Cluster Replication](https://docs.temporal.io/temporal-service/multi-cluster-replication) Every inbound request related to a Workflow Execution must have a Workflow Id, which is hashed for routing purposes. The Frontend Service has access to the hash rings that maintain service membership information, including how many nodes (instances of each service) are in the Temporal Service. Inbound call rate limiting is applied per host and per namespace. The Frontend Service talks to the Matching Service, History Service, Worker Service, the database, and Elasticsearch (if in use). * It uses the grpcPort 7233 to host the service handler. * It uses port 6933 for membership-related communication. Ports are configurable in the Temporal Service configuration. What is a History Service?[​](https://docs.temporal.io/temporal-service/temporal-server#history-service "Direct link to What is a History Service?") ----------------------------------------------------------------------------------------------------------------------------------------------------- The History Service is responsible for persisting Workflow Execution state to the Workflow History. When the Workflow Execution is able to progress, the History Service adds a Task with the Workflow's updated history to the Task Queue. From there, a Worker can poll for work, receive this updated history, and resume execution. ![Block diagram of how the History Service relates to the other services of the Temporal Server and to the Temporal Service](https://docs.temporal.io/diagrams/temporal-history-service.svg) Block diagram of how the History Service relates to the other services of the Temporal Server and to the Temporal Service The total number of History Service processes can be between 1 and the total number of [History Shards](https://docs.temporal.io/temporal-service/temporal-server#history-shard) . An individual History Service can support many History Shards. Temporal recommends starting at a ratio of 1 History Service process for every 500 History Shards. Although the total number of History Shards remains static for the life of the Temporal Service, the number of History Service processes can change. The History Service talks to the Matching Service and the database. * It uses grpcPort 7234 to host the service handler. * It uses port 6934 for membership-related communication. Ports are configurable in the Temporal Service configuration. ### What is a History Shard?[​](https://docs.temporal.io/temporal-service/temporal-server#history-shard "Direct link to What is a History Shard?") A History Shard is an important unit within a Temporal Service by which concurrent Workflow Execution throughput can be scaled. Each History Shard maps to a single persistence partition. A History Shard assumes that only one concurrent operation can be within a partition at a time. In essence, the number of History Shards represents the number of concurrent database operations that can occur for a Temporal Service. This means that the number of History Shards in a Temporal Service plays a significant role in the performance of your Temporal Application. Before integrating a database, the total number of History Shards for the Temporal Service must be chosen and set in the Temporal Service's configuration (see [persistence](https://docs.temporal.io/references/configuration#persistence) ). After the Shard count is configured and the database integrated, the total number of History Shards for the Temporal Service cannot be changed. In theory, a Temporal Service can operate with an unlimited number of History Shards, but each History Shard adds compute overhead to the Temporal Service. The Temporal Service has operated successfully using anywhere from 1 to 128K History Shards, with each Shard responsible for tens of thousands of Workflow Executions. One Shard is useful only in small scale setups designed for testing, while 128k Shards is useful only in very large scale production environments. The correct number of History Shards for any given Temporal Service depends entirely on the Temporal Application that it is supporting and the type of database. A History Shard is represented as a hashed integer. Each Workflow Execution is automatically assigned to a History Shard. The assignment algorithm hashes Workflow Execution metadata such as Workflow Id and Namespace and uses that value to match a History Shard. Each History Shard maintains the Workflow Execution Event History, Workflow Execution mutable state, and the following internal Task Queues: * Internal Transfer Task Queue: Transfers internal tasks to the Matching Service. Whenever a new Workflow Task needs to be scheduled, the History Service's Transfer Task Queue Processor transactionally dispatches it to the Matching Service. * Internal Timer Task Queue: Durably persists Timers. * Internal Replicator Task Queue: Asynchronously replicates Workflow Executions from active Clusters to other passive Clusters. (Relies on the experimental Multi-Cluster feature.) * Internal Visibility Task Queue: Pushes data to the [Advanced Visibility](https://docs.temporal.io/visibility#advanced-visibility) index. What is a Matching Service?[​](https://docs.temporal.io/temporal-service/temporal-server#matching-service "Direct link to What is a Matching Service?") -------------------------------------------------------------------------------------------------------------------------------------------------------- The Matching Service is responsible for hosting user-facing [Task Queues](https://docs.temporal.io/task-queue) for Task dispatching. ![Matching Service](https://docs.temporal.io/diagrams/temporal-matching-service.svg) Matching Service It is responsible for matching Workers to Tasks and routing new Tasks to the appropriate queue. This service can scale internally by having multiple instances. It talks to the Frontend Service, History Service, and the database. * It uses grpcPort 7235 to host the service handler. * It uses port 6935 for membership related communication. Ports are configurable in the Temporal Service configuration. What is a Worker Service?[​](https://docs.temporal.io/temporal-service/temporal-server#worker-service "Direct link to What is a Worker Service?") -------------------------------------------------------------------------------------------------------------------------------------------------- The Worker Service runs background processing for the replication queue, system Workflows, and (in versions older than 1.5.0) the Kafka visibility processor. Worker Service ![Worker Service](https://docs.temporal.io/diagrams/temporal-worker-service.svg) It talks to the Frontend Service. * It uses port 6939 for membership-related communication. Ports are configurable in the Temporal Service configuration. What is a Retention Period?[​](https://docs.temporal.io/temporal-service/temporal-server#retention-period "Direct link to What is a Retention Period?") -------------------------------------------------------------------------------------------------------------------------------------------------------- Retention Period is the duration for which the Temporal Service stores data associated with closed Workflow Executions on a Namespace in the Persistence store. * [How to set the Retention Period for a Namespace](https://docs.temporal.io/cli/operator#create) * [How to set the Retention Period for a Namespace using the Go SDK](https://docs.temporal.io/develop/go/client/namespaces) * [How to set the Retention Period for a Namespace using the Java SDK](https://docs.temporal.io/develop/java/client/namespaces) A Retention Period applies to all closed Workflow Executions within a [Namespace](https://docs.temporal.io/namespaces) and is set when the Namespace is registered. The Temporal Service triggers a Timer task at the end of the Retention Period that cleans up the data associated with the closed Workflow Execution on that Namespace. The minimum Retention Period is 1 day. On Temporal Service version 1.18 and later, the maximum Retention Period value for Namespaces can be set to anything over the minimum requirement of 1 day. Ensure that your Persistence store has enough capacity for the storage. On Temporal Service versions 1.17 and earlier, the maximum Retention Period you can set is 30 days. Setting the Retention Period to 0 results in the error _A valid retention period is not set on request_. If you don't set the Retention Period value when using the [`temporal operator namespace create`](https://docs.temporal.io/cli/operator#create) command, it defaults to 3 days. If you don't set the Retention Period value when using the Register Namespace Request API, it returns an error. When changing the Retention Period (with [`temporal operator namespace update`](https://docs.temporal.io/cli/operator#update) or the `UpdateNamespace` API), the new duration applies to Workflow Executions that close after the change is saved. info Changing the Retention Period does NOT affect existing closed Workflow Executions: they retain their original cleanup timers based on the Retention Period that was in effect when they closed. ### Manual cleanup of closed Workflow Executions[​](https://docs.temporal.io/temporal-service/temporal-server#manual-cleanup-of-closed-workflow-executions "Direct link to Manual cleanup of closed Workflow Executions") For cases where you need to remove closed Workflow Executions before their retention timer expires, you can use [`temporal workflow delete`](https://docs.temporal.io/cli/workflow#delete) or the `DeleteWorkflowExecution` command. This is particularly useful along with reducing the Retention Period to clean up previously closed Workflow Executions to reduce storage costs. * [What is the Temporal Server?](https://docs.temporal.io/temporal-service/temporal-server#temporal-server) * [Versions and support](https://docs.temporal.io/temporal-service/temporal-server#versions-and-support) * [What is a Frontend Service?](https://docs.temporal.io/temporal-service/temporal-server#frontend-service) * [What is a History Service?](https://docs.temporal.io/temporal-service/temporal-server#history-service) * [What is a History Shard?](https://docs.temporal.io/temporal-service/temporal-server#history-shard) * [What is a Matching Service?](https://docs.temporal.io/temporal-service/temporal-server#matching-service) * [What is a Worker Service?](https://docs.temporal.io/temporal-service/temporal-server#worker-service) * [What is a Retention Period?](https://docs.temporal.io/temporal-service/temporal-server#retention-period) * [Manual cleanup of closed Workflow Executions](https://docs.temporal.io/temporal-service/temporal-server#manual-cleanup-of-closed-workflow-executions) --- # Cloud automation - Temporal feature | Temporal Platform Documentation [Skip to main content](https://docs.temporal.io/evaluate/development-production-features/cloud-automation#__docusaurus_skipToContent_fallback) Temporal Cloud Automation changes how you manage and scale your cloud infrastructure. Its features enable you to automate critical tasks like user and namespace management, mTLS certificate rotation, and access control, ensuring security and operational efficiency. Cloud Automation offers secure authentication across all interfaces, reducing errors and enhancing security. **Key Features:** * [Secure API Keys](https://docs.temporal.io/cloud/api-keys) : Manage resources securely with Temporal Cloud API Keys. * [Temporal Cloud CLI (tcld)](https://docs.temporal.io/cloud/tcld) : Automate operations directly from the command line. * [Terraform Provider for Cloud](https://docs.temporal.io/cloud/terraform-provider#prerequisites) : Scale effortlessly with infrastructure-as-code. Related 📚 * [API Keys documentation](https://docs.temporal.io/cloud/api-keys) * [Cloud Ops API documentation](https://docs.temporal.io/ops?_gl=1*1yf937l*_gcl_au*MTg1MTAxMTEwNC4xNzEzOTcxMjYw*_ga*MTgwODU1MzQyNi4xNzA3NzA4ODIz*_ga_R90Q9SJD3D*MTcyMTI0MTAyNy41MjIuMS4xNzIxMjQ5NTYxLjAuMC4w) * [Temporal Cloud CLI](https://docs.temporal.io/cloud/tcld) * [Terraform Provider for Cloud](https://docs.temporal.io/cloud/terraform-provider) From centralizing cloud operations and automating certificate rotation to streamlining user management and onboarding new teams, Temporal's Cloud Automation features cover a wide range of use cases that enhance efficiency and security across your organization. --- # Interrupt a Workflow - Cancellation and Termination | Temporal Platform Documentation [Skip to main content](https://docs.temporal.io/evaluate/development-production-features/interrupt-workflow#__docusaurus_skipToContent_fallback) Discover how Temporal enables you to gracefully handle Workflow interruptions through cancellations and terminations. Understand how to stop a Workflow cleanly with cancellation, allowing for proper cleanup and state management. For situations where a Workflow is stuck, termination provides an immediate solution, ensuring your applications remain robust and responsive. Related 📚 * [![](https://docs.temporal.io/img/sdks/svgs/golang.svg)Handling Cancellation and Termination using the Go SDK](https://docs.temporal.io/develop/go/workflows/cancellation) feature-guide * [![](https://docs.temporal.io/img/sdks/svgs/java.svg)Handling Cancellation and Termination using the Java SDK](https://docs.temporal.io/develop/java/workflows/cancellation) feature-guide * [![](https://docs.temporal.io/img/sdks/svgs/php.svg)Handling Cancellation and Termination using the PHP SDK](https://docs.temporal.io/develop/php/workflows/cancellation) feature-guide * [![](https://docs.temporal.io/img/sdks/svgs/python.svg)Handling Cancellation and Termination using the Python SDK](https://docs.temporal.io/develop/python/workflows/cancellation) feature-guide * [![](https://docs.temporal.io/img/sdks/svgs/typescript.svg)Handling Cancellation and Termination using the TypeScript SDK](https://docs.temporal.io/develop/typescript/workflows/cancellation) feature-guide * [![](https://docs.temporal.io/img/sdks/svgs/dotnet.svg)Handling Cancellation and Termination using the .NET SDK](https://docs.temporal.io/develop/dotnet/workflows/cancellation) feature-guide * [![](https://docs.temporal.io/img/sdks/svgs/ruby.svg)Handling Cancellation and Termination using the Ruby SDK](https://docs.temporal.io/develop/ruby/workflows/cancellation) feature-guide --- # Standalone Activity | Temporal Platform Documentation [Skip to main content](https://docs.temporal.io/standalone-activity#__docusaurus_skipToContent_fallback) On this page SUPPORT, STABILITY, and DEPENDENCY INFO Standalone Activities are available as a [Pre-release](https://docs.temporal.io/evaluate/development-production-features/release-stages#pre-release) feature in [Temporal Cloud](https://docs.temporal.io/standalone-activity#temporal-cloud-support) and in a [special release](https://docs.temporal.io/standalone-activity#temporal-cli-support) of the Temporal CLI. See [limitations](https://docs.temporal.io/standalone-activity#pre-release-limitations) below. What is a Standalone Activity?[​](https://docs.temporal.io/standalone-activity#standalone-activity "Direct link to What is a Standalone Activity?") ---------------------------------------------------------------------------------------------------------------------------------------------------- A top-level [Activity Execution](https://docs.temporal.io/activity-execution) that is started directly by a [Client](https://docs.temporal.io/encyclopedia/temporal-sdks#temporal-client) , without using a Workflow, is called a Standalone Activity. Standalone Activities are Temporal's [job queue](https://docs.temporal.io/evaluate/development-production-features/job-queue) . If you need to orchestrate multiple Activities, then you should use a Workflow. But if you just need to execute a single Activity, then you can use a Standalone Activity. This will result in fewer [Billable Actions](https://docs.temporal.io/cloud/actions#actions-in-workflows) in Temporal Cloud than using a Workflow to run a single Activity. If your Activity Execution is short-lived, then you will also notice lower latency, since there are fewer Worker round-trips than when executing the Activity in a Workflow. Standalone Activities support the same retry policies and timeouts as Workflow Activities, and you write your Activity Functions in the same way for both. In fact, an Activity Function can be executed both as a Standalone Activity and as a Workflow Activity. Use cases[​](https://docs.temporal.io/standalone-activity#use-cases "Direct link to Use cases") ------------------------------------------------------------------------------------------------ Standalone Activities can be used for [durable job processing use cases](https://docs.temporal.io/evaluate/development-production-features/job-queue) such as sending an email, processing a webhook, syncing data, or executing a single function reliably with built-in retries and timeouts. Key features[​](https://docs.temporal.io/standalone-activity#key-features "Direct link to Key features") --------------------------------------------------------------------------------------------------------- * Execute any Temporal Activity as a top-level primitive without the overhead of a Workflow * Native async job processing model: schedule -> dispatch -> process -> result * No head-of-line blocking - a slow job doesn’t block the dispatch of other Tasks * Arbitrary length jobs with heartbeats for liveness and checkpointing progress * At-least-once execution by default with native retry policy and timeouts * At-most-once execution if retry max attempts is 1 * Addressable - get an Activity ID / Run ID and get the result, cancel, and terminate * Deduplication - with conflict policy: (USE\_EXISTING, …), reuse policy: (REJECT\_DUPLICATES, …) * Separate ID space from Workflows - Standalone Activities are a different kind of top-level execution * Priority and fairness - multi-tenant fairness, weighted priority tiers, and safeguards against starvation of lower-weighted tasks * Visibility - list Activity Executions and view status, retry count, and last error * Manual completion by ID (or token): ignore activity return and wait for external completion * Activity metrics - including counts for success, failure, timeout, and cancel * Dual use - execute Activities within a Workflow or standalone with no Worker code changes Pre-release limitations[​](https://docs.temporal.io/standalone-activity#pre-release-limitations "Direct link to Pre-release limitations") ------------------------------------------------------------------------------------------------------------------------------------------ The pre-release of Standalone Activities is recommended for experimental use only and has some known limitations. General limitations: * Not suitable for production use cases during Pre-release * Delete, pause, reset, and update options are not supported yet * The `TerminateExisting` conflict policy is not supported yet Temporal Cloud limitations: * Standalone Activities are free for evaluation purposes during Pre-release, so we reserve the right to limit usage if it exceeds a reasonable amount * 1 day max retention for Standalone Activities * We recommend enabling Standalone Activities on a new namespace for dev/test experimental use only Temporal CLI support[​](https://docs.temporal.io/standalone-activity#temporal-cli-support "Direct link to Temporal CLI support") --------------------------------------------------------------------------------------------------------------------------------- A [special release of the Temporal CLI](https://github.com/temporalio/cli/releases/tag/v1.6.2-standalone-activity) supports Standalone Activities during Pre-release: * The `temporal activity` subcommand supports Standalone Activities with commands including: `start`, `result`, and `list` * Temporal Dev Server has Standalone Activities enabled by default in this special CLI release for local testing Temporal Cloud support[​](https://docs.temporal.io/standalone-activity#temporal-cloud-support "Direct link to Temporal Cloud support") --------------------------------------------------------------------------------------------------------------------------------------- Contact your account team to enable Standalone Activities in Temporal Cloud as a Pre-release feature. Get started[​](https://docs.temporal.io/standalone-activity#get-started "Direct link to Get started") ------------------------------------------------------------------------------------------------------ RESOURCES * [Go SDK - Standalone Activities quick start and code sample](https://docs.temporal.io/develop/go/activities/standalone-activities) * [Python SDK - Standalone Activities quick start and code sample](https://docs.temporal.io/develop/python/activities/standalone-activities) * [.NET SDK - Standalone Activities quick start and code sample](https://docs.temporal.io/develop/dotnet/activities/standalone-activities) * [What is a Standalone Activity?](https://docs.temporal.io/standalone-activity#standalone-activity) * [Use cases](https://docs.temporal.io/standalone-activity#use-cases) * [Key features](https://docs.temporal.io/standalone-activity#key-features) * [Pre-release limitations](https://docs.temporal.io/standalone-activity#pre-release-limitations) * [Temporal CLI support](https://docs.temporal.io/standalone-activity#temporal-cli-support) * [Temporal Cloud support](https://docs.temporal.io/standalone-activity#temporal-cloud-support) * [Get started](https://docs.temporal.io/standalone-activity#get-started) --- # Security in Temporal Nexus | Temporal Platform Documentation [Skip to main content](https://docs.temporal.io/nexus/security#__docusaurus_skipToContent_fallback) On this page SUPPORT, STABILITY, and DEPENDENCY INFO Temporal Nexus is now [Generally Available](https://docs.temporal.io/evaluate/development-production-features/release-stages#general-availability) for [Temporal Cloud](https://docs.temporal.io/cloud/nexus) and [self-hosted deployments](https://docs.temporal.io/production-deployment/self-hosted-guide/nexus) . Temporal Cloud provides built-in Endpoint access controls and secure connectivity across Namespaces. Self-hosted deployments can implement [custom Authorizers](https://docs.temporal.io/self-hosted-guide/security#authorizer-plugin) . Runtime access controls[​](https://docs.temporal.io/nexus/security#runtime-access-controls "Direct link to Runtime access controls") ------------------------------------------------------------------------------------------------------------------------------------- In Temporal Cloud, each Endpoint has an access control policy: an allowlist of caller Namespaces. ![Nexus Security](https://docs.temporal.io/img/cloud/nexus/nexus-workers-short.png) Nexus Security Workers authenticate with their Namespace using mTLS or API key. When a caller Workflow executes a Nexus Operation, Temporal Cloud verifies the caller's Namespace is in the Endpoint's allowlist before routing the request to the handler. Temporal Cloud acts as a trusted broker across Namespace boundaries. See [Configure runtime access controls](https://docs.temporal.io/nexus/registry#configure-runtime-access-controls) . Secure connectivity[​](https://docs.temporal.io/nexus/security#secure-connectivity "Direct link to Secure connectivity") ------------------------------------------------------------------------------------------------------------------------- info Temporal Cloud has built-in secure connectivity across all Namespaces in an Account. Self-hosted deployments rely on the Temporal Cluster being secure. Temporal Cloud secures all Nexus communication: * Workers authenticate to their Namespace using mTLS or API key. * mTLS encrypts all cross-Namespace Nexus traffic (start, cancel, and completion callbacks) across cells and regions. * Endpoints are only accessible from within a Temporal Cloud Account through the Temporal SDK - not externally accessible. Payload encryption and Data Converter[​](https://docs.temporal.io/nexus/security#payload-encryption-data-converter "Direct link to Payload encryption and Data Converter") --------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Nexus uses the same Data Converter as Workflows and Activities - JSON, Proto, and binary payloads are all supported. If you use a Codec for encryption, it also encrypts Nexus payloads. Caller and handler Workers must have compatible Data Converters. Payloads are encrypted by the sender (caller encrypts input, handler encrypts result). Three common approaches for cross-Namespace payload encryption: ### Option 1: Same encryption key[​](https://docs.temporal.io/nexus/security#same-encryption-key "Direct link to Option 1: Same encryption key") Both Namespaces share the same encryption key. Simplest approach - no additional configuration needed. ### Option 2: Pass KMS key ID in payload metadata[​](https://docs.temporal.io/nexus/security#kms-key-id-metadata "Direct link to Option 2: Pass KMS key ID in payload metadata") Each Namespace uses its own encryption key, with the KMS key ID passed in Temporal payload metadata. The receiver reads the key ID from metadata and decrypts using KMS IAM permissions. Works bi-directionally: caller encrypts input with the caller's key, handler decrypts using the key ID from metadata, then encrypts the result with the handler's key. The Codec Server needs KMS decrypt permissions for all relevant keys. See the [encryption sample](https://github.com/temporalio/samples-go/blob/main/encryption/data_converter.go) and the [reference-app-orders-go data converter](https://github.com/temporalio/reference-app-orders-go/blob/main/app/temporalutil/data_converter.go) . ### Option 3: Wrapper types for endpoint-specific encryption keys[​](https://docs.temporal.io/nexus/security#endpoint-specific-keys "Direct link to Option 3: Wrapper types for endpoint-specific encryption keys") Use wrapper types (for example, `EndpointValue`) so the Data Converter selects an Endpoint-specific encryption key. This encrypts only Nexus traffic with a dedicated key, without sharing Namespace keys across teams. See the [draft endpoint-based encryption sample](https://github.com/temporalio/samples-go/compare/main...bergundy:samples-go:nexus-encryption-by-endpoint) . ### Choosing an approach[​](https://docs.temporal.io/nexus/security#choosing-an-approach "Direct link to Choosing an approach") Options 1 and 2, where both sides share the same key or flow the KMS key ID in payload metadata, work with the standard Data Converter. Option 3 is more advanced and is intended for teams that don't want to share their Namespace encryption keys with other teams. Nexus Registry security[​](https://docs.temporal.io/nexus/security#managing-nexus-endpoints "Direct link to Nexus Registry security") -------------------------------------------------------------------------------------------------------------------------------------- See [Nexus Registry Roles and Permissions](https://docs.temporal.io/nexus/registry#roles-and-permissions) . * [Runtime access controls](https://docs.temporal.io/nexus/security#runtime-access-controls) * [Secure connectivity](https://docs.temporal.io/nexus/security#secure-connectivity) * [Payload encryption and Data Converter](https://docs.temporal.io/nexus/security#payload-encryption-data-converter) * [Option 1: Same encryption key](https://docs.temporal.io/nexus/security#same-encryption-key) * [Option 2: Pass KMS key ID in payload metadata](https://docs.temporal.io/nexus/security#kms-key-id-metadata) * [Option 3: Wrapper types for endpoint-specific encryption keys](https://docs.temporal.io/nexus/security#endpoint-specific-keys) * [Choosing an approach](https://docs.temporal.io/nexus/security#choosing-an-approach) * [Nexus Registry security](https://docs.temporal.io/nexus/security#managing-nexus-endpoints) --- # Nexus services | Temporal Platform Documentation [Skip to main content](https://docs.temporal.io/nexus/services#__docusaurus_skipToContent_fallback) SUPPORT, STABILITY, and DEPENDENCY INFO Temporal Nexus is now [Generally Available](https://docs.temporal.io/evaluate/development-production-features/release-stages#general-availability) for [Temporal Cloud](https://docs.temporal.io/cloud/nexus) and [self-hosted deployments](https://docs.temporal.io/production-deployment/self-hosted-guide/nexus) . [Nexus Services](https://docs.temporal.io/glossary#nexus-service) are named collections of [Nexus Operations](https://docs.temporal.io/nexus/operations) that provide a contract for sharing across team boundaries. A [Nexus Endpoint](https://docs.temporal.io/nexus/endpoints) exposes Services for callers to use. Services are registered in a Worker that polls the Endpoint's target Task Queue. Multiple Services can run in the same Worker. Services typically run alongside the Workflows they abstract, or in a dedicated router Worker using the [router-queue pattern](https://docs.temporal.io/nexus/patterns#router-queue-pattern) . Callers reference a Service by name when executing a Nexus Operation. --- # Detecting Workflow failures | Temporal Platform Documentation [Skip to main content](https://docs.temporal.io/encyclopedia/detecting-workflow-failures#__docusaurus_skipToContent_fallback) On this page Each Workflow Timeout controls the maximum duration of a different aspect of a Workflow Execution. Workflow Timeouts are set when starting the Workflow Execution. Before we continue, we want to note that we generally do not recommend setting Workflow Timeouts, because Workflows are designed to be long-running and resilient. Instead, setting a Timeout can limit its ability to handle unexpected delays or long-running processes. If you need to perform an action inside your Workflow after a specific period of time, we recommend using a Timer. * [Workflow Execution Timeout](https://docs.temporal.io/encyclopedia/detecting-workflow-failures#workflow-execution-timeout) * [Workflow Run Timeout](https://docs.temporal.io/encyclopedia/detecting-workflow-failures#workflow-run-timeout) * [Workflow Task Timeout](https://docs.temporal.io/encyclopedia/detecting-workflow-failures#workflow-task-timeout) Workflow Execution Timeout[​](https://docs.temporal.io/encyclopedia/detecting-workflow-failures#workflow-execution-timeout "Direct link to Workflow Execution Timeout") ------------------------------------------------------------------------------------------------------------------------------------------------------------------------ **What is a Workflow Execution Timeout in Temporal?** A Workflow Execution Timeout is the maximum time that a Workflow Execution can be executing (have an Open status) including retries and any usage of Continue As New. Related 📚 * [![](https://docs.temporal.io/img/sdks/svgs/golang.svg)Set a Workflow Execution Timeout using the Go SDK](https://docs.temporal.io/develop/go/workflows/timeouts) feature-guide * [![](https://docs.temporal.io/img/sdks/svgs/java.svg)Set a Workflow Execution Timeout using the Java SDK](https://docs.temporal.io/develop/java/workflows/timeouts) feature-guide * [![](https://docs.temporal.io/img/sdks/svgs/php.svg)Set a Workflow Execution Timeout using the PHP SDK](https://docs.temporal.io/develop/php/workflows/timeouts#workflow-timeouts) feature-guide * [![](https://docs.temporal.io/img/sdks/svgs/python.svg)Set a Workflow Execution Timeout using the Python SDK](https://docs.temporal.io/develop/python/workflows/timeouts) feature-guide * [![](https://docs.temporal.io/img/sdks/svgs/typescript.svg)Set a Workflow Execution Timeout using the TypeScript SDK](https://docs.temporal.io/develop/typescript/workflows/timeouts) feature-guide * [![](https://docs.temporal.io/img/sdks/svgs/dotnet.svg)Set a Workflow Execution Timeout using the .NET SDK](https://docs.temporal.io/develop/dotnet/workflows/timeouts) feature-guide ![Workflow Execution Timeout period](https://docs.temporal.io/diagrams/workflow-execution-timeout.svg) Workflow Execution Timeout period **The default value is ∞ (infinite).** If this timeout is reached, the Workflow Execution changes to a Timed Out status. This timeout is different from the [Workflow Run Timeout](https://docs.temporal.io/encyclopedia/detecting-workflow-failures#workflow-run-timeout) . This timeout is most commonly used for stopping the execution of a [Temporal Cron Job](https://docs.temporal.io/cron-job) after a certain amount of time has passed. Workflow Run Timeout[​](https://docs.temporal.io/encyclopedia/detecting-workflow-failures#workflow-run-timeout "Direct link to Workflow Run Timeout") ------------------------------------------------------------------------------------------------------------------------------------------------------ **What is a Workflow Run Timeout in Temporal?** A Workflow Run is the instance of a specific Workflow Execution. Due to the potential for Workflow Retries or Continue-as-New, a Workflow Execution may have multiple Workflow runs. For example, if a Workflow that specifies a Retry Policy initially fails and then succeeds during the next retry attempt, there is a single Workflow Execution that spans two Workflow Runs. Both runs will share the same Workflow ID but have a unique Run ID to distinguish them. A Workflow Run Timeout restricts the maximum duration of a single Workflow Run. If the Workflow Run Timeout is reached, the Workflow Execution will be Timed Out. Because this Timeout only applies to an individual Workflow Run, this does not include retries or Continue-As-New. Related 📚 * [![](https://docs.temporal.io/img/sdks/svgs/golang.svg)Set a Workflow Run Timeout using the Go SDK](https://docs.temporal.io/develop/go/workflows/timeouts) feature-guide * [![](https://docs.temporal.io/img/sdks/svgs/java.svg)Set a Workflow Run Timeout using the Java SDK](https://docs.temporal.io/develop/java/workflows/timeouts) feature-guide * [![](https://docs.temporal.io/img/sdks/svgs/php.svg)Set a Workflow Run Timeout using the PHP SDK](https://docs.temporal.io/develop/php/workflows/timeouts#workflow-timeouts) feature-guide * [![](https://docs.temporal.io/img/sdks/svgs/python.svg)Set a Workflow Run Timeout using the Python SDK](https://docs.temporal.io/develop/python/workflows/timeouts) feature-guide * [![](https://docs.temporal.io/img/sdks/svgs/typescript.svg)Set a Workflow Run Timeout using the TypeScript SDK](https://docs.temporal.io/develop/typescript/workflows/timeouts) feature-guide * [![](https://docs.temporal.io/img/sdks/svgs/dotnet.svg)Set a Workflow Timeout using the .NET SDK](https://docs.temporal.io/develop/dotnet/workflows/timeouts) feature-guide ![Workflow Run Timeout period](https://docs.temporal.io/diagrams/workflow-run-timeout.svg) Workflow Run Timeout period **The default is set to the same value as the [Workflow Execution Timeout](https://docs.temporal.io/encyclopedia/detecting-workflow-failures#workflow-execution-timeout) .** This timeout is most commonly used to limit the execution time of a single [Temporal Cron Job Execution](https://docs.temporal.io/cron-job) . If the Workflow Run Timeout is reached, the Workflow Execution will be Timed Out. Workflow Task Timeout[​](https://docs.temporal.io/encyclopedia/detecting-workflow-failures#workflow-task-timeout "Direct link to Workflow Task Timeout") --------------------------------------------------------------------------------------------------------------------------------------------------------- **What is a Workflow Task Timeout in Temporal?** A Workflow Task Timeout is the maximum amount of time allowed for a [Worker](https://docs.temporal.io/workers#worker) to execute a [Workflow Task](https://docs.temporal.io/tasks#workflow-task) after the Worker has pulled that Workflow Task from the [Task Queue](https://docs.temporal.io/task-queue) . This Timeout is primarily available to recognize whether a Worker has gone down so that the Workflow Execution can be recovered on a different Worker. Related 📚 * [![](https://docs.temporal.io/img/sdks/svgs/golang.svg)Set a Workflow Task Timeout using the Go SDK](https://docs.temporal.io/develop/go/workflows/timeouts) feature-guide * [![](https://docs.temporal.io/img/sdks/svgs/java.svg)Set a Workflow Task Timeout using the Java SDK](https://docs.temporal.io/develop/java/workflows/timeouts) feature-guide * [![](https://docs.temporal.io/img/sdks/svgs/php.svg)Set a Workflow Task Timeout using the PHP SDK](https://docs.temporal.io/develop/php/workflows/timeouts#workflow-timeouts) feature-guide * [![](https://docs.temporal.io/img/sdks/svgs/python.svg)Set a Workflow Task Timeout using the Python SDK](https://docs.temporal.io/develop/python/workflows/timeouts) feature-guide * [![](https://docs.temporal.io/img/sdks/svgs/typescript.svg)Set a Workflow Task Timeout using the TypeScript SDK](https://docs.temporal.io/develop/typescript/workflows/timeouts) feature-guide * [![](https://docs.temporal.io/img/sdks/svgs/dotnet.svg)Set a Workflow Task Timeout using the .NET SDK](https://docs.temporal.io/develop/dotnet/workflows/timeouts) feature-guide ![Workflow Task Timeout period](https://docs.temporal.io/diagrams/workflow-task-timeout.svg) Workflow Task Timeout period **The default value is 10 seconds.** This timeout is primarily available to recognize whether a Worker has gone down so that the Workflow Execution can be recovered on a different Worker. The main reason for increasing the default value is to accommodate a Workflow Execution that has an extensive Workflow Execution History, requiring more than 10 seconds for the Worker to load. It's worth mentioning that although you can extend the timeout up to the maximum value of 120 seconds, it's not recommended to move beyond the default value. Detecting Workflow Task Failures[​](https://docs.temporal.io/encyclopedia/detecting-workflow-failures#detecting-workflow-task-failures "Direct link to Detecting Workflow Task Failures") ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ Use the `TemporalReportedProblems` Search Attribute to detect Workflows with failed Workflow Tasks. A failed Workflow Task does not cause the Workflow to fail. Some Tasks within a Workflow may be intended to fail. For example, a Workflow Task may check a remote data source for new messages. If there aren't any, the Task will fail as intended. If your Task has code to handle the failure, the Workflow will proceed. However, if your Workflow has a Task that fails and the failure is not handled, the Workflow will continue to run, but will not complete. Detecting Workflows in this state is a common troubleshooting issue. To identify Workflows with Task failures, you can use the Temporal Web UI. See [Task Failures View](https://docs.temporal.io/web-ui/#task-failures-view) for more details. You can also detect Workflows with Task failures by searching for the `TemporalReportedProblems` search attribute with your observability tools. Activating Workflow Task Failure To enable the Task Failures View for a Namespace, you need to update the Dynamic Config for that Namespace. See [Activating Task Failures View](https://docs.temporal.io/web-ui/#activate-task-failures-view) . * [Workflow Execution Timeout](https://docs.temporal.io/encyclopedia/detecting-workflow-failures#workflow-execution-timeout) * [Workflow Run Timeout](https://docs.temporal.io/encyclopedia/detecting-workflow-failures#workflow-run-timeout) * [Workflow Task Timeout](https://docs.temporal.io/encyclopedia/detecting-workflow-failures#workflow-task-timeout) * [Detecting Workflow Task Failures](https://docs.temporal.io/encyclopedia/detecting-workflow-failures#detecting-workflow-task-failures) --- # Detecting Activity failures | Temporal Platform Documentation [Skip to main content](https://docs.temporal.io/encyclopedia/detecting-activity-failures#__docusaurus_skipToContent_fallback) On this page A Workflow can detect different kinds of Activity Execution failures through the following timeouts: * [Schedule-To-Start Timeout](https://docs.temporal.io/encyclopedia/detecting-activity-failures#schedule-to-start-timeout) * [Start-To-Close Timeout](https://docs.temporal.io/encyclopedia/detecting-activity-failures#start-to-close-timeout) * [Schedule-To-Close Timeout](https://docs.temporal.io/encyclopedia/detecting-activity-failures#schedule-to-close-timeout) * [Activity Heartbeats](https://docs.temporal.io/encyclopedia/detecting-activity-failures#activity-heartbeat) Schedule-To-Start Timeout[​](https://docs.temporal.io/encyclopedia/detecting-activity-failures#schedule-to-start-timeout "Direct link to Schedule-To-Start Timeout") --------------------------------------------------------------------------------------------------------------------------------------------------------------------- **What is a Schedule-To-Start Timeout in Temporal?** A Schedule-To-Start Timeout is the maximum amount of time that is allowed from when an [Activity Task](https://docs.temporal.io/tasks#activity-task) is scheduled (that is, placed in a Task Queue) to when a [Worker](https://docs.temporal.io/workers#worker) starts (that is, picks up from the Task Queue) that Activity Task. In other words, it's a limit for how long an Activity Task can be enqueued. Related 📚 * [![](https://docs.temporal.io/img/sdks/svgs/golang.svg)Set a Schedule-To-Start Timeout using the Go SDK](https://docs.temporal.io/develop/go/activities/timeouts) feature-guide * [![](https://docs.temporal.io/img/sdks/svgs/java.svg)Set a Schedule-To-Start Timeout using the Java SDK](https://docs.temporal.io/develop/java/activities/timeouts) feature-guide * [![](https://docs.temporal.io/img/sdks/svgs/php.svg)Set a Schedule-To-Start Timeout using the PHP SDK](https://docs.temporal.io/develop/php/activities/timeouts#activity-timeouts) feature-guide * [![](https://docs.temporal.io/img/sdks/svgs/python.svg)Set a Schedule-To-Start Timeout using the Python SDK](https://docs.temporal.io/develop/python/activities/timeouts) feature-guide * [![](https://docs.temporal.io/img/sdks/svgs/typescript.svg)Set a Schedule-To-Start Timeout using the TypeScript SDK](https://docs.temporal.io/develop/typescript/activities/timeouts) feature-guide * [![](https://docs.temporal.io/img/sdks/svgs/dotnet.svg)Set a Schedule-To-Start Timeout using the .NET SDK](https://docs.temporal.io/develop/dotnet/activities/timeouts) feature-guide The moment that the Task is picked by the Worker, from the Task Queue, is considered to be the start of the Activity Task Execution for the purposes of the Schedule-To-Start Timeout and associated metrics. This definition of "Start" avoids issues that a clock difference between the Temporal Service and a Worker might create. ![Schedule-To-Start Timeout period](https://docs.temporal.io/diagrams/schedule-to-start-timeout.svg) Schedule-To-Start Timeout period "Schedule" in Schedule-To-Start and Schedule-To-Close have different frequency guarantees. The Schedule-To-Start Timeout is enforced for each Activity Task, whereas the Schedule-To-Close Timeout is enforced once per Activity Execution. Thus, "Schedule" in Schedule-To-Start refers to the scheduling moment of _every_ Activity Task in the sequence of Activity Tasks that make up the Activity Execution, while "Schedule" in Schedule-To-Close refers to the _first_ Activity Task in that sequence. A [Retry Policy](https://docs.temporal.io/encyclopedia/retry-policies) attached to an Activity Execution retries an Activity Task. ![Start-To-Close Timeout period with retries](https://docs.temporal.io/diagrams/schedule-to-start-timeout-with-retry.svg) Start-To-Close Timeout period with retries This timeout has two primary use cases: 1. Detect whether an individual Worker has crashed. 2. Detect whether the fleet of Workers polling the Task Queue is not able to keep up with the rate of Activity Tasks. **The default Schedule-To-Start Timeout is ∞ (infinity).** If this timeout is used, we recommend setting this timeout to the maximum time a Workflow Execution is willing to wait for an Activity Execution in the presence of all possible Worker outages, and have a concrete plan in place to reroute Activity Tasks to a different Task Queue. This timeout **does not** trigger any retries regardless of the Retry Policy, as a retry would place the Activity Task back into the same Task Queue. We do not recommend using this timeout unless you know what you are doing. In most cases, we recommend monitoring the `temporal_activity_schedule_to_start_latency` metric to know when Workers slow down picking up Activity Tasks, instead of setting this timeout. Start-To-Close Timeout[​](https://docs.temporal.io/encyclopedia/detecting-activity-failures#start-to-close-timeout "Direct link to Start-To-Close Timeout") ------------------------------------------------------------------------------------------------------------------------------------------------------------ **What is a Start-To-Close Timeout in Temporal?** A Start-To-Close Timeout is the maximum time allowed for a single [Activity Task Execution](https://docs.temporal.io/tasks#activity-task-execution) . Related 📚 * [![](https://docs.temporal.io/img/sdks/svgs/golang.svg)Set a Start-To-Close Timeout using the Go SDK](https://docs.temporal.io/develop/go/activities/timeouts) feature-guide * [![](https://docs.temporal.io/img/sdks/svgs/java.svg)Set a Start-To-Close Timeout using the Java SDK](https://docs.temporal.io/develop/java/activities/timeouts) feature-guide * [![](https://docs.temporal.io/img/sdks/svgs/php.svg)Set a Start-To-Close Timeout using the PHP SDK](https://docs.temporal.io/develop/php/activities/timeouts#activity-timeouts) feature-guide * [![](https://docs.temporal.io/img/sdks/svgs/python.svg)Set a Start-To-Close Timeout using the Python SDK](https://docs.temporal.io/develop/python/activities/timeouts) feature-guide * [![](https://docs.temporal.io/img/sdks/svgs/typescript.svg)Set a Start-To-Close Timeout using the TypeScript SDK](https://docs.temporal.io/develop/typescript/activities/timeouts) feature-guide * [![](https://docs.temporal.io/img/sdks/svgs/dotnet.svg)Set a Start-To-Close Timeout using the .NET SDK](https://docs.temporal.io/develop/dotnet/activities/timeouts) feature-guide **The default Start-To-Close Timeout is the same as the default [Schedule-To-Close Timeout](https://docs.temporal.io/encyclopedia/detecting-activity-failures#schedule-to-close-timeout) .** An Activity Execution must have either this timeout (Start-To-Close) or the [Schedule-To-Close Timeout](https://docs.temporal.io/encyclopedia/detecting-activity-failures#schedule-to-close-timeout) set. We recommend always setting this timeout; however, make sure that Start-To-Close Timeout is always set to be longer than the maximum possible time for the Activity Execution to complete. For long running Activity Executions, we recommend also using [Activity Heartbeats](https://docs.temporal.io/encyclopedia/detecting-activity-failures#activity-heartbeat) and [Heartbeat Timeouts](https://docs.temporal.io/encyclopedia/detecting-activity-failures#heartbeat-timeout) . tip We strongly recommend setting a Start-To-Close Timeout. The Temporal Server doesn't detect failures when a Worker loses communication with the Server or crashes. Therefore, the Temporal Server relies on the Start-To-Close Timeout to force Activity retries. The main use case for the Start-To-Close timeout is to detect when a Worker crashes after it has started executing an Activity Task. ![Start-To-Close Timeout period](https://docs.temporal.io/diagrams/start-to-close-timeout.svg) Start-To-Close Timeout period A [Retry Policy](https://docs.temporal.io/encyclopedia/retry-policies) attached to an Activity Execution retries an Activity Task Execution. Thus, the Start-To-Close Timeout is applied to each Activity Task Execution within an Activity Execution. If the first Activity Task Execution returns an error the first time, then the full Activity Execution might look like this: ![Start-To-Close Timeout period with retries](https://docs.temporal.io/diagrams/start-to-close-timeout-with-retry.svg) Start-To-Close Timeout period with retries If this timeout is reached, the following actions occur: * An [ActivityTaskTimedOut](https://docs.temporal.io/references/events#activitytasktimedout) Event is written to the Workflow Execution's mutable state. * If a Retry Policy dictates a retry, the Temporal Service schedules another Activity Task. * The attempt count increments by 1 in the Workflow Execution's mutable state. * The Start-To-Close Timeout timer is reset. Schedule-To-Close Timeout[​](https://docs.temporal.io/encyclopedia/detecting-activity-failures#schedule-to-close-timeout "Direct link to Schedule-To-Close Timeout") --------------------------------------------------------------------------------------------------------------------------------------------------------------------- **What is a Schedule-To-Close Timeout in Temporal?** A Schedule-To-Close Timeout is the maximum amount of time allowed for the overall [Activity Execution](https://docs.temporal.io/activity-execution) , from when the first [Activity Task](https://docs.temporal.io/tasks#activity-task) is scheduled to when the last Activity Task, in the chain of Activity Tasks that make up the Activity Execution, reaches a Closed status. Related 📚 * [![](https://docs.temporal.io/img/sdks/svgs/golang.svg)Set a Schedule-To-Close Timeout using the Go SDK](https://docs.temporal.io/develop/go/activities/timeouts) feature-guide * [![](https://docs.temporal.io/img/sdks/svgs/java.svg)Set a Schedule-To-Close Timeout using the Java SDK](https://docs.temporal.io/develop/java/activities/timeouts) feature-guide * [![](https://docs.temporal.io/img/sdks/svgs/php.svg)Set a Schedule-To-Close Timeout using the PHP SDK](https://docs.temporal.io/develop/php/activities/timeouts#activity-timeouts) feature-guide * [![](https://docs.temporal.io/img/sdks/svgs/python.svg)Set a Schedule-To-Close Timeout using the Python SDK](https://docs.temporal.io/develop/python/activities/timeouts) feature-guide * [![](https://docs.temporal.io/img/sdks/svgs/typescript.svg)Set a Schedule-To-Close Timeout using the TypeScript SDK](https://docs.temporal.io/develop/typescript/activities/timeouts) feature-guide * [![](https://docs.temporal.io/img/sdks/svgs/dotnet.svg)Set a Schedule-To-Close Timeout using the .NET SDK](https://docs.temporal.io/develop/dotnet/activities/timeouts) feature-guide ![Schedule-To-Close Timeout period](https://docs.temporal.io/diagrams/schedule-to-close-timeout.svg) Schedule-To-Close Timeout period Example Schedule-To-Close Timeout period for an Activity Execution that has a chain Activity Task Executions: ![Schedule-To-Close Timeout period with a retry](https://docs.temporal.io/diagrams/schedule-to-close-timeout-with-retry.svg) Schedule-To-Close Timeout period with a retry **The default Schedule-To-Close Timeout is ∞ (infinity).** An Activity Execution must have either this timeout (Schedule-To-Close) or [Start-To-Close](https://docs.temporal.io/encyclopedia/detecting-activity-failures#start-to-close-timeout) set. This timeout can be used to control the overall duration of an Activity Execution in the face of failures (repeated Activity Task Executions), without altering the Maximum Attempts field of the Retry Policy. tip We strongly recommend setting a Start-To-Close Timeout. The Temporal Server doesn't detect failures when a Worker loses communication with the Server or crashes. Therefore, the Temporal Server relies on the Start-To-Close Timeout to force Activity retries. Activity Heartbeat[​](https://docs.temporal.io/encyclopedia/detecting-activity-failures#activity-heartbeat "Direct link to Activity Heartbeat") ------------------------------------------------------------------------------------------------------------------------------------------------ **What is an Activity Heartbeat in Temporal?** An Activity Heartbeat is a ping from the Worker that is executing the Activity to the Temporal Service. Each ping informs the Temporal Service that the Activity Execution is making progress and the Worker has not crashed. Related 📚 * [![](https://docs.temporal.io/img/sdks/svgs/golang.svg)Heartbeat an Activity using the Go SDK](https://docs.temporal.io/develop/go/activities/timeouts#activity-heartbeats) feature-guide * [![](https://docs.temporal.io/img/sdks/svgs/java.svg)Heartbeat an Activity using the Java SDK](https://docs.temporal.io/develop/java/activities/timeouts#activity-heartbeats) feature-guide * [![](https://docs.temporal.io/img/sdks/svgs/php.svg)Heartbeat an Activity using the PHP SDK](https://docs.temporal.io/develop/php/activities/timeouts#activity-heartbeats) feature-guide * [![](https://docs.temporal.io/img/sdks/svgs/python.svg)Heartbeat an Activity using the Python SDK](https://docs.temporal.io/develop/python/activities/timeouts#activity-heartbeats) feature-guide * [![](https://docs.temporal.io/img/sdks/svgs/typescript.svg)Heartbeat an Activity using the TypeScript SDK](https://docs.temporal.io/develop/typescript/activities/timeouts#activity-heartbeat-timeout) feature-guide * [![](https://docs.temporal.io/img/sdks/svgs/dotnet.svg)Heartbeat an Activity using the .NET SDK](https://docs.temporal.io/develop/dotnet/activities/timeouts#activity-heartbeats) feature-guide Activity Heartbeats work in conjunction with a [Heartbeat Timeout](https://docs.temporal.io/encyclopedia/detecting-activity-failures#heartbeat-timeout) . Activity Heartbeats are implemented within the Activity Definition. Custom progress information can be included in the Heartbeat which can then be used by the Activity Execution should a retry occur. An Activity Heartbeat can be recorded as often as needed (e.g. once a minute or every loop iteration). It is often a good practice to Heartbeat on anything but the shortest Activity Execution. Temporal SDKs control the rate at which Heartbeats are sent to the Temporal Service. Heartbeating is not required from [Local Activities](https://docs.temporal.io/local-activity) , and does nothing. For _long-running_ Activities, we recommend using a relatively short Heartbeat Timeout and a frequent Heartbeat. That way if a Worker fails it can be handled in a timely manner. A Heartbeat can include an application layer payload that can be used to _save_ Activity Execution progress. If an [Activity Task Execution](https://docs.temporal.io/tasks#activity-task-execution) times out due to a missed Heartbeat, the next Activity Task can access and continue with that payload. Activity Cancellations are delivered to Activities from the Temporal Service when they Heartbeat. Activities that don't Heartbeat can't receive a Cancellation. Heartbeat throttling may lead to Cancellation getting delivered later than expected. ### Throttling[​](https://docs.temporal.io/encyclopedia/detecting-activity-failures#throttling "Direct link to Throttling") Heartbeats may not always be sent to the Temporal Service—they may be throttled by the Worker. The throttle interval is the smaller of the following: * If `heartbeatTimeout` is provided, `heartbeatTimeout * 0.8`; otherwise, `defaultHeartbeatThrottleInterval` * `maxHeartbeatThrottleInterval` `defaultHeartbeatThrottleInterval` is 30 seconds by default, and `maxHeartbeatThrottleInterval` is 60 seconds by default. Each can be set in Worker options. Throttling is implemented as follows: * After sending a Heartbeat, the Worker sets a timer for the throttle interval. * The Worker stops sending Heartbeats, but continues receiving Heartbeats from the Activity and remembers the most recent one. * When the timer fires, the Worker: * Sends the most recent Heartbeat. * Sets the timer again. Throttling allows the Worker to reduce network traffic and load on the Temporal Service by suppressing Heartbeats that aren’t necessary to prevent a Heartbeat Timeout. Throttling does not apply to the final Heartbeat message in the case of Activity Failure. If an Activity fails just after recording progress information in a Heartbeat message, that progress information will be available during the next retry attempt, provided that the Worker itself did not crash before delivering it to the Temporal Service. ### Which Activities should Heartbeat?[​](https://docs.temporal.io/encyclopedia/detecting-activity-failures#which-activities-should-heartbeat "Direct link to Which Activities should Heartbeat?") Heartbeating is best thought about not in terms of time, but in terms of "How do you know you are making progress?" For short-term operations, progress updates are not a requirement. However, checking the progress and status of Activity Executions that run over long periods is almost always useful. Consider the following when setting Activity Heartbeats: * Your underlying task must be able to report definite progress. Note that your Workflow cannot read this progress information while the Activity is still executing (or it would have to store it in Event History). You can report progress to external sources if you need it exposed to the user. * Your Activity Execution is long-running, and you need to verify whether the Worker that is processing your Activity is still alive and has not run out of memory or silently crashed. For example, the following scenarios are suitable for Heartbeating: * Reading a large file from Amazon S3. * Running a ML training job on some local GPUs. And the following scenarios are not suitable for Heartbeating: * Making a quick API call. * Reading a small file from disk. ### Heartbeat Timeout[​](https://docs.temporal.io/encyclopedia/detecting-activity-failures#heartbeat-timeout "Direct link to Heartbeat Timeout") **What is a Heartbeat Timeout in Temporal?** A Heartbeat Timeout is the maximum time between [Activity Heartbeats](https://docs.temporal.io/encyclopedia/detecting-activity-failures#activity-heartbeat) . Related 📚 * [![](https://docs.temporal.io/img/sdks/svgs/golang.svg)Set a Heartbeat Timeout using the Go SDK](https://docs.temporal.io/develop/go/activities/timeouts#activity-heartbeats) feature-guide * [![](https://docs.temporal.io/img/sdks/svgs/java.svg)Set a Heartbeat Timeout using the Java SDK](https://docs.temporal.io/develop/java/activities/timeouts#heartbeat-timeout) feature-guide * [![](https://docs.temporal.io/img/sdks/svgs/php.svg)Set a Heartbeat Timeout using the PHP SDK](https://docs.temporal.io/develop/php/activities/timeouts#heartbeat-timeout) feature-guide * [![](https://docs.temporal.io/img/sdks/svgs/python.svg)Set a Heartbeat Timeout using the Python SDK](https://docs.temporal.io/develop/python/activities/timeouts#heartbeat-timeout) feature-guide * [![](https://docs.temporal.io/img/sdks/svgs/typescript.svg)Set a Heartbeat Timeout using the TypeScript SDK](https://docs.temporal.io/develop/typescript/activities/timeouts#activity-heartbeat-timeout) feature-guide * [![](https://docs.temporal.io/img/sdks/svgs/dotnet.svg)Set a Heartbeat Timeout using the .NET SDK](https://docs.temporal.io/develop/dotnet/activities/timeouts#heartbeat-timeout) feature-guide ![Heartbeat Timeout periods](https://docs.temporal.io/diagrams/heartbeat-timeout.svg) Heartbeat Timeout periods If this timeout is reached, the Activity Task fails and a retry occurs if a [Retry Policy](https://docs.temporal.io/encyclopedia/retry-policies) dictates it. * [Schedule-To-Start Timeout](https://docs.temporal.io/encyclopedia/detecting-activity-failures#schedule-to-start-timeout) * [Start-To-Close Timeout](https://docs.temporal.io/encyclopedia/detecting-activity-failures#start-to-close-timeout) * [Schedule-To-Close Timeout](https://docs.temporal.io/encyclopedia/detecting-activity-failures#schedule-to-close-timeout) * [Activity Heartbeat](https://docs.temporal.io/encyclopedia/detecting-activity-failures#activity-heartbeat) * [Throttling](https://docs.temporal.io/encyclopedia/detecting-activity-failures#throttling) * [Which Activities should Heartbeat?](https://docs.temporal.io/encyclopedia/detecting-activity-failures#which-activities-should-heartbeat) * [Heartbeat Timeout](https://docs.temporal.io/encyclopedia/detecting-activity-failures#heartbeat-timeout) --- # Debugging - Temporal feature | Temporal Platform Documentation [Skip to main content](https://docs.temporal.io/evaluate/development-production-features/debugging#__docusaurus_skipToContent_fallback) Temporal offers powerful and efficient debugging capabilities for both development and production. These capabilities help developers inspect and troubleshoot Workflows and Activities with precision, ensuring that Workflows perform as expected. By leveraging detailed event histories and intuitive tooling, you can trace the execution path of Workflows, identify issues, and understand the state of your application at any given point in time. Jump straight to a Temporal SDK feature guide. Related 📚 * [![](https://docs.temporal.io/img/sdks/svgs/golang.svg)Debugging using the Go SDK](https://docs.temporal.io/develop/go/best-practices/debugging) feature-guide * [![](https://docs.temporal.io/img/sdks/svgs/java.svg)Debugging using the Java SDK](https://docs.temporal.io/develop/java/best-practices/debugging) feature-guide * [![](https://docs.temporal.io/img/sdks/svgs/php.svg)Debugging using the PHP SDK](https://docs.temporal.io/develop/php/best-practices/debugging) feature-guide * [![](https://docs.temporal.io/img/sdks/svgs/python.svg)Debugging using the Python SDK](https://docs.temporal.io/develop/python/best-practices/debugging) feature-guide * [![](https://docs.temporal.io/img/sdks/svgs/typescript.svg)Debugging using the TypeScript SDK](https://docs.temporal.io/develop/typescript/best-practices/debugging) feature-guide * [![](https://docs.temporal.io/img/sdks/svgs/dotnet.svg)Debugging using the .NET SDK](https://docs.temporal.io/develop/dotnet/best-practices/debugging) feature-guide * [![](https://docs.temporal.io/img/sdks/svgs/ruby.svg)Debugging using the Ruby SDK](https://docs.temporal.io/develop/ruby/best-practices/debugging) feature-guide --- # Client - TypeScript SDK | Temporal Platform Documentation [Skip to main content](https://docs.temporal.io/develop/typescript/client#__docusaurus_skipToContent_fallback) On this page ![TypeScript SDK Banner](https://docs.temporal.io/assets/images/banner-typescript-temporal-d8a24070726a0d14cb4d1aab011db927.png) Temporal Client[​](https://docs.temporal.io/develop/typescript/client#temporal-client "Direct link to Temporal Client") ------------------------------------------------------------------------------------------------------------------------ * [Temporal Client](https://docs.temporal.io/develop/typescript/client/temporal-client) * [Namespaces](https://docs.temporal.io/develop/typescript/client/namespaces) * [Temporal Client](https://docs.temporal.io/develop/typescript/client#temporal-client) --- # AI integrations | Temporal Platform Documentation [Skip to main content](https://docs.temporal.io/develop/typescript/integrations#__docusaurus_skipToContent_fallback) The following AI framework integrations are available for the Temporal TypeScript SDK: | Framework | SDK docs | Integration guide | | --- | --- | --- | | AI SDK by Vercel | [ai-sdk.dev](https://ai-sdk.dev/docs/introduction) | [Guide](https://docs.temporal.io/develop/typescript/integrations/ai-sdk) | | Braintrust | [braintrust.dev](https://braintrust.dev/docs) | [Guide](https://www.braintrust.dev/docs/integrations/sdk-integrations/temporal#typescript) | --- # Best Practices - Ruby SDK | Temporal Platform Documentation [Skip to main content](https://docs.temporal.io/develop/ruby/best-practices#__docusaurus_skipToContent_fallback) On this page ![Ruby SDK Banner](https://docs.temporal.io/assets/images/banner-ruby-temporal-be833f13b8e3655d7a8d4e50119b7da2.png) Best practices[​](https://docs.temporal.io/develop/ruby/best-practices#best-practices "Direct link to Best practices") ----------------------------------------------------------------------------------------------------------------------- * [Error handling](https://docs.temporal.io/develop/ruby/best-practices/error-handling) * [Testing](https://docs.temporal.io/develop/ruby/best-practices/testing-suite) * [Debugging](https://docs.temporal.io/develop/ruby/best-practices/debugging) * [Converters and encryption](https://docs.temporal.io/develop/ruby/best-practices/converters-and-encryption) * [Best practices](https://docs.temporal.io/develop/ruby/best-practices#best-practices) --- # Enriching the user interface - TypeScript SDK | Temporal Platform Documentation [Skip to main content](https://docs.temporal.io/develop/typescript/platform/enriching-ui#__docusaurus_skipToContent_fallback) On this page Temporal supports adding context to Workflows and Events with metadata. This helps users identify and understand Workflows and their operations. Adding Summary and Details to Workflows[​](https://docs.temporal.io/develop/typescript/platform/enriching-ui#adding-summary-and-details-to-workflows "Direct link to Adding Summary and Details to Workflows") --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- ### Starting a Workflow[​](https://docs.temporal.io/develop/typescript/platform/enriching-ui#starting-a-workflow "Direct link to Starting a Workflow") When starting a Workflow, you can provide a static summary and details to help identify the workflow in the UI: import { Client } from '@temporalio/client';const client = new Client();// Start a workflow with static summary and detailsconst handle = await client.workflow.start(yourWorkflow, { args: ['workflow input'], taskQueue: 'your-task-queue', workflowId: 'your-workflow-id', staticSummary: 'Order processing for customer #12345', staticDetails: 'Processing premium order with expedited shipping'}); `staticSummary` is a single-line description that appears in the workflow list view, limited to 200 bytes. `staticDetails` can be multi-line and provides more comprehensive information that appears in the workflow details view, with a larger limit of 20K bytes. The input format is standard Markdown excluding images, HTML, and scripts. You can also use the `execute` method with the same parameters: const result = await client.workflow.execute(yourWorkflow, { args: ['workflow input'], taskQueue: 'your-task-queue', workflowId: 'your-workflow-id', staticSummary: 'Order processing for customer #12345', staticDetails: 'Processing premium order with expedited shipping'}); ### Inside the Workflow[​](https://docs.temporal.io/develop/typescript/platform/enriching-ui#inside-the-workflow "Direct link to Inside the Workflow") Within a Workflow, you can get and set the _current workflow details_. Unlike static summary/details set at Workflow start, this value can be updated throughout the life of the Workflow. Current Workflow details also takes Markdown format (excluding images, HTML, and scripts) and can span multiple lines. import { getCurrentDetails, setCurrentDetails } from '@temporalio/workflow';export async function yourWorkflow(input: string): Promise { // Get the current details const currentDetails = getCurrentDetails(); console.log(`Current details: ${currentDetails}`); // Set/update the current details setCurrentDetails('Updated workflow details with new status'); return 'Workflow completed';} ### Adding Summary to Activities and Timers[​](https://docs.temporal.io/develop/typescript/platform/enriching-ui#adding-summary-to-activities-and-timers "Direct link to Adding Summary to Activities and Timers") You can attach a `summary` to activities by using `executeWithOptions` when calling them: import { proxyActivities } from '@temporalio/workflow';import type * as activities from './activities';const { yourActivity } = proxyActivities({ startToCloseTimeout: '10 seconds'});export async function yourWorkflow(input: string): Promise { // Execute an activity with a summary using executeWithOptions const result = await yourActivity.executeWithOptions( { staticSummary: 'Processing user data' }, [input] // Note: arguments must be passed as an array ); return result;} Similarly, you can attach a `summary` to timers within a workflow: import { sleep } from '@temporalio/workflow';export async function yourWorkflow(input: string): Promise { // Create a timer with a summary await sleep('5 minutes', { summary: 'Waiting for payment confirmation' }); return 'Timer completed';} The input format for `summary` is a string, and limited to 200 bytes. Viewing Summary and Details in the UI[​](https://docs.temporal.io/develop/typescript/platform/enriching-ui#viewing-summary-and-details-in-the-ui "Direct link to Viewing Summary and Details in the UI") --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Once you've added summaries and details to your Workflows, Activities, and Timers, you can view this enriched information in the Temporal Web UI. Navigate to your Workflow's details page to see the metadata displayed in two key locations: ### Workflow Overview Section[​](https://docs.temporal.io/develop/typescript/platform/enriching-ui#workflow-overview-section "Direct link to Workflow Overview Section") At the top of the workflow details page, you'll find the workflow-level metadata: * **Summary & Details** - Displays the static summary and static details set when starting the workflow * **Current Details** - Displays the dynamic details that can be updated during workflow execution All Workflow details support standard Markdown formatting (excluding images, HTML, and scripts), allowing you to create rich, structured information displays. ### Event History[​](https://docs.temporal.io/develop/typescript/platform/enriching-ui#event-history "Direct link to Event History") Individual events in the Workflow's Event History display their associated summaries when available. Workflow, Activity and Timer summaries appear in purple text next to their corresponding Events, providing immediate context without requiring you to expand the event details. When you do expand an Event, the summary is also prominently displayed in the detailed view. * [Adding Summary and Details to Workflows](https://docs.temporal.io/develop/typescript/platform/enriching-ui#adding-summary-and-details-to-workflows) * [Starting a Workflow](https://docs.temporal.io/develop/typescript/platform/enriching-ui#starting-a-workflow) * [Inside the Workflow](https://docs.temporal.io/develop/typescript/platform/enriching-ui#inside-the-workflow) * [Adding Summary to Activities and Timers](https://docs.temporal.io/develop/typescript/platform/enriching-ui#adding-summary-to-activities-and-timers) * [Viewing Summary and Details in the UI](https://docs.temporal.io/develop/typescript/platform/enriching-ui#viewing-summary-and-details-in-the-ui) * [Workflow Overview Section](https://docs.temporal.io/develop/typescript/platform/enriching-ui#workflow-overview-section) * [Event History](https://docs.temporal.io/develop/typescript/platform/enriching-ui#event-history) --- # Dynamic handler | Temporal Platform Documentation [Skip to main content](https://docs.temporal.io/dynamic-handler#__docusaurus_skipToContent_fallback) On this page This page discusses [Dynamic Handler](https://docs.temporal.io/dynamic-handler#dynamic-handler) . What is a Dynamic Handler?[​](https://docs.temporal.io/dynamic-handler#dynamic-handler "Direct link to What is a Dynamic Handler?") ------------------------------------------------------------------------------------------------------------------------------------ Temporal supports Dynamic Workflows, Activities, Signals, and Queries. note Currently, the Temporal SDKs that support Dynamic Handlers are: * [Java](https://docs.temporal.io/develop/java/workflows/message-passing#dynamic-handler) * [Python](https://docs.temporal.io/develop/python/workflows/message-passing#dynamic-handler) * [.NET](https://docs.temporal.io/develop/dotnet/workflows/message-passing#dynamic-handler) * [Go](https://docs.temporal.io/develop/go/workflows/dynamic-workflow) * [Ruby](https://docs.temporal.io/develop/ruby/workflows/message-passing#dynamic-handler) These are unnamed handlers that are invoked if no other statically defined handler with the given name exists. Dynamic Handlers provide flexibility to handle cases where the names of Workflows, Activities, Signals, or Queries aren't known at run time. caution Dynamic Handlers should be used judiciously as a fallback mechanism rather than the primary approach. Overusing them can lead to maintainability and debugging issues down the line. Instead, Workflows, Activities, Signals, and Queries should be defined statically whenever possible, with clear names that indicate their purpose. Use static definitions as the primary way of structuring your Workflows. Reserve Dynamic Handlers for cases where the handler names are not known at compile time and need to be looked up dynamically at runtime. They are meant to handle edge cases and act as a catch-all, not as the main way of invoking logic. * [What is a Dynamic Handler?](https://docs.temporal.io/dynamic-handler#dynamic-handler) --- # Entity pattern - TypeScript SDK | Temporal Platform Documentation [Skip to main content](https://docs.temporal.io/develop/typescript/best-practices/entity-pattern#__docusaurus_skipToContent_fallback) On this page ### Single-entity design pattern in TypeScript[​](https://docs.temporal.io/develop/typescript/best-practices/entity-pattern#single-entity-pattern "Direct link to Single-entity design pattern in TypeScript") The following is a simple pattern that represents a single entity. It tracks the number of iterations regardless of frequency, and calls `continueAsNew` while properly handling pending updates from Signals. interface Input { /* Define your Workflow input type here */}interface Update { /* Define your Workflow update type here */}const MAX_ITERATIONS = 1;export async function entityWorkflow( input: Input, isNew = true,): Promise { try { const pendingUpdates = Array(); setHandler(updateSignal, (updateCommand) => { pendingUpdates.push(updateCommand); }); if (isNew) { await setup(input); } for (let iteration = 1; iteration <= MAX_ITERATIONS; ++iteration) { // Ensure that we don't block the Workflow Execution forever waiting // for updates, which means that it will eventually Continue-As-New // even if it does not receive updates. await condition(() => pendingUpdates.length > 0, '1 day'); while (pendingUpdates.length) { const update = pendingUpdates.shift(); await runAnActivityOrChildWorkflow(update); } } } catch (err) { if (isCancellation(err)) { await CancellationScope.nonCancellable(async () => { await cleanup(); }); } throw err; } await continueAsNew(input, false);} * [Single-entity design pattern in TypeScript](https://docs.temporal.io/develop/typescript/best-practices/entity-pattern#single-entity-pattern) --- # Multi-tenancy - Temporal feature | Temporal Platform Documentation [Skip to main content](https://docs.temporal.io/evaluate/development-production-features/multi-tenancy#__docusaurus_skipToContent_fallback) On this page Multi-tenancy in Temporal operates at two levels: Namespace isolation[​](https://docs.temporal.io/evaluate/development-production-features/multi-tenancy#namespace-isolation "Direct link to Namespace isolation") ----------------------------------------------------------------------------------------------------------------------------------------------------------------- [Namespaces](https://docs.temporal.io/namespaces) are Temporal's unit of isolation, providing logical separation for multi-tenant deployments in both open source Temporal and Temporal Cloud. ### Open source Temporal[​](https://docs.temporal.io/evaluate/development-production-features/multi-tenancy#open-source-temporal "Direct link to Open source Temporal") Namespaces in self-hosted Temporal provide: * **Workflow ID uniqueness**: Temporal guarantees unique Workflow IDs within a Namespace. Different Namespaces can have Workflows with the same ID without conflict. * **Resource isolation**: Traffic from one Namespace does not impact other Namespaces on the same Temporal Service. * **Configuration boundaries**: Settings like [Retention Period](https://docs.temporal.io/temporal-service/temporal-server#retention-period) and [Archival](https://docs.temporal.io/temporal-service/archival) destination are configured per Namespace. * **Access control**: Use a custom [Authorizer](https://docs.temporal.io/self-hosted-guide/security#authorization) on your Frontend Service to restrict who can access each Namespace. * **Inter-namespace communication**: Use [Nexus](https://docs.temporal.io/evaluate/nexus) for controlled communication between Namespaces. ### Temporal Cloud[​](https://docs.temporal.io/evaluate/development-production-features/multi-tenancy#temporal-cloud "Direct link to Temporal Cloud") Temporal Cloud builds on these capabilities with additional isolation guarantees: * **Independent authentication** via [API keys](https://docs.temporal.io/cloud/api-keys) or [mTLS certificates](https://docs.temporal.io/cloud/certificates) * **Built-in [role-based access controls](https://docs.temporal.io/cloud/manage-access/roles-and-permissions#namespace-level-permissions) ** without custom Authorizer configuration * **Separate [rate limits](https://docs.temporal.io/cloud/limits#namespace-level) ** to prevent noisy neighbor problems * **[High availability replication](https://docs.temporal.io/cloud/high-availability) ** across regions Related 📚 * [Namespace Isolation Details](https://docs.temporal.io/cloud/security#namespace-isolation) * [Temporal Cloud Pricing](https://docs.temporal.io/cloud/pricing) Application multi-tenancy[​](https://docs.temporal.io/evaluate/development-production-features/multi-tenancy#application-multi-tenancy "Direct link to Application multi-tenancy") ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Many organizations use Temporal to power their own multi-tenant SaaS applications, isolating their customers' workloads using Task Queues, Search Attributes, and Worker design patterns. See the [multi-tenant application patterns guide](https://docs.temporal.io/production-deployment/multi-tenant-patterns) for detailed recommendations on architecting multi-tenant applications with Temporal. * [Namespace isolation](https://docs.temporal.io/evaluate/development-production-features/multi-tenancy#namespace-isolation) * [Open source Temporal](https://docs.temporal.io/evaluate/development-production-features/multi-tenancy#open-source-temporal) * [Temporal Cloud](https://docs.temporal.io/evaluate/development-production-features/multi-tenancy#temporal-cloud) * [Application multi-tenancy](https://docs.temporal.io/evaluate/development-production-features/multi-tenancy#application-multi-tenancy) --- # Schedules - Temporal feature | Temporal Platform Documentation [Skip to main content](https://docs.temporal.io/evaluate/development-production-features/schedules#__docusaurus_skipToContent_fallback) Temporal Schedules is a feature that allows you to "schedule" Temporal Workflows at specified times or intervals, adjusting for peak use. It offers a flexible way to automate and manage your Temporal Workflows, ensuring your business processes run smoothly and efficiently especially when handling time-sensitive tasks. 1. **Automate Repetitive Tasks:** Schedules automate repetitive tasks, reducing manual intervention and ensuring timely execution of business processes. 2. **Enhanced Workflow Control and Observability:** Gain complete control over your automation processes. With Schedules, you can create, backfill, delete, describe, list, pause, trigger, and update Workflow Executions. 3. **Flexible Timing:** Schedule Workflow Executions to run at regular intervals or specific future times, ensuring they execute precisely when needed. 4. **Reliable and Scalable:** Designed for reliability and scalability, Temporal Schedules handle the complexities of distributed systems while ensuring your Workflows run as intended, even during failures. 5. **Eliminate External Dependencies:** Schedules remove the need to integrate external scheduling systems. Jump straight to a Temporal SDK feature guide. Related 📚 * [![](https://docs.temporal.io/img/sdks/svgs/golang.svg)Schedules using the Go SDK](https://docs.temporal.io/develop/go/workflows/schedules) feature-guide * [![](https://docs.temporal.io/img/sdks/svgs/java.svg)Schedules using the Java SDK](https://docs.temporal.io/develop/java/workflows/schedules) feature-guide * [![](https://docs.temporal.io/img/sdks/svgs/php.svg)Schedules using the PHP SDK](https://docs.temporal.io/develop/php/workflows/schedules) feature-guide * [![](https://docs.temporal.io/img/sdks/svgs/python.svg)Schedules using the Python SDK](https://docs.temporal.io/develop/python/workflows/schedules) feature-guide * [![](https://docs.temporal.io/img/sdks/svgs/typescript.svg)Schedules using the TypeScript SDK](https://docs.temporal.io/develop/typescript/workflows/schedules) feature-guide * [![](https://docs.temporal.io/img/sdks/svgs/dotnet.svg)Schedules using the .NET SDK](https://docs.temporal.io/develop/dotnet/workflows/schedules) feature-guide * [![](https://docs.temporal.io/img/sdks/svgs/ruby.svg)Schedules using the Ruby SDK](https://docs.temporal.io/develop/ruby/workflows/schedules) feature-guide --- # Temporal Nexus - Temporal feature | Temporal Platform Documentation [Skip to main content](https://docs.temporal.io/evaluate/nexus#__docusaurus_skipToContent_fallback) On this page SUPPORT, STABILITY, and DEPENDENCY INFO Temporal Nexus is now [Generally Available](https://docs.temporal.io/evaluate/development-production-features/release-stages#general-availability) for [Temporal Cloud](https://docs.temporal.io/cloud/nexus) and [self-hosted deployments](https://docs.temporal.io/production-deployment/self-hosted-guide/nexus) . As Temporal adoption grows across teams, organizations partition their applications into isolated Namespaces for security and fault isolation. Nexus bridges these boundaries, connecting Temporal applications across Namespaces, regions, and clouds with built-in durable execution, observability, and access control. Each team retains ownership of their own Namespace while sharing capabilities through clean service contracts. Watch the [Nexus overview](https://www.youtube.com/watch?v=tJ1OwSFokOg&t=117s) for a walkthrough. Before Nexus[​](https://docs.temporal.io/evaluate/nexus#before-nexus "Direct link to Before Nexus") ---------------------------------------------------------------------------------------------------- Connecting Namespaces was possible, but painful. It required extensive configuration, added operational overhead, and often depended on additional infrastructure * **Child Workflows** - Limited to the same Namespace. Cross-Namespace use leaks underlying implementation details, requiring callers to manage the target Namespace, Task Queue, and Workflow options. * **Activity wrappers** - Require per-target mTLS clients, adding configuration and certificate management overhead. Often over-permissioned, lack built-in observability, and require error-prone boilerplate for async results. * **Extra gateway infrastructure** - Not durable, difficult to debug across services, and adds another service to manage and patch. Nexus replaces all of these with a clean service contract between caller and handler, reducing code, and providing first-class observability. Benefits[​](https://docs.temporal.io/evaluate/nexus#benefits "Direct link to Benefits") ---------------------------------------------------------------------------------------- Connect Temporal Applications across teams, domains, regions, and clouds with: * **Stronger security posture** - Built-in access controls for service contracts instead of broad Namespace access. Each team controls their own Namespace, Workers, and deployment lifecycle. * **Higher reliability** - Durable, atomic handoffs eliminate lost requests. Faults are isolated so misbehaving Workers don't impact other teams. * **Easier to build and maintain** - Less boilerplate code, custom retry and deduplication logic, and ongoing maintenance. Teams focus on business logic instead of infrastructure. * **Scalable platform patterns** - Enables cross-team and cross-region orchestration without centralizing ownership. * **Lower barriers to cross-team use cases** - Makes it easy to incrementally build and adopt shared services, with built-in discoverability. * **Compliance and data isolation** - Isolated Namespaces support auditability, data residency requirements, and dedicated encryption and access controls for sensitive data (PCI, PII). What customers are using Nexus for[​](https://docs.temporal.io/evaluate/nexus#what-customers-are-using-nexus-for "Direct link to What customers are using Nexus for") ---------------------------------------------------------------------------------------------------------------------------------------------------------------------- * **Duolingo** - Self-service infrastructure ([Case study](https://temporal.io/resources/case-studies/duolingo-temporal-nexus) | [Webinar](https://www.youtube.com/watch?v=tJ1OwSFokOg&t=524s) ) * **Netflix** - Infrastructure orchestration ([Replay talk](https://www.youtube.com/watch?v=izR9dQ_eIe4&t=470s) | [Webinar](https://www.youtube.com/watch?v=At1FfqGQiu0&t=1295s) ) * **Miro** - Cross-region data migration ([Replay talk](https://youtu.be/YLmFR-IAC3M?feature=shared&t=1488) ) Should I use Nexus?[​](https://docs.temporal.io/evaluate/nexus#should-i-use-nexus "Direct link to Should I use Nexus?") ------------------------------------------------------------------------------------------------------------------------ Use the following decision tree to help determine if Nexus is right for your use case: [![Should I use Nexus? Decision tree](https://docs.temporal.io/diagrams/nexusadoptionlight.svg)![Should I use Nexus? Decision tree](https://docs.temporal.io/diagrams/nexusadoptiondark.svg)](https://docs.temporal.io/diagrams/nexusadoptionlight.svg) Get started[​](https://docs.temporal.io/evaluate/nexus#learn-more "Direct link to Get started") ------------------------------------------------------------------------------------------------ Join the [#nexus](https://temporalio.slack.com/archives/C07LQN0JK9B) channel in [Temporal Slack](https://t.mp/slack) to connect with the Nexus community. Related 📚 * [![](https://docs.temporal.io/img/assets/link-preview-icon.svg)Nexus concepts and architecture](https://docs.temporal.io/nexus) encyclopedia * [![](https://docs.temporal.io/img/sdks/svgs/golang.svg)Go SDK - Nexus quick start](https://docs.temporal.io/develop/go/nexus) feature-guide * [![](https://docs.temporal.io/img/sdks/svgs/java.svg)Java SDK - Nexus quick start](https://docs.temporal.io/develop/java/nexus) feature-guide * [![](https://docs.temporal.io/img/sdks/svgs/python.svg)Python SDK - Nexus quick start](https://docs.temporal.io/develop/python/nexus) feature-guide * [![](https://docs.temporal.io/img/sdks/svgs/typescript.svg)TypeScript SDK - Nexus quick start](https://docs.temporal.io/develop/typescript/nexus) feature-guide * [![](https://docs.temporal.io/img/sdks/svgs/dotnet.svg).NET SDK - Nexus quick start](https://docs.temporal.io/develop/dotnet/nexus) feature-guide * [Temporal Cloud](https://docs.temporal.io/cloud/nexus) feature-guide * [Self-hosted deployment](https://docs.temporal.io/production-deployment/self-hosted-guide/nexus) feature-guide * [Before Nexus](https://docs.temporal.io/evaluate/nexus#before-nexus) * [Benefits](https://docs.temporal.io/evaluate/nexus#benefits) * [What customers are using Nexus for](https://docs.temporal.io/evaluate/nexus#what-customers-are-using-nexus-for) * [Should I use Nexus?](https://docs.temporal.io/evaluate/nexus#should-i-use-nexus) * [Get started](https://docs.temporal.io/evaluate/nexus#learn-more) --- # Manage Interceptors - TypeScript SDK | Temporal Platform Documentation [Skip to main content](https://docs.temporal.io/develop/typescript/workers/interceptors#__docusaurus_skipToContent_fallback) On this page Interceptors are SDK hooks that let you intercept inbound and outbound Temporal calls. You use them to apply shared behavior across many calls, such as tracing and authorization, before calls reach the application code and after they return. This is similar to middleware in other frameworks. There are two main types of interceptors--inbound and outbound. * Outbound interceptors wrap network calls, running before they reach the network and after they return. * Inbound interceptors run after the network hop, wrapping application code and running before it starts and after it returns. Those further break down into concrete Interceptor types--see below. How to implement interceptors in TypeScript[​](https://docs.temporal.io/develop/typescript/workers/interceptors#interceptors "Direct link to How to implement interceptors in TypeScript") ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Interceptors run as a chain. Each interceptor wraps the entire inner call: your code runs before the call, invokes `next` to execute the rest of the chain, and then runs after the call completes. This means you can inspect or modify both the `input` and the result, handle errors, and perform side effects at either stage. The TypeScript SDK comes with an optional interceptor package that adds tracing with [OpenTelemetry](https://www.npmjs.com/package/@temporalio/interceptors-opentelemetry) . See how to use it in the [interceptors-opentelemetry](https://github.com/temporalio/samples-typescript/tree/main/interceptors-opentelemetry) code sample. * [WorkflowInboundCallsInterceptor](https://typescript.temporal.io/api/interfaces/workflow.WorkflowInboundCallsInterceptor/) : Intercept Workflow inbound calls like execution, Signals, and Queries. * [WorkflowOutboundCallsInterceptor](https://typescript.temporal.io/api/interfaces/workflow.WorkflowOutboundCallsInterceptor/) : Intercept Workflow outbound calls to Temporal APIs like scheduling Activities and starting Timers. * [ActivityInboundCallsInterceptor](https://typescript.temporal.io/api/interfaces/worker.ActivityInboundCallsInterceptor) : Intercept inbound calls to an Activity (such as `execute`). * [WorkflowClientInterceptor](https://typescript.temporal.io/api/interfaces/client.WorkflowClientInterceptor/) : Intercept workflow-related methods of [`Client`](https://typescript.temporal.io/api/classes/client.Client/) and [`WorkflowHandle`](https://typescript.temporal.io/api/interfaces/client.WorkflowHandle) like starting or signaling a Workflow. * [NexusInboundCallsInterceptor](https://typescript.temporal.io/api/interfaces/worker.NexusInboundCallsInterceptor) : Intercept inbound Nexus Operation calls like `startOperation` and `cancelOperation`. * [NexusOutboundCallsInterceptor](https://typescript.temporal.io/api/interfaces/worker.NexusOutboundCallsInterceptor) : Intercept outbound calls from Nexus Operations, such as enriching log attributes and metric tags. All interceptor methods are optional—it's up to the implementor to choose which methods to intercept. Interceptor examples[​](https://docs.temporal.io/develop/typescript/workers/interceptors#interceptor-examples "Direct link to Interceptor examples") ----------------------------------------------------------------------------------------------------------------------------------------------------- **Log start and completion of Activities** import { ActivityInput, Next, WorkflowOutboundCallsInterceptor } from '@temporalio/workflow';export class ActivityLogInterceptor implements WorkflowOutboundCallsInterceptor { constructor(public readonly workflowType: string) {} async scheduleActivity( input: ActivityInput, next: Next ): Promise { console.log('Starting activity', { activityType: input.activityType }); try { return await next(input); } finally { console.log('Completed activity', { workflow: this.workflowType, activityType: input.activityType, }); } }} **Log Nexus Operations** import type { NexusInboundCallsInterceptor, NexusStartOperationInput, NexusStartOperationOutput, Next,} from '@temporalio/worker';export class NexusOperationLogInterceptor implements NexusInboundCallsInterceptor { async startOperation( input: NexusStartOperationInput, next: Next ): Promise { console.log('Starting Nexus operation', { service: input.ctx.service, operation: input.ctx.operation, }); const output = await next(input); console.log('Nexus operation started', { service: input.ctx.service, operation: input.ctx.operation, async: output.result.isAsync, }); return output; }} Register an Interceptor[​](https://docs.temporal.io/develop/typescript/workers/interceptors#register-interceptor "Direct link to Register an Interceptor") ----------------------------------------------------------------------------------------------------------------------------------------------------------- Registering an interceptor means providing it to the SDK so Temporal can invoke it when matching Client or Worker calls occur. Once registered, it runs in the call path and can observe or modify request and response data. ### Register via a Plugin[​](https://docs.temporal.io/develop/typescript/workers/interceptors#register-via-a-plugin "Direct link to Register via a Plugin") If you're building a reusable library or want to bundle interceptors with other primitives, you can register them through a [Plugin](https://docs.temporal.io/develop/plugins-guide#interceptors) . ### Activity and client interceptors registration[​](https://docs.temporal.io/develop/typescript/workers/interceptors#activity-and-client-interceptors-registration "Direct link to Activity and client interceptors registration") * Activity interceptors are registered on Worker creation by passing an array of [ActivityInboundCallsInterceptor factory functions](https://typescript.temporal.io/api/interfaces/worker.ActivityInboundCallsInterceptorFactory) through [WorkerOptions](https://typescript.temporal.io/api/interfaces/worker.WorkerOptions#interceptors) . * Client interceptors are registered on `Client` construction by passing an array of [WorkflowClientInterceptor](https://typescript.temporal.io/api/interfaces/client.WorkflowClientInterceptor) via [ClientOptions.interceptors](https://typescript.temporal.io/api/interfaces/client.ClientOptions#interceptors) . ### Workflow interceptors registration[​](https://docs.temporal.io/develop/typescript/workers/interceptors#workflow-interceptors-registration "Direct link to Workflow interceptors registration") Workflow interceptor registration is different from the other interceptors because they run in the Workflow isolate. To register Workflow interceptors, export an `interceptors` function from a file located in the `workflows` directory and provide the name of that file to the Worker on creation via [WorkerOptions](https://typescript.temporal.io/api/interfaces/worker.WorkerOptions#interceptors) . At the time of construction, the Workflow context is already initialized for the current Workflow. You may call the [`workflowInfo()`](https://typescript.temporal.io/api/namespaces/workflow#workflowinfo) function to access Workflow-specific information from an interceptor. `src/workflows/your-interceptors.ts` import { workflowInfo } from '@temporalio/workflow';export const interceptors = () => ({ outbound: [new ActivityLogInterceptor(workflowInfo().workflowType)], inbound: [],}); `src/worker/index.ts` const worker = await Worker.create({ workflowsPath: require.resolve('./workflows'), interceptors: { workflowModules: [require.resolve('./workflows/your-interceptors')], },}); ### Nexus interceptor registration[​](https://docs.temporal.io/develop/typescript/workers/interceptors#nexus-interceptor-registration "Direct link to Nexus interceptor registration") Nexus interceptors are registered on Worker creation via [WorkerOptions](https://typescript.temporal.io/api/interfaces/worker.WorkerOptions#interceptors) . Pass an array of factory functions to `interceptors.nexus`. Each factory receives an [`OperationContext`](https://typescript.temporal.io/api/classes/nexus.OperationContext) and returns an object with optional `inbound` and `outbound` interceptors. `src/worker/index.ts` import { NexusOperationLogInterceptor } from './nexus-interceptors';const worker = await Worker.create({ // ... nexusServices: [/* your Nexus services */], interceptors: { nexus: [ (_ctx) => ({ inbound: new NexusOperationLogInterceptor(), }), ], },}); * [How to implement interceptors in TypeScript](https://docs.temporal.io/develop/typescript/workers/interceptors#interceptors) * [Interceptor examples](https://docs.temporal.io/develop/typescript/workers/interceptors#interceptor-examples) * [Register an Interceptor](https://docs.temporal.io/develop/typescript/workers/interceptors#register-interceptor) --- # Set up your local with the Typescript SDK | Temporal Platform Documentation [Skip to main content](https://docs.temporal.io/develop/typescript/set-up-your-local-typescript#__docusaurus_skipToContent_fallback) Configure your local development environment to get started developing with Temporal. Install Node.js[​](https://docs.temporal.io/develop/typescript/set-up-your-local-typescript#install-nodejs "Direct link to Install Node.js") --------------------------------------------------------------------------------------------------------------------------------------------- The TypeScript SDK requires Node.js 20 or later. Install Node.js via your package manager by following the official Node.js instructions. The TypeScript SDK requires Node.js 20 or later. Install Node.js via your package manager by following the [official Node.js instructions](https://nodejs.org/en/download/) . Install the Temporal TypeScript SDK[​](https://docs.temporal.io/develop/typescript/set-up-your-local-typescript#install-the-temporal-typescript-sdk "Direct link to Install the Temporal TypeScript SDK") ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- You can create a new project with the Temporal SDK: If you're creating a new project using `npx @temporalio/create`, the required SDK packages will be installed automatically. To add Temporal to an existing project, install the required packages manually with `npm install @temporalio/client @temporalio/worker @temporalio/workflow`. Next, you'll configure a local Temporal Service for development. npx @temporalio/create@latest ./my-app When prompted to select a sample, choose the **hello-world** sample. Install Temporal CLI[​](https://docs.temporal.io/develop/typescript/set-up-your-local-typescript#install-temporal-cli "Direct link to Install Temporal CLI") ------------------------------------------------------------------------------------------------------------------------------------------------------------- The fastest way to get a development version of the Temporal Service running on your local machine is to use [Temporal CLI](https://docs.temporal.io/cli) . Choose your operating system to install Temporal CLI. * macOS * Windows * Linux Install the Temporal CLI using Homebrew: brew install temporal Download the Temporal CLI archive for your architecture: * [Windows amd64](https://temporal.download/cli/archive/latest?platform=windows&arch=amd64) * [Windows arm64](https://temporal.download/cli/archive/latest?platform=windows&arch=arm64) Extract it and add `temporal.exe` to your PATH. Download the Temporal CLI for your architecture: * [Linux amd64](https://temporal.download/cli/archive/latest?platform=linux&arch=amd64) * [Linux arm64](https://temporal.download/cli/archive/latest?platform=linux&arch=arm64) Extract the archive and move the `temporal` binary into your PATH, for example: sudo mv temporal /usr/local/bin Start the development server[​](https://docs.temporal.io/develop/typescript/set-up-your-local-typescript#start-the-development-server "Direct link to Start the development server") ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Once you've installed Temporal CLI and added it to your PATH, open a new Terminal window and run the following command. This command starts a local Temporal Service. It starts the Web UI, creates the default Namespace, and uses an in-memory database. The Temporal Service will be available on localhost:7233. The Temporal Web UI will be available at [http://localhost:8233](http://localhost:8233/) . Leave the local Temporal Service running as you work through tutorials and other projects. You can stop the Temporal Service at any time by pressing CTRL+C. Once you have everything installed, you're ready to build apps with Temporal on your local machine. After installing, open a new Terminal window and start the development server: temporal server start-dev #### Change the Web UI port The Temporal Web UI may be on a different port in some examples or tutorials. To change the port for the Web UI, use the `--ui-port` option when starting the server: temporal server start-dev --ui-port 8080 The Temporal Web UI will now be available at http://localhost:8080. Run Hello World: Test Your Installation[​](https://docs.temporal.io/develop/typescript/set-up-your-local-typescript#run-hello-world-test-your-installation "Direct link to Run Hello World: Test Your Installation") --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Now let's verify your setup is working by creating and running a complete Temporal application with both a Workflow and Activity. This test will confirm that: * The Temporal TypeScript SDK is properly installed * Your local Temporal Service is running * You can successfully create and execute Workflows and Activities * The communication between components is functioning correctly ### 1\. Create the Activity[​](https://docs.temporal.io/develop/typescript/set-up-your-local-typescript#1-create-the-activity "Direct link to 1. Create the Activity") Create an Activity file (activities.ts): export async function greet(name: string): Promise { return `Hello, ${name}!`;} An Activity is a normal function or method that executes a single, well-defined action (either short or long running), which often involve interacting with the outside world, such as sending emails, making network requests, writing to a database, or calling an API, which are prone to failure. If an Activity fails, Temporal automatically retries it based on your configuration. ### 2\. Create the Workflow[​](https://docs.temporal.io/develop/typescript/set-up-your-local-typescript#2-create-the-workflow "Direct link to 2. Create the Workflow") Create a Workflow file (workflows.ts): import { proxyActivities } from '@temporalio/workflow';// Only import the activity typesimport type * as activities from './activities';const { greet } = proxyActivities({ startToCloseTimeout: '1 minute',});/** A workflow that simply calls an activity */export async function example(name: string): Promise { return await greet(name);} Workflows orchestrate Activities and contain the application logic. Temporal Workflows are resilient. They can run and keep running for years, even if the underlying infrastructure fails. If the application itself crashes, Temporal will automatically recreate its pre-failure state so it can continue right where it left off. ### 3\. Create and Run the Worker[​](https://docs.temporal.io/develop/typescript/set-up-your-local-typescript#3-create-and-run-the-worker "Direct link to 3. Create and Run the Worker") Create a Worker file (worker.ts): import { NativeConnection, Worker } from '@temporalio/worker';import * as activities from './activities';async function run() { // Step 1: Establish a connection with Temporal server. // // Worker code uses `@temporalio/worker.NativeConnection`. // (But in your application code it's `@temporalio/client.Connection`.) const connection = await NativeConnection.connect({ address: 'localhost:7233', // TLS and gRPC metadata configuration goes here. }); try { // Step 2: Register Workflows and Activities with the Worker. const worker = await Worker.create({ connection, namespace: 'default', taskQueue: 'hello-world', // Workflows are registered using a path as they run in a separate JS context. workflowsPath: require.resolve('./workflows'), activities, }); // Step 3: Start accepting tasks on the `hello-world` queue // // The worker runs until it encounters an unexpected error or the process receives a shutdown signal registered on // the SDK Runtime object. // // By default, worker logs are written via the Runtime logger to STDERR at INFO level. // // See https://typescript.temporal.io/api/classes/worker.Runtime#install to customize these defaults. await worker.run(); } finally { // Close the connection once the worker has stopped await connection.close(); }}run().catch((err) => { console.error(err); process.exit(1);}); Run the Worker and keep this terminal running: npm run start With your Activity and Workflow defined, you need a Worker to execute them. A Worker polls a Task Queue, that you configure it to poll, looking for work to do. Once the Worker dequeues a Workflow or Activity task from the Task Queue, it then executes that task. Workers are a crucial part of your Temporal application as they're what actually execute the tasks defined in your Workflows and Activities. For more information on Workers, see [Understanding Temporal](https://docs.temporal.io/evaluate/understanding-temporal#workers) and a [deep dive into Workers](https://docs.temporal.io/workers) . ### 4\. Execute the Workflow[​](https://docs.temporal.io/develop/typescript/set-up-your-local-typescript#4-execute-the-workflow "Direct link to 4. Execute the Workflow") Now that your Worker is running, it's time to start a Workflow Execution. This final step will validate that everything is working correctly with your file labeled `client.ts`. Create a separate file called `client.ts`. import { Client, Connection } from '@temporalio/client';import { nanoid } from 'nanoid';import { example } from './workflows';async function run() { // Connect to the default Server location const connection = await Connection.connect({ address: 'localhost:7233' }); // In production, pass options to configure TLS and other settings: // { // address: 'foo.bar.tmprl.cloud', // tls: {} // } const client = new Client({ connection, // namespace: 'foo.bar', // connects to 'default' namespace if not specified }); const handle = await client.workflow.start(example, { taskQueue: 'hello-world', // type inference works! args: [name: string] args: ['Temporal'], // in practice, use a meaningful business ID, like customerId or transactionId workflowId: 'workflow-' + nanoid(), }); console.log(`Started workflow ${handle.workflowId}`); // optional: wait for client result console.log(await handle.result()); // Hello, Temporal!}run().catch((err) => { console.error(err); process.exit(1);}); Then run: npm run workflow ### Verify Success[​](https://docs.temporal.io/develop/typescript/set-up-your-local-typescript#verify-success "Direct link to Verify Success") If everything is working correctly, you should see: * Worker processing the workflow and activity * Output: `Workflow result: Hello, Temporal!` * Workflow Execution details in the [Temporal Web UI](http://localhost:8233/) **Additional details about Workflow Execution** * Temporal clients are not explicitly closed. * To enable TLS, set the `tls` option to `true` for default settings or pass a [`TLSConfig`](https://typescript.temporal.io/api/interfaces/worker.TLSConfig) for custom configuration. * Calling `client.workflow.start()` and `client.workflow.execute()` send a command to Temporal Server to schedule a new Workflow Execution on the specified Task Queue. * If you started a Workflow with `client.workflow.start()`, you can choose to wait for the result anytime with handle.result(). * Using a Workflow Handle isn't necessary with `client.workflow.execute()`. [### Next: Run your first Temporal Application\ \ Create a basic Workflow and run it with the Temporal TypeScript SDK\ \ →](https://learn.temporal.io/getting_started/typescript/first_program_in_typescript/) --- # Tasks | Temporal Platform Documentation [Skip to main content](https://docs.temporal.io/tasks#__docusaurus_skipToContent_fallback) On this page This page discusses the following: * [Task](https://docs.temporal.io/tasks#task) * [Workflow Task](https://docs.temporal.io/tasks#workflow-task) * [When are Workflow Tasks scheduled?](https://docs.temporal.io/tasks#when-workflow-tasks-scheduled) * [How does a Worker process a Workflow Task?](https://docs.temporal.io/tasks#how-worker-processes-workflow-task) * [How does the SDK know which code to run?](https://docs.temporal.io/tasks#how-worker-processes-workflow-task) * [Workflow Tasks and Determinism](https://docs.temporal.io/tasks#workflow-task-failures-vs-execution-failures) * [Performance characteristics](https://docs.temporal.io/tasks#workflow-task-execution) * [Workflow Task Execution](https://docs.temporal.io/tasks#workflow-task-execution) * [Workflow Task Failures vs Workflow Execution Failures](https://docs.temporal.io/tasks#workflow-task-failures-vs-execution-failures) * [Activity Task](https://docs.temporal.io/tasks#activity-task) * [Activity Task Execution](https://docs.temporal.io/tasks#activity-task-execution) * [Nexus Task](https://docs.temporal.io/tasks#nexus-task) * [Nexus Task Execution](https://docs.temporal.io/tasks#nexus-task-execution) What is a Task?[​](https://docs.temporal.io/tasks#task "Direct link to What is a Task?") ----------------------------------------------------------------------------------------- A Task is the context that a Worker needs to progress with a specific [Workflow Execution](https://docs.temporal.io/workflow-execution) , [Activity Execution](https://docs.temporal.io/activity-execution) , or a [Nexus Task Execution](https://docs.temporal.io/tasks#nexus-task-execution) . There are three types of Tasks: * [Workflow Task](https://docs.temporal.io/tasks#workflow-task) * [Activity Task](https://docs.temporal.io/tasks#activity-task) * [Nexus Task](https://docs.temporal.io/tasks#nexus-task) What is a Workflow Task?[​](https://docs.temporal.io/tasks#workflow-task "Direct link to What is a Workflow Task?") -------------------------------------------------------------------------------------------------------------------- A Workflow Task is a Task that contains the context needed to make progress with a Workflow Execution. ### When are Workflow Tasks scheduled?[​](https://docs.temporal.io/tasks#when-workflow-tasks-scheduled "Direct link to When are Workflow Tasks scheduled?") The Temporal Service creates and schedules a new Workflow Task whenever one of the following occurs: * The Workflow Execution is started * A Signal is sent to the Workflow * An Update is sent to the Workflow * An Activity completes (successfully or with a failure) * A Timer fires * A Child Workflow completes * A Workflow Task fails and needs to be retried Any event that might affect the Workflow's state triggers a new Workflow Task. The Workflow Task bundles together all new events that have occurred since the last Workflow Task completed. ### How does a Worker process a Workflow Task?[​](https://docs.temporal.io/tasks#how-worker-processes-workflow-task "Direct link to How does a Worker process a Workflow Task?") When a Worker picks up a Workflow Task, it replays the entire Workflow Execution from the beginning using the Event History. * The Worker receives the Workflow Task, which contains the complete Event History for the Workflow Execution * The Workflow Worker replays the Workflow code from the start, using the Event History to recreate the Workflow's state * During replay, previously executed operations (like Activity calls or Timers) return their results immediately from the Event History instead of executing again * The replay continues until the Worker reaches a point where it needs to make new progress (a new Activity to schedule, a new Timer to set, etc.) * The Workflow code executes any new decisions and generates Commands * The Worker sends these Commands back to the Temporal Service, completing the Workflow Task * The Temporal Service persists the Commands as new Events in the Event History This replay mechanism makes Temporal Workflows durable and fault-tolerant. If a Worker crashes mid-execution, another Worker can pick up the Workflow Task and replay the entire history to reconstruct the exact state before continuing ### What is a Workflow Task Execution?[​](https://docs.temporal.io/tasks#workflow-task-execution "Direct link to What is a Workflow Task Execution?") A Workflow Task Execution occurs when a [Worker](https://docs.temporal.io/workers#worker-entity) picks up a [Workflow Task](https://docs.temporal.io/tasks#workflow-task) and uses it to make progress on the execution of a [Workflow Definition](https://docs.temporal.io/workflow-definition) (also known as a Workflow function). Workflow Task Execution is typically very fast (milliseconds). The Worker replays code and makes decisions based on the Event History. No actual I/O operations occur during replay (Activity results come from history). The time spent in a Workflow Task is unrelated to how long Activities or Timers take. Workflow Task Failures vs Workflow Execution Failures[​](https://docs.temporal.io/tasks#workflow-task-failures-vs-execution-failures "Direct link to Workflow Task Failures vs Workflow Execution Failures") ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Understanding the difference between Workflow Task failures and Workflow Execution failures is essential to working with Temporal at a deeper level. **Workflow Task failure** means a Worker can't successfully process a Workflow Task due to infrastructure, Workflow code, or execution environment issues (not business logic). Common causes include non-determinism, unhandled exceptions, task timeouts, invalid Commands, or bad binary checksums. The Service automatically retries the task with exponential backoff, and the Workflow Execution stays Open until a task completes, an operator terminates it, or the Workflow Execution Timeout is reached. Fixes typically involve correcting code, scaling Workers, or resolving infrastructure problems. **Workflow Execution failure** means the Workflow's business logic determines it can't complete. It occurs when Workflow code throws or returns an error, an Activity failure propagates uncaught, or an external system terminates/cancels the Workflow. The Workflow closes with a Failed status and does not automatically retry; if a Retry Policy is configured, the Service starts a new Run with the same Workflow ID and continues retrying until success or exhaustion. Each retry is a separate Run with its own Event History. The table summarizes the differences: | Aspect | Workflow Task Failure | Workflow Execution Failure | | --- | --- | --- | | **What failed** | Infrastructure or Workflow code has a bug | Business logic determined the Workflow cannot succeed | | **Workflow state** | Workflow Execution remains Open | Workflow Execution closes (Failed, Terminated, etc.) | | **Automatic retry** | Always retried automatically by the Service | Only retried if a Workflow Retry Policy is configured | | **Event History** | Same Event History continues to grow | Each retry run has a separate Event History | | **How to resolve** | Fix code/infrastructure and redeploy | May require business logic changes or external intervention | | **Visibility** | Shows as Workflow Task failures in history and metrics | Shows as a Failed Workflow Execution in the UI | **Workflow Task failure example:** A new deployment introduces non-determinism, existing Workflows fail Workflow Tasks, and the executions stay Open and retry. After deploying a fix, the Workflows automatically continue. **Workflow Execution failure example:** A payment Activity fails due to a declined card, the failure propagates uncaught, and the Workflow closes as Failed. The customer updates payment details and restarts the order. What is an Activity Task?[​](https://docs.temporal.io/tasks#activity-task "Direct link to What is an Activity Task?") ---------------------------------------------------------------------------------------------------------------------- An Activity Task contains the context needed to proceed with an [Activity Task Execution](https://docs.temporal.io/tasks#activity-task-execution) . Activity Tasks largely represent the Activity Task Scheduled Event, which contains the data needed to execute an Activity Function. If Heartbeat data is being passed, an Activity Task will also contain the latest Heartbeat details. ### What is an Activity Task Execution?[​](https://docs.temporal.io/tasks#activity-task-execution "Direct link to What is an Activity Task Execution?") An Activity Task Execution occurs when a [Worker](https://docs.temporal.io/workers#worker-entity) uses the context provided from the [Activity Task](https://docs.temporal.io/tasks#activity-task) and executes the [Activity Definition](https://docs.temporal.io/activity-definition) (also known as the Activity Function). The [ActivityTaskScheduled Event](https://docs.temporal.io/references/events#activitytaskscheduled) corresponds to when the Temporal Service puts the Activity Task into the Task Queue. The [ActivityTaskStarted Event](https://docs.temporal.io/references/events#activitytaskstarted) corresponds to when the Worker picks up the Activity Task from the Task Queue. Either [ActivityTaskCompleted](https://docs.temporal.io/references/events#activitytaskcompleted) or one of the other Closed Activity Task Events corresponds to when the Worker has yielded back to the Temporal Service. The API to schedule an Activity Execution provides an "effectively once" experience, even though there may be several Activity Task Executions that take place to successfully complete an Activity. Once an Activity Task finishes execution, the Worker responds to the Temporal Service with a specific Event: * ActivityTaskCanceled * ActivityTaskCompleted * ActivityTaskFailed * ActivityTaskTerminated * ActivityTaskTimedOut What is a Nexus Task?[​](https://docs.temporal.io/tasks#nexus-task "Direct link to What is a Nexus Task?") ----------------------------------------------------------------------------------------------------------- A Nexus Task represents a single Nexus request to start or cancel a Nexus Operation. The Nexus Task includes details such as the Nexus Service and Nexus Operation names, and other information required to process the Nexus request. The Temporal Worker triggers the registered Operation handler based on the Nexus task information. ### What is a Nexus Task Execution?[​](https://docs.temporal.io/tasks#nexus-task-execution "Direct link to What is a Nexus Task Execution?") A Nexus Task Execution occurs when a Worker uses the context provided from the Nexus Task and executes an action associated with a Nexus Operation which commonly includes starting a Nexus Operation using its Nexus Operation handler plus many additional actions that may be performed on a Nexus Operation. The NexusOperationScheduled Event corresponds to when the Temporal Service records the Workflow's intent to schedule an operation. The NexusOperationStarted Event corresponds to when the Worker picks up the Nexus Task from the Task Queue, starts an asynchronous Nexus Operation, and returns an Operation token to the caller indicating the asynchronous Nexus Operation has started. Either NexusOperationCompleted or one of the other Closed Nexus Operation Events corresponds to when the Nexus Operation has reached a final state due to successfully completing the operation or unsuccessfully completing the operation in the case of a failure, timeout, or cancellation. A Nexus Operation Execution appears to the caller Workflow as a single RPC, while under the hood the Temporal Service may issue several Nexus Tasks to attempt to start the Operation. Hence, a Nexus Operation Handler implementation should be idempotent. The WorkflowRunOperation provided by the SDK leverages Workflow ID based deduplication to ensure idempotency and provide an "effectively once" experience. A Nexus Task Execution completes when a Worker responds to the Temporal Service with either a RespondNexusTaskCompleted or RespondNexusTaskFailed call, or when the Task times out. The Temporal Service interprets the outcome and determines whether to retry the Task or record the progress in a History Event: * NexusTaskCompleted * NexusTaskFailed * [What is a Task?](https://docs.temporal.io/tasks#task) * [What is a Workflow Task?](https://docs.temporal.io/tasks#workflow-task) * [When are Workflow Tasks scheduled?](https://docs.temporal.io/tasks#when-workflow-tasks-scheduled) * [How does a Worker process a Workflow Task?](https://docs.temporal.io/tasks#how-worker-processes-workflow-task) * [What is a Workflow Task Execution?](https://docs.temporal.io/tasks#workflow-task-execution) * [Workflow Task Failures vs Workflow Execution Failures](https://docs.temporal.io/tasks#workflow-task-failures-vs-execution-failures) * [What is an Activity Task?](https://docs.temporal.io/tasks#activity-task) * [What is an Activity Task Execution?](https://docs.temporal.io/tasks#activity-task-execution) * [What is a Nexus Task?](https://docs.temporal.io/tasks#nexus-task) * [What is a Nexus Task Execution?](https://docs.temporal.io/tasks#nexus-task-execution) --- # Workflow Execution limits | Temporal Platform Documentation [Skip to main content](https://docs.temporal.io/workflow-execution/limits#__docusaurus_skipToContent_fallback) On this page This page discusses [Workflow Execution limits](https://docs.temporal.io/workflow-execution/limits#workflow-execution-limits) , [Workflow Execution Callback limits](https://docs.temporal.io/workflow-execution/limits#workflow-execution-callback-limits) , and [Nexus Operation limits](https://docs.temporal.io/workflow-execution/limits#workflow-execution-nexus-operation-limits) . Limits[​](https://docs.temporal.io/workflow-execution/limits#workflow-execution-limits "Direct link to Limits") ---------------------------------------------------------------------------------------------------------------- There is no limit to the number of concurrent Workflow Executions, albeit you must abide by the Workflow Execution's Event History limit. caution As a precautionary measure, the Workflow Execution's Event History is limited to [51,200 Events](https://github.com/temporalio/temporal/blob/e3496b1c51bfaaae8142b78e4032cc791de8a76f/service/history/configs/config.go#L382) or [50 MB](https://github.com/temporalio/temporal/blob/e3496b1c51bfaaae8142b78e4032cc791de8a76f/service/history/configs/config.go#L380) and will warn you after 10,240 Events or 10 MB. There is also a limit to the number of certain types of incomplete operations. Each in-progress Activity generates a metadata entry in the Workflow Execution's mutable state. Too many entries in a single Workflow Execution's mutable state causes unstable persistence. To protect the system, Temporal enforces a maximum number of incomplete Activities, Child Workflows, Signals, or Cancellation requests per Workflow Execution (by default, 2,000 for each type of operation). Once the limit is reached for a type of operation, if the Workflow Execution attempts to start another operation of that type (by producing a `ScheduleActivityTask`, `StartChildWorkflowExecution`, `SignalExternalWorkflowExecution`, or `RequestCancelExternalWorkflowExecution` Command), it will be unable to (the Workflow Task Execution will fail and get retried). These limits are set with the following [dynamic configuration keys](https://github.com/temporalio/temporal/blob/main/service/history/configs/config.go) : * `NumPendingActivitiesLimit` * `NumPendingChildExecutionsLimit` * `NumPendingSignalsLimit` * `NumPendingCancelRequestsLimit` Workflow Execution Callback limits[​](https://docs.temporal.io/workflow-execution/limits#workflow-execution-callback-limits "Direct link to Workflow Execution Callback limits") --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- There is a limit to the total number of Workflow Callbacks that may be attached to a single Workflow Execution. Attaching [multiple Nexus callers to a handler Workflow](https://docs.temporal.io/nexus/operations#attaching-multiple-nexus-callers) may exceed these limits. These limits can be set with the following dynamic configuration keys: * [MaxCallbacksPerWorkflow](https://github.com/temporalio/temporal/blob/3b626075691c483871630d4a4df266e783f86328/common/dynamicconfig/constants.go#L998) * [MaxCHASMCallbacksPerWorkflow](https://github.com/temporalio/temporal/blob/3b626075691c483871630d4a4df266e783f86328/common/dynamicconfig/constants.go#L1005) Workflow Execution Nexus Operation Limits[​](https://docs.temporal.io/workflow-execution/limits#workflow-execution-nexus-operation-limits "Direct link to Workflow Execution Nexus Operation Limits") ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ There is a limit to the maximum number of Nexus Operations in a Workflow before Continue-As-New is required. Each in-progress Nexus Operation generates a metadata entry in the Workflow Execution's mutable state. Too many entries in a single Workflow Execution's mutable state causes unstable persistence. To protect the system, Temporal enforces a maximum number of incomplete Nexus Operation requests per Workflow Execution (by default, 30 Nexus Operations). Once the limit is reached for a type of operation, if the Workflow Execution attempts to start another Nexus operation (by producing a ScheduleNexusOperation), it will be unable to do so (the Workflow Task Execution will fail and get retried). These limits are set with the following [dynamic configuration keys](https://github.com/temporalio/temporal/blob/de7c8879e103be666a7b067cc1b247f0ac63c25c/components/nexusoperations/config.go#L38) : * MaxConcurrentOperations * [Limits](https://docs.temporal.io/workflow-execution/limits#workflow-execution-limits) * [Workflow Execution Callback limits](https://docs.temporal.io/workflow-execution/limits#workflow-execution-callback-limits) * [Workflow Execution Nexus Operation Limits](https://docs.temporal.io/workflow-execution/limits#workflow-execution-nexus-operation-limits) --- # Temporal Platform security features | Temporal Platform Documentation [Skip to main content](https://docs.temporal.io/self-hosted-guide/security#__docusaurus_skipToContent_fallback) On this page General company security For information about the general security habits of Temporal Technologies, see our [trust page](https://trust.temporal.io/) . Cloud security For information about Temporal Cloud security features, see our [Cloud security page](https://docs.temporal.io/cloud/security) The Temporal Platform is designed with security in mind, and there are many features that you can use to keep both the Platform itself and your user's data secure. A secured Temporal Server has its network communication encrypted and has authentication and authorization protocols set up for API calls made to it. Without these, your server could be accessed by unwanted entities. What is documented on this page are the built-in opt-in security measures that come with Temporal. However users may also choose to design their own security architecture with reverse proxies or run unsecured instances inside of a VPC environment. ### Server Samples[​](https://docs.temporal.io/self-hosted-guide/security#server-samples "Direct link to Server Samples") The [https://github.com/temporalio/samples-server](https://github.com/temporalio/samples-server) repo offers two examples, which are further explained below: * **TLS:** how to configure Transport Layer Security (TLS) to secure network communication with and within a Temporal Service. * **Authorizer:** how to inject a low-level authorizer component that can control access to all API calls. ### Encryption in transit with mTLS[​](https://docs.temporal.io/self-hosted-guide/security#encryption-in-transit-with-mtls "Direct link to Encryption in transit with mTLS") Temporal supports Mutual Transport Layer Security (mTLS) as a way of encrypting network traffic between the services of a Temporal Service and also between application processes and a Temporal Service. Self-signed or properly minted certificates can be used for mTLS. mTLS is set in Temporal's [TLS configuration](https://docs.temporal.io/references/configuration#tls) . The configuration includes two sections such that intra-Temporal Service and external traffic can be encrypted with different sets of certificates and settings: * `internode`: Configuration for encrypting communication between nodes in the Temporal Service. * `frontend`: Configuration for encrypting the Frontend's public endpoints. A customized configuration can be passed using either the [WithConfig](https://docs.temporal.io/references/server-options#withconfig) or [WithConfigLoader](https://docs.temporal.io/references/server-options#withconfigloader) Server options. See [TLS configuration reference](https://docs.temporal.io/references/configuration#tls) for more details. ### Authentication[​](https://docs.temporal.io/self-hosted-guide/security#authentication "Direct link to Authentication") There are a few authentication protocols available to prevent unwanted access such as authentication of servers, clients, and users. ### Servers[​](https://docs.temporal.io/self-hosted-guide/security#servers "Direct link to Servers") To prevent spoofing and [MITM attacks](https://en.wikipedia.org/wiki/Man-in-the-middle_attack) you can specify the `serverName` in the `client` section of your respective mTLS configuration. This enables established connections to authenticate the endpoint, ensuring that the server certificate presented to any connecting Client has the appropriate server name in its CN property. It can be used for both `internode` and `frontend` endpoints. More guidance on mTLS setup can be found in [the `samples-server` repo](https://github.com/temporalio/samples-server/tree/main/tls) and you can reach out to us for further guidance. ### Client connections[​](https://docs.temporal.io/self-hosted-guide/security#client-connections "Direct link to Client connections") To restrict a client's network access to Temporal Service endpoints you can limit it to clients with certificates issued by a specific Certificate Authority (CA). Use the `clientCAFiles`/ `clientCAData` and `requireClientAuth` properties in both the `internode` and `frontend` sections of the [mTLS configuration](https://docs.temporal.io/references/configuration#tls) . ### Users[​](https://docs.temporal.io/self-hosted-guide/security#users "Direct link to Users") To restrict access to specific users, authentication and authorization is performed through extensibility points and plugins as described in the [Authorization](https://docs.temporal.io/self-hosted-guide/security#authorization) section below. #### Authorization[​](https://docs.temporal.io/self-hosted-guide/security#authorization "Direct link to Authorization") note Information regarding [`Authorizer`](https://docs.temporal.io/self-hosted-guide/security#authorizer-plugin) and [`ClaimMapper`](https://docs.temporal.io/self-hosted-guide/security#claim-mapper) has been moved to another location. Temporal offers two plugin interfaces for implementing API call authorization: * [`ClaimMapper`](https://docs.temporal.io/self-hosted-guide/security#claim-mapper) * [`Authorizer`](https://docs.temporal.io/self-hosted-guide/security#authorizer-plugin) The authorization and claim mapping logic is customizable, making it available to a variety of use cases and identity schemes. When these are provided the frontend invokes the implementation of these interfaces before executing the requested operation. See [https://github.com/temporalio/samples-server/blob/main/extensibility/authorizer](https://github.com/temporalio/samples-server/blob/main/extensibility/authorizer) for a sample implementation. ![Front-end authorization order of operations](https://docs.temporal.io/diagrams/frontend-authorization-order-of-operations.png) Front-end authorization order of operations ### Single sign-on integration[​](https://docs.temporal.io/self-hosted-guide/security#single-sign-on-integration "Direct link to Single sign-on integration") Temporal can be integrated with a single sign-on (SSO) experience by using the `ClaimMapper` and `Authorizer` plugins. The default JWT `ClaimMapper` implementation can be used as is or as a base for a custom implementation of a similar plugin. ### Temporal UI[​](https://docs.temporal.io/self-hosted-guide/security#temporal-ui "Direct link to Temporal UI") To enable SSO authentication in the Temporal UI using environment credentials, you need to configure the UI container with specific environment variables that define your identity provider and OAuth settings. In your docker-compose.yaml, set `TEMPORAL_AUTH_ENABLED=true` to activate authentication. Next, specify the required OAuth credentials and endpoints using environment variables such as: * `TEMPORAL_AUTH_CLIENT_ID` * `TEMPORAL_AUTH_CLIENT_SECRET` * `TEMPORAL_AUTH_PROVIDER_URL` * `TEMPORAL_AUTH_CALLBACK_URL` These values correspond to the client credentials and endpoints provided by your OAuth identity provider (such as Google, Auth0, Okta). When properly configured, Temporal UI will redirect users to your SSO login page and enforce authentication on access. This approach does not require any additional configuration files, making it ideal for containerized environments using secure environment variable injection. temporal-ui: container_name: temporal-ui depends_on: - temporal environment: - TEMPORAL_GRPC_ENDPOINT=temporal:7233 - TEMPORAL_ADDRESS=temporal:7233 - TEMPORAL_AUTH_ENABLED=true - TEMPORAL_AUTH_PROVIDER_URL=https://example.com - TEMPORAL_AUTH_CLIENT_ID=xxxxxxxxxxxxxx - TEMPORAL_AUTH_CLIENT_SECRET=xxxxxxxxxxxxxx - TEMPORAL_AUTH_CALLBACK_URL=https://your-domain/auth/sso/callback - TEMPORAL_AUTH_SCOPES=openid profile email image: temporalio/ui:latest networks: - temporal-network ports: - 8080:8080 For more general guidance for configuration, refer to the [Temporal UI README](https://github.com/temporalio/ui?tab=readme-ov-file#configuration) . For more details on configuration with Docker, refer to [Temporal UI Config](https://github.com/temporalio/ui/blob/c95265ee6431fd0f6cf78ae06373885d66a8ee0c/server/docker/config-template.yaml) . Temporal Service plugins[​](https://docs.temporal.io/self-hosted-guide/security#plugins "Direct link to Temporal Service plugins") ----------------------------------------------------------------------------------------------------------------------------------- The Temporal Service supports some pluggable components. ### What is a ClaimMapper Plugin?[​](https://docs.temporal.io/self-hosted-guide/security#claim-mapper "Direct link to What is a ClaimMapper Plugin?") The Claim Mapper component is a pluggable component that extracts Claims from JSON Web Tokens (JWTs). This process is achieved with the method `GetClaims`, which translates `AuthInfo` structs from the caller into `Claims` about the caller's roles within Temporal. A `Role` (within Temporal) is a bit mask that combines one or more of the role constants. In the following example, the role is assigned constants that allow the caller to read and write information. role := authorization.RoleReader | authorization.RoleWriter `GetClaims` is customizable and can be modified with the `temporal.WithClaimMapper` server option. Temporal also offers a default JWT `ClaimMapper` for your use. A typical approach is for `ClaimMapper` to interpret custom `Claims` from a caller's JWT, such as membership in groups, and map them to Temporal roles for the user. The subject information from the caller's mTLS certificate can also be a parameter in determining roles. #### `AuthInfo`[​](https://docs.temporal.io/self-hosted-guide/security#authinfo "Direct link to authinfo") `AuthInfo` is a struct that is passed to `GetClaims`. `AuthInfo` contains an authorization token extracted from the `authorization` header of the gRPC request. `AuthInfo` includes a pointer to the `pkix.Name` struct. This struct contains an [x.509](https://www.ibm.com/docs/en/ibm-mq/7.5?topic=certificates-distinguished-names) Distinguished Name from the caller's mTLS certificate. #### `Claims`[​](https://docs.temporal.io/self-hosted-guide/security#claims "Direct link to claims") `Claims` is a struct that contains information about permission claims granted to the caller. `Authorizer` assumes that the caller has been properly authenticated, and trusts the `Claims` when making an authorization decision. #### Default JWT ClaimMapper[​](https://docs.temporal.io/self-hosted-guide/security#default-jwt-claimmapper "Direct link to Default JWT ClaimMapper") Temporal offers a default JWT `ClaimMapper` that extracts the information needed to form Temporal `Claims`. This plugin requires a public key to validate digital signatures. To get an instance of the default JWT `ClaimMapper`, call `NewDefaultJWTClaimMapper` and provide it with the following: * a `TokenKeyProvider` instance * a `config.Authorization` pointer * a logger The code for the default `ClaimMapper` can also be used to build a custom `ClaimMapper`. #### Token key provider[​](https://docs.temporal.io/self-hosted-guide/security#token-key-provider "Direct link to Token key provider") A `TokenKeyProvider` obtains public keys from specified issuers' URIs that adhere to a specific format. The default JWT `ClaimMapper` uses this component to obtain and refresh public keys over time. Temporal provides a `defaultTokenKeyProvider`. This component dynamically obtains public keys that follow the [JWKS format](https://tools.ietf.org/html/rfc7517) . It supports formats such as `RSA` and `ECDSA`. provider := authorization.NewDefaultTokenKeyProvider(cfg, logger) note `KeySourceURIs` are the HTTP endpoints that return public keys of token issuers in the [JWKS format](https://tools.ietf.org/html/rfc7517) . `RefreshInterval` defines how frequently keys should be refreshed. For example, [Auth0](https://auth0.com/) exposes endpoints such as `https://YOUR_DOMAIN/.well-known/jwks.json`. By default, "permissions" is used to name the `permissionsClaimName` value. Configure the plugin with `config.Config.Global.Authorization.JWTKeyProvider`. #### JSON Web Token format[​](https://docs.temporal.io/self-hosted-guide/security#json-web-token-format "Direct link to JSON Web Token format") The default JWT `ClaimMapper` expects authorization tokens to be formatted as follows: Bearer The Permissions Claim in the JWT Token is expected to be a collection of Individual Permission Claims. Each Individual Permission Claim must be formatted as follows: : These permissions are then converted into Temporal roles for the caller. This can be one of Temporal's four values: * read * write * worker * admin Multiple permissions for the same Namespace are overridden by the `ClaimMapper`. ##### Example of a payload for the default JWT ClaimMapper[​](https://docs.temporal.io/self-hosted-guide/security#example-of-a-payload-for-the-default-jwt-claimmapper "Direct link to Example of a payload for the default JWT ClaimMapper") { "permissions":[ "temporal-system:read", "namespace1:write" ], "aud":[ "audience" ], "exp":1630295722, "iss":"Issuer"} ### What is an Authorizer Plugin?[​](https://docs.temporal.io/self-hosted-guide/security#authorizer-plugin "Direct link to What is an Authorizer Plugin?") The `Authorizer` plugin contains a single `Authorize` method, which is invoked for each incoming API call. `Authorize` receives information about the API call, along with the role and permission claims of the caller. `Authorizer` allows for a wide range of authorization logic, including call target, role/permissions claims, and other data available to the system. #### Configuration[​](https://docs.temporal.io/self-hosted-guide/security#configuration "Direct link to Configuration") The following arguments must be passed to `Authorizer`: * `context.Context`: General context of the call. * `authorization.Claims`: Claims about the roles assigned to the caller. Its intended use is described in the [`Claims`](https://docs.temporal.io/self-hosted-guide/security#claims) section earlier on this page. * `authorization.CallTarget`: Target of the API call. `Authorizer` then returns one of two decisions: * `DecisionDeny`: the requested API call is not invoked and an error is returned to the caller. * `DecisionAllow`: the requested API call is invoked. Security Warning If you do **not** explicitly configure an `Authorizer`, Temporal uses the default `noopAuthorizer`. This default allows **every** API request, with no authentication or access control. Anyone who can reach your Temporal Server can invoke any API, including sensitive administrative operations. This is **not secure** for production or for any environment that is accessible to untrusted clients (such as over the internet). **To protect your Temporal Server, you must configure an `Authorizer` plugin with a corresponding `ClaimMapper`.** Without this, your deployment is effectively open to anyone with network access. Configure your `Authorizer` with the [`temporal.WithAuthorizer`](https://docs.temporal.io/references/server-options#withauthorizer) server option, and your `ClaimMapper` with the [`temporal.WithClaimMapper`](https://docs.temporal.io/references/server-options#withclaimmapper) server option. temporalServer, err := temporal.NewServer( temporal.WithAuthorizer(newCustomAuthorizer()), temporal.WithClaimMapper(func(cfg *config.Config) authorization.ClaimMapper { return newCustomClaimMapper(cfg) }),) #### How to authorize SDK API calls[​](https://docs.temporal.io/self-hosted-guide/security#authorize-api-calls "Direct link to How to authorize SDK API calls") When authentication is enabled, you can authorize API calls made to the Frontend Service. The Temporal Server [expects](https://docs.temporal.io/self-hosted-guide/security#authentication) an `authorization` gRPC header with an authorization token to be passed with API calls if [requests authorization](https://docs.temporal.io/self-hosted-guide/security#authorization) is configured. Authorization Tokens may be provided to the Temporal Java SDK by implementing a `io.temporal.authorization.AuthorizationTokenSupplier` interface. The implementation should be used to create `io.temporal.authorization.AuthorizationGrpcMetadataProvider` that may be configured on ServiceStub gRPC interceptors list. The implementation is called for each SDK gRPC request and may supply dynamic tokens. **JWT** One of the token types that may be passed this way are JWT tokens. Temporal Server provides a [default implementation of JWT authentication](https://docs.temporal.io/self-hosted-guide/security#default-jwt-claimmapper) . **Example** AuthorizationTokenSupplier tokenSupplier = //your implementation of token supplier () -> "Bearer ";WorkflowServiceStubsOptions serviceStubOptions = WorkflowServiceStubsOptions.newBuilder() //other service stub options .addGrpcMetadataProvider(new AuthorizationGrpcMetadataProvider(tokenSupplier)) .build();WorkflowServiceStubs service = WorkflowServiceStubs.newServiceStubs(serviceStubOptions);WorkflowClient client = WorkflowClient.newInstance(service); Related read: * [How to secure a Temporal Service](https://docs.temporal.io/security) Data Converter[​](https://docs.temporal.io/self-hosted-guide/security#data-converter "Direct link to Data Converter") ---------------------------------------------------------------------------------------------------------------------- Each Temporal SDK provides a [Data Converter](https://docs.temporal.io/dataconversion) that can be customized with a custom [Payload Codec](https://docs.temporal.io/payload-codec) to encode and secure your data. For details on what data can be encoded, how to secure it, and what to consider when using encryption, see [Data encryption](https://docs.temporal.io/production-deployment/data-encryption) . #### Codec Server[​](https://docs.temporal.io/self-hosted-guide/security#codec-server "Direct link to Codec Server") You can use a [Codec Server](https://docs.temporal.io/codec-server) with your custom Payload Codec to decode the data you see on your Web UI and CLI locally through remote endpoints. However, ensure that you consider all security implications of [remote data encoding](https://docs.temporal.io/remote-data-encoding) before using a Codec Server. For details on how to set up a Codec Server, see [Codec Server setup](https://docs.temporal.io/production-deployment/data-encryption#codec-server-setup) . Configuration Options for Restricting and Securing Access[​](https://docs.temporal.io/self-hosted-guide/security#configuration-options-for-restricting-and-securing-access "Direct link to Configuration Options for Restricting and Securing Access") ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Temporal provides extensive configuration options to limit and control which callers can access your services and how those services behave in different environments. These options help you enforce stricter security boundaries in self-hosted deployments and tune behavior for production use cases. For more details on available settings and how to configure them, see the following references: * [Cluster & Server configuration](https://docs.temporal.io/references/configuration) — Static configuration settings for servers and services. * [Dynamic configuration](https://docs.temporal.io/references/dynamic-configuration) — Runtime-updatable settings for tuning and behavior control. * [Client environment configuration](https://docs.temporal.io/references/client-environment-configuration) — Environment variable settings for configuring Temporal clients. * [Server Samples](https://docs.temporal.io/self-hosted-guide/security#server-samples) * [Encryption in transit with mTLS](https://docs.temporal.io/self-hosted-guide/security#encryption-in-transit-with-mtls) * [Authentication](https://docs.temporal.io/self-hosted-guide/security#authentication) * [Servers](https://docs.temporal.io/self-hosted-guide/security#servers) * [Client connections](https://docs.temporal.io/self-hosted-guide/security#client-connections) * [Users](https://docs.temporal.io/self-hosted-guide/security#users) * [Authorization](https://docs.temporal.io/self-hosted-guide/security#authorization) * [Single sign-on integration](https://docs.temporal.io/self-hosted-guide/security#single-sign-on-integration) * [Temporal UI](https://docs.temporal.io/self-hosted-guide/security#temporal-ui) * [Temporal Service plugins](https://docs.temporal.io/self-hosted-guide/security#plugins) * [What is a ClaimMapper Plugin?](https://docs.temporal.io/self-hosted-guide/security#claim-mapper) * [`AuthInfo`](https://docs.temporal.io/self-hosted-guide/security#authinfo) * [`Claims`](https://docs.temporal.io/self-hosted-guide/security#claims) * [Default JWT ClaimMapper](https://docs.temporal.io/self-hosted-guide/security#default-jwt-claimmapper) * [Token key provider](https://docs.temporal.io/self-hosted-guide/security#token-key-provider) * [JSON Web Token format](https://docs.temporal.io/self-hosted-guide/security#json-web-token-format) * [What is an Authorizer Plugin?](https://docs.temporal.io/self-hosted-guide/security#authorizer-plugin) * [Configuration](https://docs.temporal.io/self-hosted-guide/security#configuration) * [How to authorize SDK API calls](https://docs.temporal.io/self-hosted-guide/security#authorize-api-calls) * [Data Converter](https://docs.temporal.io/self-hosted-guide/security#data-converter) * [Codec Server](https://docs.temporal.io/self-hosted-guide/security#codec-server) * [Configuration Options for Restricting and Securing Access](https://docs.temporal.io/self-hosted-guide/security#configuration-options-for-restricting-and-securing-access) --- # Testing - Ruby SDK | Temporal Platform Documentation [Skip to main content](https://docs.temporal.io/develop/ruby/best-practices/testing-suite#__docusaurus_skipToContent_fallback) On this page This page shows how to do the following: * [Understand types of tests](https://docs.temporal.io/develop/ruby/best-practices/testing-suite#types-of-tests) * [Use compatible test frameworks](https://docs.temporal.io/develop/ruby/best-practices/testing-suite#test-frameworks) * [Test Workflows](https://docs.temporal.io/develop/ruby/best-practices/testing-suite#testing-workflows) * [Test Activities](https://docs.temporal.io/develop/ruby/best-practices/testing-suite#test-activities) * [Replay tests](https://docs.temporal.io/develop/ruby/best-practices/testing-suite#replay-test) The Ruby test-suite feature guide describes the frameworks that facilitate Workflow and integration testing. Types of Tests[​](https://docs.temporal.io/develop/ruby/best-practices/testing-suite#types-of-tests "Direct link to Types of Tests") ------------------------------------------------------------------------------------------------------------------------------------- In the context of Temporal, you can create these types of automated tests: * **End-to-end:** Running a Temporal Server and Worker with all its Workflows and Activities; starting and interacting with Workflows from a Client. * **Integration:** Anything between end-to-end and unit testing. * Running Activities with mocked Context and other SDK imports (and usually network requests). * Running Workers with mock Activities, and using a Client to start Workflows. * Running Workflows with mocked SDK imports. * **Unit:** Running a piece of Workflow or Activity code and mocking any code it calls. We generally recommend writing the majority of your tests as integration tests. Because the test server supports skipping time, use the test server for both end-to-end and integration tests with Workers. Test frameworks[​](https://docs.temporal.io/develop/ruby/best-practices/testing-suite#test-frameworks "Direct link to Test frameworks") ---------------------------------------------------------------------------------------------------------------------------------------- **Compatible testing frameworks** The Ruby SDK is compatible with any testing framework and does not have a specific recommendation. Most Ruby SDK samples use [minitest](https://github.com/minitest/minitest) . Testing Workflows[​](https://docs.temporal.io/develop/ruby/best-practices/testing-suite#testing-workflows "Direct link to Testing Workflows") ---------------------------------------------------------------------------------------------------------------------------------------------- Workflow testing can be done in an integration-test fashion against a real server, however it is hard to simulate timeouts and other long time-based code. Using the time-skipping Workflow test environment can help there. ### Testing Workflows with standard server[​](https://docs.temporal.io/develop/ruby/best-practices/testing-suite#testing-workflows-with-standard-server "Direct link to Testing Workflows with standard server") A non-time-skipping `Temporalio::Testing::WorkflowEnvironment` can be started via `start_local` which supports all standard Temporal features. It is actually the real Temporal dev server packaged in the Temporal CLI, lazily downloaded on first use, and run as a sub-process in the background. Assuming tests properly use separate Task Queues, the same server can and should be reused across tests. Here's a simple example of a Workflow: class SimpleWorkflow < Temporalio::Workflow::Definition def execute(name) "Hello, #{name}!" endend Here's how a test of that Workflow may appear in minitest: def test_simple_workflow # Start local server that is stopped when block is done Temporalio::Testing::WorkflowEnvironment.start_local do |env| # Start worker that is stopped when block is done worker = Temporalio::Worker.new( env.client, task_queue: "tq-#{SecureRandom.uuid}", workflows: [SimpleWorkflow] ) worker.run do # Execute workflow and check result result = env.client.execute_workflow( SimpleWorkflow, 'some-name', id: "wf-#{SecureRandom.uuid}", task_queue: worker.task_queue ) assert_equal 'Hello, some-name!', result end endend While this is just a demonstration, a local server is often used as a fixture across many tests. In minitest for instance, users often start the environment lazily (with no block), and shut it down inside a block passed to `Minitest.after_run`. ### Testing Workflows with time skipping[​](https://docs.temporal.io/develop/ruby/best-practices/testing-suite#testing-workflows-with-time-skipping "Direct link to Testing Workflows with time skipping") Sometimes there is a need to test Workflows that run a long time or to test that timeouts occur. A time-skipping `Temporalio::Testing::WorkflowEnvironment` can be started via `start_time_skipping` which is a reimplementation of the Temporal server with special time skipping capabilities. Like `start_local`, this also lazily downloads the process to run when first called. Note, unlike `start_local`, this class is not thread safe nor safe for use with independent tests. It can be technically be reused, but only for one test at a time because time skipping is locked/unlocked at the environment level. Developers are encouraged to run it per test needed. #### Automatic time skipping[​](https://docs.temporal.io/develop/ruby/best-practices/testing-suite#automatic-time-skipping "Direct link to Automatic time skipping") Here's a simple example of a Workflow that waits a day: class WaitADayWorkflow < Temporalio::Workflow::Definition def execute Temporalio::Workflow.sleep(1 * 24 * 60 * 60) 'all done' endend A regular integration test of this Workflow on a normal server would be way too slow. However, the time-skipping server automatically skips to the next event when we wait on the result. Here's a test for that Workflow in minitest: def test_wait_a_day_workflow # Start time-skipping test server that is stopped when block is done Temporalio::Testing::WorkflowEnvironment.start_time_skipping do |env| # Start worker that is stopped when block is done worker = Temporalio::Worker.new( env.client, task_queue: "tq-#{SecureRandom.uuid}", workflows: [WaitADayWorkflow] ) worker.run do # Execute workflow and check result result = env.client.execute_workflow( WaitADayWorkflow, id: "wf-#{SecureRandom.uuid}", task_queue: worker.task_queue ) assert_equal 'all done', result end endend This test will run almost instantly. This is because by calling `execute_workflow` on our client, we are actually calling `start_workflow` + `result`, and `result` automatically skips time as much as it can (basically until the end of the workflow or until an activity is run). To disable automatic time-skipping while waiting for a workflow result, run code in a block passed to `env.auto_time_skipping_disabled`. #### Manual time skipping[​](https://docs.temporal.io/develop/ruby/best-practices/testing-suite#manual-time-skipping "Direct link to Manual time skipping") Until a Workflow is waited on, all time skipping in the time-skipping environment is done manually via `WorkflowEnvironment#sleep`. Here's a Workflow that waits for a Signal or times out: class SignalWorkflow < Temporalio::Workflow::Definition def execute # Wait for signal or timeout in 45 seconds Temporalio::Workflow.timeout(45 * 60) do Temporalio::Workflow.wait_condition { @signal_received } end 'got signal' rescue Timeout::Error 'got timeout' end workflow_signal def some_signal @signal_received = true endend To test a normal Signal in minitest, you might: def test_signal_workflow Temporalio::Testing::WorkflowEnvironment.start_time_skipping do |env| worker = Temporalio::Worker.new( env.client, task_queue: "tq-#{SecureRandom.uuid}", workflows: [SignalWorkflow] ) worker.run do handle = env.client.start_workflow( SignalWorkflow, id: "wf-#{SecureRandom.uuid}", task_queue: worker.task_queue ) handle.signal(SignalWorkflow.some_signal) assert_equal 'got signal', handle.result end endend But how would you test the timeout part? Like so: def test_signal_workflow_timeout Temporalio::Testing::WorkflowEnvironment.start_time_skipping do |env| worker = Temporalio::Worker.new( env.client, task_queue: "tq-#{SecureRandom.uuid}", workflows: [SignalWorkflow] ) worker.run do handle = env.client.start_workflow( SignalWorkflow, id: "wf-#{SecureRandom.uuid}", task_queue: worker.task_queue ) # Advance 50 seconds env.sleep(50) assert_equal 'got timeout', handle.result end endend ### Mocking Activities[​](https://docs.temporal.io/develop/ruby/best-practices/testing-suite#mocking-activities "Direct link to Mocking Activities") When testing Workflows, often you don't want to actually run the Activities. Activities are just classes that extend `Temporalio::Activity::Definition`. Simply write different/empty/fake/asserting ones and pass those to the Worker to have different activities called during the test. Testing Activities[​](https://docs.temporal.io/develop/ruby/best-practices/testing-suite#test-activities "Direct link to Testing Activities") ---------------------------------------------------------------------------------------------------------------------------------------------- Unit testing an Activity or any code that could run in an Activity is done via the `Temporalio::Testing::ActivityEnvironment` class. Simply instantiate the class, and any code inside the block to `run` will be invoked inside the activity context. Several things about the activity environment can be customized via parameters when constructing the environment including setting the info, providing a proc to call back on each heartbeat, setting the cancellation to be used, etc. Replay test[​](https://docs.temporal.io/develop/ruby/best-practices/testing-suite#replay-test "Direct link to Replay test") ---------------------------------------------------------------------------------------------------------------------------- Given a Workflow's history, it can be replayed locally to check for things like non-determinism errors. For example, assuming the `history_json` parameter below is given a JSON string of history exported from the CLI or web UI for workflow `MyWorkflow`, the following method will replay it: def replay_from_json(history_json) # Create a replayer replayer = Temporalio::Worker::WorkflowReplayer.new(workflows: [MyWorkflow]) # Replay the history history = Temporalio::WorkflowHistory.from_history_json(history_json) replayer.replay_workflow(history)end If there is a non-determinism, this will raise an exception. Workflow history can be loaded from more than just JSON. It can be fetched individually from a Workflow handle, or even in a list. For example, the following code will check that all Workflow histories for a certain Workflow type (i.e. workflow class) are safe with the current Workflow code. # Create a replayerreplayer = Temporalio::Worker::WorkflowReplayer.new(workflows: [MyWorkflow])# Replay all workflows from a listreplayer.replay_workflows(client.list_workflows("WorkflowType = 'MyWorkflow'")).each do |result| # Raise if any failed (could have just set raise_on_replay_failure: true, but this # demonstrates iterating over the results) raise result.replay_failure if result.replay_failureend * [Types of Tests](https://docs.temporal.io/develop/ruby/best-practices/testing-suite#types-of-tests) * [Test frameworks](https://docs.temporal.io/develop/ruby/best-practices/testing-suite#test-frameworks) * [Testing Workflows](https://docs.temporal.io/develop/ruby/best-practices/testing-suite#testing-workflows) * [Testing Workflows with standard server](https://docs.temporal.io/develop/ruby/best-practices/testing-suite#testing-workflows-with-standard-server) * [Testing Workflows with time skipping](https://docs.temporal.io/develop/ruby/best-practices/testing-suite#testing-workflows-with-time-skipping) * [Automatic time skipping](https://docs.temporal.io/develop/ruby/best-practices/testing-suite#automatic-time-skipping) * [Manual time skipping](https://docs.temporal.io/develop/ruby/best-practices/testing-suite#manual-time-skipping) * [Mocking Activities](https://docs.temporal.io/develop/ruby/best-practices/testing-suite#mocking-activities) * [Testing Activities](https://docs.temporal.io/develop/ruby/best-practices/testing-suite#test-activities) * [Replay test](https://docs.temporal.io/develop/ruby/best-practices/testing-suite#replay-test) --- # Data encryption - Temporal feature | Temporal Platform Documentation [Skip to main content](https://docs.temporal.io/evaluate/development-production-features/data-encryption#__docusaurus_skipToContent_fallback) Data Converters in Temporal are SDK components that handle the serialization and encoding of data transmitted and received by a Temporal Client. Workflow input and output need to be serialized and deserialized so they can be sent as JSON to the Temporal Service. Temporal provides its own default Data Converter logic, which is not apparent to a user if payloads contain plain text or JSON data. For enhanced security, you can implement your own encryption standards using a Codec Server. Temporal's data encryption capabilities ensure the security and confidentiality of your Workflows and provides protection without compromising performance. Jump straight to a Temporal SDK feature guide. Related 📚 * [![](https://docs.temporal.io/img/sdks/svgs/golang.svg)Data Encryption using the Go SDK](https://docs.temporal.io/develop/go/data-handling/data-encryption) feature-guide * [![](https://docs.temporal.io/img/sdks/svgs/java.svg)Data Encryption using the Java SDK](https://docs.temporal.io/develop/java/best-practices/converters-and-encryption) feature-guide * [![](https://docs.temporal.io/img/sdks/svgs/python.svg)Data Encryption using the Python SDK](https://docs.temporal.io/develop/python/data-handling/data-encryption) feature-guide * [![](https://docs.temporal.io/img/sdks/svgs/typescript.svg)Data Encryption using the TypeScript SDK](https://docs.temporal.io/develop/typescript/converters-and-encryption) feature-guide * [![](https://docs.temporal.io/img/sdks/svgs/dotnet.svg)Data Encryption using the .NET SDK](https://docs.temporal.io/develop/dotnet/best-practices/converters-and-encryption) feature-guide * [![](https://docs.temporal.io/img/sdks/svgs/ruby.svg)Data Encryption using the Ruby SDK](https://docs.temporal.io/develop/ruby/best-practices/converters-and-encryption) feature-guide --- # Archival | Temporal Platform Documentation [Skip to main content](https://docs.temporal.io/temporal-service/archival#__docusaurus_skipToContent_fallback) On this page This page discusses [Archival](https://docs.temporal.io/temporal-service/archival#archival) . What is Archival?[​](https://docs.temporal.io/temporal-service/archival#archival "Direct link to What is Archival?") --------------------------------------------------------------------------------------------------------------------- Use Archival to copy closed Workflow Execution [Event Histories](https://docs.temporal.io/workflow-execution/event#event-history) and Visibility records from Temporal Service persistence to blob storage. * [How to create a custom Archiver](https://docs.temporal.io/self-hosted-guide/archival#custom-archiver) * [How to set up Archival](https://docs.temporal.io/self-hosted-guide/archival#set-up-archival) When a Workflow Execution closes, Temporal schedules close-processing tasks for both Visibility records and Event History archival. Archival then runs asynchronously after a randomized delay. By default, that delay is up to 5 minutes (`history.archivalProcessorArchiveDelay`), capped by the Namespace [Retention Period](https://docs.temporal.io/temporal-service/temporal-server#retention-period) . The closed execution still stays in Temporal persistence until retention cleanup runs. For some time, the same closed execution can exist in both persistence and archival storage. Archival enables Workflow Execution data to persist beyond retention without overwhelming the Temporal Service persistence store. This feature is helpful for compliance and debugging. Temporal's Archival feature is considered **experimental** and not subject to normal [versioning and support policy](https://docs.temporal.io/temporal-service/temporal-server#versions-and-support) . Archival is not supported when running Temporal through Docker. It's disabled by default when installing the system manually and when deploying through [helm charts](https://github.com/temporalio/helm-charts/blob/main/charts/temporal/templates/server-configmap.yaml) . It can be enabled in the [config](https://github.com/temporalio/temporal/blob/main/config/development.yaml) . * [What is Archival?](https://docs.temporal.io/temporal-service/archival#archival) --- # Temporal Events reference | Temporal Platform Documentation [Skip to main content](https://docs.temporal.io/references/events#__docusaurus_skipToContent_fallback) On this page [Events](https://docs.temporal.io/workflow-execution/event#event) are created by the [Temporal Service](https://docs.temporal.io/temporal-service) in response to external occurrences and [Commands](https://docs.temporal.io/workflow-execution#command) generated by a [Workflow Execution](https://docs.temporal.io/workflow-execution) . All possible Events that could appear in a Workflow Execution [Event History](https://docs.temporal.io/workflow-execution/event#event-history) are listed below. ### WorkflowExecutionStarted[​](https://docs.temporal.io/references/events#workflowexecutionstarted "Direct link to WorkflowExecutionStarted") This is always the first [Event](https://docs.temporal.io/workflow-execution/event#event) in a Workflow Execution Event History. It indicates that the Temporal Service received a request to spawn the Workflow Execution. | Field | Description | | --- | --- | | workflow\_type | The [Name](https://docs.temporal.io/workflow-definition#workflow-type)
of [Workflow](https://docs.temporal.io/workflows)
that was initiated. | | parent\_workflow\_namespace | The [Namespace](https://docs.temporal.io/namespaces)
of the Parent [Workflow Execution](https://docs.temporal.io/workflow-execution)
, if applicable. | | parent\_workflow\_execution | Identifies the parent Workflow and the execution run. | | parent\_initiated\_event\_id | Id of the [StartWorkflowExecutionInitiated](https://docs.temporal.io/references/events#startchildworkflowexecutioninitiated)
Event this Event corresponds to. | | task\_queue | The [Task Queue](https://docs.temporal.io/task-queue)
that this [Workflow Task](https://docs.temporal.io/tasks#workflow-task)
was enqueued in. | | input | Information that is deserialized by the SDK to provide arguments to the Workflow. | | workflow\_execution\_timeout | The total timeout period for a [Workflow Execution](https://docs.temporal.io/workflow-execution)
, including retries and continue-as-new. | | workflow\_run\_timeout | Timeout of a single Workflow run. | | workflow\_task\_timeout | Timeout of a single Workflow Task. | | continued\_execution\_run\_id | [Run Id](https://docs.temporal.io/workflow-execution/workflowid-runid#run-id)
of the previous Workflow which continued-as-new, retried or was executed by Cron into this Workflow. | | initiator | Allows the Workflow to continue as a new Workflow Execution. | | continued\_failure | Serialized result of a failure. | | last\_completion\_result | Information from the previously completed [Task](https://docs.temporal.io/tasks#task)
, if applicable. | | original\_execution\_run\_id | The [Run Id](https://docs.temporal.io/workflow-execution/workflowid-runid#run-id)
of the original Workflow started. | | identity | The Id of the [Client](https://docs.temporal.io/self-hosted-guide/security#client-connections)
or parent Workflow [Worker](https://docs.temporal.io/workers#worker)
that requested the start of this Workflow. | | first\_execution\_run\_id | The first [Run Id](https://docs.temporal.io/workflow-execution/workflowid-runid#run-id)
, along the chain of [Continue-As-New](https://docs.temporal.io/workflow-execution/continue-as-new)
Runs and Reset. | | retry\_policy | The amount of retries as determined by the service's dynamic configuration. Retries will happen until 'schedule\_to\_close\_timeout' is reached. | | attempt | The number of attempts that have been made to complete this Task. | | workflow\_execution\_expiration\_time | The absolute time at which the Workflow Execution will [time out](https://docs.temporal.io/encyclopedia/detecting-workflow-failures#workflow-execution-timeout)
. | | cron\_schedule | Displays the Workflow's [Cron Schedule](https://docs.temporal.io/cron-job)
, if applicable. | | first\_workflow\_task\_backoff | Contains the amount of time between when this iteration of the Workflow was scheduled, and when it should run next. Applies to Cron Scheduling. | | memo | Non-indexed information to show in the Workflow. | | search\_attributes | Provides data for setting up a Workflow's [Search Attributes](https://docs.temporal.io/search-attribute)
. | | prev\_auto\_reset\_points | | | header | Information passed by the sender of the [Signal](https://docs.temporal.io/sending-messages#sending-signals)
that is copied into the [Workflow Task](https://docs.temporal.io/tasks#workflow-task)
. | | completion\_callbacks | Completion callbacks attached when this workflow was started. | ### WorkflowExecutionCompleted[​](https://docs.temporal.io/references/events#workflowexecutioncompleted "Direct link to WorkflowExecutionCompleted") This indicates that the [Workflow Execution](https://docs.temporal.io/workflow-execution) has successfully completed. The [Event](https://docs.temporal.io/workflow-execution/event#event) contains Workflow Execution results. | Field | Description | | --- | --- | | result | Serialized result of completed [Workflow](https://docs.temporal.io/workflows)
. | | workflow\_task\_completed\_event\_id | The Id of the [WorkflowTaskCompleted](https://docs.temporal.io/references/events#workflowtaskcompleted)
that the Event was reported with. | | new\_execution\_run\_id | The [Run Id](https://docs.temporal.io/workflow-execution/workflowid-runid#run-id)
of the new Workflow Execution started as a result of a [Cron Schedule](https://docs.temporal.io/cron-job)
. | ### WorkflowExecutionFailed[​](https://docs.temporal.io/references/events#workflowexecutionfailed "Direct link to WorkflowExecutionFailed") This [Event](https://docs.temporal.io/workflow-execution/event#event) indicates that the [Workflow Execution](https://docs.temporal.io/workflow-execution) has unsuccessfully completed and contains the Workflow Execution error. | Field | Description | | --- | --- | | failure | Serialized result of a [Workflow](https://docs.temporal.io/workflows)
failure. | | retry\_state | The reason provided for whether the [Task](https://docs.temporal.io/tasks#task)
should or shouldn't be retried. | | workflow\_task\_completed\_event\_id | The [Run Id](https://docs.temporal.io/workflow-execution/workflowid-runid#run-id)
of the [WorkflowTaskCompleted](https://docs.temporal.io/references/events#workflowtaskcompleted)
that the Event was reported with. | | new\_execution\_run\_id | The [Run Id](https://docs.temporal.io/workflow-execution/workflowid-runid#run-id)
of the new Workflow started by Cron or [Retry](https://docs.temporal.io/encyclopedia/retry-policies)
. | ### WorkflowExecutionTimedOut[​](https://docs.temporal.io/references/events#workflowexecutiontimedout "Direct link to WorkflowExecutionTimedOut") This [Event](https://docs.temporal.io/workflow-execution/event#event) type indicates that the [Workflow Execution](https://docs.temporal.io/workflow-execution) has timed out by the [Temporal Server](https://docs.temporal.io/temporal-service/temporal-server) due to the [Workflow](https://docs.temporal.io/workflows) having not been completed within [timeout](https://docs.temporal.io/encyclopedia/detecting-workflow-failures#workflow-execution-timeout) settings. | Field | Description | | --- | --- | | retry\_state | The reason provided for whether the [Task](https://docs.temporal.io/tasks#task)
should or shouldn't be retried. | | new\_execution\_run\_id | The [Run Id](https://docs.temporal.io/workflow-execution/workflowid-runid#run-id)
of the new Workflow started by Cron or [Retry](https://docs.temporal.io/encyclopedia/retry-policies)
. | ### WorkflowExecutionCancelRequested[​](https://docs.temporal.io/references/events#workflowexecutioncancelrequested "Direct link to WorkflowExecutionCancelRequested") This [Event](https://docs.temporal.io/workflow-execution/event#event) type indicates that a request has been made to cancel the [Workflow Execution](https://docs.temporal.io/workflow-execution) . | Field | Description | | --- | --- | | cause | The user-provided reason for the cancelation request. | | external\_initiated\_event\_id | The [Run Id](https://docs.temporal.io/workflow-execution/workflowid-runid#run-id)
of the Event in the [Workflow](https://docs.temporal.io/workflows)
that requested cancelation, if applicable. | | external\_workflow\_execution | Identifies the external Workflow and the run of the its execution. | | identity | Id of the [Worker](https://docs.temporal.io/workers#worker)
that requested cancelation. | ### WorkflowExecutionCanceled[​](https://docs.temporal.io/references/events#workflowexecutioncanceled "Direct link to WorkflowExecutionCanceled") This [Event](https://docs.temporal.io/workflow-execution/event#event) type indicates that the client has confirmed the cancelation request and the [Workflow Execution](https://docs.temporal.io/workflow-execution) has been canceled. | Field | Description | | --- | --- | | workflow\_task\_completed\_event\_id | The Id of the [WorkflowTaskCompleted](https://docs.temporal.io/references/events#workflowtaskcompleted)
that the Event was reported with. | | details | Additional information reported by the [Workflow](https://docs.temporal.io/workflows)
upon cancelation. | ### WorkflowExecutionSignaled[​](https://docs.temporal.io/references/events#workflowexecutionsignaled "Direct link to WorkflowExecutionSignaled") This [Event](https://docs.temporal.io/workflow-execution/event#event) type indicates the [Workflow](https://docs.temporal.io/workflows) has received a [Signal](https://docs.temporal.io/sending-messages#sending-signals) Event. The Event type contains the Signal name and a Signal payload. | Field | Description | | --- | --- | | signal\_name | The name/type of Signal to be fired. | | input | Information that is deserialized by the SDK to provide arguments to the Workflow function. | | identity | Identifies the [Worker](https://docs.temporal.io/workers#worker)
that signaled to the Workflow. | | header | Information passed by the sender of the Signal that is copied into the [Workflow Task](https://docs.temporal.io/tasks#workflow-task)
. | ### WorkflowExecutionTerminated[​](https://docs.temporal.io/references/events#workflowexecutionterminated "Direct link to WorkflowExecutionTerminated") This [Event](https://docs.temporal.io/workflow-execution/event#event) type indicates that the [Workflow Execution](https://docs.temporal.io/workflow-execution) has been forcefully terminated and that likely the terminate Workflow API was called. | Field | Description | | --- | --- | | reason | Information provided by the user or client for Workflow termination. | | details | Additional information reported by the Workflow upon termination. | | identity | Identifies the Worker that requested termination. | ### WorkflowExecutionContinuedAsNew[​](https://docs.temporal.io/references/events#workflowexecutioncontinuedasnew "Direct link to WorkflowExecutionContinuedAsNew") This [Event](https://docs.temporal.io/workflow-execution/event#event) type indicates that the Workflow has successfully completed, and a new Workflow has been started within the same transaction. This Event type contains last [Workflow Execution](https://docs.temporal.io/workflow-execution) results as well as new Workflow Execution inputs. | Field | Description | | --- | --- | | new\_execution\_run\_id | The [Run Id](https://docs.temporal.io/workflow-execution/workflowid-runid#run-id)
of the new Workflow started by this Continue-As-New Event. | | workflow\_type | The name/type of Workflow that was started by this Event. | | task\_queue | The [Task Queue](https://docs.temporal.io/task-queue)
that this [Workflow Task](https://docs.temporal.io/tasks#workflow-task)
was enqueued in. | | input | Information that is deserialized by the SDK to provide arguments to the Workflow. | | workflow\_run\_timeout | Timeout of a single Workflow run. | | workflow\_task\_timeout | Timeout of a single Workflow Task. | | workflow\_task\_completed\_event\_id | The Id of the [WorkflowTaskCompleted](https://docs.temporal.io/references/events#workflowtaskcompleted)
that the Event command was reported with. | | backoff\_start\_interval | The amount of time to delay the beginning of the [ContinuedAsNew](https://docs.temporal.io/references/events#workflowexecutioncontinuedasnew)
Workflow. | | initiator | Allows the Workflow to continue as a new execution. | | last\_completion\_result | Information passed by the previously completed Task to the ongoing execution. | | header | Information passed by the sender of the Signal that is copied into the Workflow Task. | | memo | Non-indexed information to show in the Workflow. | | search\_attributes | Provides data for setting up a Workflow's [Search Attributes](https://docs.temporal.io/search-attribute)
. | ### WorkflowExecutionOptionsUpdated[​](https://docs.temporal.io/references/events#workflowexecutionoptionsupdated "Direct link to WorkflowExecutionOptionsUpdated") This [Event](https://docs.temporal.io/workflow-execution/event#event) type indicates that the Workflow options have been updated. The Event type contains updated options such as a versioning override or attached completion callbacks. | Field | Description | | --- | --- | | versioning\_override | Versioning override upserted in this event. Ignored if nil or if unset\_versioning\_override is true. | | unset\_versioning\_override | Versioning override removed in this event. | | attached\_request\_id | Request ID attached to the running workflow execution so subsequent requests with the same request ID will be deduped. | | attached\_completion\_callbacks | Completion callbacks attached to the running workflow execution. | ### WorkflowTaskScheduled[​](https://docs.temporal.io/references/events#workflowtaskscheduled "Direct link to WorkflowTaskScheduled") This [Event](https://docs.temporal.io/workflow-execution/event#event) type indicates that the [Workflow Task](https://docs.temporal.io/tasks#workflow-task) has been scheduled. The SDK client should now be able to process any new history events. | Field | Description | | --- | --- | | task\_queue | The [Task Queue](https://docs.temporal.io/task-queue)
that this Workflow Task was enqueued in. | | start\_to\_close\_timeout | The time that the [Worker](https://docs.temporal.io/workers#worker)
takes to process this Task once it's received. | | attempt | The number of attempts that have been made to complete this Task. | ### WorkflowTaskStarted[​](https://docs.temporal.io/references/events#workflowtaskstarted "Direct link to WorkflowTaskStarted") This [Event](https://docs.temporal.io/workflow-execution/event#event) type indicates that the [Workflow Task](https://docs.temporal.io/tasks#workflow-task) has started. The SDK client has picked up the Workflow Task and is processing new history events. | Field | Description | | --- | --- | | scheduled\_event\_id | The Id of the [WorkflowTaskScheduled](https://docs.temporal.io/references/events#workflowtaskscheduled)
Event that this Workflow Task corresponds to. | | identity | Identifies the [Worker](https://docs.temporal.io/workers#worker)
that started this Task. | | request\_id | Identifies the Workflow Task request. | ### WorkflowTaskCompleted[​](https://docs.temporal.io/references/events#workflowtaskcompleted "Direct link to WorkflowTaskCompleted") This [Event](https://docs.temporal.io/workflow-execution/event#event) type indicates that the [Workflow Task](https://docs.temporal.io/tasks#workflow-task) completed. | Field | Description | | --- | --- | | scheduled\_event\_id | The Id of the [WorkflowTaskScheduled](https://docs.temporal.io/references/events#workflowtaskscheduled)
Event that this Workflow Task corresponds to. | | started\_event\_id | The Id of the [WorkflowTaskStarted](https://docs.temporal.io/references/events#workflowtaskstarted)
Event that this Task corresponds to. | | identity | Identity of the [Worker](https://docs.temporal.io/workers#worker)
that completed this Task. | | binary\_checksum | Binary Id of the Worker that completed this Task. | The SDK client picked up the Workflow Task, processed new history events, and may or may not ask the [Temporal Server](https://docs.temporal.io/temporal-service/temporal-server) to do additional work. It is possible for the following events to still occur: * [ActivityTaskScheduled](https://docs.temporal.io/references/events#activitytaskscheduled) * [TimerStarted](https://docs.temporal.io/references/events#timerstarted) * [UpsertWorkflowSearchAttributes](https://docs.temporal.io/references/events#upsertworkflowsearchattributes) * [MarkerRecorded](https://docs.temporal.io/references/events#markerrecorded) * [StartChildWorkflowExecutionInitiated](https://docs.temporal.io/references/events#startchildworkflowexecutioninitiated) * [RequestCancelExternalWorkflowExecutionInitiated](https://docs.temporal.io/references/events#requestcancelexternalworkflowexecutioninitiated) * [SignalExternalWorkflowExecutionInitiated](https://docs.temporal.io/references/events#signalexternalworkflowexecutioninitiated) * [WorkflowExecutionCompleted](https://docs.temporal.io/references/events#workflowexecutioncompleted) * [WorkflowExecutionFailed](https://docs.temporal.io/references/events#workflowexecutionfailed) * [WorkflowExecutionCanceled](https://docs.temporal.io/references/events#workflowexecutioncanceled) * [WorkflowExecutionContinuedAsNew](https://docs.temporal.io/references/events#workflowexecutioncontinuedasnew) ### WorkflowTaskTimedOut[​](https://docs.temporal.io/references/events#workflowtasktimedout "Direct link to WorkflowTaskTimedOut") This [Event](https://docs.temporal.io/workflow-execution/event#event) type indicates that the [Workflow Task](https://docs.temporal.io/tasks#workflow-task) encountered a [timeout](https://docs.temporal.io/encyclopedia/detecting-workflow-failures#workflow-task-timeout) . Either an SDK client with a local cache was not available at the time, or it took too long for the SDK client to process the Task. | Field | Description | | --- | --- | | scheduled\_event\_id | The Id of the [WorkflowTaskScheduled](https://docs.temporal.io/references/events#workflowtaskscheduled)
Event that this Workflow Task corresponds to. | | started\_event\_id | The Id of the [WorkflowTaskStarted](https://docs.temporal.io/references/events#workflowtaskstarted)
Event that this Task corresponds to. | | timeout\_type | The type of timeout that has occurred. | ### WorkflowTaskFailed[​](https://docs.temporal.io/references/events#workflowtaskfailed "Direct link to WorkflowTaskFailed") This [Event](https://docs.temporal.io/workflow-execution/event#event) type indicates that the [Workflow Task](https://docs.temporal.io/tasks#workflow-task) encountered a failure. Usually this means that the Workflow was non-deterministic. However, the Workflow reset functionality also uses this Event. | Field | Description | | --- | --- | | scheduled\_event\_id | The Id of the [WorkflowTaskScheduled](https://docs.temporal.io/references/events#workflowtaskscheduled)
Event that this Workflow Task corresponds to. | | started\_event\_id | The Id of the [WorkflowTaskStarted](https://docs.temporal.io/references/events#workflowtaskstarted)
Event that this Workflow Task corresponds to. | | failure | Details for the Workflow Task's failure. | | identity | The identity of the [Worker](https://docs.temporal.io/workers#worker)
that failed this Task. The Worker must be explicitly defined to return a value for this field. | | base\_run\_id | The original [Run Id](https://docs.temporal.io/workflow-execution/workflowid-runid#run-id)
of the Workflow. | | new\_run\_id | The Run Id of the reset Workflow. | | fork\_event\_version | Identifies the Event version that was forked off to the reset Workflow. | | binary\_checksum | The Binary Id of the Worker that failed this Task. The Worker must be explicitly defined to return a value for this field. | ### ActivityTaskScheduled[​](https://docs.temporal.io/references/events#activitytaskscheduled "Direct link to ActivityTaskScheduled") This [Event](https://docs.temporal.io/workflow-execution/event#event) type indicates that an [Activity Task](https://docs.temporal.io/tasks#activity-task) was scheduled. The SDK client should pick up this Activity Task and execute. This Event type contains Activity inputs, as well as Activity Timeout configurations. | Field | Description | | --- | --- | | activity\_id | The identifier assigned to this Activity by a [Worker](https://docs.temporal.io/workers#worker)
or user. | | activity\_type | The [type of Activity](https://docs.temporal.io/activity-definition#activity-type)
that was scheduled. | | namespace | Namespace of the Workflow that the [Activity](https://docs.temporal.io/activities)
resides in. | | task\_queue | The [Task Queue](https://docs.temporal.io/task-queue)
that this Activity Task was enqueued in. | | header | Information passed by the sender of the [Signal](https://docs.temporal.io/sending-messages#sending-signals)
that is copied into the [Workflow Task](https://docs.temporal.io/tasks#workflow-task)
. | | input | Information that is deserialized by the SDK to provide arguments to the [Workflow](https://docs.temporal.io/workflows)
function. | | schedule\_to\_close\_timeout | The amount of time that a caller will wait for Activity completion. Limits the amount of time that retries will be attempted for this Activity. | | schedule\_to\_start\_timeout | Limits the time that an Activity Task can stay in a Task Queue. This timeout cannot be retried. | | start\_to\_close\_timeout | Maximum amount of execution time that an Activity is allowed after being picked up by a Worker. This timeout is retryable. | | heartbeat\_timeout | Maximum amount of time allowed between successful Worker heartbeats. | | workflow\_task\_completed\_event\_id | The Id of the [WorkflowTaskCompleted](https://docs.temporal.io/references/events#workflowtaskcompleted)
that the Event was reported with. | | retry\_policy | The amount of retries as determined by the service's dynamic configuration. Retries will happen until `schedule_to_close_timeout` is reached. | ### ActivityTaskStarted[​](https://docs.temporal.io/references/events#activitytaskstarted "Direct link to ActivityTaskStarted") This [Event](https://docs.temporal.io/workflow-execution/event#event) type indicates that an [Activity Task Execution](https://docs.temporal.io/tasks#activity-task-execution) was started. The SDK Worker picked up the Activity Task and started processing the [Activity](https://docs.temporal.io/activities) invocation. `ActivityTaskStarted` is generated by the server when the Task is dispatched to the Worker, not when the Worker starts executing the Task. Note, however, that this Event is not written to History until the terminal Event (like [ActivityTaskCompleted](https://docs.temporal.io/references/events#activitytaskcompleted) or [ActivityTaskFailed](https://docs.temporal.io/references/events#activitytaskfailed) ) occurs. | Field | Description | | --- | --- | | scheduled\_event\_id | The Id of the [ActivityTaskScheduled](https://docs.temporal.io/references/events#activitytaskscheduled)
Event that this Task corresponds to. | | identity | Identifies the [Worker](https://docs.temporal.io/workers#worker)
that started the Task. | | request\_id | Identifies the Activity Task request. | | attempt | The number of attempts that have been made to complete this Task. | | last\_failure | Details from the most recent failure Event. Only assigned values if the Task has previously failed and been retried. | ### ActivityTaskCompleted[​](https://docs.temporal.io/references/events#activitytaskcompleted "Direct link to ActivityTaskCompleted") This [Event](https://docs.temporal.io/workflow-execution/event#event) type indicates that the [Activity Task](https://docs.temporal.io/tasks#activity-task) has completed. The SDK client has picked up and successfully completed the Activity Task. This Event type contains [Activity Execution](https://docs.temporal.io/activity-execution) results. | Field | Description | | --- | --- | | result | Serialized result of a completed [Activity](https://docs.temporal.io/activities)
. | | scheduled\_event\_id | The Id of the [ActivityTaskScheduled](https://docs.temporal.io/references/events#activitytaskscheduled)
Event that this completion Event corresponds to. | | started\_event\_id | The Id of the [ActivityTaskStarted](https://docs.temporal.io/references/events#activitytaskstarted)
Event that this Task corresponds to. | | identity | Identity of the [Worker](https://docs.temporal.io/workers#worker)
that completed this Task. | ### ActivityTaskFailed[​](https://docs.temporal.io/references/events#activitytaskfailed "Direct link to ActivityTaskFailed") This [Event](https://docs.temporal.io/workflow-execution/event#event) type indicates that the [Activity Task](https://docs.temporal.io/tasks#activity-task) has failed. The SDK client picked up the Activity Task but unsuccessfully completed it. This Event type contains [Activity Execution](https://docs.temporal.io/activity-execution) errors. | Field | Description | | --- | --- | | failure | Serialized result of a [Workflow](https://docs.temporal.io/workflows)
failure. | | scheduled\_event\_id | The Id of the [ActivityTaskScheduled](https://docs.temporal.io/references/events#activitytaskscheduled)
Event that this failure Event corresponds to. | | started\_event\_id | The Id of the [ActivityTaskStarted](https://docs.temporal.io/references/events#activitytaskstarted)
Event that this failure corresponds to. | | retry\_state | The reason provided for whether the Task should or shouldn't be retried. | ### ActivityTaskTimedOut[​](https://docs.temporal.io/references/events#activitytasktimedout "Direct link to ActivityTaskTimedOut") This [Event](https://docs.temporal.io/workflow-execution/event#event) type indicates that the Activity has timed out according to the [Temporal Server](https://docs.temporal.io/temporal-service/temporal-server) , due to one of these [Activity](https://docs.temporal.io/activities) timeouts: [Schedule-to-Close Timeout](https://docs.temporal.io/encyclopedia/detecting-activity-failures#schedule-to-close-timeout) and [Schedule-to-Start Timeout](https://docs.temporal.io/encyclopedia/detecting-activity-failures#schedule-to-start-timeout) . | Field | Description | | --- | --- | | failure | Serialized result of a [Workflow](https://docs.temporal.io/workflows)
failure. | | scheduled\_event\_id | The Id of the [ActivityTaskScheduled](https://docs.temporal.io/references/events#activitytaskscheduled)
Event that this timeout Event corresponds to. | | started\_event\_id | The Id of the [ActivityTaskStarted](https://docs.temporal.io/references/events#activitytaskstarted)
Event that this timeout corresponds to. | | retry\_state | The reason provided for whether the Task should or shouldn't be retried. | | timeout\_type | The type of timeout that led to this Event, e.g., Start-to-Close, Schedule-to-Close, Schedule-to-Start. | You can run a Workflow containing an Activity Execution that takes longer than the Start-to-Close Timeout you set and use a RetryPolicy that sets MaxAttempts to 1 so it does not retry indefinitely. When the Activity times out, you will observe that the `ActivityTaskTimedOut` Event contains other attributes missing from the documentation, including the type of timeout that led to the Event. ### ActivityTaskCancelRequested[​](https://docs.temporal.io/references/events#activitytaskcancelrequested "Direct link to ActivityTaskCancelRequested") This [Event](https://docs.temporal.io/workflow-execution/event#event) type indicates that a request to [cancel](https://docs.temporal.io/activity-execution#cancellation) the [Activity](https://docs.temporal.io/activities) has occurred. | Field | Description | | --- | --- | | scheduled\_event\_id | The Id of the [ActivityTaskScheduled](https://docs.temporal.io/references/events#activitytaskscheduled)
Event that this cancel Event corresponds to. | | workflow\_task\_completed\_event\_id | The Id of the [WorkflowTaskCompleted](https://docs.temporal.io/references/events#workflowtaskcompleted)
that the Event was reported with. | ### ActivityTaskCanceled[​](https://docs.temporal.io/references/events#activitytaskcanceled "Direct link to ActivityTaskCanceled") This [Event](https://docs.temporal.io/workflow-execution/event#event) type indicates that the [Activity](https://docs.temporal.io/activities) has been [canceled](https://docs.temporal.io/activity-execution#cancellation) . | Field | Description | | --- | --- | | details | Additional information reported by the Activity upon confirming cancelation. | | latest\_cancel\_requested\_event\_id | Id of the most recent [ActivityTaskCancelRequested](https://docs.temporal.io/references/events#activitytaskcancelrequested)
Event which refers to the same Activity. | | scheduled\_event\_id | The Id of the [ActivityTaskScheduled](https://docs.temporal.io/references/events#activitytaskscheduled)
Event that this cancelation corresponds to. | | started\_event\_id | The Id of the [ActivityTaskStarted](https://docs.temporal.io/references/events#activitytaskstarted)
Event that this cancelation corresponds to. | | identity | Identifies the [Worker](https://docs.temporal.io/workers#worker)
that requested cancelation. | ### TimerStarted[​](https://docs.temporal.io/references/events#timerstarted "Direct link to TimerStarted") This [Event](https://docs.temporal.io/workflow-execution/event#event) type indicates a timer has started. | Field | Description | | --- | --- | | timer\_id | The Id assigned for the timer by a [Worker](https://docs.temporal.io/workers#worker)
or user. | | start\_to\_fire\_timeout | Amount of time to elapse before the timer fires. | | workflow\_task\_completed\_event\_id | The Id of the [WorkflowTaskCompleted](https://docs.temporal.io/references/events#workflowtaskcompleted)
that the Event was reported with. | ### TimerFired[​](https://docs.temporal.io/references/events#timerfired "Direct link to TimerFired") This [Event](https://docs.temporal.io/workflow-execution/event#event) type indicates a timer has fired. | Field | Description | | --- | --- | | timer\_id | The Id assigned for the timer by a [Worker](https://docs.temporal.io/workers#worker)
or user. | | started\_event\_id | The Id of the [TimerStarted](https://docs.temporal.io/references/events#timerstarted)
Event itself. | ### TimerCanceled[​](https://docs.temporal.io/references/events#timercanceled "Direct link to TimerCanceled") This [Event](https://docs.temporal.io/workflow-execution/event#event) type indicates a Timer has been canceled. | Field | Description | | --- | --- | | timer\_id | The Id assigned for the timer by a [Worker](https://docs.temporal.io/workers#worker)
or user. | | started\_event\_id | The Id of the [TimerStarted](https://docs.temporal.io/references/events#timerstarted)
Event itself. | | workflow\_task\_completed\_event\_id | The Id of the [WorkflowTaskCompleted](https://docs.temporal.io/references/events#workflowtaskcompleted)
that the Event was reported with. | ### RequestCancelExternalWorkflowExecutionInitiated[​](https://docs.temporal.io/references/events#requestcancelexternalworkflowexecutioninitiated "Direct link to RequestCancelExternalWorkflowExecutionInitiated") This [Event](https://docs.temporal.io/workflow-execution/event#event) type indicates that a [Workflow](https://docs.temporal.io/workflows) has requested that the [Temporal Server](https://docs.temporal.io/temporal-service/temporal-server) try to cancel another Workflow. | Field | Description | | --- | --- | | workflow\_task\_completed\_event\_id | The Id of the [WorkflowTaskCompleted](https://docs.temporal.io/references/events#workflowtaskcompleted)
that the Event was reported with. | | namespace | [Namespace](https://docs.temporal.io/namespaces)
of the Workflow that\`s going to be signaled for execution. | | workflow\_execution | Identifies the Workflow and the run of the [Workflow Execution](https://docs.temporal.io/workflow-execution)
. | | child\_workflow\_only | Set to true if this Workflow is a child of the Workflow which issued the cancelation request. | | reason | Information provided by the user or client for Workflow cancelation. | ### RequestCancelExternalWorkflowExecutionFailed[​](https://docs.temporal.io/references/events#requestcancelexternalworkflowexecutionfailed "Direct link to RequestCancelExternalWorkflowExecutionFailed") This [Event](https://docs.temporal.io/workflow-execution/event#event) type indicates that [Temporal Server](https://docs.temporal.io/temporal-service/temporal-server) could not cancel the targeted [Workflow](https://docs.temporal.io/workflows) . This is usually because the target Workflow could not be found. | Field | Description | | --- | --- | | workflow\_task\_completed\_event\_id | The Id of the [WorkflowTaskCompleted](https://docs.temporal.io/references/events#workflowtaskcompleted)
that the Event was reported with. | | namespace | [Namespace](https://docs.temporal.io/namespaces)
of the Workflow that failed to cancel. | | workflow\_execution | Identifies the Workflow and the run of the [Workflow Execution](https://docs.temporal.io/workflow-execution)
. | | initiated\_event\_id | Id of the \[RequestCancelExternalWorkflowExecutionInitiated\] Event this failure corresponds to. | ### ExternalWorkflowExecutionCancelRequested[​](https://docs.temporal.io/references/events#externalworkflowexecutioncancelrequested "Direct link to ExternalWorkflowExecutionCancelRequested") This [Event](https://docs.temporal.io/workflow-execution/event#event) type indicates that the [Temporal Server](https://docs.temporal.io/temporal-service/temporal-server) has successfully requested the cancelation of the target [Workflow](https://docs.temporal.io/workflows) . | Field | Description | | --- | --- | | initiated\_event\_id | Id of the [RequestCancelExternalWorkflowExecutionInitiated](https://docs.temporal.io/references/events#requestcancelexternalworkflowexecutioninitiated)
Event that this cancelation request corresponds to. | | namespace | [Namespace](https://docs.temporal.io/namespaces)
of the Workflow that was requested to cancel. | | workflow\_execution | Identifies the Workflow and the run of the [Workflow Execution](https://docs.temporal.io/workflow-execution)
. | ### ExternalWorkflowExecutionSignaled[​](https://docs.temporal.io/references/events#externalworkflowexecutionsignaled "Direct link to ExternalWorkflowExecutionSignaled") This [Event](https://docs.temporal.io/workflow-execution/event#event) type indicates that the [Temporal Server](https://docs.temporal.io/temporal-service/temporal-server) has successfully [Signaled](https://docs.temporal.io/sending-messages#sending-signals) the targeted [Workflow](https://docs.temporal.io/workflows) . | Field | Description | | --- | --- | | initiated\_event\_id | Id of the [SignalExternalWorkflowExecutionInitiated](https://docs.temporal.io/references/events#signalexternalworkflowexecutioninitiated)
Event this Event corresponds to. | | namespace | [Namespace](https://docs.temporal.io/namespaces)
of the Workflow that was signaled to. | | workflow\_execution | Identifies the Workflow and the run of the [Workflow Execution](https://docs.temporal.io/workflow-execution)
. | ### MarkerRecorded[​](https://docs.temporal.io/references/events#markerrecorded "Direct link to MarkerRecorded") This [Event](https://docs.temporal.io/workflow-execution/event#event) type is transparent to the [Temporal Server](https://docs.temporal.io/temporal-service/temporal-server) . The Server will only store it and will not try to understand it. The SDK client may use it for local activities or side effects. | Field | Description | | --- | --- | | marker\_name | Identifies various markers. | | details | Serialized information recorded in the marker. | | workflow\_task\_completed\_event\_id | The Id of the [WorkflowTaskCompleted](https://docs.temporal.io/references/events#workflowtaskcompleted)
that the Event was reported with. | | header | Information passed by the sender of the [Signal](https://docs.temporal.io/sending-messages#sending-signals)
that is copied into the marker. | | failure | Serialized result of a [Workflow](https://docs.temporal.io/workflows)
failure. | ### StartChildWorkflowExecutionInitiated[​](https://docs.temporal.io/references/events#startchildworkflowexecutioninitiated "Direct link to StartChildWorkflowExecutionInitiated") This [Event](https://docs.temporal.io/workflow-execution/event#event) type indicates that the [Temporal Server](https://docs.temporal.io/temporal-service/temporal-server) will try to start a Child Workflow. | Field | Description | | --- | --- | | namespace | [Namespace](https://docs.temporal.io/namespaces)
of the Child Workflow. | | workflow\_id | Identifies the Child Workflow. | | workflow\_type | The name/type of Workflow that was initiated. | ### StartChildWorkflowExecutionFailed[​](https://docs.temporal.io/references/events#startchildworkflowexecutionfailed "Direct link to StartChildWorkflowExecutionFailed") This [Event](https://docs.temporal.io/workflow-execution/event#event) type indicates a [Child Workflow Execution](https://docs.temporal.io/child-workflows) cannot be started / triggered. It is usually due to a Child Workflow Id collision. | Field | Description | | --- | --- | | namespace | [Namespace](https://docs.temporal.io/namespaces)
of the Child Workflow. | | workflow\_id | Identifies the Child Workflow. | | workflow\_type | The name/type of Workflow that has failed. | | initiated\_event\_id | Id of the [StartChildWorkflowExecutionInitiated](https://docs.temporal.io/references/events#startchildworkflowexecutioninitiated)
Event this Event corresponds to. | | workflow\_task\_completed\_event\_id | The Id of the [WorkflowTaskCompleted](https://docs.temporal.io/references/events#workflowtaskcompleted)
that the Event was reported with. | ### ChildWorkflowExecutionStarted[​](https://docs.temporal.io/references/events#childworkflowexecutionstarted "Direct link to ChildWorkflowExecutionStarted") This [Event](https://docs.temporal.io/workflow-execution/event#event) type indicates a [Child Workflow Execution](https://docs.temporal.io/child-workflows) has successfully started / triggered. This would also cause the [WorkflowExecutionStarted](https://docs.temporal.io/references/events#workflowexecutionstarted) to be recorded for the Workflow that has started. | Field | Description | | --- | --- | | namespace | [Namespace](https://docs.temporal.io/namespaces)
of the Child Workflow. | | initiated\_event\_id | Id of the [StartChildWorkflowExecutionInitiated](https://docs.temporal.io/references/events#startchildworkflowexecutioninitiated)
Event this Event corresponds to. | | workflow\_execution | Identifies the Workflow and the run of the Workflow Execution. | | workflow\_type | The name/type of Workflow that has started execution. | | header | Information passed by the sender of the [Signal](https://docs.temporal.io/sending-messages#sending-signals)
that is copied into the Child Workflow Task. | ### ChildWorkflowExecutionCompleted[​](https://docs.temporal.io/references/events#childworkflowexecutioncompleted "Direct link to ChildWorkflowExecutionCompleted") This [Event](https://docs.temporal.io/workflow-execution/event#event) type indicates that the [Child Workflow Execution](https://docs.temporal.io/child-workflows) has successfully completed. This would also cause the [WorkflowExecutionCompleted](https://docs.temporal.io/references/events#workflowexecutioncompleted) to be recorded for the [Workflow](https://docs.temporal.io/workflows) that has completed. | Field | Description | | --- | --- | | result | Serialized result of the completed Child Workflow. | | namespace | [Namespace](https://docs.temporal.io/namespaces)
of the completed Child Workflow. | | workflow\_execution | Identifies the Workflow and the run of the [Workflow Execution](https://docs.temporal.io/workflow-execution)
. | | workflow\_type | The name/type of Workflow that was completed. | | initiated\_event\_id | Id of the [StartChildWorkflowExecutionInitiated](https://docs.temporal.io/references/events#startchildworkflowexecutioninitiated)
Event this Event corresponds to. | | started\_event\_id | Id of the [ChildWorkflowExecutionStarted](https://docs.temporal.io/references/events#childworkflowexecutionstarted)
Event this Event corresponds to. | ### ChildWorkflowExecutionFailed[​](https://docs.temporal.io/references/events#childworkflowexecutionfailed "Direct link to ChildWorkflowExecutionFailed") This [Event](https://docs.temporal.io/workflow-execution/event#event) type indicates that the [Child Workflow Execution](https://docs.temporal.io/child-workflows) has unsuccessfully completed. This would also cause the [WorkflowExecutionFailed](https://docs.temporal.io/references/events#workflowexecutionfailed) to be recorded for the Workflow that has failed. | Field | Description | | --- | --- | | failure | Serialized result of a [Workflow](https://docs.temporal.io/workflows)
failure. | | namespace | [Namespace](https://docs.temporal.io/namespaces)
of the Child Workflow that failed. | | workflow\_execution | Identifies the Workflow and the run of the [Workflow Execution](https://docs.temporal.io/workflow-execution)
. | | workflow\_type | The name/type of Workflow that has failed. | | initiated\_event\_id | Id of the [StartChildWorkflowExecutionInitiated](https://docs.temporal.io/references/events#startchildworkflowexecutioninitiated)
Event this Event corresponds to. | | started\_event\_id | Id of the [ChildWorkflowExecutionStarted](https://docs.temporal.io/references/events#childworkflowexecutionstarted)
Event this failure corresponds to. | | retry\_state | The reason provided for whether the Task should or shouldn't be retried. | ### ChildWorkflowExecutionCanceled[​](https://docs.temporal.io/references/events#childworkflowexecutioncanceled "Direct link to ChildWorkflowExecutionCanceled") This [Event](https://docs.temporal.io/workflow-execution/event#event) type indicates that the Child Workflow Execution has been canceled. This would also cause the [WorkflowExecutionCanceled](https://docs.temporal.io/references/events#workflowexecutioncanceled) to be recorded for the Workflow that was canceled. | Field | Description | | --- | --- | | details | Additional information reported by the Child Workflow upon cancelation. | | namespace | [Namespace](https://docs.temporal.io/namespaces)
of the Child Workflow that was canceled. | | workflow\_execution | Identifies the Workflow and the run of the [Workflow Execution](https://docs.temporal.io/workflow-execution)
. | | workflow\_type | The name/type of Workflow that was canceled. | | initiated\_event\_id | Id of the [StartChildWorkflowExecutionInitiated](https://docs.temporal.io/references/events#startchildworkflowexecutioninitiated)
Event this Event corresponds to. | | started\_event\_id | Id of the [ChildWorkflowExecutionStarted](https://docs.temporal.io/references/events#childworkflowexecutionstarted)
Event this cancelation corresponds to. | ### ChildWorkflowExecutionTimedOut[​](https://docs.temporal.io/references/events#childworkflowexecutiontimedout "Direct link to ChildWorkflowExecutionTimedOut") This Event type indicates that the [Child Workflow Execution](https://docs.temporal.io/child-workflows) has timed out by the [Temporal Server](https://docs.temporal.io/temporal-service/temporal-server) . This would also cause the [WorkflowExecutionTimeOut](https://docs.temporal.io/references/events#workflowexecutiontimedout) to be recorded for the Workflow that timed out. | Field | Description | | --- | --- | | namespace | [Namespace](https://docs.temporal.io/namespaces)
of the Child Workflow. | | workflow\_execution | Identifies the Workflow and the run of the Workflow Execution. | | workflow\_type | The name/type of Workflow that has timed out. | | initiated\_event\_id | Id of the [StartChildWorkflowExecutionInitiated](https://docs.temporal.io/references/events#startchildworkflowexecutioninitiated)
Event this Event corresponds to. | | started\_event\_id | Id of the [ChildWorkflowExecutionStarted](https://docs.temporal.io/references/events#childworkflowexecutionstarted)
Event that this timeout corresponds to. | | retry\_state | The reason provided for whether the Task should or shouldn't be retried. | ### ChildWorkflowExecutionTerminated[​](https://docs.temporal.io/references/events#childworkflowexecutionterminated "Direct link to ChildWorkflowExecutionTerminated") This [Event](https://docs.temporal.io/workflow-execution/event#event) type indicates that the Child Workflow Execution has been terminated. This would also cause the [WorkflowExecutionTerminated](https://docs.temporal.io/references/events#workflowexecutionterminated) to be recorded for the Workflow that was terminated. | Field | Description | | --- | --- | | namespace | [Namespace](https://docs.temporal.io/namespaces)
of the Child Workflow. | | workflow\_execution | Identifies the Workflow and the run of the Workflow Execution. | | workflow\_type | The name/type of Workflow that was terminated. | | initiated\_event\_id | Id of the [StartChildWorkflowExecutionInitiated](https://docs.temporal.io/references/events#startchildworkflowexecutioninitiated)
Event this Event corresponds to. | | started\_event\_id | Id of the [ChildWorkflowExecutionStarted](https://docs.temporal.io/references/events#childworkflowexecutionstarted)
Event that this termination corresponds to. | | retry\_state | The reason provided for whether the Task should or shouldn't be retried. | ### SignalExternalWorkflowExecutionInitiated[​](https://docs.temporal.io/references/events#signalexternalworkflowexecutioninitiated "Direct link to SignalExternalWorkflowExecutionInitiated") This [Event](https://docs.temporal.io/workflow-execution/event#event) type indicates that the [Temporal Server](https://docs.temporal.io/temporal-service/temporal-server) will try to [Signal](https://docs.temporal.io/sending-messages#sending-signals) the targeted [Workflow](https://docs.temporal.io/workflows) . This Event type contains the Signal name, as well as a Signal payload. | Field | Description | | --- | --- | | workflow\_task\_completed\_event\_id | The Id of the [WorkflowTaskCompleted](https://docs.temporal.io/references/events#workflowtaskcompleted)
that the Event was reported with. | | namespace | [Namespace](https://docs.temporal.io/namespaces)
of the Workflow that's to be signaled. | | workflow\_execution | Identifies the Workflow and the run of the [Workflow Execution](https://docs.temporal.io/workflow-execution)
. | | signal\_name | The name/type of Signal to be fired. | | input | Information that is deserialized by the SDK to provide arguments to the Workflow Function. | | child\_workflow\_only | Set to true if this Workflow is a child of the Workflow which issued the cancelation request. | | header | Information to be passed from the Signal to the targeted Workflow. | ### SignalExternalWorkflowExecutionFailed[​](https://docs.temporal.io/references/events#signalexternalworkflowexecutionfailed "Direct link to SignalExternalWorkflowExecutionFailed") This [Event](https://docs.temporal.io/workflow-execution/event#event) type indicates that the [Temporal Server](https://docs.temporal.io/temporal-service/temporal-server) cannot Signal the targeted [Workflow](https://docs.temporal.io/workflows) , usually because the Workflow could not be found. | Field | Description | | --- | --- | | workflow\_task\_completed\_event\_id | The Id of the [WorkflowTaskCompleted](https://docs.temporal.io/references/events#workflowtaskcompleted)
that the Event was reported with. | | namespace | [Namespace](https://docs.temporal.io/namespaces)
of the Workflow that failed to execute. | | workflow\_execution | Identifies the Workflow and the run of the [Workflow Execution](https://docs.temporal.io/workflow-execution)
. | | initiated\_event\_id | Id of the [RequestCancelExternalWorkflowExecutionInitiated](https://docs.temporal.io/references/events#requestcancelexternalworkflowexecutioninitiated)
Event this failure [signal](https://docs.temporal.io/sending-messages#sending-signals)
corresponds to. | ### UpsertWorkflowSearchAttributes[​](https://docs.temporal.io/references/events#upsertworkflowsearchattributes "Direct link to UpsertWorkflowSearchAttributes") This [Event](https://docs.temporal.io/workflow-execution/event#event) type indicates that the Workflow [Search Attributes](https://docs.temporal.io/search-attribute) should be updated and synchronized with the visibility store. | Field | Description | | --- | --- | | workflow\_task\_completed\_event\_id | The [WorkflowTaskCompleted](https://docs.temporal.io/references/events#workflowtaskcompleted)
Event reported the Event with this Id. | | search\_attributes | Provides data for setting up a Workflow's [Search Attributes](https://docs.temporal.io/search-attribute)
. | ### WorkflowExecutionUpdateAcceptedEvent[​](https://docs.temporal.io/references/events#workflowexecutionupdateacceptedevent "Direct link to WorkflowExecutionUpdateAcceptedEvent") This [Event](https://docs.temporal.io/workflow-execution/event#event) type indicates that a [Workflow Execution](https://docs.temporal.io/workflow-execution) has accepted an [Update](https://docs.temporal.io/sending-messages#sending-updates) for execution. The original request input payload is both indicated and stored by this Event, as it generates no Event when initially requesting an Update. | Field | Description | | --- | --- | | protocol\_instance\_id | The instance of the Update protocol with this Id is executing this Update. | | accepted\_request\_message\_id | The Id of the request message sent by [Temporal Server](https://docs.temporal.io/temporal-service/temporal-server)
to the [Worker](https://docs.temporal.io/workers#worker)
. | | accepted\_request\_sequencing\_event\_id | Execute this Update after the Event with this Id. | | accepted\_request | The request input and metadata initially provided by the invoker of the Update and subsequently relayed by Temporal Server to the Worker for acceptance and execution. | ### WorkflowExecutionUpdateCompletedEvent[​](https://docs.temporal.io/references/events#workflowexecutionupdatecompletedevent "Direct link to WorkflowExecutionUpdateCompletedEvent") This [Event](https://docs.temporal.io/workflow-execution/event#event) type indicates that a [Workflow Execution](https://docs.temporal.io/workflow-execution) has executed an [Update](https://docs.temporal.io/sending-messages#sending-updates) to completion. | Field | Description | | --- | --- | | meta | The metadata associated with this Update, sourced from the initial request. | | accepted\_event\_id | The Id of the [WorkflowExecutionUpdateAcceptedEvent](https://docs.temporal.io/references/events#workflowexecutionupdateacceptedevent)
The Platform accepted this Update for execution. | | outcome | The outcome of execution of this Update whether the execution resulted in a success or a failure. | ### NexusOperationScheduled[​](https://docs.temporal.io/references/events#nexusoperationscheduled "Direct link to NexusOperationScheduled") This Event type indicates that a Nexus Operation scheduled by a caller Workflow. The caller's [Nexus Machinery](https://docs.temporal.io/glossary#nexus-machinery) will attempt to start the Nexus Operation. This Event type contains Nexus Operation input and the Operation request ID. | Field | Description | | --- | --- | | endpoint | Endpoint name, must exist in the endpoint registry. | | service | Service name. | | operation | Operation name. | | input | Input for the operation. The server converts this into Nexus request content and the appropriate content headers internally when sending the StartOperation request. On the handler side, if it is also backed by Temporal, the content is transformed back to the original Payload stored in this event. | | schedule\_to\_close\_timeout | Schedule-to-close timeout for this operation. Indicates how long the caller is willing to wait for operation completion. Calls are retried internally by the server. | | nexus\_header | Header to attach to the Nexus request. Note these headers are not the same as Temporal headers on internal activities and child Workflows, these are transmitted to Nexus operations that may be external and are not traditional payloads. | | workflow\_task\_completed\_event\_id | The ID of the [WorkflowTaskCompleted](https://docs.temporal.io/references/events#workflowtaskcompleted)
event that the corresponding ScheduleNexusOperation command was reported with. | | request\_id | A unique ID generated by the History Service upon creation of this event. The ID will be transmitted with all Nexus StartOperation requests and is used as an idempotency key. | | endpoint\_id | Endpoint ID as resolved in the endpoint registry at the time this event was generated. This is stored on the event and used internally by the server in case the endpoint is renamed from the time the event was originally scheduled. | ### NexusOperationStarted[​](https://docs.temporal.io/references/events#nexusoperationstarted "Direct link to NexusOperationStarted") This Event type indicates that a Nexus Operation Execution was started. This Event is added to the caller's Workflow History for Asynchronous Nexus Operations, for example those that are backed by a Workflow. The Event is not added to the caller's Workflow History for Synchronous Nexus Operations, since they transition directly to [NexusOperationCompleted](https://docs.temporal.io/references/events#nexusoperationcompleted) or another final state such as [NexusOperationFailed](https://docs.temporal.io/references/events#nexusoperationfailed) when the response is provided synchronously by the Nexus handler. | Field | Description | | --- | --- | | scheduled\_event\_id | The ID of the [NexusOperationScheduled](https://docs.temporal.io/references/events#nexusoperationscheduled)
event this task corresponds to. | | operation\_token | The operation token returned by the Nexus handler in the response to the StartOperation request. This token is used when canceling the operation. | | request\_id | The request ID allocated at schedule time. | ### NexusOperationCompleted[​](https://docs.temporal.io/references/events#nexusoperationcompleted "Direct link to NexusOperationCompleted") This Event type indicates that a Nexus Operation has completed successfully. The caller's Workflow History records the result of a successful Nexus Operation with this event for synchronous and asynchronous Nexus Operations. This Event type contains Nexus Operation results. | Field | Description | | --- | --- | | scheduled\_event\_id | The ID of the [NexusOperationScheduled](https://docs.temporal.io/references/events#nexusoperationscheduled)
event. Uniquely identifies this operation. | | result | Serialized result of the Nexus operation. The response of the Nexus handler. Delivered either via a completion callback or as a response to a synchronous operation. | | request\_id | The request ID allocated at schedule time. | ### NexusOperationFailed[​](https://docs.temporal.io/references/events#nexusoperationfailed "Direct link to NexusOperationFailed") This Event type indicates that a Nexus Operation has failed. The caller's Workflow History records a failed Nexus Operation with this event both for synchronous and asynchronous Nexus Operations. For example, when a Nexus Handler responds synchronously with a non-retryable error or when a Workflow that backs an Operation fails, resulting in a [WorkflowExecutionFailed](https://docs.temporal.io/references/events#workflowexecutionfailed) Event. When an SDK client picks up a Nexus Operation, the Nexus handler asynchronously starts an underlying Workflow, which subsequently results in [WorkflowExecutionFailed](https://docs.temporal.io/references/events#workflowexecutionfailed) . This Event type contains a Nexus Operation failure. | Field | Description | | --- | --- | | scheduled\_event\_id | The ID of the [NexusOperationScheduled](https://docs.temporal.io/references/events#nexusoperationscheduled)
\` event. Uniquely identifies this operation. | | failure | Failure details. A NexusOperationFailureInfo wrapping an ApplicationFailureInfo. | | request\_id | The request ID allocated at schedule time. | ### NexusOperationTimedOut[​](https://docs.temporal.io/references/events#nexusoperationtimedout "Direct link to NexusOperationTimedOut") This Event type indicates that a Nexus Operation has timed out according to the Temporal Server, due to one of these Nexus Operation timeouts: Schedule-to-Close Timeout. | Field | Description | | --- | --- | | scheduled\_event\_id | The ID of the [NexusOperationScheduled](https://docs.temporal.io/references/events#nexusoperationscheduled)
\` event. Uniquely identifies this operation. | | failure | Failure details. A NexusOperationFailureInfo wrapping a CanceledFailureInfo. | | request\_id | The request ID allocated at schedule time. | ### NexusOperationCancelRequested[​](https://docs.temporal.io/references/events#nexusoperationcancelrequested "Direct link to NexusOperationCancelRequested") This Event type indicates that the Workflow that scheduled a Nexus Operation requested to cancel it. | Field | Description | | --- | --- | | scheduled\_event\_id | The id of the [NexusOperationScheduled](https://docs.temporal.io/references/events#nexusoperationscheduled)
\` event this cancel request corresponds to. | | workflow\_task\_completed\_event\_id | The [WorkflowTaskCompleted](https://docs.temporal.io/references/events#workflowtaskcompleted)
event that the corresponding RequestCancelNexusOperation command was reported with. | ### NexusOperationCanceled[​](https://docs.temporal.io/references/events#nexusoperationcanceled "Direct link to NexusOperationCanceled") This Event type indicates that a Nexus Operation has resolved as canceled. | Field | Description | | --- | --- | | scheduled\_event\_id | The ID of the [NexusOperationScheduled](https://docs.temporal.io/references/events#nexusoperationscheduled)
\` event. Uniquely identifies this operation. | | failure | Cancellation details. | | request\_id | The request ID allocated at schedule time. | * [WorkflowExecutionStarted](https://docs.temporal.io/references/events#workflowexecutionstarted) * [WorkflowExecutionCompleted](https://docs.temporal.io/references/events#workflowexecutioncompleted) * [WorkflowExecutionFailed](https://docs.temporal.io/references/events#workflowexecutionfailed) * [WorkflowExecutionTimedOut](https://docs.temporal.io/references/events#workflowexecutiontimedout) * [WorkflowExecutionCancelRequested](https://docs.temporal.io/references/events#workflowexecutioncancelrequested) * [WorkflowExecutionCanceled](https://docs.temporal.io/references/events#workflowexecutioncanceled) * [WorkflowExecutionSignaled](https://docs.temporal.io/references/events#workflowexecutionsignaled) * [WorkflowExecutionTerminated](https://docs.temporal.io/references/events#workflowexecutionterminated) * [WorkflowExecutionContinuedAsNew](https://docs.temporal.io/references/events#workflowexecutioncontinuedasnew) * [WorkflowExecutionOptionsUpdated](https://docs.temporal.io/references/events#workflowexecutionoptionsupdated) * [WorkflowTaskScheduled](https://docs.temporal.io/references/events#workflowtaskscheduled) * [WorkflowTaskStarted](https://docs.temporal.io/references/events#workflowtaskstarted) * [WorkflowTaskCompleted](https://docs.temporal.io/references/events#workflowtaskcompleted) * [WorkflowTaskTimedOut](https://docs.temporal.io/references/events#workflowtasktimedout) * [WorkflowTaskFailed](https://docs.temporal.io/references/events#workflowtaskfailed) * [ActivityTaskScheduled](https://docs.temporal.io/references/events#activitytaskscheduled) * [ActivityTaskStarted](https://docs.temporal.io/references/events#activitytaskstarted) * [ActivityTaskCompleted](https://docs.temporal.io/references/events#activitytaskcompleted) * [ActivityTaskFailed](https://docs.temporal.io/references/events#activitytaskfailed) * [ActivityTaskTimedOut](https://docs.temporal.io/references/events#activitytasktimedout) * [ActivityTaskCancelRequested](https://docs.temporal.io/references/events#activitytaskcancelrequested) * [ActivityTaskCanceled](https://docs.temporal.io/references/events#activitytaskcanceled) * [TimerStarted](https://docs.temporal.io/references/events#timerstarted) * [TimerFired](https://docs.temporal.io/references/events#timerfired) * [TimerCanceled](https://docs.temporal.io/references/events#timercanceled) * [RequestCancelExternalWorkflowExecutionInitiated](https://docs.temporal.io/references/events#requestcancelexternalworkflowexecutioninitiated) * [RequestCancelExternalWorkflowExecutionFailed](https://docs.temporal.io/references/events#requestcancelexternalworkflowexecutionfailed) * [ExternalWorkflowExecutionCancelRequested](https://docs.temporal.io/references/events#externalworkflowexecutioncancelrequested) * [ExternalWorkflowExecutionSignaled](https://docs.temporal.io/references/events#externalworkflowexecutionsignaled) * [MarkerRecorded](https://docs.temporal.io/references/events#markerrecorded) * [StartChildWorkflowExecutionInitiated](https://docs.temporal.io/references/events#startchildworkflowexecutioninitiated) * [StartChildWorkflowExecutionFailed](https://docs.temporal.io/references/events#startchildworkflowexecutionfailed) * [ChildWorkflowExecutionStarted](https://docs.temporal.io/references/events#childworkflowexecutionstarted) * [ChildWorkflowExecutionCompleted](https://docs.temporal.io/references/events#childworkflowexecutioncompleted) * [ChildWorkflowExecutionFailed](https://docs.temporal.io/references/events#childworkflowexecutionfailed) * [ChildWorkflowExecutionCanceled](https://docs.temporal.io/references/events#childworkflowexecutioncanceled) * [ChildWorkflowExecutionTimedOut](https://docs.temporal.io/references/events#childworkflowexecutiontimedout) * [ChildWorkflowExecutionTerminated](https://docs.temporal.io/references/events#childworkflowexecutionterminated) * [SignalExternalWorkflowExecutionInitiated](https://docs.temporal.io/references/events#signalexternalworkflowexecutioninitiated) * [SignalExternalWorkflowExecutionFailed](https://docs.temporal.io/references/events#signalexternalworkflowexecutionfailed) * [UpsertWorkflowSearchAttributes](https://docs.temporal.io/references/events#upsertworkflowsearchattributes) * [WorkflowExecutionUpdateAcceptedEvent](https://docs.temporal.io/references/events#workflowexecutionupdateacceptedevent) * [WorkflowExecutionUpdateCompletedEvent](https://docs.temporal.io/references/events#workflowexecutionupdatecompletedevent) * [NexusOperationScheduled](https://docs.temporal.io/references/events#nexusoperationscheduled) * [NexusOperationStarted](https://docs.temporal.io/references/events#nexusoperationstarted) * [NexusOperationCompleted](https://docs.temporal.io/references/events#nexusoperationcompleted) * [NexusOperationFailed](https://docs.temporal.io/references/events#nexusoperationfailed) * [NexusOperationTimedOut](https://docs.temporal.io/references/events#nexusoperationtimedout) * [NexusOperationCancelRequested](https://docs.temporal.io/references/events#nexusoperationcancelrequested) * [NexusOperationCanceled](https://docs.temporal.io/references/events#nexusoperationcanceled) --- # Event History walkthrough with the Java SDK | Temporal Platform Documentation [Skip to main content](https://docs.temporal.io/encyclopedia/event-history/event-history-java#__docusaurus_skipToContent_fallback) On this page In order to understand how Workflow Replay works, this page will go through the following walkthroughs: 1. [How Workflow Code Maps to Commands](https://docs.temporal.io/encyclopedia/event-history/event-history-java#How-Workflow-Code-Maps-To-Commands) 2. [How Workflow Commands Map to Events](https://docs.temporal.io/encyclopedia/event-history/event-history-java#How-Workflow-Commands-Map-To-Events) 3. [How History Replay Provides Durable Execution](https://docs.temporal.io/encyclopedia/event-history/event-history-java#How-History-Replay-Provides-Durable-Execution) 4. [Example of a Non-Deterministic Workflow](https://docs.temporal.io/encyclopedia/event-history/event-history-java#Example-of-Non-Deterministic-Workflow) How Workflow Code Maps to Commands[​](https://docs.temporal.io/encyclopedia/event-history/event-history-java#How-Workflow-Code-Maps-To-Commands "Direct link to How Workflow Code Maps to Commands") ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- This walkthrough will cover how the Workflow code maps to Commands that get sent to the Temporal Service, letting the Temporal Service know what to do. Image 1 out of 11 ![Slide 1](https://learn.temporal.io/courses/temporal-102/java/event-history-walkthrough/code-commands/code-commands.001.jpeg)❮❯ How Workflow Commands Map to Events[​](https://docs.temporal.io/encyclopedia/event-history/event-history-java#How-Workflow-Commands-Map-To-Events "Direct link to How Workflow Commands Map to Events") -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- The Commands that are sent to the Temporal Service are then turned into Events, which build up the Event History. The Event History is a detailed log of Events that occur during the lifecycle of a Workflow Execution, such as the execution of Workflow Tasks or Activity Tasks. Event Histories are persisted to the database used by the Temporal Service, so they're durable, and will even survive a crash of the Temporal Service itself. These Events are what are used to recreate a Workflow Execution's state in the case of failure. Image 1 out of 14 ![Slide 1](https://learn.temporal.io/courses/temporal-102/java/event-history-walkthrough/commands-events/commands-events.001.jpeg)❮❯ How History Replay Provides Durable Execution[​](https://docs.temporal.io/encyclopedia/event-history/event-history-java#How-History-Replay-Provides-Durable-Execution "Direct link to How History Replay Provides Durable Execution") -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Now that you have seen how code maps to Commands, and how Commands map to Events, this next walkthrough will take a look at how Temporal uses Replay with the Events to provide Durable Execution and restore a Workflow Execution in the case of a failure. This code walkthrough will begin by walking through a Workflow Execution, describing how the code maps to Commands and Events. There will then be a Worker crash halfway through, explaining how Temporal uses Replay to recover the state of the Workflow Execution, ultimately resulting in a completed execution that's identical to one that had not crashed. Image 1 out of 60 ![Slide 1](https://learn.temporal.io/courses/temporal-102/java/event-history-walkthrough/history-replay/history-replay.001.jpeg)❮❯ Example of a Non-Deterministic Workflow[​](https://docs.temporal.io/encyclopedia/event-history/event-history-java#Example-of-Non-Deterministic-Workflow "Direct link to Example of a Non-Deterministic Workflow") ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ Now that Replay has been covered, this section will explain why Workflows need to be [deterministic](https://docs.temporal.io/workflow-definition#deterministic-constraints) in order for Replay to work. A Workflow is deterministic if every execution of its Workflow Definition produces the same Commands in the same sequence given the same input. As mentioned in the [`How History Replay Provides Durable Execution`](https://docs.temporal.io/encyclopedia/event-history/event-history-java#How-History-Replay-Provides-Durable-Execution) walkthrough, in the case of a failure, a Worker requests the Event History to replay it. During Replay, the Worker runs the Workflow code again to produce a set of Commands which is compared against the sequence of Commands in the Event History. When there’s a mismatch between the expected sequence of Commands the Worker expects based on the Event History and the actual sequence produced during Replay (due to non-determinism), Replay will be unable to continue. To better understand why Workflows need to be deterministic, it's helpful to look at a Workflow Definition that violates it. In this case, this code will walk through a Workflow Definition that breaks the determinism constraint with a random number generator. Image 1 out of 13 ![Slide 1](https://learn.temporal.io/courses/temporal-102/java/event-history-walkthrough/nondeterministic-workflow/nondeterministic-workflow.001.jpeg)❮❯ Note that non-deterministic failures do not fail the Workflow Execution by default. A non-deterministic failure is considered a [Workflow Task Failure](https://docs.temporal.io/references/failures#workflow-task-failures) which is considered a transient failure, meaning it retries over and over. Users can also fix the source of non-determinism, perhaps by removing the Activity, and then restart the Workers. This means that this type of failure can recover by itself. You can also use a strategy called versioning to address this non-determinism error. See [versioning](https://docs.temporal.io/develop/java/versioning) to learn more. For more information on how Temporal handles Durable Execution or to see these slides in a video format with more explanation, check out our free, self-paced courses: [Temporal 102](https://learn.temporal.io/courses/temporal_102/) and [Versioning Workflows](https://learn.temporal.io/courses/versioning/) . Temporal Applications Support Non-Deterministic Operations[​](https://docs.temporal.io/encyclopedia/event-history/event-history-java#temporal-applications-support-non-deterministic-operations "Direct link to Temporal Applications Support Non-Deterministic Operations") ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- We want to emphasize that although your Workflows themselves need to be deterministic, your application itself does not! Remember that pretty much anything that interacts with the external world is inherently non-deterministic: * Calling LLM APIs * Querying databases * Reading or writing files * Making HTTP requests to external services **Good news**: Your Temporal application can absolutely handle all of these operations. While your Workflow must be deterministic, your application absolutely can handle any type of non-deterministic operation, including those listed above. This gives you the best of both worlds—the crash-proof reliability of a Workflow and the resiliency of Activities which have built-in support for retries. * [How Workflow Code Maps to Commands](https://docs.temporal.io/encyclopedia/event-history/event-history-java#How-Workflow-Code-Maps-To-Commands) * [How Workflow Commands Map to Events](https://docs.temporal.io/encyclopedia/event-history/event-history-java#How-Workflow-Commands-Map-To-Events) * [How History Replay Provides Durable Execution](https://docs.temporal.io/encyclopedia/event-history/event-history-java#How-History-Replay-Provides-Durable-Execution) * [Example of a Non-Deterministic Workflow](https://docs.temporal.io/encyclopedia/event-history/event-history-java#Example-of-Non-Deterministic-Workflow) * [Temporal Applications Support Non-Deterministic Operations](https://docs.temporal.io/encyclopedia/event-history/event-history-java#temporal-applications-support-non-deterministic-operations) --- # Manage Namespaces - TypeScript SDK | Temporal Platform Documentation [Skip to main content](https://docs.temporal.io/develop/typescript/client/namespaces#__docusaurus_skipToContent_fallback) On this page How to create and manage Namespaces[​](https://docs.temporal.io/develop/typescript/client/namespaces#namespaces "Direct link to How to create and manage Namespaces") ---------------------------------------------------------------------------------------------------------------------------------------------------------------------- You can create, update, deprecate or delete your [Namespaces](https://docs.temporal.io/namespaces) using either the Temporal CLI or SDK APIs. Use Namespaces to isolate your Workflow Executions according to your needs. For example, you can use Namespaces to match the development lifecycle by having separate `dev` and `prod` Namespaces. You could also use them to ensure Workflow Executions between different teams never communicate - such as ensuring that the `teamA` Namespace never impacts the `teamB` Namespace. On Temporal Cloud, use the [Temporal Cloud UI](https://docs.temporal.io/cloud/namespaces#create-a-namespace) to create and manage a Namespace from the UI, or [tcld commands](https://docs.temporal.io/cloud/tcld/namespace/) to manage Namespaces from the command-line interface. On self-hosted Temporal Service, you can register and manage your Namespaces using the Temporal CLI (recommended) or programmatically using APIs. Note that these APIs and Temporal CLI commands will not work with Temporal Cloud. Use a custom [Authorizer](https://docs.temporal.io/self-hosted-guide/security#authorizer-plugin) on your Frontend Service in the Temporal Service to set restrictions on who can create, update, or deprecate Namespaces. You must register a Namespace with the Temporal Service before setting it in the Temporal Client. ### How to register Namespaces[​](https://docs.temporal.io/develop/typescript/client/namespaces#register-namespace "Direct link to How to register Namespaces") Registering a Namespace creates a Namespace on the Temporal Service or Temporal Cloud. On Temporal Cloud, use the [Temporal Cloud UI](https://docs.temporal.io/cloud/namespaces#create-a-namespace) or [tcld commands](https://docs.temporal.io/cloud/tcld/namespace/) to create Namespaces. On self-hosted Temporal Service, you can register your Namespaces using the Temporal CLI (recommended) or programmatically using APIs. Note that these APIs and Temporal CLI commands will not work with Temporal Cloud. Use a custom [Authorizer](https://docs.temporal.io/self-hosted-guide/security#authorizer-plugin) on your Frontend Service in the Temporal Service to set restrictions on who can create, update, or deprecate Namespaces. ### How to manage Namespaces[​](https://docs.temporal.io/develop/typescript/client/namespaces#manage-namespaces "Direct link to How to manage Namespaces") You can get details for your Namespaces, update Namespace configuration, and deprecate or delete your Namespaces. On Temporal Cloud, use the [Temporal Cloud UI](https://docs.temporal.io/cloud/namespaces#create-a-namespace) or [tcld commands](https://docs.temporal.io/cloud/tcld/namespace/) to manage Namespaces. On self-hosted Temporal Service, you can manage your registered Namespaces using the Temporal CLI (recommended) or programmatically using APIs. Note that these APIs and Temporal CLI commands will not work with Temporal Cloud. Use a custom [Authorizer](https://docs.temporal.io/self-hosted-guide/security#authorizer-plugin) on your Frontend Service in the Temporal Service to set restrictions on who can create, update, or deprecate Namespaces. You must register a Namespace with the Temporal Service before setting it in the Temporal Client. * [How to create and manage Namespaces](https://docs.temporal.io/develop/typescript/client/namespaces#namespaces) --- # Best Practices - TypeScript SDK | Temporal Platform Documentation [Skip to main content](https://docs.temporal.io/develop/typescript/best-practices#__docusaurus_skipToContent_fallback) On this page ![TypeScript SDK Banner](https://docs.temporal.io/assets/images/banner-typescript-temporal-d8a24070726a0d14cb4d1aab011db927.png) Best practices[​](https://docs.temporal.io/develop/typescript/best-practices#best-practices "Direct link to Best practices") ----------------------------------------------------------------------------------------------------------------------------- * [Testing](https://docs.temporal.io/develop/typescript/best-practices/testing-suite) * [Debugging](https://docs.temporal.io/develop/typescript/best-practices/debugging) * [Converters and encryption](https://docs.temporal.io/develop/typescript/converters-and-encryption) * [Entity pattern](https://docs.temporal.io/develop/typescript/best-practices/entity-pattern) * [Best practices](https://docs.temporal.io/develop/typescript/best-practices#best-practices) --- # Payload Converter | Temporal Platform Documentation [Skip to main content](https://docs.temporal.io/payload-converter#__docusaurus_skipToContent_fallback) On this page This page discusses [Payload Converter](https://docs.temporal.io/payload-converter#payload-converter) . What is a Payload Converter?[​](https://docs.temporal.io/payload-converter#payload-converter "Direct link to What is a Payload Converter?") -------------------------------------------------------------------------------------------------------------------------------------------- A Payload Converter serializes data, converting values to bytes and back. When you initiate a Workflow Execution through a Client and pass data as input, the input is serialized using a Data Converter that runs it through a set of Payload Converters. When your Workflow Execution starts, this data input is deserialized and passed as input to your Workflow. ### Composite Data Converters[​](https://docs.temporal.io/payload-converter#composite-data-converters "Direct link to Composite Data Converters") A Composite Data Converter is used to apply custom, type-specific Payload Converters in a specified order. A Composite Data Converter can be comprised of custom rules that you created, and it can also leverage the default Data Converters built into Temporal. In fact, the default Data Converter logic is implemented internally in the Temporal source as a Composite Data Converter. It defines these rules in this order: defaultDataConverter = NewCompositeDataConverter( NewNilPayloadConverter(), NewByteSlicePayloadConverter(), NewProtoJSONPayloadConverter(), NewProtoPayloadConverter(), NewJSONPayloadConverter(),) The order in which the Payload Converters are applied is important. During serialization, the Data Converter tries the Payload Converters in that specific order until a Payload Converter returns a non-nil Payload. A custom PayloadConverter must implement the functions: * `FromPayload` (for a single value) or * `FromPayloads` (for a list of values) to convert to values from a Payload, and * `ToPayload` (for a single value) or * `ToPayloads` (for a list of values) to convert values to a Payload. Defining a new Composite Data Converter is not always necessary to implement custom data handling. Each SDK allows you to override or configure the default Converter with a custom Payload Codec. * [What is a Payload Converter?](https://docs.temporal.io/payload-converter#payload-converter) * [Composite Data Converters](https://docs.temporal.io/payload-converter#composite-data-converters) --- # Nexus Registry | Temporal Platform Documentation [Skip to main content](https://docs.temporal.io/nexus/registry#__docusaurus_skipToContent_fallback) On this page SUPPORT, STABILITY, and DEPENDENCY INFO Temporal Nexus is now [Generally Available](https://docs.temporal.io/evaluate/development-production-features/release-stages#general-availability) for [Temporal Cloud](https://docs.temporal.io/cloud/nexus) and [self-hosted deployments](https://docs.temporal.io/production-deployment/self-hosted-guide/nexus) . The [Nexus Registry](https://docs.temporal.io/glossary#nexus-registry) manages [Nexus Endpoints](https://docs.temporal.io/nexus/endpoints) . Developers can advertise available Endpoints and Services, so others can find and use them in their caller Workflows. Adding an Endpoint to the Registry deploys it for immediate runtime use. Endpoint names must be unique within the Registry. In Temporal Cloud, the Registry is global across your entire Account, spanning all Namespaces. In self-hosted deployments, it is scoped to a Cluster. View and manage Nexus Endpoints[​](https://docs.temporal.io/nexus/registry#view-and-manage-nexus-endpoints "Direct link to View and manage Nexus Endpoints") ------------------------------------------------------------------------------------------------------------------------------------------------------------- Manage Endpoints using the Temporal UI, CLI, Terraform provider, or [Cloud Ops API](https://docs.temporal.io/ops) . RESOURCES * [Terraform support](https://docs.temporal.io/cloud/terraform-provider#manage-temporal-cloud-nexus-endpoints-with-terraform) for Temporal Cloud. * [tcld nexus](https://docs.temporal.io/cloud/tcld/nexus) for Temporal Cloud. * [temporal operator nexus](https://docs.temporal.io/cli/operator#nexus) for self-hosted deployments. ### Search for a Nexus Endpoint[​](https://docs.temporal.io/nexus/registry#search-for-a-nexus-endpoint "Direct link to Search for a Nexus Endpoint") Search by Endpoint name or target Namespace. ![Nexus Endpoints](https://docs.temporal.io/img/cloud/nexus/nexus-endpoints-ss.png) Nexus Endpoints The details page shows the target Namespace, Task Queue, and description rendered as markdown. ![Nexus Billing](https://docs.temporal.io/img/cloud/nexus/nexus-billing-ss.png) Nexus Billing ### Create a Nexus Endpoint[​](https://docs.temporal.io/nexus/registry#create-a-nexus-endpoint "Direct link to Create a Nexus Endpoint") Creating an Endpoint includes setting an Access Policy - the allowlist of caller Namespaces permitted to use the Endpoint. No callers are allowed by default, even if in the same Namespace as the Endpoint target. ![Create Nexus Endpoint](https://docs.temporal.io/img/cloud/nexus/create-nexus-endpoint-ss.png) Create Nexus Endpoint ### Edit a Nexus Endpoint[​](https://docs.temporal.io/nexus/registry#edit-a-nexus-endpoint "Direct link to Edit a Nexus Endpoint") Everything except the Endpoint name can be edited. New Operations route to the updated target immediately. Changing the target Namespace * **In-flight async Operations** - Completion callbacks point to the original handler Namespace and are unaffected, but Cancel requests route to the new target. * **Workflow ID uniqueness** - IDs are scoped per Namespace. Signal-With-Start creates a new Workflow in the new target even if the same ID is active in the old target, resulting in potential duplicates. * **Recommendation:** Drain existing Nexus Operations and underlying handler Workflows before changing the target Namespace. ![Edit Nexus Endpoint](https://docs.temporal.io/img/cloud/nexus/target-namespace-ss.png) Edit Nexus Endpoint ### Configure runtime access controls[​](https://docs.temporal.io/nexus/registry#configure-runtime-access-controls "Direct link to Configure runtime access controls") The Access Policy controls which caller Namespaces can use an Endpoint at runtime. No callers are allowed by default. See [Runtime Access Controls](https://docs.temporal.io/nexus/security#runtime-access-controls) for details. ![Configure runtime access controls](https://docs.temporal.io/img/cloud/nexus/create-nexus-endpoint-ss.png) Configure runtime access controls Roles and permissions[​](https://docs.temporal.io/nexus/registry#roles-and-permissions "Direct link to Roles and permissions") ------------------------------------------------------------------------------------------------------------------------------- info The Nexus Registry uses default roles in Temporal Cloud. For self-hosted deployments, you can implement \[custom Authorizers\](/self-hosted-guide/security#authorizer-plugin to restrict access). In Temporal Cloud, Nexus Registry respects RBAC permissions, and restricts functionality based on user role: | Action | Required permissions | | --- | --- | | View or search Endpoints | Read-only role (or higher) at the Account level | | Manage Endpoints | Developer role (or higher) and Namespace Admin on target Namespace | See [Nexus security in Temporal Cloud](https://docs.temporal.io/cloud/nexus/security) . Automate Nexus Endpoint provisioning and lifecycle management[​](https://docs.temporal.io/nexus/registry#automate-nexus-endpoint-provisioning-and-lifecycle-management "Direct link to Automate Nexus Endpoint provisioning and lifecycle management") ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- There are two ways to automate endpoint provisioning and lifecycle management: Terraform and the Operator API. RESOURCES * [Terraform support](https://docs.temporal.io/cloud/terraform-provider#manage-temporal-cloud-nexus-endpoints-with-terraform) for Temporal Cloud. * [Cloud Ops API](https://docs.temporal.io/ops) for Temporal Cloud. * [Operator API](https://github.com/temporalio/api/blob/master/temporal/api/operatorservice/v1/service.proto) for self-hosted deployments. * [View and manage Nexus Endpoints](https://docs.temporal.io/nexus/registry#view-and-manage-nexus-endpoints) * [Search for a Nexus Endpoint](https://docs.temporal.io/nexus/registry#search-for-a-nexus-endpoint) * [Create a Nexus Endpoint](https://docs.temporal.io/nexus/registry#create-a-nexus-endpoint) * [Edit a Nexus Endpoint](https://docs.temporal.io/nexus/registry#edit-a-nexus-endpoint) * [Configure runtime access controls](https://docs.temporal.io/nexus/registry#configure-runtime-access-controls) * [Roles and permissions](https://docs.temporal.io/nexus/registry#roles-and-permissions) * [Automate Nexus Endpoint provisioning and lifecycle management](https://docs.temporal.io/nexus/registry#automate-nexus-endpoint-provisioning-and-lifecycle-management) --- # Worker Versioning | Temporal Platform Documentation [Skip to main content](https://docs.temporal.io/worker-versioning#__docusaurus_skipToContent_fallback) On this page This page defines some of the underlying concepts used in [Worker Versioning](https://docs.temporal.io/production-deployment/worker-deployments/worker-versioning) : * [Worker Deployments](https://docs.temporal.io/worker-versioning#deployments) * [Worker Deployment Versions](https://docs.temporal.io/worker-versioning#deployment-versions) * [Versioning Behaviors](https://docs.temporal.io/worker-versioning#versioning-behaviors) * [Versioning Definitions](https://docs.temporal.io/worker-versioning#versioning-definitions) * [Versioning Statuses](https://docs.temporal.io/worker-versioning#versioning-statuses) * [Continue-as-new, Child Workflow, and Retry Semantics](https://docs.temporal.io/worker-versioning#inheritance-semantics) Worker Deployments[​](https://docs.temporal.io/worker-versioning#deployments "Direct link to Worker Deployments") ------------------------------------------------------------------------------------------------------------------ A Worker Deployment is a logical service that groups similar Workers together for unified management. Each Deployment has a name (such as your service name) and supports versioning through a series of Worker Deployment Versions. Worker Deployment Versions[​](https://docs.temporal.io/worker-versioning#deployment-versions "Direct link to Worker Deployment Versions") ------------------------------------------------------------------------------------------------------------------------------------------ A Worker Deployment Version represents an iteration of a Worker Deployment. Each Deployment Version consists of Workers that share the same code build and environment. When a Worker starts polling for Workflow and Activity Tasks, it reports its Deployment Version to the Temporal Server. Versioning Behaviors[​](https://docs.temporal.io/worker-versioning#versioning-behaviors "Direct link to Versioning Behaviors") ------------------------------------------------------------------------------------------------------------------------------- You can declare each Workflow type to have a **Versioning Behavior**, either Pinned or Auto-Upgrade, in your Workflow configuration using an SDK or the CLI. To learn more about implementing Worker Versioning, see our [Worker Versioning in production](https://docs.temporal.io/production-deployment/worker-deployments/worker-versioning) page. ### Pinned Workflows[​](https://docs.temporal.io/worker-versioning#pinned "Direct link to Pinned Workflows") A **Pinned** Workflow is guaranteed to complete on a single Worker Deployment Version. You can mark a Workflow Type as pinned when you register it by adding an additional Pinned parameter. If you need to move a pinned Workflow to a new version, use [`temporal workflow update-options`](https://docs.temporal.io/cli/workflow#update-options) . ### Auto-Upgrade Workflows[​](https://docs.temporal.io/worker-versioning#auto-upgrade "Direct link to Auto-Upgrade Workflows") An **Auto-Upgrade** Workflow will move to the latest Worker Deployment Version automatically whenever you change the current version. Auto-upgrade Workflows are not restricted to a single Deployment Version and need to be kept replay-safe manually, i.e. with [patching](https://docs.temporal.io/workflow-definition#workflow-versioning) . ### Activity behavior across versions[​](https://docs.temporal.io/worker-versioning#activity-behavior-across-versions "Direct link to Activity behavior across versions") There are a few scenarios to consider for your Activities when you're handling your Worker Deployment versions. * Activities generally start on the Worker Deployment Version of their Workflow which means: * For Pinned Workflows, an Activity starts on the pinned version. * For Auto-Upgrade Workflows, an Activity starts on the Target Worker Deployment Version of the Workflow. In this case, Workflow Execution moves to its Target Version immediately before starting the Activity if the Target Version is different from the last used Version. The Target Worker Deployment Version of a Workflow is the Current or Ramping Version of the Workflow's Task Queue, depending on the Ramp Percentage and Workflow ID. There is an exception where you will have **Independent Activities**. Independent Activities are specific to Worker Versioning. They start on the Current or Ramping Version of their own Task Queue independently from their Workflow. * For a Pinned Workflow, Independent Activities are Activities that start on a Task Queue that's not a member of the calling Workflow's Pinned Worker Deployment Version. * For an Auto-Upgrade Workflow, Independent Activities are Activities that start on a Task Queue that's not a member of the calling Workflow's Target Worker Deployment Version. Since Independent Activities aren't part of a Workflow's version, they can run in a few different ways: * The Activity Task Queue is running in a separate Worker Deployment that only has the Independent Activity. * The Independent Activity is in an unversioned Task Queue. * The Independent Activity is in a separate Worker Deployment that has its own Workflows, but other Workflows reuse the Activity from other Worker Deployments. Versioning Definitions[​](https://docs.temporal.io/worker-versioning#versioning-definitions "Direct link to Versioning Definitions") ------------------------------------------------------------------------------------------------------------------------------------- * **Current Worker Deployment Version**: The version where Workflows are routed to unless they were previously pinned on a different version. Other versions can continue polling to allow pinned Workflows to finish executing or in case you need to roll back. If no current version is specified, the default is unversioned. * **Ramping Worker Deployment Version**: The version where a configurable percentage of Workflows are routed to unless they were previously pinned on a different version. The ramp percentage can be in the range \[0, 100\]. Workflows that don't go to the Ramping Version will go to the Current Version. If no Ramping Version is specified, 100% of new Workflows and Auto-Upgrade Workflows will go to the Current Version. * **Target Worker Deployment Version**: The version your Workflow will upgrade to next. This could be the Deployment's Current Version or the Ramping Version. For example, if an Auto-Upgrade Workflow was running on Version A, the Current Version is B, and there is a 5% ramp to C, there is a 95% chance that its Target Version is B and 5% that it's C. Workflow ID determines whether the workflow falls into the 95% group or the 5% group. Versioning Statuses[​](https://docs.temporal.io/worker-versioning#versioning-statuses "Direct link to Versioning Statuses") ---------------------------------------------------------------------------------------------------------------------------- A Worker Deployment Version moves through the following states: 1. **Inactive**: The version exists because a Worker with that version has polled the server. If this version never becomes Active, it will never be Draining or Drained. 2. **Active**: The version is either Current or Ramping, so it is accepting new Workflows and existing auto-upgrade Workflows. 3. **Draining**: The version has open pinned Workflows running on it, but stopped being Current or Ramping, usually because a newer version has been deployed. It is possible to be Draining and have no open pinned Workflows for a short time, since the drainage status is updated only periodically. 4. **Drained**: The version was draining and now all the pinned Workflows that were running on it are closed. Closed Workflows may still re-run some code paths if they are [Queried](https://docs.temporal.io/sending-messages#sending-queries) within their [Retention Period](https://docs.temporal.io/temporal-service/temporal-server#retention-period) and Workers with that version are still polling. Continue-as-new, Child Workflow, and Retry Semantics[​](https://docs.temporal.io/worker-versioning#inheritance-semantics "Direct link to Continue-as-new, Child Workflow, and Retry Semantics") ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ When Workflows start new runs (e.g. by continuing-as-new or retrying) the new run may inherit their versioning behavior. This section explains how inheritance works across different Workflow execution patterns. ### Ways Workflows Start New Runs[​](https://docs.temporal.io/worker-versioning#ways-workflows-start-new-runs "Direct link to Ways Workflows Start New Runs") A Workflow can start a new run through: * Starting a [Child Workflow](https://docs.temporal.io/child-workflows) * Invoking [Continue-As-New](https://docs.temporal.io/workflow-execution/continue-as-new) * Retrying per its [Retry Policy](https://docs.temporal.io/encyclopedia/retry-policies) * Starting another iteration of a [Cron Job](https://docs.temporal.io/cron-job) (superseded by [Schedules](https://docs.temporal.io/schedule) ) ### Inheritance Rules Overview[​](https://docs.temporal.io/worker-versioning#inheritance-rules-overview "Direct link to Inheritance Rules Overview") Auto-upgrade Workflows never inherit versions. By default, Pinned workflows will pass their version to any Pinned children. This section provides more detail on specific inheritance scenarios. ### Inheritance by Scenario[​](https://docs.temporal.io/worker-versioning#inheritance-by-scenario "Direct link to Inheritance by Scenario") #### Child Workflows[​](https://docs.temporal.io/worker-versioning#child-workflows "Direct link to Child Workflows") **When Parent is Pinned:** * Child inherits the parent's version if the child's Task Queue belongs to that version * Child's first Workflow task executes in the same version as its parent * If child is also Pinned: child remains Pinned to the inherited version for its lifetime * If child is Auto-Upgrade: child's behavior changes to Auto-Upgrade after the first task completes * If child's Task Queue is not in the same Worker Deployment as parent: no inheritance occurs, child starts on Current Version of its task queue **When Parent is Auto-upgrade:** * Child inherits no initial Versioning Behavior * Child starts on the Current Version of its Worker Deployment like all new Workflow executions #### Continue-As-New[​](https://docs.temporal.io/worker-versioning#continue-as-new "Direct link to Continue-As-New") **When Original Workflow is Pinned:** * The Pinned version is inherited across the Continue-As-New chain * If the new run's Task Queue is not in the same Worker Deployment as the original Workflow: no inheritance occurs, new run starts on Current Version of its task queue **When Original Workflow is Auto-upgrade:** * No version inheritance occurs #### Retries[​](https://docs.temporal.io/worker-versioning#retries "Direct link to Retries") **Inheritance Conditions (all must be met):** * The retried run is effectively pinned at the time of retry * The retried run inherited a pinned version when it started (i.e., it is a child of a pinned parent, or a Continue-As-New of a pinned run) * The retried run is running on a Task Queue in the inherited version **When Conditions Not Met:** * No version inheritance occurs #### Cron Jobs[​](https://docs.temporal.io/worker-versioning#cron-jobs "Direct link to Cron Jobs") * **Never inherit** versioning behavior or version ### Versioning Override Inheritance[​](https://docs.temporal.io/worker-versioning#versioning-override-inheritance "Direct link to Versioning Override Inheritance") * Children, crons, retries, and continue-as-new inherit the source run's override **if**: * The override is pinned, **AND** * The new Workflow's Task Queue belongs to the override version * Override inheritance is evaluated separately and takes precedence over inherited base version * [Worker Deployments](https://docs.temporal.io/worker-versioning#deployments) * [Worker Deployment Versions](https://docs.temporal.io/worker-versioning#deployment-versions) * [Versioning Behaviors](https://docs.temporal.io/worker-versioning#versioning-behaviors) * [Pinned Workflows](https://docs.temporal.io/worker-versioning#pinned) * [Auto-Upgrade Workflows](https://docs.temporal.io/worker-versioning#auto-upgrade) * [Activity behavior across versions](https://docs.temporal.io/worker-versioning#activity-behavior-across-versions) * [Versioning Definitions](https://docs.temporal.io/worker-versioning#versioning-definitions) * [Versioning Statuses](https://docs.temporal.io/worker-versioning#versioning-statuses) * [Continue-as-new, Child Workflow, and Retry Semantics](https://docs.temporal.io/worker-versioning#inheritance-semantics) * [Ways Workflows Start New Runs](https://docs.temporal.io/worker-versioning#ways-workflows-start-new-runs) * [Inheritance Rules Overview](https://docs.temporal.io/worker-versioning#inheritance-rules-overview) * [Inheritance by Scenario](https://docs.temporal.io/worker-versioning#inheritance-by-scenario) * [Child Workflows](https://docs.temporal.io/worker-versioning#child-workflows) * [Continue-As-New](https://docs.temporal.io/worker-versioning#continue-as-new) * [Retries](https://docs.temporal.io/worker-versioning#retries) * [Cron Jobs](https://docs.temporal.io/worker-versioning#cron-jobs) * [Versioning Override Inheritance](https://docs.temporal.io/worker-versioning#versioning-override-inheritance) --- # Errors | Temporal Platform Documentation [Skip to main content](https://docs.temporal.io/references/errors#__docusaurus_skipToContent_fallback) On this page This reference lists possible [Workflow Task](https://docs.temporal.io/tasks#workflow-task) errors and how to resolve them. > For other types of errors, see [Temporal Failures](https://docs.temporal.io/kb/failures) > . Each of the below errors corresponds with a [WorkflowTaskFailedCause](https://api-docs.temporal.io/#temporal.api.enums.v1.WorkflowTaskFailedCause) , which appears in [Events](https://docs.temporal.io/workflow-execution/event#event) under `workflow_task_failed_event_attributes`. Bad Cancel Timer Attributes[​](https://docs.temporal.io/references/errors#bad-cancel-timer-attributes "Direct link to Bad Cancel Timer Attributes") ---------------------------------------------------------------------------------------------------------------------------------------------------- This error indicates that the [Workflow Task](https://docs.temporal.io/tasks#workflow-task) failed while attempting to cancel a Timer. Check your Timer attributes for a missing Timer Id value. Add a valid Timer Id and redeploy the code. Bad Cancel Workflow Execution Attributes[​](https://docs.temporal.io/references/errors#bad-cancel-workflow-execution-attributes "Direct link to Bad Cancel Workflow Execution Attributes") ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- The [Workflow Task](https://docs.temporal.io/tasks#workflow-task) failed due to unset [CancelWorkflowExecution](https://docs.temporal.io/references/commands#cancelworkflowexecution) attributes. Reset any missing attributes and redeploy the Workflow Task. Bad Complete Workflow Execution Attributes[​](https://docs.temporal.io/references/errors#bad-complete-workflow-execution-attributes "Direct link to Bad Complete Workflow Execution Attributes") ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- This error indicates that the [Workflow Task](https://docs.temporal.io/tasks#workflow-task) failed due to unset attributes on [CompleteWorkflowExecution](https://docs.temporal.io/references/commands#completeworkflowexecution) . Reset any missing attributes. Adjust the size of your Payload if it exceeds size limits. Bad Continue as New Attributes[​](https://docs.temporal.io/references/errors#bad-continue-as-new-attributes "Direct link to Bad Continue as New Attributes") ------------------------------------------------------------------------------------------------------------------------------------------------------------- This error indicates that the [Workflow Task](https://docs.temporal.io/tasks#workflow-task) failed to validate a [ContinueAsNew](https://docs.temporal.io/references/commands#continueasnewworkflowexecution) attribute. The attribute could be unset or invalid. Reset any missing attributes. If the payload or memo exceeded size limits, adjust the input size. Check that the [Workflow](https://docs.temporal.io/workflows) is validating search attributes after unaliasing keys. Bad Fail Workflow Execution Attributes[​](https://docs.temporal.io/references/errors#bad-fail-workflow-execution-attributes "Direct link to Bad Fail Workflow Execution Attributes") ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- This error indicates that the [Workflow Task](https://docs.temporal.io/tasks#workflow-task) failed due to unset [FailWorkflowExecution](https://docs.temporal.io/references/commands#failworkflowexecution) attributes. If you encounter this error, make sure that `StartToCloseTimeout` or `ScheduleToCloseTimeout` are set. Restart the [Worker](https://docs.temporal.io/workers) that the [Workflow](https://docs.temporal.io/workflows) and [Activity](https://docs.temporal.io/activities) are registered to. Bad Modify Workflow Properties Attributes[​](https://docs.temporal.io/references/errors#bad-modify-workflow-properties-attributes "Direct link to Bad Modify Workflow Properties Attributes") ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- This error indicates that the [Workflow Task](https://docs.temporal.io/tasks#workflow-task) failed to validate attributes on a property in the Upsert Memo or in a payload. These attributes are either unset or exceeding size limits. Reset any unset and empty attributes. Adjust the size of the [Memo](https://docs.temporal.io/workflow-execution#memo) or payload to fit within the system's limits. Bad Record Marker Attributes[​](https://docs.temporal.io/references/errors#bad-record-marker-attributes "Direct link to Bad Record Marker Attributes") ------------------------------------------------------------------------------------------------------------------------------------------------------- This error indicates that the [Workflow Task](https://docs.temporal.io/tasks#workflow-task) failed due to an unset or incorrect [Marker](https://docs.temporal.io/references/events#markerrecorded) name. Enter a valid Marker name and redeploy the Task. Bad Request Cancel Activity Attributes[​](https://docs.temporal.io/references/errors#bad-request-cancel-activity-attributes "Direct link to Bad Request Cancel Activity Attributes") ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- This error either indicates the possibility of unset attributes for [RequestCancelActivity](https://docs.temporal.io/references/commands#requestcancelactivitytask) , or an invalid History Builder state. Update the [Temporal SDK](https://docs.temporal.io/encyclopedia/temporal-sdks) to the most recent release. Reset any unset attributes before retrying the [Workflow Task](https://docs.temporal.io/tasks#workflow-task) . If you continue to see this error, review your code for [nondeterministic causes](https://docs.temporal.io/workflow-definition#non-deterministic-change) . Bad Request Cancel External Workflow Execution Attributes[​](https://docs.temporal.io/references/errors#bad-request-cancel-external-workflow-execution "Direct link to Bad Request Cancel External Workflow Execution Attributes") ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- This error indicates that the [Workflow Task](https://docs.temporal.io/tasks#workflow-task) failed while trying to cancel an external Workflow. Unset or invalid attributes can cause this to occur. Reset any missing attributes, such as Workflow Id or Run Id. Adjust any fields that exceed length limits. If [Child Workflow](https://docs.temporal.io/child-workflows) is set to `Start` and `RequestCancel`, remove one of these attributes. A Child Workflow cannot perform both actions in the same Workflow Task. Bad Schedule Activity Attributes[​](https://docs.temporal.io/references/errors#bad-schedule-activity-attributes "Direct link to Bad Schedule Activity Attributes") ------------------------------------------------------------------------------------------------------------------------------------------------------------------- This error indicates unset or invalid attributes for [`ScheduleActivityTask`](https://docs.temporal.io/references/commands#scheduleactivitytask) or [`CompleteWorkflowExecution`](https://docs.temporal.io/references/commands#completeworkflowexecution) . Reset any unset or empty attributes. Adjust the size of the received payload to stay within the given size limit. Bad Schedule Nexus Operation Attributes[​](https://docs.temporal.io/references/errors#bad-schedule-nexus-operation-attributes "Direct link to Bad Schedule Nexus Operation Attributes") ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- This error indicates unset or invalid attributes for ScheduleNexusOperation, for example if the Nexus Endpoint name used in the caller Workflow doesn't exist. Inspect the reason given in the error for mitigation when possible. Bad Search Attributes[​](https://docs.temporal.io/references/errors#bad-search-attributes "Direct link to Bad Search Attributes") ---------------------------------------------------------------------------------------------------------------------------------- This error indicates that the [Workflow Task](https://docs.temporal.io/tasks#workflow-task) has unset or invalid [Search Attributes](https://docs.temporal.io/search-attribute) . This can cause Workflow Tasks to continue to retry without success. Make sure that all attributes are defined before retrying the Task. Adjust the size of the Payload to fit within the system's size limits. Bad Signal Input Size[​](https://docs.temporal.io/references/errors#bad-signal-input-size "Direct link to Bad Signal Input Size") ---------------------------------------------------------------------------------------------------------------------------------- This error indicates that the Payload has exceeded the [Signal's](https://docs.temporal.io/sending-messages#sending-signals) available input size. Adjust the size of the Payload, and redeploy the [Workflow Task](https://docs.temporal.io/tasks#workflow-task) . Bad Signal Workflow Execution Attributes[​](https://docs.temporal.io/references/errors#bad-signal-workflow-execution-attributes "Direct link to Bad Signal Workflow Execution Attributes") ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- This error indicates that the [Workflow Task](https://docs.temporal.io/tasks#workflow-task) failed to validate attributes for [SignalExternalWorkflowExecution](https://docs.temporal.io/references/commands#signalexternalworkflowexecution) . Reset any unset, missing, nil, or invalid attributes. Adjust the input to fit within the system's size limits. Bad Start Child Execution Attributes[​](https://docs.temporal.io/references/errors#bad-start-child-execution-attributes "Direct link to Bad Start Child Execution Attributes") ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- This error indicates that the [Workflow Task](https://docs.temporal.io/tasks#workflow-task) failed to validate attributes for [`StartChildWorkflowExecution`](https://docs.temporal.io/references/commands#startchildworkflowexecution) Adjust the input size of the attributes to fall within the system's size limits. Make sure that [Search Attribute](https://docs.temporal.io/search-attribute) validation is performed after unaliasing keys. Bad Start Timer Attributes[​](https://docs.temporal.io/references/errors#bad-start-timer-attributes "Direct link to Bad Start Timer Attributes") ------------------------------------------------------------------------------------------------------------------------------------------------- This error indicates that the scheduled [Event](https://docs.temporal.io/workflow-execution/event#event) is missing a Timer Id. Set a valid Timer Id and retry the [Workflow Task](https://docs.temporal.io/tasks#workflow-task) . Cause Bad Binary[​](https://docs.temporal.io/references/errors#cause-bad-binary "Direct link to Cause Bad Binary") ------------------------------------------------------------------------------------------------------------------- This error indicates that the [Worker](https://docs.temporal.io/workers) deployment returned a bad binary checksum. Cause Bad Update[​](https://docs.temporal.io/references/errors#cause-bad-update "Direct link to Cause Bad Update") ------------------------------------------------------------------------------------------------------------------- This error indicates that a [Workflow Execution](https://docs.temporal.io/workflow-execution) tried to complete before receiving an Update. `BadUpdate` can happen when a [Worker](https://docs.temporal.io/workers#worker) generates a [Workflow Task Completed](https://docs.temporal.io/references/events#workflowtaskcompleted) message with missing fields or an invalid Update response format. This error might indicate usage of an unsupported SDK. Make sure you're using a [supported SDK](https://docs.temporal.io/encyclopedia/temporal-sdks) . Cause Reset Workflow[​](https://docs.temporal.io/references/errors#cause-reset-workflow "Direct link to Cause Reset Workflow") ------------------------------------------------------------------------------------------------------------------------------- This error indicates that the [Workflow Task](https://docs.temporal.io/tasks#workflow-task) failed due to a request to reset the [Workflow](https://docs.temporal.io/workflows) . If the system hasn't started a new Workflow, manually reset the Workflow. Cause Unhandled Update[​](https://docs.temporal.io/references/errors#cause-unhandled-update "Direct link to Cause Unhandled Update") ------------------------------------------------------------------------------------------------------------------------------------- `UnhandledUpdate` occurs when a Workflow Update is received by the Temporal Server while a Workflow Task being processed on a Worker produces a Command that would cause the Workflow to transition to a closed state. Temporal rejects the Workflow Task completion to guarantee that the Update is eventually handled by Workflow code and rewinds the Workflow so it can handle the pending Update. This error can happen when the Workflow receives frequent Updates. Cause Unspecified[​](https://docs.temporal.io/references/errors#cause-unspecified "Direct link to Cause Unspecified") ---------------------------------------------------------------------------------------------------------------------- This error indicates that the [Workflow Task](https://docs.temporal.io/tasks#workflow-task) has failed for an unknown reason. If you see this error, examine your Workflow Definition. Failover Close Command[​](https://docs.temporal.io/references/errors#failover-close-command "Direct link to Failover Close Command") ------------------------------------------------------------------------------------------------------------------------------------- This error indicates that a [Namespace](https://docs.temporal.io/namespaces) failover forced the [Workflow Task](https://docs.temporal.io/tasks#workflow-task) to close. The system automatically schedules a retry when this error occurs. Force Close Command[​](https://docs.temporal.io/references/errors#force-close-command "Direct link to Force Close Command") ---------------------------------------------------------------------------------------------------------------------------- This error indicates that the [Workflow Task](https://docs.temporal.io/tasks#workflow-task) was forced to close. A retry will be scheduled if the error is recoverable. gRPC Message Too Large[​](https://docs.temporal.io/references/errors#grpc-message-too-large "Direct link to gRPC Message Too Large") ------------------------------------------------------------------------------------------------------------------------------------- This error occurs when the Workflow Task response exceeds the gRPC message size limit of 4 MB. The Workflow Execution is automatically terminated because this is a non-recoverable error. This typically happens when a Workflow schedules too many Activities, Child Workflows, or commands in a single Workflow Task, or when a Workflow returns a large result. To resolve this error, fix your Workflow code and start a new Workflow Execution. Break work into smaller batches, reduce the size of Workflow returns, use Continue-As-New for long-running Workflows, or compress large payloads with a custom Payload Codec. See the [BlobSizeLimitError troubleshooting guide](https://docs.temporal.io/troubleshooting/blob-size-limit-error) for detailed resolution strategies. Nondeterminism Error[​](https://docs.temporal.io/references/errors#non-deterministic-error "Direct link to Nondeterminism Error") ---------------------------------------------------------------------------------------------------------------------------------- The [Workflow Task](https://docs.temporal.io/tasks#workflow-task) failed due to a [nondeterminism error](https://docs.temporal.io/workflow-definition#non-deterministic-change) . Pending Activities Limit Exceeded[​](https://docs.temporal.io/references/errors#pending-activities-limit-exceeded "Direct link to Pending Activities Limit Exceeded") ---------------------------------------------------------------------------------------------------------------------------------------------------------------------- The [Workflow](https://docs.temporal.io/workflows) has reached capacity for pending [Activities](https://docs.temporal.io/activities) . Therefore, the [Workflow Task](https://docs.temporal.io/tasks#workflow-task) was failed to prevent the creation of another Activity. Let the Workflow complete any current Activities before redeploying the code. Pending Child Workflows Limit Exceeded[​](https://docs.temporal.io/references/errors#pending-child-workflows-limit-exceeded "Direct link to Pending Child Workflows Limit Exceeded") ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- This error indicates that the [Workflow](https://docs.temporal.io/workflows) has reached capacity for pending [Child Workflows](https://docs.temporal.io/child-workflows) . Therefore, the [Workflow Task](https://docs.temporal.io/tasks#workflow-task) was failed to prevent additional Child Workflows from being added. Wait for the system to finish any currently running Child Workflows before redeploying this Task. Pending Nexus Operations Limit Exceeded[​](https://docs.temporal.io/references/errors#pending-nexus-operations-limit-exceeded "Direct link to Pending Nexus Operations Limit Exceeded") ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- The Workflow has reached capacity for pending Nexus Operations. Therefore, the Workflow Task was failed to prevent the creation of another Nexus Operation. Let the Workflow complete any current Nexus Operation before retrying the Task. See [Per Workflow Nexus Operation Limits](https://docs.temporal.io/cloud/limits#per-workflow-nexus-operation-limits) for details. Pending Request Cancel Limit Exceeded[​](https://docs.temporal.io/references/errors#pending-request-cancel-limit-exceeded "Direct link to Pending Request Cancel Limit Exceeded") ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- This error indicates that the [Workflow Task](https://docs.temporal.io/tasks#workflow-task) failed after attempting to add more cancel requests. The [Workflow](https://docs.temporal.io/workflows) has reached capacity for pending requests to cancel other Workflows, and cannot accept more requests. If you see this error, give the system time to process pending requests before retrying the Task. Pending Signals Limit Exceeded[​](https://docs.temporal.io/references/errors#pending-signals-limit-exceeded "Direct link to Pending Signals Limit Exceeded") ------------------------------------------------------------------------------------------------------------------------------------------------------------- The Workflow has reached capacity for pending Signals. Therefore, the [Workflow Task](https://docs.temporal.io/tasks#workflow-task) was failed after attempting to add more [Signals](https://docs.temporal.io/sending-messages#sending-signals) to an external Workflow. Wait for Signals to be processed by the Workflow before retrying the Task. Reset Sticky Task Queue[​](https://docs.temporal.io/references/errors#reset-sticky-task-queue "Direct link to Reset Sticky Task Queue") ---------------------------------------------------------------------------------------------------------------------------------------- This error indicates that the Sticky [Task Queue](https://docs.temporal.io/task-queue) needs to be reset. If you see this error, reset the Sticky Task Queue. The system will retry automatically. Resource Exhausted Cause Concurrent Limit[​](https://docs.temporal.io/references/errors#resource-exhausted-cause-concurrent-limit "Direct link to Resource Exhausted Cause Concurrent Limit") ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- This error indicates that the concurrent [poller count](https://docs.temporal.io/develop/worker-performance#poller-count) has been exhausted. Adjust the poller count per [Worker](https://docs.temporal.io/workers) . Resource Exhausted Cause Persistence Limit[​](https://docs.temporal.io/references/errors#resource-exhausted-cause-persistence-limit "Direct link to Resource Exhausted Cause Persistence Limit") ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- This error indicates that the persistence rate limit has been reached. Resource Exhausted Cause RPS Limit[​](https://docs.temporal.io/references/errors#resource-exhausted-cause-rps-limit "Direct link to Resource Exhausted Cause RPS Limit") ------------------------------------------------------------------------------------------------------------------------------------------------------------------------- This error indicates that the [Workflow](https://docs.temporal.io/workflows) has exhausted its RPS limit. Resource Exhausted Cause System Overload[​](https://docs.temporal.io/references/errors#resource-exhausted-cause-system-overload "Direct link to Resource Exhausted Cause System Overload") ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- This error indicates that the system is overloaded and cannot allocate further resources to [Workflow Tasks](https://docs.temporal.io/tasks#workflow-task) . Resource Exhausted Cause Unspecified[​](https://docs.temporal.io/references/errors#resource-exhausted-cause-unspecified "Direct link to Resource Exhausted Cause Unspecified") ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- This error indicates that an unknown cause is preventing resources from being allocated to further [Workflow Tasks](https://docs.temporal.io/tasks#workflow-task) . Schedule Activity Duplicate Id[​](https://docs.temporal.io/references/errors#schedule-activity-duplicate-id "Direct link to Schedule Activity Duplicate Id") ------------------------------------------------------------------------------------------------------------------------------------------------------------- The [Workflow Task](https://docs.temporal.io/tasks#workflow-task) failed because the [Activity](https://docs.temporal.io/activities) Id is already in use. Check your code to see if you've already specified the same Activity Id in your [Workflow](https://docs.temporal.io/workflows) . Enter another Activity Id, and try running the Workflow Task again. Start Timer Duplicate Id[​](https://docs.temporal.io/references/errors#start-timer-duplicate-id "Direct link to Start Timer Duplicate Id") ------------------------------------------------------------------------------------------------------------------------------------------- This error indicates that a Timer with the given Timer Id has already started. Try entering a different Timer Id, and retry the [Workflow Task](https://docs.temporal.io/tasks#workflow-task) . Unhandled Command[​](https://docs.temporal.io/references/errors#unhandled-command "Direct link to Unhandled Command") ---------------------------------------------------------------------------------------------------------------------- This error indicates new available [Events](https://docs.temporal.io/references/events) since the last [Workflow Task](https://docs.temporal.io/tasks#workflow-task) started. The Workflow Task was failed because the [Workflow](https://docs.temporal.io/workflows) attempted to close itself without handling the new Events. `UnhandledCommand` can happen when the Workflow is receiving a high number of [Signals](https://docs.temporal.io/sending-messages#sending-signals) . If the Workflow doesn't have enough time to handle these Signals, a RetryWorkflow Task is scheduled to handle these new Events. To prevent this error, drain the Signal Channel with the ReceiveAsync function. If you continue to see this error, check your logs for failing Workflow Tasks. The Workflow may have been picked up by a different [Worker](https://docs.temporal.io/workers#worker) . Workflow Worker Unhandled Failure[​](https://docs.temporal.io/references/errors#workflow-worker-unhandled-failure "Direct link to Workflow Worker Unhandled Failure") ---------------------------------------------------------------------------------------------------------------------------------------------------------------------- This error indicates that the [Workflow Task](https://docs.temporal.io/tasks#workflow-task) encountered an unhandled failure from the [Workflow Definition](https://docs.temporal.io/workflow-definition) . * [Bad Cancel Timer Attributes](https://docs.temporal.io/references/errors#bad-cancel-timer-attributes) * [Bad Cancel Workflow Execution Attributes](https://docs.temporal.io/references/errors#bad-cancel-workflow-execution-attributes) * [Bad Complete Workflow Execution Attributes](https://docs.temporal.io/references/errors#bad-complete-workflow-execution-attributes) * [Bad Continue as New Attributes](https://docs.temporal.io/references/errors#bad-continue-as-new-attributes) * [Bad Fail Workflow Execution Attributes](https://docs.temporal.io/references/errors#bad-fail-workflow-execution-attributes) * [Bad Modify Workflow Properties Attributes](https://docs.temporal.io/references/errors#bad-modify-workflow-properties-attributes) * [Bad Record Marker Attributes](https://docs.temporal.io/references/errors#bad-record-marker-attributes) * [Bad Request Cancel Activity Attributes](https://docs.temporal.io/references/errors#bad-request-cancel-activity-attributes) * [Bad Request Cancel External Workflow Execution Attributes](https://docs.temporal.io/references/errors#bad-request-cancel-external-workflow-execution) * [Bad Schedule Activity Attributes](https://docs.temporal.io/references/errors#bad-schedule-activity-attributes) * [Bad Schedule Nexus Operation Attributes](https://docs.temporal.io/references/errors#bad-schedule-nexus-operation-attributes) * [Bad Search Attributes](https://docs.temporal.io/references/errors#bad-search-attributes) * [Bad Signal Input Size](https://docs.temporal.io/references/errors#bad-signal-input-size) * [Bad Signal Workflow Execution Attributes](https://docs.temporal.io/references/errors#bad-signal-workflow-execution-attributes) * [Bad Start Child Execution Attributes](https://docs.temporal.io/references/errors#bad-start-child-execution-attributes) * [Bad Start Timer Attributes](https://docs.temporal.io/references/errors#bad-start-timer-attributes) * [Cause Bad Binary](https://docs.temporal.io/references/errors#cause-bad-binary) * [Cause Bad Update](https://docs.temporal.io/references/errors#cause-bad-update) * [Cause Reset Workflow](https://docs.temporal.io/references/errors#cause-reset-workflow) * [Cause Unhandled Update](https://docs.temporal.io/references/errors#cause-unhandled-update) * [Cause Unspecified](https://docs.temporal.io/references/errors#cause-unspecified) * [Failover Close Command](https://docs.temporal.io/references/errors#failover-close-command) * [Force Close Command](https://docs.temporal.io/references/errors#force-close-command) * [gRPC Message Too Large](https://docs.temporal.io/references/errors#grpc-message-too-large) * [Nondeterminism Error](https://docs.temporal.io/references/errors#non-deterministic-error) * [Pending Activities Limit Exceeded](https://docs.temporal.io/references/errors#pending-activities-limit-exceeded) * [Pending Child Workflows Limit Exceeded](https://docs.temporal.io/references/errors#pending-child-workflows-limit-exceeded) * [Pending Nexus Operations Limit Exceeded](https://docs.temporal.io/references/errors#pending-nexus-operations-limit-exceeded) * [Pending Request Cancel Limit Exceeded](https://docs.temporal.io/references/errors#pending-request-cancel-limit-exceeded) * [Pending Signals Limit Exceeded](https://docs.temporal.io/references/errors#pending-signals-limit-exceeded) * [Reset Sticky Task Queue](https://docs.temporal.io/references/errors#reset-sticky-task-queue) * [Resource Exhausted Cause Concurrent Limit](https://docs.temporal.io/references/errors#resource-exhausted-cause-concurrent-limit) * [Resource Exhausted Cause Persistence Limit](https://docs.temporal.io/references/errors#resource-exhausted-cause-persistence-limit) * [Resource Exhausted Cause RPS Limit](https://docs.temporal.io/references/errors#resource-exhausted-cause-rps-limit) * [Resource Exhausted Cause System Overload](https://docs.temporal.io/references/errors#resource-exhausted-cause-system-overload) * [Resource Exhausted Cause Unspecified](https://docs.temporal.io/references/errors#resource-exhausted-cause-unspecified) * [Schedule Activity Duplicate Id](https://docs.temporal.io/references/errors#schedule-activity-duplicate-id) * [Start Timer Duplicate Id](https://docs.temporal.io/references/errors#start-timer-duplicate-id) * [Unhandled Command](https://docs.temporal.io/references/errors#unhandled-command) * [Workflow Worker Unhandled Failure](https://docs.temporal.io/references/errors#workflow-worker-unhandled-failure) --- # Activities - Ruby SDK | Temporal Platform Documentation [Skip to main content](https://docs.temporal.io/develop/ruby/activities#__docusaurus_skipToContent_fallback) On this page ![Ruby SDK Banner](https://docs.temporal.io/assets/images/banner-ruby-temporal-be833f13b8e3655d7a8d4e50119b7da2.png) Activities[​](https://docs.temporal.io/develop/ruby/activities#activities "Direct link to Activities") ------------------------------------------------------------------------------------------------------- * [Activity basics](https://docs.temporal.io/develop/ruby/activities/basics) * [Activity execution](https://docs.temporal.io/develop/ruby/activities/execution) * [Timeouts](https://docs.temporal.io/develop/ruby/activities/timeouts) * [Asynchronous Activity completion](https://docs.temporal.io/develop/ruby/activities/asynchronous-activity) * [Dynamic Activity](https://docs.temporal.io/develop/ruby/activities/dynamic-activity) * [Benign exceptions](https://docs.temporal.io/develop/ruby/activities/benign-exceptions) * [Activities](https://docs.temporal.io/develop/ruby/activities#activities) --- # Local Activity | Temporal Platform Documentation [Skip to main content](https://docs.temporal.io/local-activity#__docusaurus_skipToContent_fallback) On this page This page discusses [Local Activity](https://docs.temporal.io/local-activity#local-activity) . What is a Local Activity?[​](https://docs.temporal.io/local-activity#local-activity "Direct link to What is a Local Activity?") -------------------------------------------------------------------------------------------------------------------------------- A Local Activity is an [Activity Execution](https://docs.temporal.io/activity-execution) that executes in the same process as the [Workflow Execution](https://docs.temporal.io/workflow-execution) that spawns it. Some Activity Executions are very short-living and do not need the queuing semantic, flow control, rate limiting, and routing capabilities. For this case, Temporal supports the Local Activity feature. The main benefit of Local Activities is that they use less Temporal Service resources (for example, fewer History events) and have much lower latency overhead (because no need to roundtrip to the Temporal Service) compared to normal Activity Executions. However, Local Activities are subject to shorter durations and a lack of rate limiting. Consider using Local Activities for functions that are the following: * can be implemented in the same binary as the Workflow that calls them. * do not require global rate limiting. * do not require routing to a specific Worker or Worker pool. * no longer than a few seconds, inclusive of retries. If it takes longer than 80% of the Workflow Task Timeout (which is 10 seconds by default), the Worker will ask the Temporal Service to create a new Workflow Task to extend the "lease" for processing the Local Activity. The Worker will continue doing so until the Local Activity has completed. This is called Workflow Task Heartbeating. The drawbacks of long-running Local Activities are: * Each new Workflow Task results in 3 more Events in History. * The Workflow won't get notified of new events like Signals and completions until the next Workflow Task Heartbeat. * New Commands created by the Workflow concurrently with the Local Activity will not be sent to the Temporal Service until either the Local Activity completes or the next Workflow Task Heartbeat. Using a Local Activity without understanding its limitations can cause various production issues. **We recommend using regular Activities unless your use case requires very high throughput and large Activity fan outs of very short-lived Activities.** More guidance in choosing between [Local Activity vs Activity](https://community.temporal.io/t/local-activity-vs-activity/290/3) is available in our forums. * [What is a Local Activity?](https://docs.temporal.io/local-activity#local-activity) --- # Temporal Namespace | Temporal Platform Documentation [Skip to main content](https://docs.temporal.io/namespaces#__docusaurus_skipToContent_fallback) On this page Open source and Temporal Cloud This page covers core namespace concepts that apply to both open source Temporal and Temporal Cloud. Temporal Cloud namespaces include additional capabilities, such as [API key](https://docs.temporal.io/cloud/api-keys) and [mTLS authentication](https://docs.temporal.io/cloud/certificates) , [built-in role-based access controls](https://docs.temporal.io/cloud/manage-access/roles-and-permissions#namespace-level-permissions) , [high availability replication](https://docs.temporal.io/cloud/high-availability) , and [namespace tags](https://docs.temporal.io/cloud/namespaces#tag-a-namespace) . Moving from self-hosting to Cloud, or the reverse, requires zero code changes and incurs zero downtime. A Namespace is a unit of isolation within the [Temporal Platform](https://docs.temporal.io/temporal#temporal-platform) . [Task Queues](https://docs.temporal.io/task-queue) and [Workflow Executions](https://docs.temporal.io/workflow-execution) belong to a Namespace. When a Workflow Execution is spawned, it does so within a specific Namespace. Usage[​](https://docs.temporal.io/namespaces#usage "Direct link to Usage") --------------------------------------------------------------------------- * **Workflow ID uniqueness**: Temporal guarantees a unique Workflow Id within a Namespace. Workflow Executions may have the same Workflow Id if they are in different Namespaces. * **Resource isolation**: Heavy traffic from one Namespace will not impact other Namespaces running on the same Temporal Service. * **Configuration boundaries**: Options like the [Retention Period](https://docs.temporal.io/temporal-service/temporal-server#retention-period) and [Archival](https://docs.temporal.io/temporal-service/archival) destination are configured per Namespace. * **Default Namespace**: If no Namespace is specified, the Temporal Service uses the Namespace "default" for all Temporal SDKs and the Temporal CLI. You must create a Namespace before using it in your Client. * **Multi-tenancy**: A single Namespace is still multi-tenant. Multiple applications or teams can share a Namespace, but must coordinate on Workflow ID and Task Queue naming to avoid conflicts. Namespace operations[​](https://docs.temporal.io/namespaces#namespace-operations "Direct link to Namespace operations") ------------------------------------------------------------------------------------------------------------------------ For how to create and manage Namespaces: * **Open source Temporal**: [Managing Namespaces](https://docs.temporal.io/self-hosted-guide/namespaces) * **Temporal Cloud**: [Temporal Cloud Namespaces](https://docs.temporal.io/cloud/namespaces) * [Usage](https://docs.temporal.io/namespaces#usage) * [Namespace operations](https://docs.temporal.io/namespaces#namespace-operations) --- # Observability - Temporal feature | Temporal Platform Documentation [Skip to main content](https://docs.temporal.io/evaluate/development-production-features/observability#__docusaurus_skipToContent_fallback) Temporal's observability feature helps you track the state of your Workflows in real-time, providing tools for detailed metrics, tracing, comprehensive logging, and visibility into your application state. Monitor performance, trace Activity and Workflow Executions, debug, and filter Workflow Executions to gain deeper insights into your Workflows. **Key Components of Temporal's Observability and Visibility** * **Metrics**: Detailed performance metrics to track the health and efficiency of your Temporal Service and Workflows. * **Tracing**: End-to-end tracing of Workflow and Activity Executions to understand the flow and timing of operations. * **Logging**: Comprehensive logging capabilities for debugging and auditing purposes. * **Search Attributes**: Custom attributes that can be used to enhance searchability and provide additional context to Workflow Executions. * **Web UI**: A user-friendly interface for visualizing and interacting with your Workflows and Temporal Service state. **Benefits of Temporal's Observability and Visibility Features** * **Real-time Monitoring**: Track the state and progress of your Workflows as they execute. * **Performance Optimization**: Identify bottlenecks and optimize your Workflow and Activity implementations. * **Effective Debugging**: Quickly locate and diagnose issues in your Temporal applications. * **Compliance and Auditing**: Maintain detailed records of all Workflow executions for compliance and auditing purposes. * **Operational Insights**: Gain a deep understanding of your application's behavior and usage patterns. * **Scalability Management**: Monitor and manage the scalability of your Temporal Service effectively. Jump straight into the Temporal SDK feature guide. Related 📚 * [![](https://docs.temporal.io/img/sdks/svgs/golang.svg)Observability using the Go SDK](https://docs.temporal.io/develop/go/platform/observability) feature-guide * [![](https://docs.temporal.io/img/sdks/svgs/dotnet.svg)Observability using the .NET SDK](https://docs.temporal.io/develop/dotnet/platform/observability) feature-guide * [![](https://docs.temporal.io/img/sdks/svgs/java.svg)Observability using the Java SDK](https://docs.temporal.io/develop/java/platform/observability) feature-guide * [![](https://docs.temporal.io/img/sdks/svgs/php.svg)Observability using the PHP SDK](https://docs.temporal.io/develop/php/platform/observability) feature-guide * [![](https://docs.temporal.io/img/sdks/svgs/python.svg)Observability using the Python SDK](https://docs.temporal.io/develop/python/platform/observability) feature-guide * [![](https://docs.temporal.io/img/sdks/svgs/typescript.svg)Observability using the TypeScript SDK](https://docs.temporal.io/develop/typescript/platform/observability) feature-guide * [![](https://docs.temporal.io/img/sdks/svgs/ruby.svg)Observability using the Ruby SDK](https://docs.temporal.io/develop/ruby/platform/observability) feature-guide --- # AI SDK by Vercel integration | Temporal Platform Documentation [Skip to main content](https://docs.temporal.io/develop/typescript/integrations/ai-sdk#__docusaurus_skipToContent_fallback) On this page Temporal's integration with [Vercel's AI SDK](https://ai-sdk.dev/) lets you use the AI SDK's API directly in Workflow code while Temporal handles Durable Execution. Like all API calls, LLM API calls are non-deterministic. In a [Temporal Application](https://docs.temporal.io/glossary#temporal-application) , that means you cannot make LLM calls directly from a [Workflow](https://docs.temporal.io/glossary#workflow) ; they must run as [Activities](https://docs.temporal.io/glossary#activity) . The AI SDK plugin handles this automatically: when you call methods in the AI SDK such as `generateText()`, the plugin wraps those calls in Activities behind the scenes. This preserves the Vercel AI SDK's developer experience that you are already familiar with while Temporal handles Durable Execution for you. All code snippets in this guide are taken from the TypeScript SDK [ai-sdk samples](https://github.com/temporalio/samples-typescript/tree/main/ai-sdk) . Refer to the samples for the complete code and run them locally. info The Vercel AI SDK Integration is in Public Preview. Refer to the [Temporal product release stages guide](https://docs.temporal.io/evaluate/development-production-features/release-stages) for more information. Prerequisites[​](https://docs.temporal.io/develop/typescript/integrations/ai-sdk#prerequisites "Direct link to Prerequisites") ------------------------------------------------------------------------------------------------------------------------------- * This guide assumes you are already familiar with the Vercel AI SDK. If you aren't, refer to the [Vercel AI SDK documentation](https://ai-sdk.dev/) for more details. * If you are new to Temporal, we also recommend you read the [Understanding Temporal](https://docs.temporal.io/evaluate/understanding-temporal) document or take the [Temporal 101](https://learn.temporal.io/courses/temporal_101/) course to understand the basics of Temporal. * Ensure you have set up your local development environment by following the [Set up your local with the TypeScript SDK](https://docs.temporal.io/develop/typescript/set-up-your-local-typescript) guide. When you are done, leave the Temporal Development Server running if you want to test your code locally. Configure Workers to use the AI SDK[​](https://docs.temporal.io/develop/typescript/integrations/ai-sdk#configure-workers-to-use-the-ai-sdk "Direct link to Configure Workers to use the AI SDK") ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Workers are the compute layer of a Temporal Application. They are responsible for executing the code that defines your [Workflows](https://docs.temporal.io/glossary#workflow) and [Activities](https://docs.temporal.io/glossary#activity) . Before you can execute a Workflow or Activity with the Vercel AI SDK, you need to create a Worker and configure it to use the AI SDK plugin. Follow the steps below to configure your Worker. 1. Install the `@temporalio/ai-sdk` package. npm install @temporalio/ai-sdk 2. Create a `worker.ts` file and configure the Worker to use the AI SDK plugin. import { openai } from '@ai-sdk/openai';import { AiSdkPlugin } from '@temporalio/ai-sdk';//... other import statements, initializing a connection// to the Temporal Service to be used by the Workerconst worker = await Worker.create({ plugins: [ new AiSdkPlugin({ modelProvider: openai, }), ], connection, namespace: 'default', taskQueue: 'ai-sdk', workflowsPath: require.resolve('./workflows'), activities,});// ... code that runs the worker The `modelProvider` specifies which AI provider to use when creating models. Choose the provider that best suits your needs. In the Worker options, you are also specifying that the Worker polls the `ai-sdk` Task Queue for work in the `default` Namespace. Make sure that you configure your Client application to use the same Task Queue and Namespace. 3. Run the Worker. This Worker will now poll the Temporal Service for work on the `ai-sdk` Task Queue in the `default` Namespace until you stop it. nodemon worker.ts You must ensure the Worker process has access to your API credentials. Most provider SDKs read credentials from environment variables. Refer to the [Vercel AI SDK documentation](https://ai-sdk.dev/providers/ai-sdk-providers) for instructions on how to set up your environment variables for the provider you chose. tip You only need to give provider credentials to the Worker process. The client application, meaning the application that sends requests to the Temporal Service to start Workflow Executions, doesn't need to know about the credentials. See the full example at [ai-sdk samples](https://github.com/temporalio/samples-typescript/tree/main/ai-sdk) . Develop a Simple Haiku Agent[​](https://docs.temporal.io/develop/typescript/integrations/ai-sdk#develop-a-simple-haiku-agent "Direct link to Develop a Simple Haiku Agent") ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- To help you get started, you can develop a simple Haiku Agent that generates haikus based on a prompt. If you weren't using Temporal, you would write code like this to generate a haiku: import { generateText } from 'ai';import { openai } from '@ai-sdk/openai';async function haikuAgent(prompt: string): Promise { const result = await generateText({ model: openai('gpt-4o-mini'), prompt, system: 'You only respond in haikus.', }); return result.text;} To add Durable Execution to your agent, implement the agent as a Temporal Workflow. Use the AI SDK as you normally would, but pass `temporalProvider.languageModel()` as the model. The string you provide (like `'gpt-4o-mini'`) is passed to your configured `modelProvider` to create the model. import { generateText } from 'ai';import { temporalProvider } from '@temporalio/ai-sdk';export async function haikuAgent(prompt: string): Promise { const result = await generateText({ model: temporalProvider.languageModel('gpt-4o-mini'), prompt, system: 'You only respond in haikus.', }); return result.text;} With only two line changes, you have added Durable Execution to your agent. Your agent now gets automatic retries, timeouts, and the ability to run for extended periods without losing state if the process crashes. Provide your durable agent with tools[​](https://docs.temporal.io/develop/typescript/integrations/ai-sdk#provide-your-durable-agent-with-tools "Direct link to Provide your durable agent with tools") ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- The Vercel AI SDK lets you provide tools to your agents, and when the model calls them, they execute in the Workflow. Since tool functions run in Workflow context, they must follow Workflow rules. That means they must call Activities or Child Workflows to perform non-deterministic operations like API calls. For example, if you want to call an external API to get the weather, you would implement it as an Activity and call it from the tool function. The following is an example of an Activity that gets the weather for a given location: [ai-sdk/src/activities.ts](https://github.com/temporalio/samples-typescript/blob/main/ai-sdk/src/activities.ts) export async function getWeather(input: { location: string;}): Promise<{ city: string; temperatureRange: string; conditions: string }> { console.log('Activity execution'); return { city: input.location, temperatureRange: '14-20C', conditions: 'Sunny with wind.', };} Then in your agent implementation, provide the tool to the model using the `tools` option and instruct the model to use the tool when needed. import { proxyActivities } from '@temporalio/workflow';import { generateText, tool } from 'ai';import { temporalProvider } from '@temporalio/ai-sdk';import { z } from 'zod';const { getWeather } = proxyActivities({ startToCloseTimeout: '1 minute',});export async function toolsAgent(question: string): Promise { const result = await generateText({ model: temporalProvider.languageModel('gpt-4o-mini'), prompt: question, system: 'You are a helpful agent.', tools: { getWeather: tool({ description: 'Get the weather for a given city', inputSchema: z.object({ location: z.string().describe('The location to get the weather for'), }), execute: getWeather, }), }, stopWhen: stepCountIs(5), }); return result.text;} Integrate with Model Context Protocol (MCP) servers[​](https://docs.temporal.io/develop/typescript/integrations/ai-sdk#integrate-with-model-context-protocol-mcp-servers "Direct link to Integrate with Model Context Protocol (MCP) servers") ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- [Model Context Protocol (MCP)](https://modelcontextprotocol.io/) is an open standard that lets AI applications connect to external tools and data sources. Calls to MCP servers, being calls to external APIs, are non-deterministic and would usually need to be implemented as Activities. The Temporal AI SDK integration handles this for you and provides a built-in implementation of a stateless MCP client that you can use inside Workflows. Follow the steps below to integrate your agent with an MCP server. 1. Create a connection to the MCP servers using the `experimental_createMCPClient` function from the `@ai-sdk/mcp` package. You can register multiple MCP servers by providing multiple factory functions in `mcpClientFactories`. import { experimental_createMCPClient as createMCPClient } from '@ai-sdk/mcp';import { StdioClientTransport } from '@modelcontextprotocol/sdk/client/stdio.js';const mcpClientFactories = { testServer: () => createMCPClient({ transport: new StdioClientTransport({ command: 'node', args: ['lib/mcp-server.js'], }), }),}; The example uses `StdioClientTransport` as the transport mechanisms for client-server communication. Each time the Worker processes a Task that requires communication with the MCP server, it will start the server process and connect to it as required by the Task. 2. Configure the Worker to use the MCP client factories. const worker = await Worker.create({ plugins: [ new AiSdkPlugin({ modelProvider: openai, mcpClientFactories }), ]}, ...); 3. In your agent Workflow, use `TemporalMCPClient` to get tools from the MCP server by referencing it by name: import { TemporalMCPClient, temporalProvider } from '@temporalio/ai-sdk';export async function mcpAgent(prompt: string): Promise { const mcpClient = new TemporalMCPClient({ name: 'testServer' }); const tools = await mcpClient.tools(); const result = await generateText({ model: temporalProvider.languageModel('gpt-4o-mini'), prompt, tools, system: 'You are a helpful agent, You always use your tools when needed.', stopWhen: stepCountIs(5), }); return result.text;} Both listing tools and calling them run as Activities behind the scenes, giving you automatic retries, timeouts, and full observability. * [Prerequisites](https://docs.temporal.io/develop/typescript/integrations/ai-sdk#prerequisites) * [Configure Workers to use the AI SDK](https://docs.temporal.io/develop/typescript/integrations/ai-sdk#configure-workers-to-use-the-ai-sdk) * [Develop a Simple Haiku Agent](https://docs.temporal.io/develop/typescript/integrations/ai-sdk#develop-a-simple-haiku-agent) * [Provide your durable agent with tools](https://docs.temporal.io/develop/typescript/integrations/ai-sdk#provide-your-durable-agent-with-tools) * [Integrate with Model Context Protocol (MCP) servers](https://docs.temporal.io/develop/typescript/integrations/ai-sdk#integrate-with-model-context-protocol-mcp-servers) --- # Error handling - Ruby SDK | Temporal Platform Documentation [Skip to main content](https://docs.temporal.io/develop/ruby/best-practices/error-handling#__docusaurus_skipToContent_fallback) On this page Raise and Handle Exceptions[​](https://docs.temporal.io/develop/ruby/best-practices/error-handling#exception-handling "Direct link to Raise and Handle Exceptions") -------------------------------------------------------------------------------------------------------------------------------------------------------------------- In each Temporal SDK, error handling is implemented idiomatically, following the conventions of the language. Temporal uses several different error classes internally — for example, [`CancelledError`](https://ruby.temporal.io/Temporalio/Error/CanceledError.html) in the Ruby SDK, to handle a Workflow cancellation. You should not raise or otherwise implement these manually, as they are tied to Temporal platform logic. The one Temporal error class that you will typically raise deliberately is [`ApplicationError`](https://ruby.temporal.io/Temporalio/Error/ApplicationError.html) . In fact, _any_ other exceptions that are raised from your Ruby code in a Temporal Activity will be converted to an `ApplicationError` internally. This way, an error's type, severity, and any additional details can be sent to the Temporal Service, indexed by the Web UI, and even serialized across language boundaries. In other words, these two code samples do the same thing: class MyError < StandardErrorendclass SomethingThatFails < Temporalio::Activity::Definition def execute(details) Temporalio::Activity::Context.current.logger.info( "We have a problem." ) raise MyError.new('Simulated failure') endend class SomethingThatFails < Temporalio::Activity::Definition def execute(details) Temporalio::Activity::Context.current.logger.info( "We have a problem." ) raise Temporalio::Error::ApplicationError.new('Simulated failure', type: 'MyError') endend Depending on your implementation, you may decide to use either method. One reason to use the Temporal `ApplicationError` class is because it allows you to set an additional `non_retryable` parameter. This way, you can decide whether an error should not be retried automatically by Temporal. This can be useful for deliberately failing a Workflow due to bad input data, rather than waiting for a timeout to elapse: class SomethingThatFails < Temporalio::Activity::Definition def execute(details) Temporalio::Activity::Context.current.logger.info( "We have a problem." ) raise Temporalio::Error::ApplicationError.new('Simulated failure', non_retryable: true) endend You can alternately specify a list of errors that are non-retryable in your Activity [Retry Policy](https://docs.temporal.io/develop/ruby/activities/timeouts#activity-retries) . Failing Workflows[​](https://docs.temporal.io/develop/ruby/best-practices/error-handling#workflow-failure "Direct link to Failing Workflows") ---------------------------------------------------------------------------------------------------------------------------------------------- One of the core design principles of Temporal is that an Activity Failure will never directly cause a Workflow Failure — a Workflow should never return as Failed unless deliberately. The default retry policy associated with Temporal Activities is to retry them until reaching a certain timeout threshold. Activities will not actually _return_ a failure to your Workflow until this condition or another non-retryable condition is met. At this point, you can decide how to handle an error returned by your Activity the way you would in any other program. For example, you could implement a [Saga Pattern](https://github.com/temporalio/samples-ruby/tree/main/saga) that uses `rescue` blocks to "unwind" some of the steps your Workflow has performed up to the point of Activity Failure. **You will only fail a Workflow by manually raising an `ApplicationError` from the Workflow code.** You could do this in response to an Activity Failure, if the failure of that Activity means that your Workflow should not continue: class SagaWorkflow < Temporalio::Workflow::Definition def execute(details) Temporalio::Workflow.execute_activity(Activities::SomethingThatFails, details,start_to_close_timeout: 30) rescue StandardError raise Temporalio::Error::ApplicationError.new('Fail the Workflow') This works differently in a Workflow than raising exceptions from Activities. In an Activity, any Ruby exceptions or custom exceptions are converted to a Temporal `ApplicationError`. In a Workflow, any exceptions that are raised other than an explicit Temporal `ApplicationError` will only fail that particular [Workflow Task](https://docs.temporal.io/tasks#workflow-task-execution) and be retried. This includes any typical Ruby `RuntimeError`s that are raised automatically. These errors are treated as bugs that can be corrected with a fixed deployment, rather than a reason for a Temporal Workflow Execution to return unexpectedly. * [Raise and Handle Exceptions](https://docs.temporal.io/develop/ruby/best-practices/error-handling#exception-handling) * [Failing Workflows](https://docs.temporal.io/develop/ruby/best-practices/error-handling#workflow-failure) --- # Workflows - TypeScript SDK | Temporal Platform Documentation [Skip to main content](https://docs.temporal.io/develop/typescript/activities#__docusaurus_skipToContent_fallback) On this page ![TypeScript SDK Banner](https://docs.temporal.io/assets/images/banner-typescript-temporal-d8a24070726a0d14cb4d1aab011db927.png) Activities[​](https://docs.temporal.io/develop/typescript/activities#activities "Direct link to Activities") ------------------------------------------------------------------------------------------------------------- * [Activity basics](https://docs.temporal.io/develop/typescript/activities/basics) * [Activity execution](https://docs.temporal.io/develop/typescript/activities/execution) * [Timeouts](https://docs.temporal.io/develop/typescript/activities/timeouts) * [Asynchronous Activity](https://docs.temporal.io/develop/typescript/activities/asynchronous-activity) * [Benign exceptions](https://docs.temporal.io/develop/typescript/activities/benign-exceptions) * [Activities](https://docs.temporal.io/develop/typescript/activities#activities) --- # Persistence | Temporal Platform Documentation [Skip to main content](https://docs.temporal.io/temporal-service/persistence#__docusaurus_skipToContent_fallback) On this page This page discusses the following: * [Persistence](https://docs.temporal.io/temporal-service/persistence#persistence) * [Dependency Versions](https://docs.temporal.io/temporal-service/persistence#dependency-versions) What is Persistence?[​](https://docs.temporal.io/temporal-service/persistence#persistence "Direct link to What is Persistence?") --------------------------------------------------------------------------------------------------------------------------------- The Temporal Persistence store is a database used by the [Temporal Server](https://docs.temporal.io/temporal-service/temporal-server) to persist events generated and processed in your Temporal Service and SDK. A Temporal Service's only required dependency for basic operation is the Persistence database. Multiple types of databases are supported. ![Persistence](https://docs.temporal.io/diagrams/temporal-database.svg) Persistence The database stores the following types of data: * Tasks: Tasks to be dispatched. * State of Workflow Executions: * Execution table: A capture of the mutable state of Workflow Executions. * History table: An append-only log of Workflow Execution History Events. * Namespace metadata: Metadata of each Namespace in the Temporal Service. * [Visibility](https://docs.temporal.io/temporal-service/visibility) data: Enables operations like "show all running Workflow Executions". For production environments, we recommend using Elasticsearch as your Visibility store. An Elasticsearch database must be configured in a self-hosted Temporal Service to enable [advanced Visibility](https://docs.temporal.io/visibility#advanced-visibility) on Temporal Server versions 1.19.1 and earlier. With Temporal Server version 1.20 and later, advanced Visibility features are available on SQL databases like MySQL (version 8.0.17 and later), PostgreSQL (version 12 and later), SQLite (v3.31.0 and later), and Elasticsearch. ### Dependency versions[​](https://docs.temporal.io/temporal-service/persistence#dependency-versions "Direct link to Dependency versions") Temporal tests compatibility by spanning the minimum and maximum stable major versions for each supported database. The following versions are used in our test pipelines and actively tested before we release any version of Temporal: * **Cassandra v3.11 and v4.0** * **PostgreSQL 13.18, 14.15, 15.10 and 16.6** * **MySQL v5.7 and v8.0** (specifically 8.0.19+ due to a bug) You can verify supported databases in the [Temporal Server release notes](https://github.com/temporalio/temporal/releases) . * Because Temporal Server primarily relies on core database functionality, we do not expect compatibility to break often. * We do not run tests with vendors like Vitess and CockroachDB. * Temporal also supports SQLite v3.x persistence, but this is meant only for development and testing, not production usage. * [What is Persistence?](https://docs.temporal.io/temporal-service/persistence#persistence) * [Dependency versions](https://docs.temporal.io/temporal-service/persistence#dependency-versions) --- # Payload Codec | Temporal Platform Documentation [Skip to main content](https://docs.temporal.io/payload-codec#__docusaurus_skipToContent_fallback) On this page This page discusses [Payload Codec](https://docs.temporal.io/payload-codec#payload-codec) . What is a Payload Codec?[​](https://docs.temporal.io/payload-codec#payload-codec "Direct link to What is a Payload Codec?") ---------------------------------------------------------------------------------------------------------------------------- A Payload Codec transforms an array of [Payloads](https://docs.temporal.io/dataconversion#payload) (for example, a list of Workflow arguments) into another array of Payloads. When serializing to Payloads, the Payload Converter is applied first to convert your objects to bytes, followed by codecs that convert bytes to bytes. When deserializing from Payloads, codecs are applied first to last to reverse the effect, followed by the Payload Converter. Use a custom Payload Codec to transform your Payloads; for example, implementing compression and/or encryption on your Workflow Execution data. ### Encryption[​](https://docs.temporal.io/payload-codec#encryption "Direct link to Encryption") Using end-to-end encryption in your custom Data Converter ensures that sensitive application data is secure when handled by the Temporal Server. Apply your encryption logic in a custom Payload Codec and use it locally to encrypt data. You maintain all the encryption keys, and the Temporal Server sees only encrypted data. Refer to [What is Key Management?](https://docs.temporal.io/key-management) for more guidance. Your data exists unencrypted only on the Client and the Worker process that is executing the Workflows and Activities, on hosts that you control. For details, see [Securing your data](https://docs.temporal.io/production-deployment/data-encryption) . The following samples use encryption (AES GCM with 256-bit key) in a custom Data Converter: * [Go sample](https://github.com/temporalio/samples-go/tree/main/encryption) * [Java sample](https://github.com/temporalio/samples-java/tree/main/core/src/main/java/io/temporal/samples/encryptedpayloads) * [Python sample](https://github.com/temporalio/samples-python/tree/main/encryption) * [TypeScript sample](https://github.com/temporalio/samples-typescript/tree/main/encryption) * [What is a Payload Codec?](https://docs.temporal.io/payload-codec#payload-codec) * [Encryption](https://docs.temporal.io/payload-codec#encryption) --- # Workflow Id and Run Id | Temporal Platform Documentation [Skip to main content](https://docs.temporal.io/workflow-execution/workflowid-runid#__docusaurus_skipToContent_fallback) On this page This page discusses the following: * [Run Id](https://docs.temporal.io/workflow-execution/workflowid-runid#run-id) * [Operations leading to non-determinism](https://docs.temporal.io/workflow-execution/workflowid-runid#run-id-non-determinism) * [Workflow Id](https://docs.temporal.io/workflow-execution/workflowid-runid#workflow-id) * [Workflow Id Reuse Policy](https://docs.temporal.io/workflow-execution/workflowid-runid#workflow-id-reuse-policy) * [Workflow Id Conflict Policy](https://docs.temporal.io/workflow-execution/workflowid-runid#workflow-id-conflict-policy) Each Workflow Execution is associated with a user-defined [Workflow ID](https://docs.temporal.io/workflow-execution/workflowid-runid#workflow-id) , a value which typically carries some business meaning (such as an order number or customer number). Temporal guarantees that there can be at most one Workflow Execution with a given ID running at any point in time, a constraint that helps to protect against unexpected duplication. In some cases, such as when running the same Workflow at recurring intervals using the Schedules features, there can be multiple "runs" of a single Workflow Execution over a period of time. In this case, all runs will have the same Workflow ID. However, each run will have a unique system-generated [Run ID](https://docs.temporal.io/workflow-execution/workflowid-runid#run-id) . What is a Run Id?[​](https://docs.temporal.io/workflow-execution/workflowid-runid#run-id "Direct link to What is a Run Id?") ----------------------------------------------------------------------------------------------------------------------------- A Run Id is a globally unique, platform-level identifier for a [Workflow Execution](https://docs.temporal.io/workflow-execution) . The current Run Id is mutable and can change during a [Workflow Retry](https://docs.temporal.io/encyclopedia/retry-policies) . You shouldn't rely on storing the current Run Id, or using it for any logical choices, because a Workflow Retry changes the Run Id and can lead to non-determinism issues. Temporal guarantees that only one Workflow Execution with a given [Workflow Id](https://docs.temporal.io/workflow-execution/workflowid-runid#workflow-id) can be in an Open state at any given time. But when a Workflow Execution reaches a Closed state, it is possible to have another Workflow Execution in an Open state with the same Workflow Id. For example, a Temporal Cron Job is a chain of Workflow Executions that all have the same Workflow Id. Each Workflow Execution within the chain is considered a _Run_. A Run Id uniquely identifies a Workflow Execution even if it shares a Workflow Id with other Workflow Executions. ### Which operations lead to non-determinism issues?[​](https://docs.temporal.io/workflow-execution/workflowid-runid#run-id-non-determinism "Direct link to Which operations lead to non-determinism issues?") An operation like `ContinueAsNew`, `Retry`, `Cron`, and `Reset` creates a [Workflow Execution Chain](https://docs.temporal.io/workflow-execution#workflow-execution-chain) as identified by the [`first_execution_run_id`](https://github.com/temporalio/api/blob/master/temporal/api/history/v1/message.proto) . Each operation creates a new Workflow Execution inside a chain run and saves its information as `first_execution_run_id`. Thus, the Run Id is updated during each operation on a Workflow Execution. * The `first_execution_run_id` is the Run Id of the first Workflow Execution in a Chain run. * The `original_execution_run_id` is the Run Id when the `WorkflowExecutionStarted` Event occurs. A Workflow `Reset` changes the first execution Run Id, but preserves the original execution Run Id. For example, when a new Workflow Execution in the chain starts, it stores its Run Id in `original_execution_run_id`. A reset doesn't change that field, but the current Run Id is updated. caution Because of this behavior, you shouldn't rely on the current Run Id in your code to make logical choices. **Learn more** For more information, see the following link. * [`message.proto`](https://github.com/temporalio/api/blob/master/temporal/api/history/v1/message.proto#L75-L82) What is a Workflow Id?[​](https://docs.temporal.io/workflow-execution/workflowid-runid#workflow-id "Direct link to What is a Workflow Id?") -------------------------------------------------------------------------------------------------------------------------------------------- A Workflow Id is a customizable, application-level identifier for a [Workflow Execution](https://docs.temporal.io/workflow-execution) that is unique to an Open Workflow Execution within a [Namespace](https://docs.temporal.io/namespaces) . * [How to set a Workflow Id](https://docs.temporal.io/develop/go/client/temporal-client#workflow-id) A Workflow Id is meant to be a business-process identifier, such as customer identifier or order identifier. Do not use sensitive data or PII in Workflow Ids Do not include sensitive data, secrets, or personally identifiable information (PII) as a Workflow Id. Workflow Ids are stored in plain text, are **not** processed by a custom [Payload Codec](https://docs.temporal.io/payload-codec#payload-codec) , and are visible in the Temporal Web UI, CLI output, Event History, and system logs. The same applies to other user-defined identifiers such as Workflow Type names, Task Queue names, Activity names, and Signal/Query/Update names. Using sensitive data in these identifiers risks exposure to anyone with Namespace access and may violate data protection regulations such as GDPR, HIPAA, or SOC 2. The Temporal Platform guarantees uniqueness of the Workflow Id within a [Namespace](https://docs.temporal.io/namespaces) based on the Workflow Id Reuse Policy. A [Workflow Id Reuse Policy](https://docs.temporal.io/workflow-execution/workflowid-runid#workflow-id-reuse-policy) can be used to manage whether a Workflow Id from a Closed Workflow can be re-used. A [Workflow Id Conflict Policy](https://docs.temporal.io/workflow-execution/workflowid-runid#workflow-id-conflict-policy) can be used to decide how to resolve a Workflow Id conflict with a Running Workflow. A Workflow Execution can be uniquely identified across all Namespaces by its [Namespace](https://docs.temporal.io/namespaces) , Workflow Id, and [Run Id](https://docs.temporal.io/workflow-execution/workflowid-runid#run-id) . ### What is a Workflow Id Reuse Policy?[​](https://docs.temporal.io/workflow-execution/workflowid-runid#workflow-id-reuse-policy "Direct link to What is a Workflow Id Reuse Policy?") A Workflow Id Reuse Policy determines whether a Workflow Execution is allowed to spawn with a particular Workflow Id, if that Workflow Id has been used with a previous, and now Closed, Workflow Execution. It is not possible for a new Workflow Execution to spawn with the same Workflow Id as another Open Workflow Execution, regardless of the Workflow Id Reuse Policy. See [Workflow Id Conflict Policy](https://docs.temporal.io/workflow-execution/workflowid-runid#workflow-id-conflict-policy) for resolving a Workflow Id conflict. The Workflow Id Reuse Policy can have one of the following values: * **Allow Duplicate:** The Workflow Execution is allowed to exist regardless of the Closed status of a previous Workflow Execution with the same Workflow Id. **This is the default policy, if one is not specified.** Use this when it is OK to have a Workflow Execution with the same Workflow Id as a previous, but now Closed, Workflow Execution. * **Allow Duplicate Failed Only:** The Workflow Execution is allowed to exist only if a previous Workflow Execution with the same Workflow Id does not have a Completed status. Use this policy when there is a need to re-execute a Failed, Timed Out, Terminated, or Cancelled Workflow Execution and guarantee that the Completed Workflow Execution will not be re-executed. * **Reject Duplicate:** The Workflow Execution cannot exist if a previous Workflow Execution has the same Workflow Id, regardless of the Closed status. Use this when there can only be one Workflow Execution per Workflow Id within a Namespace for the given retention period. The first three values (Allow Duplicate, Allow Duplicate Failed Only, and Reject Duplicate) of the Workflow Id Reuse Policy apply to Closed Workflow Executions that are retained within the Namespace. For example, given a default Retention Period, the Temporal Service can only check the Workflow Id of the spawning Workflow Execution based on the Workflow Id Reuse Policy against the Closed Workflow Executions for the last _30 days_. If you need to start a Workflow for a particular implementation only if it hasn't started yet, ensure that your Retention Period is long enough to check against. If this becomes unwieldy, consider using [Workflow message passing](https://docs.temporal.io/encyclopedia/workflow-message-passing) instead of trying to start Workflows atomically. The fourth value of the Workflow Id Reuse Policy, Terminate if Running, only applies to a Workflow Execution that is currently open within the Namespace. For Terminate if Running, the Retention Period is not a consideration for this policy. If there is an attempt to spawn a Workflow Execution with a Workflow Id Reuse Policy that won't allow it, the Server will prevent the Workflow Execution from spawning. ### What is a Workflow Id Conflict Policy?[​](https://docs.temporal.io/workflow-execution/workflowid-runid#workflow-id-conflict-policy "Direct link to What is a Workflow Id Conflict Policy?") A Workflow Id Conflict Policy determines how to resolve a conflict when spawning a new Workflow Execution with a particular Workflow Id used by an existing Open Workflow Execution. See [Workflow Id Reuse Policy](https://docs.temporal.io/workflow-execution/workflowid-runid#workflow-id-reuse-policy) for managing the reuse of a Workflow Id of a Closed Workflow. By default, this results in a `Workflow execution already started` error. note The default [StartWorkflowOptions](https://pkg.go.dev/go.temporal.io/sdk/internal#StartWorkflowOptions) behavior in the Go SDK is to not return an error when a new Workflow Execution is attempted with the same Workflow Id as an Open Workflow Execution. Instead, it returns a WorkflowRun instance representing the current or last run of the Open Workflow Execution. To return the `Workflow execution already started` error, set `WorkflowExecutionErrorWhenAlreadyStarted` to `true`. The Workflow Id Conflict Policy can have one of the following values: * **Fail:** Prevents the Workflow Execution from spawning and returns a `Workflow execution already started` error. **This is the default policy, if one isn't specified.** * **Use Existing:** Prevents the Workflow Execution from spawning and returns a successful response with the Open Workflow Execution's Run Id. * **Terminate Existing:** Terminates the Open Workflow Execution then spawns the new Workflow Execution with the same Workflow Id. * [What is a Run Id?](https://docs.temporal.io/workflow-execution/workflowid-runid#run-id) * [Which operations lead to non-determinism issues?](https://docs.temporal.io/workflow-execution/workflowid-runid#run-id-non-determinism) * [What is a Workflow Id?](https://docs.temporal.io/workflow-execution/workflowid-runid#workflow-id) * [What is a Workflow Id Reuse Policy?](https://docs.temporal.io/workflow-execution/workflowid-runid#workflow-id-reuse-policy) * [What is a Workflow Id Conflict Policy?](https://docs.temporal.io/workflow-execution/workflowid-runid#workflow-id-conflict-policy) --- # Workers - Ruby SDK | Temporal Platform Documentation [Skip to main content](https://docs.temporal.io/develop/ruby/workers#__docusaurus_skipToContent_fallback) On this page ![Ruby SDK Banner](https://docs.temporal.io/assets/images/banner-ruby-temporal-be833f13b8e3655d7a8d4e50119b7da2.png) Workers[​](https://docs.temporal.io/develop/ruby/workers#workers "Direct link to Workers") ------------------------------------------------------------------------------------------- * [Worker processes](https://docs.temporal.io/develop/ruby/workers/run-worker-process) * [Workers](https://docs.temporal.io/develop/ruby/workers#workers) --- # Asynchronous Activity - TypeScript SDK | Temporal Platform Documentation [Skip to main content](https://docs.temporal.io/develop/typescript/activities/asynchronous-activity#__docusaurus_skipToContent_fallback) On this page How to asynchronously complete an Activity[​](https://docs.temporal.io/develop/typescript/activities/asynchronous-activity#asynchronous-activity-completion "Direct link to How to asynchronously complete an Activity") ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- [Asynchronous Activity Completion](https://docs.temporal.io/activity-execution#asynchronous-activity-completion) enables the Activity Function to return without the Activity Execution completing. There are three steps to follow: 1. The Activity provides the external system with identifying information needed to complete the Activity Execution. Identifying information can be a [Task Token](https://docs.temporal.io/activity-execution#task-token) , or a combination of Namespace, Workflow Id, and Activity Id. 2. The Activity Function completes in a way that identifies it as waiting to be completed by an external system. 3. The Temporal Client is used to Heartbeat and complete the Activity. To asynchronously complete an Activity, call [`AsyncCompletionClient.complete`](https://typescript.temporal.io/api/classes/client.AsyncCompletionClient#complete) . [activities-examples/src/activities/async-completion.ts](https://github.com/temporalio/samples-typescript/blob/main/activities-examples/src/activities/async-completion.ts) import { CompleteAsyncError, activityInfo } from '@temporalio/activity';import { Client } from '@temporalio/client';export async function doSomethingAsync(): Promise { const taskToken = activityInfo().taskToken; setTimeout(() => doSomeWork(taskToken), 1000); throw new CompleteAsyncError();}// this work could be done in a different process or on a different machineasync function doSomeWork(taskToken: Uint8Array): Promise { const client = new Client(); // does some work... await client.activity.complete(taskToken, "Job's done!");} Local Activities[​](https://docs.temporal.io/develop/typescript/activities/asynchronous-activity#local-activities "Direct link to Local Activities") ----------------------------------------------------------------------------------------------------------------------------------------------------- To call [Local Activities](https://docs.temporal.io/local-activity) in TypeScript, use [`proxyLocalActivities`](https://typescript.temporal.io/api/namespaces/workflow/#proxylocalactivities) . import * as workflow from '@temporalio/workflow';const { getEnvVar } = workflow.proxyLocalActivities({ startToCloseTimeout: '2 seconds',});export async function yourWorkflow(): Promise { const someSetting = await getEnvVar('SOME_SETTING'); // ...} Local Activities must be registered with the Worker the same way non-local Activities are. * [How to asynchronously complete an Activity](https://docs.temporal.io/develop/typescript/activities/asynchronous-activity#asynchronous-activity-completion) * [Local Activities](https://docs.temporal.io/develop/typescript/activities/asynchronous-activity#local-activities) --- # Cloud Ops API | Temporal Platform Documentation [Skip to main content](https://docs.temporal.io/ops#__docusaurus_skipToContent_fallback) On this page Support, stability, and dependency info The Temporal Cloud Operations API is in [Public Preview](https://docs.temporal.io/evaluate/development-production-features/release-stages#public-preview) . The Temporal Cloud Operations API, or the Cloud Ops API, is an open source, public [HTTP API](https://saas-api.tmprl.cloud/docs/httpapi.html#description/introduction) and [gRPC API](https://github.com/temporalio/cloud-api/tree/main) for programmatically managing Temporal Cloud control plane resources, including [Namespaces](https://docs.temporal.io/cloud/namespaces) , [Users](https://docs.temporal.io/cloud/users) , [Service Accounts](https://docs.temporal.io/cloud/service-accounts) , [API keys](https://docs.temporal.io/cloud/api-keys) , and others. The Temporal Cloud [Terraform Provider](https://docs.temporal.io/cloud/terraform-provider) , [tcld CLI](https://docs.temporal.io/cloud/tcld) , and Web UI all use the Cloud Ops API. Develop applications with the Cloud Ops API[​](https://docs.temporal.io/ops#develop-applications-with-the-cloud-ops-api "Direct link to Develop applications with the Cloud Ops API") -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- You can use the HTTP API or the gRPC API depending on how you need to integrate with your platform. The URL to access both the HTTP and gRPC Cloud Ops API is `saas-api.tmprl.cloud`. ### Prerequisites[​](https://docs.temporal.io/ops#prerequisites "Direct link to Prerequisites") These prerequisites are required for using either HTTP or gRPC. * [Temporal Cloud user account](https://docs.temporal.io/cloud/get-started) * [API Key](https://docs.temporal.io/cloud/tcld/apikey#create) for authentication ### Use cases[​](https://docs.temporal.io/ops#use-cases "Direct link to Use cases") Some common reasons you might use the API are to: * Provision Namespaces per environment or tenant via pipelines. * Bootstrap new projects by creating users, assigning roles, and creating Namespaces via custom scripts. * Rotate service account keys on a schedule with a job. * Audit and report access across orgs with scheduled HTTP requests. ### Using HTTP[​](https://docs.temporal.io/ops#using-http "Direct link to Using HTTP") [The HTTP API](https://saas-api.tmprl.cloud/docs/httpapi.html#description/introduction) supports the same operations as the [gRPC API](https://docs.temporal.io/ops#using-grpc) , but it's usable via standard HTTP methods and authentication. This may be a more convenient option if you are writing automation scripts for CI/CD or you can't use gRPC due to network policies, proxies, tooling gaps, or language/runtime constraints. Since it's standard HTTP, it's language agnostic giving you the ability to run cloud operations consistently. note This _does not_ allow interaction with individual Workflows or Activities via HTTP. ### Using gRPC[​](https://docs.temporal.io/ops#using-grpc "Direct link to Using gRPC") _For Go developers:_ * Use the [Go SDK](https://github.com/temporalio/cloud-sdk-go) for the simplest setup experience _For other programming languages:_ * Basic familiarity with gRPC and Protocol Buffers (protobuf) * [Protocol Buffers](https://github.com/protocolbuffers/protobuf/releases) * [gRPC](https://grpc.io/docs/languages/) in your preferred programming language You can use the provided proto files to generate client libraries in your desired programming language, and then use that client to access the gRPC API. You can also find the [full gRPC docs on Buf](https://buf.build/temporalio/cloud-api/docs/main:temporal.api.cloud.cloudservice.v1#temporal.api.cloud.cloudservice.v1.CloudService) . #### Using the Go SDK[​](https://docs.temporal.io/ops#using-the-go-sdk "Direct link to Using the Go SDK") If you're developing in Go, we recommend using the [Go SDK](https://github.com/temporalio/cloud-sdk-go) which provides pre-compiled Go bindings and a more idiomatic interface. The Go SDK handles all the protobuf compilation and provides ready-to-use Go types and client interfaces. You can also use the [Go samples](https://github.com/temporalio/cloud-samples-go) to help you get started with the Cloud Ops API using the Go SDK. To start using the Go SDK with the Cloud Ops API, follow these steps: 1. Install the Go SDK: go get github.com/temporalio/cloud-sdk-go 2. Import and use the SDK: import ( "github.com/temporalio/cloud-sdk-go/client") 3. The Go SDK provides pre-built client interfaces that handle authentication and connection setup. Refer to the [Go samples](https://github.com/temporalio/cloud-samples-go) for detailed usage examples. The Go SDK eliminates the need to work directly with generated protobuf files and provides a more idiomatic Go experience. #### Compile the API and use the generated code (For other languages)[​](https://docs.temporal.io/ops#compile-the-api-and-use-the-generated-code-for-other-languages "Direct link to Compile the API and use the generated code (For other languages)") For programming languages other than Go, download the gRPC protobufs from the [Cloud Ops API repository](https://github.com/temporalio/cloud-api/tree/main/temporal/api/cloud) and compile them manually. Use [gRPC](https://grpc.io/docs/) to compile and generate code in your preferred [programming language](https://grpc.io/docs/#official-support) . The steps below use Python as an example and require [Python's gRPC tools](https://grpc.io/docs/languages/python/quickstart/#grpc-tools) to be installed, but the approach can be adapted for other supported programming languages. 1. Clone the Temporal Cloud API repository: git clone https://github.com/temporalio/cloud-api.gitcd cloud-api 2. Copy Protobuf files: * Navigate to the `temporal` directory. * Copy the protobuf files to your project directory. 3. Compile the Protobuf files: python -m grpc_tools.protoc -I./ --python_out=./ --grpc_python_out=./ *.proto * `-I` specifies the directory of the `.proto` files. * `--python_out=` sets the output directory for generated Python classes. * `--grpc_python_out=` sets the output directory for generated gRPC service classes. * `*.proto` processes all `.proto` files. After compiling the Protobuf files, you will have generated code files in your project directory. These files enable interaction with the Temporal Cloud API in your chosen programming language. 4. Import the Generated Files: * Locate the Python files (.py) generated in your project directory. * Import these files into your Python application where you intend to interact with the Temporal Cloud API. 5. Use the API: * Use the classes and methods defined in the imported files to communicate with the Temporal Cloud services. * Ensure to handle any required authentication or configuration as needed for Temporal Cloud. This approach can be adapted for other programming languages by following their respective import and usage conventions for the generated code files. Usage guidelines[​](https://docs.temporal.io/ops#usage-guidelines "Direct link to Usage guidelines") ----------------------------------------------------------------------------------------------------- When interacting with the Temporal Cloud Ops API, follow these guidelines: * API version header: * Always include the `temporal-cloud-api-version` header in your requests, specifying the API version identifier. * The current API version can be found [here](https://github.com/temporalio/cloud-api/blob/main/VERSION#L1C1-L1C14) . * Connection URL: * Connect to the Temporal Cloud using the gRPC URL: `saas-api.tmprl.cloud:443`. * Engagement steps: * Generate API key: * Obtain an [API Key for authentication](https://docs.temporal.io/cloud/api-keys#manage-api-keys) . Note that many operations may require Admin privileges. * Set up client: * Establish a secure connection to the Temporal Cloud. Refer to the example [Client setup in Go](https://github.com/temporalio/cloud-samples-go/blob/main/client/temporal/client.go) for guidance. * Execute operations: * For operation specifics, refer to the `cloudservice/v1/request_response.proto` for gRPC messages and `cloudservice/v1/service.proto` for gRPC services. These steps provide a structured approach to using the Temporal Cloud Ops API effectively, ensuring proper authentication and connection setup. Rate limits[​](https://docs.temporal.io/ops#rate-limits "Direct link to Rate limits") -------------------------------------------------------------------------------------- The Temporal Cloud Operations API implements rate limiting to ensure system stability and fair usage across all users. Rate limits are applied based on identity type, with different limits for users and service accounts. ### Account-level rate limit[​](https://docs.temporal.io/ops#account-level-rate-limit "Direct link to Account-level rate limit") **Total rate limit: 160 requests per second (RPS)** This limit applies to all requests made to the Temporal Cloud control plane by any client (tcld, UI, Cloud Ops API) or identity type (user, service account) within your account. The total account throughput cannot exceed the limit regardless of the number of users or service accounts making requests. ### Per-identity rate limits[​](https://docs.temporal.io/ops#per-identity-rate-limits "Direct link to Per-identity rate limits") **User rate limit: 40 RPS per user** This limit applies to all requests made by each user through any client (tcld, UI, Cloud Ops API), regardless of the authentication method used (SSO or API keys). **Service account rate limit: 80 RPS per service account** This limit applies to all requests made by each service account through any client (tcld, Cloud Ops API). **Asynchronous Operations: 10 concurrent operations at a time** This limits the number of concurrent asynchronous operations that can be in-flight at any given time. ### Important considerations[​](https://docs.temporal.io/ops#important-considerations "Direct link to Important considerations") * Rate limits are enforced across all Temporal Cloud control plane operations * Multiple clients used by the same identity (user or service account) share the same rate limit * Authentication method (SSO, API keys) does not affect rate limiting * These limits help ensure system stability and prevent any single account or identity from overwhelming the service ### Request limit increases[​](https://docs.temporal.io/ops#request-limit-increases "Direct link to Request limit increases") If your use case requires higher rate limits, you can request an increase by [submitting a support ticket](https://docs.temporal.io/cloud/support#support-ticket) . When requesting a limit increase, please provide: * Your current usage patterns and requirements * The specific limits you need increased * A description of your use case and why higher limits are necessary ### Provide feedback[​](https://docs.temporal.io/ops#provide-feedback "Direct link to Provide feedback") Your input is valuable! You can provide feedback through the following channels: * Submit request or feedback through a [support ticket](https://docs.temporal.io/cloud/support#support-ticket) * Open an issue in the [GitHub Repo](https://github.com/temporalio/cloud-api) * [Develop applications with the Cloud Ops API](https://docs.temporal.io/ops#develop-applications-with-the-cloud-ops-api) * [Prerequisites](https://docs.temporal.io/ops#prerequisites) * [Use cases](https://docs.temporal.io/ops#use-cases) * [Using HTTP](https://docs.temporal.io/ops#using-http) * [Using gRPC](https://docs.temporal.io/ops#using-grpc) * [Using the Go SDK](https://docs.temporal.io/ops#using-the-go-sdk) * [Compile the API and use the generated code (For other languages)](https://docs.temporal.io/ops#compile-the-api-and-use-the-generated-code-for-other-languages) * [Usage guidelines](https://docs.temporal.io/ops#usage-guidelines) * [Rate limits](https://docs.temporal.io/ops#rate-limits) * [Account-level rate limit](https://docs.temporal.io/ops#account-level-rate-limit) * [Per-identity rate limits](https://docs.temporal.io/ops#per-identity-rate-limits) * [Important considerations](https://docs.temporal.io/ops#important-considerations) * [Request limit increases](https://docs.temporal.io/ops#request-limit-increases) * [Provide feedback](https://docs.temporal.io/ops#provide-feedback) --- # Remote data encoding | Temporal Platform Documentation [Skip to main content](https://docs.temporal.io/remote-data-encoding#__docusaurus_skipToContent_fallback) On this page This page discusses [Remote Data Encoding](https://docs.temporal.io/remote-data-encoding#remote-data-encoding) . What is remote data encoding?[​](https://docs.temporal.io/remote-data-encoding#remote-data-encoding "Direct link to What is remote data encoding?") ---------------------------------------------------------------------------------------------------------------------------------------------------- Remote data encoding is exposing your Payload Codec via HTTP endpoints to support remote encoding and decoding. Running your encoding remotely allows you to use it with the [Temporal CLI](https://docs.temporal.io/cli) to encode/decode data for several commands including `temporal workflow show` and with Temporal Web UI to decode data in your Workflow Execution details view. To run data encoding/decoding remotely, use a [Codec Server](https://docs.temporal.io/codec-server) . A Codec Server is an HTTP server that uses your custom Codec logic to decode your data remotely. The Codec Server is independent of the Temporal Service and decodes your encrypted payloads through predefined endpoints. You create, operate, and manage access to your Codec Server in your own environment. The Temporal CLI and the Web UI in turn provide built-in hooks to call the Codec Server to decode encrypted payloads on demand. ### Encoding data on the Web UI and CLI[​](https://docs.temporal.io/remote-data-encoding#encoding-data-on-the-web-ui-and-cli "Direct link to Encoding data on the Web UI and CLI") You can perform some operations on your Workflow Execution using the Temporal CLI and the Web UI. For example, you can start or signal an active Workflow Execution from the Temporal CLI or cancel a Workflow Execution from the Web UI, which might require inputs that contain sensitive data. To encode this data, specify your [Codec Server endpoints](https://docs.temporal.io/codec-server) with the `codec-endpoint` parameter in [the Temporal CLI](https://docs.temporal.io/cli) and configure your Web UI to use the Codec Server endpoints. ### Decoding data on the Web UI and CLI[​](https://docs.temporal.io/remote-data-encoding#decoding-data-on-the-web-ui-and-cli "Direct link to Decoding data on the Web UI and CLI") If you use custom encoding, Payload data handled by the Temporal Service is stored encoded. Since the Web UI uses the [Visibility](https://docs.temporal.io/temporal-service/visibility) database to show events and data stored on the Temporal Server, all data in the Workflow Execution History in your Web UI is displayed in the encoded format. To decode output when using the Web UI and the Temporal CLI, use a [Codec Server](https://docs.temporal.io/codec-server) . Note that a remote data encoder is a separate system with access to your encryption keys and exposes APIs to encode and decode any data. Evaluate and ensure that your remote data encoder endpoints are secured and only authorized users have access to them. Samples: * [Go](https://github.com/temporalio/samples-go/tree/main/codec-server) * [Java](https://github.com/temporalio/sdk-java/tree/master/temporal-remote-data-encoder) * [Python](https://github.com/temporalio/samples-python/tree/main/encryption) * [TypeScript](https://github.com/temporalio/samples-typescript/tree/main/encryption) * [What is remote data encoding?](https://docs.temporal.io/remote-data-encoding#remote-data-encoding) * [Encoding data on the Web UI and CLI](https://docs.temporal.io/remote-data-encoding#encoding-data-on-the-web-ui-and-cli) * [Decoding data on the Web UI and CLI](https://docs.temporal.io/remote-data-encoding#decoding-data-on-the-web-ui-and-cli) --- # What is a Temporal Worker? | Temporal Platform Documentation [Skip to main content](https://docs.temporal.io/workers#__docusaurus_skipToContent_fallback) On this page This page discusses the following: * [Worker](https://docs.temporal.io/workers#worker) * [Worker Program](https://docs.temporal.io/workers#worker-program) * [Worker Entity](https://docs.temporal.io/workers#worker-entity) * [Worker Identity](https://docs.temporal.io/workers#worker-identity) * [Worker Process](https://docs.temporal.io/workers#worker-process) What is a Worker?[​](https://docs.temporal.io/workers#worker "Direct link to What is a Worker?") ------------------------------------------------------------------------------------------------- In day-to-day conversations, the term Worker is used to denote either a [Worker Program](https://docs.temporal.io/workers#worker-program) , a [Worker Process](https://docs.temporal.io/workers#worker-process) , or a [Worker Entity](https://docs.temporal.io/workers#worker-entity) . Temporal documentation aims to be explicit and differentiate between them. What is a Worker Program?[​](https://docs.temporal.io/workers#worker-program "Direct link to What is a Worker Program?") ------------------------------------------------------------------------------------------------------------------------- A Worker Program is the static code that defines the constraints of the Worker Process, developed using the APIs of a Temporal SDK. info * [How to run a development Worker using the Go SDK](https://docs.temporal.io/develop/go/workers/run-worker-process#develop-worker) * [How to run a development Worker using the Java SDK](https://docs.temporal.io/develop/java/workers/run-worker-process) * [How to run a development Worker using the PHP SDK](https://docs.temporal.io/develop/php/workers/run-worker-process#run-a-dev-worker) * [How to run a development Worker using the Python SDK](https://docs.temporal.io/develop/python/workers/run-worker-process#run-a-dev-worker) * [How to run a development Worker using the TypeScript SDK](https://docs.temporal.io/develop/typescript/workers/run-worker-process#run-a-dev-worker) * [How to run a development Worker using the .NET SDK](https://docs.temporal.io/develop/dotnet/workers/run-worker-process) * [How to run a Temporal Cloud Worker using the Go SDK](https://docs.temporal.io/develop/go/workers/cloud-worker) * [How to run a Temporal Cloud Worker using the TypeScript SDK](https://docs.temporal.io/develop/typescript/workers/run-worker-process#run-a-temporal-cloud-worker) What is a Worker Entity?[​](https://docs.temporal.io/workers#worker-entity "Direct link to What is a Worker Entity?") ---------------------------------------------------------------------------------------------------------------------- A Worker Entity is the individual Worker within a Worker Process that listens to a specific Task Queue. A Worker Entity listens and polls on a single Task Queue. A Worker Entity contains a Workflow Worker and/or an Activity Worker, which makes progress on Workflow Executions and Activity Executions, respectively. **Can a Worker handle more Workflow Executions than its cache size or number of supported threads?** Yes it can. However, the trade off is added latency. Workers are stateless, so any Workflow Execution in a blocked state can be safely removed from a Worker. Later on, it can be resurrected on the same or different Worker when the need arises (in the form of an external event). Therefore, a single Worker can handle millions of open Workflow Executions, assuming it can handle the update rate and that a slightly higher latency is not a concern. **Operation guides:** * [How to tune Workers](https://docs.temporal.io/develop/worker-performance) * [Worker tuning quick reference](https://docs.temporal.io/develop/worker-tuning-reference) - SDK defaults and metrics What is a Worker Identity?[​](https://docs.temporal.io/workers#worker-identity "Direct link to What is a Worker Identity?") ---------------------------------------------------------------------------------------------------------------------------- Workers have an associated identifier that helps identify the specific Worker instance. By default, Temporal SDKs set a Worker Identity to `${process.pid}@${os.hostname()}`, which combines the Worker's process ID (`process.pid`) and the hostname of the machine running the Worker (`os.hostname()`). The Worker Identity is visible in various contexts, such as Workflow History and the list of pollers on a Task Queue. You can use the Worker Identity to aid in debugging operational issues. By providing a user assigned identifier, you can trace issues back to specific Worker instances. **What are some limitations of the default identity?** While the default identity format may seem sensible, it often proves to be of limited usefulness in cloud environments. Some common issues include: * **Docker containers**: When running Workers inside Docker containers, the process ID is always `1`, as each container typically runs a single process. This makes the process identifier meaningless for identification purposes. * **Random hostnames**: In some cloud environments, such as Amazon ECS (Elastic Container Service), the hostname is a randomly generated string that does not provide any meaningful information about the Worker's execution context. * **Ephemeral IP addresses**: In certain cases, the hostname might be set to an ephemeral IP address, which can change over time and does not uniquely identify a Worker instance. **What are some recommended approaches?** It is recommended that you ensure that the Worker Identity can be linked back to the corresponding machine, process, execution context, or log stream. In some execution environments, this might require that you explicitly specify the Worker Identity. Here are some approaches: * **Use environment-specific identifiers**: Choose an identifier that is specific to your execution environment. For example, when running Workers on Amazon ECS, you can set the Worker Identity to the ECS Task ID, which uniquely identifies the task running the Worker. * **Include relevant context**: Incorporate information that helps establish the context of the Worker, such as the deployment environment (`staging` or `production`), region, or any other relevant details. * **Ensure uniqueness**: Make sure that the Worker Identity is unique within your system to avoid ambiguity when debugging issues. * **Keep it concise**: While including relevant information is important, try to keep the Worker Identity concise and easily readable to facilitate quick identification and troubleshooting. What is a Worker Process?[​](https://docs.temporal.io/workers#worker-process "Direct link to What is a Worker Process?") ------------------------------------------------------------------------------------------------------------------------- ![Component diagram of a Worker Process and the Temporal Server](https://docs.temporal.io/diagrams/worker-and-server-component.svg) Component diagram of a Worker Process and the Temporal Server A Worker Process is responsible for polling a [Task Queue](https://docs.temporal.io/task-queue) , dequeueing a [Task](https://docs.temporal.io/tasks#task) , executing your code in response to a Task, and responding to the [Temporal Service](https://docs.temporal.io/temporal-service) with the results. More formally, a Worker Process is any process that implements the Task Queue Protocol and the Task Execution Protocol. * A Worker Process is a Workflow Worker Process if the process implements the Workflow Task Queue Protocol and executes the Workflow Task Execution Protocol to make progress on a Workflow Execution. A Workflow Worker Process can listen on an arbitrary number of Workflow Task Queues and can execute an arbitrary number of Workflow Tasks. * A Worker Process is an Activity Worker Process if the process implements the Activity Task Queue Protocol and executes the Activity Task Processing Protocol to make progress on an Activity Execution. An Activity Worker Process can listen on an arbitrary number of Activity Task Queues and can execute an arbitrary number of Activity Tasks. **Worker Processes are external to a Temporal Service.** Temporal Application developers are responsible for developing [Worker Programs](https://docs.temporal.io/workers#worker-program) and operating Worker Processes. Said another way, the [Temporal Service](https://docs.temporal.io/temporal-service) (including the Temporal Cloud) doesn't execute any of your code (Workflow and Activity Definitions) on Temporal Service machines. The Temporal Service is solely responsible for orchestrating [State Transitions](https://docs.temporal.io/workflow-execution#state-transition) and providing Tasks to the next available [Worker Entity](https://docs.temporal.io/workers#worker-entity) . While data transferred in Event Histories is [secured by mTLS](https://docs.temporal.io/self-hosted-guide/security#encryption-in-transit-with-mtls) , by default, it is still readable at rest in the Temporal Service. To solve this, Temporal SDKs offer a [Data Converter API](https://docs.temporal.io/dataconversion) that you can use to customize the serialization of data going out of and coming back in to a Worker Entity, with the net effect of guaranteeing that the Temporal Service cannot read sensitive business data. In many of our tutorials, we show you how to run both a Temporal Service and one Worker on the same machine for local development. However, a production-grade Temporal Application typically has a _fleet_ of Worker Processes, all running on hosts external to the Temporal Service. A Temporal Application can have as many Worker Processes as needed. A Worker Process can be both a Workflow Worker Process and an Activity Worker Process. Many SDKs support the ability to have multiple Worker Entities in a single Worker Process. (Worker Entity creation and management differ between SDKs.) A single Worker Entity can listen to only a single Task Queue. But if a Worker Process has multiple Worker Entities, the Worker Process could be listening to multiple Task Queues. ![Entity relationship diagram (meta model) of Worker Processes, Task Queues, and Tasks](https://docs.temporal.io/diagrams/worker-and-server-entity-relationship.svg) Entity relationship diagram (meta model) of Worker Processes, Task Queues, and Tasks Worker Processes executing Activity Tasks must have access to any resources needed to execute the actions that are defined in Activity Definitions, such as the following: * Network access for external API calls. * Credentials for infrastructure provisioning. * Specialized GPUs for machine learning utilities. The Temporal Service itself has [internal workers](https://temporal.io/blog/workflow-engine-principles/#system-workflows-1910) for system Workflow Executions. However, these internal workers are not visible to the developer. * [What is a Worker?](https://docs.temporal.io/workers#worker) * [What is a Worker Program?](https://docs.temporal.io/workers#worker-program) * [What is a Worker Entity?](https://docs.temporal.io/workers#worker-entity) * [What is a Worker Identity?](https://docs.temporal.io/workers#worker-identity) * [What is a Worker Process?](https://docs.temporal.io/workers#worker-process) --- # Temporal Workflow Execution overview | Temporal Platform Documentation [Skip to main content](https://docs.temporal.io/workflow-execution#__docusaurus_skipToContent_fallback) On this page This page provides an overview of Workflow Execution: * [What is a Workflow Execution?](https://docs.temporal.io/workflow-execution#workflow-execution) * [Replay](https://docs.temporal.io/workflow-execution#replay) * [Commands and awaitables](https://docs.temporal.io/workflow-execution#commands-awaitables) * [What is a Command?](https://docs.temporal.io/workflow-execution#command) * [Checking Workflow Execution Status](https://docs.temporal.io/workflow-execution#workflow-execution-status) * [Workflow Execution Chain](https://docs.temporal.io/workflow-execution#workflow-execution-chain) * [Memo](https://docs.temporal.io/workflow-execution#memo) * [State Transition](https://docs.temporal.io/workflow-execution#state-transition) What is a Workflow Execution?[​](https://docs.temporal.io/workflow-execution#workflow-execution "Direct link to What is a Workflow Execution?") ------------------------------------------------------------------------------------------------------------------------------------------------ While the Workflow Definition is the code that defines the Workflow, the Workflow Execution is created by executing that code. A Temporal Workflow Execution is a durable, reliable, and scalable function execution. It is the main unit of execution of a [Temporal Application](https://docs.temporal.io/temporal#temporal-application) . * [How to start a Workflow Execution using temporal](https://docs.temporal.io/cli/workflow#start) * [How to start a Workflow Execution using the Go SDK](https://docs.temporal.io/develop/go/client/temporal-client#start-workflow-execution) * [How to start a Workflow Execution using the Java SDK](https://docs.temporal.io/develop/java/client/temporal-client#start-workflow-execution) * [How to start a Workflow Execution using the PHP SDK](https://docs.temporal.io/develop/php/client/temporal-client#start-workflow-execution) * [How to start a Workflow Execution using the Python SDK](https://docs.temporal.io/develop/python/client/temporal-client#start-workflow-execution) * [How to start a Workflow Execution using the TypeScript SDK](https://docs.temporal.io/develop/typescript/client/temporal-client#start-workflow-execution) * [How to start a Workflow Execution using the .NET SDK](https://docs.temporal.io/develop/dotnet/client/temporal-client#start-workflow) Each Temporal Workflow Execution has exclusive access to its local state. It executes concurrently to all other Workflow Executions, and communicates with other Workflow Executions through [Signals](https://docs.temporal.io/sending-messages#sending-signals) and the environment through [Activities](https://docs.temporal.io/activities) . While a single Workflow Execution has limits on size and throughput, a Temporal Application can consist of millions to billions of Workflow Executions. **Durability** Durability is the absence of an imposed time limit. A Workflow Execution is durable because it executes a Temporal Workflow Definition (also called a Temporal Workflow Function), your application code, effectively once and to completion—whether your code executes for seconds or years. **Reliability** Reliability is responsiveness in the presence of failure. A Workflow Execution is reliable, because it is fully recoverable after a failure. The Temporal Platform ensures the state of the Workflow Execution persists in the face of failures and outages and resumes execution from the latest state. **Scalability** Scalability is responsiveness in the presence of load. A single Workflow Execution is limited in size and throughput but is scalable because it can [Continue-As-New](https://docs.temporal.io/workflow-execution/continue-as-new) in response to load. A Temporal Application is scalable because the Temporal Platform is capable of supporting millions to billions of Workflow Executions executing concurrently, which is realized by the design and nature of the [Temporal Service](https://docs.temporal.io/temporal-service) and [Worker Processes](https://docs.temporal.io/workers#worker-process) . ### Replays[​](https://docs.temporal.io/workflow-execution#replay "Direct link to Replays") A Replay is the method by which a Workflow Execution resumes making progress. During a Replay the Commands that are generated are checked against an existing Event History. Replays are necessary and often happen to give the effect that Workflow Executions are resumable, reliable, and durable. For more information, see [Deterministic constraints](https://docs.temporal.io/workflow-definition#deterministic-constraints) . If a failure occurs, the Workflow Execution picks up where the last recorded event occurred in the Event History. * [How to use Replay APIs using the Go SDK](https://docs.temporal.io/develop/go/best-practices/testing-suite#replay) * [How to use Replay APIs using the Java SDK](https://docs.temporal.io/develop/java/best-practices/testing-suite#replay) * [How to use Replay APIs using the Python SDK](https://docs.temporal.io/develop/python/best-practices/testing-suite#replay) * [How to use Replay APIs using the TypeScript SDK](https://docs.temporal.io/develop/typescript/best-practices/testing-suite#replay) * [How to use Replay APIs using the .NET SDK](https://docs.temporal.io/develop/dotnet/best-practices/testing-suite#replay) ### Commands and awaitables[​](https://docs.temporal.io/workflow-execution#commands-awaitables "Direct link to Commands and awaitables") A Workflow Execution does two things: 1. Issue [Commands](https://docs.temporal.io/workflow-execution#command) . 2. Wait on an Awaitables (often called Futures). ![Command generation and waiting](https://docs.temporal.io/diagrams/workflow-execution-progession-simple.svg) Command generation and waiting Commands are issued and Awaitables are provided by the use of Workflow APIs in the [Workflow Definition](https://docs.temporal.io/workflow-definition) . Commands are generated whenever the Workflow Function is executed. The Worker Process supervises the Command generation and makes sure that it maps to the current Event History. (For more information, see [Deterministic constraints](https://docs.temporal.io/workflow-definition#deterministic-constraints) .) The Worker Process batches the Commands and then suspends progress to send the Commands to the Temporal Service whenever the Workflow Function reaches a place where it can no longer progress without a result from an Awaitable. A Workflow Execution may only ever block progress on an Awaitable that is provided through a Temporal SDK API. Awaitables are provided when using APIs for the following: * Awaiting: Progress can block using explicit "Await" APIs. * Requesting cancellation of another Workflow Execution: Progress can block on confirmation that the other Workflow Execution is cancelled. * Sending a [Signal](https://docs.temporal.io/sending-messages#sending-signals) : Progress can block on confirmation that the Signal sent. * Spawning a [Child Workflow Execution](https://docs.temporal.io/child-workflows) : Progress can block on confirmation that the Child Workflow Execution started, and on the result of the Child Workflow Execution. * Spawning an [Activity Execution](https://docs.temporal.io/activity-execution) : Progress can block on the result of the Activity Execution. * Starting a Timer: Progress can block until the Timer fires. ### What is a Command?[​](https://docs.temporal.io/workflow-execution#command "Direct link to What is a Command?") A Command is a requested action issued by a [Worker](https://docs.temporal.io/workers#worker) to the [Temporal Service](https://docs.temporal.io/temporal-service) after a [Workflow Task Execution](https://docs.temporal.io/tasks#workflow-task-execution) completes. The action that the Temporal Service takes is recorded in the [Workflow Execution's](https://docs.temporal.io/workflow-execution#workflow-execution) [Event History](https://docs.temporal.io/workflow-execution/event#event-history) as an [Event](https://docs.temporal.io/workflow-execution/event) . The Workflow Execution can await on some of the Events that come as a result from some of the Commands. Commands are generated by the use of Workflow APIs in your code. During a Workflow Task Execution there may be several Commands that are generated. The Commands are batched and sent to the Temporal Service as part of the Workflow Task Execution completion request, after the Workflow Task has progressed as far as it can with the Workflow function. There will always be [WorkflowTaskStarted](https://docs.temporal.io/references/events#workflowtaskstarted) and [WorkflowTaskCompleted](https://docs.temporal.io/references/events#workflowtaskcompleted) Events in the Event History when there is a Workflow Task Execution completion request. ![Commands are generated by the use of Workflow APIs in your code](https://docs.temporal.io/diagrams/commands.svg) Commands are generated by the use of Workflow APIs in your code Commands are described in the [Command reference](https://docs.temporal.io/references/commands) and are defined in the [Temporal gRPC API](https://github.com/temporalio/api/blob/master/temporal/api/command/v1/message.proto) . ### Status[​](https://docs.temporal.io/workflow-execution#workflow-execution-status "Direct link to Status") A Workflow Execution can be either _Open_ or _Closed_. ![Workflow Execution statuses](https://docs.temporal.io/diagrams/workflow-execution-statuses.svg) Workflow Execution statuses #### Open[​](https://docs.temporal.io/workflow-execution#open "Direct link to Open") An _Open_ status means that the Workflow Execution is able to make progress. * Running: The only Open status for a Workflow Execution. When the Workflow Execution is Running, it is either actively progressing or is waiting on something. #### Closed[​](https://docs.temporal.io/workflow-execution#closed "Direct link to Closed") A _Closed_ status means that the Workflow Execution cannot make further progress because of one of the following reasons: * Cancelled: The Workflow Execution successfully handled a cancellation request. * Completed: The Workflow Execution has completed successfully. * Continued-As-New: The Workflow Execution [Continued-As-New](https://docs.temporal.io/workflow-execution/continue-as-new) . * Failed: The Workflow Execution returned an error and failed. * Terminated: The Workflow Execution was terminated. * Timed Out: The Workflow Execution reached a timeout limit. ### Workflow Execution Chain[​](https://docs.temporal.io/workflow-execution#workflow-execution-chain "Direct link to Workflow Execution Chain") A Workflow Execution Chain is a sequence of Workflow Executions that share the same Workflow Id. Each link in the Chain is often called a Workflow Run. Each Workflow Run in the sequence is connected by one of the following: * [Continue-As-New](https://docs.temporal.io/workflow-execution/continue-as-new) * [Retries](https://docs.temporal.io/encyclopedia/retry-policies) * [Temporal Cron Job](https://docs.temporal.io/cron-job) A Workflow Execution is uniquely identified by its [Namespace](https://docs.temporal.io/namespaces) , [Workflow Id](https://docs.temporal.io/workflow-execution/workflowid-runid#workflow-id) , and [Run Id](https://docs.temporal.io/workflow-execution/workflowid-runid#run-id) . The [Workflow Execution Timeout](https://docs.temporal.io/encyclopedia/detecting-workflow-failures#workflow-execution-timeout) applies to a Workflow Execution Chain. The [Workflow Run Timeout](https://docs.temporal.io/encyclopedia/detecting-workflow-failures#workflow-run-timeout) applies to a single Workflow Execution (Workflow Run). What is a Memo?[​](https://docs.temporal.io/workflow-execution#memo "Direct link to What is a Memo?") ------------------------------------------------------------------------------------------------------ A Memo is a non-indexed set of Workflow Execution metadata that developers supply at start time or in Workflow code and that is returned when you describe or list Workflow Executions. The primary purpose of using a Memo is to enhance the organization and management of Workflow Executions. Add your own metadata, such as notes or descriptions, to a Workflow Execution, which lets you annotate and categorize Workflow Executions based on developer-defined criteria. This feature is particularly useful when dealing with numerous Workflow Executions because it facilitates the addition of context, reminders, or any other relevant information that aids in understanding or tracking the Workflow Execution. Use Memos judiciously Memos shouldn't store data that's critical to the execution of a Workflow, for some of the following reasons: * Unlike Workflow inputs, Memos lack type safety * Memos are subject to eventual consistency and may not be immediately available * Excessive reliance on Memos hides mutable state from the Workflow Execution History What is a State Transition?[​](https://docs.temporal.io/workflow-execution#state-transition "Direct link to What is a State Transition?") ------------------------------------------------------------------------------------------------------------------------------------------ A State Transition is a unit of progress made by a [Workflow Execution](https://docs.temporal.io/workflow-execution#workflow-execution) . Each State Transition is recorded in a persistence store. Some operations, such as [Activity Heartbeats](https://docs.temporal.io/encyclopedia/detecting-activity-failures#activity-heartbeat) , require only one or two State Transitions each. With an Activity Heartbeat, there are two: the Activity Heartbeat and a Timer. Most operations require multiple State Transitions. For example, a simple Workflow with two sequential [Activity Tasks](https://docs.temporal.io/tasks#activity-task) (and no retries) produces 11 State Transitions: two for Workflow start, four for each Activity, and one for Workflow completion. NEXT STEPS For more information on Workflow Execution, please refer to the following subpages: * [Event](https://docs.temporal.io/workflow-execution/event) * [Workflow Id and Run Id](https://docs.temporal.io/workflow-execution/workflowid-runid) * [Limits](https://docs.temporal.io/workflow-execution/limits) * [Continue-as-New](https://docs.temporal.io/workflow-execution/continue-as-new) * [Timers and Start Delay](https://docs.temporal.io/workflow-execution/timers-delays) * [What is a Workflow Execution?](https://docs.temporal.io/workflow-execution#workflow-execution) * [Replays](https://docs.temporal.io/workflow-execution#replay) * [Commands and awaitables](https://docs.temporal.io/workflow-execution#commands-awaitables) * [What is a Command?](https://docs.temporal.io/workflow-execution#command) * [Status](https://docs.temporal.io/workflow-execution#workflow-execution-status) * [Open](https://docs.temporal.io/workflow-execution#open) * [Closed](https://docs.temporal.io/workflow-execution#closed) * [Workflow Execution Chain](https://docs.temporal.io/workflow-execution#workflow-execution-chain) * [What is a Memo?](https://docs.temporal.io/workflow-execution#memo) * [What is a State Transition?](https://docs.temporal.io/workflow-execution#state-transition) --- # Temporal Service | Temporal Platform Documentation [Skip to main content](https://docs.temporal.io/temporal-service#__docusaurus_skipToContent_fallback) info Please note an important update in our terminology. We now refer to the Temporal Cluster as the Temporal Service. This guide provides a comprehensive technical overview of a Temporal Service. A Temporal Service is the group of services, known as the [Temporal Server](https://docs.temporal.io/temporal-service/temporal-server) , combined with [Persistence](https://docs.temporal.io/temporal-service/persistence) and [Visibility](https://docs.temporal.io/temporal-service/visibility) stores, that together act as a component of the Temporal Platform. See the Self-hosted Temporal Service [production deployment guide](https://docs.temporal.io/self-hosted-guide) for implementation guidance. ![A Temporal Service (Server + persistence)](https://docs.temporal.io/diagrams/temporal-cluster.svg) A Temporal Service (Server + persistence) --- # Events and Event History | Temporal Platform Documentation [Skip to main content](https://docs.temporal.io/workflow-execution/event#__docusaurus_skipToContent_fallback) On this page This page discusses the following: * [Events](https://docs.temporal.io/workflow-execution/event#event) * [Activity Events](https://docs.temporal.io/workflow-execution/event#activity-events) * [Event History](https://docs.temporal.io/workflow-execution/event#event-history) * [Event Loop](https://docs.temporal.io/workflow-execution/event#event-loop) * [Time Constraints](https://docs.temporal.io/workflow-execution/event#time-constraints) * [Reset](https://docs.temporal.io/workflow-execution/event#reset) * [Side Effect](https://docs.temporal.io/workflow-execution/event#side-effect) The Temporal Service tracks the progress of each Workflow Execution by appending information about Events, such as when the Workflow Execution began or ended, to the Event History associated with that execution. This information not only enables developers to know what took place, but is also essential for providing Durable Execution, since it enables the Workflow Execution to recover from a crash and continue making progress. In order to maintain high performance, the Temporal Service places limits on both the number and size of items in the Event History for each Workflow Execution. What is an Event?[​](https://docs.temporal.io/workflow-execution/event#event "Direct link to What is an Event?") ----------------------------------------------------------------------------------------------------------------- Events are created by the Temporal Service in response to external occurrences and Commands generated by a Workflow Execution. Each Event corresponds to an `enum` that is defined in the [Server API](https://github.com/temporalio/api/blob/master/temporal/api/enums/v1/event_type.proto) . All Events are recorded in the [Event History](https://docs.temporal.io/workflow-execution/event#event-history) . A list of all possible Events that could appear in a Workflow Execution Event History is provided in the [Event reference](https://docs.temporal.io/references/events) . ### Activity Events[​](https://docs.temporal.io/workflow-execution/event#activity-events "Direct link to Activity Events") Seven Activity-related Events are added to Event History at various points in an Activity Execution: * After a [Workflow Task Execution](https://docs.temporal.io/tasks#activity-task-execution) reaches a line of code that starts/executes an Activity, the Worker sends the Activity Type and arguments to the Temporal Service, and the Temporal Service adds an [ActivityTaskScheduled](https://docs.temporal.io/references/events#activitytaskscheduled) Event to Event History. * When `ActivityTaskScheduled` is added to History, the Temporal Service adds a corresponding Activity Task to the Task Queue. * A Worker polling that Task Queue picks up the Activity Task and runs the Activity function or method. * If the Activity function returns, the Worker reports completion to the Temporal Service, and the Temporal Service adds [ActivityTaskStarted](https://docs.temporal.io/references/events#activitytaskstarted) and [ActivityTaskCompleted](https://docs.temporal.io/references/events#activitytaskcompleted) to Event History. * If the Activity function throws a [non-retryable Failure](https://docs.temporal.io/references/failures#non-retryable) , the Temporal Service adds [ActivityTaskStarted](https://docs.temporal.io/references/events#activitytaskstarted) and [ActivityTaskFailed](https://docs.temporal.io/references/events#activitytaskfailed) to Event History. * If the Activity function throws an error or retryable Failure, the Temporal Service schedules an Activity Task retry to be added to the Task Queue (unless you’ve reached the Maximum Attempts value of the [Retry Policy](https://docs.temporal.io/encyclopedia/retry-policies) , in which case the Temporal Service adds [ActivityTaskStarted](https://docs.temporal.io/references/events#activitytaskstarted) and [ActivityTaskFailed](https://docs.temporal.io/references/events#activitytaskfailed) to Event History). * If the Activity’s [Start-to-Close Timeout](https://docs.temporal.io/encyclopedia/detecting-activity-failures#start-to-close-timeout) passes before the Activity function returns or throws, the Temporal Service schedules a retry. * If the Activity’s [Schedule-to-Close Timeout](https://docs.temporal.io/encyclopedia/detecting-activity-failures#schedule-to-close-timeout) passes before Activity Execution is complete, or if [Schedule-to-Start Timeout](https://docs.temporal.io/encyclopedia/detecting-activity-failures#schedule-to-start-timeout) passes before a Worker gets the Activity Task, the Temporal Service writes [ActivityTaskTimedOut](https://docs.temporal.io/references/events#activitytasktimedout) to Event History. * If the Activity is [canceled](https://docs.temporal.io/activity-execution#cancellation) , the Temporal Service writes [ActivityTaskCancelRequested](https://docs.temporal.io/references/events#activitytaskcancelrequested) to Event History, and if the Activity accepts cancellation, the Temporal Service writes [ActivityTaskCanceled](https://docs.temporal.io/references/events#activitytaskcanceled) . note While the Activity is running and retrying, [ActivityTaskScheduled](https://docs.temporal.io/references/events#activitytaskscheduled) is the only Activity-related Event in History: [ActivityTaskStarted](https://docs.temporal.io/references/events#activitytaskstarted) is written along with a terminal Event like [ActivityTaskCompleted](https://docs.temporal.io/references/events#activitytaskcompleted) or [ActivityTaskFailed](https://docs.temporal.io/references/events#activitytaskfailed) . ### What is an Event History?[​](https://docs.temporal.io/workflow-execution/event#event-history "Direct link to What is an Event History?") An append-only log of [Events](https://docs.temporal.io/workflow-execution/event#event) for your application. * Event History is durably persisted by the Temporal service, enabling seamless recovery of your application state from crashes or failures. * It also serves as an audit log for debugging. ### Event History limits[​](https://docs.temporal.io/workflow-execution/event#event-history-limits "Direct link to Event History limits") The Temporal Service stores the complete Event History for the entire lifecycle of a Workflow Execution. The Temporal Service logs a [warning after 10,240 Events](https://docs.temporal.io/workflow-execution/limits) and periodically logs additional warnings as new Events are added. The Workflow Execution is terminated when the Event History: * exceeds 51,200 Events. * contains more than 2000 Updates. * contains more than 10000 Signals. To avoid hitting these limits, you can use the [Continue-As-New](https://docs.temporal.io/workflow-execution/continue-as-new) feature to close the current Workflow Execution and create a new one. ### Event loop[​](https://docs.temporal.io/workflow-execution/event#event-loop "Direct link to Event loop") A Workflow Execution is made up of a sequence of [Events](https://docs.temporal.io/workflow-execution/event#event) called an [Event History](https://docs.temporal.io/workflow-execution/event#event-history) . Events are created by the Temporal Service in response to either Commands or actions requested by a Temporal Client (such as a request to spawn a Workflow Execution). ![Workflow Execution](https://docs.temporal.io/diagrams/workflow-execution-swim-lane-01.svg) Workflow Execution Time constraints[​](https://docs.temporal.io/workflow-execution/event#time-constraints "Direct link to Time constraints") -------------------------------------------------------------------------------------------------------------------------- **Is there a limit to how long Workflows can run?** No, there is no time constraint on how long a Workflow Execution can run. However, if your Workflow will perform many actions, or will receive many messages, it can run into [Event History limits](https://docs.temporal.io/workflow-execution/event#event-history-limits) . It can also hit [Workflow Versioning](https://docs.temporal.io/workflow-definition#workflow-versioning) and other backwards incompatibility problems. For these reasons, it can be a good idea to [Continue-As-New](https://docs.temporal.io/workflow-execution/continue-as-new) periodically. What is a Reset?[​](https://docs.temporal.io/workflow-execution/event#reset "Direct link to What is a Reset?") --------------------------------------------------------------------------------------------------------------- A Reset terminates a [Workflow Execution](https://docs.temporal.io/workflow-execution) and creates a new Workflow Execution with the same [Workflow Type](https://docs.temporal.io/workflow-definition#workflow-type) and [Workflow ID](https://docs.temporal.io/workflow-execution/workflowid-runid) . The [Event History](https://docs.temporal.io/workflow-execution/event#event-history) is copied from the original execution up to and including the reset point. The new execution continues from the reset point. Valid reset points are: `WorkflowTaskStarted`, `WorkflowTaskCompleted`, `WorkflowTaskTimedOut`, and `WorkflowTaskFailed`. Signals in the original history can be optionally copied to the new history, whether they appear after the reset point or not. What is a Side Effect?[​](https://docs.temporal.io/workflow-execution/event#side-effect "Direct link to What is a Side Effect?") --------------------------------------------------------------------------------------------------------------------------------- note Side Effects are included in the Go, Java, and PHP SDKs. They are not included in other SDKs. [Local Activities](https://docs.temporal.io/local-activity) fit the same use case and are slightly less resource intensive. A Side Effect is a way to execute a short, non-deterministic code snippet, such as generating a UUID, that executes the provided function once and records its result into the Workflow Execution Event History. A Side Effect does not re-execute upon replay, but instead returns the recorded result. Do not ever have a Side Effect that could fail, because failure could result in the Side Effect function executing more than once. If there is any chance that the code provided to the Side Effect could fail, use an Activity. * [What is an Event?](https://docs.temporal.io/workflow-execution/event#event) * [Activity Events](https://docs.temporal.io/workflow-execution/event#activity-events) * [What is an Event History?](https://docs.temporal.io/workflow-execution/event#event-history) * [Event History limits](https://docs.temporal.io/workflow-execution/event#event-history-limits) * [Event loop](https://docs.temporal.io/workflow-execution/event#event-loop) * [Time constraints](https://docs.temporal.io/workflow-execution/event#time-constraints) * [What is a Reset?](https://docs.temporal.io/workflow-execution/event#reset) * [What is a Side Effect?](https://docs.temporal.io/workflow-execution/event#side-effect) --- # External Storage | Temporal Platform Documentation [Skip to main content](https://docs.temporal.io/external-storage#__docusaurus_skipToContent_fallback) On this page Release, stability, and dependency info External Storage is in [Pre-Release](https://docs.temporal.io/evaluate/development-production-features/release-stages#pre-release) . APIs and configuration may change before the stable release. Join the [#large-payloads Slack channel](https://temporalio.slack.com/archives/C09VA2DE15Y) to provide feedback or ask for help. External Storage offloads payloads to an external store (such as Amazon S3) and passes a small reference token through the Event History instead. This is called the [claim check pattern](https://dataengineering.wiki/Concepts/Software+Engineering/Claim+Check+Pattern) . For SDK-specific usage guides, see: * [Go SDK: Large payload storage](https://docs.temporal.io/develop/go/data-handling/external-storage) * [Python SDK: Large payload storage](https://docs.temporal.io/develop/python/data-handling/external-storage) Why use External Storage[​](https://docs.temporal.io/external-storage#why-use-external-storage "Direct link to Why use External Storage") ------------------------------------------------------------------------------------------------------------------------------------------ The Temporal Service enforces a maximum per-payload size. The default and recommended limit is 2 MB. Self-hosted users can [configure this limit](https://docs.temporal.io/self-hosted-guide/defaults) , but it is fixed at 2 MB on Temporal Cloud. Payloads that exceed this limit fail the operation. Without External Storage, you must restructure your code to work around the limit, for example by splitting data across multiple Workflows. Even when individual payloads stay under the hard limit, payload data accumulates in Event History. Every Activity input and output is persisted, so Workflows that pass data through many Activities can see history size grow quickly. Large histories degrade Workflow Task latency. You may use [Continue-as-New](https://docs.temporal.io/workflow-execution/continue-as-new) to work around this problem, but that comes with other tradeoffs. External Storage addresses several common scenarios: * **Data processing pipelines.** Workflows that process documents, images, or other large blobs can exceed the per-payload limit. * **AI agent conversations.** Long conversation histories grow with each turn, and the cumulative size can degrade Workflow performance. * **Spiky data sizes.** Some Workflows handle data that is usually small but occasionally large. The Claim check pattern handles these spikes transparently, offloading only the payloads that exceed the size threshold. * **Migration to Temporal Cloud.** Self-hosted deployments may have higher configured payload limits. External Storage lets you migrate to Cloud without restructuring Workflows that exceed the 2 MB limit. * **Data governance.** While Temporal supports end-to-end client-side encryption, some organizations prefer to store payload data in infrastructure they control. Set the offload size threshold to zero to externalize all payloads regardless of size. For SDK-specific usage guides, see: * [Go SDK: Large payload storage](https://docs.temporal.io/develop/go/data-handling/external-storage) * [Python SDK: Large payload storage](https://docs.temporal.io/develop/python/data-handling/external-storage) How External Storage fits in the data conversion pipeline[​](https://docs.temporal.io/external-storage#data-pipeline "Direct link to How External Storage fits in the data conversion pipeline") ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- During [Data Conversion](https://docs.temporal.io/dataconversion) , External Storage sits at the end of the pipeline, after both the [Payload Converter](https://docs.temporal.io/payload-converter) and the [Payload Codec](https://docs.temporal.io/payload-codec) : ![The Flow of Data through a Data Converter](https://docs.temporal.io/diagrams/data-converter-flow-with-external-storage.svg)![The Flow of Data through a Data Converter](https://docs.temporal.io/diagrams/data-converter-flow-dark.svg) The Flow of Data through a Data Converter When a Temporal Client sends a payload that exceeds the configured size threshold, the storage driver uploads the payload to your external store and replaces it with a lightweight reference. Payloads below the threshold stay inline in the Event History. When the Temporal Service dispatches Tasks to the Worker, the process reverses. The Worker downloads the referenced payloads from external storage in parallel, then passes them back through the Payload Codec and Payload Converter to reconstruct the original data. The SDK parallelizes uploads and downloads to minimize latency. When a single Workflow Task involves multiple payloads that exceed the threshold, the SDK uploads or downloads all of them concurrently rather than one at a time. This allows external storage operations to scale well even when a Task carries many large payloads. Because External Storage runs after the Payload Codec, if you use an encryption codec, payloads are already encrypted before upload to your store. Storage drivers[​](https://docs.temporal.io/external-storage#storage-drivers "Direct link to Storage drivers") --------------------------------------------------------------------------------------------------------------- A storage driver connects External Storage to a backing store. Each driver provides two operations: * **Store**. Upload payloads and return a claim, which is a set of key-value pairs the driver uses to locate the payload later. * **Retrieve**. Download payloads using the claims that `store` produced. Temporal SDKs include built-in drivers for common storage systems like Amazon S3. You can configure multiple storage drivers and use a selector function to route payloads to different drivers based on size, type, or other criteria such as hot and cold storage tiers. ### Custom storage drivers[​](https://docs.temporal.io/external-storage#custom-storage-drivers "Direct link to Custom storage drivers") If the built-in drivers don't support your storage backend, you can implement a custom driver. For SDK-specific examples, see: * [Go SDK: Implement a custom storage driver](https://docs.temporal.io/develop/go/data-handling/external-storage#implement-a-custom-storage-driver) * [Python SDK: Implement a custom storage driver](https://docs.temporal.io/develop/python/data-handling/external-storage#implement-a-custom-storage-driver) Key configuration settings[​](https://docs.temporal.io/external-storage#key-configuration-settings "Direct link to Key configuration settings") ------------------------------------------------------------------------------------------------------------------------------------------------ Configure External Storage on the Data Converter. The key settings are: * **Size threshold**. The driver offloads payloads larger than this value, which defaults to 256 KiB. * **Drivers**. One or more storage driver implementations. * **Driver selector**. When using multiple drivers, you must provide a function that chooses which driver handles each payload. Lifecycle management for external storage[​](https://docs.temporal.io/external-storage#lifecycle "Direct link to Lifecycle management for external storage") ------------------------------------------------------------------------------------------------------------------------------------------------------------- Temporal does not automatically delete payloads from your external store. Payloads can also be orphaned if a request fails after the upload completes. We recommend you configure a lifecycle policy that both ensures these payloads are eventually cleaned up and provides a grace period for debugging and recovery. Your TTL must be long enough that payloads remain available for the entire lifetime of the Workflow plus its retention window: TTL > Maximum Workflow Run Timeout + Namespace Retention Period For example, if your longest-running Workflow has a Run Timeout of 14 days and your Namespace retention period is 30 days, configure your lifecycle rule to expire objects after at least 44 days. If your Workflows run indefinitely (no Run Timeout), there is no finite TTL that guarantees safety. Set a generous TTL based on your operational needs. Use [Continue-as-New](https://docs.temporal.io/workflow-execution/continue-as-new) for Workflows that need to run longer. The new run uploads fresh payloads, and the old run's payloads only need to survive through its retention period. * [Why use External Storage](https://docs.temporal.io/external-storage#why-use-external-storage) * [How External Storage fits in the data conversion pipeline](https://docs.temporal.io/external-storage#data-pipeline) * [Storage drivers](https://docs.temporal.io/external-storage#storage-drivers) * [Custom storage drivers](https://docs.temporal.io/external-storage#custom-storage-drivers) * [Key configuration settings](https://docs.temporal.io/external-storage#key-configuration-settings) * [Lifecycle management for external storage](https://docs.temporal.io/external-storage#lifecycle) --- # Worker processes - TypeScript SDK | Temporal Platform Documentation [Skip to main content](https://docs.temporal.io/develop/typescript/workers/run-worker-process#__docusaurus_skipToContent_fallback) On this page How to run Worker Processes[​](https://docs.temporal.io/develop/typescript/workers/run-worker-process#run-a-dev-worker "Direct link to How to run Worker Processes") --------------------------------------------------------------------------------------------------------------------------------------------------------------------- The [Worker Process](https://docs.temporal.io/workers#worker-process) is where Workflow Functions and Activity Functions are executed. * Each [Worker Entity](https://docs.temporal.io/workers#worker-entity) in the Worker Process must register the exact Workflow Types and Activity Types it may execute. * Each Worker Entity must also associate itself with exactly one [Task Queue](https://docs.temporal.io/task-queue) . * Each Worker Entity polling the same Task Queue must be registered with the same Workflow Types and Activity Types. A [Worker Entity](https://docs.temporal.io/workers#worker-entity) is the component within a Worker Process that listens to a specific Task Queue. Although multiple Worker Entities can be in a single Worker Process, a single Worker Entity Worker Process may be perfectly sufficient. For more information, see the [Worker tuning guide](https://docs.temporal.io/develop/worker-performance) . A Worker Entity contains a Workflow Worker and/or an Activity Worker, which makes progress on Workflow Executions and Activity Executions, respectively. How to run a Worker on Docker in TypeScript[​](https://docs.temporal.io/develop/typescript/workers/run-worker-process#run-a-worker-on-docker "Direct link to How to run a Worker on Docker in TypeScript") ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- note To improve worker startup time, we recommend preparing workflow bundles ahead-of-time. See our [productionsample](https://github.com/temporalio/samples-typescript/tree/main/production) for details. Workers based on the TypeScript SDK can be deployed and run as Docker containers. We recommend an LTS Node.js release such as 18, 20, 22, or 24. Both `amd64` and `arm64` architectures are supported. A glibc-based image is required; musl-based images are _not_ supported (see below). The easiest way to deploy a TypeScript SDK Worker on Docker is to start with the `node:20-bullseye` image. For example: FROM node:20-bullseye# For better cache utilization, copy package.json and lock file first and install the dependencies before copying the# rest of the application and building.COPY . /appWORKDIR /app# Alternatively, run npm ci, which installs only dependencies specified in the lock file and is generally faster.RUN npm install --only=production \ && npm run buildCMD ["npm", "start"] For smaller images and/or more secure deployments, it is also possible to use `-slim` Docker image variants (like `node:20-bullseye-slim`) or `distroless/nodejs` Docker images (like `gcr.io/distroless/nodejs20-debian11`) with the following caveats. ### Using `node:slim` images[​](https://docs.temporal.io/develop/typescript/workers/run-worker-process#using-nodeslim-images "Direct link to using-nodeslim-images") `node:slim` images do not contain some of the common packages found in regular images. This results in significantly smaller images. However, TypeScript SDK requires the presence of root TLS certificates (the `ca-certificates` package), which are not included in `slim` images. The `ca-certificates` package is required even when connecting to a local Temporal Server or when using a server connection config that doesn't explicitly use TLS. For this reason, the `ca-certificates` package must be installed during the construction of the Docker image. For example: FROM node:20-bullseye-slimRUN apt-get update \ && apt-get install -y ca-certificates \ && rm -rf /var/lib/apt/lists/*# ... same as with regular image Failure to install this dependency results in a `[TransportError: transport error]` runtime error, because the certificates cannot be verified. ### Using `distroless/nodejs` images[​](https://docs.temporal.io/develop/typescript/workers/run-worker-process#using-distrolessnodejs-images "Direct link to using-distrolessnodejs-images") `distroless/nodejs` images include only the files that are strictly required to execute `node`. This results in even smaller images (approximately half the size of `node:slim` images). It also significantly reduces the surface of potential security issues that could be exploited by a hacker in the resulting Docker images. It is generally possible and safe to execute TypeScript SDK Workers using `distroless/nodejs` images (unless your code itself requires dependencies that are not included in `distroless/nodejs`). However, some tools required for the build process (notably the `npm` command) are _not_ included in the `distroless/nodejs` image. This might result in various error messages during the Docker build. The recommended solution is to use a multi-step Dockerfile. For example: # -- BUILD STEP --FROM node:20-bullseye AS builderCOPY . /appWORKDIR /appRUN npm install --only=production \ && npm run build# -- RESULTING IMAGE --FROM gcr.io/distroless/nodejs20-debian11COPY --from=builder /app /appWORKDIR /appCMD ["node", "build/worker.js"] ### Properly configure Node.js memory in Docker[​](https://docs.temporal.io/develop/typescript/workers/run-worker-process#properly-configure-nodejs-memory-in-docker "Direct link to Properly configure Node.js memory in Docker") By default, `node` configures its maximum old-gen memory to 25% of the _physical memory_ of the machine on which it is executing, with a maximum of 4 GB. This is likely inappropriate when running Node.js in a Docker environment and can result in either underusage of available memory (`node` only uses a fraction of the memory allocated to the container) or overusage (`node` tries to use more memory than what is allocated to the container, which will eventually lead to the process being killed by the operating system). Therefore we recommended that you always explicitly set the `--max-old-space-size` `node` argument to approximately 80% of the maximum size (in megabytes) that you want to allocate the `node` process. You might need some experimentation and adjustment to find the most appropriate value based on your specific application. In practice, it is generally easier to provide this argument through the [`NODE_OPTIONS` environment variable](https://nodejs.org/api/cli.html#node_optionsoptions) . ### Do not use Alpine[​](https://docs.temporal.io/develop/typescript/workers/run-worker-process#do-not-use-alpine "Direct link to Do not use Alpine") Alpine replaces glibc with musl, which is incompatible with the Rust core of the TypeScript SDK. If you receive errors like the following, it's probably because you are using Alpine. Error: Error loading shared library ld-linux-x86-64.so.2: No such file or directory (needed by /opt/app/node_modules/@temporalio/core-bridge/index.node) Or like this: Error: Error relocating /opt/app/node_modules/@temporalio/core-bridge/index.node: __register_atfork: symbol not found How to run a Temporal Cloud Worker[​](https://docs.temporal.io/develop/typescript/workers/run-worker-process#run-a-temporal-cloud-worker "Direct link to How to run a Temporal Cloud Worker") ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- To run a Worker that uses [Temporal Cloud](https://docs.temporal.io/cloud) , you need to provide additional connection and client options that include the following: * An address that includes your [Cloud Namespace Name](https://docs.temporal.io/namespaces) and a port number: `..tmprl.cloud:`. * mTLS CA certificate. * mTLS private key. For more information about managing and generating client certificates for Temporal Cloud, see [How to manage certificates in Temporal Cloud](https://docs.temporal.io/cloud/certificates) . For more information about configuring TLS to secure inter- and intra-network communication for a Temporal Service, see [Temporal Customization Samples](https://github.com/temporalio/samples-server) . ### How to register types[​](https://docs.temporal.io/develop/typescript/workers/run-worker-process#register-types "Direct link to How to register types") All Workers listening to the same Task Queue name must be registered to handle the exact same Workflows Types and Activity Types. If a Worker polls a Task for a Workflow Type or Activity Type it does not know about, it fails that Task. However, the failure of the Task does not cause the associated Workflow Execution to fail. In development, use [`workflowsPath`](https://typescript.temporal.io/api/interfaces/worker.WorkerOptions/#workflowspath) : [snippets/src/worker.ts](https://github.com/temporalio/samples-typescript/blob/main/snippets/src/worker.ts) import { Worker } from '@temporalio/worker';import * as activities from './activities';async function run() { const worker = await Worker.create({ workflowsPath: require.resolve('./workflows'), taskQueue: 'snippets', activities, }); await worker.run();} In this snippet, the Worker bundles the Workflow code at runtime. In production, you can improve your Worker's startup time by bundling in advance: as part of your production build, call `bundleWorkflowCode`: [production/src/scripts/build-workflow-bundle.ts](https://github.com/temporalio/samples-typescript/blob/main/production/src/scripts/build-workflow-bundle.ts) import { bundleWorkflowCode } from '@temporalio/worker';import { writeFile } from 'fs/promises';import path from 'path';async function bundle() { const { code } = await bundleWorkflowCode({ workflowsPath: require.resolve('../workflows'), }); const codePath = path.join(__dirname, '../../workflow-bundle.js'); await writeFile(codePath, code); console.log(`Bundle written to ${codePath}`);} Then the bundle can be passed to the Worker: [production/src/worker.ts](https://github.com/temporalio/samples-typescript/blob/main/production/src/worker.ts) const workflowOption = () => process.env.NODE_ENV === 'production' ? { workflowBundle: { codePath: require.resolve('../workflow-bundle.js'), }, } : { workflowsPath: require.resolve('./workflows') };async function run() { const worker = await Worker.create({ ...workflowOption(), activities, taskQueue: 'production-sample', }); await worker.run();} How to shut down a Worker and track its state[​](https://docs.temporal.io/develop/typescript/workers/run-worker-process#shut-down-a-worker "Direct link to How to shut down a Worker and track its state") ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Workers shut down if they receive any of the Signals enumerated in [shutdownSignals](https://typescript.temporal.io/api/interfaces/worker.RuntimeOptions#shutdownsignals) : `'SIGINT'`, `'SIGTERM'`, `'SIGQUIT'`, and `'SIGUSR2'`. In development, we shut down Workers with `Ctrl+C` (`SIGINT`) or [nodemon](https://github.com/temporalio/samples-typescript/blob/c37bae3ea235d1b6956fcbe805478aa46af973ce/hello-world/package.json#L10) (`SIGUSR2`). In production, you usually want to give Workers time to finish any in-progress Activities by setting [shutdownGraceTime](https://typescript.temporal.io/api/interfaces/worker.WorkerOptions#shutdowngracetime) . As soon as a Worker receives a shutdown Signal or request, the Worker stops polling for new Tasks and allows in-flight Tasks to complete until `shutdownGraceTime` is reached. Any Activities that are still running at that time will stop running and will be rescheduled by Temporal Server when an Activity timeout occurs. If you must guarantee that the Worker eventually shuts down, you can set [shutdownForceTime](https://typescript.temporal.io/api/interfaces/worker.WorkerOptions#shutdownforcetime) . You might want to programmatically shut down Workers (with [Worker.shutdown()](https://typescript.temporal.io/api/classes/worker.Worker#shutdown) ) in integration tests or when automating a fleet of Workers. ### Worker states[​](https://docs.temporal.io/develop/typescript/workers/run-worker-process#worker-states "Direct link to Worker states") At any time, you can Query Worker state with [Worker.getState()](https://typescript.temporal.io/api/classes/worker.Worker#getstate) . A Worker is always in one of seven states: * `INITIALIZED`: The initial state of the Worker after calling [Worker.create()](https://typescript.temporal.io/api/classes/worker.Worker#create) and successfully connecting to the server. * `RUNNING`: [Worker.run()](https://typescript.temporal.io/api/classes/worker.Worker#run) was called and the Worker is polling Task Queues. * `FAILED`: The Worker encountered an unrecoverable error; `Worker.run()` should reject with the error. * The last four states are related to the Worker shutdown process: * `STOPPING`: The Worker received a shutdown Signal or `Worker.shutdown()` was called. The Worker will forcefully shut down after `shutdownGraceTime` expires. * `DRAINING`: All Workflow Tasks have been drained; waiting for Activities and cached Workflows eviction. * `DRAINED`: All Activities and Workflows have completed; ready to shut down. * `STOPPED`: Shutdown complete; `worker.run()` resolves. If you need more visibility into internal Worker state, see the [Worker class](https://typescript.temporal.io/api/classes/worker.Worker) in the API reference. * [How to run Worker Processes](https://docs.temporal.io/develop/typescript/workers/run-worker-process#run-a-dev-worker) * [How to run a Worker on Docker in TypeScript](https://docs.temporal.io/develop/typescript/workers/run-worker-process#run-a-worker-on-docker) * [Using `node:slim` images](https://docs.temporal.io/develop/typescript/workers/run-worker-process#using-nodeslim-images) * [Using `distroless/nodejs` images](https://docs.temporal.io/develop/typescript/workers/run-worker-process#using-distrolessnodejs-images) * [Properly configure Node.js memory in Docker](https://docs.temporal.io/develop/typescript/workers/run-worker-process#properly-configure-nodejs-memory-in-docker) * [Do not use Alpine](https://docs.temporal.io/develop/typescript/workers/run-worker-process#do-not-use-alpine) * [How to run a Temporal Cloud Worker](https://docs.temporal.io/develop/typescript/workers/run-worker-process#run-a-temporal-cloud-worker) * [How to register types](https://docs.temporal.io/develop/typescript/workers/run-worker-process#register-types) * [How to shut down a Worker and track its state](https://docs.temporal.io/develop/typescript/workers/run-worker-process#shut-down-a-worker) * [Worker states](https://docs.temporal.io/develop/typescript/workers/run-worker-process#worker-states) --- # Benign exceptions - TypeScript SDK | Temporal Platform Documentation [Skip to main content](https://docs.temporal.io/develop/typescript/activities/benign-exceptions#__docusaurus_skipToContent_fallback) When Activities throw errors that are expected or not severe, they can create noise in your logs, metrics, and OpenTelemetry traces, making it harder to identify real issues. By marking these errors as benign, you can exclude them from your observability data while still handling them in your Workflow logic. To mark an error as benign, set the `category` field to `ApplicationFailureCategory.BENIGN` when creating an [`ApplicationFailure`](https://typescript.temporal.io/api/classes/common.ApplicationFailure) . Benign errors: * Have Activity failure logs downgraded to DEBUG level * Do not emit Activity failure metrics * Do not set the OpenTelemetry failure status to ERROR import { ApplicationFailure, ApplicationFailureCategory,} from '@temporalio/common';export async function myActivity(): Promise { try { return await callExternalService(); } catch (err) { const message = err instanceof Error ? err.message : String(err); throw ApplicationFailure.create({ message, // Mark this error as benign since it's expected category: ApplicationFailureCategory.BENIGN, }); }} Use benign exceptions for Activity errors that occur regularly as part of normal operations, such as polling an external service that isn't ready yet, or handling expected transient failures that will be retried. --- # Client - Ruby SDK | Temporal Platform Documentation [Skip to main content](https://docs.temporal.io/develop/ruby/client#__docusaurus_skipToContent_fallback) On this page ![.NET SDK Banner](https://docs.temporal.io/assets/images/banner-ruby-temporal-be833f13b8e3655d7a8d4e50119b7da2.png) Temporal Client[​](https://docs.temporal.io/develop/ruby/client#temporal-client "Direct link to Temporal Client") ------------------------------------------------------------------------------------------------------------------ * [Temporal Client](https://docs.temporal.io/develop/ruby/client/temporal-client) * [Temporal Client](https://docs.temporal.io/develop/ruby/client#temporal-client) --- # Integrations - Ruby SDK | Temporal Platform Documentation [Skip to main content](https://docs.temporal.io/develop/ruby/integrations#__docusaurus_skipToContent_fallback) On this page ![Ruby SDK Banner](https://docs.temporal.io/assets/images/banner-ruby-temporal-be833f13b8e3655d7a8d4e50119b7da2.png) Integrations[​](https://docs.temporal.io/develop/ruby/integrations#integrations "Direct link to Integrations") --------------------------------------------------------------------------------------------------------------- * [Rails integration](https://docs.temporal.io/develop/ruby/integrations/rails-integration) * [Integrations](https://docs.temporal.io/develop/ruby/integrations#integrations) --- # Activity execution - TypeScript SDK | Temporal Platform Documentation [Skip to main content](https://docs.temporal.io/develop/typescript/activities/execution#__docusaurus_skipToContent_fallback) On this page How to start an Activity Execution[​](https://docs.temporal.io/develop/typescript/activities/execution#activity-execution "Direct link to How to start an Activity Execution") ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Calls to spawn [Activity Executions](https://docs.temporal.io/activity-execution) are written within a [Workflow Definition](https://docs.temporal.io/workflow-definition) . In TypeScript, you never call an Activity function directly. Instead, you pass in the _types_ of your Activities and Activity options to the `proxyActivities` function. This will give you an _Activity Handle_, a type-safe proxy object with the same function names and signatures as your real activities. From the Activity Handle, you can call your Activities as if they were normal async functions. import { proxyActivities } from '@temporalio/workflow';// Only import the activity types, not the functions themselvesimport type * as activities from './activities';// Retrieve the Activity Handle by passing in the Activity types and optionsconst activityHandle = proxyActivities({ startToCloseTimeout: '1 minute',});// Deconstruct the individual Activity functions from the Activity Handleconst { greet } = activityHandle;// A workflow that calls an activityexport async function example(name: string): Promise { return await greet(name);} When you call a proxied function, the Workflow does not execute the Activity code directly. Instead, it schedules an Activity Task. After the Activity Task is scheduled, it becomes available for a Worker to pick up and execute. This results in the set of three [Activity Task](https://docs.temporal.io/tasks#activity-task) related Events: [ActivityTaskScheduled](https://docs.temporal.io/references/events#activitytaskscheduled) , [ActivityTaskStarted](https://docs.temporal.io/references/events#activitytaskstarted) , and [ActivityTaskCompleted](https://docs.temporal.io/references/events#activitytaskcompleted) in your Workflow Execution Event History. The Worker may run many Activity executions at the same time, all using the same Activity function code. Temporal can also retry an Activity if it fails or times out. For this reason, you should write Activities to be [idempotent](https://docs.temporal.io/activity-definition#idempotency) : calling them multiple times with the same input should have the same effect as calling them once. Every Activity call you make is recorded in the Workflow’s execution history, including the parameters you pass in and the value that comes back. This history is what allows Temporal to recover a Workflow after a failure. Because the entire history must be stored and replayed, avoid passing large objects as Activity inputs or return values. Keeping payloads small will help your Workflows replay and recover efficiently. ::: How to set the required Activity Timeouts[​](https://docs.temporal.io/develop/typescript/activities/execution#required-timeout "Direct link to How to set the required Activity Timeouts") ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Activity Execution semantics rely on several parameters. The only required value that needs to be set is either a [Schedule-To-Close Timeout](https://docs.temporal.io/encyclopedia/detecting-activity-failures#schedule-to-close-timeout) or a [Start-To-Close Timeout](https://docs.temporal.io/encyclopedia/detecting-activity-failures#start-to-close-timeout) . These values are set in the Activity Options. How to get the results of an Activity Execution[​](https://docs.temporal.io/develop/typescript/activities/execution#get-activity-results "Direct link to How to get the results of an Activity Execution") ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- The call to spawn an [Activity Execution](https://docs.temporal.io/activity-execution) generates the [ScheduleActivityTask](https://docs.temporal.io/references/commands#scheduleactivitytask) Command and provides the Workflow with an Awaitable. Workflow Executions can either block progress until the result is available through the Awaitable or continue progressing, making use of the result when it becomes available. Since Activities are referenced by their string name, you can reference them dynamically to get the result of an Activity Execution. export async function DynamicWorkflow(activityName, ...args) { const acts = proxyActivities(/* activityOptions */); // these are equivalent await acts.activity1(); await acts['activity1'](); let result = await acts[activityName](...args); return result;} The `proxyActivities()` returns an object that calls the Activities in the function. `acts[activityName]()` references the Activity using the Activity name, then it returns the results. * [How to start an Activity Execution](https://docs.temporal.io/develop/typescript/activities/execution#activity-execution) * [How to set the required Activity Timeouts](https://docs.temporal.io/develop/typescript/activities/execution#required-timeout) * [How to get the results of an Activity Execution](https://docs.temporal.io/develop/typescript/activities/execution#get-activity-results) --- # Error Handling - Temporal Nexus | Temporal Platform Documentation [Skip to main content](https://docs.temporal.io/nexus/error-handling#__docusaurus_skipToContent_fallback) On this page SUPPORT, STABILITY, and DEPENDENCY INFO Temporal Nexus is now [Generally Available](https://docs.temporal.io/evaluate/development-production-features/release-stages#general-availability) for [Temporal Cloud](https://docs.temporal.io/cloud/nexus) and [self-hosted deployments](https://docs.temporal.io/production-deployment/self-hosted-guide/nexus) . Nexus Operations can return errors for a caller Workflow to handle. Errors from an asynchronous Operation's underlying Workflow propagate back to the caller. Errors in Nexus handlers[​](https://docs.temporal.io/nexus/error-handling#errors-in-nexus-handlers "Direct link to Errors in Nexus handlers") ---------------------------------------------------------------------------------------------------------------------------------------------- Nexus handlers may return [different error types](https://docs.temporal.io/references/failures#nexus-errors) . By default, handler errors are retryable unless they are: * [Application Failures](https://docs.temporal.io/references/failures#nexus-errors) explicitly marked as non-retryable. * [Nexus Operation errors](https://docs.temporal.io/references/failures#nexus-errors) that resolve an Operation as failed or canceled. * [Non-retryable Nexus errors](https://docs.temporal.io/references/failures#non-retryable-nexus-errors) . When the caller's Nexus Machinery receives an error: * **Non-retryable** - A [NexusOperationFailed](https://docs.temporal.io/references/events#nexusoperationfailed) event is added to the caller's Workflow History. * **Retryable** - The Nexus Machinery [automatically retries](https://docs.temporal.io/nexus/operations#automatic-retries) . These errors surface in [Pending Operations](https://docs.temporal.io/nexus/execution-debugging/#pending-operations) . tip Return a [specific Nexus error type](https://docs.temporal.io/references/failures#nexus-errors) to avoid infinite retries. See [errors in Nexus Operations](https://docs.temporal.io/references/failures#errors-in-nexus-operations) for additional details. Nexus error handling in caller Workflows[​](https://docs.temporal.io/nexus/error-handling#nexus-error-handling-in-caller-workflows "Direct link to Nexus error handling in caller Workflows") ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- When a Nexus Operation fails, the caller receives a Nexus Operation Failure containing the operation name, token, and failure reason. The cause field indicates the type of error (for example, Application Error or Canceled Error). RESOURCES * [Errors in Nexus Operations](https://docs.temporal.io/references/failures#errors-in-nexus-operations) * [Nexus Errors](https://docs.temporal.io/references/failures#nexus-errors) * [Nexus Operation Failures](https://docs.temporal.io/references/failures#nexus-operation-failure) * [Errors in Nexus handlers](https://docs.temporal.io/nexus/error-handling#errors-in-nexus-handlers) * [Nexus error handling in caller Workflows](https://docs.temporal.io/nexus/error-handling#nexus-error-handling-in-caller-workflows) --- # Execution Debugging - Temporal Nexus | Temporal Platform Documentation [Skip to main content](https://docs.temporal.io/nexus/execution-debugging#__docusaurus_skipToContent_fallback) On this page SUPPORT, STABILITY, and DEPENDENCY INFO Temporal Nexus is now [Generally Available](https://docs.temporal.io/evaluate/development-production-features/release-stages#general-availability) for [Temporal Cloud](https://docs.temporal.io/cloud/nexus) and [self-hosted deployments](https://docs.temporal.io/production-deployment/self-hosted-guide/nexus) . Nexus supports end-to-end execution debugging across caller Workflows, Nexus Operations, and handler Workflows - even across [multi-level calls](https://docs.temporal.io/nexus#multi-level-calls) spanning multiple Namespaces. Bi-directional linking[​](https://docs.temporal.io/nexus/execution-debugging#bi-directional-linking "Direct link to Bi-directional linking") --------------------------------------------------------------------------------------------------------------------------------------------- Bidirectional links connect Nexus Operation events in the caller's Workflow History to corresponding events in the handler's Workflow History. They are automatically wired by SDK builder functions like New-Workflow-Run-Operation, enabling click-through navigation across Namespaces, regions, and clouds in the Temporal UI. ![Bi-directional linking](https://docs.temporal.io/img/cloud/nexus/nexus-bi-directional-linking.png) Bi-directional linking * **Forward**: From a caller's Nexus Operation event to the handler's Workflow. * **Backward**: From the handler's Workflow back to the caller's Nexus Operation event. Pending Operations[​](https://docs.temporal.io/nexus/execution-debugging#pending-operations "Direct link to Pending Operations") --------------------------------------------------------------------------------------------------------------------------------- Pending Nexus Operations are displayed in the UI on the Workflow details page and can be listed from the CLI using the `temporal workflow describe` command. From the UI: ![Pending Operations](https://docs.temporal.io/img/cloud/nexus/pending-nexus-operations.png) Pending Operations From the CLI: temporal workflow describePending Nexus Operations: 1 Endpoint myendpoint Service my-hello-service Operation echo OperationToken State BackingOff Attempt 6 ScheduleToCloseTimeout 0s NextAttemptScheduleTime 20 seconds from now LastAttemptCompleteTime 11 seconds ago LastAttemptFailure {"message":"handler error (INTERNAL): internal error","applicationFailureInfo":{}} [Retryable errors](https://docs.temporal.io/nexus/error-handling#errors-in-nexus-handlers) surface in the Pending Operation. Non-retryable errors resolve the Operation with a [Failed](https://docs.temporal.io/references/events#nexusoperationfailed) , [TimedOut](https://docs.temporal.io/references/events#nexusoperationtimedout) , or [Canceled](https://docs.temporal.io/references/events#nexusoperationcanceled) event. Pending Callbacks[​](https://docs.temporal.io/nexus/execution-debugging#pending-callbacks "Direct link to Pending Callbacks") ------------------------------------------------------------------------------------------------------------------------------ Nexus completion callbacks are sent from the handler's Namespace to the caller's Namespace for asynchronous Operations. These can be viewed in the UI or from the CLI using the `temporal workflow describe` command. From the UI: ![Pending Callbacks](https://docs.temporal.io/img/cloud/nexus/nexus-callback.png) Pending Callbacks From the CLI: temporal workflow describeCallbacks: 1 URL https://nexus.phil-caller-Namespace.a2dd6.cluster.tmprl.cloud:7243/Namespaces/phil-caller-Namespace.a2dd6/nexus/callback Trigger WorkflowClosed State Succeeded Attempt 1 RegistrationTime 32 minutes ago Tracing[​](https://docs.temporal.io/nexus/execution-debugging#tracing "Direct link to Tracing") ------------------------------------------------------------------------------------------------ Temporal integrates with [OpenTelemetry](https://opentelemetry.io/) and [OpenTracing](https://opentracing.io/) to visualize call graphs across Activities, Nexus Operations, and Child Workflows. Enable tracing by installing an interceptor on the Client or Worker: * [Go SDK](https://github.com/temporalio/samples-go/tree/main/opentelemetry) * [Java SDK](https://github.com/temporalio/samples-java/tree/main/core/src/main/java/io/temporal/samples/tracing) * [Python SDK](https://github.com/temporalio/samples-python/tree/main/open_telemetry) * [TypeScript](https://github.com/temporalio/samples-typescript/tree/main/interceptors-opentelemetry) * [.NET SDK](https://github.com/temporalio/samples-dotnet/tree/main/src/OpenTelemetry) * [Bi-directional linking](https://docs.temporal.io/nexus/execution-debugging#bi-directional-linking) * [Pending Operations](https://docs.temporal.io/nexus/execution-debugging#pending-operations) * [Pending Callbacks](https://docs.temporal.io/nexus/execution-debugging#pending-callbacks) * [Tracing](https://docs.temporal.io/nexus/execution-debugging#tracing) --- # Activity basics - TypeScript SDK | Temporal Platform Documentation [Skip to main content](https://docs.temporal.io/develop/typescript/activities/basics#__docusaurus_skipToContent_fallback) On this page How to develop an Activity[​](https://docs.temporal.io/develop/typescript/activities/basics#develop-activities "Direct link to How to develop an Activity") ------------------------------------------------------------------------------------------------------------------------------------------------------------ One of the primary things that Workflows do is orchestrate the execution of Activities. An Activity is a normal function or method execution that's intended to execute a single, well-defined action (either short or long-running), such as querying a database, calling a third-party API, or transcoding a media file. An Activity can interact with world outside the Temporal Platform or use a Temporal Client to interact with a Temporal Service. For the Workflow to be able to execute the Activity, we must define the [Activity Definition](https://docs.temporal.io/activity-definition) . * Activities execute in the standard Node.js environment. * Activities cannot be in the same file as Workflows and must be separately registered. * Activities may be retried repeatedly, so you may need to use idempotency keys for critical side effects. Activities are _just functions_. The following is an Activity that accepts a string parameter and returns a string. [snippets/src/activities.ts](https://github.com/temporalio/samples-typescript/blob/main/snippets/src/activities.ts) export async function greet(name: string): Promise { return `👋 Hello, ${name}!`;} How to develop Activity Parameters[​](https://docs.temporal.io/develop/typescript/activities/basics#activity-parameters "Direct link to How to develop Activity Parameters") ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------- There is no explicit limit to the total number of parameters that an [Activity Definition](https://docs.temporal.io/activity-definition) may support. However, there is a limit to the total size of the data that ends up encoded into a gRPC message Payload. A single argument is limited to a maximum size of 2 MB. And the total size of a gRPC message, which includes all the arguments, is limited to a maximum of 4 MB. Also, keep in mind that all Payload data is recorded in the [Workflow Execution Event History](https://docs.temporal.io/workflow-execution/event#event-history) and large Event Histories can affect Worker performance. This is because the entire Event History could be transferred to a Worker Process with a [Workflow Task](https://docs.temporal.io/tasks#workflow-task) . Some SDKs require that you pass context objects, others do not. When it comes to your application data—that is, data that is serialized and encoded into a Payload—we recommend that you use a single object as an argument that wraps the application data passed to Activities. This is so that you can change what data is passed to the Activity without breaking a function or method signature. This Activity takes a single `name` parameter of type `string`. [snippets/src/activities.ts](https://github.com/temporalio/samples-typescript/blob/main/snippets/src/activities.ts) export async function greet(name: string): Promise { return `👋 Hello, ${name}!`;} How to define Activity return values[​](https://docs.temporal.io/develop/typescript/activities/basics#activity-return-values "Direct link to How to define Activity return values") ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ All data returned from an Activity must be serializable. Activity return values are subject to payload size limits in Temporal. The default payload size limit is 2MB, and there is a hard limit of 4MB for any gRPC message size in the Event History transaction ([see Cloud limits here](https://docs.temporal.io/cloud/limits#per-message-grpc-limit) ). Keep in mind that all return values are recorded in a [Workflow Execution Event History](https://docs.temporal.io/workflow-execution/event#event-history) . In TypeScript, the return value is always a Promise. In the following example, `Promise` is the return value. export async function greet(name: string): Promise { return `👋 Hello, ${name}!`;} How to customize your Activity Type[​](https://docs.temporal.io/develop/typescript/activities/basics#activity-type "Direct link to How to customize your Activity Type") ------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Activities have a Type that are referred to as the Activity name. The following examples demonstrate how to set a custom name for your Activity Type. You can customize the name of the Activity when you register it with the Worker. In the following example, the Activity Name is `activityFoo`. [snippets/src/worker-activity-type-custom.ts](https://github.com/temporalio/samples-typescript/blob/main/snippets/src/worker-activity-type-custom.ts) import { Worker } from '@temporalio/worker';import { greet } from './activities';async function run() { const worker = await Worker.create({ workflowsPath: require.resolve('./workflows'), taskQueue: 'snippets', activities: { activityFoo: greet, }, }); await worker.run();} Important design patterns for Activities[​](https://docs.temporal.io/develop/typescript/activities/basics#activity-design-patterns "Direct link to Important design patterns for Activities") ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- The following are some important (and frequently requested) patterns for using our Activities APIs. These patterns address common needs and use cases. ### Share dependencies in Activity functions (dependency injection)[​](https://docs.temporal.io/develop/typescript/activities/basics#share-dependencies-in-activity-functions-dependency-injection "Direct link to Share dependencies in Activity functions (dependency injection)") Because Activities are "just functions," you can also create functions that create Activities. This is a helpful pattern for using closures to do the following: * Store expensive dependencies for sharing, such as database connections. * Inject secret keys (such as environment variables) from the Worker to the Activity. [activities-dependency-injection/src/activities.ts](https://github.com/temporalio/samples-typescript/blob/main/activities-dependency-injection/src/activities.ts) export interface DB { get(key: string): Promise;}export const createActivities = (db: DB) => ({ async greet(msg: string): Promise { const name = await db.get('name'); // simulate read from db return `${msg}: ${name}`; }, async greet_es(mensaje: string): Promise { const name = await db.get('name'); // simulate read from db return `${mensaje}: ${name}`; },}); See full example When you register these in the Worker, pass your shared dependencies accordingly: import { createActivities } from './activities';async function run() { // Mock DB connection initialization in Worker const db = { async get(_key: string) { return 'Temporal'; }, }; const worker = await Worker.create({ taskQueue: 'dependency-injection', workflowsPath: require.resolve('./workflows'), activities: createActivities(db), }); await worker.run();}run().catch((err) => { console.error(err); process.exit(1);}); Because Activities are always referenced by name, inside the Workflow they can be proxied as normal, although the types need some adjustment: [activities-dependency-injection/src/workflows.ts](https://github.com/temporalio/samples-typescript/blob/main/activities-dependency-injection/src/workflows.ts) import type { createActivities } from './activities';// Note usage of ReturnType<> generic since createActivities is a factory functionconst { greet, greet_es } = proxyActivities>({ startToCloseTimeout: '30 seconds',}); ### Import multiple Activities simultaneously[​](https://docs.temporal.io/develop/typescript/activities/basics#import-multiple-activities-simultaneously "Direct link to Import multiple Activities simultaneously") You can proxy multiple Activities from the same `proxyActivities` call if you want them to share the same timeouts, retries, and options: export async function Workflow(name: string): Promise { // destructuring multiple activities with the same options const { act1, act2, act3 } = proxyActivities(); /* activityOptions */ await act1(); await Promise.all([act2, act3]);} ### Dynamically reference Activities[​](https://docs.temporal.io/develop/typescript/activities/basics#dynamically-reference-activities "Direct link to Dynamically reference Activities") Because Activities are referenced only by their string names, you can reference them dynamically if needed: export async function DynamicWorkflow(activityName, ...args) { const acts = proxyActivities(/* activityOptions */); // these are equivalent await acts.activity1(); await acts['activity1'](); // dynamic reference to activities using activityName let result = await acts[activityName](...args);} Type safety is still supported here, but we encourage you to validate and handle mismatches in Activity names. An invalid Activity name leads to a `NotFoundError` with a message that looks like this: ApplicationFailure: Activity function actC is not registered on this Worker, available activities: ["actA", "actB"] * [How to develop an Activity](https://docs.temporal.io/develop/typescript/activities/basics#develop-activities) * [How to develop Activity Parameters](https://docs.temporal.io/develop/typescript/activities/basics#activity-parameters) * [How to define Activity return values](https://docs.temporal.io/develop/typescript/activities/basics#activity-return-values) * [How to customize your Activity Type](https://docs.temporal.io/develop/typescript/activities/basics#activity-type) * [Important design patterns for Activities](https://docs.temporal.io/develop/typescript/activities/basics#activity-design-patterns) * [Share dependencies in Activity functions (dependency injection)](https://docs.temporal.io/develop/typescript/activities/basics#share-dependencies-in-activity-functions-dependency-injection) * [Import multiple Activities simultaneously](https://docs.temporal.io/develop/typescript/activities/basics#import-multiple-activities-simultaneously) * [Dynamically reference Activities](https://docs.temporal.io/develop/typescript/activities/basics#dynamically-reference-activities) --- # Activity Timeouts - Ruby SDK | Temporal Platform Documentation [Skip to main content](https://docs.temporal.io/develop/ruby/activities/timeouts#__docusaurus_skipToContent_fallback) On this page Activity timeouts[​](https://docs.temporal.io/develop/ruby/activities/timeouts#activity-timeouts "Direct link to Activity timeouts") ------------------------------------------------------------------------------------------------------------------------------------- Each Activity Timeout controls a different aspect of how long an Activity Execution can take: * **[Schedule-To-Close Timeout](https://docs.temporal.io/encyclopedia/detecting-activity-failures#schedule-to-close-timeout) ** * **[Start-To-Close Timeout](https://docs.temporal.io/encyclopedia/detecting-activity-failures#start-to-close-timeout) ** * **[Schedule-To-Start Timeout](https://docs.temporal.io/encyclopedia/detecting-activity-failures#schedule-to-start-timeout) ** At least one of `start_to_close_timeout` or `schedule_to_close_timeout` is required. Temporalio::Workflow.execute_activity( MyActivity, { greeting: 'Hello', name: }, start_to_close_timeout: 5 * 60) ### Activity Retry Policy[​](https://docs.temporal.io/develop/ruby/activities/timeouts#activity-retries "Direct link to Activity Retry Policy") By default, Activities use a system Retry Policy. You can override it by specifying a custom Retry Policy. To create an Activity Retry Policy in Ruby, set the `retry_policy` parameter when executing an activity. Temporalio::Workflow.execute_activity( MyActivity, { greeting: 'Hello', name: }, start_to_close_timeout: 5 * 60, retry_policy: Temporalio::RetryPolicy.new(max_interval: 10)) ### Override the retry interval with `next_retry_delay`[​](https://docs.temporal.io/develop/ruby/activities/timeouts#next-retry-delay "Direct link to next-retry-delay") If you raise an application-level error, you can override the Retry Policy's delay by specifying a new delay. raise Temporalio::Error::ApplicationError.new( 'Some error', type: 'SomeErrorType', next_retry_delay: 3 * Temporalio::Activity::Context.current.info.attempt) Heartbeat an Activity[​](https://docs.temporal.io/develop/ruby/activities/timeouts#activity-heartbeats "Direct link to Heartbeat an Activity") ----------------------------------------------------------------------------------------------------------------------------------------------- A Heartbeat is a periodic signal from the Worker to the Temporal Service indicating the Activity is still alive and making progress. * Heartbeats are used to detect Worker failure. * Cancellations are delivered via Heartbeats. * Heartbeats may contain custom progress details. class MyActivity < Temporalio::Activity::Definition def execute # This is a naive loop simulating work, but similar heartbeat logic # applies to other scenarios as well loop do # Send heartbeat Temporalio::Activity::Context.current.heartbeat # Sleep before heartbeating again sleep(3) end endend ### Heartbeat Timeout[​](https://docs.temporal.io/develop/ruby/activities/timeouts#heartbeat-timeout "Direct link to Heartbeat Timeout") The Heartbeat Timeout sets the maximum duration between Heartbeats before the Temporal Service considers the Activity failed. Temporalio::Workflow.execute_activity( MyActivity, { greeting: 'Hello', name: }, start_to_close_timeout: 5 * 60, heartbeat_timeout: 5) * [Activity timeouts](https://docs.temporal.io/develop/ruby/activities/timeouts#activity-timeouts) * [Activity Retry Policy](https://docs.temporal.io/develop/ruby/activities/timeouts#activity-retries) * [Override the retry interval with `next_retry_delay`](https://docs.temporal.io/develop/ruby/activities/timeouts#next-retry-delay) * [Heartbeat an Activity](https://docs.temporal.io/develop/ruby/activities/timeouts#activity-heartbeats) * [Heartbeat Timeout](https://docs.temporal.io/develop/ruby/activities/timeouts#heartbeat-timeout) --- # Debugging - TypeScript SDK | Temporal Platform Documentation [Skip to main content](https://docs.temporal.io/develop/typescript/best-practices/debugging#__docusaurus_skipToContent_fallback) On this page The Debugging section of the Temporal TypeScript SDK developer's guide covers tools for debugging and how to troubleshoot common issues. How to debug in a development environment[​](https://docs.temporal.io/develop/typescript/best-practices/debugging#debug-in-a-development-environment "Direct link to How to debug in a development environment") ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- In addition to the normal development tools of logging and a debugger, you can also see what's happening in your Workflow by using the [Web UI](https://docs.temporal.io/web-ui) or [Temporal CLI](https://docs.temporal.io/cli) . How to debug in a production environment[​](https://docs.temporal.io/develop/typescript/best-practices/debugging#debug-in-a-production-environment "Direct link to How to debug in a production environment") -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- You can debug production Workflows using: * [Web UI](https://docs.temporal.io/web-ui) * [Temporal CLI](https://docs.temporal.io/cli) * [Replay](https://docs.temporal.io/develop/typescript/best-practices/testing-suite#replay) * [Tracing](https://docs.temporal.io/develop/typescript/platform/observability#tracing) * [Logging](https://docs.temporal.io/develop/typescript/platform/observability#logging) You can debug and tune Worker performance with metrics and the [Worker performance guide](https://docs.temporal.io/develop/worker-performance) . For information on setting up SDK metrics, see [Metrics](https://docs.temporal.io/develop/typescript/platform/observability#metrics) in the Observability section of the TypeScript SDK developer's guide. Debug Server performance with [Cloud metrics](https://docs.temporal.io/cloud/metrics/) or [self-hosted Server metrics](https://docs.temporal.io/self-hosted-guide/production-checklist#scaling-and-metrics) . How to troubleshoot common issues in the TypeScript SDK[​](https://docs.temporal.io/develop/typescript/best-practices/debugging#troubleshoot-common-issues "Direct link to How to troubleshoot common issues in the TypeScript SDK") ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- ### Two locations to watch[​](https://docs.temporal.io/develop/typescript/best-practices/debugging#two-locations-to-watch "Direct link to Two locations to watch") * Workflow Errors are reflected in Temporal Web. * Worker errors and logs are reflected in the terminal. If something isn't behaving the way you expect, make sure to check both locations for helpful error messages. ### Stale Workflows[​](https://docs.temporal.io/develop/typescript/best-practices/debugging#stale-workflows "Direct link to Stale Workflows") If you are developing Workflows and finding that code isn't executing as expected, the first place to look is whether old Workflows are still running. If those old Workflows have the same name and are on the same task queue, Temporal will try to continue executing them on your new code by design. You may get errors that make no sense to you because * Temporal is trying to execute old Workflow code that no longer exists in your codebase, or * your new Client code is expecting Temporal to execute old Workflow/Activity code it doesn't yet know about. The biggest sign that this is happening is if you notice Temporal is acting non-deterministically: running the same Workflow twice gets different results. Stale workflows are usually a non-issue because the errors generated are just noise from code you no longer want to run. If you need to terminate old stale Workflows, you can do so with Temporal Web or the Temporal CLI. ### Workflow/Activity registration errors[​](https://docs.temporal.io/develop/typescript/best-practices/debugging#workflowactivity-registration-errors "Direct link to Workflow/Activity registration errors") **If your Workflows or Activities are not imported or spelled correctly**, here are some errors we've seen: * `ApplicationFailure: 'MyFunction' is not a function` * `Workflow did not register a handler for MyQuery` Double check that your Workers are registering the right Workflow and Activity Definitions (function names) on the right Task Queues. **If you are running Temporal in a monorepo**, then your `node_modules` may be in a different location than where Temporal expects to find it by default, which results in errors like: [ERROR] Module not found: Error: Can't resolve '@temporalio/workflow/lib/worker-interface.js' in '/src' Our [Next.js tutorial](https://learn.temporal.io/tutorials/typescript/nextjs) is written for people setting up Temporal **within an existing monorepo**, which may be of use here. When you pass a `workflowsPath`, our Webpack config expects to find `node_modules` in the same or a parent/ancestor directory. **If you are custom bundling your own Workflows** you may get errors like these: [ERROR] Failed to activate workflow { runId: 'aaf84a83-51ce-462a-9ab7-6a641a703bff', error: ReferenceError: exports is not defined, workflowExists: false} Temporal Workflow Bundles need to [export a set of methods that fit the compiled `worker-interface.ts` from `@temporalio/workflow`](https://github.com/temporalio/sdk-typescript/blob/eaa2d205c9bc5ff4a3b17c0b34f2dcf6b1e0264a/packages/worker/src/workflow/bundler.ts#L81) as an entry point. We do offer a `bundleWorkflowCode` method to assist you with this, though it uses our Webpack settings. For more information, see the [Register types](https://docs.temporal.io/develop/typescript/workers/run-worker-process#register-types) section. ### Webpack errors[​](https://docs.temporal.io/develop/typescript/best-practices/debugging#webpack-errors "Direct link to Webpack errors") The TypeScript SDK's Worker bundles Workflows based on `workflowsPath` with [Webpack](https://webpack.js.org/) and run them inside v8 isolates. If Webpack fails to create the bundle, the SDK will throw an error and emit webpack logs using the SDK's [logger](https://docs.temporal.io/develop/typescript/platform/observability#logging) . If you do not see Webpack output in your terminal make sure that you have not disabled SDK logging (see reference to `Runtime.install()` in the link above). **A common mistake for newcomers to the TypeScript SDK is trying to use Node.js built-ins and modules in their Workflow code.** Usually, the best thing to do is move that code to an Activity. Some common examples that will **not** work in the Workflow isolate: Importing node built-in modules Antipattern import fs from 'fs';const config = fs.readFileSync('config.json', 'utf8'); This is invalid because reading from the filesystem is a non-deterministic operation: the file may change from the time of the original Workflow Execution to when the Workflow is replayed. You'll typically see an error in this form in the Webpack output: 2021-10-14T19:22:00.606Z [INFO] Module not found: Error: Can't resolve 'fs' in '/Users/you/your-project/src'2021-10-14T19:22:00.606Z [INFO] resolve 'fs' in '/Users/you/your-project/src'2021-10-14T19:22:00.606Z [INFO] Parsed request is a module2021-10-14T19:22:00.606Z [INFO] using description file: /Users/you/your-project/package.json (relative path: ./src)2021-10-14T19:22:00.606Z [INFO] Field 'browser' doesn't contain a valid alias configuration Importing and calling Activities directly from Workflow code Antipattern import { makeHTTPRequest } from './activities';export async function yourWorkflow(): Promise { return await makeHTTPRequest('https://temporal.io');} This is invalid because activity implementations should not be directly referenced by Workflow code. Activities are used by Workflows in order to make network calls and read from the filesystem, operations which are non-deterministic by nature because they rely on external state. Temporal records Activity results in the Workflow history and in case your Workflow is replayed, completed Activities will not be rerun, instead their recorded result will be delivered to the Workflow. You'll typically see an error in this form in the Webpack output: 2021-10-14T19:46:52.731Z [INFO] ERROR in ./src/activities.ts 8:31-462021-10-14T19:46:52.731Z [INFO] Module not found: Error: Can't resolve 'http' in '/Users/you/your-project/src'2021-10-14T19:46:52.731Z [INFO]2021-10-14T19:46:52.731Z [INFO] BREAKING CHANGE: webpack < 5 used to include polyfills for node.js core modules by default.2021-10-14T19:46:52.731Z [INFO] This is no longer the case. Verify if you need this module and configure a polyfill for it.2021-10-14T19:46:52.731Z [INFO]2021-10-14T19:46:52.731Z [INFO] If you want to include a polyfill, you need to:2021-10-14T19:46:52.731Z [INFO] - add a fallback 'resolve.fallback: { "http": require.resolve("stream-http") }'2021-10-14T19:46:52.731Z [INFO] - install 'stream-http'2021-10-14T19:46:52.731Z [INFO] If you don't want to include a polyfill, you can use an empty module like this:2021-10-14T19:46:52.731Z [INFO] resolve.fallback: { "http": false } To properly call your Activities from Workflow code use `proxyActivities` and make sure to only import the Activity types. import { proxyActivities } from '@temporalio/workflow';import type * as activities from './activities';const { makeHTTPRequest } = proxyActivities();export async function yourWorkflow(): Promise { return await makeHTTPRequest('https://temporal.io');} ### Works in Dev but not in Prod[​](https://docs.temporal.io/develop/typescript/best-practices/debugging#works-in-dev-but-not-in-prod "Direct link to Works in Dev but not in Prod") The two main sources of dev-prod discrepancies are in bundling and connecting. #### Production bundling[​](https://docs.temporal.io/develop/typescript/best-practices/debugging#production-bundling "Direct link to Production bundling") You may experience your Client sending stripped names as the Workflow "Type" when scheduling a Workflow. Webpack can change the Workflow's function name to something shorter. Temporal won't know how to handle the mismatch between the shorter name and the expected Workflow type. You may experience errors like this: Error: 3 INVALID_ARGUMENT: WorkflowType is not set on request. Or you may see shorter names in the Temporal Service's Web UI when Webpack changed the Workflow's function name to something shorter, in this case the single letter 's': ![Temporal Web UI showing stripped 'Workflow Type' entries, in this case the single letter 's'](https://docs.temporal.io/img/webui/stripped_workflow_types_in_webui.png) Temporal Web UI showing stripped 'Workflow Type' entries, in this case the single letter 's' This issue can happen when your bundler strips out Workflow function names. Temporal relies on those names to set the "Workflow Type" in the Service Web UI. To prevent the build process from shortening Workflow function names, modify the webpack configuration file ( `webpack.config.js`) to set the Boolean that retains the original names in the `TerserPlugin` configuration section. Setting the option (`keep_fnames`) to `true` prevents name stripping. * Webpack with Terser * ESbuild // webpack.config.jsmodule.exports = { optimization: { minimize: true, minimizer: [ new TerserPlugin({ terserOptions: { keep_fnames: true, // don't strip function names in production }, }), ], },}; require('esbuild').buildSync({ entryPoints: ['app.js'], minify: true, keepNames: true, outfile: 'out.js',}); See the [esbuild docs](https://esbuild.github.io/api/#keep-names) for more information. #### Connecting to Temporal Server[​](https://docs.temporal.io/develop/typescript/best-practices/debugging#connecting-to-temporal-server "Direct link to Connecting to Temporal Server") If you are trying to connect in production and getting this: [TransportError: transport error] It is a sign that something is wrong with your Cert/Key pair. Log it out and make sure it is an exact match with what is expected (often, the issue can be whitespace when injecting from your production secrets management environment). ### Resetting Workflows to deal with logical bugs[​](https://docs.temporal.io/develop/typescript/best-practices/debugging#resetting-workflows-to-deal-with-logical-bugs "Direct link to Resetting Workflows to deal with logical bugs") You can "rewind time" using the Temporal CLI, resetting Workflow History to some previous point in time. You can read the Temporal CLI docs on: * [Restarting and resetting Workflows by ID](https://docs.temporal.io/cli) * [Resetting all Workflows by binary checksum identifier](https://docs.temporal.io/cli) If you need to reset programmatically, the TS SDK does not have any high level APIs for this, but you can make raw gRPC calls to [resetWorkflowExecution](https://typescript.temporal.io/api/classes/proto.temporal.api.workflowservice.v1.WorkflowService-1/#resetworkflowexecution) . Resetting should only be used to deal with serious logical bugs in your code: it's not for handling transient failures, like a downstream service being unreachable. It should not be used in the course of normal application flows. ### gRPC call timeouts (context deadline exceeded)[​](https://docs.temporal.io/develop/typescript/best-practices/debugging#grpc-call-timeouts-context-deadline-exceeded "Direct link to gRPC call timeouts (context deadline exceeded)") The opaque `context deadline exceeded` error comes from `gRPC`: Error: 4 DEADLINE_EXCEEDED: context deadline exceeded at Object.callErrorFromStatus (/Users/swyx/Work/Temporal/samples-typescript/nextjs-oneclick/node_modules/@grpc/grpc-js/build/src/call.js:31:26) at Object.onReceiveStatus (/Users/swyx/Work/Temporal/samples-typescript/nextjs-oneclick/node_modules/@grpc/grpc-js/build/src/client.js:179:52) at Object.onReceiveStatus (/Users/swyx/Work/Temporal/samples-typescript/nextjs-oneclick/node_modules/@grpc/grpc-js/build/src/client-interceptors.js:336:141) at Object.onReceiveStatus (/Users/swyx/Work/Temporal/samples-typescript/nextjs-oneclick/node_modules/@grpc/grpc-js/build/src/client-interceptors.js:299:181) at /Users/swyx/Work/Temporal/samples-typescript/nextjs-oneclick/node_modules/@grpc/grpc-js/build/src/call-stream.js:145:78 at processTicksAndRejections (node:internal/process/task_queues:78:11) { code: 4, details: 'context deadline exceeded', metadata: Metadata { internalRepr: Map(1) { 'content-type' => [Array] }, options: {} }, page: '/api/getBuyState'} Several conditions can cause this error, including network hiccups, timeouts that are too short, and an overloaded server. Querying a Workflow Execution whose query handler causes an error can result in the query call timing out. Some troubleshooting actions you can take: * Verify the connection from your Worker to the Temporal Server is working and doesn't have unusually high latency. * If you are running Temporal Server yourself, check your [server metrics](https://docs.temporal.io/self-hosted-guide/production-checklist#scaling-and-metrics) to ensure it's not overloaded. * If what's timing out is a query, check the logs of your Workers to see if they are having issues handling the query. If none of the preceding actions help you discover why timeouts are occurring, please try to produce a minimal repro and we'll be glad to help. * [How to debug in a development environment](https://docs.temporal.io/develop/typescript/best-practices/debugging#debug-in-a-development-environment) * [How to debug in a production environment](https://docs.temporal.io/develop/typescript/best-practices/debugging#debug-in-a-production-environment) * [How to troubleshoot common issues in the TypeScript SDK](https://docs.temporal.io/develop/typescript/best-practices/debugging#troubleshoot-common-issues) * [Two locations to watch](https://docs.temporal.io/develop/typescript/best-practices/debugging#two-locations-to-watch) * [Stale Workflows](https://docs.temporal.io/develop/typescript/best-practices/debugging#stale-workflows) * [Workflow/Activity registration errors](https://docs.temporal.io/develop/typescript/best-practices/debugging#workflowactivity-registration-errors) * [Webpack errors](https://docs.temporal.io/develop/typescript/best-practices/debugging#webpack-errors) * [Works in Dev but not in Prod](https://docs.temporal.io/develop/typescript/best-practices/debugging#works-in-dev-but-not-in-prod) * [Production bundling](https://docs.temporal.io/develop/typescript/best-practices/debugging#production-bundling) * [Connecting to Temporal Server](https://docs.temporal.io/develop/typescript/best-practices/debugging#connecting-to-temporal-server) * [Resetting Workflows to deal with logical bugs](https://docs.temporal.io/develop/typescript/best-practices/debugging#resetting-workflows-to-deal-with-logical-bugs) * [gRPC call timeouts (context deadline exceeded)](https://docs.temporal.io/develop/typescript/best-practices/debugging#grpc-call-timeouts-context-deadline-exceeded) --- # Converters and encryption - Ruby SDK | Temporal Platform Documentation [Skip to main content](https://docs.temporal.io/develop/ruby/best-practices/converters-and-encryption#__docusaurus_skipToContent_fallback) On this page Temporal's security model is designed around client-side encryption of Payloads. A client may encrypt Payloads before sending them to the server, and decrypt them after receiving them from the server. This provides a high degree of confidentiality because the Temporal Server itself has absolutely no knowledge of the actual data. It also gives implementers more power and more freedom regarding which client is able to read which data -- they can control access with keys, algorithms, or other security measures. A Temporal developer adds client-side encryption of Payloads by providing a Custom Payload Codec to its Client. Depending on business needs, a complete implementation of Payload Encryption may involve selecting appropriate encryption algorithms, managing encryption keys, restricting a subset of their users from viewing payload output, or a combination of these. The server itself never adds encryption over Payloads. Therefore, unless client-side encryption is implemented, Payload data will be persisted in non-encrypted form to the data store, and any Client that can make requests to a Temporal namespace (including the Temporal UI and CLI) will be able to read Payloads contained in Workflows. When working with sensitive data, you should always implement Payload encryption. Custom Payload Codec[​](https://docs.temporal.io/develop/ruby/best-practices/converters-and-encryption#custom-payload-codec "Direct link to Custom Payload Codec") ------------------------------------------------------------------------------------------------------------------------------------------------------------------- Custom Data Converters can change the default Temporal Data Conversion behavior by adding hooks, sending payloads to external storage, or performing different encoding steps. If you only need to change the encoding performed on your payloads -- by adding compression or encryption -- you can override the default Data Converter to use a new `PayloadCodec`. The Payload Codec needs to extend `Temporalio::Converters::PayloadCodec` and implement `encode` and `decode` methods. These should convert the given payloads as needed into new payloads, using the `"encoding"` metadata field. Do not mutate the existing payloads. Here is an example of an encryption codec that just uses base64 in each direction: class Base64Codec < Temporalio::Converters::PayloadCodec def encode(payloads) payloads.map do |p| Temporalio::Api::Common::V1::Payload.new( # Set our specific encoding. We may also want to add a key ID in here for use by # the decode side metadata: { 'encoding' => 'binary/my-payload-encoding' }, data: Base64.strict_encode64(p.to_proto) ) end end def decode(payloads) payloads.map do |p| # Ignore if it doesn't have our expected encoding next p unless p.metadata['encoding'] == 'binary/my-payload-encoding' Temporalio::Api::Common::V1::Payload.decode( Base64.strict_decode64(p.data) ) end endend **Set Data Converter to use custom Payload Codec** When creating a client, the default `DataConverter` can be updated with the payload codec like so: my_client = Temporalio::Client.connect( 'localhost:7233', 'my-namespace', data_converter: Temporalio::Converters::DataConverter.new(payload_codec: Base64Codec.new)) * Data **encoding** is performed by the client using the converters and codecs provided by Temporal or your custom implementation when passing input to the Temporal Cluster. For example, plain text input is usually serialized into a JSON object, and can then be compressed or encrypted. * Data **decoding** may be performed by your application logic during your Workflows or Activities as necessary, but decoded Workflow results are never persisted back to the Temporal Cluster. Instead, they are stored encoded on the Cluster, and you need to provide an additional parameter when using the [temporal workflow show](https://docs.temporal.io/cli/workflow#show) command or when browsing the Web UI to view output. ### Using a Codec Server[​](https://docs.temporal.io/develop/ruby/best-practices/converters-and-encryption#using-a-codec-server "Direct link to Using a Codec Server") A Codec Server is an HTTP server that uses your custom Codec logic to decode your data remotely. The Codec Server is independent of the Temporal Cluster and decodes your encrypted payloads through predefined endpoints. You create, operate, and manage access to your Codec Server in your own environment. The Temporal CLI and the Web UI in turn provide built-in hooks to call the Codec Server to decode encrypted payloads on demand. Refer to the [Codec Server](https://docs.temporal.io/production-deployment/data-encryption) documentation for information on how to design and deploy a Codec Server. Payload conversion[​](https://docs.temporal.io/develop/ruby/best-practices/converters-and-encryption#custom-payload-converter "Direct link to Payload conversion") ------------------------------------------------------------------------------------------------------------------------------------------------------------------- Temporal SDKs provide a default [Payload Converter](https://docs.temporal.io/payload-converter) that can be customized to convert a custom data type to [Payload](https://docs.temporal.io/dataconversion#payload) and back. ### Conversion sequence[​](https://docs.temporal.io/develop/ruby/best-practices/converters-and-encryption#conversion-sequence "Direct link to Conversion sequence") The order in which your encoding Payload Converters are applied depend on the order given to the Data Converter. You can set multiple encoding Payload Converters to run your conversions. When the Data Converter receives a value for conversion, it passes through each Payload Converter in sequence until the converter that handles the data type does the conversion. Payload Converters can be customized independently of a Payload Codec. Temporal's Converter architecture looks like this: ![Temporal converter architecture](https://docs.temporal.io/img/info/converter-architecture.png) Temporal converter architecture ### Supported Data Types[​](https://docs.temporal.io/develop/ruby/best-practices/converters-and-encryption#supported-data-types "Direct link to Supported Data Types") Data converters are used to convert raw Temporal payloads to/from actual Ruby types. A custom data converter can be set via the `data_converter` keyword argument when creating a client. Data converters are a combination of payload converters, payload codecs, and failure converters. Payload converters convert Ruby values to/from serialized bytes. Payload codecs convert bytes to bytes (e.g. for compression or encryption). Failure converters convert exceptions to/from serialized failures. Data converters are in the `Temporalio::Converters` module. The default data converter uses a default payload converter, which supports the following types: * `nil` * "bytes" (i.e. `String` with `Encoding::ASCII_8BIT` encoding) * `Google::Protobuf::MessageExts` instances * [JSON module](https://docs.ruby-lang.org/en/master/JSON.html) for everything else This means that normal Ruby objects will use `JSON.generate` when serializing and `JSON.parse` when deserializing (with `create_additions: true` set by default). So a Ruby object will often appear as a hash when deserialized. Also, hashes that are passed in with symbol keys end up with string keys when deserialized. While "JSON Additions" are supported, it is not cross-SDK-language compatible since this is a Ruby-specific construct. The default payload converter is a collection of "encoding payload converters". On serialize, each encoding converter will be tried in order until one accepts (default falls through to the JSON one). The encoding converter sets an `encoding` metadata value which is used to know which converter to use on deserialize. Custom encoding converters can be created, or even the entire payload converter can be replaced with a different implementation. **NOTE:** For ActiveRecord, or other general/ORM models that are used for a different purpose, it is not recommended to try to reuse them as Temporal models. Eventually model purposes diverge and models for a Temporal workflows/activities should be specific to their use for clarity and compatibility reasons. Also many Ruby ORMs do many lazy things and therefore provide unclear serialization semantics. Instead, consider having models specific for workflows/activities and translate to/from existing models as needed. See the next section on how to do this with ActiveModel objects. #### ActiveModel[​](https://docs.temporal.io/develop/ruby/best-practices/converters-and-encryption#active-model "Direct link to ActiveModel") By default, ActiveModel objects do not natively support the `JSON` module. A mixin can be created to add this support for ActiveModel, for example: module ActiveModelJSONSupport extend ActiveSupport::Concern include ActiveModel::Serializers::JSON included do def as_json(*) super.merge(::JSON.create_id => self.class.name) end def to_json(*args) as_json.to_json(*args) end def self.json_create(object) object = object.dup object.delete(::JSON.create_id) new(**object.symbolize_keys) end endend Now if `include ActiveModelJSONSupport` is present on any ActiveModel class, on serialization `to_json` will be used which will use `as_json` which calls the super `as_json` but also includes the fully qualified class name as the JSON `create_id` key. On deserialization, Ruby JSON then uses this key to know what class to call `json_create` on. * [Custom Payload Codec](https://docs.temporal.io/develop/ruby/best-practices/converters-and-encryption#custom-payload-codec) * [Using a Codec Server](https://docs.temporal.io/develop/ruby/best-practices/converters-and-encryption#using-a-codec-server) * [Payload conversion](https://docs.temporal.io/develop/ruby/best-practices/converters-and-encryption#custom-payload-converter) * [Conversion sequence](https://docs.temporal.io/develop/ruby/best-practices/converters-and-encryption#conversion-sequence) * [Supported Data Types](https://docs.temporal.io/develop/ruby/best-practices/converters-and-encryption#supported-data-types) --- # About Temporal SDKs | Temporal Platform Documentation [Skip to main content](https://docs.temporal.io/encyclopedia/temporal-sdks#__docusaurus_skipToContent_fallback) On this page Temporal SDKs (software development kits) are an open source collection of tools, libraries, and APIs that enable Temporal Application development. They offer a [Temporal Client](https://docs.temporal.io/encyclopedia/temporal-sdks#temporal-client) to interact with the [Temporal Service](https://docs.temporal.io/temporal-service) , APIs to develop your [Temporal Application](https://docs.temporal.io/encyclopedia/temporal-sdks#temporal-application) , and APIs to run horizontally scalable [Workers](https://docs.temporal.io/workers#worker) . SDKs are more than just a development tool, however. The SDK APIs enable developers to write code in a particular pattern that mirrors real world processes. The SDK's internal implementation, working in collaboration with the Temporal Service, steps through that code, guaranteeing execution progression during application runtime. Temporal Applications[​](https://docs.temporal.io/encyclopedia/temporal-sdks#temporal-application "Direct link to Temporal Applications") ------------------------------------------------------------------------------------------------------------------------------------------ A Temporal Application is the code you write, comprised of [Workflow Definitions](https://docs.temporal.io/workflow-definition) , [Activity Definitions](https://docs.temporal.io/workflow-definition) , code used to configure [Temporal Clients](https://docs.temporal.io/encyclopedia/temporal-sdks#temporal-client) , and code used to configure and start [Workers](https://docs.temporal.io/workers#worker) . Developers create Temporal Applications using an [official Temporal SDK](https://docs.temporal.io/encyclopedia/temporal-sdks#official-sdks) . Consider that the Workflow Definition code can be executed repeatedly. The Temporal Platform can concurrently support millions to billions of Workflow Executions, each of which representing an invoked Workflow Definition. Additionally, a Temporal Workflow Execution is both resumable and recoverable, and it can react to external events. * Resumable: The ability of a process to resume execution after suspending on an _awaitable_. * Recoverable: The ability of a process to resume execution after suspending due to a _failure_. * Reactive: The ability of a process to respond to external events. Hence, a Temporal Application can run for seconds or years in the presence of arbitrary load and failures. Official SDKs[​](https://docs.temporal.io/encyclopedia/temporal-sdks#official-sdks "Direct link to Official SDKs") ------------------------------------------------------------------------------------------------------------------- **What are the officially supported SDKs?** Each Temporal SDK targets a specific programming language. * [Go SDK feature guides](https://docs.temporal.io/develop/go) * [Java SDK feature guides](https://docs.temporal.io/develop/java) * [Python SDK feature guides](https://docs.temporal.io/develop/python/) * [TypeScript SDK feature guides](https://docs.temporal.io/develop/typescript/) * [.NET SDK feature guides](https://docs.temporal.io/develop/dotnet) * [Ruby SDK feature guides](https://docs.temporal.io/develop/ruby/) * [PHP SDK feature guides](https://docs.temporal.io/develop/php) Despite supporting multiple languages, and supporting many features, Temporal SDKs aim to make developers feel at home in their language. ### Third-party SDKs[​](https://docs.temporal.io/encyclopedia/temporal-sdks#third-party-sdks "Direct link to Third-party SDKs") The following third-party SDKs exist but are not officially supported by Temporal: * [Swift](https://github.com/apple/swift-temporal-sdk) from [@Swift Community](https://github.com/apple) * [Haskell](https://github.com/MercuryTechnologies/hs-temporal-sdk) from [@MercuryTechnologies](https://github.com/MercuryTechnologies) * [Clojure](https://github.com/manetu/temporal-clojure-sdk) from [@Manetu](https://github.com/manetu) * [Scala](https://github.com/vitaliihonta/zio-temporal) from [@vitaliihonta](https://github.com/vitaliihonta) Why use a Temporal SDK?[​](https://docs.temporal.io/encyclopedia/temporal-sdks#why-use-an-sdk "Direct link to Why use a Temporal SDK?") ---------------------------------------------------------------------------------------------------------------------------------------- Temporal SDKs empower developers to concentrate on creating dependable and scalable business logic, alleviating the need to build home-grown supervisor systems to ensure reliability and fault-tolerance. This is possible because the Temporal SDK provides a unified library that abstracts the intricacies of how Temporal handles distributed systems. ### Development pattern[​](https://docs.temporal.io/encyclopedia/temporal-sdks#development-pattern "Direct link to Development pattern") By abstracting complexities and streamlining boilerplate code, developers can craft straightforward code that directly aligns with their business logic, enhancing code readability and bolstering developer productivity. Consider a bank loan application. Developers can design the business logic of a bank loan using the Temporal SDK. The Workflow defines the overarching business logic, encompassing tasks such as validating applicant information, credit checks, loan approval, and applicant notifications, as Activities. Do not copy and use code The following is pseudocode. For tested samples see your language SDK's developer's guide. func LoanApplicationWorkflow { sdk.ExecuteActivity(CreditCheck) sdk.ExecuteActivity(AutomatedApproval) sdk.ExecuteActivity(NotifyApplicant) // ...} For instance, Temporal SDKs have built-in support for handling failures, timeouts, and retries. In the event of an Activity failure, the SDK automatically initiates retries according to configurable policies established by the developer within the SDK. This streamlined process simplifies the integration of fault-tolerance mechanisms into applications. Do not copy and use code The following is pseudocode. For tested samples see your language SDK's developer's guide. func LoanApplicationWorkflow { options = { MaxAttempts: 3, StartToCloseTimeout: 30min, HeartbeatTimeout: 10min, } sdk.ExecuteActivity(CreditCheck, options) sdk.ExecuteActivity(AutomatedApproval) sdk.ExecuteActivity(NotifyApplicant) // ...} ### Replays[​](https://docs.temporal.io/encyclopedia/temporal-sdks#replays "Direct link to Replays") Another quality of the SDKs lies in their ability to replay Workflow Executions, a complex operation that contributes significantly to the Platform's promised reliability. ![The SDKs Replay code execution to continue from the last step](https://docs.temporal.io/diagrams/replay-basic.svg) The SDKs Replay code execution to continue from the last step We will delve into this idea more later, but for now, it signifies that the SDKs can automatically continue a process from the point of interruption, should a failure occur. This capability stems from the SDK's ability to persist each step the program takes. Temporal SDKs major components[​](https://docs.temporal.io/encyclopedia/temporal-sdks#major-components "Direct link to Temporal SDKs major components") -------------------------------------------------------------------------------------------------------------------------------------------------------- **What are the major components of Temporal SDKs?** Temporal SDKs offer developers the following: * A Temporal Client to communicate with a Temporal Service * APIs to develop application code (Workflows & Activities) * APIs to configure and run Workers ![Temporal SDK components create a runtime across your environment and a Temporal Service](https://docs.temporal.io/diagrams/temporal-sdk-components.svg) Temporal SDK components create a runtime across your environment and a Temporal Service Let's break down each one. ### Temporal Client[​](https://docs.temporal.io/encyclopedia/temporal-sdks#temporal-client "Direct link to Temporal Client") A Temporal Client acts as the bridge for communication between your applications and the Temporal Service. The Client performs key functions that facilitate the execution of, management of, and communication with Workflows. The most common operations that a Temporal Client enables you to perform are the following: * Get the result of Workflow Execution. * List Workflow Executions. * Query a Workflow Execution. * Signal a Workflow Execution. * Start a Workflow Execution. The following code is an example using the Go SDK. It showcases how to initialize a Temporal Client, create a connection to a local Temporal Service, and start a Workflow Execution: Do not copy and use code The following code is for example purposes only. For tested code samples and best practices, use your preferred language SDK's developer's guide. * [Go SDK Temporal Client feature guide](https://docs.temporal.io/develop/go/client/temporal-client) * [Java SDK Temporal Client feature guide](https://docs.temporal.io/develop/java/client/temporal-client) * [PHP SDK Temporal Client feature guide](https://docs.temporal.io/develop/php/client/temporal-client) * [Python SDK Temporal Client feature guide](https://docs.temporal.io/develop/python/client) * [TypeScript SDK Temporal Client feature guide](https://docs.temporal.io/develop/typescript/client) package mainimport ( "context" "go.temporal.io/sdk/client")func main() { // Temporal Client setup code c, err := client.NewClient(client.Options{}) if err != nil { log.Fatalln("Unable to create client", err) } defer c.Close() // Prepare Workflow option and parameters workflowOptions := client.StartWorkflowOptions{ ID: "loan-application-1", TaskQueue: "loan-application-task-queue", } applicantDetails := ApplicantDetails{ // ... } // Start the Workflow workflowRun, err := c.ExecuteWorkflow(context.Background(), workflowOptions, "loan-application-workflow", applicantDetails) if err != nil { // ... } // ...} Developers can then use the Client as the main entry point for interacting with the application through Temporal. Using that Client, developers may for example start or Signal Workflows, Query a Workflow's state, etc. We can see in the example above how the developer has used `ExecuteWorkflow` API to start a Workflow. ### APIs to Develop Workflows[​](https://docs.temporal.io/encyclopedia/temporal-sdks#apis-to-develop-workflows "Direct link to APIs to Develop Workflows") Workflows are defined as code: either a function or an object method, depending on the language. For example, the following is a valid Temporal Workflow in Go: Do not copy and use code The following code is for example purposes only. For tested code samples and best practices, use your preferred language SDK's developer's guide. func LoanApplication(ctx context.Context) (error) { // ... return nil} The Workflow code uses Temporal SDK APIs to orchestrate the steps of the application. Do not copy and use code The following code is for example purposes only. For tested code samples and best practices, use your preferred language SDK's developer's guide. func LoanApplication(ctx workflow.Context, input *LoanApplicationWorkflowInput) (*LoanApplicationWorkflowResult, error) { // ... var result activities.CreditCheckResult f := workflow.ExecuteActivity(ctx, a.CreditCheck, CreditCheckInput(*input)) err := f.Get(ctx, &result) // ... // Return the results return &loanApplicationResults, nil} A Workflow executes Activities (other functions that interact with external systems), handles and sends messages (Queries, Signals, Updates), and interacts with other Workflows. This Workflow code, while executing, can be paused, resumed, and migrated across physical machines without losing state. When a Workflow calls the API to execute an Activity, the Worker sends a [Command](https://docs.temporal.io/references/commands) back to the Temporal Service. The Temporal Service creates Activity Tasks in response which the same or a different Worker can then pick up and begin executing. In this way, the Worker and Temporal Service work together to incrementally execute Workflow code in a reliable way. We discuss this more in detail in [The SDK and Temporal Service relationship](https://docs.temporal.io/encyclopedia/temporal-sdks#sdk-and-cluster-relationship) section. The SDK APIs also enable developers to write code that more genuinely maps to their process. This is because without a specialized SDK, developers might have to write a lot of boilerplate code. This can lead to code that's hard to maintain, difficult to understand, or that doesn't directly correspond to the underlying business process. For example, the bank loan application Workflow might actually look like this: Do not copy and use code The following code is for example purposes only. For tested code samples and best practices, use your preferred language SDK's developer's guide. // LoanApplicationWorkflow is the workflow definition.func LoanApplicationWorkflow(ctx workflow.Context, applicantName string, loanAmount int) (string, error) { // Step 1: Notify the applicant that the application process has started err := workflow.ExecuteActivity(ctx, NotifyApplicantActivity, applicantName, "Application process started").Get(ctx, nil) if err != nil { return "", err } // Step 2: Perform a credit check var creditCheckResult string err = workflow.ExecuteActivity(ctx, LoanCreditCheckActivity, loanAmount).Get(ctx, &creditCheckResult) if err != nil { return "", err } // Step 3: Perform an automatic approval check var approvalCheckResult string err = workflow.ExecuteActivity(ctx, AutomaticApprovalCheckActivity, creditCheckResult).Get(ctx, &approvalCheckResult) if err != nil { return "", err } // Step 4: Notify the applicant of the decision var notificationResult string err = workflow.ExecuteActivity(ctx, NotifyApplicantActivity, applicantName, approvalCheckResult).Get(ctx, ¬ificationResult) if err != nil { return "", err } return notificationResult, nil} The level of abstraction that APIs offer enables the developer to focus on business logic without having to worry about the intricacies of distributed computing such as retries, or having to explicitly maintain a state machine and the intermediate state for each step of the process. Additionally, the state of the Workflow is automatically persisted so if a failure does occur, it resumes right where it left off. ### APIs to create and manage Worker Processes[​](https://docs.temporal.io/encyclopedia/temporal-sdks#apis-to-create-and-manage-worker-processes "Direct link to APIs to create and manage Worker Processes") Workers are responsible for executing Workflow and Activity code (application code). The SDK provides APIs for configuring and starting Workers, enabling developers to control how the code is executed. Workers are horizontally scalable, often run with systems like Kubernetes, and configured according to the application's needs. Here is an example of how you could initialize a Worker using the Go SDK. Do not copy and use code The following code is for example purposes only. For tested code samples and best practices, use your preferred language SDK's developer's guide. func main() { // Create the client object just once per process c, err := client.NewClient(client.Options{}) if err != nil { log.Fatalln("Unable to create Temporal client", err) } defer c.Close() // Create the Worker instance w := worker.New(c, "loan-application-task-queue", worker.Options{}) // Register the workflow and activity with the worker w.RegisterWorkflow(LoanApplicationWorkflow) w.RegisterActivity(LoanCreditCheck) // Start listening to the Task Queue err = w.Run(worker.InterruptCh()) if err != nil { log.Fatalln("Unable to start Worker", err) }} The Worker polls on the specified Task Queue, processing those Tasks, and reporting the results back to the Temporal Service. They execute both the Workflows and Activities, and the SDK ensures that they perform these tasks efficiently and reliably. ### APIs to customize Activity Execution behavior[​](https://docs.temporal.io/encyclopedia/temporal-sdks#apis-to-customize-activity-execution-behavior "Direct link to APIs to customize Activity Execution behavior") Activities in Temporal are individual units of work that often represent non-deterministic parts of the code logic, such as querying a database or calling an external service. The SDK provides APIs to customize the behavior of an Activity Execution. By default, if an Activity attempts to communicate with another system and encounters a transient failure like a network issue, Temporal ensures the Activity is tried again automatically. However, Temporal enables developers to control a variety of timeouts, a Retry Policy, Heartbeat monitoring, and asynchronous completion. The following code is an example of a custom set of Activity Execution options that affect the timeout and retry behavior of the execution, should the Activity encounter a failure. Do not copy and use code The following code is for example purposes only. For tested code samples and best practices, use your preferred language SDK's developer's guide. // LoanApplicationWorkflow is the Workflow Definition.func LoanApplicationWorkflow(ctx workflow.Context, applicantName string, loanAmount int) (string, error) { // ... var creditCheckResult string // set a Retry Policy ao := workflow.ActivityOptions{ ScheduleToCloseTimeout: time.Hour, HeartbeatTimeout: time.Minute, RetryPolicy: &temporal.RetryPolicy{ InitialInterval: time.Second, BackoffCoefficient: 2, MaximumInterval: time.Minute, MaximumAttempts: 5, }, } ctx = workflow.WithActivityOptions(ctx, ao) err = workflow.ExecuteActivity(ctx, LoanCreditCheckActivity, loanAmount).Get(ctx, &creditCheckResult) if err != nil { return "", err } // ... return notificationResult, nil}// LoanCreditCheckActivity is an Activity function that performs a credit check.func LoanCreditCheckActivity(ctx context.Context, loanAmount int) (string, error) { // ... your logic here ... return "Credit check passed", nil} The SDK and Temporal Service relationship[​](https://docs.temporal.io/encyclopedia/temporal-sdks#sdk-and-cluster-relationship "Direct link to The SDK and Temporal Service relationship") ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ **How do the Temporal SDKs work with the Temporal Service?** The Temporal Service functions more as a choreographer than a conductor. Rather than directly assigning tasks to Workers, the Temporal Service arranges the Tasks into a Task Queue while Workers poll the Task Queue. Developers may create a fleet of Workers and tune them so that a Task is picked up as soon as it is available. If a Worker goes down, Tasks can wait until the next Worker is available. A Workflow might request to execute an Activity, start a Timer, or start a Child Workflow, each of which translates into a Command, dispatched to the Temporal Service. In addition to acting on these Commands, the Temporal Service documents that interaction by appending their corresponding Events into the Workflow Execution's Event History. Take for instance the call to execute an Activity. When a Workflow invokes it, the Worker doesn't immediately execute that Activity code. Instead, it generates a ScheduleActivityTask Command, dispatching it to the Cluster. In response, the Cluster queues up a new Activity Task. Only when a Worker finds itself free, it collects the task and begins executing the Activity code. The Temporal Service persists Workflow Execution Event History, so that if there is a failure, the SDK Worker is able to Replay the execution and resume where it left off. This is where the deterministic constraints of the Workflow code comes into play, requiring the use of Activities to create side effects and interact with the outside world. Let's look at an example Workflow with a single Activity. func LoanApplication(ctx workflow.Context, input *LoanApplicationWorkflowInput) (*LoanApplicationWorkflowResult, error) { ctx = workflow.WithActivityOptions(ctx, workflow.ActivityOptions{ StartToCloseTimeout: time.Minute, }) var result activities.NotifyApplicantActivityResult f := workflow.ExecuteActivity(ctx, a.NotifyApplicantActivity, NotifyApplicantActivityInput(*input)) err := f.Get(ctx, &result) // Return the results return &l.LoanApplicationState, nil}type Activities struct {}func (a *Activities) NotifyApplicantActivity(ctx context.Context, input *NotifyApplicantActivityInput) (*NotifyApplicantActivityResult, error) { var result NotifyApplicantActivityResult // Call the thirdparty API and handle the result return &result, err} The Activity above is performing a single call to an external API. Since the call can fail due to transient issues, we define it outside of the Workflow and provide it with retry options. When you create a new Worker process, the Worker creates a long-lasting connection to the Temporal Service, polling a Task Queue for Tasks that are related to the code it is capable of executing. ![A Worker long polls for Tasks](https://docs.temporal.io/diagrams/how-sdk-works-1.svg) A Worker long polls for Tasks Although the Worker is now running, unless a Workflow is explicitly started, the Task Queue doesn't have any Tasks on it and so, no code executes. We can use a Temporal Client (available in Temporal SDKs and the Temporal CLI) to start a new Workflow. ![Start a Workflow using a Temporal Client](https://docs.temporal.io/diagrams/how-sdk-works-2.svg) Start a Workflow using a Temporal Client Starting a Workflow Execution creates a new Event, WorkflowExecutionStarted, and adds it to the Workflow Execution's Event History. The Temporal Service then schedules a Workflow Task by adding it to the Task Queue. When the Worker has capacity, it picks up this Task, and begin executing code. Each step of the Task (e.g. Scheduled, Started, and Completed), gets recorded into the Event History. * Scheduled means that the Temporal Service has added a Task to the Task Queue. * Started means that the Worker has dequeued the Task. * Completed means that the Worker finished executing the Task by responding to the Temporal Service. When the call to invoke the Activity is evaluated, the Worker suspends executing the code and sends a Command to the Temporal Service to schedule an Activity Task. ![Worker suspends code execution and sends a Command to the Temporal Service](https://docs.temporal.io/diagrams/how-sdk-works-3.svg) Worker suspends code execution and sends a Command to the Temporal Service When the Worker process can perform more work, it picks up the Activity Task and begins executing the Activity code, which includes the call to the external API. If the Activity fails, say the API goes down, Temporal will automatically retry the Activity with one second between intervals, as the configurations have defined, an infinite number of times until the Activity succeeds or is canceled. In the case where the calls succeeds, and the code completes, the Worker tells the Temporal Service the Activity Task completed. ![The Worker reports that the Activity Execution completed](https://docs.temporal.io/diagrams/how-sdk-works-activity.svg) The Worker reports that the Activity Execution completed Included is any data that was returned from the Activity (results of the API call), which is then persisted in the Workflow Execution Event History, and is now accessible to the Workflow code. The Temporal Service creates a new Workflow Task which the Worker picks up. ![The Worker picks up the new Task](https://docs.temporal.io/diagrams/how-sdk-works-1.svg) The Worker picks up the new Task This is when the SDK Worker Replays the Workflow code, using the Event History as guidance on what to expect. If the Replay encounters an Event that doesn't match up with what is expected from the code, a [non-determinism](https://docs.temporal.io/references/errors#non-deterministic-error) error gets thrown. If there is alignment, the Worker continues evaluating code. Assuming the Activity Execution is successful, the Workflow now has the result of the Activity and the Worker is able to finish evaluating and executing the Workflow code, responding to the Temporal Service when complete. The result of the Workflow can now be retrieved using a Temporal Client. ![The Temporal Client can now access the result of the Workflow](https://docs.temporal.io/diagrams/how-sdk-works-4.svg) The Temporal Client can now access the result of the Workflow And that’s how a Temporal Worker and Temporal Service work together. * [Temporal Applications](https://docs.temporal.io/encyclopedia/temporal-sdks#temporal-application) * [Official SDKs](https://docs.temporal.io/encyclopedia/temporal-sdks#official-sdks) * [Third-party SDKs](https://docs.temporal.io/encyclopedia/temporal-sdks#third-party-sdks) * [Why use a Temporal SDK?](https://docs.temporal.io/encyclopedia/temporal-sdks#why-use-an-sdk) * [Development pattern](https://docs.temporal.io/encyclopedia/temporal-sdks#development-pattern) * [Replays](https://docs.temporal.io/encyclopedia/temporal-sdks#replays) * [Temporal SDKs major components](https://docs.temporal.io/encyclopedia/temporal-sdks#major-components) * [Temporal Client](https://docs.temporal.io/encyclopedia/temporal-sdks#temporal-client) * [APIs to Develop Workflows](https://docs.temporal.io/encyclopedia/temporal-sdks#apis-to-develop-workflows) * [APIs to create and manage Worker Processes](https://docs.temporal.io/encyclopedia/temporal-sdks#apis-to-create-and-manage-worker-processes) * [APIs to customize Activity Execution behavior](https://docs.temporal.io/encyclopedia/temporal-sdks#apis-to-customize-activity-execution-behavior) * [The SDK and Temporal Service relationship](https://docs.temporal.io/encyclopedia/temporal-sdks#sdk-and-cluster-relationship) --- # Client - Ruby SDK | Temporal Platform Documentation [Skip to main content](https://docs.temporal.io/develop/ruby/platform#__docusaurus_skipToContent_fallback) On this page ![Ruby SDK Banner](https://docs.temporal.io/assets/images/banner-ruby-temporal-be833f13b8e3655d7a8d4e50119b7da2.png) Platform[​](https://docs.temporal.io/develop/ruby/platform#platform "Direct link to Platform") ----------------------------------------------------------------------------------------------- * [Observability](https://docs.temporal.io/develop/ruby/platform/observability) * [Enriching the UI](https://docs.temporal.io/develop/ruby/platform/enriching-ui) * [Platform](https://docs.temporal.io/develop/ruby/platform#platform) --- # Core application - Temporal feature | Temporal Platform Documentation [Skip to main content](https://docs.temporal.io/evaluate/development-production-features/core-application#__docusaurus_skipToContent_fallback) **Workflows**, **Activities**, and **Workers** form the core parts of a Temporal Application. **Workflows**: A Workflow defines the overall flow of the application. You write it in your programming language of choice using the Temporal SDK. Conceptually, a Workflow specifies a sequence of steps and orchestrates the execution of Activities. **Activities**: An Activity is a method or function that encapsulates business logic prone to failure (e.g., calling a service that may go down). The system can automatically retry these Activities upon some failures. Activities perform a single, well-defined action, such as calling another service, transcoding a media file, or sending an email message. **Workers**: A Worker executes your Workflow and Activity code. **Follow one of our tutorials to [Get started](https://learn.temporal.io/getting_started/) learning how to develop Workflows and Activities and run them in Worker Processes.** Or jump straight to a Temporal SDK feature guide: Related 📚 * [![](https://docs.temporal.io/img/sdks/svgs/golang.svg)Go SDK Core application feature guide](https://docs.temporal.io/develop/go) feature-guide * [![](https://docs.temporal.io/img/sdks/svgs/java.svg)Java SDK guide](https://docs.temporal.io/develop/java) feature-guide * [![](https://docs.temporal.io/img/sdks/svgs/php.svg)PHP SDK Core application feature guide](https://docs.temporal.io/develop/php) feature-guide * [![](https://docs.temporal.io/img/sdks/svgs/python.svg)Python SDK Core application feature guide](https://docs.temporal.io/develop/python) feature-guide * [![](https://docs.temporal.io/img/sdks/svgs/typescript.svg)TypeScript SDK Core application feature guide](https://docs.temporal.io/develop/typescript) feature-guide * [![](https://docs.temporal.io/img/sdks/svgs/dotnet.svg).NET SDK Core application feature guide](https://docs.temporal.io/develop/dotnet) feature-guide * [![](https://docs.temporal.io/img/sdks/svgs/ruby.svg)Ruby SDK Core application feature guide](https://docs.temporal.io/develop/ruby) feature-guide For a deep dive into Temporal Workflows, Activities, and Workers, visit the following Temporal Encyclopedia pages or enroll in one of [our courses](https://learn.temporal.io/courses/) . * [Temporal Workflows](https://docs.temporal.io/workflows) * [Temporal Activities](https://docs.temporal.io/activities) * [Temporal Workers](https://docs.temporal.io/workers) --- # Workflow message passing - Temporal feature | Temporal Platform Documentation [Skip to main content](https://docs.temporal.io/evaluate/development-production-features/workflow-message-passing#__docusaurus_skipToContent_fallback) Need to interact with your Workflow from outside of it? Think about use cases like these: * Your shipment-tracking Workflow needs to know when the item leaves the warehouse and is loaded into their truck. **Signal** your Workflow when the truck driver scans the barcode. * Folks in your company want to track the progress of their data migration Workflows. **Query** your running batch Workflow to get the data for the progress bar. * Your eCommerce shopping cart Workflow needs to know when a new item is added. **Update** it to add the item and receive back the current items to render. Temporal provides Signals, Queries, and Updates to allow rich interactivity with your running Workflows. **Signals**: Signal to send messages asynchronously to a running Workflow, changing its state or controlling its flow in real-time. **Queries**: Query to check the progress of your Workflow or debug the internal state in real-time. **Updates**: Update to send synchronous requests to your Workflow and track it in real-time. To learn more about using these powerful primitives, see our encyclopedia entry: Related 📚 [![](https://docs.temporal.io/img/assets/link-preview-icon.svg)Workflow message passing (Signals, Queries, & Updates)](https://docs.temporal.io/encyclopedia/workflow-message-passing) encyclopedia For a deeper dive into Workflow message passing, enroll in one of [our courses](https://learn.temporal.io/courses/interacting_with_workflows) . If you want to jump straight to implementation details, see the SDK feature guides. Related 📚 * [![](https://docs.temporal.io/img/sdks/svgs/golang.svg)Go SDK Workflow message passing feature guide](https://docs.temporal.io/develop/go/workflows/message-passing) feature-guide * [![](https://docs.temporal.io/img/sdks/svgs/java.svg)Java SDK Workflow message passing feature guide](https://docs.temporal.io/develop/java/workflows/message-passing) feature-guide * [![](https://docs.temporal.io/img/sdks/svgs/python.svg)Python SDK Workflow message passing feature guide](https://docs.temporal.io/develop/python/workflows/message-passing) feature-guide * [![](https://docs.temporal.io/img/sdks/svgs/typescript.svg)TypeScript SDK Workflow message passing feature guide](https://docs.temporal.io/develop/typescript/workflows/message-passing) feature-guide * [![](https://docs.temporal.io/img/sdks/svgs/php.svg)PHP SDK Workflow message passing feature guide](https://docs.temporal.io/develop/php/workflows/message-passing) feature-guide * [![](https://docs.temporal.io/img/sdks/svgs/dotnet.svg).NET SDK Workflow message passing feature guide](https://docs.temporal.io/develop/dotnet/workflows/message-passing) feature-guide * [![](https://docs.temporal.io/img/sdks/svgs/ruby.svg)Ruby SDK Workflow message passing feature guide](https://docs.temporal.io/develop/ruby/workflows/message-passing) feature-guide --- # Continue-As-New | Temporal Platform Documentation [Skip to main content](https://docs.temporal.io/workflow-execution/continue-as-new#__docusaurus_skipToContent_fallback) On this page This page discusses [Continue-As-New](https://docs.temporal.io/workflow-execution/continue-as-new#continue-as-new) and how to decide [when to use it](https://docs.temporal.io/workflow-execution/continue-as-new#when) . What is Continue-As-New?[​](https://docs.temporal.io/workflow-execution/continue-as-new#continue-as-new "Direct link to What is Continue-As-New?") --------------------------------------------------------------------------------------------------------------------------------------------------- Continue-As-New allows you to checkpoint your Workflow's state and start a fresh Workflow. There are two main reasons you might want to start a new Workflow: * A Workflow Execution with a long, or large [Event History](https://docs.temporal.io/workflow-execution/event#event-history) , such as one calling many Activities, may bog down and have performance issues. It could even generate more Events than allowed by the [Event History limits](https://docs.temporal.io/workflow-execution/event#event-history-limits) . * A Workflow Execution can hit [Workflow Versioning](https://docs.temporal.io/workflow-definition#workflow-versioning) problems if it started running on an older version of your code and then begins executing on a newer version. Your goal is to create a new Workflow with a fresh history that picks up where your last one left off. First, pass your latest relevant state into Continue-As-New. This hands it to a new Execution in the [Execution Chain](https://docs.temporal.io/workflow-execution#workflow-execution-chain) . This state is passed in as arguments to your Workflow. The parameters are typically optional and left unset by the original caller of the Workflow. The new Workflow Execution has the same Workflow Id, but a different Run Id, and starts its own Event History. You can repeat Continue-As-New as often as needed, which means that your Workflow can run forever. Workflows that do this are often called Entity Workflows because they represent durable objects, not just processes. * [How to Continue-As-New using the Go SDK](https://docs.temporal.io/develop/go/workflows/continue-as-new#how) * [How to Continue-As-New using the Java SDK](https://docs.temporal.io/develop/java/workflows/continue-as-new) * [How to Continue-As-New using the PHP SDK](https://docs.temporal.io/develop/php/workflows/continue-as-new) * [How to Continue-As-New using the Python SDK](https://docs.temporal.io/develop/python/workflows/continue-as-new#how) * [How to Continue-As-New using the TypeScript SDK](https://docs.temporal.io/develop/typescript/workflows/continue-as-new) * [How to Continue-As-New using the .NET SDK](https://docs.temporal.io/develop/dotnet/workflows/continue-as-new) When in your Workflow is it right to Continue-As-New?[​](https://docs.temporal.io/workflow-execution/continue-as-new#when "Direct link to When in your Workflow is it right to Continue-As-New?") -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Temporal will tell your Workflow when it's approaching performance or scalability problems. Find out if it's time by checking Continue-As-New Suggested in your Workflow at spots in your implementation where you are ready to checkpoint your state. To prevent long-running Workflows from running on stale versions of code, you may also want to Continue-as-New periodically, depending on how often you deploy. This makes sure you're running only a couple of versions, which avoids some backwards compatibility problems. * [Determine when to Continue-As-New using the Go SDK](https://docs.temporal.io/develop/go/workflows/continue-as-new#when) * [Determine when to Continue-As-New using the Java SDK](https://docs.temporal.io/develop/java/workflows/continue-as-new) * [Determine when to Continue-As-New using the PHP SDK](https://docs.temporal.io/develop/php/workflows/continue-as-new) * [Determine when to Continue-As-New using the Python SDK](https://docs.temporal.io/develop/python/workflows/continue-as-new) * [Determine when to Continue-As-New using the TypeScript SDK](https://docs.temporal.io/develop/typescript/workflows/continue-as-new) * [Determine when to Continue-As-New using the .NET SDK](https://docs.temporal.io/develop/dotnet/workflows/continue-as-new) * [What is Continue-As-New?](https://docs.temporal.io/workflow-execution/continue-as-new#continue-as-new) * [When in your Workflow is it right to Continue-As-New?](https://docs.temporal.io/workflow-execution/continue-as-new#when) --- # Feature guide - TypeScript SDK feature guide | Temporal Platform Documentation [Skip to main content](https://docs.temporal.io/develop/typescript/nexus/feature-guide#__docusaurus_skipToContent_fallback) On this page SUPPORT, STABILITY, and DEPENDENCY INFO Temporal TypeScript SDK support for Nexus is in [Public Preview](https://docs.temporal.io/evaluate/development-production-features/release-stages#public-preview) . Use [Temporal Nexus](https://docs.temporal.io/evaluate/nexus) to connect Temporal Applications within and across Namespaces using a Nexus Endpoint, a Nexus Service contract, and Nexus Operations. This page shows how to do the following: * [Run a development Temporal Service with Nexus enabled](https://docs.temporal.io/develop/typescript/nexus/feature-guide#run-the-temporal-nexus-development-server) * [Create caller and handler Namespaces](https://docs.temporal.io/develop/typescript/nexus/feature-guide#create-caller-handler-namespaces) * [Create a Nexus Endpoint to route requests from caller to handler](https://docs.temporal.io/develop/typescript/nexus/feature-guide#create-nexus-endpoint) * [Define the Nexus Service contract](https://docs.temporal.io/develop/typescript/nexus/feature-guide#define-nexus-service-contract) * [Develop a Nexus Service and Operation handlers](https://docs.temporal.io/develop/typescript/nexus/feature-guide#develop-nexus-service-operation-handlers) * [Develop a caller Workflow that uses a Nexus Service](https://docs.temporal.io/develop/typescript/nexus/feature-guide#develop-caller-workflow-nexus-service) * [Understand exceptions in Nexus Operations](https://docs.temporal.io/develop/typescript/nexus/feature-guide#exceptions-in-nexus-operations) * [Cancel a Nexus Operation](https://docs.temporal.io/develop/typescript/nexus/feature-guide#canceling-a-nexus-operation) * [Make Nexus calls across Namespaces in Temporal Cloud](https://docs.temporal.io/develop/typescript/nexus/feature-guide#nexus-calls-across-namespaces-temporal-cloud) note This documentation uses source code derived from the [TypeScript Nexus sample](https://github.com/temporalio/samples-typescript/tree/main/nexus-hello) . Run the Temporal Development Server with Nexus enabled[​](https://docs.temporal.io/develop/typescript/nexus/feature-guide#run-the-temporal-nexus-development-server "Direct link to Run the Temporal Development Server with Nexus enabled") --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Prerequisites: * [Install the latest Temporal CLI](https://learn.temporal.io/getting_started/typescript/dev_environment/#set-up-a-local-temporal-service-for-development-with-temporal-cli) (`v1.3.0` or higher recommended) * [Install the latest Temporal TypeScript SDK](https://learn.temporal.io/getting_started/typescript/dev_environment/#add-temporal-typescript-sdk-dependencies) (`v1.12.3` or higher) The first step in working with Temporal Nexus involves starting a Temporal Server with Nexus enabled. temporal server start-dev This command automatically starts the Temporal development server with the Web UI, and creates the `default` Namespace. It uses an in-memory database, so do not use it for real use cases. The Temporal Web UI should now be accessible at [http://localhost:8233](http://localhost:8233/) , and the Temporal Server should now be available for client connections on `localhost:7233`. Create caller and handler Namespaces[​](https://docs.temporal.io/develop/typescript/nexus/feature-guide#create-caller-handler-namespaces "Direct link to Create caller and handler Namespaces") ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ Before setting up Nexus endpoints, create separate Namespaces for the caller and handler. temporal operator namespace create --namespace my-target-namespacetemporal operator namespace create --namespace my-caller-namespace For this example, `my-target-namespace` will contain the Nexus Operation handler, and you will use a Workflow in `my-caller-namespace` to call that Operation handler. We use different namespaces to demonstrate cross-Namespace Nexus calls. Create a Nexus Endpoint to route requests from caller to handler[​](https://docs.temporal.io/develop/typescript/nexus/feature-guide#create-nexus-endpoint "Direct link to Create a Nexus Endpoint to route requests from caller to handler") --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- After establishing caller and handler Namespaces, the next step is to create a Nexus Endpoint to route requests. temporal operator nexus endpoint create \ --name my-nexus-endpoint-name \ --target-namespace my-target-namespace \ --target-task-queue my-handler-task-queue You can also use the Web UI to create the Namespaces and Nexus endpoint. Define the Nexus Service contract[​](https://docs.temporal.io/develop/typescript/nexus/feature-guide#define-nexus-service-contract "Direct link to Define the Nexus Service contract") --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Defining a clear contract for the Nexus Service is crucial for smooth communication. In this example, there is a service package that describes the Service and Operation names along with input/output types for caller Workflows to use the Nexus Endpoint. Each [Temporal SDK includes and uses a default Data Converter](https://docs.temporal.io/dataconversion) . The default data converter encodes payloads in the following order: Null, Byte array, and JSON. In a polyglot environment, that is where more than one language and SDK is being used to develop a Temporal solution, JSON is a common choice. This example uses plain TypeScript objects, serialized into JSON. Note: By default, the TypeScript SDK [does not support Protobuf JSON encoding](https://typescript.temporal.io/api/interfaces/common.PayloadConverter) . If passing Protobuf payloads use the [ProtobufJsonPayloadConverter](https://typescript.temporal.io/api/classes/protobufs.ProtobufJsonPayloadConverter) instead. [nexus-hello/src/api.ts](https://github.com/temporalio/samples-typescript/blob/main/nexus-hello/src/api.ts) import * as nexus from 'nexus-rpc';export const helloService = nexus.service('hello', { /** * Return the input message, unmodified. In the present sample, this Operation * will be implemented using the Synchronous Nexus Operation handler syntax. */ echo: nexus.operation(), /** * Return a salutation message, in the requested language. In the present sample, * this Operation will be implemented by starting the `helloWorkflow` Workflow. */ hello: nexus.operation(),});export interface EchoInput { message: string;}export interface EchoOutput { message: string;}export interface HelloInput { name: string; language: LanguageCode;}export interface HelloOutput { message: string;}export type LanguageCode = 'en' | 'fr' | 'de' | 'es' | 'tr'; Develop a Nexus Service handler and Operation handlers[​](https://docs.temporal.io/develop/typescript/nexus/feature-guide#develop-nexus-service-operation-handlers "Direct link to Develop a Nexus Service handler and Operation handlers") -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- A Nexus Service handler is defined using the `nexus-rpc`'s [`serviceHandler`](https://nexus-rpc.github.io/sdk-typescript/functions/serviceHandler.html) function. Nexus Service handlers are typically defined in the same Worker as the underlying Temporal primitives they abstract. A Service handler must provide Operation handlers for each Operation declared by the Service. Operation handlers can decide if a given Nexus Operation will be synchronous or asynchronous. They can invoke underlying Temporal primitives such as a Query, Signal, or Update using the Temporal SDK Client, or run other reliable code. Handlers should be reliable since the [circuit breaker](https://docs.temporal.io/nexus/operations#circuit-breaking) trips after 5 consecutive retryable errors, blocking all Operations from the caller to that Endpoint. The `@temporalio/nexus` package provides utilities to help create Nexus Operations that interact with a Temporal namespace: * `WorkflowRunOperationHandler` - Create an asynchronous operation handler that starts a Workflow. * `getClient()` - Get a Temporal Client connected using the same `NativeConnection` as the present Temporal Worker. It can be used to implement synchronous handlers backed by Temporal primitives such as Signals and Queries. ### Develop a Synchronous Nexus Operation handler[​](https://docs.temporal.io/develop/typescript/nexus/feature-guide#develop-a-synchronous-nexus-operation-handler "Direct link to Develop a Synchronous Nexus Operation handler") Simple RPC handlers can be implemented as synchronous Nexus Operation handlers, which is defined in TypeScript as a simple async function. Use `getClient()` from `@temporalio/nexus` to get the Temporal Client for signaling, querying, and listing Workflows. Implementations can also make other calls, but handlers should be reliable to avoid tripping the [circuit breaker](https://docs.temporal.io/nexus/operations#circuit-breaking) . [nexus-hello/src/service/handler.ts](https://github.com/temporalio/samples-typescript/blob/main/nexus-hello/src/service/handler.ts) // ...import * as nexus from 'nexus-rpc';// ...import { helloService, EchoInput, EchoOutput, HelloInput, HelloOutput } from '../api';// ...export const helloServiceHandler = nexus.serviceHandler(helloService, { echo: async (ctx, input: EchoInput): Promise => { // A simple async function can be used to defined a Synchronous Nexus Operation. // This is often sufficient for Operations that simply make arbitrary short calls to // other services or databases, or that perform simple computations such as this one. // // You may also access a Temporal Client by calling `temporalNexus.getClient()`. // That Client can be used to make arbitrary calls, such as signaling, querying, // or listing workflows. return input; },// ...}); ### Use the Temporal Client for Signals, Queries, and Updates[​](https://docs.temporal.io/develop/typescript/nexus/feature-guide#use-the-temporal-client-for-signals-queries-and-updates "Direct link to Use the Temporal Client for Signals, Queries, and Updates") A common pattern is to use the Temporal Client from within a sync handler to Signal, Query, or Update a Workflow. You can also use Signal-With-Start or Update-With-Start to ensure the Workflow is started and send it a Signal or Update. All calls must complete within the [Nexus request timeout](https://docs.temporal.io/cloud/limits#nexus-operation-request-timeout) . The handler receives an AbortSignal via `ctx.abortSignal` that is triggered when the deadline is exceeded — pass it to Temporal Client calls to ensure they are canceled if the timeout is reached. Updates should be short-lived to stay within this deadline. The handler context also exposes `ctx.requestDeadline` as an optional `Date`, representing the time by which the current request must complete. Note that this is the deadline for the current _request_, not the overall operation. Use it to make decisions about whether to start work that may not finish in time, or to set timeouts on downstream calls. ### Develop an Asynchronous Nexus Operation handler to start a Workflow[​](https://docs.temporal.io/develop/typescript/nexus/feature-guide#develop-an-asynchronous-nexus-operation-handler-to-start-a-workflow "Direct link to Develop an Asynchronous Nexus Operation handler to start a Workflow") Use `@temporalio/nexus`'s `WorkflowRunOperationHandler` helper class to easily expose a Temporal Workflow as a Nexus Operation. Note that even though a Nexus operation can only take one input parameter, if you need to pass multiple arguments through to the workflow, you can do so by using multiple properties of the input object, and placing them in the array provided to the `args` option when calling `startWorkflow`. [nexus-hello/src/service/handler.ts](https://github.com/temporalio/samples-typescript/blob/main/nexus-hello/src/service/handler.ts) import { randomUUID } from 'crypto';import * as nexus from 'nexus-rpc';import * as temporalNexus from '@temporalio/nexus';import { helloService, EchoInput, EchoOutput, HelloInput, HelloOutput } from '../api';import { helloWorkflow } from './workflows';export const helloServiceHandler = nexus.serviceHandler(helloService, {// ... hello: new temporalNexus.WorkflowRunOperationHandler( // WorkflowRunOperationHandler takes a function that receives the Operation's context and input. // That function can be used to validate and/or transform the input before passing it to // the Workflow, as well as to customize various Workflow start options as appropriate. // Call temporalNexus.startWorkflow() to actually start the Workflow from inside the // WorkflowRunOperationHandler's delegate function. async (ctx, input: HelloInput) => { return await temporalNexus.startWorkflow(ctx, helloWorkflow, { args: [input], // Workflow IDs should typically be business-meaningful IDs and are used to dedupe workflow starts. // For this example, we're using the request ID allocated by Temporal when the caller workflow schedules // the operation, this ID is guaranteed to be stable across retries of this operation. workflowId: ctx.requestId ?? randomUUID(), // Task queue defaults to the task queue this Operation is handled on. }); }, ),}); Workflow IDs should typically be business-meaningful IDs and are used to dedupe Workflow starts. In general, the ID should be passed in the Operation input as part of the Nexus Service contract. RESOURCES [Attach multiple Nexus callers to a handler Workflow](https://docs.temporal.io/nexus/operations#attaching-multiple-nexus-callers) with a Conflict-Policy of Use-Existing. ### Register your Nexus Service handler in a Worker[​](https://docs.temporal.io/develop/typescript/nexus/feature-guide#register-your-nexus-service-handler-in-a-worker "Direct link to Register your Nexus Service handler in a Worker") After developing an asynchronous Nexus Operation handler to start a Workflow, the next step is to register your Nexus Service handler in a Worker. [nexus-hello/src/service/worker.ts](https://github.com/temporalio/samples-typescript/blob/main/nexus-hello/src/service/worker.ts) import { Worker, NativeConnection } from '@temporalio/worker';import { helloServiceHandler } from './handler';// ... const namespace = 'my-target-namespace'; const serviceTaskQueue = 'my-handler-task-queue'; const worker = await Worker.create({ connection, namespace, taskQueue: serviceTaskQueue, workflowsPath: require.resolve('./workflows'), nexusServices: [helloServiceHandler], }); Develop a caller Workflow that uses the Nexus Service[​](https://docs.temporal.io/develop/typescript/nexus/feature-guide#develop-caller-workflow-nexus-service "Direct link to Develop a caller Workflow that uses the Nexus Service") --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- To execute a Nexus Operation from a Workflow, import the necessary service definition types, then use `@temporalio/workflow`'s `createNexusServiceClient` to create a Nexus client for that service. You will need to provide the Nexus Endpoint name, which you registered previously in [Create a Nexus Endpoint to route requests from caller to handler](https://docs.temporal.io/develop/typescript/nexus/feature-guide#create-nexus-endpoint) . [nexus-hello/src/caller/workflows.ts](https://github.com/temporalio/samples-typescript/blob/main/nexus-hello/src/caller/workflows.ts) import * as wf from "@temporalio/workflow";import { helloService, LanguageCode } from "../service/api";const HELLO_SERVICE_ENDPOINT = "hello-service-endpoint-name";export async function helloCallerWorkflow(name: string, language: LanguageCode): Promise { const nexusClient = wf.createNexusServiceClient({ service: helloService, endpoint: HELLO_SERVICE_ENDPOINT, }); const helloResult = await nexusClient.executeOperation( "hello", { name, language }, { scheduleToCloseTimeout: "10s" } ); return helloResult.message;} ### Register the caller Workflow in a Worker and start the caller Workflow[​](https://docs.temporal.io/develop/typescript/nexus/feature-guide#register-the-caller-workflow-in-a-worker-and-start-the-caller-workflow "Direct link to Register the caller Workflow in a Worker and start the caller Workflow") This Workflow can be registered with a Worker and started using `client.startWorkflow()` or `client.executeWorkflow()`, as usual. Refer to the [complete TypeScript sample](https://github.com/temporalio/samples-typescript/blob/main/nexus-hello) for reference. * [nexus-hello/src/caller/worker.ts](https://github.com/temporalio/samples-typescript/blob/main/nexus-hello/src/caller/worker.ts) shows how to register the caller Workflow in a Worker and run the Worker. * [nexus-hello/src/starter.ts](https://github.com/temporalio/samples-typescript/blob/main/nexus-hello/src/starter.ts) shows how to use a Temporal Client to execute the sample caller Workflow. Exceptions in Nexus operations[​](https://docs.temporal.io/develop/typescript/nexus/feature-guide#exceptions-in-nexus-operations "Direct link to Exceptions in Nexus operations") ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Temporal provides general guidance on [Errors in Nexus operations](https://docs.temporal.io/references/failures#errors-in-nexus-operations) . In TypeScript, there are three Nexus-specific exception classes: * `nexus-rpc`'s [`OperationError`](https://nexus-rpc.github.io/sdk-typescript/classes/OperationError.html) : this is the exception type you should throw in a Nexus operation to indicate that it has failed according to its own application logic and should not be retried. * `nexus-rpc`'s [`HandlerError`](https://nexus-rpc.github.io/sdk-typescript/classes/HandlerError.html) : you can throw this exception type in a Nexus operation with a specific [HandlerErrorType](https://nexus-rpc.github.io/sdk-typescript/types/HandlerErrorType.html) . The error will be marked as either retryable or non-retryable according to the type, following the [Nexus spec](https://github.com/nexus-rpc/api/blob/main/SPEC.md#predefined-handler-errors) . The non-retryable handler error types are `BAD_REQUEST`, `UNAUTHENTICATED`, `UNAUTHORIZED`, `NOT_FOUND`, `NOT_IMPLEMENTED`; the retryable types are `RESOURCE_EXHAUSTED`, `INTERNAL`, `UNAVAILABLE`, `UPSTREAM_TIMEOUT`. * `@temporalio/nexus`'s [`NexusOperationFailure`](https://typescript.temporal.io/api/classes/common.NexusOperationFailure) : this is the error thrown inside a Workflow when a Nexus operation fails for any reason. Use the `cause` attribute on the exception to access the cause chain. Canceling a Nexus Operation[​](https://docs.temporal.io/develop/typescript/nexus/feature-guide#canceling-a-nexus-operation "Direct link to Canceling a Nexus Operation") ------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Nexus Operations, just like other cancellable APIs provided by the `@temporalio/workflow` package, execute within Cancellation Scopes. Requesting cancellation of a Cancellation Scope results in requesting cancellation for all cancellable operations owned by that scope. The Workflow itself defines the root Cancellation Scope. Requesting cancellation of the Workflow therefore propagates the cancellation request to all cancellable operations started by that workflow, including Nexus Operations. To provide more granular control over cancellation of a specific Nexus Operation, you may explicitly create a new Cancellation Scope, and start the Nexus Operation from within that scope. An example demonstrating this can be found at our [nexus cancellation sample](https://github.com/temporalio/samples-typescript/tree/main/nexus-cancellation) . Only asynchronous operations can be canceled in Nexus, since cancellation is sent using an operation token. The Workflow or other resources backing the operation may choose to ignore the cancellation request. Once the caller Workflow completes, the caller's Nexus Machinery will not make any further attempts to cancel operations that are still running. It's okay to leave operations running in some use cases. To ensure cancellations are delivered, wait for all pending operations to finish before exiting the Workflow. Make Nexus calls across Namespaces in Temporal Cloud[​](https://docs.temporal.io/develop/typescript/nexus/feature-guide#nexus-calls-across-namespaces-temporal-cloud "Direct link to Make Nexus calls across Namespaces in Temporal Cloud") -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- This section assumes you are already familiar with how to connect a Worker to Temporal Cloud. The `tcld` CLI is used to create Namespaces and the Nexus Endpoint, and mTLS client certificates will be used to securely connect the caller and handler Workers to their respective Temporal Cloud Namespaces. ### Install the latest `tcld` CLI and generate certificates[​](https://docs.temporal.io/develop/typescript/nexus/feature-guide#install-the-latest-tcld-cli-and-generate-certificates "Direct link to install-the-latest-tcld-cli-and-generate-certificates") To install the latest version of the `tcld` CLI, run the following command (on macOS): brew install temporalio/brew/tcld If you don't already have certificates, you can generate them for mTLS Worker authentication using the command below: tcld gen ca --org $YOUR_ORG_NAME --validity-period 1y --ca-cert ca.pem --ca-key ca.key These certificates will be valid for one year. ### Create caller and handler Namespaces[​](https://docs.temporal.io/develop/typescript/nexus/feature-guide#create-caller-and-handler-namespaces "Direct link to Create caller and handler Namespaces") Before deploying to Temporal Cloud, ensure that the appropriate Namespaces are created for both the caller and handler. If you already have these Namespaces, you don't need to do this. tcld logintcld namespace create \ --namespace \ --cloud-provider aws \ --region us-west-2 \ --ca-certificate-file 'path/to/your/ca.pem' \ --retention-days 1tcld namespace create \ --namespace \ --cloud-provider aws \ --region us-west-2 \ --ca-certificate-file 'path/to/your/ca.pem' \ --retention-days 1 Alternatively, you can create Namespaces through the UI: [https://cloud.temporal.io/namespaces](https://cloud.temporal.io/namespaces) . ### Create a Nexus Endpoint to route requests from caller to handler[​](https://docs.temporal.io/develop/typescript/nexus/feature-guide#create-a-nexus-endpoint-to-route-requests-from-caller-to-handler "Direct link to Create a Nexus Endpoint to route requests from caller to handler") To create a Nexus Endpoint you must have a Developer account role or higher, and have NamespaceAdmin permission on the `--target-namespace`. tcld nexus endpoint create \ --name \ --target-task-queue my-handler-task-queue \ --target-namespace \ --allow-namespace \ --description-file description.md The `--allow-namespace` is used to build an Endpoint allowlist of caller Namespaces that can use the Nexus Endpoint, as described in Runtime Access Control. Alternatively, you can create a Nexus Endpoint through the UI: [https://cloud.temporal.io/nexus](https://cloud.temporal.io/nexus) . Observability[​](https://docs.temporal.io/develop/typescript/nexus/feature-guide#observability "Direct link to Observability") ------------------------------------------------------------------------------------------------------------------------------- ### Web UI[​](https://docs.temporal.io/develop/typescript/nexus/feature-guide#web-ui "Direct link to Web UI") A synchronous Nexus Operation will surface in the caller Workflow as follows, with just `NexusOperationScheduled` and `NexusOperationCompleted` events in the caller's Workflow history: ![Observability Sync](https://docs.temporal.io/img/cloud/nexus/go-sdk-observability-sync.png) Observability Sync An asynchronous Nexus Operation will surface in the caller Workflow as follows, with `NexusOperationScheduled`, `NexusOperationStarted`, and `NexusOperationCompleted`, in the caller's Workflow history: ![Observability Async](https://docs.temporal.io/img/cloud/nexus/go-sdk-observability-async.png) Observability Async ### Temporal CLI[​](https://docs.temporal.io/develop/typescript/nexus/feature-guide#temporal-cli "Direct link to Temporal CLI") Use the `workflow describe` command to show pending Nexus Operations in the caller Workflow and any attached callbacks on the handler Workflow: temporal workflow describe -w Nexus events are included in the caller's Workflow history: temporal workflow show -w For **asynchronous Nexus Operations** the following are reported in the caller's history: * `NexusOperationScheduled` * `NexusOperationStarted` * `NexusOperationCompleted` For **synchronous Nexus Operations** the following are reported in the caller's history: * `NexusOperationScheduled` * `NexusOperationCompleted` note `NexusOperationStarted` isn't reported in the caller's history for synchronous operations. ### OpenTelemetry[​](https://docs.temporal.io/develop/typescript/nexus/feature-guide#opentelemetry "Direct link to OpenTelemetry") The `@temporalio/interceptors-opentelemetry` package supports Nexus Operations, providing automatic trace context propagation across Nexus boundaries from the caller Workflow to the handler. The easiest way to enable it is with the `OpenTelemetryPlugin`, which auto-registers Nexus interceptors alongside Activity and Workflow interceptors: import { OpenTelemetryPlugin } from '@temporalio/interceptors-opentelemetry';const plugin = new OpenTelemetryPlugin({ resource: myResource, spanProcessor: mySpanProcessor,});const worker = await Worker.create({ // ... plugins: [plugin], nexusServices: [myServiceHandler],}); The plugin creates the following spans: * **Caller side:** `StartNexusOperation:service/operation` — created when the caller Workflow starts a Nexus Operation. * **Handler side:** `RunStartNexusOperation:service/operation` and `RunCancelNexusOperation:service/operation` — created when the handler processes the operation. These spans are children of the caller span, linked via trace context propagated in Nexus request headers. See the [interceptors-opentelemetry sample](https://github.com/temporalio/samples-typescript/tree/main/interceptors-opentelemetry) for a complete example. For custom interceptor logic beyond tracing (e.g., logging, authorization), see [Nexus interceptor registration](https://docs.temporal.io/develop/typescript/workers/interceptors#nexus-interceptor-registration) . Learn more[​](https://docs.temporal.io/develop/typescript/nexus/feature-guide#learn-more "Direct link to Learn more") ---------------------------------------------------------------------------------------------------------------------- * Read the high-level description of the [Temporal Nexus feature](https://docs.temporal.io/evaluate/nexus) and watch the [Nexus keynote and demo](https://youtu.be/qqc2vsv1mrU?feature=shared&t=2082) . * Learn how Nexus works in the [Nexus deep dive talk](https://www.youtube.com/watch?v=izR9dQ_eIe4) and [Encyclopedia](https://docs.temporal.io/nexus) . * Deploy Nexus Endpoints in production with [Temporal Cloud](https://docs.temporal.io/cloud/nexus) . * [Run the Temporal Development Server with Nexus enabled](https://docs.temporal.io/develop/typescript/nexus/feature-guide#run-the-temporal-nexus-development-server) * [Create caller and handler Namespaces](https://docs.temporal.io/develop/typescript/nexus/feature-guide#create-caller-handler-namespaces) * [Create a Nexus Endpoint to route requests from caller to handler](https://docs.temporal.io/develop/typescript/nexus/feature-guide#create-nexus-endpoint) * [Define the Nexus Service contract](https://docs.temporal.io/develop/typescript/nexus/feature-guide#define-nexus-service-contract) * [Develop a Nexus Service handler and Operation handlers](https://docs.temporal.io/develop/typescript/nexus/feature-guide#develop-nexus-service-operation-handlers) * [Develop a Synchronous Nexus Operation handler](https://docs.temporal.io/develop/typescript/nexus/feature-guide#develop-a-synchronous-nexus-operation-handler) * [Use the Temporal Client for Signals, Queries, and Updates](https://docs.temporal.io/develop/typescript/nexus/feature-guide#use-the-temporal-client-for-signals-queries-and-updates) * [Develop an Asynchronous Nexus Operation handler to start a Workflow](https://docs.temporal.io/develop/typescript/nexus/feature-guide#develop-an-asynchronous-nexus-operation-handler-to-start-a-workflow) * [Register your Nexus Service handler in a Worker](https://docs.temporal.io/develop/typescript/nexus/feature-guide#register-your-nexus-service-handler-in-a-worker) * [Develop a caller Workflow that uses the Nexus Service](https://docs.temporal.io/develop/typescript/nexus/feature-guide#develop-caller-workflow-nexus-service) * [Register the caller Workflow in a Worker and start the caller Workflow](https://docs.temporal.io/develop/typescript/nexus/feature-guide#register-the-caller-workflow-in-a-worker-and-start-the-caller-workflow) * [Exceptions in Nexus operations](https://docs.temporal.io/develop/typescript/nexus/feature-guide#exceptions-in-nexus-operations) * [Canceling a Nexus Operation](https://docs.temporal.io/develop/typescript/nexus/feature-guide#canceling-a-nexus-operation) * [Make Nexus calls across Namespaces in Temporal Cloud](https://docs.temporal.io/develop/typescript/nexus/feature-guide#nexus-calls-across-namespaces-temporal-cloud) * [Install the latest `tcld` CLI and generate certificates](https://docs.temporal.io/develop/typescript/nexus/feature-guide#install-the-latest-tcld-cli-and-generate-certificates) * [Create caller and handler Namespaces](https://docs.temporal.io/develop/typescript/nexus/feature-guide#create-caller-and-handler-namespaces) * [Create a Nexus Endpoint to route requests from caller to handler](https://docs.temporal.io/develop/typescript/nexus/feature-guide#create-a-nexus-endpoint-to-route-requests-from-caller-to-handler) * [Observability](https://docs.temporal.io/develop/typescript/nexus/feature-guide#observability) * [Web UI](https://docs.temporal.io/develop/typescript/nexus/feature-guide#web-ui) * [Temporal CLI](https://docs.temporal.io/develop/typescript/nexus/feature-guide#temporal-cli) * [OpenTelemetry](https://docs.temporal.io/develop/typescript/nexus/feature-guide#opentelemetry) * [Learn more](https://docs.temporal.io/develop/typescript/nexus/feature-guide#learn-more) --- # Self-hosted Visibility feature setup | Temporal Platform Documentation [Skip to main content](https://docs.temporal.io/self-hosted-guide/visibility#__docusaurus_skipToContent_fallback) On this page A [Visibility](https://docs.temporal.io/temporal-service/visibility) store is set up as a part of your [Persistence store](https://docs.temporal.io/temporal-service/persistence) to enable listing and filtering details about Workflow Executions that exist on your Temporal Service. A Visibility store is required in a Temporal Service setup because it is used by [Temporal Web UI](https://docs.temporal.io/web-ui) and [Temporal CLI](https://docs.temporal.io/cli) to pull [Workflow Execution](https://docs.temporal.io/workflow-execution) data and enables features like batch operations on a group of Workflow Executions. With the Visibility store, you can use [List Filters](https://docs.temporal.io/list-filter) with [Search Attributes](https://docs.temporal.io/search-attribute) to list and filter Workflow Executions that you want to review or act upon. Supported Visibility stores include: * Elasticsearch v7 with Temporal Server v1.7 and later * Elasticsearch v8 with Temporal Server v1.18 and later * OpenSearch 2+ with Temporal Server v1.30.1 and later * MySQL v8.0.17 and later with Temporal Server v1.20 and later * PostgreSQL v12 and later with Temporal Server v1.20 and later * SQLite v3.31.0 and later with Temporal Server v1.20 and later Current and legacy Visibility support[​](https://docs.temporal.io/self-hosted-guide/visibility#supported-databases "Direct link to Current and legacy Visibility support") --------------------------------------------------------------------------------------------------------------------------------------------------------------------------- [Advanced Visibility](https://docs.temporal.io/visibility#advanced-visibility) is the current generation of Temporal Visibility. It supports the modern query model, including [custom Search Attributes](https://docs.temporal.io/search-attribute#custom-search-attribute) . This page also includes guidance for the legacy (deprecated in Temporal Server v1.21 and removed in v1.24) [standard Visibility](https://docs.temporal.io/self-hosted-guide/visibility#legacy-standard-visibility) model for older deployments and migration work. In this context, "advanced" and "standard (legacy)" refer to the current and legacy generations of Temporal Visibility, respectively. The following compatibility matrix summarizes which generation of Visibility each store supports and the Temporal Server versions required: | Store | Advanced Visibility (current) | Standard Visibility (legacy) | | --- | --- | --- | | [Elasticsearch](https://docs.temporal.io/self-hosted-guide/visibility#elasticsearch)


Recommended for any setup that spawns more than a few Workflow Executions | v7 on Temporal Server v1.7+, v8 on Temporal Server v1.18+ | Not supported | | [OpenSearch](https://docs.temporal.io/self-hosted-guide/visibility#elasticsearch) | 2+ on Temporal Server v1.30.1+ | Not supported | | [MySQL](https://docs.temporal.io/self-hosted-guide/visibility#mysql) | v8.0.17+ on Temporal Server v1.20+ | v5.7+ on older deployments before Temporal Server v1.24 | | [PostgreSQL](https://docs.temporal.io/self-hosted-guide/visibility#postgresql) | v12+ on Temporal Server v1.20+ | v9.6+ on older deployments before Temporal Server v1.24 | | [SQLite](https://docs.temporal.io/self-hosted-guide/visibility#sqlite) | v3.31.0+ on Temporal Server v1.20+ | Not supported | | [Cassandra](https://docs.temporal.io/self-hosted-guide/visibility#cassandra) | Not supported.

To migrate from Cassandra to a supported advanced Visibility store, see [Migrating Visibility database](https://docs.temporal.io/self-hosted-guide/visibility#migrating-visibility-database)
. | Deprecated in Temporal Server v1.21, removed in Temporal Server v1.24 | You can use any combination of the supported databases for your Persistence and Visibility stores. For updates, check [Server release notes](https://github.com/temporalio/temporal/releases) . Temporal Server v1.21 introduced support for a secondary Visibility store in your Temporal Service to enable [Dual Visibility](https://docs.temporal.io/dual-visibility) . This is useful for migrating your Visibility store database. How to set up MySQL Visibility store[​](https://docs.temporal.io/self-hosted-guide/visibility#mysql "Direct link to How to set up MySQL Visibility store") ----------------------------------------------------------------------------------------------------------------------------------------------------------- Support, stability, and dependency info * MySQL v5.7 and later. * Advanced Visibility is available on MySQL v8.0.17 and later with Temporal Server v1.20 and later. * MySQL v5.7 support applied to older standard Visibility deployments before Temporal Server v1.24. You can set MySQL as your [Visibility store](https://docs.temporal.io/temporal-service/visibility) . Verify [supported versions](https://docs.temporal.io/self-hosted-guide/visibility) before you proceed. If using MySQL v8.0.17 or later as your Visibility store with Temporal Server v1.20 and later, any [custom Search Attributes](https://docs.temporal.io/search-attribute#custom-search-attribute) that you create must be associated with a Namespace in that Temporal Service. ### Persistence configuration[​](https://docs.temporal.io/self-hosted-guide/visibility#persistence-configuration "Direct link to Persistence configuration") Set your MySQL Visibility store name in the `visibilityStore` parameter in your Persistence configuration, and then define the Visibility store configuration under `datastores`. The following example shows how to set a Visibility store `mysql-visibility` and define the datastore configuration in your Temporal Service configuration YAML. #...persistence: #... visibilityStore: mysql-visibility #... datastores: default: #... mysql-visibility: sql: pluginName: 'mysql8' # For MySQL v8.0.17 and later. For earlier versions, use "mysql" plugin. databaseName: 'temporal_visibility' connectAddr: ' ' # Remote address of this database; for example, 127.0.0.0:3306 connectProtocol: ' ' # Protocol example: tcp user: 'username_for_auth' password: 'password_for_auth' maxConns: 2 maxIdleConns: 2 maxConnLifetime: '1h'#... For details on the configuration parameters and values, see [Temporal Service configuration](https://docs.temporal.io/references/configuration#sql) . To enable advanced Visibility features on your MySQL Visibility store, upgrade to MySQL v8.0.17 or later with Temporal Server v1.20 or later. See [Upgrade Server](https://docs.temporal.io/self-hosted-guide/upgrade-server#upgrade-server) on how to upgrade your Temporal Server and database schemas. For example configuration templates, see [MySQL Visibility store configuration](https://github.com/temporalio/temporal/blob/main/config/development-mysql8.yaml) . ### Database schema and setup[​](https://docs.temporal.io/self-hosted-guide/visibility#database-schema-and-setup "Direct link to Database schema and setup") Visibility data is stored in a database table called `executions_visibility` and must be created using the schema for [MySQL v8.0.17 and later](https://github.com/temporalio/temporal/tree/main/schema/mysql/v8/visibility) . The following example shows how to set up your MySQL as both your persistence and Visibility store using `temporal-sql-tool`. Refer to the [samples-server repository](https://github.com/temporalio/samples-server/tree/main/compose/scripts) for more examples with different databases. [compose/scripts/setup-mysql.sh](https://github.com/temporalio/samples-server/blob/main/compose/scripts/setup-mysql.sh) set -eu# Validate required environment variables: "${MYSQL_SEEDS:?ERROR: MYSQL_SEEDS environment variable is required}": "${MYSQL_USER:?ERROR: MYSQL_USER environment variable is required}"echo 'Starting MySQL schema setup...'echo 'Waiting for MySQL port to be available...'nc -z -w 10 ${MYSQL_SEEDS} ${DB_PORT:-3306}echo 'MySQL port is available'# Create and setup temporal databasetemporal-sql-tool --plugin mysql8 --ep ${MYSQL_SEEDS} -u ${MYSQL_USER} -p ${DB_PORT:-3306} --db temporal createtemporal-sql-tool --plugin mysql8 --ep ${MYSQL_SEEDS} -u ${MYSQL_USER} -p ${DB_PORT:-3306} --db temporal setup-schema -v 0.0temporal-sql-tool --plugin mysql8 --ep ${MYSQL_SEEDS} -u ${MYSQL_USER} -p ${DB_PORT:-3306} --db temporal update-schema -d /etc/temporal/schema/mysql/v8/temporal/versioned# Create and setup visibility databasetemporal-sql-tool --plugin mysql8 --ep ${MYSQL_SEEDS} -u ${MYSQL_USER} -p ${DB_PORT:-3306} --db temporal_visibility createtemporal-sql-tool --plugin mysql8 --ep ${MYSQL_SEEDS} -u ${MYSQL_USER} -p ${DB_PORT:-3306} --db temporal_visibility setup-schema -v 0.0temporal-sql-tool --plugin mysql8 --ep ${MYSQL_SEEDS} -u ${MYSQL_USER} -p ${DB_PORT:-3306} --db temporal_visibility update-schema -d /etc/temporal/schema/mysql/v8/visibility/versionedecho 'MySQL schema setup complete' Note that the script uses [temporal-sql-tool](https://github.com/temporalio/temporal/blob/3b982585bf0124839e697952df4bba01fe4d9543/tools/sql/main.go) to run the setup. How to set up PostgreSQL Visibility store[​](https://docs.temporal.io/self-hosted-guide/visibility#postgresql "Direct link to How to set up PostgreSQL Visibility store") -------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Support, stability, and dependency info * PostgreSQL v9.6 and later. * Advanced Visibility is available on PostgreSQL v12 and later with Temporal Server v1.20 and later. * PostgreSQL v9.6 through v11 support applied to older standard Visibility deployments before Temporal Server v1.24. We recommend upgrading to PostgreSQL 12 or later. You can set PostgreSQL as your [Visibility store](https://docs.temporal.io/temporal-service/visibility) . Verify [supported versions](https://docs.temporal.io/self-hosted-guide/visibility) before you proceed. If using PostgreSQL v12 or later as your Visibility store with Temporal Server v1.20 and later, any [custom Search Attributes](https://docs.temporal.io/search-attribute#custom-search-attribute) that you create must be associated with a Namespace in that Temporal Service. ### Persistence configuration[​](https://docs.temporal.io/self-hosted-guide/visibility#persistence-configuration-1 "Direct link to Persistence configuration") Set your PostgreSQL Visibility store name in the `visibilityStore` parameter in your Persistence configuration, and then define the Visibility store configuration under `datastores`. The following example shows how to set a Visibility store `postgres-visibility` and define the datastore configuration in your Temporal Service configuration YAML. #...persistence: #... visibilityStore: postgres-visibility #... datastores: default: #... postgres-visibility: sql: pluginName: 'postgres12' # For PostgreSQL v12 and later. For earlier versions, use "postgres" plugin. databaseName: 'temporal_visibility' connectAddr: ' ' # remote address of this database; for example, 127.0.0.0:5432 connectProtocol: ' ' # protocol example: tcp user: 'username_for_auth' password: 'password_for_auth' maxConns: 2 maxIdleConns: 2 maxConnLifetime: '1h'#... To enable advanced Visibility features on your PostgreSQL Visibility store, upgrade to PostgreSQL v12 or later with Temporal Server v1.20 or later. See [Upgrade Server](https://docs.temporal.io/self-hosted-guide/upgrade-server#upgrade-server) for details on how to upgrade your Temporal Server and database schemas. ### Database schema and setup[​](https://docs.temporal.io/self-hosted-guide/visibility#database-schema-and-setup-1 "Direct link to Database schema and setup") Visibility data is stored in a database table called `executions_visibility` and must be created using the schema for [PostgreSQL v12 and later](https://github.com/temporalio/temporal/tree/main/schema/postgresql/v12/visibility) The following example shows how to set up your PostgreSQL as both persistence and Visibility store using `temporal-sql-tool`. Refer to the [samples-server repository](https://github.com/temporalio/samples-server/tree/main/compose/scripts) for more examples with different databases. [compose/scripts/setup-postgres.sh](https://github.com/temporalio/samples-server/blob/main/compose/scripts/setup-postgres.sh) set -eu# Validate required environment variables: "${POSTGRES_SEEDS:?ERROR: POSTGRES_SEEDS environment variable is required}": "${POSTGRES_USER:?ERROR: POSTGRES_USER environment variable is required}"echo 'Starting PostgreSQL schema setup...'echo 'Waiting for PostgreSQL port to be available...'nc -z -w 10 ${POSTGRES_SEEDS} ${DB_PORT:-5432}echo 'PostgreSQL port is available'# Create and setup temporal databasetemporal-sql-tool --plugin postgres12 --ep ${POSTGRES_SEEDS} -u ${POSTGRES_USER} -p ${DB_PORT:-5432} --db temporal createtemporal-sql-tool --plugin postgres12 --ep ${POSTGRES_SEEDS} -u ${POSTGRES_USER} -p ${DB_PORT:-5432} --db temporal setup-schema -v 0.0temporal-sql-tool --plugin postgres12 --ep ${POSTGRES_SEEDS} -u ${POSTGRES_USER} -p ${DB_PORT:-5432} --db temporal update-schema -d /etc/temporal/schema/postgresql/v12/temporal/versioned# Create and setup visibility databasetemporal-sql-tool --plugin postgres12 --ep ${POSTGRES_SEEDS} -u ${POSTGRES_USER} -p ${DB_PORT:-5432} --db temporal_visibility createtemporal-sql-tool --plugin postgres12 --ep ${POSTGRES_SEEDS} -u ${POSTGRES_USER} -p ${DB_PORT:-5432} --db temporal_visibility setup-schema -v 0.0temporal-sql-tool --plugin postgres12 --ep ${POSTGRES_SEEDS} -u ${POSTGRES_USER} -p ${DB_PORT:-5432} --db temporal_visibility update-schema -d /etc/temporal/schema/postgresql/v12/visibility/versionedecho 'PostgreSQL schema setup complete' Note that the script uses [temporal-sql-tool](https://github.com/temporalio/temporal/blob/3b982585bf0124839e697952df4bba01fe4d9543/tools/sql/main.go) to run the setup. How to set up SQLite Visibility store[​](https://docs.temporal.io/self-hosted-guide/visibility#sqlite "Direct link to How to set up SQLite Visibility store") -------------------------------------------------------------------------------------------------------------------------------------------------------------- Support, stability, and dependency info * SQLite v3.31.0 and later. You can set SQLite as your [Visibility store](https://docs.temporal.io/temporal-service/visibility) . Verify [supported versions](https://docs.temporal.io/self-hosted-guide/visibility) before you proceed. Temporal supports only an in-memory database with SQLite; this means that the database is automatically created when Temporal Server starts and is destroyed when Temporal Server stops. You can change the configuration to use a file-based database so that it is preserved when Temporal Server stops. However, if you use a file-based SQLite database, upgrading your database schema to enable advanced Visibility features is not supported; in this case, you must delete the database and create it again to upgrade. If using SQLite v3.31.0 and later as your Visibility store with Temporal Server v1.20 and later, any [custom Search Attributes](https://docs.temporal.io/search-attribute#custom-search-attribute) that you create must be associated with a Namespace in that Temporal Service. ### Persistence configuration[​](https://docs.temporal.io/self-hosted-guide/visibility#persistence-configuration-2 "Direct link to Persistence configuration") Set your SQLite Visibility store name in the `visibilityStore` parameter in your Persistence configuration, and then define the Visibility store configuration under `datastores`. The following example shows how to set a Visibility store `sqlite-visibility` and define the datastore configuration in your Temporal Service configuration YAML. persistence: # ... visibilityStore: sqlite-visibility # ... datastores: # ... sqlite-visibility: sql: user: 'username_for_auth' password: 'password_for_auth' pluginName: 'sqlite' databaseName: 'default' connectAddr: 'localhost' connectProtocol: 'tcp' connectAttributes: mode: 'memory' cache: 'private' maxConns: 1 maxIdleConns: 1 maxConnLifetime: '1h' tls: enabled: false caFile: '' certFile: '' keyFile: '' enableHostVerification: false serverName: '' SQLite (v3.31.0 and later) has advanced Visibility enabled by default. ### Database schema and setup[​](https://docs.temporal.io/self-hosted-guide/visibility#database-schema-and-setup-2 "Direct link to Database schema and setup") Visibility data is stored in a database table called `executions_visibility` that must be set up according to the schemas defined (by supported versions) in [https://github.com/temporalio/temporal/blob/main/schema/sqlite/v3/visibility/schema.sql](https://github.com/temporalio/temporal/blob/main/schema/sqlite/v3/visibility/schema.sql) . For an example of setting up the SQLite schema, see [Temporalite](https://github.com/temporalio/temporalite/blob/main/server.go) setup. Legacy standard Visibility configuration[​](https://docs.temporal.io/self-hosted-guide/visibility#legacy-standard-visibility "Direct link to Legacy standard Visibility configuration") ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- The following section applies to older self-hosted deployments that still use standard Visibility. For new deployments, use one of the advanced Visibility backends described earlier on this page. ### How to set up Cassandra Visibility store[​](https://docs.temporal.io/self-hosted-guide/visibility#cassandra "Direct link to How to set up Cassandra Visibility store") Support, stability, and dependency info * Cassandra supported only standard Visibility. Standard Visibility was deprecated in Temporal Server v1.21 and removed in v1.24. For updates, check the [Temporal Server release notes](https://github.com/temporalio/temporal/releases) . * We recommend migrating from Cassandra to any of the other supported databases for Visibility. Advanced Visibility is not supported with Cassandra. To enable current Visibility features, use MySQL, PostgreSQL, SQLite, Elasticsearch, or OpenSearch as your Visibility store. We recommend Elasticsearch or OpenSearch for any Temporal Service setup that handles more than a few Workflow Executions because these backends support the Visibility request load and help optimize performance. To migrate from Cassandra to a supported SQL database, see [Migrating Visibility database](https://docs.temporal.io/self-hosted-guide/visibility#migrating-visibility-database) . ### Persistence configuration[​](https://docs.temporal.io/self-hosted-guide/visibility#persistence-configuration-3 "Direct link to Persistence configuration") Set your Cassandra Visibility store name in the `visibilityStore` parameter in your Persistence configuration, and then define the Visibility store configuration under `datastores`. The following example shows how to set a Visibility store `cass-visibility` and define the datastore configuration in your Temporal Service configuration YAML. #...persistence: #... visibilityStore: cass-visibility #... datastores: default: #... cass-visibility: cassandra: hosts: '127.0.0.1' keyspace: 'temporal_visibility'#... ### Database schema and setup[​](https://docs.temporal.io/self-hosted-guide/visibility#database-schema-and-setup-3 "Direct link to Database schema and setup") Visibility data is stored in a database table called `executions_visibility` that must be set up according to the schemas defined (by supported versions) in [https://github.com/temporalio/temporal/tree/main/schema/cassandra/visibility](https://github.com/temporalio/temporal/tree/main/schema/cassandra/visibility) . The following example shows how to set up your Cassandra Visibility store using `temporal-cassandra-tool`. For more examples with different databases, refer to the [samples-server repository](https://github.com/temporalio/samples-server/tree/main/compose/scripts) . #...# set your Cassandra environment variables: "${KEYSPACE:=temporal}": "${VISIBILITY_KEYSPACE:=temporal_visibility}": "${CASSANDRA_SEEDS:=}": "${CASSANDRA_PORT:=9042}": "${CASSANDRA_USER:=}": "${CASSANDRA_PASSWORD:=}": "${CASSANDRA_TLS_ENABLED:=}": "${CASSANDRA_CERT:=}": "${CASSANDRA_CERT_KEY:=}": "${CASSANDRA_CA:=}": "${CASSANDRA_REPLICATION_FACTOR:=1}"#...# set connection details#...# set up Cassandra schemasetup_cassandra_schema() { #... # use valid schema for the version of the database you want to set up for Visibility VISIBILITY_SCHEMA_DIR=${TEMPORAL_HOME}/schema/cassandra/visibility/versioned if [[ ${SKIP_DB_CREATE} != true ]]; then temporal-cassandra-tool --ep "${CASSANDRA_SEEDS}" create -k "${VISIBILITY_KEYSPACE}" --rf "${CASSANDRA_REPLICATION_FACTOR}" fi temporal-cassandra-tool --ep "${CASSANDRA_SEEDS}" -k "${VISIBILITY_KEYSPACE}" setup-schema -v 0.0 temporal-cassandra-tool --ep "${CASSANDRA_SEEDS}" -k "${VISIBILITY_KEYSPACE}" update-schema -d "${VISIBILITY_SCHEMA_DIR}" #...} How to integrate Elasticsearch or OpenSearch into a Temporal Service[​](https://docs.temporal.io/self-hosted-guide/visibility#elasticsearch "Direct link to How to integrate Elasticsearch or OpenSearch into a Temporal Service") ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- You can integrate Elasticsearch or OpenSearch with your Temporal Service as your Visibility store. We recommend using one of these backends for large-scale operations on the Temporal Service. To integrate Elasticsearch or OpenSearch with your Temporal Service, edit the `persistence` section of your `development.yaml` configuration file to add the search backend as the `visibilityStore`, and run the index schema setup commands. Use the following version guidance: * Elasticsearch v7 is supported with Temporal Server v1.7 and later. * Elasticsearch v8 is supported with Temporal Server v1.18 and later. * OpenSearch 2+ is supported with Temporal Server v1.30.1 and later. The examples in this section use Elasticsearch. For OpenSearch, use the same datastore configuration shape and operational flow unless a release note for your target Temporal Server version says otherwise. ### Persistence configuration[​](https://docs.temporal.io/self-hosted-guide/visibility#persistence-configuration-4 "Direct link to Persistence configuration") Set your Visibility store name in the `visibilityStore` parameter in your Persistence configuration, and then define the search backend configuration under `datastores`. The following example shows how to set a Visibility store named `es-visibility` and define the Elasticsearch datastore configuration in your Temporal Service configuration YAML. persistence: ... visibilityStore: es-visibility datastores: ... es-visibility: # Define the Elasticsearch datastore connection information under the `es-visibility` key elasticsearch: version: "v7" url: scheme: "http" host: "127.0.0.1:9200" indices: visibility: temporal_visibility_v1_dev ### Index schema and index[​](https://docs.temporal.io/self-hosted-guide/visibility#index-schema-and-index "Direct link to Index schema and index") To set up Elasticsearch as your Visibility store, use the `temporal-elasticsearch-tool` available in the `temporalio/admin-tools` image. The following example shows how to set up an Elasticsearch Visibility store with a MySQL persistence store using `temporal-elasticsearch-tool`. For more examples with different databases, refer to the [samples-server repository](https://github.com/temporalio/samples-server/tree/main/compose/scripts) . [compose/scripts/setup-mysql-es.sh](https://github.com/temporalio/samples-server/blob/main/compose/scripts/setup-mysql-es.sh) set -eu# Validate required environment variables: "${ES_SCHEME:?ERROR: ES_SCHEME environment variable is required}": "${ES_HOST:?ERROR: ES_HOST environment variable is required}": "${ES_PORT:?ERROR: ES_PORT environment variable is required}": "${ES_VISIBILITY_INDEX:?ERROR: ES_VISIBILITY_INDEX environment variable is required}": "${ES_VERSION:?ERROR: ES_VERSION environment variable is required}": "${MYSQL_SEEDS:?ERROR: MYSQL_SEEDS environment variable is required}": "${MYSQL_USER:?ERROR: MYSQL_USER environment variable is required}"echo 'Starting MySQL and Elasticsearch schema setup...'echo 'Waiting for MySQL port to be available...'nc -z -w 10 ${MYSQL_SEEDS} ${DB_PORT:-3306}echo 'MySQL port is available'# Create and setup temporal databasetemporal-sql-tool --plugin mysql8 --ep ${MYSQL_SEEDS} -u ${MYSQL_USER} -p ${DB_PORT:-3306} --db temporal createtemporal-sql-tool --plugin mysql8 --ep ${MYSQL_SEEDS} -u ${MYSQL_USER} -p ${DB_PORT:-3306} --db temporal setup-schema -v 0.0temporal-sql-tool --plugin mysql8 --ep ${MYSQL_SEEDS} -u ${MYSQL_USER} -p ${DB_PORT:-3306} --db temporal update-schema -d /etc/temporal/schema/mysql/v8/temporal/versioned# Setup Elasticsearch index# temporal-elasticsearch-tool is available in v1.30+ server releasesif [ -x /usr/local/bin/temporal-elasticsearch-tool ]; then echo 'Using temporal-elasticsearch-tool for Elasticsearch setup' temporal-elasticsearch-tool --ep "$ES_SCHEME://$ES_HOST:$ES_PORT" setup-schema temporal-elasticsearch-tool --ep "$ES_SCHEME://$ES_HOST:$ES_PORT" create-index --index $ES_VISIBILITY_INDEXelse echo 'Using curl for Elasticsearch setup' echo 'WARNING: curl will be removed from admin-tools in v1.30.' echo 'Waiting for Elasticsearch to be ready...' max_attempts=30 attempt=0 until curl -s -f "$ES_SCHEME://$ES_HOST:$ES_PORT/_cluster/health?wait_for_status=yellow&timeout=1s"; do attempt=$((attempt + 1)) if [ $attempt -ge $max_attempts ]; then echo "ERROR: Elasticsearch did not become ready after $max_attempts attempts" echo "Last error from curl:" curl "$ES_SCHEME://$ES_HOST:$ES_PORT/_cluster/health?wait_for_status=yellow&timeout=1s" 2>&1 || true exit 1 fi echo "Elasticsearch not ready yet, waiting... (attempt $attempt/$max_attempts)" sleep 2 done echo '' echo 'Elasticsearch is ready' echo 'Creating index template...' curl -X PUT --fail "$ES_SCHEME://$ES_HOST:$ES_PORT/_template/temporal_visibility_v1_template" -H 'Content-Type: application/json' --data-binary "@/etc/temporal/schema/elasticsearch/visibility/index_template_$ES_VERSION.json" echo '' echo 'Creating index...' curl --head --fail "$ES_SCHEME://$ES_HOST:$ES_PORT/$ES_VISIBILITY_INDEX" 2>/dev/null || curl -X PUT --fail "$ES_SCHEME://$ES_HOST:$ES_PORT/$ES_VISIBILITY_INDEX" echo ''fiecho 'MySQL and Elasticsearch setup complete' ### Elasticsearch privileges[​](https://docs.temporal.io/self-hosted-guide/visibility#elasticsearch-privileges "Direct link to Elasticsearch privileges") Ensure that the following privileges are granted for the Elasticsearch Temporal index: * **Read** * [index privileges](https://www.elastic.co/guide/en/elasticsearch/reference/current/security-privileges.html#privileges-list-indices) : `create`, `index`, `delete`, `read` * **Write** * [index privileges](https://www.elastic.co/guide/en/elasticsearch/reference/current/security-privileges.html#privileges-list-indices) : `write` * **Custom Search Attributes** * [index privileges](https://www.elastic.co/guide/en/elasticsearch/reference/current/security-privileges.html#privileges-list-indices) : `manage` * [cluster privileges](https://www.elastic.co/guide/en/elasticsearch/reference/current/security-privileges.html#privileges-list-cluster) : `monitor` or `manage`. How to set up Dual Visibility[​](https://docs.temporal.io/self-hosted-guide/visibility#dual-visibility "Direct link to How to set up Dual Visibility") ------------------------------------------------------------------------------------------------------------------------------------------------------- To enable [Dual Visibility](https://docs.temporal.io/dual-visibility) , set up a secondary Visibility store with your primary Visibility store, and configure your Temporal Service to enable read and/or write operations on the secondary Visibility store. With Dual Visibility, you can read from only one Visibility store at a time, but can configure your Temporal Service to write to primary only, secondary only, or to both primary and secondary stores. #### Set up secondary Visibility store[​](https://docs.temporal.io/self-hosted-guide/visibility#set-up-secondary-visibility-store "Direct link to Set up secondary Visibility store") Set the secondary store with the `secondaryVisibilityStore` configuration key in your Persistence configuration, and then define the secondary Visibility store configuration under `datastores`. You can configure any of the [supported databases](https://docs.temporal.io/self-hosted-guide/visibility) as your secondary store. Examples: To configure MySQL as a secondary store with Cassandra as your primary store, do the following. persistence: visibilityStore: cass-visibility # This is your primary Visibility store secondaryVisibilityStore: mysql-visibility # This is your secondary Visibility store datastores: cass-visibility: cassandra: hosts: '127.0.0.1' keyspace: 'temporal_primary_visibility' mysql-visibility: sql: pluginName: 'mysql8' # Verify supported versions. Use a version of SQL that supports advanced Visibility. databaseName: 'temporal_secondary_visibility' connectAddr: '127.0.0.1:3306' connectProtocol: 'tcp' user: 'temporal' password: 'temporal' To configure Elasticsearch as both your primary and secondary store, use the configuration key `elasticsearch.indices.secondary_visibility`, as shown in the following example. persistence: visibilityStore: es-visibility datastores: es-visibility: elasticsearch: version: 'v7' logLevel: 'error' url: scheme: 'http' host: '127.0.0.1:9200' indices: visibility: temporal_visibility_v1 secondary_visibility: temporal_visibility_v1_new closeIdleConnectionsInterval: 15s #### Database schema and setup[​](https://docs.temporal.io/self-hosted-guide/visibility#database-schema-and-setup-4 "Direct link to Database schema and setup") The database schema and setup for a secondary store depends on the database you plan to use. * [MySQL](https://docs.temporal.io/self-hosted-guide/visibility#mysql) * [PostgreSQL](https://docs.temporal.io/self-hosted-guide/visibility#postgresql) * [SQLite](https://docs.temporal.io/self-hosted-guide/visibility#sqlite) * [Elasticsearch](https://docs.temporal.io/self-hosted-guide/visibility#elasticsearch) For the Cassandra and MySQL configuration in the previous example, an example setup script would be as follows. #...# set your Cassandra environment variables: "${KEYSPACE:=temporal}": "${VISIBILITY_KEYSPACE:=temporal_primary_visibility}": "${CASSANDRA_SEEDS:=}": "${CASSANDRA_PORT:=9042}": "${CASSANDRA_USER:=}": "${CASSANDRA_PASSWORD:=}": "${CASSANDRA_TLS_ENABLED:=}": "${CASSANDRA_CERT:=}": "${CASSANDRA_CERT_KEY:=}": "${CASSANDRA_CA:=}": "${CASSANDRA_REPLICATION_FACTOR:=1}"#...# set connection details#...# set up Cassandra schemasetup_cassandra_schema() { #... # use valid schema for the version of the database you want to set up for Visibility VISIBILITY_SCHEMA_DIR=${TEMPORAL_HOME}/schema/cassandra/visibility/versioned if [[ ${SKIP_DB_CREATE} != true ]]; then temporal-cassandra-tool --ep "${CASSANDRA_SEEDS}" create -k "${VISIBILITY_KEYSPACE}" --rf "${CASSANDRA_REPLICATION_FACTOR}" fi temporal-cassandra-tool --ep "${CASSANDRA_SEEDS}" -k "${VISIBILITY_KEYSPACE}" setup-schema -v 0.0 temporal-cassandra-tool --ep "${CASSANDRA_SEEDS}" -k "${VISIBILITY_KEYSPACE}" update-schema -d "${VISIBILITY_SCHEMA_DIR}" #...}#...# set your MySQL environment variables: "${DBNAME:=temporal}": "${VISIBILITY_DBNAME:=temporal_secondary_visibility}": "${DB_PORT:=}": "${MYSQL_SEEDS:=}": "${MYSQL_USER:=}": "${MYSQL_PWD:=}": "${MYSQL_TX_ISOLATION_COMPAT:=false}"#...# set connection details#...# set up MySQL schemasetup_mysql_schema() { #... # use valid schema for the version of the database you want to set up for Visibility VISIBILITY_SCHEMA_DIR=${TEMPORAL_HOME}/schema/mysql/${MYSQL_VERSION_DIR}/visibility/versioned if [[ ${SKIP_DB_CREATE} != true ]]; then temporal-sql-tool --ep "${MYSQL_SEEDS}" -u "${MYSQL_USER}" -p "${DB_PORT}" "${MYSQL_CONNECT_ATTR[@]}" --db "${VISIBILITY_DBNAME}" create fi temporal-sql-tool --ep "${MYSQL_SEEDS}" -u "${MYSQL_USER}" -p "${DB_PORT}" "${MYSQL_CONNECT_ATTR[@]}" --db "${VISIBILITY_DBNAME}" setup-schema -v 0.0 temporal-sql-tool --ep "${MYSQL_SEEDS}" -u "${MYSQL_USER}" -p "${DB_PORT}" "${MYSQL_CONNECT_ATTR[@]}" --db "${VISIBILITY_DBNAME}" update-schema -d "${VISIBILITY_SCHEMA_DIR}"#...} For Elasticsearch as both primary and secondary Visibility store configuration in the previous example, an example setup script would be as follows. #...# Elasticsearch: "${ENABLE_ES:=false}": "${ES_SCHEME:=http}": "${ES_SEEDS:=}": "${ES_PORT:=9200}": "${ES_USER:=}": "${ES_PWD:=}": "${ES_VERSION:=v7}": "${ES_VIS_INDEX:=temporal_visibility_v1_dev}": "${ES_SEC_VIS_INDEX:=temporal_visibility_v1_new}": "${ES_SCHEMA_SETUP_TIMEOUT_IN_SECONDS:=0}"#...# Validate your ES environment#...# Wait for ES to start#...# Set up Elasticsearch indexsetup_es_index() { ES_SERVER="${ES_SCHEME}://${ES_SEEDS%%,*}:${ES_PORT}" # ES_SERVER is the URL of Elasticsearch server i.e. "http://localhost:9200". SETTINGS_URL="${ES_SERVER}/_cluster/settings" SETTINGS_FILE=${TEMPORAL_HOME}/schema/elasticsearch/visibility/cluster_settings_${ES_VERSION}.json TEMPLATE_URL="${ES_SERVER}/_template/temporal_visibility_v1_template" SCHEMA_FILE=${TEMPORAL_HOME}/schema/elasticsearch/visibility/index_template_${ES_VERSION}.json INDEX_URL="${ES_SERVER}/${ES_VIS_INDEX}" curl --fail --user "${ES_USER}":"${ES_PWD}" -X PUT "${SETTINGS_URL}" -H "Content-Type: application/json" --data-binary "@${SETTINGS_FILE}" --write-out "\n" curl --fail --user "${ES_USER}":"${ES_PWD}" -X PUT "${TEMPLATE_URL}" -H 'Content-Type: application/json' --data-binary "@${SCHEMA_FILE}" --write-out "\n" curl --user "${ES_USER}":"${ES_PWD}" -X PUT "${INDEX_URL}" --write-out "\n" # Checks for and sets up Elasticsearch as a secondary Visibility store if [[ ! -z "${ES_SEC_VIS_INDEX}" ]]; then SEC_INDEX_URL="${ES_SERVER}/${ES_SEC_VIS_INDEX}" curl --user "${ES_USER}":"${ES_PWD}" -X PUT "${SEC_INDEX_URL}" --write-out "\n" fi} #### Update Temporal Service configuration[​](https://docs.temporal.io/self-hosted-guide/visibility#update-temporal-service-configuration "Direct link to Update Temporal Service configuration") With the primary and secondary stores set, update the `system.secondaryVisibilityWritingMode` and `system.enableReadFromSecondaryVisibility` configuration keys in your self-hosted Temporal Service's dynamic configuration YAML file to enable read and/or write operations to the secondary Visibility store. For example, to enable write operations to both primary and secondary stores, but disable reading from the secondary store, use the following. system.secondaryVisibilityWritingMode: - value: 'dual' constraints: {}system.enableReadFromSecondaryVisibility: - value: false constraints: {} For details on the configuration options, see: * [Secondary Visibility dynamic configuration reference](https://docs.temporal.io/references/dynamic-configuration#secondary-visibility-settings) * [Migrating Visibility databases](https://docs.temporal.io/self-hosted-guide/visibility#migrating-visibility-database) How to migrate Visibility database[​](https://docs.temporal.io/self-hosted-guide/visibility#migrating-visibility-database "Direct link to How to migrate Visibility database") ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- To migrate your Visibility database, [set up a secondary Visibility store](https://docs.temporal.io/self-hosted-guide/visibility#dual-visibility) to enable [Dual Visibility](https://docs.temporal.io/dual-visibility) , and update the dynamic configuration in your Temporal Service to update the read and write operations for the Visibility store. Dual Visibility setup is optional but useful in gradually migrating your Visibility data to another database. Before you begin, verify [supported databases and versions](https://docs.temporal.io/self-hosted-guide/visibility) for a Visibility store. The following steps describe how to migrate your Visibility database. After you make any changes to your [Temporal Service configuration](https://docs.temporal.io/temporal-service/configuration) , ensure that you restart your services. #### Set up secondary Visibility store[​](https://docs.temporal.io/self-hosted-guide/visibility#set-up-secondary-visibility-store-1 "Direct link to Set up secondary Visibility store") 1. In your Temporal Service configuration, [add a secondary Visibility store](https://docs.temporal.io/references/configuration#secondaryvisibilitystore) to your Visibility setup under the Persistence configuration. Example: To migrate from Cassandra to Elasticsearch, add Elasticsearch as your secondary database and set it up. For details, see [secondary Visibility database schema and setup](https://docs.temporal.io/self-hosted-guide/visibility#dual-visibility) . persistence:visibilityStore: cass-visibilitysecondaryVisibilityStore: es-visibilitydatastores: cass-visibility: cassandra: hosts: '127.0.0.1' keyspace: 'temporal_visibility' es-visibility: elasticsearch: version: 'v7' logLevel: 'error' url: scheme: 'http' host: '127.0.0.1:9200' indices: visibility: temporal_visibility_v1_dev closeIdleConnectionsInterval: 15s 2. Update the [dynamic configuration](https://docs.temporal.io/temporal-service/configuration#dynamic-configuration) keys on your self-hosted Temporal Service to enable write operations to the secondary store and disable read operations. Example: system.secondaryVisibilityWritingMode:- value: "dual"constraints: {}system.enableReadFromSecondaryVisibility:- value: falseconstraints: {} At this point, Visibility data is read from the primary store, and all Visibility data is written to both the primary and secondary store. This setting applies only to new Visibility data generated after Dual Visibility is enabled. It does not migrate any existing data in the primary store to the secondary store. For details on write options to the secondary store, see [Secondary Visibility dynamic configuration reference](https://docs.temporal.io/references/dynamic-configuration#secondary-visibility-settings) . #### Run in dual mode[​](https://docs.temporal.io/self-hosted-guide/visibility#run-in-dual-mode "Direct link to Run in dual mode") When you enable a secondary store, only new Visibility data is written to both primary and secondary stores. The primary store still holds the Workflow Execution data from before the secondary store was set up. Running in dual mode lets you plan for closed and open Workflow Executions data from before the secondary store was set up in your self-hosted Temporal Service. Example: * To manage closed Workflow Executions data, run in dual mode until the Namespace [Retention Period](https://docs.temporal.io/temporal-service/temporal-server#retention-period) is reached. After the Retention Period, Workflow Execution data is removed from the Persistence and Visibility stores. If you want to keep the closed Workflow Executions data after the set Retention Period, you must set up [Archival](https://docs.temporal.io/self-hosted-guide/archival) . * To manage data for all open Workflow Executions, run in dual mode until all the Workflow Executions started before enabling Dual Visibility mode are closed. After the Workflow Executions are closed, verify the Retention Period and set up Archival if you need to keep the data beyond the Retention Period. You can run your Visibility setup in dual mode for an indefinite period, or until you are ready to deprecate the primary store and move completely to the secondary store without losing data. #### Deprecate primary Visibility store[​](https://docs.temporal.io/self-hosted-guide/visibility#deprecate-primary-visibility-store "Direct link to Deprecate primary Visibility store") When you are ready to deprecate your primary store, follow these steps. 1. Update the dynamic configuration YAML to enable read operations from the secondary store. Example: system.secondaryVisibilityWritingMode:- value: "dual"constraints: {}system.enableReadFromSecondaryVisibility:- value: trueconstraints: {} At this point, Visibility data is read from the secondary store only. Verify whether data on the secondary store is correct. 2. When the secondary store is vetted and ready to replace your current primary store, change your Temporal Service configuration to set the secondary store as your primary, and remove the dynamic configuration set in the previous steps. Example: persistence:visibilityStore: es-visibilitydatastores: es-visibility: elasticsearch: version: 'v7' logLevel: 'error' url: scheme: 'http' host: '127.0.0.1:9200' indices: visibility: temporal_visibility_v1_dev closeIdleConnectionsInterval: 15s Managing custom Search Attributes[​](https://docs.temporal.io/self-hosted-guide/visibility#custom-search-attributes "Direct link to Managing custom Search Attributes") ------------------------------------------------------------------------------------------------------------------------------------------------------------------------ To manage custom Search Attributes on Temporal Cloud, use the [`tcld`](https://docs.temporal.io/cloud/tcld/namespace#search-attributes) CLI tool. With Temporal Cloud, you can create and rename custom Search Attributes. If you need to delete a custom Search Attribute, contact Support at [support.temporal.io](https://support.temporal.io/) . To manage custom Search Attributes on a self-hosted Temporal Service, use the [Temporal CLI](https://docs.temporal.io/cli/operator#search-attribute) . With a self-hosted Temporal Service, you can create and remove custom Search Attributes. If you're self-hosting, verify whether your [Visibility database](https://docs.temporal.io/self-hosted-guide/visibility#supported-databases) version supports custom Search Attributes before proceeding. Do not use sensitive data or PII in Search Attributes Do not include sensitive data, secrets, or personally identifiable information (PII) in Search Attribute **names or values**. Search Attribute values are stored unencrypted in the Visibility store and are not processed by a custom [Payload Codec](https://docs.temporal.io/payload-codec#payload-codec) . The Temporal Server must be able to read these values in plain text to support filtering and ordering, so encryption is not possible without breaking search functionality. Attribute names are also visible in Namespace configuration, query expressions, and Temporal UI. Using sensitive data in either names or values risks exposure to anyone with Namespace access and may violate data protection regulations such as GDPR, HIPAA, or SOC 2. ### How to create custom Search Attributes[​](https://docs.temporal.io/self-hosted-guide/visibility#create-custom-search-attributes "Direct link to How to create custom Search Attributes") Creating a custom Search Attribute in your Visibility store makes it available to use in your Workflow metadata and [List Filters](https://docs.temporal.io/list-filter) . **On Temporal Cloud** To create custom Search Attributes on Temporal Cloud, use [`tcld namespace search-attributes add`](https://docs.temporal.io/cloud/tcld/namespace/#search-attributes) . For example, to add a custom Search Attributes "CustomSA" to your Temporal Cloud Namespace "YourNamespace", run the following command. `tcld namespace search-attributes add --namespace YourNamespace --search-attribute "CustomSA"` **On self-hosted Temporal Service** To create custom Search Attributes in your self-hosted Temporal Service Visibility store, use `temporal operator search-attribute create` with `--name` and `--type` command options. For example, to create a Search Attribute called `CustomSA` of type `Keyword`, run the following command: temporal operator search-attribute create --name="CustomSA" --type="Keyword" Note that if you use a SQL database with advanced Visibility capabilities, you are required to specify a Namespace when creating a custom Search Attribute. For example: temporal operator search-attribute create --name="CustomSA" --type="Keyword" --namespace="yournamespace" You can also create multiple custom Search Attributes when you set up your Visibility store. The following example shows how custom Search Attributes can be created during Visibility store setup for SQL databases. For setup examples, refer to the [samples-server repository](https://github.com/temporalio/samples-server) add_custom_search_attributes() { until temporal operator search-attribute list --namespace "${DEFAULT_NAMESPACE}"; do echo "Waiting for namespace cache to refresh..." sleep 1 done echo "Namespace cache refreshed." echo "Adding Custom*Field search attributes." temporal operator search-attribute create --namespace "${DEFAULT_NAMESPACE}" --yes \ --name="CustomKeywordField" --type="Keyword" \ --name="CustomStringField" --type="Text" \ --name="CustomTextField" --type="Text" \ --name="CustomIntField" --type="Int" \ --name="CustomDatetimeField" --type="Datetime" \ --name="CustomDoubleField" --type="Double" \ --name="CustomBoolField" --type="Bool"} For Temporal Server v1.19 and earlier, or if using Elasticsearch for advanced Visibility, you can create custom Search Attributes without a Namespace association, as shown in the following example. add_custom_search_attributes() { echo "Adding Custom*Field search attributes." temporal operator search-attribute create \ --name="CustomKeywordField" --type="Keyword" \ --name="CustomStringField" --type="Text" \ --name="CustomTextField" --type="Text" \ --name="CustomIntField" --type="Int" \ --name="CustomDatetimeField" --type="Datetime" \ --name="CustomDoubleField" --type="Double" \ --name="CustomBoolField" --type="Bool"} When your Visibility store is set up and running, these custom Search Attributes are available to use in your Workflow code. ### How to remove custom Search Attributes[​](https://docs.temporal.io/self-hosted-guide/visibility#remove-custom-search-attributes "Direct link to How to remove custom Search Attributes") To remove a Search Attribute key from your self-hosted Temporal Service Visibility store, use the command `temporal operator search-attribute remove`. Removing Search Attributes is not supported on Temporal Cloud. For example, if using Elasticsearch for advanced Visibility, to remove a custom Search Attribute called `CustomSA` of type Keyword use the following command: temporal operator search-attribute remove \ --name="your_custom_attribute" If you use a SQL database for advanced Visibility on Temporal Server v1.20 and later, you need to specify the Namespace in your command, as shown in the following command: temporal operator search-attribute remove \ --name="your_custom_attribute" \ --namespace="your_namespace" To check whether the Search Attribute was removed, run temporal operator search-attribute list and check the list. If you're on Temporal Server v1.20 and later, specify the Namespace from which you removed the Search Attribute. For example, temporal search-attribute list --namespace="yournamespace" Note that if you use [SQL databases](https://docs.temporal.io/self-hosted-guide/visibility) with Temporal Server v1.20 and later, a new custom Search Attribute is mapped to a database field name in the Visibility store `custom_search_attributes` table. Removing this custom Search Attribute removes the mapping with the database field name but does not remove the data. If you remove a custom Search Attribute and add a new one, the new custom Search Attribute might be mapped to the database field of the one that was recently removed. This might cause unexpected results when you use the List API to retrieve results using the new custom Search Attribute. These constraints do not apply if you use Elasticsearch. * [Current and legacy Visibility support](https://docs.temporal.io/self-hosted-guide/visibility#supported-databases) * [How to set up MySQL Visibility store](https://docs.temporal.io/self-hosted-guide/visibility#mysql) * [Persistence configuration](https://docs.temporal.io/self-hosted-guide/visibility#persistence-configuration) * [Database schema and setup](https://docs.temporal.io/self-hosted-guide/visibility#database-schema-and-setup) * [How to set up PostgreSQL Visibility store](https://docs.temporal.io/self-hosted-guide/visibility#postgresql) * [Persistence configuration](https://docs.temporal.io/self-hosted-guide/visibility#persistence-configuration-1) * [Database schema and setup](https://docs.temporal.io/self-hosted-guide/visibility#database-schema-and-setup-1) * [How to set up SQLite Visibility store](https://docs.temporal.io/self-hosted-guide/visibility#sqlite) * [Persistence configuration](https://docs.temporal.io/self-hosted-guide/visibility#persistence-configuration-2) * [Database schema and setup](https://docs.temporal.io/self-hosted-guide/visibility#database-schema-and-setup-2) * [Legacy standard Visibility configuration](https://docs.temporal.io/self-hosted-guide/visibility#legacy-standard-visibility) * [How to set up Cassandra Visibility store](https://docs.temporal.io/self-hosted-guide/visibility#cassandra) * [Persistence configuration](https://docs.temporal.io/self-hosted-guide/visibility#persistence-configuration-3) * [Database schema and setup](https://docs.temporal.io/self-hosted-guide/visibility#database-schema-and-setup-3) * [How to integrate Elasticsearch or OpenSearch into a Temporal Service](https://docs.temporal.io/self-hosted-guide/visibility#elasticsearch) * [Persistence configuration](https://docs.temporal.io/self-hosted-guide/visibility#persistence-configuration-4) * [Index schema and index](https://docs.temporal.io/self-hosted-guide/visibility#index-schema-and-index) * [Elasticsearch privileges](https://docs.temporal.io/self-hosted-guide/visibility#elasticsearch-privileges) * [How to set up Dual Visibility](https://docs.temporal.io/self-hosted-guide/visibility#dual-visibility) * [Set up secondary Visibility store](https://docs.temporal.io/self-hosted-guide/visibility#set-up-secondary-visibility-store) * [Database schema and setup](https://docs.temporal.io/self-hosted-guide/visibility#database-schema-and-setup-4) * [Update Temporal Service configuration](https://docs.temporal.io/self-hosted-guide/visibility#update-temporal-service-configuration) * [How to migrate Visibility database](https://docs.temporal.io/self-hosted-guide/visibility#migrating-visibility-database) * [Set up secondary Visibility store](https://docs.temporal.io/self-hosted-guide/visibility#set-up-secondary-visibility-store-1) * [Run in dual mode](https://docs.temporal.io/self-hosted-guide/visibility#run-in-dual-mode) * [Deprecate primary Visibility store](https://docs.temporal.io/self-hosted-guide/visibility#deprecate-primary-visibility-store) * [Managing custom Search Attributes](https://docs.temporal.io/self-hosted-guide/visibility#custom-search-attributes) * [How to create custom Search Attributes](https://docs.temporal.io/self-hosted-guide/visibility#create-custom-search-attributes) * [How to remove custom Search Attributes](https://docs.temporal.io/self-hosted-guide/visibility#remove-custom-search-attributes) --- # Dynamic Workflow - Ruby SDK | Temporal Platform Documentation [Skip to main content](https://docs.temporal.io/develop/ruby/workflows/dynamic-workflow#__docusaurus_skipToContent_fallback) On this page Set a Dynamic Workflow[​](https://docs.temporal.io/develop/ruby/workflows/dynamic-workflow#set-a-dynamic-workflow "Direct link to Set a Dynamic Workflow") ----------------------------------------------------------------------------------------------------------------------------------------------------------- A Dynamic Workflow in Temporal is a Workflow that is invoked dynamically at runtime if no other Workflow with the same name is registered. A Workflow can be made dynamic by invoking `workflow_dynamic` class method at the top of the definition. You must register the Workflow with the Worker before it can be invoked. Only one Dynamic Workflow can be present on a Worker. Often, dynamic is used in conjunction with `workflow_raw_args` which does not convert arguments but instead passes them through as a splatted array of `Temporalio::Converters::RawValue` instances. class MyDynamicWorkflow < Temporalio::Workflow::Definition # Make this the dynamic workflow and accept raw args workflow_dynamic workflow_raw_args def execute(*raw_args) # Require a single arg for our workflow raise Temporalio::Error::ApplicationError, 'One arg expected' unless raw_args.size == 1 # Use payload converter to convert it name = Temporalio::Workflow.payload_converter.from_payload(raw_args.first.payload) Temporalio::Workflow.execute_activity( MyActivity, { greeting: 'Hello', name: }, start_to_close_timeout: 100 ) endend * [Set a Dynamic Workflow](https://docs.temporal.io/develop/ruby/workflows/dynamic-workflow#set-a-dynamic-workflow) --- # Observability - Ruby SDK | Temporal Platform Documentation [Skip to main content](https://docs.temporal.io/develop/ruby/platform/observability#__docusaurus_skipToContent_fallback) On this page This page covers capabilities related to viewing the state of the application, including: * [Metrics](https://docs.temporal.io/develop/ruby/platform/observability#metrics) * [Tracing](https://docs.temporal.io/develop/ruby/platform/observability#tracing) * [Logging](https://docs.temporal.io/develop/ruby/platform/observability#logging) * [Visibility](https://docs.temporal.io/develop/ruby/platform/observability#visibility) The observability guide covers the many ways to view the current state of your [Temporal Application](https://docs.temporal.io/temporal#temporal-application) . This includes viewing [Workflow Executions](https://docs.temporal.io/workflow-execution) tracked by the [Temporal Platform](https://docs.temporal.io/temporal#temporal-platform) , as well as inspecting state at any point during execution. Emit metrics[​](https://docs.temporal.io/develop/ruby/platform/observability#metrics "Direct link to Emit metrics") -------------------------------------------------------------------------------------------------------------------- Each Temporal SDK can optionally emit metrics from either the Client or Worker process. Metrics can be scraped by systems like Prometheus, and graphs can be created using tools like Grafana. * For an overview of Prometheus and Grafana integration, refer to the [Monitoring](https://docs.temporal.io/self-hosted-guide/monitoring) guide. * For a list of metrics, see the [SDK metrics reference](https://docs.temporal.io/references/sdk-metrics) . Metrics in Ruby are configured on the `metrics` argument of the `telemetry` argument when creating a global `Temporalio::Runtime`. That object should be created globally and should be used for all clients; therefore, you should configure this before any other Temporal code. Set a Prometheus endpoint[​](https://docs.temporal.io/develop/ruby/platform/observability#set-a-prometheus-endpoint "Direct link to Set a Prometheus endpoint") ---------------------------------------------------------------------------------------------------------------------------------------------------------------- The following example exposes a Prometheus endpoint on port `9000`. Temporalio::Runtime.default = Temporalio::Runtime.new( telemetry: Temporalio::Runtime::TelemetryOptions.new( metrics: Temporalio::Runtime::MetricsOptions.new( prometheus: Temporalio::Runtime::PrometheusMetricsOptions.new( bind_address: '0.0.0.0:9000' ) ) )) ### Custom metric handling[​](https://docs.temporal.io/develop/ruby/platform/observability#custom-metric-handling "Direct link to Custom metric handling") Instead of Prometheus or OpenTelemetry, an instance of `Temporalio::Runtime::MetricBuffer` can be provided as a `buffer` argument to the `MetricsOptions`. `retrieve_updates` can then be periodically called on the buffer to get metric updates. Setup Tracing[​](https://docs.temporal.io/develop/ruby/platform/observability#tracing "Direct link to Setup Tracing") ---------------------------------------------------------------------------------------------------------------------- Tracing enables observability into the sequence of calls across your application, including Workflows and Activities. OpenTelemetry tracing for clients, activities, and workflows can be enabled using the `Temporalio::Contrib::OpenTelemetry::TracingInterceptor`. Specifically, when creating a client, set the interceptor like so: require 'opentelemetry/api'require 'opentelemetry/sdk'require 'temporalio/client'require 'temporalio/contrib/open_telemetry'# ... assumes my_otel_tracer_provider is a tracer provider created by the usermy_tracer = my_otel_tracer_provider.tracer('my-otel-tracer')my_client = Temporalio::Client.connect( 'localhost:7233', 'my-namespace', interceptors: [Temporalio::Contrib::OpenTelemetry::TracingInterceptor.new(my_tracer)]) When your Client is connected, spans are created for all Client calls, Activities, and Workflow invocations on the Worker. Spans are created and serialized through the server to give one trace for a Workflow Execution. Log from a Workflow[​](https://docs.temporal.io/develop/ruby/platform/observability#logging "Direct link to Log from a Workflow") ---------------------------------------------------------------------------------------------------------------------------------- Logging enables you to capture and persist important execution details from your Workflow and Activity code. Logging levels typically include: | Level | Use | | --- | --- | | `DEBUG` | Detailed information, typically useful for debugging purposes. | | `INFO` | General information about the application's operation. | | `WARN` | Indicates potentially harmful situations or minor issues that don't prevent the application from working. | | `ERROR` | Indicates error conditions that might still allow the application to continue running. | Logging uses the Ruby standard logging APIs. The `logger` can be set when connecting a client. The following example shows logging on the console and sets the level to `INFO`. require 'logger'require 'temporalio/client'my_client = Temporalio::Client.connect( 'localhost:7233', 'my-namespace', logger: Logger.new($stdout, level: Logger::INFO)) You can log from a Workflow using `Temporalio::Workflow.logger` which is a special instance of Ruby's `Logger` that appends workflow details to every log and does not log during replay. Temporalio::Workflow.logger.info("Some log #{some_value}") There's also one for use in activities that appends Activity details to every log: Temporalio::Activity::Context.current.logger.info("Some log #{some_value}") Use Visibility APIs[​](https://docs.temporal.io/develop/ruby/platform/observability#visibility "Direct link to Use Visibility APIs") ------------------------------------------------------------------------------------------------------------------------------------- Visibility refers to Temporal features for listing, filtering, and inspecting Workflow Executions. ### Use Search Attributes[​](https://docs.temporal.io/develop/ruby/platform/observability#search-attributes "Direct link to Use Search Attributes") * [Default Search Attributes](https://docs.temporal.io/search-attribute#default-search-attribute) like `WorkflowType`, `StartTime`, and `ExecutionStatus` are automatically indexed. * [Custom Search Attributes](https://docs.temporal.io/search-attribute#custom-search-attribute) let you store domain-specific metadata for Workflows. The typical method of retrieving a Workflow Execution is by its Workflow Id. However, sometimes you'll want to retrieve one or more Workflow Executions based on another property. For example, imagine you want to get all Workflow Executions of a certain type that have failed within a time range, so that you can start new ones with the same arguments. You can do this with [Search Attributes](https://docs.temporal.io/search-attribute) . * [Default Search Attributes](https://docs.temporal.io/search-attribute#default-search-attribute) like `WorkflowType`, `StartTime` and `ExecutionStatus` are automatically added to Workflow Executions. * [Custom Search Attributes](https://docs.temporal.io/search-attribute#custom-search-attribute) can contain their own domain-specific data (like `customerId` or `numItems`). The steps to using custom Search Attributes are: * Create a new Search Attribute in your Temporal Service in the CLI or Web UI. * For example: `temporal operator search-attribute create --name CustomKeywordField --type Text` * Replace `CustomKeywordField` with the name of your Search Attribute. * Replace `Text` with a type value associated with your Search Attribute: `Text` | `Keyword` | `Int` | `Double` | `Bool` | `Datetime` | `KeywordList` * Set the value of the Search Attribute for a Workflow Execution: * On the Client by including it as an argument when starting the Execution. * In the Workflow by calling `Temporalio::Workflow.upsert_search_attributes`. * Read the value of the Search Attribute: * On the Client by calling `describe` on a `WorkflowHandle`. * In the Workflow by looking at `Temporalio::Workflow.search_attributes`. * Query Workflow Executions by the Search Attribute using a [List Filter](https://docs.temporal.io/list-filter) : * [In the Temporal CLI](https://docs.temporal.io/cli/operator#list-2) * In code by calling `list_workflows`. ### List Workflow Executions[​](https://docs.temporal.io/develop/ruby/platform/observability#list-workflow-executions "Direct link to List Workflow Executions") Use the [list\_workflows](https://ruby.temporal.io/Temporalio/Client.html#list_workflows-instance_method) method on the Client and pass a [List Filter](https://docs.temporal.io/list-filter) as an argument to filter the listed Workflows. The result is a lazy enumerator/enumerable. my_client.list_workflows("WorkflowType='GreetingWorkflow'").each do |wf| puts "Workflow: #{wf.id}"end ### Set Custom Search Attributes[​](https://docs.temporal.io/develop/ruby/platform/observability#custom-search-attributes "Direct link to Set Custom Search Attributes") After you've created custom Search Attributes in your Temporal Service (using `temporal operator search-attribute create`or the Cloud UI), you can set the values of the custom Search Attributes when starting a Workflow. To set custom Search Attributes, use the `search_attributes` parameter for `start_workflow` or `execute_workflow`. Keys should be predefined for reuse. # Predefined search attribute key, usually a global somewhereMY_KEYWORD_KEY = Temporalio::SearchAttributes::Key.new( 'my-keyword', Temporalio::SearchAttributes::IndexedValueType::KEYWORD)# ...# Start workflow with the search attribute sethandle = my_client.start_workflow( MyWorkflow, 'some-input', id: 'my-workflow-id', task_queue: 'my-task-queue', search_attributes: Temporalio::SearchAttributes.new({ MY_KEYWORD_KEY => 'some-value' })) ### Upsert Search Attributes[​](https://docs.temporal.io/develop/ruby/platform/observability#upsert-search-attributes "Direct link to Upsert Search Attributes") You can upsert Search Attributes to add, update, or remove Search Attributes from within Workflow code. To upsert custom Search Attributes, use the [`upsert_search_attributes`](https://ruby.temporal.io/Temporalio/Workflow.html#upsert_search_attributes-class_method) method with a set of updates. Keys should be predefined for reuse. # Predefined search attribute key, usually a global somewhereMY_KEYWORD_KEY = Temporalio::SearchAttributes::Key.new( 'my-keyword', Temporalio::SearchAttributes::IndexedValueType::KEYWORD)# ...class MyWorkflow < Temporalio::Workflow::Definition def execute # ... Temporalio::Workflow.upsert_search_attributes(MY_KEYWORD_KEY.value_set('some-new-value')) # ... endend * [Emit metrics](https://docs.temporal.io/develop/ruby/platform/observability#metrics) * [Set a Prometheus endpoint](https://docs.temporal.io/develop/ruby/platform/observability#set-a-prometheus-endpoint) * [Custom metric handling](https://docs.temporal.io/develop/ruby/platform/observability#custom-metric-handling) * [Setup Tracing](https://docs.temporal.io/develop/ruby/platform/observability#tracing) * [Log from a Workflow](https://docs.temporal.io/develop/ruby/platform/observability#logging) * [Use Visibility APIs](https://docs.temporal.io/develop/ruby/platform/observability#visibility) * [Use Search Attributes](https://docs.temporal.io/develop/ruby/platform/observability#search-attributes) * [List Workflow Executions](https://docs.temporal.io/develop/ruby/platform/observability#list-workflow-executions) * [Set Custom Search Attributes](https://docs.temporal.io/develop/ruby/platform/observability#custom-search-attributes) * [Upsert Search Attributes](https://docs.temporal.io/develop/ruby/platform/observability#upsert-search-attributes) --- # Temporal Visibility | Temporal Platform Documentation [Skip to main content](https://docs.temporal.io/visibility#__docusaurus_skipToContent_fallback) On this page This page provides an overview of Temporal Visibility. The term [Visibility](https://docs.temporal.io/visibility) , within the Temporal Platform, refers to the subsystems and APIs that enable an operator to view, filter, and search for Workflow Executions that currently exist within a Temporal Service. The [Visibility store](https://docs.temporal.io/self-hosted-guide/visibility) in your Temporal Service stores persisted Workflow Execution Event History data and is set up as a part of your [Persistence store](https://docs.temporal.io/temporal-service/persistence) to enable listing and filtering details about Workflow Executions that exist on your Temporal Service. * [How to set up a Visibility store](https://docs.temporal.io/self-hosted-guide/visibility) With Temporal Server v1.21, you can set up [Dual Visibility](https://docs.temporal.io/dual-visibility) to migrate your Visibility store from one database to another. Visibility features[​](https://docs.temporal.io/visibility#advanced-visibility "Direct link to Visibility features") --------------------------------------------------------------------------------------------------------------------- Visibility enables the listing, filtering, and sorting of [Workflow Executions](https://docs.temporal.io/workflow-execution) through a custom SQL-like [List Filter](https://docs.temporal.io/list-filter) . Visibility supports [custom Search Attributes](https://docs.temporal.io/search-attribute#custom-search-attribute) for user-defined filtering beyond the default system attributes. * On SQL databases (MySQL v8.0.17+, PostgreSQL v12+), Visibility is available with Temporal Server v1.20 and later. * On Elasticsearch (v7+ with Temporal Server v1.7+, v8 with Temporal Server v1.18+) and OpenSearch (2+ with Temporal Server v1.30.1+). * On Temporal Cloud, Visibility is enabled by default for [all users](https://docs.temporal.io/cloud/users#invite-users) . For self-hosted setup and version compatibility details, see [Visibility store setup](https://docs.temporal.io/self-hosted-guide/visibility) . ### Count Workflow by ExecutionStatus[​](https://docs.temporal.io/visibility#count-workflow-by-executionstatus "Direct link to Count Workflow by ExecutionStatus") The Count API feature lets you count the number of Workflows that match a given query. For example, the command `temporal workflow count -q "WorkflowType='foo'"` returns the number of Workflows that match `WorkflowType='foo'`. You can send queries to the Count API to group by a given search attribute. For example, `-q "WorkflowType='foo' GROUP BY ExecutionStatus` returns the number of Workflows that match `WorkflowType='foo'` grouped by `ExecutionStatus`. The `GROUP BY` clause is only supported in the Count API and currently only grouping by `ExecutionStatus` is supported. note The Count API returns approximate counts. Legacy: standard Visibility[​](https://docs.temporal.io/visibility#standard-visibility "Direct link to Legacy: standard Visibility") ------------------------------------------------------------------------------------------------------------------------------------- Prior to Temporal Server v1.20, Temporal had two Visibility modes: "standard" and "advanced." Standard Visibility supported only predefined filters such as Workflow Type, Workflow Id, Run Id, and Execution Status, without custom Search Attributes. Advanced Visibility required Elasticsearch. Starting with Temporal Server v1.20, advanced Visibility became available on SQL databases. Standard Visibility was deprecated in v1.21 and removed in v1.24. All current deployments use what was formerly called advanced Visibility, now simply called Visibility. If you are running a Temporal Server version older than v1.24, see the [legacy standard Visibility section](https://docs.temporal.io/self-hosted-guide/visibility#legacy-standard-visibility) in the self-hosted guide. * [Visibility features](https://docs.temporal.io/visibility#advanced-visibility) * [Count Workflow by ExecutionStatus](https://docs.temporal.io/visibility#count-workflow-by-executionstatus) * [Legacy: standard Visibility](https://docs.temporal.io/visibility#standard-visibility) --- # Nexus Operations | Temporal Platform Documentation [Skip to main content](https://docs.temporal.io/nexus/operations#__docusaurus_skipToContent_fallback) On this page SUPPORT, STABILITY, and DEPENDENCY INFO Temporal Nexus is now [Generally Available](https://docs.temporal.io/evaluate/development-production-features/release-stages#general-availability) for [Temporal Cloud](https://docs.temporal.io/cloud/nexus) and [self-hosted deployments](https://docs.temporal.io/production-deployment/self-hosted-guide/nexus) . [Nexus Operations](https://docs.temporal.io/glossary#nexus-operation) can be synchronous or asynchronous. Unlike a traditional RPC, an asynchronous Nexus Operation has an operation token that can be used to re-attach to a long-running Operation backed by a Workflow. An Operation's lifecycle spans scheduling, reliable delivery with retries, handler execution, and result or callback completion. SDK support[​](https://docs.temporal.io/nexus/operations#sdk-support "Direct link to SDK support") --------------------------------------------------------------------------------------------------- SDK GUIDES * [Go](https://docs.temporal.io/develop/go/nexus/feature-guide) | [Java](https://docs.temporal.io/develop/java/nexus) | [Python](https://docs.temporal.io/develop/python/nexus) | [TypeScript](https://docs.temporal.io/develop/typescript/nexus) | [.NET](https://docs.temporal.io/develop/dotnet/nexus) **Caller side:** A caller Workflow executes a Nexus Operation through a [Nexus Endpoint](https://docs.temporal.io/nexus/endpoints) using the Temporal SDK. **Handler side:** [Nexus Services](https://docs.temporal.io/nexus/services) and their Operations are registered with a Worker that polls the Endpoint's target Task Queue. Operations are defined using SDK builder functions: * **New-Workflow-Run-Operation** - Start a Workflow as an asynchronous Operation. * **New-Sync-Operation** - Run a synchronous Operation: invoke a Query, Signal, or Update, or execute other reliable code using the Temporal SDK Client. Nexus Operation lifecycle[​](https://docs.temporal.io/nexus/operations#operation-lifecycle "Direct link to Nexus Operation lifecycle") --------------------------------------------------------------------------------------------------------------------------------------- When a caller Workflow executes a Nexus Operation, the command is atomically handed off to the [Nexus Machinery](https://docs.temporal.io/glossary#nexus-machinery) . The Machinery ensures [at-least-once](https://docs.temporal.io/nexus/operations#at-least-once-execution-semantics-and-idempotency) execution with [automatic retries](https://docs.temporal.io/nexus/operations#automatic-retries) and reliable result delivery. ![Nexus Overview](https://docs.temporal.io/img/cloud/nexus/nexus-overview.png) Nexus Overview ### Synchronous Operation lifecycle[​](https://docs.temporal.io/nexus/operations#synchronous-operation-lifecycle "Direct link to Synchronous Operation lifecycle") Synchronous Operations must complete within the [10-second handler deadline](https://docs.temporal.io/cloud/limits#nexus-operation-request-timeout) , as measured from the caller's Nexus Machinery. ![Nexus Sync Operation Lifecycle](https://docs.temporal.io/img/cloud/nexus/nexus-sync-operation.png) Nexus Sync Operation Lifecycle Lifecycle for a synchronous Operation (for example, to Signal, Query, or Update a Workflow, or to run other reliable code): 1. Caller Workflow executes a Nexus Operation. 2. Caller Worker issues a [ScheduleNexusOperation](https://docs.temporal.io/references/commands#schedulenexusoperation) command. 3. Caller Namespace records a [NexusOperationScheduled](https://docs.temporal.io/references/events#nexusoperationscheduled) event. 4. Caller Nexus Machinery sends the start request. 5. Handler Nexus Machinery sync-matches the request to a handler Worker. 6. Handler Worker receives a [Nexus Task](https://docs.temporal.io/tasks#nexus-task) by polling the Endpoint's target Task Queue. 7. Handler processes the task using **New-Sync-Operation**. 8. Handler responds with the Operation result. 9. Caller Namespace records a [Completed](https://docs.temporal.io/references/events#nexusoperationcompleted) or [Failed](https://docs.temporal.io/references/events#nexusoperationfailed) event. 10. Caller Worker polls for a Workflow Task. 11. Caller Workflow receives the result. ![Nexus](https://docs.temporal.io/img/cloud/nexus/nexus-workers-short-sync-op-sequence.png) Nexus tip Stay within the [request deadline](https://docs.temporal.io/cloud/limits#nexus-operation-request-timeout) to avoid timeouts. Timed-out handlers are retried until the Operation's Schedule-to-Close timeout is exceeded. ### Asynchronous Operation lifecycle[​](https://docs.temporal.io/nexus/operations#asynchronous-operation-lifecycle "Direct link to Asynchronous Operation lifecycle") Asynchronous Operations can run up to [60 days](https://docs.temporal.io/cloud/limits#nexus-operation-duration-limits) (the maximum Schedule-to-Close timeout in Temporal Cloud). Differences from the synchronous lifecycle are in **bold**. ![Nexus Async Operation Lifecycle](https://docs.temporal.io/img/cloud/nexus/nexus-async-operation.png) Nexus Async Operation Lifecycle 1. Caller Workflow executes a Nexus Operation. 2. Caller Worker issues a [ScheduleNexusOperation](https://docs.temporal.io/references/commands#schedulenexusoperation) command. 3. Caller Namespace records a [NexusOperationScheduled](https://docs.temporal.io/references/events#nexusoperationscheduled) event. 4. Caller Nexus Machinery sends the start request. 5. Handler Nexus Machinery sync-matches the request to a handler Worker. 6. Handler Worker receives a [Nexus Task](https://docs.temporal.io/tasks#nexus-task) by polling the Endpoint's target Task Queue. 7. Handler processes the task using **New-Workflow-Run-Operation**. 8. Handler responds with the **start Operation response**. 9. Caller Namespace records a **[NexusOperationStarted](https://docs.temporal.io/references/events#nexusoperationstarted) ** event. 10. **Handler Workflow completes and a [Nexus Completion Callback](https://docs.temporal.io/glossary#nexus-async-completion-callback) is delivered to the caller's Nexus Machinery.** 11. Caller Namespace records a [Completed](https://docs.temporal.io/references/events#nexusoperationcompleted) or [Failed](https://docs.temporal.io/references/events#nexusoperationfailed) event. 12. Caller Worker polls for a Workflow Task. 13. Caller Workflow receives the result. ![Nexus](https://docs.temporal.io/img/cloud/nexus/nexus-workers-short-async-op-sequence.png) Nexus ### Executing code from a synchronous handler[​](https://docs.temporal.io/nexus/operations#executing-arbitrary-code-from-a-sync-handler "Direct link to Executing code from a synchronous handler") Synchronous handlers can execute code directly but must complete within the [handler deadline](https://docs.temporal.io/cloud/limits#nexus-operation-request-timeout) . Use the Temporal SDK Client to invoke Signals, Queries, Updates, or other reliable code. caution Use [async Operations](https://docs.temporal.io/nexus/operations#asynchronous-operation-lifecycle) for long-running work. Repeated sync handler failures can trip the [circuit breaker](https://docs.temporal.io/nexus/operations#circuit-breaking) , blocking all Operations from that caller to the Endpoint. ![Nexus Operations with Arbitrary Code](https://docs.temporal.io/img/cloud/nexus/nexus-sync-operation-arbitrary-code.png) Nexus Operations with Arbitrary Code ### System interactions[​](https://docs.temporal.io/nexus/operations#system-interactions "Direct link to System interactions") Nexus uses the same queue-based Worker architecture as the rest of Temporal. Workers interact with their Namespace gRPC endpoint. Nexus Machinery on both sides handles cross-Namespace communication. ![Nexus Queue-based Worker Architecture](https://docs.temporal.io/img/cloud/nexus/nexus-workers.png) Nexus Queue-based Worker Architecture At a high level, when a caller Workflow executes a Nexus Operation: 1. The caller Worker schedules the Operation with a [ScheduleNexusOperation command](https://docs.temporal.io/references/commands#schedulenexusoperation) , atomically handing off execution to the caller's Nexus Machinery. 2. The handler Worker receives a [Nexus Task](https://docs.temporal.io/tasks#nexus-task) by polling the Endpoint's target Task Queue. 3. The handler processes the task and returns the result (synchronous) or an Operation token (asynchronous). 4. The caller's Nexus Machinery records a NexusOperation event ([Started](https://docs.temporal.io/references/events#nexusoperationstarted) , [Completed](https://docs.temporal.io/references/events#nexusoperationcompleted) , [Failed](https://docs.temporal.io/references/events#nexusoperationfailed) , [Canceled](https://docs.temporal.io/references/events#nexusoperationcanceled) , or [TimedOut](https://docs.temporal.io/references/events#nexusoperationtimedout) ) in the caller's Workflow History. Automatic retries[​](https://docs.temporal.io/nexus/operations#automatic-retries "Direct link to Automatic retries") --------------------------------------------------------------------------------------------------------------------- Once the caller Workflow schedules an Operation with the caller's Temporal Service, the caller's Nexus Machinery keeps trying to start the Operation. If a [retryable Nexus error](https://docs.temporal.io/references/failures#nexus-errors) is returned the Nexus Machinery will retry until the Nexus Operation's [Schedule-to-Start timeout](https://docs.temporal.io/nexus/operations#schedule-to-start-timeout) or [Schedule-to-close timeout](https://docs.temporal.io/nexus/operations#schedule-to-close-timeout) is exceeded. For example, if a Nexus handler returns a [retryable error](https://docs.temporal.io/references/failures#nexus-errors) , or an [upstream timeout](https://github.com/nexus-rpc/api/blob/main/SPEC.md#predefined-handler-errors) is encountered by the caller, the Nexus request will be retried up to the [default Retry Policy's](https://github.com/temporalio/temporal/blob/de7c8879e103be666a7b067cc1b247f0ac63c25c/components/nexusoperations/config.go#L111) max attempts and expiration interval. note This differs from Activity and Workflow error handling. See [errors in Activities](https://docs.temporal.io/references/failures#errors-in-activities) and [non-retryable errors](https://docs.temporal.io/references/failures#non-retryable) . To control retry behavior, return a [non-retryable Nexus error](https://docs.temporal.io/references/failures#non-retryable-nexus-errors) . See [errors in Nexus handlers](https://docs.temporal.io/nexus/error-handling#errors-in-nexus-handlers) . Timeouts[​](https://docs.temporal.io/nexus/operations#timeouts "Direct link to Timeouts") ------------------------------------------------------------------------------------------ Nexus Operations support three types of timeouts that control how long the caller is willing to wait at different stages of the Operation lifecycle. These timeouts are set by the caller when scheduling the Operation. ### Schedule-to-Close timeout[​](https://docs.temporal.io/nexus/operations#schedule-to-close-timeout "Direct link to Schedule-to-Close timeout") The Schedule-to-Close timeout limits the total duration from when the Operation is scheduled to when it completes. This is the overall timeout for the entire Operation. The Nexus Machinery [automatically retries](https://docs.temporal.io/nexus/operations#automatic-retries) failed requests internally until this timeout is exceeded, at which point the Operation fails with a [NexusOperationTimedOut](https://docs.temporal.io/references/events#nexusoperationtimedout) event. This timeout covers the full [Nexus Operation lifecycle](https://docs.temporal.io/nexus/operations#operation-lifecycle) . Asynchronous Operations are scheduled, started, and completed. Synchronous Operations don't have an intermediate started state because they complete as part of the start request. In Temporal Cloud, the [maximum Schedule-to-Close timeout is 60 days](https://docs.temporal.io/cloud/limits#nexus-operation-duration-limits) . ### Schedule-to-Start timeout[​](https://docs.temporal.io/nexus/operations#schedule-to-start-timeout "Direct link to Schedule-to-Start timeout") The Schedule-to-Start timeout limits how long the caller is willing to wait for the Operation to be started (or completed, if synchronous) by the handler. If the Operation is not started within this timeout, it fails with `TIMEOUT_TYPE_SCHEDULE_TO_START`. If not set or set to zero, no Schedule-to-Start timeout is enforced. note The Schedule-to-Start timeout requires Temporal Server version 1.31.0 or later. ### Start-to-Close timeout[​](https://docs.temporal.io/nexus/operations#start-to-close-timeout "Direct link to Start-to-Close timeout") The Start-to-Close timeout limits how long the caller is willing to wait for an asynchronous Operation to complete after it has been started. If the Operation does not complete within this timeout after starting, it fails with `TIMEOUT_TYPE_START_TO_CLOSE`. This timeout only applies to asynchronous Operations. Synchronous Operations ignore this timeout because they complete as part of the start request. If not set or set to zero, no Start-to-Close timeout is enforced. note The Start-to-Close timeout requires Temporal Server version 1.31.0 or later. Circuit breaking[​](https://docs.temporal.io/nexus/operations#circuit-breaking "Direct link to Circuit breaking") ------------------------------------------------------------------------------------------------------------------ Nexus implements circuit breaking per caller-Namespace/Endpoint pair ("destination pair"). Each destination pair trips and resets independently. By default, the circuit breaker activates after 5 consecutive [retryable errors](https://docs.temporal.io/references/failures#nexus-errors) . After tripping, the circuit breaker enters the _open_ state and stops sending requests. After 60 seconds, it transitions to _half-open_, allowing a single probe request. If the probe succeeds, the circuit breaker returns to _closed_ (normal operation). If it fails, the circuit breaker returns to _open_ for another 60 seconds. note Note that worker availability affects the circuit breaker as well. If no workers are polling the handler task queue — due to a deployment issue, crash, or scale-down — Nexus requests will time out. Consecutive timeouts count as retryable errors and will trip the circuit breaker just as application-level errors do. Ensure handler workers maintain sufficient availability to avoid unintended circuit breaker trips. ![Flow chart showing the states of the Temporal Nexus Circuit Breaker](https://docs.temporal.io/img/cloud/nexus/circuit-breaker.png) Flow chart showing the states of the Temporal Nexus Circuit Breaker Circuit breaker state surfaces in [Pending Nexus Operations](https://docs.temporal.io/nexus/execution-debugging#pending-operations) and [Pending Callbacks](https://docs.temporal.io/nexus/execution-debugging#pending-callbacks) . Check it in the UI, CLI, or `DescribeWorkflowExecution` API. When open, pending Operations show a `Blocked` state with a `BlockedReason`: ![Circuit Breaking](https://docs.temporal.io/img/cloud/nexus/circuit-breaking.png) Circuit Breaking Different Operations within the same destination pair contribute to the trip count. A given Operation may have fewer than 5 attempts when the circuit breaker opens. From the CLI: temporal workflow describe -w my-workflow-id Pending Nexus Operations: 1 Endpoint my-nexus-endpoint Service nexus-playground Operation sync-op-ok State Blocked Attempt 1 LastAttemptFailure {"message":"handler error (UPSTREAM_TIMEOUT): upstream timeout",...} BlockedReason The circuit breaker is open. Cancellation requests surface the same pattern with `CancelationState: Blocked` and `CancelationBlockedReason`. Execution Info: WorkflowId my-workflow-id ...Pending Activities: 0Pending Child Workflows: 0Pending Nexus Operations: 1 Endpoint my-nexus-endpoint Service nexus-playground Operation async-op-workflow-wait-for-cancel OperationToken eyJ2IjowLCJ0IjoxLCJucyI6Im5zIiwid2lkIjoidyJ State Started Attempt 1 ScheduleToCloseTimeout 1d 0h 0m 0s LastAttemptCompleteTime 51 seconds ago CancelationState Blocked CancelationAttempt 5 CancelationRequestedTime 37 seconds ago CancelationLastAttemptCompleteTime 27 seconds ago CancelationLastAttemptFailure {"message":"handler error (UPSTREAM_TIMEOUT): upstream timeout","cause":{"message":"upstream timeout","applicationFailureInfo":{"type":"NexusFailure"}},"applicationFailureInfo":{"type":"NexusHandlerError"}} CancelationBlockedReason The circuit breaker is open. Execution semantics[​](https://docs.temporal.io/nexus/operations#execution-semantics "Direct link to Execution semantics") --------------------------------------------------------------------------------------------------------------------------- ### At-least-once execution semantics and idempotency[​](https://docs.temporal.io/nexus/operations#at-least-once-execution-semantics-and-idempotency "Direct link to At-least-once execution semantics and idempotency") The Nexus Machinery provides reliable execution with at-least-once execution semantics for a Nexus Operation, until the caller's [Schedule-to-Close timeout](https://docs.temporal.io/nexus/operations#schedule-to-close-timeout) is exceeded, at which time the overall Nexus Operation times out. The Machinery retries on handler timeouts or retryable errors, so a handler may be invoked multiple times for the same Operation. Nexus Operation handlers should be idempotent, similar to Activities. Not strictly required in all cases, but highly recommended. ### Exactly-once execution semantics[​](https://docs.temporal.io/nexus/operations#exactly-once-execution-semantics "Direct link to Exactly-once execution semantics") To upgrade to exactly-once, back your Operation with a Workflow that uses a WorkflowIDReusePolicy of RejectDuplicates. This allows only one Workflow Execution per Workflow ID within a Namespace for the Retention Period. Cancelation[​](https://docs.temporal.io/nexus/operations#cancelation "Direct link to Cancelation") --------------------------------------------------------------------------------------------------- Cancelling a caller Workflow automatically propagates to all pending Nexus Operations and their underlying handler Workflows. A canceled handler Workflow reports a [Canceled Failure](https://docs.temporal.io/references/failures#cancelled-failure) to the caller. Termination[​](https://docs.temporal.io/nexus/operations#termination "Direct link to Termination") --------------------------------------------------------------------------------------------------- Terminating a caller Workflow abandons all pending Nexus Operations. Unlike cancellation, no cancel request is sent to the handler Namespace, so handler Workflows continue running indefinitely, consuming resources until they time out or are manually stopped. Because the handler runs in a separate Namespace, it has no signal that the caller is gone, making orphaned Operations difficult to detect and correlate. If the Nexus Operation was part of a multi-step process, termination also leaves no opportunity to run compensation logic, potentially leaving the system in a partially completed state. Prefer [cancellation](https://docs.temporal.io/nexus/operations#cancelation) when possible. Versioning[​](https://docs.temporal.io/nexus/operations#versioning "Direct link to Versioning") ------------------------------------------------------------------------------------------------ Task Routing is the simplest way to version Nexus service code. For backward-incompatible changes, use a different Service name and Task Queue (for example, `prod.payments.v2`). Callers migrate to the new version on their own deployment schedule. Attaching multiple Nexus callers to a handler Workflow[​](https://docs.temporal.io/nexus/operations#attaching-multiple-nexus-callers "Direct link to Attaching multiple Nexus callers to a handler Workflow") -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Operations started with [New-Workflow-Run-Operation](https://docs.temporal.io/nexus/operations#sdk-support) automatically attach a completion Callback to the handler Workflow. Additional callers can attach to the same handler Workflow using a [Conflict-Policy of Use-Existing](https://docs.temporal.io/workflow-execution/workflowid-runid#workflow-id-conflict-policy) . Each handler Workflow has a [Callback limit](https://docs.temporal.io/workflow-execution/limits#workflow-execution-callback-limits) (configurable for self-hosted, see [Cloud limits](https://docs.temporal.io/cloud/limits#per-workflow-callback-limits) for Temporal Cloud). Callers that exceed the limit receive an error. When a handler Workflow uses [Continue-As-New](https://docs.temporal.io/workflow-execution/continue-as-new) , existing completion Callbacks are copied to the new Execution. The previous Execution's Callbacks remain in `Standby` state indefinitely. * [SDK support](https://docs.temporal.io/nexus/operations#sdk-support) * [Nexus Operation lifecycle](https://docs.temporal.io/nexus/operations#operation-lifecycle) * [Synchronous Operation lifecycle](https://docs.temporal.io/nexus/operations#synchronous-operation-lifecycle) * [Asynchronous Operation lifecycle](https://docs.temporal.io/nexus/operations#asynchronous-operation-lifecycle) * [Executing code from a synchronous handler](https://docs.temporal.io/nexus/operations#executing-arbitrary-code-from-a-sync-handler) * [System interactions](https://docs.temporal.io/nexus/operations#system-interactions) * [Automatic retries](https://docs.temporal.io/nexus/operations#automatic-retries) * [Timeouts](https://docs.temporal.io/nexus/operations#timeouts) * [Schedule-to-Close timeout](https://docs.temporal.io/nexus/operations#schedule-to-close-timeout) * [Schedule-to-Start timeout](https://docs.temporal.io/nexus/operations#schedule-to-start-timeout) * [Start-to-Close timeout](https://docs.temporal.io/nexus/operations#start-to-close-timeout) * [Circuit breaking](https://docs.temporal.io/nexus/operations#circuit-breaking) * [Execution semantics](https://docs.temporal.io/nexus/operations#execution-semantics) * [At-least-once execution semantics and idempotency](https://docs.temporal.io/nexus/operations#at-least-once-execution-semantics-and-idempotency) * [Exactly-once execution semantics](https://docs.temporal.io/nexus/operations#exactly-once-execution-semantics) * [Cancelation](https://docs.temporal.io/nexus/operations#cancelation) * [Termination](https://docs.temporal.io/nexus/operations#termination) * [Versioning](https://docs.temporal.io/nexus/operations#versioning) * [Attaching multiple Nexus callers to a handler Workflow](https://docs.temporal.io/nexus/operations#attaching-multiple-nexus-callers) --- # Temporal Client - Ruby SDK | Temporal Platform Documentation [Skip to main content](https://docs.temporal.io/develop/ruby/client/temporal-client#__docusaurus_skipToContent_fallback) On this page A [Temporal Client](https://docs.temporal.io/encyclopedia/temporal-sdks#temporal-client) enables you to communicate with the Temporal Service. Communication with a Temporal Service lets you perform actions such as starting Workflow Executions, sending Signals and Queries to Workflow Executions, getting Workflow results, and more. This page shows you how to do the following using the Ruby SDK with the Temporal Client: * [Connect to a local development Temporal Service](https://docs.temporal.io/develop/ruby/client/temporal-client#connect-to-development-service) * [Connect to Temporal Cloud](https://docs.temporal.io/develop/ruby/client/temporal-client#connect-to-temporal-cloud) * [Start a Workflow Execution](https://docs.temporal.io/develop/ruby/client/temporal-client#start-workflow) * [Get Workflow results](https://docs.temporal.io/develop/ruby/client/temporal-client#get-workflow-results) A Temporal Client cannot be initialized and used inside a Workflow. However, it is acceptable and common to use a Temporal Client inside an Activity to communicate with a Temporal Service. Connect to development Temporal Service[​](https://docs.temporal.io/develop/ruby/client/temporal-client#connect-to-development-service "Direct link to Connect to development Temporal Service") ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Use [`Client.connect`](https://ruby.temporal.io/Temporalio/Client.html#connect-class_method) to create a client. Connection options include the Temporal Server address, Namespace, and (optionally) TLS configuration. You can provide these options directly in code, load them from **environment variables**, or a **TOML configuration file** using the [`EnvConfig`](https://ruby.temporal.io/Temporalio/EnvConfig.html) helpers. We recommend environment variables or a configuration file for secure, repeatable configuration. When you’re running a Temporal Service locally (such as with the [Temporal CLI dev server](https://docs.temporal.io/cli/server#start-dev) ), the required options are minimal. If you don't specify a host/port, most connections default to `127.0.0.1:7233` and the `default` Namespace. * Configuration File * Environment Variables * Code You can use a TOML configuration file to set connection options for the Temporal Client. The configuration file lets you configure multiple profiles, each with its own set of connection options. You can then specify which profile to use when creating the Temporal Client. You can use the environment variable `TEMPORAL_CONFIG_FILE` to specify the location of the TOML file or provide the path to the file directly in code. If you don't provide the configuration file path, the SDK looks for it at the path `~/.config/temporalio/temporal.toml` or the equivalent on your OS. Refer to [Environment Configuration](https://docs.temporal.io/references/client-environment-configuration) for more details about configuration files and profiles. info The connection options set in configuration files have lower precedence than environment variables. This means that if you set the same option in both the configuration file and as an environment variable, the environment variable value overrides the option set in the configuration file. For example, the following TOML configuration file defines two profiles: `default` and `prod`. Each profile has its own set of connection options. config.toml # Default profile for local development[profile.default]address = "localhost:7233"namespace = "default"# Optional: Add custom gRPC headers[profile.default.grpc_meta]my-custom-header = "development-value"trace-id = "dev-trace-123"# Production profile for Temporal Cloud[profile.prod]address = "your-namespace.a1b2c.tmprl.cloud:7233"namespace = "your-namespace"api_key = "your-api-key-here"# TLS configuration for production[profile.prod.tls]# TLS auto-enables when TLS config or an API key is present# disabled = falseclient_cert_path = "/etc/temporal/certs/client.pem"client_key_path = "/etc/temporal/certs/client.key"# Custom headers for production[profile.prod.grpc_meta]environment = "production"service-version = "v1.2.3" You can create a Temporal Client using a profile from the configuration file using the `ClientConfig.load_client_connect_options` function as follows. In this example, you load the `default` profile for local development: require 'temporalio/client'require 'temporalio/env_config'def main puts '--- Loading default profile from config.toml ---' # For this sample to be self-contained, we explicitly provide the path to # the config.toml file included in this directory. # By default though, the config.toml file will be loaded from # ~/.config/temporalio/temporal.toml (or the equivalent standard config directory on your OS). config_file = File.join(__dir__, 'config.toml') # load_client_connect_options is a helper that loads a profile and prepares # the configuration for Client.connect. By default, it loads the # "default" profile. args, kwargs = Temporalio::EnvConfig::ClientConfig.load_client_connect_options( config_source: Pathname.new(config_file) ) puts "Loaded 'default' profile from #{config_file}." puts " Address: #{args[0]}" puts " Namespace: #{args[1]}" puts " gRPC Metadata: #{kwargs[:rpc_metadata]}" puts "\nAttempting to connect to client..." begin client = Temporalio::Client.connect(*args, **kwargs) puts '✅ Client connected successfully!' sys_info = client.workflow_service.get_system_info(Temporalio::Api::WorkflowService::V1::GetSystemInfoRequest.new) puts "✅ Successfully verified connection to Temporal server!\n#{sys_info}" rescue StandardError => e puts "❌ Failed to connect: #{e}" endendmain if $PROGRAM_NAME == __FILE__ Use the `EnvConfig` package to set connection options for the Temporal Client using environment variables. For a list of all available environment variables and their default values, refer to [Environment Configuration](https://docs.temporal.io/references/client-environment-configuration) . For example, the following code snippet loads all environment variables and creates a Temporal Client with the options specified in those variables. If you have defined a configuration file at either the default location (`~/.config/temporalio/temporal.toml`) or a custom location specified by the `TEMPORAL_CONFIG_FILE` environment variable, this will also load the default profile in the configuration file. However, any options set via environment variables will take precedence. Set the following environment variables before running your application. Replace the placeholder values with your actual configuration. Since this is for a local development Temporal Service, the values connect to `localhost:7233` and the `default` Namespace. You may omit these variables entirely since they're the defaults. export TEMPORAL_NAMESPACE="default"export TEMPORAL_ADDRESS="localhost:7233" After setting the environment variables, you can create a Temporal Client as follows: require 'temporalio/client'require 'temporalio/env_config'def main # load_client_connect_options is a helper that loads a profile and prepares # the configuration for Client.connect. By default, it loads the # "default" profile and also reads from environment variables. The environment # variables take precedence over the config file. args, kwargs = Temporalio::EnvConfig::ClientConfig.load_client_connect_options() puts " Address: #{args[0]}" puts " Namespace: #{args[1]}" puts " gRPC Metadata: #{kwargs[:rpc_metadata]}" puts "\nAttempting to connect to client..." begin client = Temporalio::Client.connect(*args, **kwargs) puts '✅ Client connected successfully!' sys_info = client.workflow_service.get_system_info(Temporalio::Api::WorkflowService::V1::GetSystemInfoRequest.new) puts "✅ Successfully verified connection to Temporal server!\n#{sys_info}" rescue StandardError => e puts "❌ Failed to connect: #{e}" endendmain if $PROGRAM_NAME == __FILE__ If you don't want to use environment variables or a configuration file, you can specify connection options directly in code. This is convenient for local development and testing. You can also load a base configuration from environment variables or a configuration file, and then override specific options in code. Use the `connect` class method on the `Temporalio::Client` class to create and connect to a Temporal Client to the Temporal Service. client = Temporalio::Client.connect('localhost:7233', 'default') Connect to Temporal Cloud[​](https://docs.temporal.io/develop/ruby/client/temporal-client#connect-to-temporal-cloud "Direct link to Connect to Temporal Cloud") ---------------------------------------------------------------------------------------------------------------------------------------------------------------- You can connect to Temporal Cloud using either an [API key](https://docs.temporal.io/cloud/api-keys) or through mTLS. Connection to Temporal Cloud or any secured Temporal Service requires additional connection options compared to connecting to an unsecured local development instance: * Your credentials for authentication. * If you are using an API key, provide the API key value. * If you are using mTLS, provide the mTLS CA certificate and mTLS private key. * Your _Namespace and Account ID_ combination, which follows the format `.`. * The recommended _endpoint_ is the gRPC Namespace endpoint: `..tmprl.cloud:7233`. This endpoint works for all Namespaces and automatically directs traffic to the active region for Namespaces with [High Availability](https://docs.temporal.io/cloud/high-availability) . See [accessing Namespaces](https://docs.temporal.io/cloud/namespaces#access-namespaces) for more information on endpoint options. You can find the Namespace and Account ID, as well as the endpoint, on the Namespaces tab: ![The Namespace and Account ID combination on the left, and the regional endpoint on the right](https://docs.temporal.io/assets/images/namespaces-and-regional-endpoints-5d0328eb623fc5e3307226a01a5f35b1.png) For more information about managing and generating client certificates for Temporal Cloud, see [How to manage certificates in Temporal Cloud](https://docs.temporal.io/cloud/certificates) . You can provide these connection options using environment variables, a configuration file, or directly in code. * Configuration File * Environment Variables * Code You can use a TOML configuration file to set connection options for the Temporal Client. The configuration file lets you configure multiple profiles, each with its own set of connection options. You can then specify which profile to use when creating the Temporal Client. For a list of all available configuration options you can set in the TOML file, refer to [Environment Configuration](https://docs.temporal.io/references/client-environment-configuration) . You can use the environment variable `TEMPORAL_CONFIG_FILE` to specify the location of the TOML file or provide the path to the file directly in code. If you don't provide the path to the configuration file, the SDK looks for it at the default path `~/.config/temporalio/temporal.toml`. info The connection options set in configuration files have lower precedence than environment variables. This means that if you set the same option in both the configuration file and as an environment variable, the environment variable value overrides the option set in the configuration file. For example, the following TOML configuration file defines a `staging` profile with the necessary connection options to connect to Temporal Cloud via an API key: # Cloud profile for Temporal Cloud[profile.staging]address = "your-namespace.a1b2c.tmprl.cloud:7233"namespace = "your-namespace"api_key = "your-api-key-here" If you want to use mTLS authentication instead of an API key, replace the `api_key` field with your mTLS certificate and private key: # Cloud profile for Temporal Cloud[profile.staging]address = "your-namespace.a1b2c.tmprl.cloud:7233"namespace = "your-namespace"tls_client_cert_data = "your-tls-client-cert-data"tls_client_key_path = "your-tls-client-key-path" With the connections options defined in the configuration file, use the [`Client.connect` method](https://ruby.temporal.io/Temporalio/Client.html#connect-class_method) to create a Temporal Client using the `staging` profile as follows. After loading the profile, you can also programmatically override specific connection options before creating the client. require 'temporalio/client'require 'temporalio/env_config'def main puts "--- Loading 'staging' profile with programmatic overrides ---" config_file = File.join(__dir__, 'config.toml') profile_name = 'staging' puts "The 'staging' profile in config.toml has an incorrect address (localhost:9999)." puts "We'll programmatically override it to the correct address." # Load the 'staging' profile. args, kwargs = Temporalio::EnvConfig::ClientConfig.load_client_connect_options( profile: profile_name, config_source: Pathname.new(config_file) ) # Override the target host to the correct address. # This is the recommended way to override configuration values. args[0] = 'localhost:7233' puts "\nLoaded '#{profile_name}' profile from #{config_file} with overrides." puts " Address: #{args[0]} (overridden from localhost:9999)" puts " Namespace: #{args[1]}" puts "\nAttempting to connect to client..." begin client = Temporalio::Client.connect(*args, **kwargs) puts '✅ Client connected successfully!' sys_info = client.workflow_service.get_system_info(Temporalio::Api::WorkflowService::V1::GetSystemInfoRequest.new) puts "✅ Successfully verified connection to Temporal server!\n#{sys_info}" rescue StandardError => e puts "❌ Failed to connect: #{e}" endendmain if $PROGRAM_NAME == __FILE__ The following environment variables are required to connect to Temporal Cloud: * `TEMPORAL_NAMESPACE`: Your Namespace and Account ID combination in the format `.`. * `TEMPORAL_ADDRESS`: The gRPC endpoint for your Temporal Cloud Namespace. * `TEMPORAL_API_KEY`: Your API key value. Required if you are using API key authentication. * `TEMPORAL_TLS_CLIENT_CERT_DATA` or `TEMPORAL_TLS_CLIENT_CERT_PATH`: Your mTLS client certificate data or file path. Required if you are using mTLS authentication. * `TEMPORAL_TLS_CLIENT_KEY_DATA` or `TEMPORAL_TLS_CLIENT_KEY_PATH`: Your mTLS client private key data or file path. Required if you are using mTLS authentication. Ensure these environment variables exist in your environment before running your application. Require the `temporalio/env_config` module to set connection options for the Temporal Client using environment variables. The `Temporalio::EnvConfig::ClientConfig.load_client_connect_options` method will automatically load all environment variables. For a list of all available environment variables and their default values, refer to [Environment Configuration](https://docs.temporal.io/references/client-environment-configuration) . For example, the following code snippet loads all environment variables and creates a Temporal Client with the options specified in those variables. If you have defined a configuration file at either the default location (`~/.config/temporalio/temporal.toml`) or a custom location specified by the `TEMPORAL_CONFIG_FILE` environment variable, this will also load the default profile in the configuration file. However, any options set via environment variables will take precedence. After setting the environment variables, use the following code to create the Temporal Client: require 'temporalio/client'require 'temporalio/env_config'def main # load_client_connect_options is a helper that loads a profile and prepares # the configuration for Client.connect. By default, it loads the # "default" profile. It also reads from environment variables. The environment # variables take precedence over the config file. args, kwargs = Temporalio::EnvConfig::ClientConfig.load_client_connect_options() puts " Address: #{args[0]}" puts " Namespace: #{args[1]}" puts " gRPC Metadata: #{kwargs[:rpc_metadata]}" puts "\nAttempting to connect to client..." begin client = Temporalio::Client.connect(*args, **kwargs) puts '✅ Client connected successfully!' sys_info = client.workflow_service.get_system_info(Temporalio::Api::WorkflowService::V1::GetSystemInfoRequest.new) puts "✅ Successfully verified connection to Temporal server!\n#{sys_info}" rescue StandardError => e puts "❌ Failed to connect: #{e}" endendmain if $PROGRAM_NAME == __FILE__ You can also specify connection options directly in code to connect to Temporal Cloud. To create an initial connection, provide the endpoint, Namespace and Account ID combination, and API key values to the `Client.connect` method. client = Temporalio::Client.connect( '', # Endpoint '.', # Namespace api_key: '', tls: true) To connect using mTLS instead of an API key, provide the mTLS certificate and private key as follows: client = Temporalio::Client.connect( '', # Endpoint '.', # Namespace tls: Temporalio::Client::Connection::TLSOptions.new( client_cert: File.read('my-client-cert.pem'), client_private_key: File.read('my-client-key.pem') )) For more information about configuring TLS to secure inter- and intra-network communication for a Temporal Service, see [Temporal Customization Samples](https://github.com/temporalio/samples-server) . Start a Workflow[​](https://docs.temporal.io/develop/ruby/client/temporal-client#start-workflow "Direct link to Start a Workflow") ----------------------------------------------------------------------------------------------------------------------------------- To start a Workflow Execution, supply: * A Task Queue * A Workflow Type * Input arguments * Workflow options such as Workflow Id To start a Workflow Execution in Ruby, use either the `start_workflow` or `execute_workflow` methods in the Client. You must set a [Workflow Id](https://docs.temporal.io/workflow-execution/workflowid-runid#workflow-id) and [Task Queue](https://docs.temporal.io/task-queue) in the parameters given to the method. result = my_client.execute_workflow( MyWorkflow, 'some-input', id: 'my-workflow-id', task_queue: 'my-task-queue')puts "Result: #{result}" Get Workflow results[​](https://docs.temporal.io/develop/ruby/client/temporal-client#get-workflow-results "Direct link to Get Workflow results") ------------------------------------------------------------------------------------------------------------------------------------------------- Once a Workflow Execution is started, the Workflow Id and Run Id can be used to uniquely identify it. You can block until the result is available, or retrieve it later using the handle. You can also use Queries to access Workflow state and results while the Workflow is running. Use `start_workflow` or `workflow_handle` on the Client to return a Workflow handle. Then use the `result` method to await on the result of the Workflow. handle = my_client.workflow_handle('my-workflow-id')result = handle.resultputs "Result: #{result}" * [Connect to development Temporal Service](https://docs.temporal.io/develop/ruby/client/temporal-client#connect-to-development-service) * [Connect to Temporal Cloud](https://docs.temporal.io/develop/ruby/client/temporal-client#connect-to-temporal-cloud) * [Start a Workflow](https://docs.temporal.io/develop/ruby/client/temporal-client#start-workflow) * [Get Workflow results](https://docs.temporal.io/develop/ruby/client/temporal-client#get-workflow-results) --- # Workflow futures - Ruby SDK | Temporal Platform Documentation [Skip to main content](https://docs.temporal.io/develop/ruby/workflows/futures#__docusaurus_skipToContent_fallback) On this page Workflow Futures[​](https://docs.temporal.io/develop/ruby/workflows/futures#workflow-futures "Direct link to Workflow Futures") -------------------------------------------------------------------------------------------------------------------------------- `Temporalio::Workflow::Future` can be used for running things in the background or concurrently. Temporal provides Workflow-safe wrappers around some core language features in cases like these. `Temporalio::Workflow::Future` is a safe wrapper around `Fiber.schedule` for running multiple Activities at once. The Ruby SDK also provides `Workflow.wait_condition` for awaiting a result. Futures are never used implicitly, but they work with all Workflow code and constructs. For instance, to run 3 activities and wait for them all to complete, something like this can be written: # Start 3 activities in backgroundfut1 = Temporalio::Workflow::Future.new do Temporalio::Workflow.execute_activity(MyActivity1, schedule_to_close_timeout: 300)endfut2 = Temporalio::Workflow::Future.new do Temporalio::Workflow.execute_activity(MyActivity2, schedule_to_close_timeout: 300)endfut3 = Temporalio::Workflow::Future.new do Temporalio::Workflow.execute_activity(MyActivity3, schedule_to_close_timeout: 300)end# Wait for them all to completeTemporalio::Workflow::Future.all_of(fut1, fut2, fut3).waitTemporalio::Workflow.logger.info("Got: #{fut1.result}, #{fut2.result}, #{fut3.result}") Or, say, to wait on the first of 5 activities or a timeout to complete: # Start 5 activitiesact_futs = 5.times.map do |i| Temporalio::Workflow::Future.new do Temporalio::Workflow.execute_activity(MyActivity, "my-arg-#{i}", schedule_to_close_timeout: 300) endend# Start a timersleep_fut = Temporalio::Workflow::Future.new { Temporalio::Workflow.sleep(30) }# Wait for first act result or sleep futact_result = Temporalio::Workflow::Future.any_of(sleep_fut, *act_futs).wait# Fail if timer done firstraise Temporalio::Error::ApplicationError, 'Timer expired' if sleep_fut.done?# Print act result otherwiseTemporalio::Workflow.logger.info("Act result: #{act_result}") There are several other details not covered here about futures, such as how exceptions are handled, how to use a setter proc instead of a block, etc. See the [API documentation](https://ruby.temporal.io/Temporalio/Workflow/Future.html) for details. * [Workflow Futures](https://docs.temporal.io/develop/ruby/workflows/futures#workflow-futures) --- # Temporal Testing Suite - Temporal feature | Temporal Platform Documentation [Skip to main content](https://docs.temporal.io/evaluate/development-production-features/testing-suite#__docusaurus_skipToContent_fallback) In the context of Temporal, you can create these types of automated tests: 1. End-to-end: Running a Temporal Server and Worker with all its Workflows and Activities; starting and interacting with Workflows from a Client. 2. Integration: Anything between end-to-end and unit testing. Running Activities with mocked Context and other SDK imports (and usually network requests). Running Workers with mock Activities, and using a Client to start Workflows. Running Workflows with mocked SDK imports. 3. Unit: Running a piece of Workflow or Activity code and mocking any code it calls. Jump straight to a Temporal SDK feature guide. Related 📚 * [![](https://docs.temporal.io/img/sdks/svgs/golang.svg)Testing using the Go SDK](https://docs.temporal.io/develop/go/best-practices/testing-suite) feature-guide * [![](https://docs.temporal.io/img/sdks/svgs/java.svg)Testing using the Java SDK](https://docs.temporal.io/develop/java/best-practices/testing-suite) feature-guide * [![](https://docs.temporal.io/img/sdks/svgs/php.svg)Testing using the PHP SDK](https://docs.temporal.io/develop/php/best-practices/testing-suite) feature-guide * [![](https://docs.temporal.io/img/sdks/svgs/python.svg)Testing using the Python SDK](https://docs.temporal.io/develop/python/best-practices/testing-suite) feature-guide * [![](https://docs.temporal.io/img/sdks/svgs/typescript.svg)Testing using the TypeScript SDK](https://docs.temporal.io/develop/typescript/best-practices/testing-suite) feature-guide * [![](https://docs.temporal.io/img/sdks/svgs/dotnet.svg)Testing using the .NET SDK](https://docs.temporal.io/develop/dotnet/best-practices/testing-suite) feature-guide * [![](https://docs.temporal.io/img/sdks/svgs/ruby.svg)Testing using the Ruby SDK](https://docs.temporal.io/develop/ruby/best-practices/testing-suite) feature-guide --- # Nexus Patterns | Temporal Platform Documentation [Skip to main content](https://docs.temporal.io/nexus/patterns#__docusaurus_skipToContent_fallback) On this page SUPPORT, STABILITY, and DEPENDENCY INFO Temporal Nexus is now [Generally Available](https://docs.temporal.io/evaluate/development-production-features/release-stages#general-availability) for [Temporal Cloud](https://docs.temporal.io/cloud/nexus) and [self-hosted deployments](https://docs.temporal.io/production-deployment/self-hosted-guide/nexus) . There are two common patterns for building and deploying [Nexus Services](https://docs.temporal.io/nexus/services) : * **[Collocated pattern](https://docs.temporal.io/nexus/patterns#collocated-pattern) **: Runs on the same Workers as your existing Workflows and Activities. Use by default. * **[Router pattern](https://docs.temporal.io/nexus/patterns#router-queue-pattern) **: Separates Nexus routing from Workflow execution. A dedicated Nexus Worker on a “router” Task Queue routes Operations to Workflows on other Task Queues. Use when you need independent scaling, different IAM permissions per Worker fleet, or want to add Nexus to without modifying existing Workers. Collocated pattern (default)[​](https://docs.temporal.io/nexus/patterns#collocated-pattern "Direct link to Collocated pattern (default)") ------------------------------------------------------------------------------------------------------------------------------------------ The **collocated pattern** runs Nexus Operation handlers in the same Worker and on the same Task Queue as the underlying Workflows. This is the default and simplest deployment. The Nexus Endpoint targets the same Task Queue used by the underlying Workflows. A single Worker registers both Nexus Services and Workflow types, so everything runs together. ### Why start here[​](https://docs.temporal.io/nexus/patterns#why-start-here "Direct link to Why start here") * **Simplest setup:** One Worker, one Task Queue, one deployment. No extra infrastructure. * **Eager Workflow Start:** When the handler starts a Workflow in the same Worker, you can use [Eager Workflow Start](https://docs.temporal.io/develop/worker-performance#eager-workflow-start) to execute the first Workflow Task locally without an extra call to the Temporal Server - while still recording durable state. If the process crashes, the Workflow resumes on another Worker. * **Clean facade:** Operations act as a stable contract. You can change the underlying implementation (Signal today, Workflow tomorrow) without impacting callers. ### When to use this pattern[​](https://docs.temporal.io/nexus/patterns#when-to-use-this-pattern "Direct link to When to use this pattern") * Getting started with Nexus. * The same team owns both the Nexus Service and underlying Workflows. * You don't need to scale Nexus routing separately from Workflow execution. * You are setting up a simple test environment Use this pattern by default unless you have a good reason to use the Router-queue pattern below Router-queue pattern[​](https://docs.temporal.io/nexus/patterns#router-queue-pattern "Direct link to Router-queue pattern") ---------------------------------------------------------------------------------------------------------------------------- The **router-queue pattern** separates Nexus routing from Workflow execution. A dedicated Nexus Worker on a "router" Task Queue routes Operations to Workflows on other Task Queues in the same Namespace. ### When to use this pattern[​](https://docs.temporal.io/nexus/patterns#when-to-use-this-pattern-1 "Direct link to When to use this pattern") * **Separate scaling:** Scale Nexus routing independently from Workflow execution. * **Dedicated routing layer:** A single Nexus Worker routes requests to multiple Workflow types on different Task Queues. * **Different IAM permissions:** Worker fleets behind different Task Queues may have different IAM permissions to different underlying resources. * **Avoid modifying existing Workers:** Add a router Worker to a Namespace without changing any existing Workers or Workflows. ### How it works[​](https://docs.temporal.io/nexus/patterns#how-it-works "Direct link to How it works") 1. Register a Nexus Worker that polls a dedicated "router" Task Queue. 2. Configure the Nexus Endpoint's target Task Queue to point to this router Task Queue. 3. In each Nexus Operation handler, specify a different target Task Queue in the Workflow start options. 4. Existing Workers continue to poll their own Task Queues and execute the Workflows started by the router. ### Production usage[​](https://docs.temporal.io/nexus/patterns#production-usage "Direct link to Production usage") Used in production by organizations running self-service platforms where a central gateway routes requests to domain-specific Namespaces and Task Queues. The router Worker is lightweight - it only handles routing logic. * [Collocated pattern (default)](https://docs.temporal.io/nexus/patterns#collocated-pattern) * [Why start here](https://docs.temporal.io/nexus/patterns#why-start-here) * [When to use this pattern](https://docs.temporal.io/nexus/patterns#when-to-use-this-pattern) * [Router-queue pattern](https://docs.temporal.io/nexus/patterns#router-queue-pattern) * [When to use this pattern](https://docs.temporal.io/nexus/patterns#when-to-use-this-pattern-1) * [How it works](https://docs.temporal.io/nexus/patterns#how-it-works) * [Production usage](https://docs.temporal.io/nexus/patterns#production-usage) --- # Timers and Start Delays | Temporal Platform Documentation [Skip to main content](https://docs.temporal.io/workflow-execution/timers-delays#__docusaurus_skipToContent_fallback) On this page This page discusses [Timer](https://docs.temporal.io/workflow-execution/timers-delays#timer) and [Start Delay](https://docs.temporal.io/workflow-execution/timers-delays#delay-workflow-execution) . What is a Timer?[​](https://docs.temporal.io/workflow-execution/timers-delays#timer "Direct link to What is a Timer?") ----------------------------------------------------------------------------------------------------------------------- Temporal SDKs offer Timer APIs so that Workflow Executions are deterministic in their handling of time values. Timers in Temporal are persisted, meaning that even if your Worker or Temporal Service is down when the time period completes, as soon as your Worker and Temporal Service become available, the call that is awaiting the Timer in your Workflow code will resolve, causing execution to proceed. Timers are reliable and efficient. Workers consume no additional resources while waiting for a Timer to fire, so a single Worker can await millions of Timers concurrently. * [How to set Timers in Go](https://docs.temporal.io/develop/go/workflows/timers) * [How to set Timers in Java](https://docs.temporal.io/develop/java/workflows/timers) * [How to set Timers in PHP](https://docs.temporal.io/develop/php/workflows/timers) * [How to set Timers in Python](https://docs.temporal.io/develop/python/workflows/timers) * [How to set Timers in TypeScript](https://docs.temporal.io/develop/typescript/workflows/timers) * [How to set Timers in .NET](https://docs.temporal.io/develop/dotnet/workflows/timers) The duration of a Timer is fixed, and your Workflow might specify a value as short as one second or as long as several years. Although it's possible to specify an extremely precise duration, such as 36 milliseconds or 15.072 minutes, your Workflows should not rely on sub-second accuracy for Timers. We recommend that you consider the duration as a minimum time, one which will be rounded up slightly due to the latency involved with scheduling and firing the Timer. For example, setting a Timer for 11.97 seconds is guaranteed to delay execution for at least that long, but will likely be closer to 12 seconds in practice. What is a Start Delay?[​](https://docs.temporal.io/workflow-execution/timers-delays#delay-workflow-execution "Direct link to What is a Start Delay?") ------------------------------------------------------------------------------------------------------------------------------------------------------ COMPATIBILITY Start Delay Workflow Execution is incompatible with both [Schedules](https://docs.temporal.io/schedule) and [Cron Jobs](https://docs.temporal.io/cron-job) . Start Delay determines the amount of time to wait before initiating a Workflow Execution. This is useful if you have a Workflow you want to schedule out in the future, but only want it to execute once: in comparison to reoccurring Workflows using Schedules. If the Workflow receives a Signal-With-Start or Update-With-Start during the delay, it dispatches a Workflow Task and the remaining delay is bypassed. If the Workflow receives a Signal during the delay that is not a Signal-With-Start, the Signal does not interrupt the delay, and the Workflow continues to be delayed until the delay expires or a Signal-With-Start is received. You can delay the dispatch of the initial Workflow Execution by setting this option in the Workflow Options field of your chosen SDK. This delay only applies to the initial Workflow Execution and does not affect subsequent executions, such as when the Workflow Continues-as-New. * [What is a Timer?](https://docs.temporal.io/workflow-execution/timers-delays#timer) * [What is a Start Delay?](https://docs.temporal.io/workflow-execution/timers-delays#delay-workflow-execution) --- # Temporal Workflow Definition | Temporal Platform Documentation [Skip to main content](https://docs.temporal.io/workflow-definition#__docusaurus_skipToContent_fallback) On this page This page covers the following: * [What is a Workflow Definition?](https://docs.temporal.io/workflow-definition) * [Determinism and constraints](https://docs.temporal.io/workflow-definition#deterministic-constraints) * [Handling code changes and non-deterministic behavior](https://docs.temporal.io/workflow-definition#non-deterministic-change) * [Intrinsic non-determinism logic](https://docs.temporal.io/workflow-definition#intrinsic-nondeterministic-logic) * [Versioning Workflow code and Patching](https://docs.temporal.io/workflow-definition#workflow-versioning) * [Handling unreliable Worker Processes](https://docs.temporal.io/workflow-definition#unreliable-worker-processes) * [What is a Workflow Type?](https://docs.temporal.io/workflow-definition#workflow-type) A Temporal Workflow defines the overall flow of the application. Conceptually, a Workflow is a sequence of steps written in a general-purpose programming language. With Temporal, those steps are defined by writing code, known as a Workflow Definition, and are carried out by running that code, which results in a Workflow Execution. In day-to-day conversations, the term _Workflow_ might refer to [Workflow Type](https://docs.temporal.io/workflow-definition#workflow-type) , a [Workflow Definition](https://docs.temporal.io/workflow-definition) , or a [Workflow Execution](https://docs.temporal.io/workflow-execution) . Temporal documentation aims to be explicit and differentiate between them. What is a Workflow Definition?[​](https://docs.temporal.io/workflow-definition#workflow-definition "Direct link to What is a Workflow Definition?") ---------------------------------------------------------------------------------------------------------------------------------------------------- A Workflow Definition is the code that defines the Workflow. It is written with a programming language and corresponding Temporal SDK. Depending on the programming language, it's typically implemented as a function or an object method and encompasses the end-to-end series of steps of a Temporal application. Below are different ways to develop a basic Workflow Definition. * Go * Java * PHP * Python * Typescript * .NET **[Workflow Definition in Go](https://docs.temporal.io/develop/go/workflows/basics) ** func YourBasicWorkflow(ctx workflow.Context) error { // ... return nil} **[Workflow Definition in Java (Interface)](https://docs.temporal.io/develop/java/workflows/basics) ** // Workflow interface@WorkflowInterfacepublic interface YourBasicWorkflow { @WorkflowMethod String workflowMethod(Arguments args);} **[Workflow Definition in Java (Implementation)](https://docs.temporal.io/develop/java/workflows/basics) ** // Workflow implementationpublic class YourBasicWorkflowImpl implements YourBasicWorkflow { // ...} **[Workflow Definition in PHP (Interface)](https://docs.temporal.io/develop/php/workflows/basics) ** #[WorkflowInterface]interface YourBasicWorkflow { #[WorkflowMethod] public function workflowMethod(Arguments args);} **[Workflow Definition in PHP (Implementation)](https://docs.temporal.io/develop/php/workflows/basics) ** class YourBasicWorkflowImpl implements YourBasicWorkflow { // ...} **[Workflow Definition in Python](https://docs.temporal.io/develop/python/workflows/basics) ** @workflow.defnclass YourWorkflow: @workflow.run async def YourBasicWorkflow(self, input: str) -> str: # ... **[Workflow Definition in Typescript](https://docs.temporal.io/develop/typescript/workflows/basics) ** type BasicWorkflowArgs = { param: string;};export async function WorkflowExample( args: BasicWorkflowArgs,): Promise<{ result: string }> { // ...} **[Workflow Definition in C# and .NET](https://docs.temporal.io/develop/dotnet/workflows/basics) ** [Workflow]public class YourBasicWorkflow { [WorkflowRun] public async Task workflowExample(string param) { // ... }} A Workflow Definition may be also referred to as a Workflow Function. In Temporal's documentation, a Workflow Definition refers to the source for the instance of a Workflow Execution, while a Workflow Function refers to the source for the instance of a Workflow Function Execution. A Workflow Execution effectively executes once to completion, while a Workflow Function Execution occurs many times during the life of a Workflow Execution. We strongly recommend that you write a Workflow Definition in a language that has a corresponding Temporal SDK. ### Deterministic constraints[​](https://docs.temporal.io/workflow-definition#deterministic-constraints "Direct link to Deterministic constraints") A critical aspect of developing Workflow Definitions is ensuring that they are deterministic. Generally speaking, this means you must take care to ensure that any time your Workflow code is executed it makes the same Workflow API calls in the same sequence, given the same input. Some changes to those API calls are safe to make. For example, you can change: * The input parameters, return values, and execution timeouts of Child Workflows and Activities * However, it is not safe to change the types or IDs of Child Workflows or Activities * The input parameters used to Signal an external Workflow * The duration of Timers (although changing them to 0 is not safe in all SDKs) * Add or remove calls to Workflow APIs that don't produce [Commands](https://docs.temporal.io/workflow-execution#command) (For example - `workflow.GetInfo` in the Go SDK or its equivalent in other SDKs) The following Workflow API calls all can produce Commands, and thus must not be reordered, added, or removed without proper [Versioning techniques](https://docs.temporal.io/workflow-definition#workflow-versioning) : * Starting or cancelling a Timer * Scheduling or cancelling Activity Executions (including local Activities) * Starting or cancelling Child Workflow executions * Signalling or cancelling signals to external Workflow Executions * Scheduling or cancelling Nexus operations * Ending the Workflow Execution in any way (completing, failing, cancelling, or continuing-as-new) * `Patched` or `GetVersion` calls for Versioning (although they may be added or removed according to the [patching](https://docs.temporal.io/workflow-definition#workflow-patching) rules) * Upserting Workflow Search Attributes * Upserting Workflow Memos * Running a `SideEffect` or `MutableSideEffect` For a complete reference, see the [Command reference](https://docs.temporal.io/references/commands) . More formally, the use of certain Workflow APIs in the function is what generates Commands. Commands tell the Temporal Service which Events to create and add to the Workflow Execution's [Event History](https://docs.temporal.io/workflow-execution/event#event-history) . When the Workflow's code [replays](https://docs.temporal.io/workflow-execution#replay) , the Commands that are emitted are compared with the existing Event History. If a corresponding Event already exists within the Event History that matches that command, then the Execution progresses. See [Event History](https://docs.temporal.io/encyclopedia/event-history/) for a detailed walkthrough of the process. For example, using an SDK's "Execute Activity" API generates the [ScheduleActivityTask](https://docs.temporal.io/references/commands#scheduleactivitytask) Command. When this API is called upon re-execution, that Command is compared with the Event that is in the same location within the sequence. The Event in the sequence must be an [ActivityTaskScheduled](https://docs.temporal.io/references/events#activitytaskscheduled) Event, where the Activity name is the same as what is in the Command. If a generated Command doesn't match what it needs to in the existing Event History, then the Workflow Execution returns a _non-deterministic_ error. The following are the two reasons why a Command might be generated out of sequence or the wrong Command might be generated altogether: 1. Code changes are made to a Workflow Definition that is in use by a running Workflow Execution. 2. There is intrinsic non-deterministic logic (such as inline random branching). ### Code changes can cause non-deterministic behavior[​](https://docs.temporal.io/workflow-definition#non-deterministic-change "Direct link to Code changes can cause non-deterministic behavior") The Workflow Definition can change in very limited ways once there is a Workflow Execution depending on it. To alleviate non-deterministic issues that arise from code changes, we recommend using [Workflow Versioning](https://docs.temporal.io/workflow-definition#workflow-versioning) . For example, let's say we have a Workflow Definition that defines the following sequence: 1. Start and wait on a Timer/sleep. 2. Spawn and wait on an Activity Execution. 3. Complete. We start a Worker and spawn a Workflow Execution that uses that Workflow Definition. The Worker would emit the [StartTimer](https://docs.temporal.io/references/commands#starttimer) Command and the Workflow Execution would become suspended. Before the Timer is up, we change the Workflow Definition to the following sequence: 1. Spawn and wait on an Activity Execution. 2. Start and wait on a Timer/sleep. 3. Complete. When the Timer fires, the next Workflow Task will cause the Workflow Function to re-execute. The first Command the Worker sees would be ScheduleActivityTask Command, which wouldn't match up to the expected [TimerStarted](https://docs.temporal.io/references/events#timerstarted) Event. The Workflow Execution would fail and return a nondeterminism error. The following are examples of minor changes that would not result in non-determinism errors when re-executing a History which already contain the Events: * Changing the duration of a Timer, with the following exceptions: * In Java, Python, and Go, changing a Timer's duration from or to 0 is a non-deterministic behavior. * In .NET, changing a Timer's duration from or to -1 (which means "infinite") is a non-deterministic behavior. * Changing the arguments to: * The Activity Options in a call to spawn an Activity Execution (local or nonlocal). * The Child Workflow Options in a call to spawn a Child Workflow Execution. * Call to Signal an External Workflow Execution. * Adding a Signal Handler for a Signal Type that has not been sent to this Workflow Execution. ### Intrinsic non-deterministic logic[​](https://docs.temporal.io/workflow-definition#intrinsic-nondeterministic-logic "Direct link to Intrinsic non-deterministic logic") Intrinsic non-determinism is when a Workflow Function Execution might emit a different sequence of Commands on re-execution, regardless of whether all the input parameters are the same. For example, a Workflow Definition can not have inline logic that branches (emits a different Command sequence) based off a local time setting or a random number. In the representative pseudocode below, the `local_clock()` function returns the local time, rather than Temporal-defined time: fn your_workflow() { if local_clock().is_before("12pm") { await workflow.sleep(duration_until("12pm")) } else { await your_afternoon_activity() }} Each Temporal SDK offers APIs that enable Workflow Definitions to have logic that gets and uses time, random numbers, and data from unreliable resources. When those APIs are used, the results are stored as part of the Event History, which means that a re-executed Workflow Function will issue the same sequence of Commands, even if there is branching involved. In other words, all operations that do not purely mutate the Workflow Execution's state should occur through a Temporal SDK API. For SDK-specific replay-safe APIs and examples (logging, random numbers, time, replay detection), see: * [Go: Develop Workflow logic](https://docs.temporal.io/develop/go/workflows/basics#workflow-logic-requirements) * [Java: Workflow logic requirements](https://docs.temporal.io/develop/java/workflows/basics#workflow-logic-requirements) * [Python: Develop Workflow logic](https://docs.temporal.io/develop/python/workflows/basics#workflow-logic-requirements) * [TypeScript: Develop Workflow logic](https://docs.temporal.io/develop/typescript/workflows/basics#workflow-logic-requirements) * [.NET: Workflow logic requirements](https://docs.temporal.io/develop/dotnet/workflows/basics#workflow-logic-requirements) * [Ruby: Workflow Logic Requirements](https://docs.temporal.io/develop/ruby/workflows/basics#workflow-logic-requirements) ### Versioning Workflows[​](https://docs.temporal.io/workflow-definition#workflow-versioning "Direct link to Versioning Workflows") The Temporal Platform requires that Workflow code (Workflow Definitions) be deterministic in nature. This requirement means that developers should consider how they plan to handle changes to Workflow code over time. A versioning strategy is even more important if your Workflow Executions live long enough to run on multiple versions of your Worker. Temporal Platform provides Workflow Versioning APIs. Temporal offers two Versioning strategies: * [Worker Versioning](https://docs.temporal.io/workflow-definition#worker-versioning) : keep Workers tied to specific code revisions, so that old Workers can run old code paths and new Workers can run new code paths. note Support for the experimental method of Worker Versioning prior to 2025 will be removed from Temporal Server in March 2026. Refer to the [latest Worker Versioning docs](https://docs.temporal.io/worker-versioning) for guidance. * [Versioning with patching](https://docs.temporal.io/workflow-definition#workflow-patching) : make sure your code changes are compatible across versions of your Workflow. You can use either strategy, or a combination. #### Worker Versioning[​](https://docs.temporal.io/workflow-definition#worker-versioning "Direct link to Worker Versioning") This is the **recommended** way to handle versioning and users see improved error rates when adopting it. To learn more about Worker Versioning, see our [Worker Versioning in production](https://docs.temporal.io/production-deployment/worker-deployments/worker-versioning) page. #### Versioning with Patching[​](https://docs.temporal.io/workflow-definition#workflow-patching "Direct link to Versioning with Patching") When keeping Workflows compatible, you should patch and ideally how to test your running Workflows will be safe to run on a new code version. To patch: * [How to patch Workflow code in Go](https://docs.temporal.io/develop/go/workflows/versioning#patching) * [How to patch Workflow code in Java](https://docs.temporal.io/develop/java/workflows/versioning#patching) * [How to patch Workflow code in Python](https://docs.temporal.io/develop/python/workflows/versioning#patching) * [How to patch Workflow code in PHP](https://docs.temporal.io/develop/php/workflows/versioning#php-sdk-patching-api) * [How to patch Workflow code in TypeScript](https://docs.temporal.io/develop/typescript/workflows/versioning#patching) * [How to patch Workflow code in .NET](https://docs.temporal.io/develop/dotnet/workflows/versioning#patching) To test, see [Safe Deployments](https://docs.temporal.io/develop/safe-deployments) . ### Handling unreliable Worker Processes[​](https://docs.temporal.io/workflow-definition#unreliable-worker-processes "Direct link to Handling unreliable Worker Processes") You do not handle Worker Process failure or restarts in a Workflow Definition. Workflow Function Executions are completely oblivious to the Worker Process in terms of failures or downtime. The Temporal Platform ensures that the state of a Workflow Execution is recovered and progress resumes if there is an outage of either Worker Processes or the Temporal Service itself. The only reason a Workflow Execution might fail is due to the code throwing an error or exception, not because of underlying infrastructure outages. ### What is a Workflow Type?[​](https://docs.temporal.io/workflow-definition#workflow-type "Direct link to What is a Workflow Type?") A Workflow Type is a name that maps to a Workflow Definition. * A single Workflow Type can be instantiated as multiple Workflow Executions. * A Workflow Type is scoped by a Task Queue. It is acceptable to have the same Workflow Type name map to different Workflow Definitions if they are using completely different Workers. ![Workflow Type cardinality with Workflow Definitions and Workflow Executions](https://docs.temporal.io/diagrams/workflow-type-cardinality.svg) Workflow Type cardinality with Workflow Definitions and Workflow Executions * [What is a Workflow Definition?](https://docs.temporal.io/workflow-definition#workflow-definition) * [Deterministic constraints](https://docs.temporal.io/workflow-definition#deterministic-constraints) * [Code changes can cause non-deterministic behavior](https://docs.temporal.io/workflow-definition#non-deterministic-change) * [Intrinsic non-deterministic logic](https://docs.temporal.io/workflow-definition#intrinsic-nondeterministic-logic) * [Versioning Workflows](https://docs.temporal.io/workflow-definition#workflow-versioning) * [Worker Versioning](https://docs.temporal.io/workflow-definition#worker-versioning) * [Versioning with Patching](https://docs.temporal.io/workflow-definition#workflow-patching) * [Handling unreliable Worker Processes](https://docs.temporal.io/workflow-definition#unreliable-worker-processes) * [What is a Workflow Type?](https://docs.temporal.io/workflow-definition#workflow-type) --- # Workflow basics - Ruby SDK | Temporal Platform Documentation [Skip to main content](https://docs.temporal.io/develop/ruby/workflows/basics#__docusaurus_skipToContent_fallback) On this page Develop a Workflow[​](https://docs.temporal.io/develop/ruby/workflows/basics#develop-workflow "Direct link to Develop a Workflow") ----------------------------------------------------------------------------------------------------------------------------------- Workflows are the fundamental unit of a Temporal Application, and it all starts with the development of a [Workflow Definition](https://docs.temporal.io/workflow-definition) . In the Temporal Ruby SDK programming model, Workflows are defined as classes. Have the Workflow class extend `Temporalio::Workflow::Definition` to define a Workflow. The entrypoint is the `execute` method. class MyWorkflow < Temporalio::Workflow::Definition def execute(name) Temporalio::Workflow.execute_activity( MyActivity, { greeting: 'Hello', name: }, start_to_close_timeout: 100 ) endend Temporal Workflows may have any number of custom parameters. However, we strongly recommend that hashes or objects are used as parameters, so that the object's individual fields may be altered without breaking the signature of the Workflow. ### Workflow Logic Requirements[​](https://docs.temporal.io/develop/ruby/workflows/basics#workflow-logic-requirements "Direct link to Workflow Logic Requirements") Temporal Workflows [must be deterministic](https://docs.temporal.io/workflows#deterministic-constraints) , which includes Ruby workflows. This means there are several things workflows cannot do such as: * Perform IO (network, disk, stdio, etc) * Access/alter external mutable state * Do any threading * Do anything using the system clock (e.g. `Time.Now`) * Make any random calls * Make any not-guaranteed-deterministic calls To prevent illegal workflow calls, a call tracer is put on the workflow thread that raises an exception if any illegal calls are made. Which calls are illegal is configurable in the worker options. ### Customize Workflow Type[​](https://docs.temporal.io/develop/ruby/workflows/basics#workflow-type "Direct link to Customize Workflow Type") Workflows have a Type that are referred to as the Workflow name. The following examples demonstrate how to set a custom name for your Workflow Type. You can customize the Workflow name with a custom name in a `workflow_name` class method call on the class. The Workflow name defaults to the unqualified class name. class MyWorkflow < Temporalio::Workflow::Definition # Customize the name workflow_name :MyDifferentWorkflowName def execute(name) Temporalio::Workflow.execute_activity( MyActivity, { greeting: 'Hello', name: }, start_to_close_timeout: 100 ) endend * [Develop a Workflow](https://docs.temporal.io/develop/ruby/workflows/basics#develop-workflow) * [Workflow Logic Requirements](https://docs.temporal.io/develop/ruby/workflows/basics#workflow-logic-requirements) * [Customize Workflow Type](https://docs.temporal.io/develop/ruby/workflows/basics#workflow-type) --- # Child Workflows - Temporal feature | Temporal Platform Documentation [Skip to main content](https://docs.temporal.io/evaluate/development-production-features/throughput-composability#__docusaurus_skipToContent_fallback) In Temporal, **Child Workflows** enable applications to achieve another level of composability when it comes to throughput. The following example scenarios are a few reasons to use this feature: * To create a separate service that can be invoked from multiple other services or applications. * To partition a step into smaller chunks. * To manage a dedicated resource and guarantee uniqueness. * To execute logic periodically without overwhelming the parent business process. See the SDK feature guides for implementation details: Related 📚 * [![](https://docs.temporal.io/img/sdks/svgs/golang.svg)Go SDK Child Workflow feature guide](https://docs.temporal.io/develop/go/workflows/child-workflows) feature-guide * [![](https://docs.temporal.io/img/sdks/svgs/java.svg)Java SDK Child Workflow feature guide](https://docs.temporal.io/develop/java/workflows/child-workflows) feature-guide * [![](https://docs.temporal.io/img/sdks/svgs/php.svg)PHP SDK Child Workflow feature guide](https://docs.temporal.io/develop/php/workflows/child-workflows) feature-guide * [![](https://docs.temporal.io/img/sdks/svgs/python.svg)Python SDK Child Workflow feature guide](https://docs.temporal.io/develop/python/workflows/child-workflows) feature-guide * [![](https://docs.temporal.io/img/sdks/svgs/typescript.svg)TypeScript SDK Child Workflow feature guide](https://docs.temporal.io/develop/typescript/workflows/child-workflows) feature-guide * [![](https://docs.temporal.io/img/sdks/svgs/dotnet.svg).NET SDK Child Workflow feature guide](https://docs.temporal.io/develop/dotnet/workflows/child-workflows) feature-guide * [![](https://docs.temporal.io/img/sdks/svgs/ruby.svg)Ruby SDK Child Workflow feature guide](https://docs.temporal.io/develop/ruby/workflows/child-workflows) feature-guide For a deep dive into Child Workflows see the [Child Workflows Encyclopedia page](https://docs.temporal.io/child-workflows) . --- # Client - TypeScript SDK | Temporal Platform Documentation [Skip to main content](https://docs.temporal.io/develop/typescript/platform#__docusaurus_skipToContent_fallback) On this page ![TypeScript SDK Banner](https://docs.temporal.io/assets/images/banner-typescript-temporal-d8a24070726a0d14cb4d1aab011db927.png) Platform[​](https://docs.temporal.io/develop/typescript/platform#platform "Direct link to Platform") ----------------------------------------------------------------------------------------------------- * [Observability](https://docs.temporal.io/develop/typescript/platform/observability) * [Enriching the UI](https://docs.temporal.io/develop/typescript/platform/enriching-ui) * [Platform](https://docs.temporal.io/develop/typescript/platform#platform) --- # Workers - TypeScript SDK | Temporal Platform Documentation [Skip to main content](https://docs.temporal.io/develop/typescript/workers#__docusaurus_skipToContent_fallback) On this page ![TypeScript SDK Banner](https://docs.temporal.io/assets/images/banner-typescript-temporal-d8a24070726a0d14cb4d1aab011db927.png) Workers[​](https://docs.temporal.io/develop/typescript/workers#workers "Direct link to Workers") ------------------------------------------------------------------------------------------------- * [Worker processes](https://docs.temporal.io/develop/typescript/workers/run-worker-process) * [Interceptors](https://docs.temporal.io/develop/typescript/workers/interceptors) * [Workers](https://docs.temporal.io/develop/typescript/workers#workers) --- # Cancellation - Ruby SDK | Temporal Platform Documentation [Skip to main content](https://docs.temporal.io/develop/ruby/workflows/cancellation#__docusaurus_skipToContent_fallback) On this page This page shows how to interrupt a Workflow Execution. You can interrupt a Workflow Execution in one of the following ways: * [Cancel](https://docs.temporal.io/develop/ruby/workflows/cancellation#cancellation) : Canceling a Workflow provides a graceful way to stop Workflow Execution. * [Terminate](https://docs.temporal.io/develop/ruby/workflows/cancellation#termination) : Terminating a Workflow forcefully stops Workflow Execution. Terminating a Workflow forcefully stops Workflow Execution. This action resembles killing a process. * The system records a `WorkflowExecutionTerminated` event in the Workflow History. * The termination forcefully and immediately stops the Workflow Execution. * The Workflow code gets no chance to handle termination. * A Workflow Task doesn't get scheduled. In most cases, canceling is preferable because it allows the Workflow to finish gracefully. Terminate only if the Workflow is stuck and cannot be canceled normally. Cancellation[​](https://docs.temporal.io/develop/ruby/workflows/cancellation#cancellation "Direct link to Cancellation") ------------------------------------------------------------------------------------------------------------------------- To give a Workflow and its Activities the ability to be cancelled, do the following: * Handle a Cancellation request within a Workflow. * Set Activity Heartbeat Timeouts. * Listen for and handle a Cancellation request within an Activity. * Send a Cancellation request from a Temporal Client. Handle Cancellation in Workflow[​](https://docs.temporal.io/develop/ruby/workflows/cancellation#handle-cancellation-in-workflow "Direct link to Handle Cancellation in Workflow") ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Workflow Definitions can be written to respond to cancellation requests. It is common for an Activity to be run on Cancellation to perform cleanup. Cancellation Requests on Workflows cancel the `Temporalio::Workflow.cancellation` which is a `Temporalio::Cancellation` that effectively serves as a cancellation token. This is the cancellation that is implicitly used for all calls within the workflow as well (e.g. Timers, Activities, etc) and therefore cancellation is propagated to them to be handled and bubble out. class MyWorkflow < Temporalio::Workflow::Definition def execute # Whether this workflow waits on the activity to handle the cancellation or not is # dependent upon the cancellation_type parameter. We leave the default here which # sends the cancellation but does not wait on it to be handled. Temporalio::Workflow.execute_activity(MyActivity, start_to_close_timeout: 100) rescue Temporalio::Error => e # For this sample, we only want to execute cleanup when it's a cancellation raise unless Temporalio::Error.canceled?(e) # Call a cleanup activity. We have to do this with a new/detached cancellation # because the default workflow-level one is already canceled at this point. Temporalio::Workflow.execute_activity( MyCleanupActivity, start_to_close_timeout: 100, cancellation: Temporalio::Cancellation.new ) # Re-raise the original exception raise endend Handle Cancellation in an Activity[​](https://docs.temporal.io/develop/ruby/workflows/cancellation#handle-cancellation-in-an-activity "Direct link to Handle Cancellation in an Activity") ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Ensure that the Activity is [Heartbeating](https://docs.temporal.io/develop/ruby/activities/timeouts#activity-heartbeats) to receive the Cancellation request and stop execution. Also make sure that the [Heartbeat Timeout](https://docs.temporal.io/develop/ruby/activities/timeouts#heartbeat-timeout) is set on the Activity Options when calling from the Workflow. An Activity Cancellation Request raises a `Temporalio::Error::CanceledError` in the Activity. class MyActivity < Temporalio::Activity::Definition def execute # This is a naive loop simulating work, but similar heartbeat/cancellation logic # applies to other scenarios as well loop do # Send heartbeat Temporalio::Activity::Context.current.heartbeat # Sleep before heartbeating again sleep(3) end rescue Temporalio::Error::CanceledError raise 'Canceled!' endend Request Cancellation[​](https://docs.temporal.io/develop/ruby/workflows/cancellation#request-cancellation "Direct link to Request Cancellation") ------------------------------------------------------------------------------------------------------------------------------------------------- Use `cancel` on the `WorkflowHandle` to cancel a Workflow Execution. # Get a workflow handle by its workflow ID. This could be made specific to a run by# passing run ID. This could also just be a handle that is returned from# start_workflow instead.handle = my_client.workflow_handle('my-workflow-id')# Send cancellation. This returns when cancellation is received by the server. Wait on# the handle's result to wait for cancellation to be applied.handle.cancel By default, Activities are automatically cancelled when the Workflow is cancelled since the workflow cancellation is used by activities by default. To issue a cancellation explicitly, a new cancellation token can be created. class MyWorkflow < Temporalio::Workflow::Definition def execute # Create a new cancellation linked to the workflow one, so that it inherits # cancellation that comes from the workflow. Users can choose to make it # completely detached by not providing a parent. cancellation, cancel_proc = Temporalio::Cancellation.new( Temporalio::Workflow.cancellation ) # Start the activity in the background. Whether this workflow waits on the activity # to handle the cancellation or not is dependent upon the cancellation_type # parameter. We leave the default here which sends the cancellation but does not wait # on it to be handled. future = Temporalio::Future.new do Temporalio::Workflow.execute_activity( MyActivity, start_to_close_timeout: 100, cancellation: ) end # Wait 5 minutes, then cancel it Temporalio::Workflow.sleep(5 * 60) cancel_proc.call # Wait on the activity which will raise an activity error with a cause of # cancellation which will fail the workflow future.wait endend Termination[​](https://docs.temporal.io/develop/ruby/workflows/cancellation#termination "Direct link to Termination") ---------------------------------------------------------------------------------------------------------------------- To Terminate a Workflow Execution in Ruby, use the `terminate` method on the Workflow handle. # Get a workflow handle by its workflow ID. This could be made specific to a run by# passing run ID. This could also just be a handle that is returned from# start_workflow instead.handle = my_client.workflow_handle('my-workflow-id')# Terminatehandle.terminate Workflow Executions can also be Terminated directly from the WebUI. In this case, a custom note can be logged from the UI when that happens. Reset a Workflow Execution[​](https://docs.temporal.io/develop/ruby/workflows/cancellation#reset "Direct link to Reset a Workflow Execution") ---------------------------------------------------------------------------------------------------------------------------------------------- Resetting a Workflow Execution terminates the current Workflow Execution and starts a new Workflow Execution from a point you specify in its Event History. Use reset when a Workflow is blocked due to a non-deterministic error or other issues that prevent it from completing. When you reset a Workflow, the Event History up to the reset point is copied to the new Workflow Execution, and the Workflow resumes from that point with the current code. Reset only works if you've fixed the underlying issue, such as removing non-deterministic code. Any progress made after the reset point will be discarded. Provide a reason when resetting, as it will be recorded in the Event History. * Web UI * Temporal CLI 1. Navigate to the Workflow Execution details page, 2. Click the **Reset** button in the top right dropdown menu, 3. Select the Event ID to reset to, 4. Provide a reason for the reset, 5. Confirm the reset. The Web UI shows available reset points and creates a link to the new Workflow Execution after the reset completes. Use the `temporal workflow reset` command to reset a Workflow Execution: temporal workflow reset \ --workflow-id \ --event-id \ --reason "Reason for reset" For example: temporal workflow reset \ --workflow-id my-background-check \ --event-id 4 \ --reason "Fixed non-deterministic code" By default, the command resets the latest Workflow Execution in the `default` Namespace. Use `--run-id` to reset a specific run. Use `--namespace` to specify a different Namespace: temporal workflow reset \ --workflow-id my-background-check \ --event-id 4 \ --reason "Fixed non-deterministic code" \ --namespace my-namespace \ --tls-cert-path /path/to/cert.pem \ --tls-key-path /path/to/key.pem Monitor the new Workflow Execution after resetting to ensure it completes successfully. * [Cancellation](https://docs.temporal.io/develop/ruby/workflows/cancellation#cancellation) * [Handle Cancellation in Workflow](https://docs.temporal.io/develop/ruby/workflows/cancellation#handle-cancellation-in-workflow) * [Handle Cancellation in an Activity](https://docs.temporal.io/develop/ruby/workflows/cancellation#handle-cancellation-in-an-activity) * [Request Cancellation](https://docs.temporal.io/develop/ruby/workflows/cancellation#request-cancellation) * [Termination](https://docs.temporal.io/develop/ruby/workflows/cancellation#termination) * [Reset a Workflow Execution](https://docs.temporal.io/develop/ruby/workflows/cancellation#reset) --- # Nexus Endpoints | Temporal Platform Documentation [Skip to main content](https://docs.temporal.io/nexus/endpoints#__docusaurus_skipToContent_fallback) On this page SUPPORT, STABILITY, and DEPENDENCY INFO Temporal Nexus is now [Generally Available](https://docs.temporal.io/evaluate/development-production-features/release-stages#general-availability) for [Temporal Cloud](https://docs.temporal.io/cloud/nexus) and [self-hosted deployments](https://docs.temporal.io/production-deployment/self-hosted-guide/nexus) . A [Nexus Endpoint](https://docs.temporal.io/glossary#nexus-endpoint) is a fully managed reverse proxy for [Nexus Services](https://docs.temporal.io/nexus/services) . It routes requests from a caller Workflow to a target Namespace and Task Queue. Callers only need to know the Endpoint name - the target Namespace, Task Queue, and internal implementation are encapsulated. Workers handle Nexus requests by registering one or more Services and polling the Endpoint's target Task Queue. Multiple Endpoints can target different Task Queues in the same Namespace. The Endpoint description field supports markdown for documenting available Operations, contact information, or schema links. Reverse proxy for Nexus Services, not a general purpose proxy[​](https://docs.temporal.io/nexus/endpoints#reverse-proxy-for-nexus-services-not-a-general-purpose-proxy "Direct link to Reverse proxy for Nexus Services, not a general purpose proxy") ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- A Nexus Endpoint acts as a reverse proxy for a single Nexus Service, routing requests to one target Namespace and Task Queue. Unlike general-purpose proxies, it does not route to multiple backends. Instead, it provides a secure, managed connection to a specific upstream target, which can be in any region or cloud. The [EndpointSpec](https://github.com/temporalio/api/blob/2a5b3951e71565e28628edea1b3d88d69ed26607/temporal/api/nexus/v1/message.proto#L170) support the following [target type](https://github.com/temporalio/api/blob/2a5b3951e71565e28628edea1b3d88d69ed26607/temporal/api/nexus/v1/message.proto#L185) : * **Worker**: Route to a target Namespace and Task Queue. Deploying a Nexus Endpoint[​](https://docs.temporal.io/nexus/endpoints#deploying-a-nexus-endpoint "Direct link to Deploying a Nexus Endpoint") ----------------------------------------------------------------------------------------------------------------------------------------------- Adding an Endpoint to the [Nexus Registry](https://docs.temporal.io/nexus/registry) deploys it immediately. The Endpoint is available at runtime as soon as it's registered. * [Reverse proxy for Nexus Services, not a general purpose proxy](https://docs.temporal.io/nexus/endpoints#reverse-proxy-for-nexus-services-not-a-general-purpose-proxy) * [Deploying a Nexus Endpoint](https://docs.temporal.io/nexus/endpoints#deploying-a-nexus-endpoint) --- # Nexus Metrics | Temporal Platform Documentation [Skip to main content](https://docs.temporal.io/nexus/metrics#__docusaurus_skipToContent_fallback) On this page SUPPORT, STABILITY, and DEPENDENCY INFO Temporal Nexus is now [Generally Available](https://docs.temporal.io/evaluate/development-production-features/release-stages#general-availability) for [Temporal Cloud](https://docs.temporal.io/cloud/nexus) and [self-hosted deployments](https://docs.temporal.io/production-deployment/self-hosted-guide/nexus) . Nexus provides SDK metrics, Cloud metrics, and OSS Cluster metrics in addition to integrated [execution debugging](https://docs.temporal.io/nexus/execution-debugging) . SDK Metrics[​](https://docs.temporal.io/nexus/metrics#sdk-metrics "Direct link to SDK Metrics") ------------------------------------------------------------------------------------------------ [SDK metrics](https://docs.temporal.io/references/sdk-metrics) are emitted from a Nexus Worker, including: * [nexus\_poll\_no\_task](https://docs.temporal.io/references/sdk-metrics#nexus_poll_no_task) * [nexus\_task\_schedule\_to\_start\_latency](https://docs.temporal.io/references/sdk-metrics#nexus_task_schedule_to_start_latency) * [nexus\_task\_execution\_failed Worker](https://docs.temporal.io/references/sdk-metrics#nexus_task_execution_failed) * [nexus\_task\_execution\_latency](https://docs.temporal.io/references/sdk-metrics#nexus_task_execution_latency) * [nexus\_task\_endtoend\_latency](https://docs.temporal.io/references/sdk-metrics#nexus_task_endtoend_latency) Cloud Metrics[​](https://docs.temporal.io/nexus/metrics#cloud-metrics "Direct link to Cloud Metrics") ------------------------------------------------------------------------------------------------------ [Cloud metrics](https://docs.temporal.io/cloud/metrics/reference) are emitted by Temporal Cloud, including: * Caller Namespace * RespondWorkflowTaskCompleted - schedule a Nexus Operation. * Handler Namespace * PollNexusTaskQueue - get a [Nexus Task](https://docs.temporal.io/tasks#nexus-task) to process, for example to start a Nexus Operation. * RespondNexusTaskCompleted - report the Nexus Task was successful. * RespondNexusTaskFailed - report the Nexus Task failed. OSS Cluster Metrics[​](https://docs.temporal.io/nexus/metrics#oss-cluster-metrics "Direct link to OSS Cluster Metrics") ------------------------------------------------------------------------------------------------------------------------ [Cluster metrics](https://docs.temporal.io/references/cluster-metrics#nexus-metrics) are emitted from an OSS Cluster, including: * History Service metrics * Concurrency Limiter metrics * Frontend Service metrics * [SDK Metrics](https://docs.temporal.io/nexus/metrics#sdk-metrics) * [Cloud Metrics](https://docs.temporal.io/nexus/metrics#cloud-metrics) * [OSS Cluster Metrics](https://docs.temporal.io/nexus/metrics#oss-cluster-metrics) --- # Event History | Temporal Platform Documentation [Skip to main content](https://docs.temporal.io/encyclopedia/event-history#__docusaurus_skipToContent_fallback) With Temporal, your Workflows can seamlessly recover from crashes. This is made possible by the [Event History](https://docs.temporal.io/workflow-execution/event) , a complete and durable log of everything that has happened in the lifecycle of a Workflow Execution, as well as the ability of the Temporal Service to durably persist the Events during Replay. Temporal uses the Event History to record every step taken along the way. Each time your Workflow Definition makes an API call to execute an Activity or start a Timer for instance, it doesn’t perform the action directly. Instead, it sends a Command to the Temporal Service. A Command is a requested action issued by a Worker to the Temporal Service after a Workflow Task Execution completes. The Temporal Service will act on these Commands such as scheduling an Activity or scheduling a timer. These Commands are then mapped to Events which are persisted in case of failure. For example, if the Worker crashes, the Worker uses the Event History to replay the code and recreate the state of the Workflow Execution to what it was immediately before the crash. It then resumes progress from the point of failure as if the failure never occurred. For a deep dive on how the Event History works, refer to the walkthroughs in the dropdown. * [Go](https://docs.temporal.io/encyclopedia/event-history/event-history-go) * [Java](https://docs.temporal.io/encyclopedia/event-history/event-history-java) * [Python](https://docs.temporal.io/encyclopedia/event-history/event-history-python) * [Typescript](https://docs.temporal.io/encyclopedia/event-history/event-history-typescript) * [.NET](https://docs.temporal.io/encyclopedia/event-history/event-history-dotnet) --- # Timers - Ruby SDK | Temporal Platform Documentation [Skip to main content](https://docs.temporal.io/develop/ruby/workflows/timers#__docusaurus_skipToContent_fallback) This page describes how to set a Durable Timer using the Temporal Ruby SDK. A [Durable Timer](https://docs.temporal.io/workflow-execution/timers-delays) is used to pause the execution of a Workflow for a specified duration. A Workflow can sleep for days or even months. Timers are persisted, so even if your Worker or Temporal Service is down when the time period completes, as soon as your Worker and Temporal Service are back up, the Durable Timer call will resolve and your code will continue executing. Sleeping is a resource-light operation: it does not tie up the process, and you can run millions of Timers off a single Worker. To add a Timer in a Workflow, use `Temporalio::Workflow.sleep`. _Technically_ `Kernel#sleep` works, but the workflow form allows one to set a summary to view in the UI. # Sleep for 72 hoursTemporalio::Workflow.sleep(72 * 60 * 60, summary: 'my timer') There is also a `Temporalio::Workflow.timeout` method that accepts a block and works like standard Ruby `Timeout.timeout` if needing the ability to timeout a set of code. --- # Testing - TypeScript SDK | Temporal Platform Documentation [Skip to main content](https://docs.temporal.io/develop/typescript/best-practices/testing-suite#__docusaurus_skipToContent_fallback) On this page The Testing section of the Temporal Application development guide describes the frameworks that facilitate Workflow and integration testing. In the context of Temporal, you can create these types of automated tests: * **End-to-end:** Running a Temporal Server and Worker with all its Workflows and Activities; starting and interacting with Workflows from a Client. * **Integration:** Anything between end-to-end and unit testing. * Running Activities with mocked Context and other SDK imports (and usually network requests). * Running Workers with mock Activities, and using a Client to start Workflows. * Running Workflows with mocked SDK imports. * **Unit:** Running a piece of Workflow or Activity code (a function or method) and mocking any code it calls. We generally recommend writing the majority of your tests as integration tests. Because the test server supports skipping time, use the test server for both end-to-end and integration tests with Workers. Test frameworks[​](https://docs.temporal.io/develop/typescript/best-practices/testing-suite#test-frameworks "Direct link to Test frameworks") ---------------------------------------------------------------------------------------------------------------------------------------------- Some SDKs have support or examples for popular test frameworks, runners, or libraries. TypeScript has sample tests for [Jest](https://jestjs.io/) and [Mocha](https://mochajs.org/) . **Jest** * Minimum Jest version: `27.0.0` * [Sample test file](https://github.com/temporalio/samples-typescript/blob/main/activities-examples/src/workflows.test.ts) * [`jest.config.js`](https://github.com/temporalio/samples-typescript/blob/main/activities-examples/jest.config.js) (must use [`testEnvironment: 'node'`](https://jestjs.io/docs/configuration#testenvironment-string) ; `testEnvironment: 'jsdom'` is not supported) **Mocha** * [Sample test file](https://github.com/temporalio/samples-typescript/blob/main/activities-examples/src/mocha/workflows.test.ts) * Test coverage library: [`@temporalio/nyc-test-coverage`](https://github.com/temporalio/sdk-typescript/tree/main/packages/nyc-test-coverage) Testing Activities[​](https://docs.temporal.io/develop/typescript/best-practices/testing-suite#test-activities "Direct link to Testing Activities") ---------------------------------------------------------------------------------------------------------------------------------------------------- An Activity can be tested with a mock Activity environment, which provides a way to mock the Activity context, listen to Heartbeats, and cancel the Activity. This behavior allows you to test the Activity in isolation by calling it directly, without needing to create a Worker to run the Activity. ### Run an Activity[​](https://docs.temporal.io/develop/typescript/best-practices/testing-suite#run-an-activity "Direct link to Run an Activity") If an Activity references its context, you need to mock that context when testing in isolation. First, create a [`MockActivityEnvironment`](https://typescript.temporal.io/api/classes/testing.MockActivityEnvironment) . The constructor accepts an optional partial Activity [`Info`](https://typescript.temporal.io/api/interfaces/activity.Info) object in case any info fields are needed for the test. Then use [`MockActivityEnvironment.run()`](https://typescript.temporal.io/api/classes/testing.MockActivityEnvironment#run) to run a function in an Activity [Context](https://typescript.temporal.io/api/classes/activity.Context) . import { activityInfo } from '@temporalio/activity';import { MockActivityEnvironment } from '@temporalio/testing';import assert from 'assert';// A function that takes two numbers and returns a promise that resolves to the sum of the two numbers// and the current attempt.async function activityFoo(a: number, b: number): Promise { return a + b + activityInfo().attempt;}// Create a MockActivityEnvironment with attempt set to 2. Run the activityFoo// function with parameters 5 and 35. Assert that the result is 42.const env = new MockActivityEnvironment({ attempt: 2 });const result = await env.run(activityFoo, 5, 35);assert.equal(result, 42); ### Listen to Heartbeats[​](https://docs.temporal.io/develop/typescript/best-practices/testing-suite#listen-to-heartbeats "Direct link to Listen to Heartbeats") When an Activity sends a Heartbeat, be sure that you can see the Heartbeats in your test code so that you can verify them. [`MockActivityEnvironment`](https://typescript.temporal.io/api/classes/testing.MockActivityEnvironment) is an [`EventEmitter`](https://nodejs.org/api/events.html#class-eventemitter) that emits a `heartbeat` event that you can use to listen for Heartbeats emitted by the Activity. When an Activity is run by a Worker, Heartbeats are throttled to avoid overloading the server. `MockActivityEnvironment`, however, does not throttle Heartbeats. import { heartbeat } from '@temporalio/activity';import assert from 'assert';async function activityFoo(): Promise { heartbeat(6);}const env = new MockActivityEnvironment();env.on('heartbeat', (d: unknown) => { assert(d === 6);});await env.run(activityFoo); ### Cancel an Activity[​](https://docs.temporal.io/develop/typescript/best-practices/testing-suite#cancel-an-activity "Direct link to Cancel an Activity") If an Activity is supposed to react to a Cancellation, you can test whether it reacts correctly by canceling it. [`MockActivityEnvironment`](https://typescript.temporal.io/api/classes/testing.MockActivityEnvironment) exposes a [`.cancel()`](https://typescript.temporal.io/api/classes/testing.MockActivityEnvironment#cancel) method that cancels the Activity Context. import { CancelledFailure, heatbeat, sleep } from '@temporalio/activity';import { MockActivityEnvironment } from '@temporalio/testing';import assert from 'assert';async function activityFoo(): Promise { heartbeat(6); // @temporalio/activity's sleep() is Cancellation-aware, which means that on Cancellation, // CancelledFailure will be thrown from it. await sleep(100);}const env = new MockActivityEnvironment();env.on('heartbeat', (d: unknown) => { assert(d === 6);});await assert.rejects(env.run(activityFoo), (err) => { assert.ok(err instanceof CancelledFailure);}); Testing Workflows[​](https://docs.temporal.io/develop/typescript/best-practices/testing-suite#test-workflows "Direct link to Testing Workflows") ------------------------------------------------------------------------------------------------------------------------------------------------- ### How to mock Activities[​](https://docs.temporal.io/develop/typescript/best-practices/testing-suite#mock-activities "Direct link to How to mock Activities") Mock the Activity invocation when unit testing your Workflows. When integration testing Workflows with a Worker, you can mock Activities by providing mock Activity implementations to the Worker. Implement only the relevant Activities for the Workflow being tested. import type * as activities from './activities';// Creating a mock object of the activities.const mockActivities: Partial = { makeHTTPRequest: async () => '99',};// Creating a worker with the mocked activities.const worker = await Worker.create({ activities: mockActivities, // ...}); ### How to skip time[​](https://docs.temporal.io/develop/typescript/best-practices/testing-suite#skip-time "Direct link to How to skip time") Some long-running Workflows can persist for months or even years. Implementing the test framework allows your Workflow code to skip time and complete your tests in seconds rather than the Workflow's specified amount. For example, if you have a Workflow sleep for a day, or have an Activity failure with a long retry interval, you don't need to wait the entire length of the sleep period to test whether the sleep function works. Instead, test the logic that happens after the sleep by skipping forward in time and complete your tests in a timely manner. The test framework included in most SDKs is an in-memory implementation of Temporal Server that supports skipping time. Time is a global property of an instance of `TestWorkflowEnvironment`: skipping time (either automatically or manually) applies to all currently running tests. If you need different time behaviors for different tests, run your tests in a series or with separate instances of the test server. For example, you could run all tests with automatic time skipping in parallel, and then all tests with manual time skipping in series, and then all tests without time skipping in parallel. #### Set up time skipping[​](https://docs.temporal.io/develop/typescript/best-practices/testing-suite#setting-up "Direct link to Set up time skipping") Set up the time-skipping test framework in the SDK of your choice. npm install @temporalio/testing The `@temporalio/testing` package downloads the test server and exports [`TestWorkflowEnvironment`](https://typescript.temporal.io/api/classes/testing.TestWorkflowEnvironment) , which you use to connect the Client and Worker to the test server and interact with the test server. [`TestWorkflowEnvironment.createTimeSkipping`](https://typescript.temporal.io/api/classes/testing.TestWorkflowEnvironment#createtimeskipping) starts the test server. A typical test suite should set up a single instance of the test environment to be reused in all tests (for example, in a [Jest](https://jestjs.io/) `beforeAll` hook or a [Mocha](https://mochajs.org/) `before()` hook). import { TestWorkflowEnvironment } from '@temporalio/testing';let testEnv: TestWorkflowEnvironment;// beforeAll and afterAll are injected by JestbeforeAll(async () => { testEnv = await TestWorkflowEnvironment.createTimeSkipping();});afterAll(async () => { await testEnv?.teardown();}); `TestWorkflowEnvironment` has [`client`](https://typescript.temporal.io/api/classes/testing.TestWorkflowEnvironment#client) and [`nativeConnection`](https://typescript.temporal.io/api/classes/testing.TestWorkflowEnvironment#nativeconnection) for creating Workers: import { Worker } from '@temporalio/worker';import { v4 as uuid4 } from 'uuid';import { workflowFoo } from './workflows';test('workflowFoo', async () => { const worker = await Worker.create({ connection: testEnv.nativeConnection, taskQueue: 'test', ... }); const result = await worker.runUntil( testEnv.client.workflow.execute(workflowFoo, { workflowId: uuid4(), taskQueue: 'test', }) ); expect(result).toEqual('foo');}); This test uses the test connection to create a Worker, runs the Worker until the Workflow is complete, and then makes an assertion about the Workflow's result. The Workflow is executed using `testEnv.client.workflow`, which is connected to the test server. #### Skip time automatically[​](https://docs.temporal.io/develop/typescript/best-practices/testing-suite#automatic-method "Direct link to Skip time automatically") You can skip time automatically in the SDK of your choice. Start a test server process that skips time as needed. For example, in the time-skipping mode, Timers, which include sleeps and conditional timeouts, are fast-forwarded except when Activities are running. The test server starts in "normal" time. When you use `TestWorkflowEnvironment.client.workflow.execute()` or `.result()`, the test server switches to "skipped" time mode until the Workflow completes. In "skipped" mode, timers (`sleep()` calls and `condition()` timeouts) are fast-forwarded except when Activities are running. `workflows.ts` import { sleep } from '@temporalio/workflow';export async function sleeperWorkflow() { await sleep('1 day');} `test.ts` import { sleeperWorkflow } from './workflows';test('sleep completes almost immediately', async () => { const worker = await Worker.create({ connection: testEnv.nativeConnection, taskQueue: 'test', workflowsPath: require.resolve('./workflows'), }); // Does not wait an entire day await worker.runUntil( testEnv.client.workflow.execute(sleeperWorkflow, { workflowId: uuid(), taskQueue: 'test', }), );}); #### Skip time manually[​](https://docs.temporal.io/develop/typescript/best-practices/testing-suite#manual-method "Direct link to Skip time manually") Skip time manually in the SDK of your choice. You can call `testEnv.sleep()` from your test code to advance the test server's time. This is useful for testing intermediate states or indefinitely long-running Workflows. However, to use `testEnv.sleep()`, you need to avoid automatic time skipping by starting the Workflow with `.start()` instead of `.execute()` (and not calling `.result()`). `workflow.ts` import { sleep } from '@temporalio/workflow';import { defineQuery, setHandler } from '@temporalio/workflow';export const daysQuery = defineQuery('days');export async function sleeperWorkflow() { let numDays = 0; setHandler(daysQuery, () => numDays); for (let i = 0; i < 100; i++) { await sleep('1 day'); numDays++; }} `test.ts` test('sleeperWorkflow counts days correctly', async () => { const worker = await Worker.create({ connection: testEnv.nativeConnection, taskQueue: 'test', workflowsPath: require.resolve('./workflows'), }); // `start()` starts the test server in "normal" mode, not skipped time mode. // If you don't advance time using `testEnv.sleep()`, then `sleeperWorkflow()` // will run for days. handle = await testEnv.client.workflow.start(sleeperWorkflow, { workflowId: uuid4(), taskQueue, }); worker.run(); let numDays = await handle.query(daysQuery); assert.equal(numDays, 0); // Advance the test server's time by 25 hours await testEnv.sleep('25 hours'); numDays = await handle.query(daysQuery); assert.equal(numDays, 1); await testEnv.sleep('25 hours'); numDays = await handle.query(daysQuery); assert.equal(numDays, 2);}); #### Skip time in Activities[​](https://docs.temporal.io/develop/typescript/best-practices/testing-suite#skip-time-in-activities "Direct link to Skip time in Activities") Skip time in Activities in the SDK of your choice. Call [`TestWorkflowEnvironment.sleep`](https://typescript.temporal.io/api/classes/testing.TestWorkflowEnvironment#sleep) from the mock Activity. In the following test, `processOrderWorkflow` sends a notification to the user after one day. The `processOrder` mocked Activity calls `testEnv.sleep(‘2 days')`, during which the Workflow sends email (by calling the `sendNotificationEmail` Activity). Then, after the Workflow completes, we assert that `sendNotificationEmail` was called. Workflow implementation [timer-examples/src/workflows.ts](https://github.com/temporalio/samples-typescript/blob/main/timer-examples/src/workflows.ts) export async function processOrderWorkflow({ orderProcessingMS, sendDelayedEmailTimeoutMS,}: ProcessOrderOptions): Promise { let processing = true; // Dynamically define the timeout based on given input const { processOrder } = proxyActivities>({ startToCloseTimeout: orderProcessingMS, }); const processOrderPromise = processOrder().then(() => { processing = false; }); await Promise.race([processOrderPromise, sleep(sendDelayedEmailTimeoutMS)]); if (processing) { await sendNotificationEmail(); await processOrderPromise; } return 'Order completed!';} [timer-examples/src/test/workflows.test.ts](https://github.com/temporalio/samples-typescript/blob/main/timer-examples/src/test/workflows.test.ts) it('sends reminder email if processOrder does not complete in time', async () => { // This test doesn't actually take days to complete: the TestWorkflowEnvironment starts the // Test Server, which automatically skips time when there are no running Activities. let emailSent = false; const mockActivities: ReturnType = { async processOrder() { // Test server switches to "normal" time while an Activity is executing. // Call `env.sleep` to skip ahead 2 days, by which time sendNotificationEmail // should have been called. await env.sleep('2 days'); }, async sendNotificationEmail() { emailSent = true; }, }; const worker = await Worker.create({ connection: env.nativeConnection, taskQueue: 'test', workflowsPath: require.resolve('../workflows'), activities: mockActivities, }); await worker.runUntil( env.client.workflow.execute(processOrderWorkflow, { workflowId: uuid(), taskQueue: 'test', args: [{ orderProcessingMS: ms('3 days'), sendDelayedEmailTimeoutMS: ms('1 day') }], }), ); assert.ok(emailSent);}); ### Test functions in Workflow context[​](https://docs.temporal.io/develop/typescript/best-practices/testing-suite#workflow-context "Direct link to Test functions in Workflow context") For a function or method to run in the Workflow context (where it's possible to get the current Workflow info, or running inside the sandbox in the case of TypeScript or Python), it needs to be run by the Worker as if it were a Workflow. note This section is applicable in Python and TypeScript. In Python, we allow testing of Workflows only and not generic Workflow-related code. To test a function in your Workflow code that isn't a Workflow, put the file it's exported from in [WorkerOptions.workflowsPath](https://typescript.temporal.io/api/interfaces/worker.WorkerOptions#workflowspath) . Then execute the function as if it were a Workflow: `workflows/file-with-workflow-function-to-test.ts` import { sleep } from '@temporalio/workflow';export async function functionToTest(): Promise { await sleep('1 day'); return 42;} `test.ts` const worker = await Worker.create({ connection: testEnv.nativeConnection, workflowsPath: require.resolve( './workflows/file-with-workflow-function-to-test', ),});const result = await worker.runUntil( testEnv.client.workflow.execute(functionToTest, workflowOptions),);assert.equal(result, 42); If `functionToTest` starts a Child Workflow, that Workflow must be exported from the same file (so that the Worker knows about it): import { sleep } from '@temporalio/workflow';import { someWorkflowToRunAsChild } from './some-workflow';export { someWorkflowToRunAsChild };export async function functionToTest(): Promise { const result = await wf.executeChild(someWorkflowToRunAsChild); return result + 42;} ### Assert in Workflow[​](https://docs.temporal.io/develop/typescript/best-practices/testing-suite#assert-in-workflow "Direct link to Assert in Workflow") The `assert` statement is a convenient way to insert debugging assertions into the Workflow context. The `assert` method is available in Python and TypeScript. The Node.js [`assert`](https://nodejs.org/api/assert.html) module is included in Workflow bundles. By default, a failed `assert` statement throws `AssertionError`, which causes a [Workflow Task](https://docs.temporal.io/tasks#workflow-task) to fail and be indefinitely retried. To prevent this behavior, use [`workflowInterceptorModules`](https://typescript.temporal.io/api/namespaces/testing/#workflowinterceptormodules) from `@temporalio/testing`. These interceptors catch an `AssertionError` and turn it into an `ApplicationFailure` that fails the entire Workflow Execution (not just the Workflow Task). `workflows/file-with-workflow-function-to-test.ts` import assert from 'assert';export async function functionToTest() { assert.ok(false);} `test.ts` import { TestWorkflowEnvironment, workflowInterceptorModules,} from '@temporalio/testing';const worker = await Worker.create({ connection: testEnv.nativeConnection, interceptors: { workflowModules: workflowInterceptorModules, }, workflowsPath: require.resolve( './workflows/file-with-workflow-function-to-test', ),});await worker.runUntil( testEnv.client.workflow.execute(functionToTest, workflowOptions), // throws WorkflowFailedError); How to Replay a Workflow Execution[​](https://docs.temporal.io/develop/typescript/best-practices/testing-suite#replay "Direct link to How to Replay a Workflow Execution") --------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Replay recreates the exact state of a Workflow Execution. You can replay a Workflow from the beginning of its Event History. Replay succeeds only if the [Workflow Definition](https://docs.temporal.io/workflow-definition) is compatible with the provided history from a deterministic point of view. When you test changes to your Workflow Definitions, we recommend doing the following as part of your CI checks: 1. Determine which Workflow Types or Task Queues (or both) will be targeted by the Worker code under test. 2. Download the Event Histories of a representative set of recent open and closed Workflows from each Task Queue, either programmatically using the SDK client or via the Temporal CLI. 3. Run the Event Histories through replay. 4. Fail CI if any error is encountered during replay. The following are examples of fetching and replaying Event Histories: To replay a single Event History, use [worker.runReplayHistory](https://typescript.temporal.io/api/classes/worker.Worker#runreplayhistory) . When an Event History is replayed and non-determinism is detected (that is, the Workflow code is incompatible with the History), [DeterminismViolationError](https://typescript.temporal.io/api/classes/workflow.DeterminismViolationError) is thrown. If replay fails for any other reason, [ReplayError](https://typescript.temporal.io/api/classes/worker.ReplayError) is thrown. In the following example, a single Event History is loaded from a JSON file on disk (as obtained from the [Web UI](https://docs.temporal.io/web-ui) or the [Temporal CLI](https://docs.temporal.io/cli/workflow#show) ): const filePath = './history_file.json';const history = await JSON.parse(fs.promises.readFile(filePath, 'utf8'));await Worker.runReplayHistory( { workflowsPath: require.resolve('./your/workflows'), }, history,); Alternatively, we can download the Event History programmatically using a Client: const connection = await Connection.connect({ address });const client = new Client({ connection, namespace: 'your-namespace' });const handle = client.workflow.getHandle('your-workflow-id');const history = await handle.fetchHistory();await Worker.runReplayHistory( { workflowsPath: require.resolve('./your/workflows'), }, history,); To gain confidence that changes to a Workflow are safe to deploy, we recommend that you obtain Event Histories from the relevant Task Queue and replay them in bulk. You can do so by combining the [Client.workflow.list()](https://typescript.temporal.io/api/classes/client.WorkflowClient#list) and [worker.runReplayHistories()](https://typescript.temporal.io/api/classes/worker.Worker#runreplayhistories) APIs. In the following example (which, as of server 1.18, requires [Advanced Visibility](https://docs.temporal.io/visibility#advanced-visibility) to be enabled), Event Histories are downloaded from the server and then replayed by passing in a client and a set of Workflows Executions. The [results](https://typescript.temporal.io/api/interfaces/worker.ReplayResult) returned by the async iterator contain information about the Workflow Execution and whether an error occurred during replay. const executions = client.workflow.list({ query: 'TaskQueue=foo and StartTime > "2022-01-01T12:00:00"',});const histories = executions.intoHistories();const results = Worker.runReplayHistories( { workflowsPath: require.resolve('./your/workflows'), }, histories,);for await (const result of results) { if (result.error) { console.error('Replay failed', result); }} * [Test frameworks](https://docs.temporal.io/develop/typescript/best-practices/testing-suite#test-frameworks) * [Testing Activities](https://docs.temporal.io/develop/typescript/best-practices/testing-suite#test-activities) * [Run an Activity](https://docs.temporal.io/develop/typescript/best-practices/testing-suite#run-an-activity) * [Listen to Heartbeats](https://docs.temporal.io/develop/typescript/best-practices/testing-suite#listen-to-heartbeats) * [Cancel an Activity](https://docs.temporal.io/develop/typescript/best-practices/testing-suite#cancel-an-activity) * [Testing Workflows](https://docs.temporal.io/develop/typescript/best-practices/testing-suite#test-workflows) * [How to mock Activities](https://docs.temporal.io/develop/typescript/best-practices/testing-suite#mock-activities) * [How to skip time](https://docs.temporal.io/develop/typescript/best-practices/testing-suite#skip-time) * [Set up time skipping](https://docs.temporal.io/develop/typescript/best-practices/testing-suite#setting-up) * [Skip time automatically](https://docs.temporal.io/develop/typescript/best-practices/testing-suite#automatic-method) * [Skip time manually](https://docs.temporal.io/develop/typescript/best-practices/testing-suite#manual-method) * [Skip time in Activities](https://docs.temporal.io/develop/typescript/best-practices/testing-suite#skip-time-in-activities) * [Test functions in Workflow context](https://docs.temporal.io/develop/typescript/best-practices/testing-suite#workflow-context) * [Assert in Workflow](https://docs.temporal.io/develop/typescript/best-practices/testing-suite#assert-in-workflow) * [How to Replay a Workflow Execution](https://docs.temporal.io/develop/typescript/best-practices/testing-suite#replay) ---